Revision tags: v1.2, v1.1, v1.0 |
|
#
5f7821ba |
| 11-Aug-2015 |
Chiyoung Seo <chiyoung.seo@gmail.com> |
MB-16038 Support the lazy cleanup of a file's buffer cache blocks If a given file is compacted ans no longer accessed, then its cache blocks in the buffer cache should be cleaned up and
MB-16038 Support the lazy cleanup of a file's buffer cache blocks If a given file is compacted ans no longer accessed, then its cache blocks in the buffer cache should be cleaned up and moved to the free list. As this operation can takes up to several seconds depending on the number of cache blocks belonging to the file, it is better to support the lazy cleanup of these cache blocks to avoid the long blocking period in the application. Change-Id: I3d607f17e189b63c592298c822672b9178356b0a
show more ...
|
#
d6eb040e |
| 05-Aug-2015 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-15908 Support lazy deletion of old DB file - To avoid API blocking by high latency of file removal, we support lazy deletion of old DB file after compaction. - Once the ref_c
MB-15908 Support lazy deletion of old DB file - To avoid API blocking by high latency of file removal, we support lazy deletion of old DB file after compaction. - Once the ref_count of an old file becomes zero, then it is registered to the file list of background compaction daemon, and then the daemon compactor threads periodically check and remove the pending files. Change-Id: Ie0705a681a05df8ccd68e87ba87576cb81f92156
show more ...
|
#
0d423664 |
| 02-Jul-2015 |
Sundar Sridharan <sundar.sridharan@gmail.com> |
fix garbage value to thread_create in compactor init num_threads else an arbitary large value of threads can be created. Change-Id: I75bc7648757b03ab324961f4de7125b836a8dca8
|
#
7da24760 |
| 13-Jun-2015 |
Chiyoung Seo <chiyoung.seo@gmail.com> |
MB-15278 Fix to potential race issues in daemon compactor and file instance The daemon compactor refers to the list of file instances that are currently opened. However, as a file instan
MB-15278 Fix to potential race issues in daemon compactor and file instance The daemon compactor refers to the list of file instances that are currently opened. However, as a file instance is destroyed when its reference counter becomes zero, the daemon compactor should identify that case and free the corresponding task entry from its queue. Change-Id: I502cd381ca0ad96f9610413800d8c555fff57bdd
show more ...
|
#
c40231de |
| 11-Jun-2015 |
Chiyoung Seo <chiyoung.seo@gmail.com> |
MB-15278 Add the multi-threaded auto compaction support. Previously, there was only one daemon compaction thread. This can cause out-of-disk space issues if many database files are opene
MB-15278 Add the multi-threaded auto compaction support. Previously, there was only one daemon compaction thread. This can cause out-of-disk space issues if many database files are opened and accessed concurrently. This change improves the auto daemon compaction to have multiple threads perform compacting individual database files concurrently. The default number of compactor threads are set to 4, but it can be configurable upon starting the ForestDB engine. Change-Id: I70515595e4171588506cfca49523e10448190f8a
show more ...
|
#
200ec8f5 |
| 27-Apr-2015 |
Chiyoung Seo <chiyoung.seo@gmail.com> |
MB-14060 Scan the file to catch up the remaining blocks in fdb_compact_upto API. This change avoids traversing the main index for each commit header written after the compaction marker,
MB-14060 Scan the file to catch up the remaining blocks in fdb_compact_upto API. This change avoids traversing the main index for each commit header written after the compaction marker, but instead scans the file to copy the remaining data blocks and commit headers to the new file. Change-Id: Ibbd2aea70eecf2510a14174a9e70e8cf2baecd80
show more ...
|
#
85444218 |
| 04-Apr-2015 |
Chiyoung Seo <chiyoung.seo@gmail.com> |
Add more loggings to erroneous cases in daemon compactor. Change-Id: Id387fc8d59e1f669385dede322166e8185377e4b
|
#
5aa14abc |
| 11-Apr-2015 |
Sundar Sridharan <sundar.sridharan@gmail.com> |
Compiler warnings and minor bug fixes Change-Id: I753118f2f79760d43b421e63aca512449da89574
|
#
89434c6e |
| 14-Mar-2015 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-13830 Do not register compacted file in compaction daemon again - Files whose status is COMPACT_OLD or REMOVED_PENDING should not be registered in compaction daemon, since they do not
MB-13830 Do not register compacted file in compaction daemon again - Files whose status is COMPACT_OLD or REMOVED_PENDING should not be registered in compaction daemon, since they do not need to be compacted again, and will be removed soon. Change-Id: I46c5be4c601dc52ac3ecbdcd5d14542b49431a87
show more ...
|
#
0e995e68 |
| 04-Mar-2015 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
Reconstruct fullpath for DB file existence check Change-Id: I1bee07346f1869016855b7680929cfd26eacdde1
|
#
02adb116 |
| 13-Feb-2015 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
Fix a bug that DB with custom cmp cannot be auto-compacted - fdb_open_for_compactor() failed because no cmp_function is passed. - Now compaction daemon keeps list of cmp_functions f
Fix a bug that DB with custom cmp cannot be auto-compacted - fdb_open_for_compactor() failed because no cmp_function is passed. - Now compaction daemon keeps list of cmp_functions for each DB file. Change-Id: I532c385213e606339e0546187db5a01f7a5580ff
show more ...
|
#
5b75d849 |
| 24-Jan-2015 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-13186 Maintain a separate DB handle for an iterator - The current iterator handle keeps pointers to the original handle's resources such as filemgr, trie, dhandle and bhandle. If comp
MB-13186 Maintain a separate DB handle for an iterator - The current iterator handle keeps pointers to the original handle's resources such as filemgr, trie, dhandle and bhandle. If compaction is performed after creating iterator, the original handle's resources are modified or recreated so that pointers in the iterator handle becomes invalid. - To avoid this issue, iterator handle should maintain its own DB handle which is not influenced by the original handle's change. Change-Id: If9bf479cbff42dda683209ff933d85733d4493c6
show more ...
|
#
f80ddf4d |
| 22-Jan-2015 |
Chiyoung Seo <chiyoung.seo@gmail.com> |
Make internal functions static if their scopes are not global. Change-Id: Ic390a59e76b99c86c29a3fefc76a0975becdccd3
|
#
2e49b909 |
| 19-Jan-2015 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-13012 Meta file stores DB file name excluding directory path - If the file path given by user includes directory path, the compactor gets rid of the directory path and stores its file
MB-13012 Meta file stores DB file name excluding directory path - If the file path given by user includes directory path, the compactor gets rid of the directory path and stores its file name part only. Change-Id: I0610a1403fc767cfac3667a692e04da33ce63354
show more ...
|
#
84f94aa1 |
| 16-Dec-2014 |
Sundar Sridharan <sundar.sridharan@gmail.com> |
MB-12613: clang analyzer fixes for null dereference, dead store etc Change-Id: I71baac9cb49cca7473289dedfbc1c00f08fbc728
|
#
64515de4 |
| 01-Dec-2014 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-12799 Do not use existing file name as a new compacted file name Change-Id: I502ea9202031727c7617361200163d2cee83fc67
|
#
4dc52e2d |
| 12-Nov-2014 |
Trond Norbye <trond.norbye@gmail.com> |
Remove unused variables (-Wunused-variable) Change-Id: I9f63f438848d61de3ca85b4e88a900c3ae5e6ba6
|
#
f54fbdb6 |
| 10-Nov-2014 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-12591 Provide a doc count for both file and KV store levels - Now doc count and space estimation are provided for each KV store separately. - Such statistics are globally managed
MB-12591 Provide a doc count for both file and KV store levels - Now doc count and space estimation are provided for each KV store separately. - Such statistics are globally managed so that the legacy 'ndocs' and 'datasize' attributes in the fdb_kvs_handle are removed. - fdb_get_kvs_info returns the number of docs belonging to the KV store, while fdb_get_file_info returns the total number of docs in the file.
show more ...
|
#
d32401da |
| 05-Sep-2014 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-11071 Support multiple KV instances - Add three new main operations: open (create), close, remove of KV instance in a DB file handle. - All other operations (get, set, rollba
MB-11071 Support multiple KV instances - Add three new main operations: open (create), close, remove of KV instance in a DB file handle. - All other operations (get, set, rollback, snapshot, etc.) are shared with the original APIs. - Same as snapshot, KV instance uses the fdb_handle structure, but commit and compaction can be invoked only using the DB file handle. Change-Id: Iad571745c57053d46130be584c5213e98baa9b9f
show more ...
|
#
47ec8bfc |
| 27-Oct-2014 |
Sundar Sridharan <sundar.sridharan@gmail.com> |
MB-12191: Support for fdb_destroy() Will wipe out all remains of a file in case of auto compaction and will do best-effort removal of files in case of manual compaction If any fi
MB-12191: Support for fdb_destroy() Will wipe out all remains of a file in case of auto compaction and will do best-effort removal of files in case of manual compaction If any file is open, the api will error out and not delete any file Reason for best-effort in manual compaction case: FileA --> FileB --> FileC --> FileA --> FileD --> FileC -->DESTROY (In above case, FileB cannot be destroyed as its info is not reachable from FileC) Change-Id: Iab6a1b77d6cc9c4a38247b15374ef4b2eaf2d48d
show more ...
|
#
fa8408c3 |
| 24-Oct-2014 |
Sundar Sridharan <sundar.sridharan@gmail.com> |
Fix memory leak in compactor.cc opendir needs a corresponding closedir Change-Id: I5b101d4cd1f04e84365b5e08d1eee04bd8b91fe5
|
#
fef67a48 |
| 10-Sep-2014 |
Chiyoung Seo <chiyoung.seo@gmail.com> |
MB-11426 Support the in-place manual compaction. If a new file name is not given (i.e., NULL is passed) in a manual compaction mode, then a new file name will be automatically created by
MB-11426 Support the in-place manual compaction. If a new file name is not given (i.e., NULL is passed) in a manual compaction mode, then a new file name will be automatically created by appending a file revision number to the original file name. For example, if the original file name is "test.fdb" and fdb_compact(db, NULL) is invoked, then "test.fdb.1" file will be created after the compaction. Note that this new compacted file can be still opened by using the original file name (e.g., fdb_open(db, "test.fdb")). Example usage: fdb_open(db1, "test.fdb"); ... fdb_compact(db1, NULL); // "test.fdb.1" is created after compaction. // Note that "test.fdb" will be removed automatically // when its reference counter becomes zero. ... fdb_compact(db1, NULL); // "test.fdb.2" is created after compaction. // Note that "test.fdb.1" will be removed automatically // when its reference counter becomes zero. fdb_open(db2, "test.fdb"); // "test.fdb.2" is opened because that is the last // compacted file. ... fdb_close(db1); fdb_close(db2); // "test.fdb.2" is automatically renamed to the original file // name "test.fdb" because there are no database handles on // "test.fdb.2". Change-Id: I1b08f687a91947a214bc607b6966a7fbed8ae533
show more ...
|
#
7164b6de |
| 26-Aug-2014 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
Do not trigger compaction when active_data >= filesize - The (approximated) active data size can be temporarily larger than the file size during WAL flushing. - Since flushed WA
Do not trigger compaction when active_data >= filesize - The (approximated) active data size can be temporarily larger than the file size during WAL flushing. - Since flushed WAL item is lazily released, its document size is counted twice (one for HB+trie and the other one for WAL) during active data size estimation, if the estimation occurs between _wal_flush() and wal_release_flushed_items().
show more ...
|
#
4a0379f5 |
| 29-Jul-2014 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-11835 Support long path to a database file - The length of path to a DB file is supported up to 1016 bytes. - The structure of DB header is modified; the size of file name fields
MB-11835 Support long path to a database file - The length of path to a DB file is supported up to 1016 bytes. - The structure of DB header is modified; the size of file name fields in header is not fixed to 256 bytes anymore. They are dynamically changed according to the length of file name, so that DB header size is also variable (but must be smaller than a block size). Change-Id: Ic4a827ba0ea13e0c8325a4b831515e480e7fe21f
show more ...
|
#
4f2ebc0a |
| 08-Jul-2014 |
Jung-Sang Ahn <jungsang.ahn@gmail.com> |
MB-11056 Adler32 checksum support - Adler32 can replace CRC32 using a preprocessor option '__CHECKSUM_ADLER32' defined in option.h (disabled by default). - Using O2 optimization
MB-11056 Adler32 checksum support - Adler32 can replace CRC32 using a preprocessor option '__CHECKSUM_ADLER32' defined in option.h (disabled by default). - Using O2 optimization option, Adler32 is about 1.5x and +100x faster than CRC32 on MacBook Pro (i7 2.3GHz, 4 cores, 8 threads) for calculating checksum of 1GB and 1KB data respectively. Compared to CRC32 (slicing 8 bytes), Adler32 becomes much faster as the size of data gets smaller (may be due to CPU cache fetching issues). - However, the improvement is very marginal in ForestDB workload (using synchronous commit) because the overall performance is bounded by Disk I/O.
show more ...
|