History log of /4.0.0/forestdb/src/compactor.cc (Results 1 - 25 of 33)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v1.2, v1.1, v1.0
# 5f7821ba 11-Aug-2015 Chiyoung Seo <chiyoung.seo@gmail.com>

MB-16038 Support the lazy cleanup of a file's buffer cache blocks

If a given file is compacted ans no longer accessed, then its cache
blocks in the buffer cache should be cleaned up and

MB-16038 Support the lazy cleanup of a file's buffer cache blocks

If a given file is compacted ans no longer accessed, then its cache
blocks in the buffer cache should be cleaned up and moved to the free list.

As this operation can takes up to several seconds depending on the number
of cache blocks belonging to the file, it is better to support the lazy
cleanup of these cache blocks to avoid the long blocking period in the
application.

Change-Id: I3d607f17e189b63c592298c822672b9178356b0a

show more ...


# d6eb040e 05-Aug-2015 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-15908 Support lazy deletion of old DB file

- To avoid API blocking by high latency of file removal, we support
lazy deletion of old DB file after compaction.

- Once the ref_c

MB-15908 Support lazy deletion of old DB file

- To avoid API blocking by high latency of file removal, we support
lazy deletion of old DB file after compaction.

- Once the ref_count of an old file becomes zero, then it is registered
to the file list of background compaction daemon, and then the daemon
compactor threads periodically check and remove the pending files.

Change-Id: Ie0705a681a05df8ccd68e87ba87576cb81f92156

show more ...


# 0d423664 02-Jul-2015 Sundar Sridharan <sundar.sridharan@gmail.com>

fix garbage value to thread_create in compactor

init num_threads else an arbitary large value of
threads can be created.

Change-Id: I75bc7648757b03ab324961f4de7125b836a8dca8


# 7da24760 13-Jun-2015 Chiyoung Seo <chiyoung.seo@gmail.com>

MB-15278 Fix to potential race issues in daemon compactor and file instance

The daemon compactor refers to the list of file instances that are
currently opened. However, as a file instan

MB-15278 Fix to potential race issues in daemon compactor and file instance

The daemon compactor refers to the list of file instances that are
currently opened. However, as a file instance is destroyed when its
reference counter becomes zero, the daemon compactor should identify
that case and free the corresponding task entry from its queue.

Change-Id: I502cd381ca0ad96f9610413800d8c555fff57bdd

show more ...


# c40231de 11-Jun-2015 Chiyoung Seo <chiyoung.seo@gmail.com>

MB-15278 Add the multi-threaded auto compaction support.

Previously, there was only one daemon compaction thread. This can
cause out-of-disk space issues if many database files are opene

MB-15278 Add the multi-threaded auto compaction support.

Previously, there was only one daemon compaction thread. This can
cause out-of-disk space issues if many database files are opened
and accessed concurrently.

This change improves the auto daemon compaction to have multiple
threads perform compacting individual database files concurrently.

The default number of compactor threads are set to 4, but it can
be configurable upon starting the ForestDB engine.

Change-Id: I70515595e4171588506cfca49523e10448190f8a

show more ...


# 200ec8f5 27-Apr-2015 Chiyoung Seo <chiyoung.seo@gmail.com>

MB-14060 Scan the file to catch up the remaining blocks in fdb_compact_upto API.

This change avoids traversing the main index for each commit header written after
the compaction marker,

MB-14060 Scan the file to catch up the remaining blocks in fdb_compact_upto API.

This change avoids traversing the main index for each commit header written after
the compaction marker, but instead scans the file to copy the remaining data
blocks and commit headers to the new file.

Change-Id: Ibbd2aea70eecf2510a14174a9e70e8cf2baecd80

show more ...


# 85444218 04-Apr-2015 Chiyoung Seo <chiyoung.seo@gmail.com>

Add more loggings to erroneous cases in daemon compactor.

Change-Id: Id387fc8d59e1f669385dede322166e8185377e4b


# 5aa14abc 11-Apr-2015 Sundar Sridharan <sundar.sridharan@gmail.com>

Compiler warnings and minor bug fixes

Change-Id: I753118f2f79760d43b421e63aca512449da89574


# 89434c6e 14-Mar-2015 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-13830 Do not register compacted file in compaction daemon again

- Files whose status is COMPACT_OLD or REMOVED_PENDING should not
be registered in compaction daemon, since they do not

MB-13830 Do not register compacted file in compaction daemon again

- Files whose status is COMPACT_OLD or REMOVED_PENDING should not
be registered in compaction daemon, since they do not need to be
compacted again, and will be removed soon.

Change-Id: I46c5be4c601dc52ac3ecbdcd5d14542b49431a87

show more ...


# 0e995e68 04-Mar-2015 Jung-Sang Ahn <jungsang.ahn@gmail.com>

Reconstruct fullpath for DB file existence check

Change-Id: I1bee07346f1869016855b7680929cfd26eacdde1


# 02adb116 13-Feb-2015 Jung-Sang Ahn <jungsang.ahn@gmail.com>

Fix a bug that DB with custom cmp cannot be auto-compacted

- fdb_open_for_compactor() failed because no cmp_function is passed.

- Now compaction daemon keeps list of cmp_functions f

Fix a bug that DB with custom cmp cannot be auto-compacted

- fdb_open_for_compactor() failed because no cmp_function is passed.

- Now compaction daemon keeps list of cmp_functions for each DB file.

Change-Id: I532c385213e606339e0546187db5a01f7a5580ff

show more ...


# 5b75d849 24-Jan-2015 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-13186 Maintain a separate DB handle for an iterator

- The current iterator handle keeps pointers to the original handle's
resources such as filemgr, trie, dhandle and bhandle. If comp

MB-13186 Maintain a separate DB handle for an iterator

- The current iterator handle keeps pointers to the original handle's
resources such as filemgr, trie, dhandle and bhandle. If compaction is
performed after creating iterator, the original handle's resources are
modified or recreated so that pointers in the iterator handle becomes
invalid.

- To avoid this issue, iterator handle should maintain its own DB
handle which is not influenced by the original handle's change.

Change-Id: If9bf479cbff42dda683209ff933d85733d4493c6

show more ...


# f80ddf4d 22-Jan-2015 Chiyoung Seo <chiyoung.seo@gmail.com>

Make internal functions static if their scopes are not global.

Change-Id: Ic390a59e76b99c86c29a3fefc76a0975becdccd3


# 2e49b909 19-Jan-2015 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-13012 Meta file stores DB file name excluding directory path

- If the file path given by user includes directory path, the
compactor gets rid of the directory path and stores its file

MB-13012 Meta file stores DB file name excluding directory path

- If the file path given by user includes directory path, the
compactor gets rid of the directory path and stores its file name
part only.

Change-Id: I0610a1403fc767cfac3667a692e04da33ce63354

show more ...


# 84f94aa1 16-Dec-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-12613: clang analyzer fixes for null dereference, dead store etc

Change-Id: I71baac9cb49cca7473289dedfbc1c00f08fbc728


# 64515de4 01-Dec-2014 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-12799 Do not use existing file name as a new compacted file name

Change-Id: I502ea9202031727c7617361200163d2cee83fc67


# 4dc52e2d 12-Nov-2014 Trond Norbye <trond.norbye@gmail.com>

Remove unused variables (-Wunused-variable)

Change-Id: I9f63f438848d61de3ca85b4e88a900c3ae5e6ba6


# f54fbdb6 10-Nov-2014 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-12591 Provide a doc count for both file and KV store levels

- Now doc count and space estimation are provided for each KV store
separately.
- Such statistics are globally managed

MB-12591 Provide a doc count for both file and KV store levels

- Now doc count and space estimation are provided for each KV store
separately.
- Such statistics are globally managed so that the legacy 'ndocs' and
'datasize' attributes in the fdb_kvs_handle are removed.
- fdb_get_kvs_info returns the number of docs belonging to the KV store,
while fdb_get_file_info returns the total number of docs in the file.

show more ...


# d32401da 05-Sep-2014 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollba

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollback, snapshot, etc.) are shared
with the original APIs.

- Same as snapshot, KV instance uses the fdb_handle structure, but
commit and compaction can be invoked only using the DB file handle.

Change-Id: Iad571745c57053d46130be584c5213e98baa9b9f

show more ...


# 47ec8bfc 27-Oct-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-12191: Support for fdb_destroy()

Will wipe out all remains of a file in case of auto compaction
and will do best-effort removal of files in case of manual
compaction
If any fi

MB-12191: Support for fdb_destroy()

Will wipe out all remains of a file in case of auto compaction
and will do best-effort removal of files in case of manual
compaction
If any file is open, the api will error out and not delete any file
Reason for best-effort in manual compaction case:
FileA --> FileB --> FileC --> FileA --> FileD --> FileC -->DESTROY
(In above case, FileB cannot be destroyed as its info is not
reachable from FileC)
Change-Id: Iab6a1b77d6cc9c4a38247b15374ef4b2eaf2d48d

show more ...


# fa8408c3 24-Oct-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

Fix memory leak in compactor.cc

opendir needs a corresponding closedir

Change-Id: I5b101d4cd1f04e84365b5e08d1eee04bd8b91fe5


# fef67a48 10-Sep-2014 Chiyoung Seo <chiyoung.seo@gmail.com>

MB-11426 Support the in-place manual compaction.

If a new file name is not given (i.e., NULL is passed) in a manual compaction
mode, then a new file name will be automatically created by

MB-11426 Support the in-place manual compaction.

If a new file name is not given (i.e., NULL is passed) in a manual compaction
mode, then a new file name will be automatically created by appending
a file revision number to the original file name. For example, if the original
file name is "test.fdb" and fdb_compact(db, NULL) is invoked, then "test.fdb.1"
file will be created after the compaction. Note that this new compacted file
can be still opened by using the original file name
(e.g., fdb_open(db, "test.fdb")).

Example usage:

fdb_open(db1, "test.fdb");
...
fdb_compact(db1, NULL); // "test.fdb.1" is created after compaction.
// Note that "test.fdb" will be removed automatically
// when its reference counter becomes zero.
...
fdb_compact(db1, NULL); // "test.fdb.2" is created after compaction.
// Note that "test.fdb.1" will be removed automatically
// when its reference counter becomes zero.
fdb_open(db2, "test.fdb"); // "test.fdb.2" is opened because that is the last
// compacted file.
...
fdb_close(db1);
fdb_close(db2); // "test.fdb.2" is automatically renamed to the original file
// name "test.fdb" because there are no database handles on
// "test.fdb.2".

Change-Id: I1b08f687a91947a214bc607b6966a7fbed8ae533

show more ...


# 7164b6de 26-Aug-2014 Jung-Sang Ahn <jungsang.ahn@gmail.com>

Do not trigger compaction when active_data >= filesize

- The (approximated) active data size can be temporarily larger than
the file size during WAL flushing.

- Since flushed WA

Do not trigger compaction when active_data >= filesize

- The (approximated) active data size can be temporarily larger than
the file size during WAL flushing.

- Since flushed WAL item is lazily released, its document size is
counted twice (one for HB+trie and the other one for WAL) during active
data size estimation, if the estimation occurs between _wal_flush()
and wal_release_flushed_items().

show more ...


# 4a0379f5 29-Jul-2014 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-11835 Support long path to a database file

- The length of path to a DB file is supported up to 1016 bytes.
- The structure of DB header is modified; the size of file name
fields

MB-11835 Support long path to a database file

- The length of path to a DB file is supported up to 1016 bytes.
- The structure of DB header is modified; the size of file name
fields in header is not fixed to 256 bytes anymore. They are
dynamically changed according to the length of file name, so that
DB header size is also variable (but must be smaller than a block
size).

Change-Id: Ic4a827ba0ea13e0c8325a4b831515e480e7fe21f

show more ...


# 4f2ebc0a 08-Jul-2014 Jung-Sang Ahn <jungsang.ahn@gmail.com>

MB-11056 Adler32 checksum support

- Adler32 can replace CRC32 using a preprocessor option
'__CHECKSUM_ADLER32' defined in option.h (disabled by default).

- Using O2 optimization

MB-11056 Adler32 checksum support

- Adler32 can replace CRC32 using a preprocessor option
'__CHECKSUM_ADLER32' defined in option.h (disabled by default).

- Using O2 optimization option, Adler32 is about 1.5x and +100x
faster than CRC32 on MacBook Pro (i7 2.3GHz, 4 cores, 8 threads) for
calculating checksum of 1GB and 1KB data respectively. Compared to
CRC32 (slicing 8 bytes), Adler32 becomes much faster as the size of
data gets smaller (may be due to CPU cache fetching issues).

- However, the improvement is very marginal in ForestDB workload
(using synchronous commit) because the overall performance is bounded
by Disk I/O.

show more ...


12