MB-16038 Support the lazy cleanup of a file's buffer cache blocks

If a given file is compacted ans no longer accessed, then its cache
blocks in the buffer cache should be cleaned up and

MB-16038 Support the lazy cleanup of a file's buffer cache blocks

If a given file is compacted ans no longer accessed, then its cache
blocks in the buffer cache should be cleaned up and moved to the free list.

As this operation can takes up to several seconds depending on the number
of cache blocks belonging to the file, it is better to support the lazy
cleanup of these cache blocks to avoid the long blocking period in the
application.

Change-Id: I3d607f17e189b63c592298c822672b9178356b0a

show more ...

MB-15908 Support lazy deletion of old DB file

- To avoid API blocking by high latency of file removal, we support
lazy deletion of old DB file after compaction.

- Once the ref_c

MB-15908 Support lazy deletion of old DB file

- To avoid API blocking by high latency of file removal, we support
lazy deletion of old DB file after compaction.

- Once the ref_count of an old file becomes zero, then it is registered
to the file list of background compaction daemon, and then the daemon
compactor threads periodically check and remove the pending files.

Change-Id: Ie0705a681a05df8ccd68e87ba87576cb81f92156

show more ...

fix garbage value to thread_create in compactor

init num_threads else an arbitary large value of
threads can be created.

Change-Id: I75bc7648757b03ab324961f4de7125b836a8dca8

MB-15278 Fix to potential race issues in daemon compactor and file instance

The daemon compactor refers to the list of file instances that are
currently opened. However, as a file instan

MB-15278 Fix to potential race issues in daemon compactor and file instance

The daemon compactor refers to the list of file instances that are
currently opened. However, as a file instance is destroyed when its
reference counter becomes zero, the daemon compactor should identify
that case and free the corresponding task entry from its queue.

Change-Id: I502cd381ca0ad96f9610413800d8c555fff57bdd

show more ...

MB-15278 Add the multi-threaded auto compaction support.

Previously, there was only one daemon compaction thread. This can
cause out-of-disk space issues if many database files are opene

MB-15278 Add the multi-threaded auto compaction support.

Previously, there was only one daemon compaction thread. This can
cause out-of-disk space issues if many database files are opened
and accessed concurrently.

This change improves the auto daemon compaction to have multiple
threads perform compacting individual database files concurrently.

The default number of compactor threads are set to 4, but it can
be configurable upon starting the ForestDB engine.

Change-Id: I70515595e4171588506cfca49523e10448190f8a

show more ...

MB-14060 Scan the file to catch up the remaining blocks in fdb_compact_upto API.

This change avoids traversing the main index for each commit header written after
the compaction marker,

MB-14060 Scan the file to catch up the remaining blocks in fdb_compact_upto API.

This change avoids traversing the main index for each commit header written after
the compaction marker, but instead scans the file to copy the remaining data
blocks and commit headers to the new file.

Change-Id: Ibbd2aea70eecf2510a14174a9e70e8cf2baecd80

show more ...

Add more loggings to erroneous cases in daemon compactor.

Change-Id: Id387fc8d59e1f669385dede322166e8185377e4b

Compiler warnings and minor bug fixes

Change-Id: I753118f2f79760d43b421e63aca512449da89574

MB-13830 Do not register compacted file in compaction daemon again

- Files whose status is COMPACT_OLD or REMOVED_PENDING should not
be registered in compaction daemon, since they do not

MB-13830 Do not register compacted file in compaction daemon again

- Files whose status is COMPACT_OLD or REMOVED_PENDING should not
be registered in compaction daemon, since they do not need to be
compacted again, and will be removed soon.

Change-Id: I46c5be4c601dc52ac3ecbdcd5d14542b49431a87

show more ...

Reconstruct fullpath for DB file existence check

Change-Id: I1bee07346f1869016855b7680929cfd26eacdde1

Fix a bug that DB with custom cmp cannot be auto-compacted

- fdb_open_for_compactor() failed because no cmp_function is passed.

- Now compaction daemon keeps list of cmp_functions f

Fix a bug that DB with custom cmp cannot be auto-compacted

- fdb_open_for_compactor() failed because no cmp_function is passed.

- Now compaction daemon keeps list of cmp_functions for each DB file.

Change-Id: I532c385213e606339e0546187db5a01f7a5580ff

show more ...

MB-13186 Maintain a separate DB handle for an iterator

- The current iterator handle keeps pointers to the original handle's
resources such as filemgr, trie, dhandle and bhandle. If comp

MB-13186 Maintain a separate DB handle for an iterator

- The current iterator handle keeps pointers to the original handle's
resources such as filemgr, trie, dhandle and bhandle. If compaction is
performed after creating iterator, the original handle's resources are
modified or recreated so that pointers in the iterator handle becomes
invalid.

- To avoid this issue, iterator handle should maintain its own DB
handle which is not influenced by the original handle's change.

Change-Id: If9bf479cbff42dda683209ff933d85733d4493c6

show more ...

Make internal functions static if their scopes are not global.

Change-Id: Ic390a59e76b99c86c29a3fefc76a0975becdccd3

MB-13012 Meta file stores DB file name excluding directory path

- If the file path given by user includes directory path, the
compactor gets rid of the directory path and stores its file

MB-13012 Meta file stores DB file name excluding directory path

- If the file path given by user includes directory path, the
compactor gets rid of the directory path and stores its file name
part only.

Change-Id: I0610a1403fc767cfac3667a692e04da33ce63354

show more ...

MB-12613: clang analyzer fixes for null dereference, dead store etc

Change-Id: I71baac9cb49cca7473289dedfbc1c00f08fbc728

MB-12799 Do not use existing file name as a new compacted file name

Change-Id: I502ea9202031727c7617361200163d2cee83fc67

Remove unused variables (-Wunused-variable)

Change-Id: I9f63f438848d61de3ca85b4e88a900c3ae5e6ba6

MB-12591 Provide a doc count for both file and KV store levels

- Now doc count and space estimation are provided for each KV store
separately.
- Such statistics are globally managed

MB-12591 Provide a doc count for both file and KV store levels

- Now doc count and space estimation are provided for each KV store
separately.
- Such statistics are globally managed so that the legacy 'ndocs' and
'datasize' attributes in the fdb_kvs_handle are removed.
- fdb_get_kvs_info returns the number of docs belonging to the KV store,
while fdb_get_file_info returns the total number of docs in the file.

show more ...

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollba

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollback, snapshot, etc.) are shared
with the original APIs.

- Same as snapshot, KV instance uses the fdb_handle structure, but
commit and compaction can be invoked only using the DB file handle.

Change-Id: Iad571745c57053d46130be584c5213e98baa9b9f

show more ...

MB-12191: Support for fdb_destroy()

Will wipe out all remains of a file in case of auto compaction
and will do best-effort removal of files in case of manual
compaction
If any fi

MB-12191: Support for fdb_destroy()

Will wipe out all remains of a file in case of auto compaction
and will do best-effort removal of files in case of manual
compaction
If any file is open, the api will error out and not delete any file
Reason for best-effort in manual compaction case:
FileA --> FileB --> FileC --> FileA --> FileD --> FileC -->DESTROY
(In above case, FileB cannot be destroyed as its info is not
reachable from FileC)
Change-Id: Iab6a1b77d6cc9c4a38247b15374ef4b2eaf2d48d

show more ...

Fix memory leak in compactor.cc

opendir needs a corresponding closedir

Change-Id: I5b101d4cd1f04e84365b5e08d1eee04bd8b91fe5

MB-11426 Support the in-place manual compaction.

If a new file name is not given (i.e., NULL is passed) in a manual compaction
mode, then a new file name will be automatically created by

MB-11426 Support the in-place manual compaction.

If a new file name is not given (i.e., NULL is passed) in a manual compaction
mode, then a new file name will be automatically created by appending
a file revision number to the original file name. For example, if the original
file name is "test.fdb" and fdb_compact(db, NULL) is invoked, then "test.fdb.1"
file will be created after the compaction. Note that this new compacted file
can be still opened by using the original file name
(e.g., fdb_open(db, "test.fdb")).

Example usage:

fdb_open(db1, "test.fdb");
...
fdb_compact(db1, NULL); // "test.fdb.1" is created after compaction.
// Note that "test.fdb" will be removed automatically
// when its reference counter becomes zero.
...
fdb_compact(db1, NULL); // "test.fdb.2" is created after compaction.
// Note that "test.fdb.1" will be removed automatically
// when its reference counter becomes zero.
fdb_open(db2, "test.fdb"); // "test.fdb.2" is opened because that is the last
// compacted file.
...
fdb_close(db1);
fdb_close(db2); // "test.fdb.2" is automatically renamed to the original file
// name "test.fdb" because there are no database handles on
// "test.fdb.2".

Change-Id: I1b08f687a91947a214bc607b6966a7fbed8ae533

show more ...

Do not trigger compaction when active_data >= filesize

- The (approximated) active data size can be temporarily larger than
the file size during WAL flushing.

- Since flushed WA

Do not trigger compaction when active_data >= filesize

- The (approximated) active data size can be temporarily larger than
the file size during WAL flushing.

- Since flushed WAL item is lazily released, its document size is
counted twice (one for HB+trie and the other one for WAL) during active
data size estimation, if the estimation occurs between _wal_flush()
and wal_release_flushed_items().

show more ...

MB-11835 Support long path to a database file

- The length of path to a DB file is supported up to 1016 bytes.
- The structure of DB header is modified; the size of file name
fields

MB-11835 Support long path to a database file

- The length of path to a DB file is supported up to 1016 bytes.
- The structure of DB header is modified; the size of file name
fields in header is not fixed to 256 bytes anymore. They are
dynamically changed according to the length of file name, so that
DB header size is also variable (but must be smaller than a block
size).

Change-Id: Ic4a827ba0ea13e0c8325a4b831515e480e7fe21f

show more ...

MB-11056 Adler32 checksum support

- Adler32 can replace CRC32 using a preprocessor option
'__CHECKSUM_ADLER32' defined in option.h (disabled by default).

- Using O2 optimization

MB-11056 Adler32 checksum support

- Adler32 can replace CRC32 using a preprocessor option
'__CHECKSUM_ADLER32' defined in option.h (disabled by default).

- Using O2 optimization option, Adler32 is about 1.5x and +100x
faster than CRC32 on MacBook Pro (i7 2.3GHz, 4 cores, 8 threads) for
calculating checksum of 1GB and 1KB data respectively. Compared to
CRC32 (slicing 8 bytes), Adler32 becomes much faster as the size of
data gets smaller (may be due to CPU cache fetching issues).

- However, the improvement is very marginal in ForestDB workload
(using synchronous commit) because the overall performance is bounded
by Disk I/O.

show more ...