MB-15394: Avoid btree block copy in snapshot clone

Use ref counts & avoid copy of dirty btree node blocks
when cloning snapshot from an existing snapshot.
This avoids huge memory ove

MB-15394: Avoid btree block copy in snapshot clone

Use ref counts & avoid copy of dirty btree node blocks
when cloning snapshot from an existing snapshot.
This avoids huge memory overhead with large number of
cloned snapshot handles.

Change-Id: Ic29ff0fb09256d70a3fc4767f82f081daa47244a

show more ...

Compiler warnings and minor bug fixes

Change-Id: I753118f2f79760d43b421e63aca512449da89574

MB-14182 Reset sub-block info in btreeblk handle after dirty WAL flush

- After dirty WAL flush, sub-block info should be reset to prevent
blocks containing sub-blocks from being updated

MB-14182 Reset sub-block info in btreeblk handle after dirty WAL flush

- After dirty WAL flush, sub-block info should be reset to prevent
blocks containing sub-blocks from being updated later. It causes block
consistency issue if there are more than two concurrent writers that
trigger dirty WAL flush.

Change-Id: Ic5e27d08ba5142a8568628f4f137f2d5dd6ca85d

show more ...

MB-13930 Make a snapshot of dirty HB+trie nodes in in-memory snapshot

- To avoid accessing dirty HB+trie nodes which are being updated by
the concurrent writer, a snapshot of those dirty

MB-13930 Make a snapshot of dirty HB+trie nodes in in-memory snapshot

- To avoid accessing dirty HB+trie nodes which are being updated by
the concurrent writer, a snapshot of those dirty nodes are created
in each snapshot's btreeblk_handle.

- In-memory snapshot always see the copied instances of the dirty
nodes, although they are updated by the concurrent writer.

- The same fix is applied to iterator on a normal handle, which can
be treated as an in-memory snapshot if dirty nodes exist.

- Due to this fix, iterator does not need to grab filemgr_mutex
anymore since it does not access the shared non-immutable blocks.

Change-Id: I6f1fdc6dddceeac8d9d35704ec1401903f7dda34

show more ...

MB-12855 Return IO errors via btreeblk_end

btreeblk_end() may fail while trying to write dirty btreeblocks
to disk (if disk is full).
If this failure does not result in failure of th

MB-12855 Return IO errors via btreeblk_end

btreeblk_end() may fail while trying to write dirty btreeblocks
to disk (if disk is full).
If this failure does not result in failure of the
user function, then subsequent forestdb operations can crash
or abort in unpredictable ways.
Fix is to propogate the error to caller via pathways such as
wal_flush() and wal_commit() which try to do disk I/O
+ Anomaly test suite altered to reproduce this issue reliably
Change-Id: I15d4832fb7f5be6398ef7f5b2da7a9df15473f8c

show more ...

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollba

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollback, snapshot, etc.) are shared
with the original APIs.

- Same as snapshot, KV instance uses the fdb_handle structure, but
commit and compaction can be invoked only using the DB file handle.

Change-Id: Iad571745c57053d46130be584c5213e98baa9b9f

show more ...

MB-11213 Allow multiple nodes in a block for small B+trees

- When B+tree is created at first so that it consists of a single
unique node, a small sub-block (whose size is 128 bytes by de

MB-11213 Allow multiple nodes in a block for small B+trees

- When B+tree is created at first so that it consists of a single
unique node, a small sub-block (whose size is 128 bytes by default)
is allocated for the B+tree node, instead of a whole block.

- As B+tree grows, the size of its (unique) node also gets doubled
(128 -> 256 -> ..). The B+tree does not use small-sized block anymore
when its height becomes equal to or greater than 2.

- Multiple same-sized sub-blocks are packed in a single block so that
we can save the space occupied by ForestDB up to 1/32 when a block
size is 4096 bytes (128/4096), if there are lots of small B+trees as
leaf nodes of HB+trie.

Change-Id: I4ec8d181dc90be4d994aff60f7b32c127fb6b3b3

show more ...

MB-10845 Support the API to set up the error logging callback

For debugging and tracking any errors inside forestdb,
we provide the new API for registering the error logging
callback

MB-10845 Support the API to set up the error logging callback

For debugging and tracking any errors inside forestdb,
we provide the new API for registering the error logging
callback:

fdb_status fdb_set_log(fdb_handle *handle,
logCallbackFunc,
void *pCtxData);

logCallbackFunc passed by applications can be something like:

void logCallbackFunc(int err_code,
const char *err_msg,
void *pCtxData) {
fprintf(stderr, "error: %d, err_msg: %s\n", err_code, err_msg);
}

pCtxData is used for passing any application-specific context
data to the logging callback.

Change-Id: I9ac742ce579b009ab6581e885dbf4f08b9d9f7f4

show more ...

More accurate DB instance size estimation

- The estimated DB size by fdb_estimate_space_used() becomes very
inaccurate when the key distribution is not uniform.
- For more accurate e

More accurate DB instance size estimation

- The estimated DB size by fdb_estimate_space_used() becomes very
inaccurate when the key distribution is not uniform.
- For more accurate estimation, the number of live nodes is counted
for each B+Tree node operation.
- The count is stored in DB header (a new 8-byte field is added).

Change-Id: I8ca30b3f25397e332581600834709f4a51fbc583

show more ...

Windows compatibility

- Code is revised for compatibility on Windows, especially for
MinGW and Cygwin.
- OS dependent structures such as spin lock, mutex, thread, time
are genera

Windows compatibility

- Code is revised for compatibility on Windows, especially for
MinGW and Cygwin.
- OS dependent structures such as spin lock, mutex, thread, time
are generalized.
- File system operations (pread, pwrite, fsync) compatible on
Windows are added.
- Can be compiled using Visual Studio CL compiler
(*.c files have to be changed to *.cc)
(nmake should include /wd4716 option)
Change-Id: I47c03bfe88fd2601b4d03340a8df7dd917f486a3

show more ...

Adapt the copylight paragraph and README.md

Benchmark program is modified
- Couchstore_bench program is modified to support the large-sized workloads over multiple DB files. Initial population can be performed using multiple threads so tha

Benchmark program is modified
- Couchstore_bench program is modified to support the large-sized workloads over multiple DB files. Initial population can be performed using multiple threads so that we can utilize the maximum bandwidth of the storage.
- Some minor bugs are fixed on b-tree (memory leakage)

show more ...

Memory leakage fixed
- A couple of memory leakages occurred during compaction. The compaction function and FDB close function have been fixed, but not sure whether other leakages exist or not.

Memory leakage fixed
- A couple of memory leakages occurred during compaction. The compaction function and FDB close function have been fixed, but not sure whether other leakages exist or not.
- Simple memory leakage detection code has been added.

show more ...

DB header modification & b-tree block layer improvement & sequence number tree
- DB header is flushed for every commit commands. Dirty blocks will not be synchronized to file when the file is clo

DB header modification & b-tree block layer improvement & sequence number tree
- DB header is flushed for every commit commands. Dirty blocks will not be synchronized to file when the file is closed without commit.
- Small LRU cache has been added to b-tree block layer. B-tree nodes used for previous tree operations don't need to be re-read (memory copied) from file or buffer cache.
- Seq-tree has been fully functional.

show more ...

Formatting & benchmark program minor change
- Benchmark duration can be configured in config file. User can run benchmark choosing between the exact number of batches and fixed duration.

Initial commit