MB-14253: Operational Stats for ForestDB CRUD apis

Operational stats at coarse granularity added for
set, gets, commits, compacts, iterator_gets, iterator_moves
These are in-memory &

MB-14253: Operational Stats for ForestDB CRUD apis

Operational stats at coarse granularity added for
set, gets, commits, compacts, iterator_gets, iterator_moves
These are in-memory & migrated over on compaction to new file.
Snapshots can still access old_file's stats.

Change-Id: Ibbc490600dc65402b2133143edba2213b6417583

show more ...

MB-14696:Return HANDLE_BUSY err if handle is shared among threads

Enforce one handle per concurrent unit of execution by using atomics.
Return FDB_RESULT_HANDLE_BUSY if a handle is found

MB-14696:Return HANDLE_BUSY err if handle is shared among threads

Enforce one handle per concurrent unit of execution by using atomics.
Return FDB_RESULT_HANDLE_BUSY if a handle is found to be in use
in another thread. Test as many apis as possible by simulating
concurrent access using anomaly test callback framework.

Change-Id: Ia1a640dc6bd49c60be5e2b92af54d4966a3903af

show more ...

Remove new_file and new_dhandle from fdb_handle structure.

new_file and new_dhandle variables are no longer used with a
new compaciton method.

Change-Id: I8bb89e0ffca4cbbf1871ee

Remove new_file and new_dhandle from fdb_handle structure.

new_file and new_dhandle variables are no longer used with a
new compaciton method.

Change-Id: I8bb89e0ffca4cbbf1871ee58bab0f367c85d6cec

show more ...

fdb_assert() to trace & dump all forestdb handles if SIGABRT

cmake -D_TRACK_HANDLES=1 /path/to/forestdb
to turn on handle track and dump on assert
This change will help track open fi

fdb_assert() to trace & dump all forestdb handles if SIGABRT

cmake -D_TRACK_HANDLES=1 /path/to/forestdb
to turn on handle track and dump on assert
This change will help track open file handles and dump their content
on asserts() only if the macro is enabled at compile-time
To help debug golang environments where there are no useful symbols
available in core dumps
Change-Id: I7ce97ed713452c3dbe2c7eb2925d913a2c0c902b

show more ...

MB-13930 Make a snapshot of dirty HB+trie nodes in in-memory snapshot

- To avoid accessing dirty HB+trie nodes which are being updated by
the concurrent writer, a snapshot of those dirty

MB-13930 Make a snapshot of dirty HB+trie nodes in in-memory snapshot

- To avoid accessing dirty HB+trie nodes which are being updated by
the concurrent writer, a snapshot of those dirty nodes are created
in each snapshot's btreeblk_handle.

- In-memory snapshot always see the copied instances of the dirty
nodes, although they are updated by the concurrent writer.

- The same fix is applied to iterator on a normal handle, which can
be treated as an in-memory snapshot if dirty nodes exist.

- Due to this fix, iterator does not need to grab filemgr_mutex
anymore since it does not access the shared non-immutable blocks.

Change-Id: I6f1fdc6dddceeac8d9d35704ec1401903f7dda34

show more ...

MB-13186 Maintain a separate DB handle for an iterator

- The current iterator handle keeps pointers to the original handle's
resources such as filemgr, trie, dhandle and bhandle. If comp

MB-13186 Maintain a separate DB handle for an iterator

- The current iterator handle keeps pointers to the original handle's
resources such as filemgr, trie, dhandle and bhandle. If compaction is
performed after creating iterator, the original handle's resources are
modified or recreated so that pointers in the iterator handle becomes
invalid.

- To avoid this issue, iterator handle should maintain its own DB
handle which is not influenced by the original handle's change.

Change-Id: If9bf479cbff42dda683209ff933d85733d4493c6

show more ...

MB-13096 Sync root node of trie if dirty WAL flush exists

- Same as in fdb_get(), iterator APIs also need to sync the dirty root
nodes and grab lock if WAL flush was performed by other h

MB-13096 Sync root node of trie if dirty WAL flush exists

- Same as in fdb_get(), iterator APIs also need to sync the dirty root
nodes and grab lock if WAL flush was performed by other handle before
invoking fdb_commit().

Change-Id: I4535c8d2b6a8698961b9cd3e97e4d9404a73f2ef

show more ...

MB-11766 Support longer key than 3840 bytes

- The max key length is enlarged to 64KB, which is the max value of
2-byte integer.

Change-Id: I39e263883092a78f04b04bb09a9cd3f1271e5

MB-11766 Support longer key than 3840 bytes

- The max key length is enlarged to 64KB, which is the max value of
2-byte integer.

Change-Id: I39e263883092a78f04b04bb09a9cd3f1271e506b

show more ...

MB-12999: Share iterator's AVL tree key comparisons with snapshots

If iterator tries to seek on a snapshot handle, it was doing an avl
tree search over an incorrect aux handle causing cr

MB-12999: Share iterator's AVL tree key comparisons with snapshots

If iterator tries to seek on a snapshot handle, it was doing an avl
tree search over an incorrect aux handle causing crashes
Fix by using a single universal key comparison context across
both snapshot as well as wal comparisons of iterator
Test case added with code coverage
Change-Id: I4b52a3b925a12166761ca984ee0326cab6860b4a

show more ...

MB-12465: New Iteration APIs & Semantics

1) fdb_iterator_next and fdb_iterator_prev only re-position cursor.
2) fdb_iterator_get and fdb_iterator_get_metaonly retrieve items
at the c

MB-12465: New Iteration APIs & Semantics

1) fdb_iterator_next and fdb_iterator_prev only re-position cursor.
2) fdb_iterator_get and fdb_iterator_get_metaonly retrieve items
at the cursor.
3) fdb_iterator_seek_to_min positions the cursor at the smallest key.
4) fdb_iterator_seek_to_max() positions the cursor at the largest key.
5) fdb_iterator_seek() accepts a direction to indicate which key must
be selected if the seek key is not present.
6) Reduced memory usage for the iterator by use of unions where possible.
7) Simplified TEST_CMP for memcmp to report errors in test cases.
For example,
fdb_doc *rdoc;
fdb_iterator_init(...);
do {
status = fdb_iterator_get(iterator, &rdoc);
...
fdb_doc_free(rdoc);
} while (fdb_iterator_next(iterator) != FDB_RESULT_ITERATOR_FAIL);
status = fdb_iterator_seek_to_min(iterator);
status = fdb_iterator_get(iterator, &rdoc);
...
fdb_doc_free(rdoc);
...
status = fdb_iterator_seek(iterator, seek_key, seek_keylen, direction);
status = fdb_iterator_seek_to_max(iterator);
status = fdb_iterator_get(iterator, &rdoc);
...
fdb_doc_free(rdoc);
Change-Id: I3818f39316ecaa8cbc2d5bc37e421efe6cb36101

show more ...

MB-12591 Provide a doc count for both file and KV store levels

- Now doc count and space estimation are provided for each KV store
separately.
- Such statistics are globally managed

MB-12591 Provide a doc count for both file and KV store levels

- Now doc count and space estimation are provided for each KV store
separately.
- Such statistics are globally managed so that the legacy 'ndocs' and
'datasize' attributes in the fdb_kvs_handle are removed.
- fdb_get_kvs_info returns the number of docs belonging to the KV store,
while fdb_get_file_info returns the total number of docs in the file.

show more ...

Make file and KV store distinction clear in public APIs.

The following APIs and structures are renamed as follows:

APIs:
fdb_get_dbinfo --> fdb_get_file_info
fdb_get_seqnum

Make file and KV store distinction clear in public APIs.

The following APIs and structures are renamed as follows:

APIs:
fdb_get_dbinfo --> fdb_get_file_info
fdb_get_seqnum --> fdb_get_kvs_seqnum

Structure:
fdb_handle --> fdb_kvs_handle
fdb_info --> fdb_file_info

Change-Id: Ia822aa1f61a087feec4e86f07cdc9fa855911191

show more ...

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollba

MB-11071 Support multiple KV instances

- Add three new main operations: open (create), close, remove of KV
instance in a DB file handle.

- All other operations (get, set, rollback, snapshot, etc.) are shared
with the original APIs.

- Same as snapshot, KV instance uses the fdb_handle structure, but
commit and compaction can be invoked only using the DB file handle.

Change-Id: Iad571745c57053d46130be584c5213e98baa9b9f

show more ...

Fix crash on multi writers with wal_flush_before_commit option

- If wal_flush_before_commit option is turned on, WAL is flushed before
fdb_commit() is invoked. The updated HB+trie nodes

Fix crash on multi writers with wal_flush_before_commit option

- If wal_flush_before_commit option is turned on, WAL is flushed before
fdb_commit() is invoked. The updated HB+trie nodes are still writable
unless fdb_commit(), so that race condition can occur if concurrent
writers modify the node at the same time.

- The (writable) dirty index nodes are shared among concurrent writers.

show more ...

MB-10961 Reverse Iteration Feature Support

New api fdb_iterator_prev() to reverse iterate over keys
in the sequence B+ trees as well as the HB+ Tries
along with internal hbtrie_prev

MB-10961 Reverse Iteration Feature Support

New api fdb_iterator_prev() to reverse iterate over keys
in the sequence B+ trees as well as the HB+ Tries
along with internal hbtrie_prev and btree_prev routines
Each layer (B-tree, HB-Trie, ForestDB) remembers the direction
and position of iteration so the next call is as expected.
test cases: sequence btree iteration,
forward+reverse+forward with turn at either end at all 3 levels
Change-Id: I5b899e77a6223004112b3fcd1ff61b161a9104d2

show more ...

MB-11426 Support the in-place manual compaction.

If a new file name is not given (i.e., NULL is passed) in a manual compaction
mode, then a new file name will be automatically created by

MB-11426 Support the in-place manual compaction.

If a new file name is not given (i.e., NULL is passed) in a manual compaction
mode, then a new file name will be automatically created by appending
a file revision number to the original file name. For example, if the original
file name is "test.fdb" and fdb_compact(db, NULL) is invoked, then "test.fdb.1"
file will be created after the compaction. Note that this new compacted file
can be still opened by using the original file name
(e.g., fdb_open(db, "test.fdb")).

Example usage:

fdb_open(db1, "test.fdb");
...
fdb_compact(db1, NULL); // "test.fdb.1" is created after compaction.
// Note that "test.fdb" will be removed automatically
// when its reference counter becomes zero.
...
fdb_compact(db1, NULL); // "test.fdb.2" is created after compaction.
// Note that "test.fdb.1" will be removed automatically
// when its reference counter becomes zero.
fdb_open(db2, "test.fdb"); // "test.fdb.2" is opened because that is the last
// compacted file.
...
fdb_close(db1);
fdb_close(db2); // "test.fdb.2" is automatically renamed to the original file
// name "test.fdb" because there are no database handles on
// "test.fdb.2".

Change-Id: I1b08f687a91947a214bc607b6966a7fbed8ae533

show more ...

MB-11600 Fix incorrect 'last WAL flush header BID'

- WAL maintains a list of active transactions belonging to the file.
- Each transaction stores the last header's BID before the transac

MB-11600 Fix incorrect 'last WAL flush header BID'

- WAL maintains a list of active transactions belonging to the file.
- Each transaction stores the last header's BID before the transaction
begins.
- During commit, we find the earliest transaction that is not flushed
yet, and set the 'last WAL flush header BID' as the last header's BID
before the earliest transaction begins.

Change-Id: I94f8ea45e6ef52068477a03a0ed1029caebceff4

show more ...

Support read_uncommitted isolation level in a transaction.

As of this change, we support read_committed and read_uncommitted
isolation levels, and plan to support serializable and repeat

Support read_uncommitted isolation level in a transaction.

As of this change, we support read_committed and read_uncommitted
isolation levels, and plan to support serializable and repeatable
reads in the future.

Change-Id: I47a3df49f359dc2061d35ac46db50ed82c3ec116

show more ...

MB-11123 Transaction support in ForestDB

- Add transactional features to existing get/set API calls.
- Dirty updates belonging to a transaction cannot be seen by other
handles until

MB-11123 Transaction support in ForestDB

- Add transactional features to existing get/set API calls.
- Dirty updates belonging to a transaction cannot be seen by other
handles until the transaction is committed.
- Isolation level: read committed. Both non-repeatable reads and
phantoms may occur, and concurrent transactions will not be
serialized. Please refer to the following link:
http://en.wikipedia.org/wiki/Isolation_level

Change-Id: I42b60f3b65a6c169770dee7ac34d588fa10a35f1

show more ...

Get rid of __FDB_SEQTREE flag

- ForestDB already has run-time configuration option for seq tree.

Change-Id: I3449f959b6fbe1229ee3689d651f4506a9dcc43b

MB-9973 Implement compaction daemon

- Implement background compaction daemon which periodically wakes up
and automatically performs compaction on active (opened) DB instances.

C

MB-9973 Implement compaction daemon

- Implement background compaction daemon which periodically wakes up
and automatically performs compaction on active (opened) DB instances.

Change-Id: Ic816aab444d317e7274e8d30aea7fbda6e6afa4c

show more ...

MB-10965 Pass configs through fdb_config struct.

Using JSON doc to pass configs turns out to be more complicated
than using a structure. This change returns back to the old way
of pa

MB-10965 Pass configs through fdb_config struct.

Using JSON doc to pass configs turns out to be more complicated
than using a structure. This change returns back to the old way
of passing configs through fdb_config struct.

Change-Id: Ic1b1d568a3340237cfa2cc5cb99005b72c15d12d

show more ...

MB-10804: Add Rollback support to forestdb

+ fdb_rollback(fdb_handle *, fdb_seqnum_t ) reverse scans file
to locate DB header having rollback sequence number as marker
+ if found, ex

MB-10804: Add Rollback support to forestdb

+ fdb_rollback(fdb_handle *, fdb_seqnum_t ) reverse scans file
to locate DB header having rollback sequence number as marker
+ if found, existing WAL is destroyed and WAL is restored from
the rollback header
+ Roll back DB header is appended into file header
+ Disable mutations while rollback is in progress
+ code coverage test cases added
Change-Id: I9452dae4dde380b38abd911c23b0632b1cd526d9

show more ...

MB-10485: Snapshot full support in forestdb

+ fdb_snapshot_open( with a sequence number marker)
Separate AVL trees stash unflushed WAL documents
all fdb apis look up snapshot trees i

MB-10485: Snapshot full support in forestdb

+ fdb_snapshot_open( with a sequence number marker)
Separate AVL trees stash unflushed WAL documents
all fdb apis look up snapshot trees instead of wal
snapshots unaffected by WAL
+ code coverage test cases for examples of usage

Change-Id: I207b89c283848d2895eed69086794c04a06cae1d

show more ...

MB-10845 Support the API to set up the error logging callback

For debugging and tracking any errors inside forestdb,
we provide the new API for registering the error logging
callback

MB-10845 Support the API to set up the error logging callback

For debugging and tracking any errors inside forestdb,
we provide the new API for registering the error logging
callback:

fdb_status fdb_set_log(fdb_handle *handle,
logCallbackFunc,
void *pCtxData);

logCallbackFunc passed by applications can be something like:

void logCallbackFunc(int err_code,
const char *err_msg,
void *pCtxData) {
fprintf(stderr, "error: %d, err_msg: %s\n", err_code, err_msg);
}

pCtxData is used for passing any application-specific context
data to the logging callback.

Change-Id: I9ac742ce579b009ab6581e885dbf4f08b9d9f7f4

show more ...