MB-23583: Gracefully handle errors during block reclaiming

* realloc() call or decompression may cause error in block reclaiming
task, but those errors do not propagate to the upper laye

MB-23583: Gracefully handle errors during block reclaiming

* realloc() call or decompression may cause error in block reclaiming
task, but those errors do not propagate to the upper layer so it causes
crash later.

* Immediately stop block reclaiming task if any kinds of errors happens.

Change-Id: I35ccace0afc03102fa55b889c148f4d4d65d2fbe
Reviewed-on: http://review.couchbase.org/76920
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Jung-Sang Ahn <jungsang.ahn@gmail.com>

show more ...

MB-19665 Do not reclaim stale blocks corresponding to rollback point

- Rollback API appends a new commit header which re-points to the
old index nodes, which can be reclaimed although th

MB-19665 Do not reclaim stale blocks corresponding to rollback point

- Rollback API appends a new commit header which re-points to the
old index nodes, which can be reclaimed although the header is still
active.

- To avoid this issue, we remove stale-tree entries since the
rollback point so that those old index nodes are not classified as
stale blocks anymore.

- However, this approach will cause some stale block leakage (due to
the blocks written after the rollback point). It needs to be
addressed in the future.

Change-Id: I86feae1527d8bdc1d7c597b926b878b566a79b06
Reviewed-on: http://review.couchbase.org/64197
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Suppress benign warning seen in staleblock.cc

Coverity scan shows this warning in stalelock.cc:373:
CID 102895: Null pointer dereferences (FORWARD_NULL)
Assigning: "doc.met

Suppress benign warning seen in staleblock.cc

Coverity scan shows this warning in stalelock.cc:373:
CID 102895: Null pointer dereferences (FORWARD_NULL)
Assigning: "doc.meta" = "NULL".

This shouldn't be a problem, however to suppress this
warning, the entire line of initializing doc.meta to NULL
can be removed as doc is already memset to 0x0.

Change-Id: Ie2e02cf54e63b94d571ecdcab98b1865995e28a7
Reviewed-on: http://review.couchbase.org/63584
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-19173 Maintain in-memory stale block info

- Keep system docs that contain the list of stale regions in-memory
to skip scanning stale-block tree.

- After block reclaim, do not

MB-19173 Maintain in-memory stale block info

- Keep system docs that contain the list of stale regions in-memory
to skip scanning stale-block tree.

- After block reclaim, do not remove & free the remaining items from
'merge-tree', and reuse the items in the next block reclaim without
re-loading them from the corresponding system doc.

Change-Id: I34ff747d55954d284105d3ee1e8be68fa4facb97
Reviewed-on: http://review.couchbase.org/63226
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...

MB-17670 Improve error handling in doc I/O and its callers

This change adapts the return type of doc_io_read APIs to return appropriate
error codes to the callers, so that the callers ca

MB-17670 Improve error handling in doc I/O and its callers

This change adapts the return type of doc_io_read APIs to return appropriate
error codes to the callers, so that the callers can handle those errors
properly and return them to the clients.

Change-Id: I0665fe47150e0d690db7ffc38a202c70a422cd07

show more ...

MB-17004: Prune an old doc's seqnum from sequence index on WAL flush

If a key has been updated or deleted, its entry is removed from the
main index by key but the entry in the sequence t

MB-17004: Prune an old doc's seqnum from sequence index on WAL flush

If a key has been updated or deleted, its entry is removed from the
main index by key but the entry in the sequence tree/trie is not removed.
As a result, the sequence tree continues to grow indefinitely until
compaction is executed, which in turn bloats up the index node space used.
Since we do not support having duplicate values for the same key in the
same snapshot, we can prune the sequence tree to prevent file size explosion.

Change-Id: I4aa2e44261873523e41a347171fd80236d839fc0

show more ...

Fix a bug skipping removal of seq numbers in single KVS mode

- In single KVS mode, kv_info_offset is always BLK_NOT_FOUND so that
we have to handle that case separately. If we don't, sta

Fix a bug skipping removal of seq numbers in single KVS mode

- In single KVS mode, kv_info_offset is always BLK_NOT_FOUND so that
we have to handle that case separately. If we don't, stale seq number
removal is not executed in single KVS mode so that live data size
is getting larger over time.

Change-Id: Ia5378e0826f9d1c83982ff03e4d17ef817ebbc15

show more ...

Get rid of time limit for stale block reclaiming

- If stale regions are randomly scattered over DB file, partial
scanning due to time limit cannot make enough amount of reusable
bloc

Get rid of time limit for stale block reclaiming

- If stale regions are randomly scattered over DB file, partial
scanning due to time limit cannot make enough amount of reusable
blocks.

Change-Id: I121f162980173740579b2917a5dbd08b6f6338de

show more ...

MB-16219 Support superblock and circular block reusing

- Super blocks point to the up-to-date DB header.

- When configured conditions are satisfied, stale blocks are reused
in a

MB-16219 Support superblock and circular block reusing

- Super blocks point to the up-to-date DB header.

- When configured conditions are satisfied, stale blocks are reused
in a circular manner. This does not increase the DB file size, so
that we can largely reduce the overhead from compaction.

- The latest a few old versions are preserved for future snapshot
creation.

Change-Id: I6eb9209ced436c0a84ebb0216344eb04f300fca9

show more ...

MB-16219 Remove seq numbers if corresponding stale documents are reclaimed

- Although doc A is updated to doc A' so that doc A is marked as
stale, seq numbers for both A and A' still coe

MB-16219 Remove seq numbers if corresponding stale documents are reclaimed

- Although doc A is updated to doc A' so that doc A is marked as
stale, seq numbers for both A and A' still coexists in seq-tree.

- Since all updates for seq-tree are monotonic increasing insertions
only, the number of index blocks occupied by seq-tree increases over
time even though the stale blocks generated by seq-tree updates are
reclaimed periodically.

- However, removal of seq number for doc A should be postponed since
other snapshot or reader may access to doc A through the old seq number.
Hence, when stale blocks up to a commit header are reclaimed for
reusing, we then remove the postponed seq numbers, update seq-tree,
and also reclaim the stale blocks as a result of the update of
seq-tree.

Change-Id: Icd760b4adb8822d4d38fca273c028b74253d74e4

show more ...

MB-16219 Support non-consecutive document block

- Add 16-byte metadata at the end of each document block instead of
block marker, to support writing a document into the set of
non-co

MB-16219 Support non-consecutive document block

- Add 16-byte metadata at the end of each document block instead of
block marker, to support writing a document into the set of
non-consecutive blocks.

Change-Id: I5372ac04301ccf68499751c3256ec433b0d55f03

show more ...

MB-16219 Manage reference counter for commit headers

- Add a feature to get the smallest commit revnum that are being
referred by opened handles.

- Also add a feature to merge a

MB-16219 Manage reference counter for commit headers

- Add a feature to get the smallest commit revnum that are being
referred by opened handles.

- Also add a feature to merge all stale regions that are generated
before the revnum, and calculate the list of the reusable blocks.
The reusable block list will be stored in super block as a bitmap.

- This commit does not affect DB behavior yet.

Change-Id: I89ed72f97d9535e3e73d0ce7ba6a2ee18195f30d

show more ...