History log of /5.5.2/kv_engine/engines/ep/src/kvstore.h (Results 1 - 25 of 229)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3, v5.5.4, v5.5.5, v5.5.6, v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1
# de9e025d 03-May-2018 Jim Walker <jim@couchbase.com>

MB-29480, MB-29512: Fail backfills that go below purge-seqno

If a backfill is requested and it is not a backfill of everything
the start must be below the purge-seqno, otherwise a DCP cl

MB-29480, MB-29512: Fail backfills that go below purge-seqno

If a backfill is requested and it is not a backfill of everything
the start must be below the purge-seqno, otherwise a DCP client
may miss deletions which have been purged.

This is achieved by loading the purgeSeqno into the ScanContext
and getting backfill create to abort (setting the stream as dead)

The initScanContext method will have opened the data file (and kept
it open) so that the purge-seqno used in the final check won't
change again.

Change-Id: I7505529d46eb6f2b6006695baf7dd9f190237df9
Reviewed-on: http://review.couchbase.org/93690
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# a52ce0a4 10-Apr-2018 Jim Walker <jim@couchbase.com>

MB-29119: Replace revSeqno with a 48-bit counter

Prevent a value too large to be stored in couchstore
from being placed into Item/StoredValue and also the
_local document (via vbucke

MB-29119: Replace revSeqno with a 48-bit counter

Prevent a value too large to be stored in couchstore
from being placed into Item/StoredValue and also the
_local document (via vbucket_state).

Change-Id: I8613c0c51388e91612cde6216445a38c3351c190
Reviewed-on: http://review.couchbase.org/92485
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v5.0.1
# dea18910 25-Oct-2017 Jim Walker <jim@couchbase.com>

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# f9d97cba 21-Feb-2018 Jim Walker <jim@couchbase.com>

MB-21916: Update get/getWithHeader to work with namespaced keys

The rollback tests failed when run with collections_prototype and
the issue was found to be the get/getWithHeader code not

MB-21916: Update get/getWithHeader to work with namespaced keys

The rollback tests failed when run with collections_prototype and
the issue was found to be the get/getWithHeader code not doing
any namespacing, so a get would fail when it should have found a key.

Simply switch the function from taking a DocKey& to a StoredDocKey& and
then perform a namespaced lookup based if KVStore configuration requires
it.

Change-Id: I18816697962695fa354152b97472247097275b37
Reviewed-on: http://review.couchbase.org/89795
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# cf32544c 19-Jan-2018 Dave Rigby <daver@couchbase.com>

Rollback: Add additional comments to rollback flow

Change-Id: I614212ec826f516c5853a43caecb6fb68c7e0326
Reviewed-on: http://review.couchbase.org/88093
Tested-by: Build Bot <build@cou

Rollback: Add additional comments to rollback flow

Change-Id: I614212ec826f516c5853a43caecb6fb68c7e0326
Reviewed-on: http://review.couchbase.org/88093
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...


# fdd7414e 09-Jan-2018 Jim Walker <jim@couchbase.com>

MB-25631: Log more details about compaction

Add code so that we can track (via logging)

* tombstones purged
* items erased by collections
* pre/post size, items, deleted ite

MB-25631: Log more details about compaction

Add code so that we can track (via logging)

* tombstones purged
* items erased by collections
* pre/post size, items, deleted items and purge seqno

Change-Id: Iec6310f1a6534dc86e8ea88bceaf91eb9078789f
Reviewed-on: http://review.couchbase.org/87606
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v5.1.0, v5.0.0
# 1f8466c9 02-Aug-2017 Dave Rigby <daver@couchbase.com>

MB-27328: Support calling fsync() periodically during compaction

Add a new config param 'fsync_after_every_n_bytes_written'. When set
to a non-null value, configures couchstore to issue

MB-27328: Support calling fsync() periodically during compaction

Add a new config param 'fsync_after_every_n_bytes_written'. When set
to a non-null value, configures couchstore to issue a sync() after
every N bytes of compaction data is written.

The default value has been set at 16MB - local testing with 1.5GB
vBucket files suggests this is a good tradeoff between bg fetch
latency and compaction write throughput:

avg_bg_wait_time (no compaction): 50-100μs
avg_bg_wait_time (compaction, 16MB): 50-200μs

For comparision, when run with no automatic sync():

avg_bg_wait_time (compactionB): 100,000-300,000μs

The value can be configured at runtime via epctl - for example to set
a 1MB period:

cbepctl -u Administrator -p asdasd localhost:12000 -b <bucket> set flush_param fsync_after_every_n_bytes_written 1048576

Note: Only the compactor is currently affected - other writers (access
log, flusher) don't yet use this parameter.

Change-Id: Iedfd8f7237ac4b5f719d680f39edd1c3eede982c
Reviewed-on: http://review.couchbase.org/81831
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...


# 000282bd 12-Dec-2017 Jim Walker <jim@couchbase.com>

MB-16181: Remove dead KVStore::persistCollectionsManifestItem

Function added and implemented in CouchKVStore - never called.

Change-Id: I6c6bbd111bb4aa5660c83fd201bd9f111ea7c01b

MB-16181: Remove dead KVStore::persistCollectionsManifestItem

Function added and implemented in CouchKVStore - never called.

Change-Id: I6c6bbd111bb4aa5660c83fd201bd9f111ea7c01b
Reviewed-on: http://review.couchbase.org/86717
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# b086999a 13-Dec-2017 Jim Walker <jim@couchbase.com>

MB-27215: Clean-up KVStore keyStats/kvstats

Found that the publically declared structure is only ever used by
private methods of couch-kvstore, further to that parts of the
kvstats_e

MB-27215: Clean-up KVStore keyStats/kvstats

Found that the publically declared structure is only ever used by
private methods of couch-kvstore, further to that parts of the
kvstats_entry are written but never read, e.g. the second part
of the std::pair and the vbucket member of the KeyStats structure.

Note:
The internal std::unordered_map (now maps key to exists bool) could be
enhanced further to be a std::unordered_set storing only the existing
keys, a future change may need to map key to more than just bool.

Change-Id: I374ce15f5da3f8248cff68fa7dc45545054b6f53
Reviewed-on: http://review.couchbase.org/86770
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 61d9ac92 04-Dec-2017 Dave Rigby <daver@couchbase.com>

MB-26021: Add StatusCallback<> subclass

Refactor the Callback<> template so the statusCode is moved into a
seperate subclass - StatusCallback.

This shrinks the size of Callback<

MB-26021: Add StatusCallback<> subclass

Refactor the Callback<> template so the statusCode is moved into a
seperate subclass - StatusCallback.

This shrinks the size of Callback<> (and any subclasses) which don't
need the status code by at least 4 bytes (can be more due to
alignment).

Specifically, this reduces PersistenceCallback from 48 to 32 bytes.

Change-Id: I1866d0cc618af55a3d7c5809fcd768504f7204c7
Reviewed-on: http://review.couchbase.org/86323
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# bc646842 01-Dec-2017 Dave Rigby <daver@couchbase.com>

MB-26021: Add TransactionContext to KVStore [1/2]

When commiting a transaction in the KVStores, various state needs to
be maintained to ensure that ep-engine is called back when items ha

MB-26021: Add TransactionContext to KVStore [1/2]

When commiting a transaction in the KVStores, various state needs to
be maintained to ensure that ep-engine is called back when items have
successfully been persisted to disk. This is typically handled by the
store maintaining a container of PersistenceCallback objects to be
invoked when the transactionis commited. However much of the state of
PersistenceCallback is identical across all operations in a commit -
for example the VBucket, the Stats object to update.

To reduce the memory cost of maintaing this state, introduce a new
TransactionContext object. Callers of KVStore pass an instance of this
into begin(), and it is owned by the KVStore until commit() or
rollback() is called. Upon commit, all PersistenceCallacks are invoked
with a reference to the TransactionContext; allowing them to make use
of the state stored there.

This initial patch just adds the instructure for the
TransactionContext; it is not yet used by ep-engine.

Change-Id: I8c3704c968270a871849e9c6b5efda22b221d79b
Reviewed-on: http://review.couchbase.org/86285
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 99073f39 30-Nov-2017 Dave Rigby <daver@couchbase.com>

MB-26021: Change persistenceCb from list to deque<unique_ptr>

This both simplies the implementation (no manual deletion needed); and
uses fewer resources - overhead of just 1 ptr per ele

MB-26021: Change persistenceCb from list to deque<unique_ptr>

This both simplies the implementation (no manual deletion needed); and
uses fewer resources - overhead of just 1 ptr per element, compared to
3 for a std::list.

Change-Id: If5dfc345062614fc864e48ddf17037a0e012b67b
Reviewed-on: http://review.couchbase.org/86200
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...


# 6335af99 16-Nov-2017 Daniel Owen <owend@couchbase.com>

MB-26110: Add compaction failure count to KVStore stats

To help with auto-failover of disk issues, we want to add a statistic
for compaction failures. This is provided on a per shard ba

MB-26110: Add compaction failure count to KVStore stats

To help with auto-failover of disk issues, we want to add a statistic
for compaction failures. This is provided on a per shard basis similar
to other KVStore stats.

A subseqent patch will provide an aggregate of this and other KVStore
stats for use by ns_server.

Change-Id: I657ce17423439cf150e0a4705450331aa02b22db
Reviewed-on: http://review.couchbase.org/85642
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 8b25bbfd 24-Oct-2017 Eugen-Alexandru Virtan <eugen.virtan@couchbase.com>

MB-26047:[7-a] Change all stats histograms to use <chrono> durations

Part of a cross repo change.

Replace Histogram<hrtime_t> with MicrosecondHistogram and refactor
realted code

MB-26047:[7-a] Change all stats histograms to use <chrono> durations

Part of a cross repo change.

Replace Histogram<hrtime_t> with MicrosecondHistogram and refactor
realted code to work with durations.
Change various add_stat methods to accomodate the new changes to
templates introduced in histogram.h by the new patch in platform.

Change-Id: Ic061e11a79b09663b980e173dc4191d46d5aa35d
Reviewed-on: http://review.couchbase.org/84743
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>

show more ...


# 9f674027 19-Oct-2017 Paolo Cocchi <paolo.cocchi@couchbase.com>

MB-26431 [RocksDB] Fix 'ep_testsuite' tests

The 'RocksDBKVStore::getSeqnoSlice(const int64_t& seqno)' uses the
address of 'seqno' in its definition.
The function breaks when a 'uint6

MB-26431 [RocksDB] Fix 'ep_testsuite' tests

The 'RocksDBKVStore::getSeqnoSlice(const int64_t& seqno)' uses the
address of 'seqno' in its definition.
The function breaks when a 'uint64_t' is passed in input, because
the implicit conversion from 'uint64_t' to 'int64_t' creates a
temporary, which is the actual argument passed to the function.
That causes a number of 'ep_testsuite' tests to fail on Jenkins
'kv_engine-linux-master-rocks' build.
This can happen because binding a rvalue to 'reference to const' is
legal in C++. Changing the function to take a 'const int64_t*' in input
fixes the issue and avoids any further misuse of a 'reference to const'
argument.
This change fixes the following 'ep_testsuite' tests:
- ep_testsuite.value_eviction.rocksdb
- ep_testsuite.full_eviction.rocksdb
- ep_testsuite_basic.value_eviction.rocksdb
- ep_testsuite_dcp.value_eviction.rocksdb
- ep_testsuite_dcp.full_eviction.rocksdb

Change-Id: I5202cfd864de18b2d425d70626534dde03e39e0d
Reviewed-on: http://review.couchbase.org/84847
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 8789b9c0 09-Oct-2017 Eugen-Alexandru Virtan <eugen.virtan@couchbase.com>

MB-26047:[4] Migrate away from using gethrtime() (ep)

Replace uses of gethrtime() with ProcessClock::now() and refactor
related code to use typesafe <chrono> time units.

Change-

MB-26047:[4] Migrate away from using gethrtime() (ep)

Replace uses of gethrtime() with ProcessClock::now() and refactor
related code to use typesafe <chrono> time units.

Change-Id: Idb1db284f469543d0ae3b3828d5ffbdf4e19d2e3
Reviewed-on: http://review.couchbase.org/84156
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# a33e0978 20-Sep-2017 Jim Walker <jim@couchbase.com>

MB-25342: Erase deleted collection keys during in compaction

Add a new call back (using std::function) so that the compactor can
check if a key belongs to a deleted collection and should

MB-25342: Erase deleted collection keys during in compaction

Add a new call back (using std::function) so that the compactor can
check if a key belongs to a deleted collection and should be dropped
from the database.

Change-Id: Iebefd385cfcae38745c81563113c025bba90890e
Reviewed-on: http://review.couchbase.org/83559
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# c7739627 11-Sep-2017 Dave Rigby <daver@couchbase.com>

RocksDB: Fix memory leak in ~RocksDBKVStore

The destructor of RocksDBKVStore was not deleting the elements from
cachedVBStates, causing a memory leak on shutdown.

To solve this,

RocksDB: Fix memory leak in ~RocksDBKVStore

The destructor of RocksDBKVStore was not deleting the elements from
cachedVBStates, causing a memory leak on shutdown.

To solve this, change cachedVBStates from a non-owning vector to
std::vector<std::unique_ptr<vbucket_state>> so cleanup is handled
automatically.

As a side-effect, this highlighted that the KVStore & CouchKVStore
copy constructors were buggy, as they were not perfoming a deep-copy
of cachedVBStates (and hence would crash if one the objects was
deleted. Fortunatley nothing actually needed to copy CouchKVStore
objects, so just delete those constructors.

Change-Id: If8d3b659ac24f0f1820c08b7c5e3a66c0aa7980d
Reviewed-on: http://review.couchbase.org/83261
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 37758b14 30-Aug-2017 Sriram Ganesan <sriram@couchbase.com>

MB-25875: Make getMultiFsReadCount into Couchbase::RelaxedAtomic<size_t>

A data race was observed while trying to read the getMultiFsReadCount
between the background fetcher and the stat

MB-25875: Make getMultiFsReadCount into Couchbase::RelaxedAtomic<size_t>

A data race was observed while trying to read the getMultiFsReadCount
between the background fetcher and the statistics snapshot task

WARNING: ThreadSanitizer: data race (pid=9742)
Read of size 8 at 0x7d8800012160 by thread T19:
Previous write of size 8 at 0x7d8800012160 by thread T15:

Change-Id: I0df032c89d11163937cbab9625d6572b193bd7f2
Reviewed-on: http://review.couchbase.org/82926
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 5e08ba29 24-Aug-2017 Dave Rigby <daver@couchbase.com>

Move callbacks impl to callbacks.cc

Move implementation details of classes in Callbacks to a new file
callbacks.cc. Similar to previous changes, this reduces the number of
files whic

Move callbacks impl to callbacks.cc

Move implementation details of classes in Callbacks to a new file
callbacks.cc. Similar to previous changes, this reduces the number of
files which need to include the implementation details of callbacks.h;
specifically item.h which indirectly pulls in a number of header
files.

Fix up missing / incorrect #includes as a result of this.

Change-Id: Icc565894b746e688d0c59a532e7ff98e22683959
Reviewed-on: http://review.couchbase.org/82686
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# e04af9d0 24-Aug-2017 Dave Rigby <daver@couchbase.com>

KVStore: move impl-only types to kvstore_priv.h

kvstore.h currently contains two classes of code - (1) the interface
for using KVStore subclasses, and (2) types and code used by
subc

KVStore: move impl-only types to kvstore_priv.h

kvstore.h currently contains two classes of code - (1) the interface
for using KVStore subclasses, and (2) types and code used by
subclasses and to implement KVStore. This results in unnecessary code
being pulled in to files which just want to /use/ KVStore, and don't
need to implement it.

As such, create a new private header - kvstore_priv.h - to contain the
implementation details and move implementaiton code to this file.

Change-Id: Icfd721ffbb39005d7392f27806882d4b4ab97bc1
Reviewed-on: http://review.couchbase.org/82683
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# bafd9a07 24-Aug-2017 Dave Rigby <daver@couchbase.com>

Move KVStoreConfig from kvstore.{h,cc} to kvstore_config.{h,cc}

A number of files which currently #include kvstore.h only actually
need the declarion of KVStoreConfig.

Given tha

Move KVStoreConfig from kvstore.{h,cc} to kvstore_config.{h,cc}

A number of files which currently #include kvstore.h only actually
need the declarion of KVStoreConfig.

Given that kvstore.h is relatively large, move KVStoreConfig to it's
own header / source file to reduce the amount of headers other files
need to include to use it.

Change-Id: I836ac199fd8d8ae8af83193ea69f3a8cf8fccb5f
Reviewed-on: http://review.couchbase.org/82682
Reviewed-by: Oliver Downard <oliver.downard@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 32797636 21-Aug-2017 Dave Rigby <daver@couchbase.com>

MB-25661: Statistics for background fetch read amplification

Add new statistics to ep-engine related to disk reads compared to the
data being read:

* ep_io_bg_fetch_read_count -

MB-25661: Statistics for background fetch read amplification

Add new statistics to ep-engine related to disk reads compared to the
data being read:

* ep_io_bg_fetch_read_count - Number of filesystem read()s issued
while fetching documents from disk.

Using this statistic, we can derive a Read Amplification value for
background fetches:

* ep_bg_fetch_read_amplification - Ratio of filesystem read()s to
number of documents fetched.

This can be plotted (e.g. with mortimer) to see the change over time,
but for convenience a statistic is provided to give the overall ratio
since last stat reset.

Change-Id: I7c77722ddccfb860ede6f159e04ffc7c814c1b34
Reviewed-on: http://review.couchbase.org/82569
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 45fa1633 21-Aug-2017 Dave Rigby <daver@couchbase.com>

MB-25661: Rename io_read_bytes/num_read to io_bg_fetch_doc_bytes/num_read

To more accuratly reflect what is represents, modify io_read_bytes and
io_num_read:

* Rename io_read_by

MB-25661: Rename io_read_bytes/num_read to io_bg_fetch_doc_bytes/num_read

To more accuratly reflect what is represents, modify io_read_bytes and
io_num_read:

* Rename io_read_bytes to io_bg_fetch_doc_bytes, and io_num_read to
io_bg_fetch_docs_read, as these statistics refer to reads done during
background fetch (and not for example during Flushing).

* Include the size of document metadata in io_bg_fetch_doc_bytes to
more accurately reflect the data read from disk.

Change-Id: I9303fb7bf61a4dfc039bc5db234d47bbaf7da88e
Reviewed-on: http://review.couchbase.org/82568
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# a52dad0f 02-Aug-2017 Dave Rigby <daver@couchbase.com>

MB-25509: Support calling fsync() periodically during compaction

Add a new config param 'fsync_after_every_n_bytes_written'. When set
to a non-null value, configures couchstore to issue

MB-25509: Support calling fsync() periodically during compaction

Add a new config param 'fsync_after_every_n_bytes_written'. When set
to a non-null value, configures couchstore to issue a sync() after
every N bytes of compaction data is written.

The default value has been set at 16MB - local testing with 1.5GB
vBucket files suggests this is a good tradeoff between bg fetch
latency and compaction write throughput:

avg_bg_wait_time (no compaction): 50-100μs
avg_bg_wait_time (compaction, 16MB): 50-200μs

For comparision, when run with no automatic sync():

avg_bg_wait_time (compactionB): 100,000-300,000μs

The value can be configured at runtime via epctl - for example to set
a 1MB period:

cbepctl -u Administrator -p asdasd localhost:12000 -b <bucket> set flush_param fsync_after_every_n_bytes_written 1048576

Note: Only the compactor is currently affected - other writers (access
log, flusher) don't yet use this parameter.

Change-Id: Iedfd8f7237ac4b5f719d680f39edd1c3eede982c
Reviewed-on: http://review.couchbase.org/81759
Reviewed-by: Will Gardner <willg@rdner.io>
Tested-by: Build Bot <build@couchbase.com>

show more ...


12345678910