MB-29675: Move ObjectRegistry stats to CoreLocalStats

ObjectRegistry updates stats on the number and size of various object
classes whenever those classes are constructed/destructed. Cur

MB-29675: Move ObjectRegistry stats to CoreLocalStats

ObjectRegistry updates stats on the number and size of various object
classes whenever those classes are constructed/destructed. Currently
it updates a single set of stats per Bucket. However, these statistics
are highly contended by all the frontend worker threads; as such the
various functions to update the stats show high in profiling.

For example during 20/80 R/W pillowfight run, a representative worker
thread shows the following:

Samples: 205K of event 'cycles:ppp', Event count (approx.): 110332199760, Thread: mc:worker_16
Overhead Command Shared Object Symbol
1.52% mc:worker_16 ep.so [.] ObjectRegistry::onCreateBlob
1.43% mc:worker_16 ep.so [.] ObjectRegistry::onCreateItem
1.04% mc:worker_16 ep.so [.] ObjectRegistry::onDeleteItem
0.59% mc:worker_16 ep.so [.] ObjectRegistry::onDeleteBlob

Approx ~4.5% of the thread's runtime is spent in stats
updating. Examining the asm-level, a significant number of the cycles
is spent on the atomic updates to those counters; which appears to be
due to cacheline contention.

Given these stats are frequerntly updated but rarely read, we can
reduce this cost by moving these hotly-contended counters into the new
CoreLocalStats class, so each core has it's own instance to write to.

In microbenchmarks this reduces the cost to create/delete a
StoredValue by ~20%:

Before:

Run on (8 X 2300 MHz CPU s)
2018-05-29 17:00:45
---------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------------------
VBucketBench/CreateDeleteStoredValue/threads:16 229 ns 1954 ns 368128

After:

Run on (8 X 2300 MHz CPU s)
2018-05-29 17:02:54
---------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------------------
VBucketBench/CreateDeleteStoredValue/threads:16 209 ns 1559 ns 461152

Change-Id: I668b4fb54997e32e9cd9229fcfaf5d411b11305d
Reviewed-on: http://review.couchbase.org/94853
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-29707: Add checkpoint memory overhead stats

Change-Id: If1e8666d043d76a1fae64f2e7909d36be24790e9
Reviewed-on: http://review.couchbase.org/94307
Tested-by: Build Bot <build@couchba

MB-29707: Add checkpoint memory overhead stats

Change-Id: If1e8666d043d76a1fae64f2e7909d36be24790e9
Reviewed-on: http://review.couchbase.org/94307
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-29675: Change Task::getDescription() back to std::string

Task::getDescription() currently returns a non-owning string
(const_char_buffer), instead of an owning std::string. This was c

MB-29675: Change Task::getDescription() back to std::string

Task::getDescription() currently returns a non-owning string
(const_char_buffer), instead of an owning std::string. This was chocen
to minimise string copying as getDescription was called multiple times
per task execution (for example for taskLog / slowTaskLog - see
d26ea7d1f12aa021e74663a304d81d1f9e512097 which originally changed to
const_char_buffer.

However now taskLog / slowTaskLog have been removed, we will no longer
call getDescription anywhere near as frequently - short-running tasks
will likely never have their description requested. As such
pre-calculating the description (and having to ensure it's lifetime is
as long as any caller requesting it) is now less efficient than simply
generating it on-demand.

Change getDescription back to std::string, only creating a string
object on demand when it is requested.

Change-Id: I5f003b76d0ac73b2abc3ac5cd3d985a6ceb867c3
Reviewed-on: http://review.couchbase.org/94283
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-29040: [2/2] Sanitise delete with value via DcpConsumer

1) Allow a wider range of datatypes to be received by adjusting the
mcbp validator.

2) Update the DcpConsumer so that

MB-29040: [2/2] Sanitise delete with value via DcpConsumer

1) Allow a wider range of datatypes to be received by adjusting the
mcbp validator.

2) Update the DcpConsumer so that delete with values are checked and
sanitised, this means even a deleted marked as 'xattr' needs checking
because the source may be sending an incorrect xattr with raw body +
user xattrs which will need deleting.

3) Add a DcpConsumer test which checks we strip the faulty input.

Change-Id: I219f21df9a63bc6b1c004fa382bd1f32f94a3e90
Reviewed-on: http://review.couchbase.org/93041
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-29240: Add checkpoint memory stats to 'stats all'

Change-Id: Idb9d80b23b34ad22114d0c52f62b7966bd97ed8d
Reviewed-on: http://review.couchbase.org/92619
Tested-by: Build Bot <build@c

MB-29240: Add checkpoint memory stats to 'stats all'

Change-Id: Idb9d80b23b34ad22114d0c52f62b7966bd97ed8d
Reviewed-on: http://review.couchbase.org/92619
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-29040: [1/2] Correctly strip user xattrs

A bug in the expiry path means a document with user xattrs is not
having the body removed. Now when we rebalance and such a deleted
docume

MB-29040: [1/2] Correctly strip user xattrs

A bug in the expiry path means a document with user xattrs is not
having the body removed. Now when we rebalance and such a deleted
document is transferred, it fails validation because the datatype
is xattr and json.

Change-Id: Ie5a71e6fc640bfd614c8bbd68c91b99fbdf8aed2
Reviewed-on: http://review.couchbase.org/92616
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-28941: Make tracing of BG_WAIT/LOAD cheaper by reusing timestamps

The session tracing for bgfetch currently has a couple of issues:

1. It takes it's own timings of the span of BG

MB-28941: Make tracing of BG_WAIT/LOAD cheaper by reusing timestamps

The session tracing for bgfetch currently has a couple of issues:

1. It takes it's own timings of the span of BG fetch, which is costly.

2. As a result of taking it's own timings, it results in different
durations compared to the existing bgWait / bgLoad histograms.

Address both of these issues by re-using the timestamps already
recorded for bgLoadHisto / bgWaitHisto; and splitting into two events
(one corresponding to bgLoad; the other to bgWait).

Change-Id: I1cc21fadbf89d7c268270d85abb233d64c99cb67
Reviewed-on: http://review.couchbase.org/92550
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>

show more ...

MB-28631: Add statistic to compute uncompressed item size

Adding this statistic to the hash table would allow us to
compare item memory size due to compression and the memory
size if

MB-28631: Add statistic to compute uncompressed item size

Adding this statistic to the hash table would allow us to
compare item memory size due to compression and the memory
size if the data was stored as uncompressed.

Change-Id: I85f10e6eef5d3ffd434b214c27dc2293f509dac2
Reviewed-on: http://review.couchbase.org/91947
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Add getNumOfVBucketsInState method

Adds functionality to KV buckets to be able to retrieve the number
of underlying vbuckets that are in a given state, i.e. active, replica,
dead, an

Add getNumOfVBucketsInState method

Adds functionality to KV buckets to be able to retrieve the number
of underlying vbuckets that are in a given state, i.e. active, replica,
dead, and pending.

This functionality will be used in the future by the hifi_mfu hash
table eviction algorithm to determine the percentage of replica
documents to evict.

Change-Id: Id19e3d28ca3c99306dddc12b7dfe8ee50d84926c
Reviewed-on: http://review.couchbase.org/92042
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-28941: Tracing: Remove begin/end indirection via TRACING_API

To record a trace start/end, currently the following indirection
occurs (starting from the ScopedTracer RAII object):

MB-28941: Tracing: Remove begin/end indirection via TRACING_API

To record a trace start/end, currently the following indirection
occurs (starting from the ScopedTracer RAII object):

ScopedTracer -> serverAPI.begin_trace -> cookie.getTracer().begin()

However, the indirection via serverAPI is essentially pointless - the
actual Tracer object is owned by the Cookie, so it's unnecessary to
bounce via the Server API to access it.

Remove this intermediate step. This results in the following ~7% speedup
(aside from the overall simplification):

Before:

Run on (8 X 2300 MHz CPU s)
2018-03-29 14:47:00
------------------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------------------
SessionTracingRecordMutationSpan 502 ns 500 ns 1331153

After:

Run on (8 X 2300 MHz CPU s)
2018-03-29 14:55:33
------------------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------------------
SessionTracingRecordMutationSpan 467 ns 467 ns 1458251

Change-Id: I331045a264bff1873f96410056e803fa357bbae1
Reviewed-on: http://review.couchbase.org/91836
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove default parameter values from VBCBAdaptor

The VBCBAdaptor constructor take multiple parameters with the last two
(sleep and shutdown) being given default parameters. The sleep

Remove default parameter values from VBCBAdaptor

The VBCBAdaptor constructor take multiple parameters with the last two
(sleep and shutdown) being given default parameters. The sleep
parameter is always provided and therefore there is no point in
providing a default value.

The VBCBAdaptor constructor is only invoked from two places. In one
case the shutdown parameter is provided (true). In the other case the
default value (false) is used. Instead of relying on the default
value we can just pass in false. This is useful for a future patch that
extends the VBCBAdaptor constructor parameters.

Change-Id: I1b5611a81ffdacc0464b67cff08571e279f2b68f
Reviewed-on: http://review.couchbase.org/92023
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-28230: Move active compression to a separate ItemCompressorTask

The DefragmenterTask operates only on a smaller chunk of items at
a time, so that the hash table lock isn't held for to

MB-28230: Move active compression to a separate ItemCompressorTask

The DefragmenterTask operates only on a smaller chunk of items at
a time, so that the hash table lock isn't held for too long that
will block front-end threads. Also, the defragmenter operates on a
10 second interval which is not active enough for active compression.
Moving it to a separate task would also allow for further customizations
and optimizations on when the task can get scheduled.

Run on (8 X 2500 MHz CPU s)
2018-03-23 08:58:47
———————————————————————————————————————————————————————————————————————————————---
Benchmark Time CPU Iterations
UserCounters…——————————————————————————————————————————————————————————————————----
ItemCompressorBench/Visit/0 6941595 ns 6937531 ns 96 6.8733M items/s ValueOnly
ItemCompressorBench/Visit/1 6652186 ns 6651641 ns 103 7.16871M items/s FullEviction
—————————————————————————————————————————————————————————————————————————————————--

Change-Id: Ieed20fc4985936663595c4116504fc8e44c4e043
Reviewed-on: http://review.couchbase.org/90996
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-28737: UBSan: Fix integer overflows

UBSan flags a number of cases where we overflow an integer type - for
example:

signed integer overflow: 1073741824 * 1024 cannot be re

MB-28737: UBSan: Fix integer overflows

UBSan flags a number of cases where we overflow an integer type - for
example:

signed integer overflow: 1073741824 * 1024 cannot be represented in type 'int'
#0 0x1029596c0 in parse_config(char const*, config_item*, __sFILE*) config_parser.cc:131

Change-Id: Ifc762bb25b598ce1db2bdee58b1636ff87436bd1
Reviewed-on: http://review.couchbase.org/91017
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-27661 [9/n]: Fix MSVC warnings - C4101, C4273, C4279, C4312

C4101 - identifier : unreferenced local variable. The local variable is
never used.
C4273 - function : inconsis

MB-27661 [9/n]: Fix MSVC warnings - C4101, C4273, C4279, C4312

C4101 - identifier : unreferenced local variable. The local variable is
never used.
C4273 - function : inconsistent DLL linkage. Two definitions in a file
differ in their use of dllimport.
C4297 - function : function assumed not to throw an exception but does.
C4312 - operation : conversion from 'type1' to 'type2' of greater size.
This warning detects an attempt to assign a 32-bit value to a
64-bit pointer type.

Change-Id: Ib025ccea825b8ae49c27d17483b314f2fa5701eb
Reviewed-on: http://review.couchbase.org/90059
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-28327: Initialize all storedValue with correct frequency count

Changed the constructor of an Item from taking a NRU value, which
almost always uses the default value, to instead be in

MB-28327: Initialize all storedValue with correct frequency count

Changed the constructor of an Item from taking a NRU value, which
almost always uses the default value, to instead be initialized directly
to INITIAL_NRU_VALUE. Then in the two places that we used to initialize
with a different value, we explictly set the NRU value.

Wth the above change in place, we initialise the FrequencyCounter in a
similar manner to the NRU value. This ensures that storedValue is
initialized with the correct frequency count.

The frequency counter should be initialised to
ItemEviction::initialFreqCounter.

Change-Id: I34096cf72290298e8bf8bc84665778b3d7208690
Reviewed-on: http://review.couchbase.org/90227
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Use unique_ptr for Configuration::addValueChangedListener

addValueChangedListener() takes ownership of the passed-in listener
object, but only if the specified key exists in the config.

Use unique_ptr for Configuration::addValueChangedListener

addValueChangedListener() takes ownership of the passed-in listener
object, but only if the specified key exists in the config. If not,
then it results in a memory-leak as the listener object is passed as a
raw pointer.

Fix this problem (and in general make resource ownership more
explicit) by passing by a unique_ptr<>.

Change-Id: Id2007cee95d1ea0ee46abd420e5ab693ecd1492e
Reviewed-on: http://review.couchbase.org/89664
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-22010: Create ItemFreqDecayerTask

Creates an ItemFreqDecayerTask on the initialization of a KVBucket.
It is used to ensure that the frequency counters of items stored in the
hash

MB-22010: Create ItemFreqDecayerTask

Creates an ItemFreqDecayerTask on the initialization of a KVBucket.
It is used to ensure that the frequency counters of items stored in the
hash table do not all become saturated. When the task runs it will then
snooze for int max and will only be woken up when the frequency counter
of an item in the hash table becomes saturated.

Change-Id: I9ae51dfa6717c6349e43ddb08dde0759e43ca16b
Reviewed-on: http://review.couchbase.org/88761
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Remove ForestKVStore

Remove support for ForestDB in KV-Engine. This code has already
bit-rotted (doesn't currently compile).

Change-Id: I3d4679da46d1b105be0b234f2193033b3edc3866

Remove ForestKVStore

Remove support for ForestDB in KV-Engine. This code has already
bit-rotted (doesn't currently compile).

Change-Id: I3d4679da46d1b105be0b234f2193033b3edc3866
Reviewed-on: http://review.couchbase.org/88863
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-24860: Make mem_used_merge_threshold_percent dynamic

Makes changes so that set_param (cbepctl) can change this value.

Change-Id: Ia9b763aca763c5dc605b94d5a21f282b017e2218
Rev

MB-24860: Make mem_used_merge_threshold_percent dynamic

Makes changes so that set_param (cbepctl) can change this value.

Change-Id: Ia9b763aca763c5dc605b94d5a21f282b017e2218
Reviewed-on: http://review.couchbase.org/88765
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-24860: Rename getTotalMemUsed to getEstimatedTotalMemoryUsed

The name better suits what the function returns, the value may read
more or less than the real amount allocation (with tha

MB-24860: Rename getTotalMemUsed to getEstimatedTotalMemoryUsed

The name better suits what the function returns, the value may read
more or less than the real amount allocation (with that +/-
controlled by the mem_used_merge_threshold_percent).

Change-Id: I7f82775b2bab9de9a25064dc7ea8660555653792
Reviewed-on: http://review.couchbase.org/88611
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-24860: Make mem_used merge threshold a percent of max_size

Update stats so that the per core allocated merge threshold is
a percentage of max_size, thus mem_used should have an error

MB-24860: Make mem_used merge threshold a percent of max_size

Update stats so that the per core allocated merge threshold is
a percentage of max_size, thus mem_used should have an error
bar +/- that percentage of max_size.

Change-Id: I16e97ea4eba3c0f129644552e2af452efda40d01
Reviewed-on: http://review.couchbase.org/88610
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>

show more ...

Remove trailing newline in log comments

The previous logger trimmed away the (extra) trailing newline,
but with spdlogger it returns in an extra blank line.

Change-Id: Idc298fb1

Remove trailing newline in log comments

The previous logger trimmed away the (extra) trailing newline,
but with spdlogger it returns in an extra blank line.

Change-Id: Idc298fb1447433e84ea88e59429e533547d1d5b5
Reviewed-on: http://review.couchbase.org/88856
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-24860: Switch memory tracking to be more read efficient

The initial CoreStore integration makes getTotalMemUsed a more heavy
weight function compared to the TLS version. The old getTo

MB-24860: Switch memory tracking to be more read efficient

The initial CoreStore integration makes getTotalMemUsed a more heavy
weight function compared to the TLS version. The old getTotalMemUsed
would always aggregate the current core counters, so means a loop
and some cache crossing.

This patch reinstates a similar technique used in TLS which means that
the read of mem_used is 'cheap', a plain and 'global' relaxed atomic
is read and now the write side (memAlloc/memDealloc) have a merge
threshold. When the core counter accumulates a abs(value) which
exceeds the threshold, the core value is accumulated into the
'global', thus getTotalMemUsed returns a value which may lag the real
total, but that matches more closely the TLS behaviour.

Note1: the original TLS code had an updates threshold and a bytes
allocated threshold, only the bytes allocated threshold remains as
doing n 1 byte allocations doesn't really need to trigger a merge.

Note2: the configuration has been updated with
mem_used_merge_threshold only so that some of ep_testsuite can pass
with this commit as they rely on mem_used being more 'sensitive' than
it currently is. Upstream a percentage threshold will replace this
hard threshold and later mem_used will be driven by
EPStats::getPreciseTotalMemoryUsed rather than getTotalMemory which
lags behind.

An updated benchmark has been written and the original TLS code was
re-benchmarked with a similar benchmark.

Test AllocNRead1 (write heavy test and benchmark output trimmed):

TLS code (github/jimwwalker/kv_engine @ daa00adb):

Run on (24 X 1232.7 MHz CPU s)
2018-01-31 09:03:48
-------------------------------------------------------------------
Benchmark Time CPU Iterations
-------------------------------------------------------------------
AllocNRead1/0/threads:96 0 ns 4 ns 163069344
AllocNRead1/1/threads:96 0 ns 46 ns 15367776
AllocNRead1/2/threads:96 1 ns 87 ns 8211072
AllocNRead1/4/threads:96 2 ns 166 ns 4326816
AllocNRead1/8/threads:96 4 ns 333 ns 2219808
AllocNRead1/16/threads:96 7 ns 635 ns 1079040
AllocNRead1/32/threads:96 13 ns 1268 ns 561216
AllocNRead1/64/threads:96 27 ns 2477 ns 282048
AllocNRead1/128/threads:96 60 ns 5569 ns 139872
AllocNRead1/256/threads:96 118 ns 11096 ns 66432
AllocNRead1/512/threads:96 213 ns 20396 ns 34656
AllocNRead1/1024/threads:96 483 ns 45152 ns 17760
AllocNRead1/2048/threads:96 884 ns 84550 ns 7968
AllocNRead1/4000/threads:96 1851 ns 169866 ns 4032

CoreStore (i.e. this commit):

Run on (24 X 1237.67 MHz CPU s)
2018-02-01 14:18:38
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------
Benchmark Time CPU Iterations
-------------------------------------------------------------------
AllocNRead1/0/threads:96 0 ns 5 ns 126621984
AllocNRead1/1/threads:96 0 ns 41 ns 17480736
AllocNRead1/2/threads:96 1 ns 87 ns 8632128
AllocNRead1/4/threads:96 2 ns 168 ns 3970560
AllocNRead1/8/threads:96 4 ns 352 ns 2025696
AllocNRead1/16/threads:96 7 ns 684 ns 1026528
AllocNRead1/32/threads:96 15 ns 1389 ns 512832
AllocNRead1/64/threads:96 29 ns 2735 ns 259584
AllocNRead1/128/threads:96 63 ns 5915 ns 125568
AllocNRead1/256/threads:96 119 ns 10876 ns 63936
AllocNRead1/512/threads:96 267 ns 23952 ns 31776
AllocNRead1/1024/threads:96 524 ns 47242 ns 14688
AllocNRead1/2048/threads:96 967 ns 92247 ns 7584
AllocNRead1/4000/threads:96 1977 ns 184717 ns 4032

AllocNReadM (read heavy output trimmed):

Test shown does 512 or 1024 writes and increasing reads (128, 256 upto
4000).

TLS code (github/jimwwalker/kv_engine @ daa00adb):

Run on (24 X 1199.97 MHz CPU s)
2018-01-31 11:17:57
-------------------------------------------------------------------
Benchmark Time CPU Iterations
-------------------------------------------------------------------
AllocNReadM/512/128/threads:96 257 ns 23747 ns 27936
AllocNReadM/512/256/threads:96 265 ns 24671 ns 30528
AllocNReadM/512/512/threads:96 286 ns 25831 ns 30144
AllocNReadM/512/1024/threads:96 271 ns 25536 ns 26208
AllocNReadM/512/2048/threads:96 267 ns 24820 ns 23904
AllocNReadM/512/4000/threads:96 367 ns 34593 ns 19488
AllocNReadM/1024/128/threads:96 476 ns 45633 ns 13248
AllocNReadM/1024/256/threads:96 475 ns 44718 ns 14976
AllocNReadM/1024/512/threads:96 460 ns 43996 ns 13536
AllocNReadM/1024/1024/threads:96 626 ns 58202 ns 9600
AllocNReadM/1024/2048/threads:96 526 ns 49646 ns 11616
AllocNReadM/1024/4000/threads:96 590 ns 55348 ns 13056

CoreStore (i.e. this commit):

Run on (24 X 1203.98 MHz CPU s)
2018-01-31 11:18:07
------------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------------
AllocNReadM/512/128/threads:96 260 ns 24193 ns 25344
AllocNReadM/512/256/threads:96 276 ns 25273 ns 28128
AllocNReadM/512/512/threads:96 295 ns 25984 ns 26880
AllocNReadM/512/1024/threads:96 306 ns 27264 ns 28896
AllocNReadM/512/2048/threads:96 328 ns 28684 ns 25824
AllocNReadM/512/4000/threads:96 340 ns 31127 ns 22560
AllocNReadM/1024/128/threads:96 582 ns 53013 ns 10560
AllocNReadM/1024/256/threads:96 540 ns 49227 ns 14496
AllocNReadM/1024/512/threads:96 572 ns 52553 ns 15072
AllocNReadM/1024/1024/threads:96 534 ns 50390 ns 13824
AllocNReadM/1024/2048/threads:96 576 ns 54154 ns 9600
AllocNReadM/1024/4000/threads:96 597 ns 56172 ns 13152

Change-Id: I50fd34f6d40c0da4e102220ae86944a10d075572
Reviewed-on: http://review.couchbase.org/88609
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-27554: [BP] Move flushing code from KVBucket to EPBucket

Originally d440338b81a768e58ccd6237b2d64fae8a8f78ea

flushVBucket and related code is only applicable to EP buckets, not t

MB-27554: [BP] Move flushing code from KVBucket to EPBucket

Originally d440338b81a768e58ccd6237b2d64fae8a8f78ea

flushVBucket and related code is only applicable to EP buckets, not to
Ephemeral buckets. As such, move all the flushing code from KVBucket
to EPBucket class.

Change-Id: Ie0c55e7ff8e67d8ef0a3276bdc20c727ae554b16
Reviewed-on: http://review.couchbase.org/88379
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...