MB-37096: Debug CRT: Reduce ep_perfsuite workload size further

ep_perfsuite tests still runn quite slowly under Windows Debug
CRT, of the order of 600s on CV machines.

Given we

MB-37096: Debug CRT: Reduce ep_perfsuite workload size further

ep_perfsuite tests still runn quite slowly under Windows Debug
CRT, of the order of 600s on CV machines.

Given we only run this in Debug mode for functional testing, reduce the
ize of the workload (vbucket couns, iterations of stats tests) so it
runs in a more reasonable duration.

This change reduces the runtime of ep_perfsuite.value_eviction from 200s
to 20s.

Change-Id: Ib26819f013d66d0cd2dd0c36964aa1d961b26541
Reviewed-on: http://review.couchbase.org/122106
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-37096: Debug CRT: Reduce ep_perfsuite iterations by 20x

ep_perfsuite tests run quite slowly under Windows Debug
CRT, frequently hitting the test suite timeout (1500s) when run on CV

MB-37096: Debug CRT: Reduce ep_perfsuite iterations by 20x

ep_perfsuite tests run quite slowly under Windows Debug
CRT, frequently hitting the test suite timeout (1500s) when run on CV
machines.

Given we only run this in Debug mode for functional testing, reduce the
iteration count by 20x compared to the normal number.

(Note: the code _did_ previously attempt to run at a reduced iteration
count for Debug builds, however it used an #ifdef which is not set on
Windows. Change to use folly's Portability header which handles these
differences.

With this change, ep_perfsuite completes in ~200s on Windows Debug mode.

Change-Id: I22c6fbe5a4eb2b1d877366903cb9535f9e4d027c
Reviewed-on: http://review.couchbase.org/121868
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-37096: Debug CRT: Reduce ep_testsuite default #vbuckets to 4

ep_testsuite tests run quite slowly under Windows Debug CRT, frequently
hitting the test suite timeout when run on CV mach

MB-37096: Debug CRT: Reduce ep_testsuite default #vbuckets to 4

ep_testsuite tests run quite slowly under Windows Debug CRT, frequently
hitting the test suite timeout when run on CV machines.

Profiling highlights that one reason for this is initializing and then
destroying each engine instance, specifically data for all 1024 vBuckets
which _could_ be used, however few fests use more than 1 or 2.

Change the default maximum number of vBuckets in ep_testsuite tests from
1024 to 4. This reduces the runtime of the ep_testsuite suite from 197s
to 134s under Windows Debug config.

Change-Id: I9bd656590189084f09fb3bb16b8d8efa4de31e8a
Reviewed-on: http://review.couchbase.org/121637
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-37013: Update DcpSnapShotMarker V2 to allow for an extra seqno

To address MB-36948 a new seqno is required in the snapshot marker.

This commit redefines the V2 format so that the

MB-37013: Update DcpSnapShotMarker V2 to allow for an extra seqno

To address MB-36948 a new seqno is required in the snapshot marker.

This commit redefines the V2 format so that the V2 extras is a single
version field (1 byte). The 1 byte version field is used to describe how
the packet is to be decoded, in this MB one V2 encoding is created.

V2.0: The 'value' stores all of the snapshot marker data.
* start (u64)
* end (u64)
* flags (u32)
* maxVisibleSeqno (u64)
* highCompletedSeqno (u64)

With this commit only the V1 and V2.0 encodings are transmitted, but the
V2.0 encoding is only currently transmitted for snapshots with the disk
flag set (and when a sync write has occurred in the vbucket). Because
there is no path yet for the DcpProducer to push down the
maxVisibleSeqno, with this commit V2.0 always transmits 0 for the
maxVisibleSeqno.

On the receipt of a V2.0 message, this patch does tunnel the
maxVisibleSeqno up to KV-engine DCP, this is so the consumer can
calculate the correct number of bytes for buffer acknowledgement.

Note on removal of the following from dcp_snapshot_marker_executor.cc

// HCS should never be sent as 0 or a pre-condition will throw in
// the replicas flusher
if (v2Payload->getHighCompletedSeqno() == 0) {
// Not success so just disconnect
connection.setState(StateMachine::State::closing);
return;
}

This disconnect clause is no longer required because analysis of the
ActiveStream::markDiskSnapshot shows that it would never push down a 0
HCS to the Connection::marker send code.

Extended testing of the committed patch was performed using a multi
node cluster and various failover/rebalance steps to check the
disk-snapshot with the v2.0 encoding was transmitted and successfully
received.

Note: further testing was performed where the Connection::marker local
'maxVisibleSeqno' was initialised to a value to confirm that if all
snapshots used the V2.0 encoding, there is no issue.

Change-Id: I886503d6353d01b284b04af730d581f6be6784c9
Reviewed-on: http://review.couchbase.org/118387
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-35930: Add support for providing stat args

Allow the STAT call to contain a value containing a JSON
payload containing "arguments and options" to the given
STAT call.

It

MB-35930: Add support for providing stat args

Allow the STAT call to contain a value containing a JSON
payload containing "arguments and options" to the given
STAT call.

It is up to the stat call to define the schema for the
JSON provided to each stat group.

Change-Id: I8e88cddbcb8e7c9ceea22e368f5cc71238618f20
Reviewed-on: http://review.couchbase.org/114585
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-34017: Add HCS to SnapshotMarker

To correct the replica on disk HCS we need to tell it what pass it
the HCS value from the active when we send a disk snapshot. Add a
HCS field to

MB-34017: Add HCS to SnapshotMarker

To correct the replica on disk HCS we need to tell it what pass it
the HCS value from the active when we send a disk snapshot. Add a
HCS field to SnapshotMarker. In this patch, we should never send the
SnapshotMarkerV2 or expect to receive it.

Change-Id: I9963e49301574ee9b5ae22b043932c09e6e0ab5b
Reviewed-on: http://review.couchbase.org/113134
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

ep_perfsuite: Reduce ITERATIONS for Debug builds

Similary to TSan / ASan, run with a reduced iteration count under
Debug builds to speed runtime.

Change-Id: Ib480dbe6b75ed02af16

ep_perfsuite: Reduce ITERATIONS for Debug builds

Similary to TSan / ASan, run with a reduced iteration count under
Debug builds to speed runtime.

Change-Id: Ib480dbe6b75ed02af1667d684c11a090cb4cf81e
Reviewed-on: http://review.couchbase.org/112924
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Include what you use: platform/platform_thread.h

Some files get this header included from other headers. Include
the file explicitly to reduce the size of the patch which cleans
up t

Include what you use: platform/platform_thread.h

Some files get this header included from other headers. Include
the file explicitly to reduce the size of the patch which cleans
up the affected header.

Change-Id: I9a3e03a54031029688db94340d2f6cde1c916dc2
Reviewed-on: http://review.couchbase.org/112158
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Include what you use

Include what you use, and remove the C linkage
in cbsasl/util.h (we're all C++)

(prepare to get rid of platform/platform.h)

Change-Id: I6be2912761a

Include what you use

Include what you use, and remove the C linkage
in cbsasl/util.h (we're all C++)

(prepare to get rid of platform/platform.h)

Change-Id: I6be2912761a34318163f05a1e3be38cf00909320
Reviewed-on: http://review.couchbase.org/105904
Reviewed-by: Daniel Owen <owend@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-30040: Use platform_time.h in kv_engine

Change-Id: Ie1bd9deafda7362f472ab10622575c15fb9b25cc
Reviewed-on: http://review.couchbase.org/105548
Tested-by: Build Bot <build@couchbase.

MB-30040: Use platform_time.h in kv_engine

Change-Id: Ie1bd9deafda7362f472ab10622575c15fb9b25cc
Reviewed-on: http://review.couchbase.org/105548
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Audit uses of condition_variable.wait(lock) in tests

The single argument overload of std::condition_variable::wait (without
a predicate) is error-prone, as it can result in lost notifica

Audit uses of condition_variable.wait(lock) in tests

The single argument overload of std::condition_variable::wait (without
a predicate) is error-prone, as it can result in lost notifications if
the condition variable is spuriously woken up. See
http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rconc-wait

Audit our uses of wait() without a predicate in test code; fixing any
instances where they are incorrect, and simplifying APIs to make it
harder to use incorrectly.

Change-Id: Ic401f2d17103603c1ea1805359477a25945897f8
Reviewed-on: http://review.couchbase.org/104897
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Refactor: Move Dcp Open Flag #defines to constants with limited scope

Change-Id: Ie1f3b67e6e20466fb47ff1882de24d12644e4717
Reviewed-on: http://review.couchbase.org/102114
Tested-by:

Refactor: Move Dcp Open Flag #defines to constants with limited scope

Change-Id: Ie1f3b67e6e20466fb47ff1882de24d12644e4717
Reviewed-on: http://review.couchbase.org/102114
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Refactor: Move dcp_last_ variables to producers class

Whenever tests want to check the DCP producer state there's
a bit of boiler plate in extern linking to the mock_dcp.cc
'global'

Refactor: Move dcp_last_ variables to producers class

Whenever tests want to check the DCP producer state there's
a bit of boiler plate in extern linking to the mock_dcp.cc
'global' variables. Also, they're globals which limit the
tests we could write.

We can move the variables to be owned by the MockDcpMessageProducer
to fix the problems above. Tests can now be written to use multiple
producers and further work could be done to associate the variables
with streams for a more complete DCP test structure.

Change-Id: Ib9ef225cfff23b6928649756adcda9ceb5268a7b
Reviewed-on: http://review.couchbase.org/102064
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Simplify DCP Control API

We used to pass this a void* and size. We always
pass strings here, so we can might as well use
a cb::const_char_buffer.

Change-Id: If9f5a

Refactor: Simplify DCP Control API

We used to pass this a void* and size. We always
pass strings here, so we can might as well use
a cb::const_char_buffer.

Change-Id: If9f5a71a74e0fae85c5569cc62e29689d7eec33d
Reviewed-on: http://review.couchbase.org/101296
Reviewed-by: Chris Farman <christopher.farman@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Refactor: Use the scoped enums cb::mcbp::ClientOpcode in ep-engine tests

Instead of the uint8_t definitions in PROTOCOL_BINARY_CMD_XXX

Change-Id: If1b5839ce6a3fcde3cf6c3a70f10288f54

Refactor: Use the scoped enums cb::mcbp::ClientOpcode in ep-engine tests

Instead of the uint8_t definitions in PROTOCOL_BINARY_CMD_XXX

Change-Id: If1b5839ce6a3fcde3cf6c3a70f10288f54c40ad5
Reviewed-on: http://review.couchbase.org/101270
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-31688: Remove the use of [cb::]ProcessClock

It is defined as:

using ProcessClock = std::chrono::steady_clock

The name is shorter, but it adds an extra lookup to figu

MB-31688: Remove the use of [cb::]ProcessClock

It is defined as:

using ProcessClock = std::chrono::steady_clock

The name is shorter, but it adds an extra lookup to figure out
what it is. It used to be different things when not all of the
compilers supported std::chrono::steady_clock, but now that it
is no longer the case it should be removed.

Change-Id: Iebef35e07ec2e0d62495853434f16156b7cde529
Reviewed-on: http://review.couchbase.org/100802
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Refactor: Use cb::mcbp::Status in ep-engine instead of uint16_t

This makes sure that we don't try to use undefined status
codes in any of the response packets generated.

Change-

Refactor: Use cb::mcbp::Status in ep-engine instead of uint16_t

This makes sure that we don't try to use undefined status
codes in any of the response packets generated.

Change-Id: I5fce55208b408712a92e7621ce3795e847d1da10
Reviewed-on: http://review.couchbase.org/100045
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-30552: Convert type of vBucket id's from uint16_t to Vbid [8/n]

(Stage 8 of n) This commit changes to the type of Vbid for
ep's tests, including mock.
Vbid allows greater control

MB-30552: Convert type of vBucket id's from uint16_t to Vbid [8/n]

(Stage 8 of n) This commit changes to the type of Vbid for
ep's tests, including mock.
Vbid allows greater control over what operations can be performed
on a vBucket id, as well as ensuring the output formatting is
consistent. To support the splitting of this into multiple stages,
Vbid can currently be implicitly converted back to uint16_t to
cover elements that have not yet been converted. During this,
there may be logs or prints with "vb:vb:", which should be fixed
within this set of changes.

Change-Id: Ie86933239982b0dd882746fcc5ea0e490b79b3e7
Reviewed-on: http://review.couchbase.org/99627
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-26618: [3/n] Remove optional JSON parameter from open

DCP open now does not accept the optional value. This
patch also incorporates MB-24547, the code which would
force replicatio

MB-26618: [3/n] Remove optional JSON parameter from open

DCP open now does not accept the optional value. This
patch also incorporates MB-24547, the code which would
force replication streams into being collection-aware.
ns_server now performs that functionality and since
the JSON value was part of that temporary code, the
dcpOpen retry logic is dropped in this patch.

Change-Id: Iae1cf8cf0f390e44b6a25dac420b2fd2d4247bb5
Reviewed-on: http://review.couchbase.org/99374
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-26618: [1/n] Add the optional "string" parameter to stream_request

This patch series will move the optional string parameter from
dcp_open to dcp_stream_request

The first ste

MB-26618: [1/n] Add the optional "string" parameter to stream_request

This patch series will move the optional string parameter from
dcp_open to dcp_stream_request

The first step is to add the parameter to stream_request

Change-Id: I61e342d793ab2a5113ba50faa3298d29499ce358
Reviewed-on: http://review.couchbase.org/99310
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-30865: [5/n] Change DocKey API to prepare for leb encoded keys

This patch does a simple API change so that the bulk of the
leb128 (in-memory) patch is smaller.

The primary ch

MB-30865: [5/n] Change DocKey API to prepare for leb encoded keys

This patch does a simple API change so that the bulk of the
leb128 (in-memory) patch is smaller.

The primary change is that DocKey (which is a view of a key)
needs to know if the key it is viewing encodes a collection-ID.

So we no longer are able to set the Collection-ID via the constructor,
we simply state if the key we are viewing does or does not
encode a collection-ID. However because this patch doesn't introduce
in-memory encoded keys, internally the DocKey constructor just sets
the Collection-ID to the default collection. However any where in
the code that does require a collection-ID other than default, the
patch adds some temporary constructors so that the ID can be specified.

The next patch in this series, removes those temporary consructors
and up-dates DocKey/StoredDocKey/Serialised/DocKey to use the
encoded flag to properly store and decode a leb128 prefixed key.

Change-Id: I11d0c5e2c6a2290a6ac5968d072c66acab99e288
Reviewed-on: http://review.couchbase.org/98758
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-29262: When scanning the seqno index, use the persisted manifest

When a scan is performed we check if items belong to a deleted
collection by calling isLogicallyDeleted, this previous

MB-29262: When scanning the seqno index, use the persisted manifest

When a scan is performed we check if items belong to a deleted
collection by calling isLogicallyDeleted, this previously always
used the in-memory manifest to determine the state of the key. This
however is dangerous, particularly when the result of a scan is used
to feed DCP (e.g. backfill_disk).

The in-memory manifest can easily lose updates if a node crashes
allowing a situation when we can backfill a client and drop keys
before a crash, but then after the node restarts a client can stream
and now receive keys which would of been dropped before the crash.

A simple example is a client asking for seqno ranges.

First they ask for all collection keys from seqno 0-10, assume seqno
3 and 4 are for the 'beer' collection which has been deleted from
the memory manifest, but not yet flushed to disk.

* Backfill will send 0 to 9 but drop 3 and 4 (3 is 'create-beer')
* Scan completes and client has received all keys.
* memcached crashes before the beer delete fact has been persisted.
* memcached restarts
* Client reconnects and now asks for 10-19, assume 15 and 16 are
'beer' mutations
* Backfill will now send every key, none are logically deleted
* Client gets beer keys and has never seen a create-beer or any prior
'beer' key, an incorrect view of the data.

If the scan drops keys based on the persisted manifest, then this
situation will not occur.

Test-code required some tweaking as we have clash with boost/list.hpp
and ep_test_apis.h both defining check(), this occurred because
callbacks.h now includes vbucket_manifest (and then boost).
callbacks.h has been removed from test_helpers.h

Change-Id: I742e5cccf861baa895f4e72c1ed2b94f2847f4dd
Reviewed-on: http://review.couchbase.org/92774
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-30149: Drop duplicate parameter to test functions [29/n]

This patch drops the second parameter to the test function which
used to be a pointer to the first interface version of the en

MB-30149: Drop duplicate parameter to test functions [29/n]

This patch drops the second parameter to the test function which
used to be a pointer to the first interface version of the engine
interface.

Change-Id: Ic42e7089c0d3b6c9167c9b7c5974ee1ffc4259fc
Reviewed-on: http://review.couchbase.org/97915
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-30149: Drop duplicate parameter to test functions [27/n]

* add_sentinel_doc
* perf_background_sets
* perf_load_client
* perf_dcp_client
* single_dcp_latency_bw_t

MB-30149: Drop duplicate parameter to test functions [27/n]

* add_sentinel_doc
* perf_background_sets
* perf_load_client
* perf_dcp_client
* single_dcp_latency_bw_test
* perf_dcp_latency_and_bandwidth

Change-Id: Ib375305950d23d4f98f358b8c13607c47f46620e
Reviewed-on: http://review.couchbase.org/97905
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...