MB-37374: Producer sends user-xattrs within normal delete

Producer must send user-xattrs when IncludeDeleteUserXattrs::Yes.
Differently, in the next patch we will ensure that user-xattrs

MB-37374: Producer sends user-xattrs within normal delete

Producer must send user-xattrs when IncludeDeleteUserXattrs::Yes.
Differently, in the next patch we will ensure that user-xattrs are
pruned before streaming when IncludeDeleteUserXattrs::No.

Note: Sync Delete will be addressed in follow-up patches.

Change-Id: Ia5d858edda5684f057c156f161377795d56fd725
Reviewed-on: http://review.couchbase.org/c/kv_engine/+/126708
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-37374: Add new IncludeDeletedUserXattrs flag at DCP_OPEN

The flag is used at DCP_OPEN(Producer) to enable the related feature.
Ignored at DCP_OPEN(Consumer).

Change-Id: I1e06

MB-37374: Add new IncludeDeletedUserXattrs flag at DCP_OPEN

The flag is used at DCP_OPEN(Producer) to enable the related feature.
Ignored at DCP_OPEN(Consumer).

Change-Id: I1e06f230e0be9c248a61d66db8b50d232848177b
Reviewed-on: http://review.couchbase.org/c/kv_engine/+/126116
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-37680: Add support for sequential DCP Backfill

Add a new DCP_CONTROL option to allow DCP consumers to request that
backfills are scheduled in sequential order, instead of the default

MB-37680: Add support for sequential DCP Backfill

Add a new DCP_CONTROL option to allow DCP consumers to request that
backfills are scheduled in sequential order, instead of the default
round-robin order.

The new DCP_CONTROL message 'backfill_order' allows the DCP client to
choose from the following values:

* `round-robin` - vBuckets should be backfilled in round-robin order,
reading chunk of data from each in turn. This is the default value
(and the behaviour of DCP BackfillManager before this change).

* `sequential` - vBuckets should be backfilled sequentially - _all_
data from the first vBucket should be read from disk before
advancing to the next vBucket.

Change-Id: I15ae56a6c917960f0efb9ac2de9b0eec49745612
Reviewed-on: http://review.couchbase.org/c/kv_engine/+/128375
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

Minimise includes of dcp/ headers

Change-Id: Ic9a737cfd50b8f665b52f90dc59f38b71eeaac99
Reviewed-on: http://review.couchbase.org/113439
Tested-by: Build Bot <build@couchbase.com>

Minimise includes of dcp/ headers

Change-Id: Ic9a737cfd50b8f665b52f90dc59f38b71eeaac99
Reviewed-on: http://review.couchbase.org/113439
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-34789: Send mutation instead of commit for backfilled commits

Send a DCP Mutation instead of a DCP Commit when sending a commit
with prepare seqno > requested stream start seqno if th

MB-34789: Send mutation instead of commit for backfilled commits

Send a DCP Mutation instead of a DCP Commit when sending a commit
with prepare seqno > requested stream start seqno if the commit
has been backfilled from disk. We need to do this because the
de-duplication of keys on disk can result in a commit being the
first seqno a replica will see. In this case we must send a mutation
because the consumer needs to know what to commit. Currently, the
code does this if the prepare seqno is less than the requested
start seqno. This does not solve the case where we de-dupe a prepare
within the disk snapshot though. Instead, send the mutation instead
of the commit if the replica did not receive the prepare before it
connected whilst we are backfilling from disk.

*** EXAMPLE ***

XXXX = Nothing sent/received

Active: P:k1 C:k1 M:k2 P:k1 C:k1 P:k1
Replica: P:k1 Disconnect XXXX M:k2 XXXX C:k1 P:k1

Replica disconnects and then streams from disk.

For the first commit the replica receives we must send a
mutation so that we can commit the correct value. The previous
code would send a commit as the prepare seqno was greater than the
first seqno sent.

Change-Id: I25bb8a11a920c035bfd25f656320642d4aba751e
Reviewed-on: http://review.couchbase.org/111435
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Minimise #include of dcp/response.h

Remove unnecessary #includes of dcp/response.h:

- replace with forward declarations

- Move some inlined code to .cc to remove need for r

Minimise #include of dcp/response.h

Remove unnecessary #includes of dcp/response.h:

- replace with forward declarations

- Move some inlined code to .cc to remove need for response.h in
header files.

Change-Id: Ie882294a686e9a27ebe2d42adf9b6c99434408f4
Reviewed-on: http://review.couchbase.org/107953
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

De-inline DCP stream-related functions

De-inline a number of DCP stream / mock stream functions to minimise
the coupling between different files. Note all de-inlined functions
are ei

De-inline DCP stream-related functions

De-inline a number of DCP stream / mock stream functions to minimise
the coupling between different files. Note all de-inlined functions
are either test code (where we don't care about perfomance), or are
not (believed to be) hot code.

Where possible remove now-unnecessary header #includes or replace with
forward-decls.

Measuring the build times for this complete sequence of patches:

* 364c8a9c0 2019-01-25 | De-inline DCP stream-related functions (HEAD -> build_optimise) [Dave Rigby]
* 97096b098 2019-01-25 | Move VBucket-related declarations to vbucket_fwd.h [Dave Rigby]
* 5e58cfb02 2019-01-25 | Move PermittedVBStates to its own file [Dave Rigby]
* 94abc0620 2019-01-25 | De-inline MockDcpProducer methods [Dave Rigby]
* f2287529f 2019-01-25 | Move Collections json-related types to .cc [Dave Rigby]
* 38ba2f323 2019-01-25 | Move vbucket_state to its own file [Dave Rigby]
* 563074cd3 2019-01-25 | Minimise where statwriter.h is #include'd [Dave Rigby]
* cd60141f7 2019-01-25 | Replace <?stream> includes with <iosfwd> where possible (m/master, couchbase/master) [Dave Rigby]

Gives the following results - user+sys time for clean build (best of 3
runs) on MacBook Pro 2018 (2.9GHz Core i9).

Before: 2271s, after: 2153s
119s reduction in time.

Not measured, but I believe incremental builds should show a greater
speedup due to the reductions in header dependancies.

Change-Id: Ie5f46be1e34ce912ef9ca6312cadf13d98967ab4
Reviewed-on: http://review.couchbase.org/104093
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-30926: Remove MustSendCheckpointEnd

As TAP has now been factored out, the use of MustSendCheckpointEnd
is now obsolete legacy so should be removed.

Change-Id: I67ee959ab5ca0f

MB-30926: Remove MustSendCheckpointEnd

As TAP has now been factored out, the use of MustSendCheckpointEnd
is now obsolete legacy so should be removed.

Change-Id: I67ee959ab5ca0f43ca1f84054e53dde089b188b4
Reviewed-on: http://review.couchbase.org/99253
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-30149: Remove ENGINE_HANDLE_V1 and ENGINE_HANDLE

Replaced all of the use of the old aliases with the new name
EngineIface by using sed.

Change-Id: I03989740baf6b60a0d0a3267c3

MB-30149: Remove ENGINE_HANDLE_V1 and ENGINE_HANDLE

Replaced all of the use of the old aliases with the new name
EngineIface by using sed.

Change-Id: I03989740baf6b60a0d0a3267c3e2fc286c8a1b57
Reviewed-on: http://review.couchbase.org/97048
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-29500: Change client's cursor from a string to be a weak_ptr

Rather than using a std::string (chosen by the cursor owner) for
looking up cursors (allowing for issues such as two clien

MB-29500: Change client's cursor from a string to be a weak_ptr

Rather than using a std::string (chosen by the cursor owner) for
looking up cursors (allowing for issues such as two clients hitting
the same cursor), use a weak_ptr to locate the cursor.

The use of a weak_ptr allows for a clients cursor to become degraded
simply by removing it from the core cursor list (which stores
shared_ptr).

Change-Id: I3779fcf1db751bde55ee1b2a90715935b79b19b6
Reviewed-on: http://review.couchbase.org/93187
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...