MB-29675: Optimize ActiveStream::makeResponseFromItem

makeResponseFromItem is called for every item to be sent out over DCP;
and it shows up high in 'linux perf' profiles (approx 18% of

MB-29675: Optimize ActiveStream::makeResponseFromItem

makeResponseFromItem is called for every item to be sent out over DCP;
and it shows up high in 'linux perf' profiles (approx 18% of AuxIO
threads). Optimize a number of areas of it:

- Remove unnecessary refcount updates by removing unneeded temporary
ref-counted ptrs

- Using std:move where applicable when passing queued_items to
sub-functions (so refcount is unchanged).

With these changes makeResponseFromItem drops to 6% of AuxIO threads.

Change-Id: I0368dd1dd93c91d3ec47d9d12c9aea7efc59a7f0
Reviewed-on: http://review.couchbase.org/94215
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID (and can persist/warm up from it).

Change-Id: I66e4ad4274558992b27c5ab02adb42295e761091
Reviewed-on: http://review.couchbase.org/91178
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-28502: Add delete_time awareness to flow control acking

DcpProducer tracks how many bytes it sends and the DcpConsumer must
ack how much it receives. With the 'delete-time' feature th

MB-28502: Add delete_time awareness to flow control acking

DcpProducer tracks how many bytes it sends and the DcpConsumer must
ack how much it receives. With the 'delete-time' feature the delete
packet is larger and as such we must account for this extra data on
DCP streams that are configured with delete-time.

Change-Id: I8d7a785fda6652554100813a8cfdf6cff0e64f39
Reviewed-on: http://review.couchbase.org/90655
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-27661 [4/n]: Fix MSVC warnings - C4018, C4146

This patch addresses the following generated warnings:

C4018 - signed/unsigned mismatch
C4146 - unary minus operator applied to

MB-27661 [4/n]: Fix MSVC warnings - C4018, C4146

This patch addresses the following generated warnings:

C4018 - signed/unsigned mismatch
C4146 - unary minus operator applied to unsigned type,
result still unsigned

It also introduces a new constant, LOCKED_CAS which represents
the value given for the CAS of an item which is under lock.

Change-Id: I1cb00e37daf41c73e942b2fc2a25e2f9c9a09e2e
Reviewed-on: http://review.couchbase.org/89815
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream_end.

Change-Id: If09cc37e113f3019a3a1c743e60ea9bb2537691b
Reviewed-on: http://review.couchbase.org/84603
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-26255[5]: Process the final DCP item at response creation time

The item that is received from a backfill or from the checkpoint,
isn't the final item that is sent over the wire to the

MB-26255[5]: Process the final DCP item at response creation time

The item that is received from a backfill or from the checkpoint,
isn't the final item that is sent over the wire to the DCP consumer.
The item's value/xattrs is pruned or the value is compressed or
decompressed. So, before creating the final DCP response, create the
final item and store it in the response.

Change-Id: Icd96a54e89d4952e1099741c878d28e51eff1695
Reviewed-on: http://review.couchbase.org/84809
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Send collection UID in network byte order.

The DCP system event 'value' is data created by ep-engine, in the
case of collections we encode the collection UID in the value. Prio

MB-16181: Send collection UID in network byte order.

The DCP system event 'value' is data created by ep-engine, in the
case of collections we encode the collection UID in the value. Prior
to this patch, that data was not encoded in network byte order.

Change-Id: Ib00e55ede356cf92aa358a5f1904cb84cc0a19c6
Reviewed-on: http://review.couchbase.org/83351
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These eve

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These events
mapped to hidden documents (System namespace) and had unique keys.
Thus the life-span of collection beer, uid:1 may have a seqno 'stream'
as follows (e.g. if backfill from seqno 0).

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker

Now due to the way we support multiple generations of collections
before the prior one is fully purged, this approach leads us to have a
seqno stream as follows (e.g. backfill from seqno 0)

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker
seqno 6, key = $collections:create:beer:2 <- create marker
seqno 7, key = beer::document <- document in the collection
seqno 8, key = $collections:delete:beer:2 <- delete marker

With this approach (unique-keys) we have to maintain an increasing
amount of meta-data to enable correct clean-up of the old generations.
This growing meta-data would need to be in memory and the per VB
JSON manifest.

So to avoid this unbounded 'list' metadata this commit adjusts the way
collections spans are managed in-terms of checkpoints and persisted
data.

1. No longer do we have a Create and separate Delete event - a single
SystemEvent::Collection will be used to cover create and delete.

a) A single key represents all creates/deletes of a collection.
b) A delete of a collection is just a delete of that system key.

2. Collection create/delete now closes the checkpoint to ensure we
never de-dup create/delete/create into a single create, ensuring
a client tracking a vbucket sees explicit creates and deletes and
doesn't have to handle de-duplication E.g. "I've been told of
beer:2 but never saw delete(beer:1)?"

So the life span of a collection now generates sequence numbers as
follows:

seqno 3, key = $collections:beer <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index we see:

seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index after re-creating the collection,
we see.

seqno 4, key = beer::document <- document in the collection
seqno 6, key = $collections:beer <- create marker

And if we had performed cycles of create/delete we can see that there
are no longer many keys associated with the collection resulting in no
extra meta data to store so we can remove old markers.

Note in these examples, DCP wouldn't just replicate the by-seqno index
as seqno 4 can be identified as being logically deleted (it has a
seqno < start of beer). Additionally for a backfill from 0 in the
first example seqno 5 doesn't need to be sent either because the
client cannot know about the create. These enhancements are not part
of this commit.

Change-Id: I32f04e512ce039dce7d780323d9bf3642cc507fa
Reviewed-on: http://review.couchbase.org/83350
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identi

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identify a
collection. This is now cleaned up so that a new class is used when
identifying a specific 'generation' of a collection.

As part of this new class we are removing the use of 'revision' and
replacing it with 'uid', however we use the Manifest::revision as
the uid and later changes will change the Manifest format so that each
collection is assigned a uid.

As part of the clean-up of revision usage this commit fixes some short
comings in the VB::Manifest management where a delete of a collection
was updating it's revision, forcing later callers of completeDeletion
to use the newer revision, instead of referring to the collection by
the correct name:rev from when it was created.

This commit additionally cleans up the separator changed path which
was needlessly passing the revision through the various code paths and
even over DCP. There is no need it was set and never read.

Change-Id: I0d81ccb9a7c7b7505d0f021b284b368f7a60f88e
Reviewed-on: http://review.couchbase.org/81984
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-21682: Add ability to close DCP producer due to vBucket rollback

This is not exposed externally, and maps to END_STREAM_STATE on the wire
(vBucket state changed). However, the additio

MB-21682: Add ability to close DCP producer due to vBucket rollback

This is not exposed externally, and maps to END_STREAM_STATE on the wire
(vBucket state changed). However, the addition of END_STREAM_ROLLBACK
means we can log a good reason why the stream was closed.

Change-Id: Ic3352b270c845cb8a75ccda0c2707889b5ac0f44
Reviewed-on: http://review.couchbase.org/80178
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-24631: Ensure that bufferedBackfill.items is decremented correctly

There was an underflow in bufferedBackfill.items because it was
incremented for non meta DCP items were read, but de

MB-24631: Ensure that bufferedBackfill.items is decremented correctly

There was an underflow in bufferedBackfill.items because it was
incremented for non meta DCP items were read, but decremented always
when an item was sent.

This commit fixes that and adds a test check for the same.

Change-Id: I077999cc3d3827980abd3811c0c78489f3080b8b
Reviewed-on: http://review.couchbase.org/79626
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-24037 / MB-23605 Strip xattrs and value and record message size

Moves the stripping of xattrs and value (if want key only) from
memcached to ep-engine. Also ensures that the mutation

MB-24037 / MB-23605 Strip xattrs and value and record message size

Moves the stripping of xattrs and value (if want key only) from
memcached to ep-engine. Also ensures that the mutationResponse message
size, which will be sent over the wire, is updated correctly.

Change-Id: I293592a5ed1e5dc2cb47c8fc78c1be09e70645de
Reviewed-on: http://review.couchbase.org/79051
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
c

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
changes to SystemEvent shouldn't result in changes to the values
we transit.

Change-Id: I67c8e5876e10299eb69a52e89c7f18ff4981c09f
Reviewed-on: http://review.couchbase.org/78640
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Move ep-engine to engines/ep

MB-16181: Make ActiveStream track the current collections separator

This patch updates the ActiveStream so that it stores a copy of the current
separator and tracks changes to as they ar

MB-16181: Make ActiveStream track the current collections separator

This patch updates the ActiveStream so that it stores a copy of the current
separator and tracks changes to as they are transmitted through the checkpoint
via the separator changed SystemEvent.

Change-Id: Ie3ea87d006b0bbab3e0edd8895a4756c7c5d9fe8
Reviewed-on: http://review.couchbase.org/75502
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-22738: Move stripping of value from DCP stream back into ep-engine

A revert of the http://review.couchbase.org/#/c/72398/
with the addition that determining whether whether to retriev

MB-22738: Move stripping of value from DCP stream back into ep-engine

A revert of the http://review.couchbase.org/#/c/72398/
with the addition that determining whether whether to retrieve only
is made on the connection level, (as opposed to the stream level).

Change-Id: I641978c2be6c67e6a9d96c0a229ff21688c74055
Reviewed-on: http://review.couchbase.org/76418
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Coverity:CID165843: Unitialised member in SystemEventConsumerMessage

vbid member was not initialised

Change-Id: I0e685cf3008e2bc446ced39a460b94e024596073
Reviewed-on: http://rev

Coverity:CID165843: Unitialised member in SystemEventConsumerMessage

vbid member was not initialised

Change-Id: I0e685cf3008e2bc446ced39a460b94e024596073
Reviewed-on: http://review.couchbase.org/76393
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Collections DcpProducer

1. Enable SystemEvent Items in "Item::shouldReplicate"
2. Update DcpProducer so that it grabs the various bits of collection
data for the transfe

MB-16181: Collections DcpProducer

1. Enable SystemEvent Items in "Item::shouldReplicate"
2. Update DcpProducer so that it grabs the various bits of collection
data for the transferred events and calls through the engine's dcp
system_event callback

An update synchrounous engine test now links a MockDcpConsumer to
a MockDcpProducer and drives events from one to the other.

Change-Id: I83052aa956afbe7495c15420889c84d920a5b7fc
Reviewed-on: http://review.couchbase.org/75603
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Refactor buffered backfill accounting.

The general aim of this commit is to be more abstract in how we
account the resources used during in backfill.

Currently the ass

MB-16181: Refactor buffered backfill accounting.

The general aim of this commit is to be more abstract in how we
account the resources used during in backfill.

Currently the assumption is that MutationResponses will be the only
object to account for, thus on the read increment we use Item::size
and on the send decrement we access the Item::size by casting to a
MutationResponse.

With Collections, SystemEvents will be part of the backfill and should
be accounted for, but a SystemEvent in the ActiveStream's readyQueue
means we need a cleaner way of asking a DcpResponse for its size, as
opposed to more if (event == ...) {static_cast...}

1. Add a method to the abstract DcpResponse class which by default
returns 0, so for example a SnapShotMarker can be unconditionally
applied to the backfill accounting without changing it.

2. Implement in MutationResponse and (upstream) SystemEvent the method
so it returns a value that better represents the resources that
DcpResponse is holding, that is the size of the Item + any extra
resources.

Change-Id: I7c57572e2f09ed80d84a7505b36734def5c75580
Reviewed-on: http://review.couchbase.org/75433
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Collections DcpConsumer

Add the functional paths that allow a replica VBucket's manifest to
be updated from incoming DCP messages. Note that the DCP protocol
has not been w

MB-16181: Collections DcpConsumer

Add the functional paths that allow a replica VBucket's manifest to
be updated from incoming DCP messages. Note that the DCP protocol
has not been written, but the API defined here shows what data needs
to be transfered over-the-wire.

Add a unit test to use MockDcpConsumer to drive a create and delete

Change-Id: Ida8171b2cde104ccd452fafcd6193a72455b404b
Reviewed-on: http://review.couchbase.org/74609
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Add getBySeqno to DcpResponse and return OptionalSeqno

This commit is useful up-stream where more DcpResponse sub-classes
now can return a seqno. This cleans up code where we c

MB-16181: Add getBySeqno to DcpResponse and return OptionalSeqno

This commit is useful up-stream where more DcpResponse sub-classes
now can return a seqno. This cleans up code where we check the event
type, then cast and pull out the seqno.

Upstream we would need more if's and a second cast to a different
response sub-class, but with this change we can just ask abstract
class for the seqno and check if it has a value.

Change-Id: Ib92390980ef394e94c6523131a5312b480bc8e44
Reviewed-on: http://review.couchbase.org/75396
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Rename DCPEvent to Event and move into DCPResponse

The Event should be inside the DCPResponse class.
The Event to_string method should also be inside the DCPResponse
class.

Refactor: Rename DCPEvent to Event and move into DCPResponse

The Event should be inside the DCPResponse class.
The Event to_string method should also be inside the DCPResponse
class.

Change-Id: I58194277ff455a2d3eefde2b130b2ac88cc4a03a
Reviewed-on: http://review.couchbase.org/74689
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MutationResponse modernisation

* Replace bare-pointer with unique_ptr
* Use const static initialisation inline with class

Change-Id: I65f153659f001a602e5caf6a0316668d603dfc20

MutationResponse modernisation

* Replace bare-pointer with unique_ptr
* Use const static initialisation inline with class

Change-Id: I65f153659f001a602e5caf6a0316668d603dfc20
Reviewed-on: http://review.couchbase.org/73615
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Use enum class for dcp event type

Change-Id: I95a13d9083999330ad24e0cf8611d5c0b40e2217
Reviewed-on: http://review.couchbase.org/72900
Reviewed-by: Dave Rigby <daver@couchba

Refactor: Use enum class for dcp event type

Change-Id: I95a13d9083999330ad24e0cf8611d5c0b40e2217
Reviewed-on: http://review.couchbase.org/72900
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...

Remove support for DCP NO_VALUE (moved to core)

As part of adding support for XATTRs over DCP we can't strip
off the value part in ep-engine. The memcached core will
snoop the DCP OP

Remove support for DCP NO_VALUE (moved to core)

As part of adding support for XATTRs over DCP we can't strip
off the value part in ep-engine. The memcached core will
snoop the DCP OPEN and ADD STREAM requests and pick up the
flag to strip off bodies.

The removed test cases tested the cases where we want to
strip off the values (which we no longer can do due to the
fact that the frontend may need them in order to pick out
the xattrs).

Note that this patch depend on the patch in memcached
which implement stripping of the value. (given that
there isn't a compile time dependency I'm not reusing
the Change-Id as jenkins don't play very well with
patches based on top of the linked patch)

Change-Id: I86ed8c2844f70d5aa1630339ae5859a731e557bd
Reviewed-on: http://review.couchbase.org/72398
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...