MB-36948: Update dcp.h marker() to take maxVisibleSeqno

dcp.h has the API signature for transmitting a DCP snapshot, update this
to include the optional maxVisibleSeqno.

Change-

MB-36948: Update dcp.h marker() to take maxVisibleSeqno

dcp.h has the API signature for transmitting a DCP snapshot, update this
to include the optional maxVisibleSeqno.

Change-Id: I8ecfb324d4bad30354e715cf5d1673a109a2cc4a
Reviewed-on: http://review.couchbase.org/118459
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Jim Walker <jim@couchbase.com>

show more ...

MB-36858: DcpProducer::itemsSent track Prepare/Commit/Abort

We should track these items in DcpProducer::itemSent as this stat
can be very useful for debugging DCP connections.

C

MB-36858: DcpProducer::itemsSent track Prepare/Commit/Abort

We should track these items in DcpProducer::itemSent as this stat
can be very useful for debugging DCP connections.

Change-Id: Ie194d394ebdd64435980f1769dd38241c264b7c0
Reviewed-on: http://review.couchbase.org/117728
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove meta section from DcpDeletion API

It is always being sent as { nullptr, 0 }.

Change-Id: Idd267d531343334dc2778d8493b68a31a9c01108
Reviewed-on: http://review.couchbase.org

Remove meta section from DcpDeletion API

It is always being sent as { nullptr, 0 }.

Change-Id: Idd267d531343334dc2778d8493b68a31a9c01108
Reviewed-on: http://review.couchbase.org/115327
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove meta section from DcpMutation API

It is always being sent as { nullptr, 0 } so we don't
need it in the API

Change-Id: I1f5162cfb978aa2ced8dd4e11cfb5f0c0ccc03ec
Review

Remove meta section from DcpMutation API

It is always being sent as { nullptr, 0 } so we don't
need it in the API

Change-Id: I1f5162cfb978aa2ced8dd4e11cfb5f0c0ccc03ec
Reviewed-on: http://review.couchbase.org/115324
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-34017: Add HCS to SnapshotMarker

To correct the replica on disk HCS we need to tell it what pass it
the HCS value from the active when we send a disk snapshot. Add a
HCS field to

MB-34017: Add HCS to SnapshotMarker

To correct the replica on disk HCS we need to tell it what pass it
the HCS value from the active when we send a disk snapshot. Add a
HCS field to SnapshotMarker. In this patch, we should never send the
SnapshotMarkerV2 or expect to receive it.

Change-Id: I9963e49301574ee9b5ae22b043932c09e6e0ab5b
Reviewed-on: http://review.couchbase.org/113134
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-34313: Remove timeout value for DCP PREPARE

The durability_timeout field in the DCP_PREPARE message is
unused - once a Prepare has been sent over DCP, the timeout
can no longer be

MB-34313: Remove timeout value for DCP PREPARE

The durability_timeout field in the DCP_PREPARE message is
unused - once a Prepare has been sent over DCP, the timeout
can no longer be applied if the replica was to be promoted,
as it may have already been committed by the old active.

Change-Id: I7f744c2fe9682b096ab58e25e34a55d1f8d32faa
Reviewed-on: http://review.couchbase.org/111998
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Handle seqno ack in MockDcpMessageProducers

ep_testsuite_dcp unconditionally enables sync replication in
add_stream_for_consumer, but returns ENGINE_ENOTSUP on receipt of a
seqno ack

Handle seqno ack in MockDcpMessageProducers

ep_testsuite_dcp unconditionally enables sync replication in
add_stream_for_consumer, but returns ENGINE_ENOTSUP on receipt of a
seqno ack.

We should handle seqno acks like other ops, allowing tests to assert
that a seqno ack occurred.

Change-Id: I7de77d38334b0c1e5534ef9643594ef091dd3025
Reviewed-on: http://review.couchbase.org/111874
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Use cb::unique_item_ptr in dcp_message_producers

The dcp_message_producers interface passes abstract `item` objects for
the implementing class (normally memcached Connection) to send out

Use cb::unique_item_ptr in dcp_message_producers

The dcp_message_producers interface passes abstract `item` objects for
the implementing class (normally memcached Connection) to send out. It
is is the responsiblity of the implementing class to release these
items. Currently this has to be done manually - the interface just
uses a raw `item*` for Item objects.

In addition to being error-prone (however known leaks exist), this
increases the boilerplate needed in other implmeentations (e.g. mocks
used for testing) - they must ensure they call engine->release(item).

Specifically, a subsequent patch adds a GoogleMock implementation of
dcp_message_producers for more flexible / powerful testing of
expecations of the DCP interface. In this case a large amount of extra
boilerplate (adding WillByDefault custom handlers to all methods)
would be needed.

Given a bunch of changes are needed one way or another to support
GoogleMock, update the interface to use the cb::unique_item_ptr (a
unique_ptr with a custom deleter). This gives RAII-style resource
management for `item` objects, removing the need to manually release
them.

Change-Id: I9ba77c4cc695e1f1d2944fd3f506198c54bdcb2e
Reviewed-on: http://review.couchbase.org/108332
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>

show more ...

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-32784: 3/5 Switch to new meta-data for dropping collections

Note this patch maintains two concurrent collection's meta-data
mechanisms and partially changing over to use the new. This

MB-32784: 3/5 Switch to new meta-data for dropping collections

Note this patch maintains two concurrent collection's meta-data
mechanisms and partially changing over to use the new. This patch
deliberately does not delete all of the code which is managing the old.
Complete removal of the old mechanism will happen in a later patch to
keep the diff size more manageable.

This patch switches over to the new persisted meta-data with two major
areas affected.

The VB::Manifest now has a more stream-lined task, it is no longer
re-used in VB::ScanContext and VB::EraseContext for collection drop
and backfill 'pruning', the object now just manages the flow of CRUD
operations at the front of house. The VB::Manifest no longer has to
manage a collection state from open -> deleting -> gone, the
VB::Manifest now only stores open collections. The fallout from this
change is the removal of a number of bits of the VB::Manifest such
as completeDeletion(), nDeletingCollections and the greatestEndSeqno
"optimisation" (never proven to make a difference anyway...). The
CRUD paths now just need to lock the manifest and if the requested
collection-ID is in the map, the collection is writable.
isLogicallyDeleted still utilises seqno comparison for certain paths so
we can support collection flush.

The second major affected is KVStore, with couch-kvstore now
utilising the new meta-data for dropping a collection from storage.

Change-Id: Ib0aca6c9cded8f3ee01f2dfecec93b180adea045
Reviewed-on: http://review.couchbase.org/104784
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-31438: Send vbucket manifest UID in resuming stream requests

Update the DCP Consumer to create PassiveStreams with a vbucket
manifest uid json string that will be sent to the correspo

MB-31438: Send vbucket manifest UID in resuming stream requests

Update the DCP Consumer to create PassiveStreams with a vbucket
manifest uid json string that will be sent to the corresponding DCP
Producer. This string will be parsed by the pre-existing vbucket
filter code.

See MB-31326 for further details.

Change-Id: I02344f22431801f19356126cb912463db67de5af
Reviewed-on: http://review.couchbase.org/103580
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-32336 [SR]: Skip pending_sync_write if not supported on DCP stream

DCP connections which do not advertise support for synchronous
replication should not be sent DcpPrepare messages. T

MB-32336 [SR]: Skip pending_sync_write if not supported on DCP stream

DCP connections which do not advertise support for synchronous
replication should not be sent DcpPrepare messages. This was initially
implemented by skipping such event types in DcpProducer::step(),
however that is too late in the sequence as we've already added the
item to the Stream's readyQ.

Instead adopt a similar approach to how Collections events are
filtered - in ActiveStream::shouldProcessItem() check if:

- the Item is a synchronous replication one, and
- if the ActiveStream supports such messages.

If not, then don't process the item.

Also adds initial unit tests for DCP synchronous replication
streams. This required enhancing MockDcpMessageProducers to support
the prepare() method.

Change-Id: I25046a9b979fd96b60b34b925ae0909e46c79110
Reviewed-on: http://review.couchbase.org/102852
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>

show more ...

MB-31978: Add DcpStreamId into DCP API

DCP producer API relates to those methods which will send a message from
server to client only for DCP producer connections.

* snapshot-ma

MB-31978: Add DcpStreamId into DCP API

DCP producer API relates to those methods which will send a message from
server to client only for DCP producer connections.

* snapshot-marker
* mutation
* deletion
* expiration
* system event
* stream-end

This commit adds to those function prototypes the DcpStreamId parameter
so that the DcpProducer can pass down the ID of the stream that
generated the message and will later on be included in the outgoing
packet.

From client to server the close_stream method can accept a DcpStreamId
so the client can close a stream by vb/sid identification.

Note that flow-control (ack) is not stream level, clients ack to the
producer so we never consider stream-id in any acking functions.

Change-Id: I81a45bf01340821e9e38079dbc9c2520145e6232
Reviewed-on: http://review.couchbase.org/102197
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Prepare DCP Mutations for framing extras

Change-Id: Ica4065d2019836cd8fedfcfcc9d9370f9e01627e
Reviewed-on: http://review.couchbase.org/102142
Tested-by: Build Bot <build@co

Refactor: Prepare DCP Mutations for framing extras

Change-Id: Ica4065d2019836cd8fedfcfcc9d9370f9e01627e
Reviewed-on: http://review.couchbase.org/102142
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Refactor: Prepare Dcp Expiration for Frame Extras

Change-Id: Ib27aa56f4ce6dc18866c35600d2d212304b23fd4
Reviewed-on: http://review.couchbase.org/102147
Tested-by: Build Bot <build@cou

Refactor: Prepare Dcp Expiration for Frame Extras

Change-Id: Ib27aa56f4ce6dc18866c35600d2d212304b23fd4
Reviewed-on: http://review.couchbase.org/102147
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-32124: drop/add scope need to update the manifestUID

Spotted whilst updating pydcp that the final manifest we
computed from DCP was one UID below the get_collection's UID.

Si

MB-32124: drop/add scope need to update the manifestUID

Spotted whilst updating pydcp that the final manifest we
computed from DCP was one UID below the get_collection's UID.

Simple add the variable updates and update a unit test to check.

Change-Id: Iac797e04bae9719e7d487889a603423bd7a3ce61
Reviewed-on: http://review.couchbase.org/102068
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

Refactor: Move dcp_last_ variables to producers class

Whenever tests want to check the DCP producer state there's
a bit of boiler plate in extern linking to the mock_dcp.cc
'global'

Refactor: Move dcp_last_ variables to producers class

Whenever tests want to check the DCP producer state there's
a bit of boiler plate in extern linking to the mock_dcp.cc
'global' variables. Also, they're globals which limit the
tests we could write.

We can move the variables to be owned by the MockDcpMessageProducer
to fix the problems above. Tests can now be written to use multiple
producers and further work could be done to associate the variables
with streams for a more complete DCP test structure.

Change-Id: Ib9ef225cfff23b6928649756adcda9ceb5268a7b
Reviewed-on: http://review.couchbase.org/102064
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-31907: [2/2] Persist and replicate scope events

Update the VB::Manifest so that it can queue a system
event when a scope is created or dropped, this system
event is very similar t

MB-31907: [2/2] Persist and replicate scope events

Update the VB::Manifest so that it can queue a system
event when a scope is created or dropped, this system
event is very similar to collection create/drop.

The patch also updates the filter code so that a stream
which is scope filtered closes when the scope is removed.

Change-Id: I42a80e93f0f9caa8707cd9faa3384d44c03cfd2d
Reviewed-on: http://review.couchbase.org/101454
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: move off old binary protocol def for dcp_delete

We want to get rid of all of the protocol_binary_request_*
unions as we want to be able to inject frame extras before
the ex

Refactor: move off old binary protocol def for dcp_delete

We want to get rid of all of the protocol_binary_request_*
unions as we want to be able to inject frame extras before
the extra section.

Change-Id: I46e6746a68db10f91e822b8e60a7279c5cc1f1e1
Reviewed-on: http://review.couchbase.org/101746
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-30487: [1/2] Accept max_ttl field in collection entry

This commit updates the JSON parsing so that we can
accept a max_ttl field in the collection manifest.

Once a max_ttl is

MB-30487: [1/2] Accept max_ttl field in collection entry

This commit updates the JSON parsing so that we can
accept a max_ttl field in the collection manifest.

Once a max_ttl is specified it is propagated down
to the VB::Manifest during collection creation.

There is no support for changing an existing max_ttl,
only create a new collection with a max_ttl is
currently supported by this commit.

Change-Id: I13e099bccf794d44000586524689abea4da4af0f
Reviewed-on: http://review.couchbase.org/101229
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-31739: [2/2] Update create/drop system events per design changes

The create event needs to transmit
* manifest uid
* collection uid
* scope uid
* collection ttl (which is

MB-31739: [2/2] Update create/drop system events per design changes

The create event needs to transmit
* manifest uid
* collection uid
* scope uid
* collection ttl (which is optional)
* collection name

The drop event needs only to transmit
* manifest uid
* collection uid

A DCP consumer can then correctly track the entire collection
manifest using this data.

The optional max_ttl is handled by having different versions of the
create collection event, so we can cleanly trasmit the different
sized structures.

These updates are achieved by updating the flatbuffer schema and
allowing the extra data (name and ttl) to be passed through
existing layers that join collections changes to DCP.

Note that we are storing and transmitting a maxTtl, but the Manifest
does not yet accept a max_ttl field and the max_ttl values themselves
aren't applied to Items.

Change-Id: I0f66aac3000b5d95ac8cf2997e00feb6a6fce415
Reviewed-on: http://review.couchbase.org/101015
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-26907: Introduce usage of Expiration event for DCP

This commit introduces implementation of Expiration event within
DCP, which in turn triggers the Expiration code path instead of

MB-26907: Introduce usage of Expiration event for DCP

This commit introduces implementation of Expiration event within
DCP, which in turn triggers the Expiration code path instead of
the deletion path. This is done by passing the DeleteSource to
'toItem' from any code that expires a document.
This also implements the ability to output an expiry opcode from
both memory and backfill, as demonstrated by the tests included
in this patch.

Change-Id: I2809941461310fce14e5fae5786a5d2b17b87f61
Reviewed-on: http://review.couchbase.org/100561
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Refactor: Simplify DCP Control API

We used to pass this a void* and size. We always
pass strings here, so we can might as well use
a cb::const_char_buffer.

Change-Id: If9f5a

Refactor: Simplify DCP Control API

We used to pass this a void* and size. We always
pass strings here, so we can might as well use
a cb::const_char_buffer.

Change-Id: If9f5a71a74e0fae85c5569cc62e29689d7eec33d
Reviewed-on: http://review.couchbase.org/101296
Reviewed-by: Chris Farman <christopher.farman@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-26907: Update expiration protocol to include delete time

This change is due to the requirement that if DCP control has been
set to output expiry opcodes, it should also have delete ti

MB-26907: Update expiration protocol to include delete time

This change is due to the requirement that if DCP control has been
set to output expiry opcodes, it should also have delete times
enabled which is the trigger for using deletion_v2. A number of
helper functions have been added to reduce code duplication. This
patch also moves DeleteSource into memcached for use inside daemon.

Change-Id: I9a5e2a8747249b91408258b1103d3a77ac6238d8
Reviewed-on: http://review.couchbase.org/100704
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Refactor: Use the scoped enums cb::mcbp::ClientOpcode in ep-engine tests

Instead of the uint8_t definitions in PROTOCOL_BINARY_CMD_XXX

Change-Id: If1b5839ce6a3fcde3cf6c3a70f10288f54

Refactor: Use the scoped enums cb::mcbp::ClientOpcode in ep-engine tests

Instead of the uint8_t definitions in PROTOCOL_BINARY_CMD_XXX

Change-Id: If1b5839ce6a3fcde3cf6c3a70f10288f54c40ad5
Reviewed-on: http://review.couchbase.org/101270
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...