MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID (and can persist/warm up from it).

Change-Id: I66e4ad4274558992b27c5ab02adb42295e761091
Reviewed-on: http://review.couchbase.org/91178
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-28773: Save and reload the manifest UID (via VB)

When a collection change occurs (via set-manifest)
save the manifest UID into each affected VB's meta-data.

When warming up,

MB-28773: Save and reload the manifest UID (via VB)

When a collection change occurs (via set-manifest)
save the manifest UID into each affected VB's meta-data.

When warming up, reload the manifest UID into the VB.

Change-Id: I169c2e0c7ee739dea8b72a21f34f02676e08acba
Reviewed-on: http://review.couchbase.org/91177
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Persist the collection UID as hex

There's an inconsistency between what the set_collections manifest
JSON data encodes a UID (hex) and how the VB manifest stores the UID
(d

MB-16181: Persist the collection UID as hex

There's an inconsistency between what the set_collections manifest
JSON data encodes a UID (hex) and how the VB manifest stores the UID
(decimal).

Make the VB manifest store as hex for easier debugging/analysis of
collections.

Change-Id: Ia2d476e25b21d7161c9f9d40c437ba429b255643
Reviewed-on: http://review.couchbase.org/83352
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These eve

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These events
mapped to hidden documents (System namespace) and had unique keys.
Thus the life-span of collection beer, uid:1 may have a seqno 'stream'
as follows (e.g. if backfill from seqno 0).

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker

Now due to the way we support multiple generations of collections
before the prior one is fully purged, this approach leads us to have a
seqno stream as follows (e.g. backfill from seqno 0)

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker
seqno 6, key = $collections:create:beer:2 <- create marker
seqno 7, key = beer::document <- document in the collection
seqno 8, key = $collections:delete:beer:2 <- delete marker

With this approach (unique-keys) we have to maintain an increasing
amount of meta-data to enable correct clean-up of the old generations.
This growing meta-data would need to be in memory and the per VB
JSON manifest.

So to avoid this unbounded 'list' metadata this commit adjusts the way
collections spans are managed in-terms of checkpoints and persisted
data.

1. No longer do we have a Create and separate Delete event - a single
SystemEvent::Collection will be used to cover create and delete.

a) A single key represents all creates/deletes of a collection.
b) A delete of a collection is just a delete of that system key.

2. Collection create/delete now closes the checkpoint to ensure we
never de-dup create/delete/create into a single create, ensuring
a client tracking a vbucket sees explicit creates and deletes and
doesn't have to handle de-duplication E.g. "I've been told of
beer:2 but never saw delete(beer:1)?"

So the life span of a collection now generates sequence numbers as
follows:

seqno 3, key = $collections:beer <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index we see:

seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index after re-creating the collection,
we see.

seqno 4, key = beer::document <- document in the collection
seqno 6, key = $collections:beer <- create marker

And if we had performed cycles of create/delete we can see that there
are no longer many keys associated with the collection resulting in no
extra meta data to store so we can remove old markers.

Note in these examples, DCP wouldn't just replicate the by-seqno index
as seqno 4 can be identified as being logically deleted (it has a
seqno < start of beer). Additionally for a backfill from 0 in the
first example seqno 5 doesn't need to be sent either because the
client cannot know about the create. These enhancements are not part
of this commit.

Change-Id: I32f04e512ce039dce7d780323d9bf3642cc507fa
Reviewed-on: http://review.couchbase.org/83350
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identi

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identify a
collection. This is now cleaned up so that a new class is used when
identifying a specific 'generation' of a collection.

As part of this new class we are removing the use of 'revision' and
replacing it with 'uid', however we use the Manifest::revision as
the uid and later changes will change the Manifest format so that each
collection is assigned a uid.

As part of the clean-up of revision usage this commit fixes some short
comings in the VB::Manifest management where a delete of a collection
was updating it's revision, forcing later callers of completeDeletion
to use the newer revision, instead of referring to the collection by
the correct name:rev from when it was created.

This commit additionally cleans up the separator changed path which
was needlessly passing the revision through the various code paths and
even over DCP. There is no need it was set and never read.

Change-Id: I0d81ccb9a7c7b7505d0f021b284b368f7a60f88e
Reviewed-on: http://review.couchbase.org/81984
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Move ep-engine to engines/ep

MB-16181: Collections DcpConsumer

Add the functional paths that allow a replica VBucket's manifest to
be updated from incoming DCP messages. Note that the DCP protocol
has not been w

MB-16181: Collections DcpConsumer

Add the functional paths that allow a replica VBucket's manifest to
be updated from incoming DCP messages. Note that the DCP protocol
has not been written, but the API defined here shows what data needs
to be transfered over-the-wire.

Add a unit test to use MockDcpConsumer to drive a create and delete

Change-Id: Ida8171b2cde104ccd452fafcd6193a72455b404b
Reviewed-on: http://review.couchbase.org/74609
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Interface additions and manifest logic for DCP

Replica vbuckets will learn of collections from their active peer.
Add new functions which allow for addition and deletion of col

MB-16181: Interface additions and manifest logic for DCP

Replica vbuckets will learn of collections from their active peer.
Add new functions which allow for addition and deletion of collections
with DCP in mind, that is addition/deletion with pre-assigned sequence
numbers.

The manifest test code is now updated to keep a replica manifest in-sync
using the data the active places in the checkpoint. A very loose DCP
simulation.

Change-Id: If7bd2dbbf16e66118bc0759f80b8b395a7370825
Reviewed-on: http://review.couchbase.org/73508
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed,

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed, as follows:

1. No open collections exist.
2. Only the default collection exists.

A separator can be changed by two manifest "types" (when the above
rules are met)

1. A manifest that has a new "separator" field
2. A manifest that has a new "separator" field and a new "collection"

In the first case a new/dedicated SystemEvent is required that ensures
a serialised copy of the manifest is flushed.

In the second case the CreateCollection event will also flush the
new separator.

Change-Id: I6b563c32882b4c45ed5a3858da834ef733e08abc
Reviewed-on: http://review.couchbase.org/74608
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit updates the threading model to ensure concurrent sets and
collection manifest changes are interlocked.

All keys read and written belong to the default collection and because
the $default collection is pre-allocated, get and set with this patch
are successful, even with the collection policing in place.

Test code can now make use of new VBucket methods so that the
collection state can be changed and we can write tests that drive
data into collections (and check the collections policing).

A new class, SystemEventFlush, captures the extra state a single
run of the flusher needs for the handling of SystemEvents (the writing
special documents and _local docs)

This commit enables set/get only to be collection aware - each
datapath in/out method will need similar "isValid" checks.

Change-Id: Ia6e6cb14d7e0fa2c418e0f52d9ab38ab36baf12d
Reviewed-on: http://review.couchbase.org/73125
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Use cb::const_char_buffer over const std::string ref

It will be easier/cheaper to integrate the DCP side of the collections
work if we can update a collection from a DCP messag

MB-16181: Use cb::const_char_buffer over const std::string ref

It will be easier/cheaper to integrate the DCP side of the collections
work if we can update a collection from a DCP message. The "easy/cheaper"
benefit being that we can just use a cb::const_char_buffer initialised to
the packet's collection name and length data, rather than being forced to
create and copy into a std::string.

Change-Id: I58858e9a1f4fe7f0c6b382160764f68c63a6fb97
Reviewed-on: http://review.couchbase.org/73444
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-16181: Update completeDeletion to push system events

Update the completeDeletion function and the JSON generation so
that we can support the cases of completely deleting a collection

MB-16181: Update completeDeletion to push system events

Update the completeDeletion function and the JSON generation so
that we can support the cases of completely deleting a collection
or just updating a "isDeleting" collection.

Deleting a collection creates an event in the data-stream that
will trigger the flusher to write the manifest (subsequent integration
will update the flusher).

Change-Id: I79e304849fe89987c3b3f1a0ee7c3eef816da684
Reviewed-on: http://review.couchbase.org/73115
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Create and BeginDelete collection against a VB.

Update the VB::Manifest to now drive an event through a
VBucket.

The VBucket can then generate a JSON manifest from the

MB-16181: Create and BeginDelete collection against a VB.

Update the VB::Manifest to now drive an event through a
VBucket.

The VBucket can then generate a JSON manifest from the
system-event item. This is achieved by serialising a
VB::Manifest into the system-event value so that at
flushing time the correct JSON represntation can be
produced.

Change-Id: I8b7d4a1fcf39d753f28eef6cc5367ed47de9d257
Reviewed-on: http://review.couchbase.org/73114
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...