MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a int

MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a intermediate
collection delete during their downtime.

This commit updates Collections::Filter so two formats of JSON are
valid and updates Collections::VB::Filter to use the strict name/uid
checks when the client uses the name:uid format.

A subsequent patch will ensure that the client does a name:uid request
when the start-seqno is non-zero.

Change-Id: I3cae53bf855d4b8001bcc65b6f036a9053b88307
Reviewed-on: http://review.couchbase.org/85981
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID (and can persist/warm up from it).

Change-Id: I66e4ad4274558992b27c5ab02adb42295e761091
Reviewed-on: http://review.couchbase.org/91178
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-28773: Save and reload the manifest UID (via VB)

When a collection change occurs (via set-manifest)
save the manifest UID into each affected VB's meta-data.

When warming up,

MB-28773: Save and reload the manifest UID (via VB)

When a collection change occurs (via set-manifest)
save the manifest UID into each affected VB's meta-data.

When warming up, reload the manifest UID into the VB.

Change-Id: I169c2e0c7ee739dea8b72a21f34f02676e08acba
Reviewed-on: http://review.couchbase.org/91177
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Add Collections::VB::Manifest::CachingReadHandle

The caching readhandle allows limited read access to the manifest
but fits into the functional flow of front-end operations.

MB-25342: Add Collections::VB::Manifest::CachingReadHandle

The caching readhandle allows limited read access to the manifest
but fits into the functional flow of front-end operations.

When constructing the CachingReadHandle from a key, the key is scanned
and a map lookup is performed. The result of the lookup is saved so
that further down the code path, subsequent isLogicallyDeleted checks
don't need to scan the key again and perform another map lookup.

Change-Id: Icffaf8a722f4a9e7e67bce870445cd3f75f940e3
Reviewed-on: http://review.couchbase.org/84474
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-16181: Const correctness on Collections ReadHandle

The ReadHandle currently allows no changes to be made to the
Manifest, so should have every method marked const. Two methods
had

MB-16181: Const correctness on Collections ReadHandle

The ReadHandle currently allows no changes to be made to the
Manifest, so should have every method marked const. Two methods
had escaped this.

Change-Id: I1a8143907978a81ec3fcc8fcd21fad39aba4ee73
Reviewed-on: http://review.couchbase.org/84746
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Reduce map lookups by tracking the largest endSeqno

To determine if a key@seqno is logically deleted we are splitting the
key and looking up in the manifest map for the collect

MB-25342: Reduce map lookups by tracking the largest endSeqno

To determine if a key@seqno is logically deleted we are splitting the
key and looking up in the manifest map for the collection's metadata.

However if we track the greatest end-seqno we can

1) Only do map lookups when the key@seqno is part of the deleted range
2) Quickly determine the key is not logically deleted when no
collections are in the deleting state, no lookups at all.

Change-Id: Ia9d164fc2e97dc4d23e501b4f03d9f74cff84a28
Reviewed-on: http://review.couchbase.org/83570
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Erase deleted collection keys during in compaction

Add a new call back (using std::function) so that the compactor can
check if a key belongs to a deleted collection and should

MB-25342: Erase deleted collection keys during in compaction

Add a new call back (using std::function) so that the compactor can
check if a key belongs to a deleted collection and should be dropped
from the database.

Change-Id: Iebefd385cfcae38745c81563113c025bba90890e
Reviewed-on: http://review.couchbase.org/83559
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Rename and update doesKeyContainDeletingCollection

Rename this function isLogicallyDeleted to test if a key belongs to
a deleted collection.

Update the function to con

MB-16181: Rename and update doesKeyContainDeletingCollection

Rename this function isLogicallyDeleted to test if a key belongs to
a deleted collection.

Update the function to consider System keys because a Collection
event key may also be considered deleted by future callers.

Change-Id: I76e07d26e9236d4a5e48be2b7e3c0f41ba0d7221
Reviewed-on: http://review.couchbase.org/83631
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These eve

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These events
mapped to hidden documents (System namespace) and had unique keys.
Thus the life-span of collection beer, uid:1 may have a seqno 'stream'
as follows (e.g. if backfill from seqno 0).

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker

Now due to the way we support multiple generations of collections
before the prior one is fully purged, this approach leads us to have a
seqno stream as follows (e.g. backfill from seqno 0)

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker
seqno 6, key = $collections:create:beer:2 <- create marker
seqno 7, key = beer::document <- document in the collection
seqno 8, key = $collections:delete:beer:2 <- delete marker

With this approach (unique-keys) we have to maintain an increasing
amount of meta-data to enable correct clean-up of the old generations.
This growing meta-data would need to be in memory and the per VB
JSON manifest.

So to avoid this unbounded 'list' metadata this commit adjusts the way
collections spans are managed in-terms of checkpoints and persisted
data.

1. No longer do we have a Create and separate Delete event - a single
SystemEvent::Collection will be used to cover create and delete.

a) A single key represents all creates/deletes of a collection.
b) A delete of a collection is just a delete of that system key.

2. Collection create/delete now closes the checkpoint to ensure we
never de-dup create/delete/create into a single create, ensuring
a client tracking a vbucket sees explicit creates and deletes and
doesn't have to handle de-duplication E.g. "I've been told of
beer:2 but never saw delete(beer:1)?"

So the life span of a collection now generates sequence numbers as
follows:

seqno 3, key = $collections:beer <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index we see:

seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index after re-creating the collection,
we see.

seqno 4, key = beer::document <- document in the collection
seqno 6, key = $collections:beer <- create marker

And if we had performed cycles of create/delete we can see that there
are no longer many keys associated with the collection resulting in no
extra meta data to store so we can remove old markers.

Note in these examples, DCP wouldn't just replicate the by-seqno index
as seqno 4 can be identified as being logically deleted (it has a
seqno < start of beer). Additionally for a backfill from 0 in the
first example seqno 5 doesn't need to be sent either because the
client cannot know about the create. These enhancements are not part
of this commit.

Change-Id: I32f04e512ce039dce7d780323d9bf3642cc507fa
Reviewed-on: http://review.couchbase.org/83350
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Refactor exception throwing in VB::Manifest

Add helper functions so we can throw with consistent error strings.

Change-Id: I1a127872625f2a5cc49047c883080e572930af7f
Re

MB-16181: Refactor exception throwing in VB::Manifest

Add helper functions so we can throw with consistent error strings.

Change-Id: I1a127872625f2a5cc49047c883080e572930af7f
Reviewed-on: http://review.couchbase.org/83229
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of the UID is to allow KV-engine to see that a collection
was deleted and recreated without ever seeing the intermediate delete.
The collections management code will identify that it knows about a
collection, but because of the UID it can determine if the collection
it knows about is now defunct.

A secondary purpose is that DCP may allow clients to use UIDs when
filtering collections.

The UID must be a JSON string that stores a 64-bit unsigned integer
encoded in hex.

Note it is the cluster managements responsibility to ensure that a UID
is unique enough that it doesn't collide with a previous generation
before that generation has been fully deleted (note once a generation
has been fully deleted, a UID could be re-used).

Change-Id: I6f9bbde00783a7a93511bb5ee426b8c43d8c019f
Reviewed-on: http://review.couchbase.org/83227
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identi

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identify a
collection. This is now cleaned up so that a new class is used when
identifying a specific 'generation' of a collection.

As part of this new class we are removing the use of 'revision' and
replacing it with 'uid', however we use the Manifest::revision as
the uid and later changes will change the Manifest format so that each
collection is assigned a uid.

As part of the clean-up of revision usage this commit fixes some short
comings in the VB::Manifest management where a delete of a collection
was updating it's revision, forcing later callers of completeDeletion
to use the newer revision, instead of referring to the collection by
the correct name:rev from when it was created.

This commit additionally cleans up the separator changed path which
was needlessly passing the revision through the various code paths and
even over DCP. There is no need it was set and never read.

Change-Id: I0d81ccb9a7c7b7505d0f021b284b368f7a60f88e
Reviewed-on: http://review.couchbase.org/81984
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Reorder Collections::VB::Manifest update/serialise

Previous to this commit an update to the Collections::VB::Manifest
would:

1: Serialise the manifest state to a syste

MB-25342: Reorder Collections::VB::Manifest update/serialise

Previous to this commit an update to the Collections::VB::Manifest
would:

1: Serialise the manifest state to a system-event
2: Update the manifest

This ordering is creating issues for new code that is performing full
deletion of the collection data. The issue being that some new changes
being made by step 2 don't get captured in step 1.

It is much simpler to (and really should of been like this from the
start):

1: Update the manifest
2: Serialise the manifest state to a system-event

This patch refactors the add/beginDelete methods so that they return
the ManifestEntry which is affected by the update allowing us to
"patch" it with the seqno we get from queueing of the serialised
manifest.

::completeDeletion is also updated but not refactored.

VbucketManifestTest is also altered slightly so that completeDeletion
is given more test coverage, it now applies updates to the replica
and compares active with replica.

Change-Id: I7361c5675356e47888afcf751dc4db21ac03762e
Reviewed-on: http://review.couchbase.org/81983
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Will Gardner <willg@rdner.io>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Add a method to check if a key is in a deleted collection

Simple method that searches for the collection entry and checks
the key::seqno against the entry::endSeqno.

C

MB-25342: Add a method to check if a key is in a deleted collection

Simple method that searches for the collection entry and checks
the key::seqno against the entry::endSeqno.

Change-Id: I2a413852df45c30d2569bae083b59abc35c039f3
Reviewed-on: http://review.couchbase.org/81167
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>

show more ...

MB-16181: Add more ostream operators and dump methods

Add more ostream operators to the Collections classes and also give
them dump() methods so we can easily trigger output to stderr.

MB-16181: Add more ostream operators and dump methods

Add more ostream operators to the Collections classes and also give
them dump() methods so we can easily trigger output to stderr.

Finally add a hook so we can force the logging of a lot of collection
state, this will eventually be migrated to stats and more supported
debug methods.

Change-Id: I18abe41cd8f00fe18cb4fb972fcc054375f02d39
Reviewed-on: http://review.couchbase.org/78639
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Integrate filters into DcpProducer and ActiveStream

The DCP producer now adds a filter to itself when opened, the filter
is configured from the DCP open input data.

Op

MB-16181: Integrate filters into DcpProducer and ActiveStream

The DCP producer now adds a filter to itself when opened, the filter
is configured from the DCP open input data.

Opening DCP without DCP_OPEN_COLLECTIONS allows access to the
default collection only (if it exists).

Opening DCP with DCP_OPEN_COLLECTIONS allows the client to specify
a filter or if no-value is given, a passthrough filter is created.

Change-Id: I6a8c24c0af20a07ddd029325487172b516e54b00
Reviewed-on: http://review.couchbase.org/78637
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Move ep-engine to engines/ep

MB-16181: Filters with deleted collections

Upstream integration revealed an issue with the VB::Filter.
If we construct a filter with a collection "X" and "X" is actually
marked as de

MB-16181: Filters with deleted collections

Upstream integration revealed an issue with the VB::Filter.
If we construct a filter with a collection "X" and "X" is actually
marked as deleted, "X" should not be part of the filter.

We fix this by replacing VB::Manifest::doesCollectionExist with
VB::Manifest::isCollectionOpen which performs the correct checks.

Tests updated to cover this case.

Change-Id: I38d4ba025805da7653bf3a84053d4280f4f547d7
Reviewed-on: http://review.couchbase.org/78570
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will esta

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will establish a Collections::Filter
that lives for the lifetime of the DCP producer.

As the DCP producer creates streams, a Collections::VB::Filter is
assigned to the stream which contains the real set of collections to
filter (and also the actual "filter" function).

Change-Id: I2f35b1698ce977116486a2e6940437eee25faef1
Reviewed-on: http://review.couchbase.org/78137
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Make ActiveStream track the current collections separator

This patch updates the ActiveStream so that it stores a copy of the current
separator and tracks changes to as they ar

MB-16181: Make ActiveStream track the current collections separator

This patch updates the ActiveStream so that it stores a copy of the current
separator and tracks changes to as they are transmitted through the checkpoint
via the separator changed SystemEvent.

Change-Id: Ie3ea87d006b0bbab3e0edd8895a4756c7c5d9fe8
Reviewed-on: http://review.couchbase.org/75502
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Interface additions and manifest logic for DCP

Replica vbuckets will learn of collections from their active peer.
Add new functions which allow for addition and deletion of col

MB-16181: Interface additions and manifest logic for DCP

Replica vbuckets will learn of collections from their active peer.
Add new functions which allow for addition and deletion of collections
with DCP in mind, that is addition/deletion with pre-assigned sequence
numbers.

The manifest test code is now updated to keep a replica manifest in-sync
using the data the active places in the checkpoint. A very loose DCP
simulation.

Change-Id: If7bd2dbbf16e66118bc0759f80b8b395a7370825
Reviewed-on: http://review.couchbase.org/73508
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Collections warmup support

The VBucket manifest state is persisted to disk and is used in the
construction of VBucket's on the warmup path.

Includes a test which creat

MB-16181: Collections warmup support

The VBucket manifest state is persisted to disk and is used in the
construction of VBucket's on the warmup path.

Includes a test which creates collections then forces a warmup
and checks those collections are still valid.

Change-Id: I8e237a72d0571294e9a821dded9f61756cb13ac3
Reviewed-on: http://review.couchbase.org/73507
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed,

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed, as follows:

1. No open collections exist.
2. Only the default collection exists.

A separator can be changed by two manifest "types" (when the above
rules are met)

1. A manifest that has a new "separator" field
2. A manifest that has a new "separator" field and a new "collection"

In the first case a new/dedicated SystemEvent is required that ensures
a serialised copy of the manifest is flushed.

In the second case the CreateCollection event will also flush the
new separator.

Change-Id: I6b563c32882b4c45ed5a3858da834ef733e08abc
Reviewed-on: http://review.couchbase.org/74608
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit updates the threading model to ensure concurrent sets and
collection manifest changes are interlocked.

All keys read and written belong to the default collection and because
the $default collection is pre-allocated, get and set with this patch
are successful, even with the collection policing in place.

Test code can now make use of new VBucket methods so that the
collection state can be changed and we can write tests that drive
data into collections (and check the collections policing).

A new class, SystemEventFlush, captures the extra state a single
run of the flusher needs for the handling of SystemEvents (the writing
special documents and _local docs)

This commit enables set/get only to be collection aware - each
datapath in/out method will need similar "isValid" checks.

Change-Id: Ia6e6cb14d7e0fa2c418e0f52d9ab38ab36baf12d
Reviewed-on: http://review.couchbase.org/73125
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...