MB-29040: [1/2] Correctly strip user xattrs

A bug in the expiry path means a document with user xattrs is not
having the body removed. Now when we rebalance and such a deleted
docume

MB-29040: [1/2] Correctly strip user xattrs

A bug in the expiry path means a document with user xattrs is not
having the body removed. Now when we rebalance and such a deleted
document is transferred, it fails validation because the datatype
is xattr and json.

Change-Id: Ie5a71e6fc640bfd614c8bbd68c91b99fbdf8aed2
Reviewed-on: http://review.couchbase.org/92616
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose

MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose
is to allow easy identification of a buckets collection settings.

Change-Id: Ic660ae018e6782ceee07a58c14fa391741ddb4d1
Reviewed-on: http://review.couchbase.org/91176
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Fix removeKey as value evicted items still need removing

The removeKey function is only removing keys if they're resident.
resident or not, if the key is found and has a matchi

MB-25342: Fix removeKey as value evicted items still need removing

The removeKey function is only removing keys if they're resident.
resident or not, if the key is found and has a matching seqno, delete
it.

Change-Id: I57b7a5c74d2185afacd1447cf8e77ca6c30dbfee
Reviewed-on: http://review.couchbase.org/89001
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877b8b2b07386a9
Reviewed-on: http://review.couchbase.org/85980
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Reduce map lookups by tracking the largest endSeqno

To determine if a key@seqno is logically deleted we are splitting the
key and looking up in the manifest map for the collect

MB-25342: Reduce map lookups by tracking the largest endSeqno

To determine if a key@seqno is logically deleted we are splitting the
key and looking up in the manifest map for the collection's metadata.

However if we track the greatest end-seqno we can

1) Only do map lookups when the key@seqno is part of the deleted range
2) Quickly determine the key is not logically deleted when no
collections are in the deleting state, no lookups at all.

Change-Id: Ia9d164fc2e97dc4d23e501b4f03d9f74cff84a28
Reviewed-on: http://review.couchbase.org/83570
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Full eviction collections eraser tests skipping item count

The item count check was disabled for FE temporarily due to it mis
counting, we can now remove the function and have

MB-25342: Full eviction collections eraser tests skipping item count

The item count check was disabled for FE temporarily due to it mis
counting, we can now remove the function and have both VE and FE tests
check the final item counts.

Change-Id: Ia17363f36a5a3a5f8af0d010482eb3e4908d3e16
Reviewed-on: http://review.couchbase.org/84636
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Erase deleted collection keys during in compaction

Add a new call back (using std::function) so that the compactor can
check if a key belongs to a deleted collection and should

MB-25342: Erase deleted collection keys during in compaction

Add a new call back (using std::function) so that the compactor can
check if a key belongs to a deleted collection and should be dropped
from the database.

Change-Id: Iebefd385cfcae38745c81563113c025bba90890e
Reviewed-on: http://review.couchbase.org/83559
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...