MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID (and can persist/warm up from it).

Change-Id: I66e4ad4274558992b27c5ab02adb42295e761091
Reviewed-on: http://review.couchbase.org/91178
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877b8b2b07386a9
Reviewed-on: http://review.couchbase.org/85980
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Persist the collection UID as hex

There's an inconsistency between what the set_collections manifest
JSON data encodes a UID (hex) and how the VB manifest stores the UID
(d

MB-16181: Persist the collection UID as hex

There's an inconsistency between what the set_collections manifest
JSON data encodes a UID (hex) and how the VB manifest stores the UID
(decimal).

Make the VB manifest store as hex for easier debugging/analysis of
collections.

Change-Id: Ia2d476e25b21d7161c9f9d40c437ba429b255643
Reviewed-on: http://review.couchbase.org/83352
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These eve

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These events
mapped to hidden documents (System namespace) and had unique keys.
Thus the life-span of collection beer, uid:1 may have a seqno 'stream'
as follows (e.g. if backfill from seqno 0).

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker

Now due to the way we support multiple generations of collections
before the prior one is fully purged, this approach leads us to have a
seqno stream as follows (e.g. backfill from seqno 0)

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker
seqno 6, key = $collections:create:beer:2 <- create marker
seqno 7, key = beer::document <- document in the collection
seqno 8, key = $collections:delete:beer:2 <- delete marker

With this approach (unique-keys) we have to maintain an increasing
amount of meta-data to enable correct clean-up of the old generations.
This growing meta-data would need to be in memory and the per VB
JSON manifest.

So to avoid this unbounded 'list' metadata this commit adjusts the way
collections spans are managed in-terms of checkpoints and persisted
data.

1. No longer do we have a Create and separate Delete event - a single
SystemEvent::Collection will be used to cover create and delete.

a) A single key represents all creates/deletes of a collection.
b) A delete of a collection is just a delete of that system key.

2. Collection create/delete now closes the checkpoint to ensure we
never de-dup create/delete/create into a single create, ensuring
a client tracking a vbucket sees explicit creates and deletes and
doesn't have to handle de-duplication E.g. "I've been told of
beer:2 but never saw delete(beer:1)?"

So the life span of a collection now generates sequence numbers as
follows:

seqno 3, key = $collections:beer <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index we see:

seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index after re-creating the collection,
we see.

seqno 4, key = beer::document <- document in the collection
seqno 6, key = $collections:beer <- create marker

And if we had performed cycles of create/delete we can see that there
are no longer many keys associated with the collection resulting in no
extra meta data to store so we can remove old markers.

Note in these examples, DCP wouldn't just replicate the by-seqno index
as seqno 4 can be identified as being logically deleted (it has a
seqno < start of beer). Additionally for a backfill from 0 in the
first example seqno 5 doesn't need to be sent either because the
client cannot know about the create. These enhancements are not part
of this commit.

Change-Id: I32f04e512ce039dce7d780323d9bf3642cc507fa
Reviewed-on: http://review.couchbase.org/83350
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of the UID is to allow KV-engine to see that a collection
was deleted and recreated without ever seeing the intermediate delete.
The collections management code will identify that it knows about a
collection, but because of the UID it can determine if the collection
it knows about is now defunct.

A secondary purpose is that DCP may allow clients to use UIDs when
filtering collections.

The UID must be a JSON string that stores a 64-bit unsigned integer
encoded in hex.

Note it is the cluster managements responsibility to ensure that a UID
is unique enough that it doesn't collide with a previous generation
before that generation has been fully deleted (note once a generation
has been fully deleted, a UID could be re-used).

Change-Id: I6f9bbde00783a7a93511bb5ee426b8c43d8c019f
Reviewed-on: http://review.couchbase.org/83227
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identi

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identify a
collection. This is now cleaned up so that a new class is used when
identifying a specific 'generation' of a collection.

As part of this new class we are removing the use of 'revision' and
replacing it with 'uid', however we use the Manifest::revision as
the uid and later changes will change the Manifest format so that each
collection is assigned a uid.

As part of the clean-up of revision usage this commit fixes some short
comings in the VB::Manifest management where a delete of a collection
was updating it's revision, forcing later callers of completeDeletion
to use the newer revision, instead of referring to the collection by
the correct name:rev from when it was created.

This commit additionally cleans up the separator changed path which
was needlessly passing the revision through the various code paths and
even over DCP. There is no need it was set and never read.

Change-Id: I0d81ccb9a7c7b7505d0f021b284b368f7a60f88e
Reviewed-on: http://review.couchbase.org/81984
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Move ep-engine to engines/ep

MB-16181: Build SystemEvent keys with the collections separator

The keys were fixed as $collections::<event> but are now changed
so that the :: is the collections separator.

Thi

MB-16181: Build SystemEvent keys with the collections separator

The keys were fixed as $collections::<event> but are now changed
so that the :: is the collections separator.

This allows code to split the event key if they wish using the
same code they would split document keys.

Change-Id: I48575d295f8c058a79cf208fe3c9d3a9b3c9ed15
Reviewed-on: http://review.couchbase.org/78136
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed,

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed, as follows:

1. No open collections exist.
2. Only the default collection exists.

A separator can be changed by two manifest "types" (when the above
rules are met)

1. A manifest that has a new "separator" field
2. A manifest that has a new "separator" field and a new "collection"

In the first case a new/dedicated SystemEvent is required that ensures
a serialised copy of the manifest is flushed.

In the second case the CreateCollection event will also flush the
new separator.

Change-Id: I6b563c32882b4c45ed5a3858da834ef733e08abc
Reviewed-on: http://review.couchbase.org/74608
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit updates the threading model to ensure concurrent sets and
collection manifest changes are interlocked.

All keys read and written belong to the default collection and because
the $default collection is pre-allocated, get and set with this patch
are successful, even with the collection policing in place.

Test code can now make use of new VBucket methods so that the
collection state can be changed and we can write tests that drive
data into collections (and check the collections policing).

A new class, SystemEventFlush, captures the extra state a single
run of the flusher needs for the handling of SystemEvents (the writing
special documents and _local docs)

This commit enables set/get only to be collection aware - each
datapath in/out method will need similar "isValid" checks.

Change-Id: Ia6e6cb14d7e0fa2c418e0f52d9ab38ab36baf12d
Reviewed-on: http://review.couchbase.org/73125
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Use cb::const_char_buffer over const std::string ref

It will be easier/cheaper to integrate the DCP side of the collections
work if we can update a collection from a DCP messag

MB-16181: Use cb::const_char_buffer over const std::string ref

It will be easier/cheaper to integrate the DCP side of the collections
work if we can update a collection from a DCP message. The "easy/cheaper"
benefit being that we can just use a cb::const_char_buffer initialised to
the packet's collection name and length data, rather than being forced to
create and copy into a std::string.

Change-Id: I58858e9a1f4fe7f0c6b382160764f68c63a6fb97
Reviewed-on: http://review.couchbase.org/73444
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-16181: Update completeDeletion to push system events

Update the completeDeletion function and the JSON generation so
that we can support the cases of completely deleting a collection

MB-16181: Update completeDeletion to push system events

Update the completeDeletion function and the JSON generation so
that we can support the cases of completely deleting a collection
or just updating a "isDeleting" collection.

Deleting a collection creates an event in the data-stream that
will trigger the flusher to write the manifest (subsequent integration
will update the flusher).

Change-Id: I79e304849fe89987c3b3f1a0ee7c3eef816da684
Reviewed-on: http://review.couchbase.org/73115
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Add SystemEventFactory

The SystemEventFactory creates Item objects representing
the following system events.

- create collection
- begin delete collection

MB-16181: Add SystemEventFactory

The SystemEventFactory creates Item objects representing
the following system events.

- create collection
- begin delete collection

An Item which represents a SystemEvent will eventually flow through
checkpoints to the flusher and DCP.

A simple test checks the Item is as expected.

- The operation is system_event.
- The flags return the detailed event type.

Change-Id: If8bdd92f83428538125ff18344cce71dbacfc2d9
Reviewed-on: http://review.couchbase.org/73113
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Add Collections::Manifest

A class which can convert a JSON manifest into something usable
by other parts of the code.

Change-Id: I2201457539ad9172002d5ff07d767ccf8d88b

MB-16181: Add Collections::Manifest

A class which can convert a JSON manifest into something usable
by other parts of the code.

Change-Id: I2201457539ad9172002d5ff07d767ccf8d88b7ec
Reviewed-on: http://review.couchbase.org/72457
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Remove hooksApi global; reduce coupling with MemoryTracker

MemoryTracker is somewhat tightly coupled with ep_engine.cc as it uses
the getHooksApi() function to obtain the memory allocato

Remove hooksApi global; reduce coupling with MemoryTracker

MemoryTracker is somewhat tightly coupled with ep_engine.cc as it uses
the getHooksApi() function to obtain the memory allocator hooks.

Firstly this makes it hard to test - compile one file and you have to
include the other, and it's also difficult to provide a different
hooks api - either for injecting a mock one for testing, or simply to
use the 'normal' hooks API but without pulling in ep-engine.

Secondly, there is unnecessary indirection in NewHook / DeleteHook -
which are called on every new/delete so performance is relevant
there. By giving the MemoryTracker it's own copy of the alloc_hooks
(instead of calling getHooksApi() on every call) we can reduce the
amount of work in NewHook / DeleteHook by approx. 50% (measured in
terms of x64 instructions).

Change-Id: Ia0f8ebb0a5263567dc08b32fe6ff9b7ea9eefa92
Reviewed-on: http://review.couchbase.org/68210
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...

Remove AtomicValue and Mutex defines

We already removed our own custom AtomicValue and Mutex classes when
we moved to C++11; replacing them with #defines to std::atomic and
std::mute

Remove AtomicValue and Mutex defines

We already removed our own custom AtomicValue and Mutex classes when
we moved to C++11; replacing them with #defines to std::atomic and
std::mutex respectively. This patch completes that changover, using
the underlying types directly so it's easier to see what is actually
being used.

Change-Id: If3f717f766727042cd81ae7b8c65f9a19934f2fa
Reviewed-on: http://review.couchbase.org/60925
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Will Gardner <will.gardner@couchbase.com>

show more ...

Mutex modernization [2/2]: Use std::condition_variable for SyncObject

Change Mutex to a typedef to std::mutex, and use
std::condition_variable for the implementation of SyncObject.

Mutex modernization [2/2]: Use std::condition_variable for SyncObject

Change Mutex to a typedef to std::mutex, and use
std::condition_variable for the implementation of SyncObject.

Note these two changes are mutually dependent hence being done in the
same patch.

Change-Id: Ife74822e9c4e04efefff446e964a952b672a1f91
Reviewed-on: http://review.couchbase.org/57328
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Mutex modernization [1/2]: Replace with std::mutex

Update the API of Mutex to match that of std::mutex, and SyncObject to
match that of std::condition_variable in preparation for replaci

Mutex modernization [1/2]: Replace with std::mutex

Update the API of Mutex to match that of std::mutex, and SyncObject to
match that of std::condition_variable in preparation for replacing
Mutex and SyncObject with the C++11 standard library equivilents.

Change-Id: I5625d980b11144f681f7e717df87c8b5f323dc7c
Reviewed-on: http://review.couchbase.org/57327
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...

Simplify common.h: add utility.h [2/3]

Create a new utility.h header file and move some of the functions /
macros which virtually everyone uses to there. utility.h has
significantly

Simplify common.h: add utility.h [2/3]

Create a new utility.h header file and move some of the functions /
macros which virtually everyone uses to there. utility.h has
significantly fewer dependancies, and such is much cheaper to include
than common.h

Update users of these functions / macros (DISALLOW_COPY_ON_ASSIGN, LOG
etc) to use utility.h instead of common.h

Change-Id: I3f93435c82950bd2a20b416af2b2e5011a931e04
Reviewed-on: http://review.couchbase.org/56704
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11731 Reduce the lock overhead among flusher, VB deletion, and compaction

This change reduces the lock blocking overhead among flusher, vbucket
deletion, compaction, and snapshot task

MB-11731 Reduce the lock overhead among flusher, VB deletion, and compaction

This change reduces the lock blocking overhead among flusher, vbucket
deletion, compaction, and snapshot tasks, by skipping the rest of
tasks if a lock is already grabbed by one task.

Change-Id: I643d7fe74150b5ead5c1368d660deb3c64dd1d01
Reviewed-on: http://review.couchbase.org/39910
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

call cb_thread_self instead of pthread_self

Change-Id: I2509b9710b0caa0324ec39a18cf82cd8cb1751b8
Reviewed-on: http://review.couchbase.org/30092
Reviewed-by: Trond Norbye <trond.norby

call cb_thread_self instead of pthread_self

Change-Id: I2509b9710b0caa0324ec39a18cf82cd8cb1751b8
Reviewed-on: http://review.couchbase.org/30092
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

Use our own mutex abstraction

Change-Id: I12faa0016977e1d5b08cdd05f509aebf71a5bbb8
Reviewed-on: http://review.couchbase.org/30014
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Use our own mutex abstraction

Change-Id: I12faa0016977e1d5b08cdd05f509aebf71a5bbb8
Reviewed-on: http://review.couchbase.org/30014
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Use our thread abstraction layer

Change-Id: I5d52d55ea603d8e53c9b39a94fa3029cdad50748
Reviewed-on: http://review.couchbase.org/30013
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com

Use our thread abstraction layer

Change-Id: I5d52d55ea603d8e53c9b39a94fa3029cdad50748
Reviewed-on: http://review.couchbase.org/30013
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-7728: Add copyright headers to all files

This is something that our cpplint tool turns on and it makes
our product look a little bit more profesional. I have also
enabled the tool

MB-7728: Add copyright headers to all files

This is something that our cpplint tool turns on and it makes
our product look a little bit more profesional. I have also
enabled the tool to begin checking for these headers so it will
start complaining if we add a new file without a copyright header.

Change-Id: Ie956fdf7027bf56d9665ee455acd3c109fcacc76
Reviewed-on: http://review.couchbase.org/24560
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...