MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose

MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose
is to allow easy identification of a buckets collection settings.

Change-Id: Ic660ae018e6782ceee07a58c14fa391741ddb4d1
Reviewed-on: http://review.couchbase.org/91176
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-27121: [1/3] Add GET_MANIFEST support to KV Manifest class

The GET_MANIFEST command will return the last manifest which was
applied using SET_MANIFEST.

Add a toJson method to

MB-27121: [1/3] Add GET_MANIFEST support to KV Manifest class

The GET_MANIFEST command will return the last manifest which was
applied using SET_MANIFEST.

Add a toJson method to Collections::Manifest which returns a legal
JSON std::string.

Change-Id: I48f608c341c5707f888798dedcc695e15334e9ed
Reviewed-on: http://review.couchbase.org/86440
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-27122: [2/2] Do not initialise the current manifest

As described in the MB, it is risky to initialise the current manifest
to a 'default' state. It is safer to run without a manifest

MB-27122: [2/2] Do not initialise the current manifest

As described in the MB, it is risky to initialise the current manifest
to a 'default' state. It is safer to run without a manifest as we
really should only operate on what the cluster_manager tells us.

This leads to one situation in DcpOpen where we must fail the open
if the manifest hasn't yet been set, this situation is now flagged
by a new error code "no_collections_manifest" allowing what should
be a temporary failure (cluster manager should always push a manifest)
to be observed.

Change-Id: I79804bbc97da6ba00182355815abe53ad1125da1
Reviewed-on: http://review.couchbase.org/86514
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Make collection size be configurable

Current PRD stated 1,000 collections per bucket maximum, so enforce
this using a config param - "collections_max_size"

Change-Id:

MB-16181: Make collection size be configurable

Current PRD stated 1,000 collections per bucket maximum, so enforce
this using a config param - "collections_max_size"

Change-Id: I482d96a806539aaadf0a045a21b49f398bc899d3
Reviewed-on: http://review.couchbase.org/86000
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Persist the collection UID as hex

There's an inconsistency between what the set_collections manifest
JSON data encodes a UID (hex) and how the VB manifest stores the UID
(d

MB-16181: Persist the collection UID as hex

There's an inconsistency between what the set_collections manifest
JSON data encodes a UID (hex) and how the VB manifest stores the UID
(decimal).

Make the VB manifest store as hex for easier debugging/analysis of
collections.

Change-Id: Ia2d476e25b21d7161c9f9d40c437ba429b255643
Reviewed-on: http://review.couchbase.org/83352
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of the UID is to allow KV-engine to see that a collection
was deleted and recreated without ever seeing the intermediate delete.
The collections management code will identify that it knows about a
collection, but because of the UID it can determine if the collection
it knows about is now defunct.

A secondary purpose is that DCP may allow clients to use UIDs when
filtering collections.

The UID must be a JSON string that stores a 64-bit unsigned integer
encoded in hex.

Note it is the cluster managements responsibility to ensure that a UID
is unique enough that it doesn't collide with a previous generation
before that generation has been fully deleted (note once a generation
has been fully deleted, a UID could be re-used).

Change-Id: I6f9bbde00783a7a93511bb5ee426b8c43d8c019f
Reviewed-on: http://review.couchbase.org/83227
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identi

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identify a
collection. This is now cleaned up so that a new class is used when
identifying a specific 'generation' of a collection.

As part of this new class we are removing the use of 'revision' and
replacing it with 'uid', however we use the Manifest::revision as
the uid and later changes will change the Manifest format so that each
collection is assigned a uid.

As part of the clean-up of revision usage this commit fixes some short
comings in the VB::Manifest management where a delete of a collection
was updating it's revision, forcing later callers of completeDeletion
to use the newer revision, instead of referring to the collection by
the correct name:rev from when it was created.

This commit additionally cleans up the separator changed path which
was needlessly passing the revision through the various code paths and
even over DCP. There is no need it was set and never read.

Change-Id: I0d81ccb9a7c7b7505d0f021b284b368f7a60f88e
Reviewed-on: http://review.couchbase.org/81984
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-16181: Add more ostream operators and dump methods

Add more ostream operators to the Collections classes and also give
them dump() methods so we can easily trigger output to stderr.

MB-16181: Add more ostream operators and dump methods

Add more ostream operators to the Collections classes and also give
them dump() methods so we can easily trigger output to stderr.

Finally add a hook so we can force the logging of a lot of collection
state, this will eventually be migrated to stats and more supported
debug methods.

Change-Id: I18abe41cd8f00fe18cb4fb972fcc054375f02d39
Reviewed-on: http://review.couchbase.org/78639
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Move ep-engine to engines/ep

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will esta

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will establish a Collections::Filter
that lives for the lifetime of the DCP producer.

As the DCP producer creates streams, a Collections::VB::Filter is
assigned to the stream which contains the real set of collections to
filter (and also the actual "filter" function).

Change-Id: I2f35b1698ce977116486a2e6940437eee25faef1
Reviewed-on: http://review.couchbase.org/78137
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Make server deny creation of reserved collections

The design has always stated that _ and $ would be reserved for
system use. Make sure the server enforces this by checking the

MB-16181: Make server deny creation of reserved collections

The design has always stated that _ and $ would be reserved for
system use. Make sure the server enforces this by checking the
names on incoming manifests.

1. Any _ prefixed collection will cause the Manifest construction to
throw invalid_argument.

2. Any $ prefixed collection which is not $default will cause the
Manifest construction to throw invalid_argument.

Change-Id: I1e5daf2ae87cba2a8dbcdda4c9bb0be66e40ffae
Reviewed-on: http://review.couchbase.org/78074
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Add Collections::Manifest

A class which can convert a JSON manifest into something usable
by other parts of the code.

Change-Id: I2201457539ad9172002d5ff07d767ccf8d88b

MB-16181: Add Collections::Manifest

A class which can convert a JSON manifest into something usable
by other parts of the code.

Change-Id: I2201457539ad9172002d5ff07d767ccf8d88b7ec
Reviewed-on: http://review.couchbase.org/72457
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...