MB-29675: Optimize collections lookups when disabled

While collections code is present in Vulcan, it is not enabled in
production configurations - only the default collection is present.

MB-29675: Optimize collections lookups when disabled

While collections code is present in Vulcan, it is not enabled in
production configurations - only the default collection is present.

As such, optimize a number of functions which show up in 'linux perf'
profiles:

- Use std::string[] instead of string::at() do avoid the runtime size
check (and setup for throwing an exception) in StoredDocKey.

- Split Collections::VB::Filter::checkAndUpdate() into a hot and slow
path, and inline the hot path (This is called for every outgoing DCP
mutation).

- Add an explicit case for DefaultCollection in
Collections::DocKey::make() - this avoids an unnecessary call to
findCollection which otherwise needs to scan the document key.

Change-Id: I4c4eeb6ab26dc616728f4dc89ac1fd4243c21e98
Reviewed-on: http://review.couchbase.org/94214
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a int

MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a intermediate
collection delete during their downtime.

This commit updates Collections::Filter so two formats of JSON are
valid and updates Collections::VB::Filter to use the strict name/uid
checks when the client uses the name:uid format.

A subsequent patch will ensure that the client does a name:uid request
when the start-seqno is non-zero.

Change-Id: I3cae53bf855d4b8001bcc65b6f036a9053b88307
Reviewed-on: http://review.couchbase.org/85981
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream_end.

Change-Id: If09cc37e113f3019a3a1c743e60ea9bb2537691b
Reviewed-on: http://review.couchbase.org/84603
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-24572: Stop Collection aware DCP from sending empty snapshots

If a DCP stream is configured with filtering enabled and the filter
results in 0 mutations, we still send snapshot marker

MB-24572: Stop Collection aware DCP from sending empty snapshots

If a DCP stream is configured with filtering enabled and the filter
results in 0 mutations, we still send snapshot markers to the client.

"Hey here's 500 to 510" ... silence

In a needle/haystack situation, the client looking for needles will
get frequent snapshot markers triggered by the mutations made against
the haystack, it is preferential that the needle client gets nothing
until the needle collection is touched.

To recap on snapshots - a DCP snapshot is "a series of commands that
is guaranteed to contain a unique set of keys.", it isn't necessarily
representative of a disk checkpoint (commit).

Prior to this commit we may then do the following:

checkpoint: 0 1 2 3 4
[a1, b1, a2, b2, b3]

some example snapshot possibilities {start, end, commands}

unfiltered:

{0, 0, a1}, {1, 1, b1}, {2, 2, a2}, {3, 3, b2}, {4, 4, b3}
{0, 1, a1,b1}, {2, 2, a2}, {3, 3, b2}, {4, 4, b3}
{0, 1, a1,b1}, {2, 2, a2}, {3, 4, b2,b3}

filtered-a:

{0, 0, a1}, {1, 1, }, {2, 2, a2}, {3, 3, a3}, {4, 4, }
{0, 1, a1}, {2, 2, a2}, {3, 3, a3}, {4, 4, }
{0, 1, a1}, {2, 2, a2}, {3, 4, a3}

As shown some of the possible filtered permutations DCP may send

1. We send empty snapshots, a marker with no commands
2. We send partial snapshots e.g. {3, 4, a3} tells the client about
an end of 4, but never sent command 4 (which unfiltered does). A
client may expect 4 to trigger something...

With this change, filtering is moved earlier in the stream flow, now
when we build the 'readyQueue' we apply filtering allowing the
existing empty checkpoint/snapshot logic to prevent an empty snapshot
being sent and ensures correct start/end seqnos.

The examples now...

filtered-a:

{0, 0, a1}, {2, 2, a2}, {3, 3, a3}
{0, 0, a1}, {2, 2, a2}, {3, 3, a3}
{0, 0, a1}, {2, 2, a2}, {3, 3, a3}

The update shows no empty snapshots and correct start/end.

Change-Id: I3379a9ea4ff11aed4f287f6cb688c8af0eecd8db
Reviewed-on: http://review.couchbase.org/84602
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
c

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
changes to SystemEvent shouldn't result in changes to the values
we transit.

Change-Id: I67c8e5876e10299eb69a52e89c7f18ff4981c09f
Reviewed-on: http://review.couchbase.org/78640
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Move ep-engine to engines/ep

MB-16181: Filters with deleted collections

Upstream integration revealed an issue with the VB::Filter.
If we construct a filter with a collection "X" and "X" is actually
marked as de

MB-16181: Filters with deleted collections

Upstream integration revealed an issue with the VB::Filter.
If we construct a filter with a collection "X" and "X" is actually
marked as deleted, "X" should not be part of the filter.

We fix this by replacing VB::Manifest::doesCollectionExist with
VB::Manifest::isCollectionOpen which performs the correct checks.

Tests updated to cover this case.

Change-Id: I38d4ba025805da7653bf3a84053d4280f4f547d7
Reviewed-on: http://review.couchbase.org/78570
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will esta

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will establish a Collections::Filter
that lives for the lifetime of the DCP producer.

As the DCP producer creates streams, a Collections::VB::Filter is
assigned to the stream which contains the real set of collections to
filter (and also the actual "filter" function).

Change-Id: I2f35b1698ce977116486a2e6940437eee25faef1
Reviewed-on: http://review.couchbase.org/78137
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...