History log of /5.5.2/kv_engine/engines/ep/tests/module_tests/collections/filter_test.cc (Results 1 - 13 of 13)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3, v5.5.4, v5.5.5, v5.5.6, v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1
# 1f8437cf 06-Feb-2018 Jim Walker <jim@couchbase.com>

MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a int

MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a intermediate
collection delete during their downtime.

This commit updates Collections::Filter so two formats of JSON are
valid and updates Collections::VB::Filter to use the strict name/uid
checks when the client uses the name:uid format.

A subsequent patch will ensure that the client does a name:uid request
when the start-seqno is non-zero.

Change-Id: I3cae53bf855d4b8001bcc65b6f036a9053b88307
Reviewed-on: http://review.couchbase.org/85981
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...


# f1bc7c68 08-Mar-2018 Jim Walker <jim@couchbase.com>

MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose

MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose
is to allow easy identification of a buckets collection settings.

Change-Id: Ic660ae018e6782ceee07a58c14fa391741ddb4d1
Reviewed-on: http://review.couchbase.org/91176
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v5.0.1
# dea18910 25-Oct-2017 Jim Walker <jim@couchbase.com>

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# f8ac377c 11-Dec-2017 Jim Walker <jim@couchbase.com>

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream_end.

Change-Id: If09cc37e113f3019a3a1c743e60ea9bb2537691b
Reviewed-on: http://review.couchbase.org/84603
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 9628a0d6 07-Dec-2017 Jim Walker <jim@couchbase.com>

MB-27122: [2/2] Do not initialise the current manifest

As described in the MB, it is risky to initialise the current manifest
to a 'default' state. It is safer to run without a manifest

MB-27122: [2/2] Do not initialise the current manifest

As described in the MB, it is risky to initialise the current manifest
to a 'default' state. It is safer to run without a manifest as we
really should only operate on what the cluster_manager tells us.

This leads to one situation in DcpOpen where we must fail the open
if the manifest hasn't yet been set, this situation is now flagged
by a new error code "no_collections_manifest" allowing what should
be a temporary failure (cluster manager should always push a manifest)
to be observed.

Change-Id: I79804bbc97da6ba00182355815abe53ad1125da1
Reviewed-on: http://review.couchbase.org/86514
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# d0d71f8a 07-Dec-2017 Jim Walker <jim@couchbase.com>

MB-27122: [1/2] Change DcpProducer filter from pointer to object

1. A DcpProducer can never not have a filter so make it an owned
object.

2. In-order to get better error return

MB-27122: [1/2] Change DcpProducer filter from pointer to object

1. A DcpProducer can never not have a filter so make it an owned
object.

2. In-order to get better error returns for incorrect filters on
DcpOpen hoist the construction of the filter to be part of the
producer creation. We now create the filter and if success create
the producer passing the filter as an argument, this is done using
std::move so that we don't create temporaries.

3. The error checking is now geared around catching cb::engine_error
allowing the creation to return errors to the client. Note at this
patch level the Filter's throw points are unchanged, it still throws
std::invalid_argument and triggers a disconnect.

Change-Id: Ife88598830dcaf27573783228c989dcc6a31a9bc
Reviewed-on: http://review.couchbase.org/86513
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v5.1.0
# 513d8e79 19-Oct-2017 Jim Walker <jim@couchbase.com>

MB-24572: Stop Collection aware DCP from sending empty snapshots

If a DCP stream is configured with filtering enabled and the filter
results in 0 mutations, we still send snapshot marker

MB-24572: Stop Collection aware DCP from sending empty snapshots

If a DCP stream is configured with filtering enabled and the filter
results in 0 mutations, we still send snapshot markers to the client.

"Hey here's 500 to 510" ... silence

In a needle/haystack situation, the client looking for needles will
get frequent snapshot markers triggered by the mutations made against
the haystack, it is preferential that the needle client gets nothing
until the needle collection is touched.

To recap on snapshots - a DCP snapshot is "a series of commands that
is guaranteed to contain a unique set of keys.", it isn't necessarily
representative of a disk checkpoint (commit).

Prior to this commit we may then do the following:

checkpoint: 0 1 2 3 4
[a1, b1, a2, b2, b3]

some example snapshot possibilities {start, end, commands}

unfiltered:

{0, 0, a1}, {1, 1, b1}, {2, 2, a2}, {3, 3, b2}, {4, 4, b3}
{0, 1, a1,b1}, {2, 2, a2}, {3, 3, b2}, {4, 4, b3}
{0, 1, a1,b1}, {2, 2, a2}, {3, 4, b2,b3}

filtered-a:

{0, 0, a1}, {1, 1, }, {2, 2, a2}, {3, 3, a3}, {4, 4, }
{0, 1, a1}, {2, 2, a2}, {3, 3, a3}, {4, 4, }
{0, 1, a1}, {2, 2, a2}, {3, 4, a3}

As shown some of the possible filtered permutations DCP may send

1. We send empty snapshots, a marker with no commands
2. We send partial snapshots e.g. {3, 4, a3} tells the client about
an end of 4, but never sent command 4 (which unfiltered does). A
client may expect 4 to trigger something...

With this change, filtering is moved earlier in the stream flow, now
when we build the 'readyQueue' we apply filtering allowing the
existing empty checkpoint/snapshot logic to prevent an empty snapshot
being sent and ensures correct start/end seqnos.

The examples now...

filtered-a:

{0, 0, a1}, {2, 2, a2}, {3, 3, a3}
{0, 0, a1}, {2, 2, a2}, {3, 3, a3}
{0, 0, a1}, {2, 2, a2}, {3, 3, a3}

The update shows no empty snapshots and correct start/end.

Change-Id: I3379a9ea4ff11aed4f287f6cb688c8af0eecd8db
Reviewed-on: http://review.couchbase.org/84602
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 4730ffcf 27-Oct-2017 Jim Walker <jim@couchbase.com>

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877b8b2b07386a9
Reviewed-on: http://review.couchbase.org/85980
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v5.0.0
# 30aab6b7 22-Aug-2017 Jim Walker <jim@couchbase.com>

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of

MB-25342: Collection UID in the Manifest

Primarily update the JSON specification so that the 'set_collections'
command requires that collections are given a UID.

The purpose of the UID is to allow KV-engine to see that a collection
was deleted and recreated without ever seeing the intermediate delete.
The collections management code will identify that it knows about a
collection, but because of the UID it can determine if the collection
it knows about is now defunct.

A secondary purpose is that DCP may allow clients to use UIDs when
filtering collections.

The UID must be a JSON string that stores a 64-bit unsigned integer
encoded in hex.

Note it is the cluster managements responsibility to ensure that a UID
is unique enough that it doesn't collide with a previous generation
before that generation has been fully deleted (note once a generation
has been fully deleted, a UID could be re-used).

Change-Id: I6f9bbde00783a7a93511bb5ee426b8c43d8c019f
Reviewed-on: http://review.couchbase.org/83227
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 6ce93328 18-May-2017 Jim Walker <jim@couchbase.com>

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
c

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
changes to SystemEvent shouldn't result in changes to the values
we transit.

Change-Id: I67c8e5876e10299eb69a52e89c7f18ff4981c09f
Reviewed-on: http://review.couchbase.org/78640
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# ef22f9b0 25-May-2017 Dave Rigby <daver@couchbase.com>

Move ep-engine to engines/ep


# 5cf47c05 24-May-2017 Jim Walker <jim@couchbase.com>

MB-16181: Filters with deleted collections

Upstream integration revealed an issue with the VB::Filter.
If we construct a filter with a collection "X" and "X" is actually
marked as de

MB-16181: Filters with deleted collections

Upstream integration revealed an issue with the VB::Filter.
If we construct a filter with a collection "X" and "X" is actually
marked as deleted, "X" should not be part of the filter.

We fix this by replacing VB::Manifest::doesCollectionExist with
VB::Manifest::isCollectionOpen which performs the correct checks.

Tests updated to cover this case.

Change-Id: I38d4ba025805da7653bf3a84053d4280f4f547d7
Reviewed-on: http://review.couchbase.org/78570
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v4.6.2_ep
# 042bf504 14-Mar-2017 Jim Walker <jim@couchbase.com>

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will esta

MB-16181: Add Collections Filter classes and test

Two classes exist for filtering.

Collections::Filter
Collections::VB::Filter

The idea is that a DCP producer will establish a Collections::Filter
that lives for the lifetime of the DCP producer.

As the DCP producer creates streams, a Collections::VB::Filter is
assigned to the stream which contains the real set of collections to
filter (and also the actual "filter" function).

Change-Id: I2f35b1698ce977116486a2e6940437eee25faef1
Reviewed-on: http://review.couchbase.org/78137
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...