Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3, v5.5.4, v5.5.5, v5.5.6, v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1
# 1f8437cf 06-Feb-2018 Jim Walker <jim@couchbase.com>

MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a int

MB-26972: JSON DCP Filter must also accept name and uid format

If a client wants to re-connect a DCP stream (non-zero) they must
be specific about the collection so they don't miss a intermediate
collection delete during their downtime.

This commit updates Collections::Filter so two formats of JSON are
valid and updates Collections::VB::Filter to use the strict name/uid
checks when the client uses the name:uid format.

A subsequent patch will ensure that the client does a name:uid request
when the start-seqno is non-zero.

Change-Id: I3cae53bf855d4b8001bcc65b6f036a9053b88307
Reviewed-on: http://review.couchbase.org/85981
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...


# cd24b1b3 09-Mar-2018 Jim Walker <jim@couchbase.com>

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID (and can persist/warm up from it).

Change-Id: I66e4ad4274558992b27c5ab02adb42295e761091
Reviewed-on: http://review.couchbase.org/91178
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# f1bc7c68 08-Mar-2018 Jim Walker <jim@couchbase.com>

MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose

MB-28773: Require UID in the Manifest

Update the Manifest JSON code so that we expect a UID in the manifest

The UID follows the same format as the collection UID and its purpose
is to allow easy identification of a buckets collection settings.

Change-Id: Ic660ae018e6782ceee07a58c14fa391741ddb4d1
Reviewed-on: http://review.couchbase.org/91176
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v5.0.1
# dea18910 25-Oct-2017 Jim Walker <jim@couchbase.com>

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# f53cb5f9 24-Jan-2018 Jim Walker <jim@couchbase.com>

MB-27457: [6/n] Allow DCP producer to send the delete time

The DCP producer will need to send any delete_time it gets from
disk to clients which have enabled the feature.

1. Add

MB-27457: [6/n] Allow DCP producer to send the delete time

The DCP producer will need to send any delete_time it gets from
disk to clients which have enabled the feature.

1. Add a new DCP open flag to the protocol
1.1 This flag is not yet known to the mcbp validator, so this WIP
feature is disabled to real clients.
2. Have the producer track the state of this flag (on/off)
3. When on, an alternative dcp_delete callback is used which can
accept the delete_time parameter.

This commit only updates KV-engine, subsequent commits will link up
to the real connection.

A test is added using the DcpProducer to show that we can retrieve
the delete-time of backfilled items, in-memory deletes and expired
items. When an expired item is produced (as a dcp-delete) the time
it has is the logical time it expired, i.e. its expiry-time, not
the actual time ep-engine deleted the item.

Change-Id: I76ddbbdbd33f14fcbabde4eb77d29713a1a9fba6
Reviewed-on: http://review.couchbase.org/88302
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 9f812d82 22-Jan-2018 Sriram Ganesan <sriram@couchbase.com>

MB-27669: [Restore] Retrieve documents as compressed in passive/active modes

This reverts commit fbe7aa5e77cb5eb31726ef763d249f00c9e49d0e.
This is restoring a change. This change was rev

MB-27669: [Restore] Retrieve documents as compressed in passive/active modes

This reverts commit fbe7aa5e77cb5eb31726ef763d249f00c9e49d0e.
This is restoring a change. This change was reverted due to a build break.

Change-Id: I571e4df0e50f41767ddf5df5a373910061d56533
Reviewed-on: http://review.couchbase.org/88210
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# fbe7aa5e 22-Jan-2018 Sriram Ganesan <sriram@couchbase.com>

Revert "MB-27669: Retrieve documents as compressed in passive/active modes"

This reverts commit a4b5da7e9a168a5a30d7302a98b67facdd3c492f.

Change-Id: I88cc56b78f2046b7c01d0d9f46901a4

Revert "MB-27669: Retrieve documents as compressed in passive/active modes"

This reverts commit a4b5da7e9a168a5a30d7302a98b67facdd3c492f.

Change-Id: I88cc56b78f2046b7c01d0d9f46901a4a07893df6
Reviewed-on: http://review.couchbase.org/88205
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>

show more ...


# a4b5da7e 19-Jan-2018 Sriram Ganesan <sriram@couchbase.com>

MB-27669: Retrieve documents as compressed in passive/active modes

During warmup, if a document is stored as compressed on disk, then
it should be retrieved as compressed to store in the

MB-27669: Retrieve documents as compressed in passive/active modes

During warmup, if a document is stored as compressed on disk, then
it should be retrieved as compressed to store in the hashtable
for passive and active modes

Change-Id: Id479dfd28d18f3568abc2c0f49d3138c5d973bb6
Reviewed-on: http://review.couchbase.org/88155
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...


# f8ac377c 11-Dec-2017 Jim Walker <jim@couchbase.com>

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream

MB-25240: Close DCP streams when all collections are removed

Ensure that a stream with an empty filter (because it has processed
deletes of all filtered collections) progresses to stream_end.

Change-Id: If09cc37e113f3019a3a1c743e60ea9bb2537691b
Reviewed-on: http://review.couchbase.org/84603
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 9628a0d6 07-Dec-2017 Jim Walker <jim@couchbase.com>

MB-27122: [2/2] Do not initialise the current manifest

As described in the MB, it is risky to initialise the current manifest
to a 'default' state. It is safer to run without a manifest

MB-27122: [2/2] Do not initialise the current manifest

As described in the MB, it is risky to initialise the current manifest
to a 'default' state. It is safer to run without a manifest as we
really should only operate on what the cluster_manager tells us.

This leads to one situation in DcpOpen where we must fail the open
if the manifest hasn't yet been set, this situation is now flagged
by a new error code "no_collections_manifest" allowing what should
be a temporary failure (cluster manager should always push a manifest)
to be observed.

Change-Id: I79804bbc97da6ba00182355815abe53ad1125da1
Reviewed-on: http://review.couchbase.org/86514
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v5.1.0
# 513d8e79 19-Oct-2017 Jim Walker <jim@couchbase.com>

MB-24572: Stop Collection aware DCP from sending empty snapshots

If a DCP stream is configured with filtering enabled and the filter
results in 0 mutations, we still send snapshot marker

MB-24572: Stop Collection aware DCP from sending empty snapshots

If a DCP stream is configured with filtering enabled and the filter
results in 0 mutations, we still send snapshot markers to the client.

"Hey here's 500 to 510" ... silence

In a needle/haystack situation, the client looking for needles will
get frequent snapshot markers triggered by the mutations made against
the haystack, it is preferential that the needle client gets nothing
until the needle collection is touched.

To recap on snapshots - a DCP snapshot is "a series of commands that
is guaranteed to contain a unique set of keys.", it isn't necessarily
representative of a disk checkpoint (commit).

Prior to this commit we may then do the following:

checkpoint: 0 1 2 3 4
[a1, b1, a2, b2, b3]

some example snapshot possibilities {start, end, commands}

unfiltered:

{0, 0, a1}, {1, 1, b1}, {2, 2, a2}, {3, 3, b2}, {4, 4, b3}
{0, 1, a1,b1}, {2, 2, a2}, {3, 3, b2}, {4, 4, b3}
{0, 1, a1,b1}, {2, 2, a2}, {3, 4, b2,b3}

filtered-a:

{0, 0, a1}, {1, 1, }, {2, 2, a2}, {3, 3, a3}, {4, 4, }
{0, 1, a1}, {2, 2, a2}, {3, 3, a3}, {4, 4, }
{0, 1, a1}, {2, 2, a2}, {3, 4, a3}

As shown some of the possible filtered permutations DCP may send

1. We send empty snapshots, a marker with no commands
2. We send partial snapshots e.g. {3, 4, a3} tells the client about
an end of 4, but never sent command 4 (which unfiltered does). A
client may expect 4 to trigger something...

With this change, filtering is moved earlier in the stream flow, now
when we build the 'readyQueue' we apply filtering allowing the
existing empty checkpoint/snapshot logic to prevent an empty snapshot
being sent and ensures correct start/end seqnos.

The examples now...

filtered-a:

{0, 0, a1}, {2, 2, a2}, {3, 3, a3}
{0, 0, a1}, {2, 2, a2}, {3, 3, a3}
{0, 0, a1}, {2, 2, a2}, {3, 3, a3}

The update shows no empty snapshots and correct start/end.

Change-Id: I3379a9ea4ff11aed4f287f6cb688c8af0eecd8db
Reviewed-on: http://review.couchbase.org/84602
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 4730ffcf 27-Oct-2017 Jim Walker <jim@couchbase.com>

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877

MB-16181: Collection separator tidy-up

The default separator should be : not ::, so correct those tests
which have been written using ::

Change-Id: If44dc77744cb08d405092f030877b8b2b07386a9
Reviewed-on: http://review.couchbase.org/85980
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# aacf24fd 23-Nov-2017 Trond Norbye <trond.norbye@gmail.com>

Refactor: Use gsl::not_null in engine API

This is an initial refactor. At some point we want to refactor
the cookie argument from a const void pointer to Cookie&

Change-Id: Ie7d

Refactor: Use gsl::not_null in engine API

This is an initial refactor. At some point we want to refactor
the cookie argument from a const void pointer to Cookie&

Change-Id: Ie7d5b15f5d19725c4da2e3cbf760ced319026c0b
Reviewed-on: http://review.couchbase.org/85960
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 1456e3a9 08-Nov-2017 Manu Dhundi <manu@couchbase.com>

MB-26470: Use shared_ptr for DcpProducer, DcpConsumer, ConnHandler

References to DcpProducer, DcpConsumer, ConnHandler class objects
are held via SingleThreadedRCPtr. SingleThreadedRCPtr

MB-26470: Use shared_ptr for DcpProducer, DcpConsumer, ConnHandler

References to DcpProducer, DcpConsumer, ConnHandler class objects
are held via SingleThreadedRCPtr. SingleThreadedRCPtr can cause
cyclic references, cannot detect std::bad_weak_ptr exceptions,
cannot dynamically cast to runtime type etc. Hence it is better
to use stl shared_ptr and weak_ptr instead.

This commit just converts all SingleThreadedRCPtr references to
std::shared_ptr reference. The next commit will introduce
std::weak_ptr.

Change-Id: I58668e563be9ea1ca16f619ff6fccfaa265f93c0
Reviewed-on: http://review.couchbase.org/85036
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...


# 7183204b 11-Oct-2017 Jim Walker <jim@couchbase.com>

MB-25342: Hide logically deleted keys when DCP backfilling

When DCP backfills, it must not forward onto the DCP stream any
logically deleted keys as a client must never see them.

MB-25342: Hide logically deleted keys when DCP backfilling

When DCP backfills, it must not forward onto the DCP stream any
logically deleted keys as a client must never see them.

Change-Id: If6501c3200fae0355798d687d9c61ff353676a89
Reviewed-on: http://review.couchbase.org/83632
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# fc1270f5 22-Sep-2017 Jim Walker <jim@couchbase.com>

MB-16181: Move DCP tests to their own file

Keep the DCP tests together so they're not mixed in with the other
tests.

Change-Id: I5a7ef932bb822d2b319099bb72d7759f1d0c277e
Rev

MB-16181: Move DCP tests to their own file

Keep the DCP tests together so they're not mixed in with the other
tests.

Change-Id: I5a7ef932bb822d2b319099bb72d7759f1d0c277e
Reviewed-on: http://review.couchbase.org/83652
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...