History log of /6.0.3/kv_engine/engines/ep/src/systemevent.cc (Results 1 - 21 of 21)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v6.0.3, v5.5.4, v5.5.5, v5.5.3, v6.0.0, v5.5.2, v5.5.1, v5.1.2, v5.1.1
# db9b7df5 15-May-2018 Dave Rigby <daver@couchbase.com>

MB-29675: Optimize ActiveStream::makeResponseFromItem

makeResponseFromItem is called for every item to be sent out over DCP;
and it shows up high in 'linux perf' profiles (approx 18% of

MB-29675: Optimize ActiveStream::makeResponseFromItem

makeResponseFromItem is called for every item to be sent out over DCP;
and it shows up high in 'linux perf' profiles (approx 18% of AuxIO
threads). Optimize a number of areas of it:

- Remove unnecessary refcount updates by removing unneeded temporary
ref-counted ptrs

- Using std:move where applicable when passing queued_items to
sub-functions (so refcount is unchanged).

With these changes makeResponseFromItem drops to 6% of AuxIO threads.

Change-Id: I0368dd1dd93c91d3ec47d9d12c9aea7efc59a7f0
Reviewed-on: http://review.couchbase.org/94215
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...


# cd24b1b3 09-Mar-2018 Jim Walker <jim@couchbase.com>

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID

MB-28773: Transfer the manifest UID to the replica VB

When a system event propagates a collection change, embed
the manifest UID in the event so that the replica is aware
of the UID (and can persist/warm up from it).

Change-Id: I66e4ad4274558992b27c5ab02adb42295e761091
Reviewed-on: http://review.couchbase.org/91178
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v5.0.1
# dea18910 25-Oct-2017 Jim Walker <jim@couchbase.com>

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v5.1.0, v5.0.0
# 7fdc2e2d 13-Sep-2017 Jim Walker <jim@couchbase.com>

MB-16181: Send collection UID in network byte order.

The DCP system event 'value' is data created by ep-engine, in the
case of collections we encode the collection UID in the value. Prio

MB-16181: Send collection UID in network byte order.

The DCP system event 'value' is data created by ep-engine, in the
case of collections we encode the collection UID in the value. Prior
to this patch, that data was not encoded in network byte order.

Change-Id: Ib00e55ede356cf92aa358a5f1904cb84cc0a19c6
Reviewed-on: http://review.couchbase.org/83351
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# a9dea599 21-Sep-2017 Jim Walker <jim@couchbase.com>

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These eve

MB-25342: Update how system events manage collections

Prior to this commit collection lifespans were managed with two system
events, CreateCollection and BeginDeleteCollection. These events
mapped to hidden documents (System namespace) and had unique keys.
Thus the life-span of collection beer, uid:1 may have a seqno 'stream'
as follows (e.g. if backfill from seqno 0).

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker

Now due to the way we support multiple generations of collections
before the prior one is fully purged, this approach leads us to have a
seqno stream as follows (e.g. backfill from seqno 0)

seqno 3, key = $collections:create:beer:1 <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:delete:beer:1 <- delete marker
seqno 6, key = $collections:create:beer:2 <- create marker
seqno 7, key = beer::document <- document in the collection
seqno 8, key = $collections:delete:beer:2 <- delete marker

With this approach (unique-keys) we have to maintain an increasing
amount of meta-data to enable correct clean-up of the old generations.
This growing meta-data would need to be in memory and the per VB
JSON manifest.

So to avoid this unbounded 'list' metadata this commit adjusts the way
collections spans are managed in-terms of checkpoints and persisted
data.

1. No longer do we have a Create and separate Delete event - a single
SystemEvent::Collection will be used to cover create and delete.

a) A single key represents all creates/deletes of a collection.
b) A delete of a collection is just a delete of that system key.

2. Collection create/delete now closes the checkpoint to ensure we
never de-dup create/delete/create into a single create, ensuring
a client tracking a vbucket sees explicit creates and deletes and
doesn't have to handle de-duplication E.g. "I've been told of
beer:2 but never saw delete(beer:1)?"

So the life span of a collection now generates sequence numbers as
follows:

seqno 3, key = $collections:beer <- create marker
seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index we see:

seqno 4, key = beer::document <- document in the collection
seqno 5, key = $collections:beer del = true <- delete marker

Now if we dump the by-seqno index after re-creating the collection,
we see.

seqno 4, key = beer::document <- document in the collection
seqno 6, key = $collections:beer <- create marker

And if we had performed cycles of create/delete we can see that there
are no longer many keys associated with the collection resulting in no
extra meta data to store so we can remove old markers.

Note in these examples, DCP wouldn't just replicate the by-seqno index
as seqno 4 can be identified as being logically deleted (it has a
seqno < start of beer). Additionally for a backfill from 0 in the
first example seqno 5 doesn't need to be sent either because the
client cannot know about the create. These enhancements are not part
of this commit.

Change-Id: I32f04e512ce039dce7d780323d9bf3642cc507fa
Reviewed-on: http://review.couchbase.org/83350
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 862dece9 07-Sep-2017 Jim Walker <jim@couchbase.com>

Revert "MB-25381: Make begin collection delete flush a system event"

This reverts commit 84a02ac934ffb33bdded8dd6b982cb0ad83afab9.

84a02ac was added as all events now flushed a 'mar

Revert "MB-25381: Make begin collection delete flush a system event"

This reverts commit 84a02ac934ffb33bdded8dd6b982cb0ad83afab9.

84a02ac was added as all events now flushed a 'marker' to disk.
However this is about to change in that the DeleteHard/Soft events
need the flusher skip functionality. So we are reverting 84a02ac so
that upstream changes can re-enable skipping for certain events.

Change-Id: I9fdfceb64b9d18c6fccb7889e9bd7f4d6aa559e8
Reviewed-on: http://review.couchbase.org/83230
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 9160b844 04-Aug-2017 Jim Walker <jim@couchbase.com>

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identi

MB-25342: Introduce Collection::uid to the VB::Manifest

Many functions on VB::Manifest required the caller to specify the name
and a uint32_t as separate parameters when trying to identify a
collection. This is now cleaned up so that a new class is used when
identifying a specific 'generation' of a collection.

As part of this new class we are removing the use of 'revision' and
replacing it with 'uid', however we use the Manifest::revision as
the uid and later changes will change the Manifest format so that each
collection is assigned a uid.

As part of the clean-up of revision usage this commit fixes some short
comings in the VB::Manifest management where a delete of a collection
was updating it's revision, forcing later callers of completeDeletion
to use the newer revision, instead of referring to the collection by
the correct name:rev from when it was created.

This commit additionally cleans up the separator changed path which
was needlessly passing the revision through the various code paths and
even over DCP. There is no need it was set and never read.

Change-Id: I0d81ccb9a7c7b7505d0f021b284b368f7a60f88e
Reviewed-on: http://review.couchbase.org/81984
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 84a02ac9 25-Jul-2017 Jim Walker <jim@couchbase.com>

MB-25381: Make begin collection delete flush a system event

BeginDeleteCollection only updated the vb state document and skipped
flushing a system event.

However:

1) Th

MB-25381: Make begin collection delete flush a system event

BeginDeleteCollection only updated the vb state document and skipped
flushing a system event.

However:

1) This leads to a seqno discrepancy (vb highSeqno above what the
persisted highseqno)
2) DCP backfill can't determine when a collection was deleted, which
is inconsistent with an in-memory stream that propagates the begin
delete as it comes down the checkpoint queue.

Change-Id: Icb2d3928c46842e9f103343d831edbeb06b4ae2e
Reviewed-on: http://review.couchbase.org/81420
Reviewed-by: Oliver Downard <oliver.downard@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# b7acc401 18-May-2017 Dave Rigby <daver@couchbase.com>

Move VBucketBGFetchItem to own header file

Move VBucketBGFetchItem and assocated types from kvstore.h to a new
header file vbucket_bgfetch_item.h. This removes the need for files
whi

Move VBucketBGFetchItem to own header file

Move VBucketBGFetchItem and assocated types from kvstore.h to a new
header file vbucket_bgfetch_item.h. This removes the need for files
which don't need to use bgfetch types from including kvstore.h

Change-Id: I8af795f12def4ac07c11ca92c36a2bf23131414f
Reviewed-on: http://review.couchbase.org/78654
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 6ce93328 18-May-2017 Jim Walker <jim@couchbase.com>

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
c

MB-16181: Map SystemEvent to mcbp::systemevent

This commit maps the ep-engine SystemEvent enum value to mcbp ones.
Not all SystemEvent entries are things we replicate using dcp and
changes to SystemEvent shouldn't result in changes to the values
we transit.

Change-Id: I67c8e5876e10299eb69a52e89c7f18ff4981c09f
Reviewed-on: http://review.couchbase.org/78640
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# ef22f9b0 25-May-2017 Dave Rigby <daver@couchbase.com>

Move ep-engine to engines/ep


Revision tags: v4.6.2_ep
# e57f09f4 23-Mar-2017 Jim Walker <jim@couchbase.com>

MB-16181: Build SystemEvent keys with the collections separator

The keys were fixed as $collections::<event> but are now changed
so that the :: is the collections separator.

Thi

MB-16181: Build SystemEvent keys with the collections separator

The keys were fixed as $collections::<event> but are now changed
so that the :: is the collections separator.

This allows code to split the event key if they wish using the
same code they would split document keys.

Change-Id: I48575d295f8c058a79cf208fe3c9d3a9b3c9ed15
Reviewed-on: http://review.couchbase.org/78136
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 4885256f 21-Apr-2017 Jim Walker <jim@couchbase.com>

MB-16181: Make ActiveStream track the current collections separator

This patch updates the ActiveStream so that it stores a copy of the current
separator and tracks changes to as they ar

MB-16181: Make ActiveStream track the current collections separator

This patch updates the ActiveStream so that it stores a copy of the current
separator and tracks changes to as they are transmitted through the checkpoint
via the separator changed SystemEvent.

Change-Id: Ie3ea87d006b0bbab3e0edd8895a4756c7c5d9fe8
Reviewed-on: http://review.couchbase.org/75502
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v4.6.2_mc, v4.6.1_ep
# 3fd9c0fc 14-Feb-2017 Jim Walker <jim@couchbase.com>

MB-16181: Collections DcpProducer

1. Enable SystemEvent Items in "Item::shouldReplicate"
2. Update DcpProducer so that it grabs the various bits of collection
data for the transfe

MB-16181: Collections DcpProducer

1. Enable SystemEvent Items in "Item::shouldReplicate"
2. Update DcpProducer so that it grabs the various bits of collection
data for the transferred events and calls through the engine's dcp
system_event callback

An update synchrounous engine test now links a MockDcpConsumer to
a MockDcpProducer and drives events from one to the other.

Change-Id: I83052aa956afbe7495c15420889c84d920a5b7fc
Reviewed-on: http://review.couchbase.org/75603
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# cc92fc76 02-Mar-2017 Jim Walker <jim@couchbase.com>

MB-16181: Interface additions and manifest logic for DCP

Replica vbuckets will learn of collections from their active peer.
Add new functions which allow for addition and deletion of col

MB-16181: Interface additions and manifest logic for DCP

Replica vbuckets will learn of collections from their active peer.
Add new functions which allow for addition and deletion of collections
with DCP in mind, that is addition/deletion with pre-assigned sequence
numbers.

The manifest test code is now updated to keep a replica manifest in-sync
using the data the active places in the checkpoint. A very loose DCP
simulation.

Change-Id: If7bd2dbbf16e66118bc0759f80b8b395a7370825
Reviewed-on: http://review.couchbase.org/73508
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 3a33f09a 16-Mar-2017 Jim Walker <jim@couchbase.com>

MB-16181: Restore documents to the System namespace

Have the Item constructor set the operation to system_event for
documents with a System namespace. This allows warmup and backfill

MB-16181: Restore documents to the System namespace

Have the Item constructor set the operation to system_event for
documents with a System namespace. This allows warmup and backfill
to correctly identify the system events.

Change-Id: I31d8994820c0d88857a5869bb3eada8351cddc29
Reviewed-on: http://review.couchbase.org/75274
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 10dad95b 01-Mar-2017 Jim Walker <jim@couchbase.com>

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed,

MB-16181: Enable separator modifications

Up until now the separator was fixed as the default value of "::".
This commit allows incoming manifests to change the separator if
allowed, as follows:

1. No open collections exist.
2. Only the default collection exists.

A separator can be changed by two manifest "types" (when the above
rules are met)

1. A manifest that has a new "separator" field
2. A manifest that has a new "separator" field and a new "collection"

In the first case a new/dedicated SystemEvent is required that ensures
a serialised copy of the manifest is flushed.

In the second case the CreateCollection event will also flush the
new separator.

Change-Id: I6b563c32882b4c45ed5a3858da834ef733e08abc
Reviewed-on: http://review.couchbase.org/74608
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# d7615449 06-Feb-2017 Jim Walker <jim@couchbase.com>

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit

MB-16181: Flusher integration with set/get support

This commit ties together collections code so that system events
and manifest updates are now applied to storage.

This commit updates the threading model to ensure concurrent sets and
collection manifest changes are interlocked.

All keys read and written belong to the default collection and because
the $default collection is pre-allocated, get and set with this patch
are successful, even with the collection policing in place.

Test code can now make use of new VBucket methods so that the
collection state can be changed and we can write tests that drive
data into collections (and check the collections policing).

A new class, SystemEventFlush, captures the extra state a single
run of the flusher needs for the handling of SystemEvents (the writing
special documents and _local docs)

This commit enables set/get only to be collection aware - each
datapath in/out method will need similar "isValid" checks.

Change-Id: Ia6e6cb14d7e0fa2c418e0f52d9ab38ab36baf12d
Reviewed-on: http://review.couchbase.org/73125
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# ddd0da82 03-Mar-2017 Jim Walker <jim@couchbase.com>

MB-16181: Create/Begin using incorrect keys

The keys are wrong in our SystemEvents

The create event is correct:

A create event for collection "beer" stores

1. A sp

MB-16181: Create/Begin using incorrect keys

The keys are wrong in our SystemEvents

The create event is correct:

A create event for collection "beer" stores

1. A special document in the system-namespace
"$collections::create:beer<rev>"
2. Writes a new JSON manifest in _local.
3. Triggers DCP (it will do upstream)

The begin delete event is incorrect, it currently is set to delete
the special document written by the create, it is too soon to do that
as Items in the collection still exist.

The begin delete event needs to.
1. Trigger a DCP notification (tell a consumer X is deleting)
2. An update to the JSON manifest in _local.

The delete events (those that are created when we know we've purged
all items of a collection) are incorrect.

The delete events (hard/soft) need to:
1. Update the JSON manifest in the _local.
2. Delete the special document written by the create event.

Note the updating of documents and triggering of DCP is handled by
commits that add collection support in those areas, so this commit
only affects the SystemEvent keys.

Change-Id: Ibccd14e3da096b495c638303c4189bca0928d4c4
Reviewed-on: http://review.couchbase.org/74607
Reviewed-by: Daniel Owen <owend@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# d87fd7e6 25-Jan-2017 Jim Walker <jim@couchbase.com>

MB-16181: Update completeDeletion to push system events

Update the completeDeletion function and the JSON generation so
that we can support the cases of completely deleting a collection

MB-16181: Update completeDeletion to push system events

Update the completeDeletion function and the JSON generation so
that we can support the cases of completely deleting a collection
or just updating a "isDeleting" collection.

Deleting a collection creates an event in the data-stream that
will trigger the flusher to write the manifest (subsequent integration
will update the flusher).

Change-Id: I79e304849fe89987c3b3f1a0ee7c3eef816da684
Reviewed-on: http://review.couchbase.org/73115
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 2f12cd1a 18-Jan-2017 Jim Walker <jim@couchbase.com>

MB-16181: Add SystemEventFactory

The SystemEventFactory creates Item objects representing
the following system events.

- create collection
- begin delete collection

MB-16181: Add SystemEventFactory

The SystemEventFactory creates Item objects representing
the following system events.

- create collection
- begin delete collection

An Item which represents a SystemEvent will eventually flow through
checkpoints to the flusher and DCP.

A simple test checks the Item is as expected.

- The operation is system_event.
- The flags return the detailed event type.

Change-Id: If8bdd92f83428538125ff18344cce71dbacfc2d9
Reviewed-on: http://review.couchbase.org/73113
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...