Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3, v5.5.4, v5.5.5, v5.5.6, v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1 |
|
#
cd24b1b3 |
| 09-Mar-2018 |
Jim Walker <jim@couchbase.com> |
MB-28773: Transfer the manifest UID to the replica VB When a system event propagates a collection change, embed the manifest UID in the event so that the replica is aware of the UID
MB-28773: Transfer the manifest UID to the replica VB When a system event propagates a collection change, embed the manifest UID in the event so that the replica is aware of the UID (and can persist/warm up from it). Change-Id: I66e4ad4274558992b27c5ab02adb42295e761091 Reviewed-on: http://review.couchbase.org/91178 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dave Rigby <daver@couchbase.com>
show more ...
|
#
f9444d17 |
| 09-Mar-2018 |
Jim Walker <jim@couchbase.com> |
MB-28773: Save and reload the manifest UID (via VB) When a collection change occurs (via set-manifest) save the manifest UID into each affected VB's meta-data. When warming up,
MB-28773: Save and reload the manifest UID (via VB) When a collection change occurs (via set-manifest) save the manifest UID into each affected VB's meta-data. When warming up, reload the manifest UID into the VB. Change-Id: I169c2e0c7ee739dea8b72a21f34f02676e08acba Reviewed-on: http://review.couchbase.org/91177 Reviewed-by: Dave Rigby <daver@couchbase.com> Tested-by: Build Bot <build@couchbase.com>
show more ...
|
Revision tags: v5.0.1, v5.1.0, v5.0.0 |
|
#
cd98cc03 |
| 12-Sep-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Persist the collection UID as hex There's an inconsistency between what the set_collections manifest JSON data encodes a UID (hex) and how the VB manifest stores the UID (d
MB-16181: Persist the collection UID as hex There's an inconsistency between what the set_collections manifest JSON data encodes a UID (hex) and how the VB manifest stores the UID (decimal). Make the VB manifest store as hex for easier debugging/analysis of collections. Change-Id: Ia2d476e25b21d7161c9f9d40c437ba429b255643 Reviewed-on: http://review.couchbase.org/83352 Reviewed-by: Dave Rigby <daver@couchbase.com> Tested-by: Build Bot <build@couchbase.com>
show more ...
|
#
a9dea599 |
| 21-Sep-2017 |
Jim Walker <jim@couchbase.com> |
MB-25342: Update how system events manage collections Prior to this commit collection lifespans were managed with two system events, CreateCollection and BeginDeleteCollection. These eve
MB-25342: Update how system events manage collections Prior to this commit collection lifespans were managed with two system events, CreateCollection and BeginDeleteCollection. These events mapped to hidden documents (System namespace) and had unique keys. Thus the life-span of collection beer, uid:1 may have a seqno 'stream' as follows (e.g. if backfill from seqno 0). seqno 3, key = $collections:create:beer:1 <- create marker seqno 4, key = beer::document <- document in the collection seqno 5, key = $collections:delete:beer:1 <- delete marker Now due to the way we support multiple generations of collections before the prior one is fully purged, this approach leads us to have a seqno stream as follows (e.g. backfill from seqno 0) seqno 3, key = $collections:create:beer:1 <- create marker seqno 4, key = beer::document <- document in the collection seqno 5, key = $collections:delete:beer:1 <- delete marker seqno 6, key = $collections:create:beer:2 <- create marker seqno 7, key = beer::document <- document in the collection seqno 8, key = $collections:delete:beer:2 <- delete marker With this approach (unique-keys) we have to maintain an increasing amount of meta-data to enable correct clean-up of the old generations. This growing meta-data would need to be in memory and the per VB JSON manifest. So to avoid this unbounded 'list' metadata this commit adjusts the way collections spans are managed in-terms of checkpoints and persisted data. 1. No longer do we have a Create and separate Delete event - a single SystemEvent::Collection will be used to cover create and delete. a) A single key represents all creates/deletes of a collection. b) A delete of a collection is just a delete of that system key. 2. Collection create/delete now closes the checkpoint to ensure we never de-dup create/delete/create into a single create, ensuring a client tracking a vbucket sees explicit creates and deletes and doesn't have to handle de-duplication E.g. "I've been told of beer:2 but never saw delete(beer:1)?" So the life span of a collection now generates sequence numbers as follows: seqno 3, key = $collections:beer <- create marker seqno 4, key = beer::document <- document in the collection seqno 5, key = $collections:beer del = true <- delete marker Now if we dump the by-seqno index we see: seqno 4, key = beer::document <- document in the collection seqno 5, key = $collections:beer del = true <- delete marker Now if we dump the by-seqno index after re-creating the collection, we see. seqno 4, key = beer::document <- document in the collection seqno 6, key = $collections:beer <- create marker And if we had performed cycles of create/delete we can see that there are no longer many keys associated with the collection resulting in no extra meta data to store so we can remove old markers. Note in these examples, DCP wouldn't just replicate the by-seqno index as seqno 4 can be identified as being logically deleted (it has a seqno < start of beer). Additionally for a backfill from 0 in the first example seqno 5 doesn't need to be sent either because the client cannot know about the create. These enhancements are not part of this commit. Change-Id: I32f04e512ce039dce7d780323d9bf3642cc507fa Reviewed-on: http://review.couchbase.org/83350 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dave Rigby <daver@couchbase.com>
show more ...
|
#
9160b844 |
| 04-Aug-2017 |
Jim Walker <jim@couchbase.com> |
MB-25342: Introduce Collection::uid to the VB::Manifest Many functions on VB::Manifest required the caller to specify the name and a uint32_t as separate parameters when trying to identi
MB-25342: Introduce Collection::uid to the VB::Manifest Many functions on VB::Manifest required the caller to specify the name and a uint32_t as separate parameters when trying to identify a collection. This is now cleaned up so that a new class is used when identifying a specific 'generation' of a collection. As part of this new class we are removing the use of 'revision' and replacing it with 'uid', however we use the Manifest::revision as the uid and later changes will change the Manifest format so that each collection is assigned a uid. As part of the clean-up of revision usage this commit fixes some short comings in the VB::Manifest management where a delete of a collection was updating it's revision, forcing later callers of completeDeletion to use the newer revision, instead of referring to the collection by the correct name:rev from when it was created. This commit additionally cleans up the separator changed path which was needlessly passing the revision through the various code paths and even over DCP. There is no need it was set and never read. Change-Id: I0d81ccb9a7c7b7505d0f021b284b368f7a60f88e Reviewed-on: http://review.couchbase.org/81984 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dave Rigby <daver@couchbase.com>
show more ...
|
#
ef22f9b0 |
| 25-May-2017 |
Dave Rigby <daver@couchbase.com> |
Move ep-engine to engines/ep
|
Revision tags: v4.6.2_ep, v4.6.2_mc |
|
#
3c80abae |
| 05-Mar-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Collections DcpConsumer Add the functional paths that allow a replica VBucket's manifest to be updated from incoming DCP messages. Note that the DCP protocol has not been w
MB-16181: Collections DcpConsumer Add the functional paths that allow a replica VBucket's manifest to be updated from incoming DCP messages. Note that the DCP protocol has not been written, but the API defined here shows what data needs to be transfered over-the-wire. Add a unit test to use MockDcpConsumer to drive a create and delete Change-Id: Ida8171b2cde104ccd452fafcd6193a72455b404b Reviewed-on: http://review.couchbase.org/74609 Reviewed-by: Dave Rigby <daver@couchbase.com> Tested-by: Build Bot <build@couchbase.com>
show more ...
|
Revision tags: v4.6.1_ep |
|
#
cc92fc76 |
| 02-Mar-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Interface additions and manifest logic for DCP Replica vbuckets will learn of collections from their active peer. Add new functions which allow for addition and deletion of col
MB-16181: Interface additions and manifest logic for DCP Replica vbuckets will learn of collections from their active peer. Add new functions which allow for addition and deletion of collections with DCP in mind, that is addition/deletion with pre-assigned sequence numbers. The manifest test code is now updated to keep a replica manifest in-sync using the data the active places in the checkpoint. A very loose DCP simulation. Change-Id: If7bd2dbbf16e66118bc0759f80b8b395a7370825 Reviewed-on: http://review.couchbase.org/73508 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dave Rigby <daver@couchbase.com>
show more ...
|
#
10dad95b |
| 01-Mar-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Enable separator modifications Up until now the separator was fixed as the default value of "::". This commit allows incoming manifests to change the separator if allowed,
MB-16181: Enable separator modifications Up until now the separator was fixed as the default value of "::". This commit allows incoming manifests to change the separator if allowed, as follows: 1. No open collections exist. 2. Only the default collection exists. A separator can be changed by two manifest "types" (when the above rules are met) 1. A manifest that has a new "separator" field 2. A manifest that has a new "separator" field and a new "collection" In the first case a new/dedicated SystemEvent is required that ensures a serialised copy of the manifest is flushed. In the second case the CreateCollection event will also flush the new separator. Change-Id: I6b563c32882b4c45ed5a3858da834ef733e08abc Reviewed-on: http://review.couchbase.org/74608 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dave Rigby <daver@couchbase.com>
show more ...
|
#
d7615449 |
| 06-Feb-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Flusher integration with set/get support This commit ties together collections code so that system events and manifest updates are now applied to storage. This commit
MB-16181: Flusher integration with set/get support This commit ties together collections code so that system events and manifest updates are now applied to storage. This commit updates the threading model to ensure concurrent sets and collection manifest changes are interlocked. All keys read and written belong to the default collection and because the $default collection is pre-allocated, get and set with this patch are successful, even with the collection policing in place. Test code can now make use of new VBucket methods so that the collection state can be changed and we can write tests that drive data into collections (and check the collections policing). A new class, SystemEventFlush, captures the extra state a single run of the flusher needs for the handling of SystemEvents (the writing special documents and _local docs) This commit enables set/get only to be collection aware - each datapath in/out method will need similar "isValid" checks. Change-Id: Ia6e6cb14d7e0fa2c418e0f52d9ab38ab36baf12d Reviewed-on: http://review.couchbase.org/73125 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dave Rigby <daver@couchbase.com>
show more ...
|
#
99b0b58b |
| 09-Feb-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Use cb::const_char_buffer over const std::string ref It will be easier/cheaper to integrate the DCP side of the collections work if we can update a collection from a DCP messag
MB-16181: Use cb::const_char_buffer over const std::string ref It will be easier/cheaper to integrate the DCP side of the collections work if we can update a collection from a DCP message. The "easy/cheaper" benefit being that we can just use a cb::const_char_buffer initialised to the packet's collection name and length data, rather than being forced to create and copy into a std::string. Change-Id: I58858e9a1f4fe7f0c6b382160764f68c63a6fb97 Reviewed-on: http://review.couchbase.org/73444 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Manu Dhundi <manu@couchbase.com> Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
show more ...
|
#
d87fd7e6 |
| 25-Jan-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Update completeDeletion to push system events Update the completeDeletion function and the JSON generation so that we can support the cases of completely deleting a collection
MB-16181: Update completeDeletion to push system events Update the completeDeletion function and the JSON generation so that we can support the cases of completely deleting a collection or just updating a "isDeleting" collection. Deleting a collection creates an event in the data-stream that will trigger the flusher to write the manifest (subsequent integration will update the flusher). Change-Id: I79e304849fe89987c3b3f1a0ee7c3eef816da684 Reviewed-on: http://review.couchbase.org/73115 Reviewed-by: Trond Norbye <trond.norbye@gmail.com> Tested-by: Build Bot <build@couchbase.com>
show more ...
|
#
96803ff8 |
| 24-Jan-2017 |
Jim Walker <jim@couchbase.com> |
MB-16181: Create and BeginDelete collection against a VB. Update the VB::Manifest to now drive an event through a VBucket. The VBucket can then generate a JSON manifest from the
MB-16181: Create and BeginDelete collection against a VB. Update the VB::Manifest to now drive an event through a VBucket. The VBucket can then generate a JSON manifest from the system-event item. This is achieved by serialising a VB::Manifest into the system-event value so that at flushing time the correct JSON represntation can be produced. Change-Id: I8b7d4a1fcf39d753f28eef6cc5367ed47de9d257 Reviewed-on: http://review.couchbase.org/73114 Reviewed-by: Dave Rigby <daver@couchbase.com> Tested-by: Build Bot <build@couchbase.com>
show more ...
|