History log of /6.6.0/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore-metadata.h (Results 1 - 19 of 19)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3
# a2111545 15-Aug-2019 Ben Huddleston <ben.huddleston@couchbase.com>

MB-35496: Remove completed prepares at compaction

When we compact a database file we can remove all completed
prepares to save space. A prepare is only needed for the lifetime
of a S

MB-35496: Remove completed prepares at compaction

When we compact a database file we can remove all completed
prepares to save space. A prepare is only needed for the lifetime
of a SyncWrite to ensure that we still have it should we shutdown
and restart. We don't need to worry about the metadata purge
interval when it comes to prepares because we stream Mutations
instead of Commits when streaming from disk.

Change-Id: I50925f11e72f3db38d92f5e0129d9a8dfe87b014
Reviewed-on: http://review.couchbase.org/113383
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 942d6f62 27-Jun-2019 James Harrison <james.harrison@couchbase.com>

MB-34768: Only check committed items for expiry

Problem:
The compaction expiry hook tries to expire sync delete prepares.

Cause:
The expiry field is used to indicate deletio

MB-34768: Only check committed items for expiry

Problem:
The compaction expiry hook tries to expire sync delete prepares.

Cause:
The expiry field is used to indicate deletion time for deletes, and is
set to the current time when processing the op. This is also applied to
sync delete prepares, but as they are not actually deletes,
couch-kvstore misinterprets them as an expired item.

Solution:
Ensure items are not candidates for expiry unless they are committed.
Prepares are never eligible for expiry, so this is valid.

Change-Id: Ic25b7fa44fcd6c50666e1db80797ac80004899c2
Reviewed-on: http://review.couchbase.org/111347
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 002492c6 20-Jun-2019 Ben Huddleston <ben.huddleston@couchbase.com>

MB-34676: Persist the prepareSeqno of commits and aborts

This requires add a prepareSeqno to:
- Couchstore MetaDataV3 (used for prepares, commits and aborts)
- RocksDB MetaData (us

MB-34676: Persist the prepareSeqno of commits and aborts

This requires add a prepareSeqno to:
- Couchstore MetaDataV3 (used for prepares, commits and aborts)
- RocksDB MetaData (used for all values)
- Magma MetaData (used for all values)
- OrderedStoredValue (used for all values)

We need the prepareSeqno to be correct when backfilling from disk
to keep our sanity checks on replica and avoid sending mutations
instead of commits wherever possible. Without this, we would need to
send a mutation instead of a commit for every backfilled commit. The
prepareSeqno will also be used for aborts to simplify some error
checking on the replica when receiving a backfilled abort.

Change-Id: If203af661b976ca3145eec1873ae9f30bce80c6c
Reviewed-on: http://review.couchbase.org/110998
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# d0933f63 07-May-2019 Dave Rigby <daver@couchbase.com>

Fix build break (missing durability_spec.h #include)

The cleanup of item.h and durability_spec.h headers resulted in a
couple of instances of missing headers (the two patches were develo

Fix build break (missing durability_spec.h #include)

The cleanup of item.h and durability_spec.h headers resulted in a
couple of instances of missing headers (the two patches were developed
independantly and not based on top of each other).

Fix the build break by adding back the missing #includes

Change-Id: Ied6b51567ba3439eaadec09fe0a349675f9475ff
Reviewed-on: http://review.couchbase.org/108772
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v5.5.4
# 64341951 16-Apr-2019 Dave Rigby <daver@couchbase.com>

Minimise #include of item.h

Remove unnecessary #includes of item.h - replace with forward
declarations or simply remove where not needed.

Change-Id: Ic88f001e7f459cb2f541462c034

Minimise #include of item.h

Remove unnecessary #includes of item.h - replace with forward
declarations or simply remove where not needed.

Change-Id: Ic88f001e7f459cb2f541462c03434f9bc6e34dce
Reviewed-on: http://review.couchbase.org/107952
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...


# cfff162e 25-Apr-2019 Dave Rigby <daver@couchbase.com>

MB-32734 [SR]: Add syncDelete flag to KVStore metadata

Prepared SyncDeletes require slightly different handling to "normal"
deletes - they represent a _potential_ delete assuming the key

MB-32734 [SR]: Add syncDelete flag to KVStore metadata

Prepared SyncDeletes require slightly different handling to "normal"
deletes - they represent a _potential_ delete assuming the key is
Committed - but are not technically deleted themselves.

This distinction is most seen on disk; where we need to persist
Prepared SyncDeletes such that they can be subsequently loaded from
disk (e.g. during warmup), but they should _not_ be subject to things
like tombstone puring which "real" deletes are.

As such, Prepared SyncDeletes should be persisted to disk as valid,
not-yet-deleted documents (in the separate Prepare namespace). To
achieve this, but still be able to distinguish them from SyncWrites,
we need to add a new flag in each KVStore's metadata. When reading
Prepared SyncDeletes from disk, if this flag is set then the returned
Item should be marked as a (Prepared) delete.

Change-Id: I15fcddc2fcbbc61a7b934d667c5d94cd4c29b49a
Reviewed-on: http://review.couchbase.org/108275
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>

show more ...


Revision tags: v5.5.5
# def65694 04-Apr-2019 Dave Rigby <daver@couchbase.com>

[SR] Persist Committed SyncWrites with correct metadata

Ensure that when persisting CommittedViaPrepare mutations, that the
metadata is correctly set so we can read it back as the same i

[SR] Persist Committed SyncWrites with correct metadata

Ensure that when persisting CommittedViaPrepare mutations, that the
metadata is correctly set so we can read it back as the same item
type.

Change-Id: Ia48912e5a8aaee00b5a108eb209b8a410a9bd1f1
Reviewed-on: http://review.couchbase.org/107312
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>

show more ...


# 8b88e592 27-Mar-2019 Dave Rigby <daver@couchbase.com>

Fix misleading placement of '#pragma pack(1)'

As identified via http://review.couchbase.org/#/c/106745/, '#pragma
pack(1)' must be placed before the definition of structs & classes -

Fix misleading placement of '#pragma pack(1)'

As identified via http://review.couchbase.org/#/c/106745/, '#pragma
pack(1)' must be placed before the definition of structs & classes -
placing it before specific members of a class does not have any
effect.

A number of uses inside kv-engine - specifically in the KVStore
backends was incorrect - the pragma occurred in the middle of the
class. However, in all cases the misplaced pragma was either benign
because either:

- The natural alignment of the object was already packed
(couch-kvstore-metadata.h) - note there are static_asserts()
checking the size; or

- The struct(s) didn't _need_ to be packed (rocksdb-kvstore.cc /
magma-kvstore.cc) - because we don't (yet) need backwards compatible
on-disk layout.

Nethertheless, fix all incorrect usages of #pragma pack to not mislead
anyone. (magma-kvstore.cc is undergoing parallel changes so not
modifying those in this patch). Additionally reduce the size of the
type used for sequence numbers in rockskv::MetaData as we only need
48bits.

Change-Id: I6f9d53a5668066210ad0f80250d14c2bfaea2820
Reviewed-on: http://review.couchbase.org/106846
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v5.5.6
# a81c119d 14-Mar-2019 Paolo Cocchi <paolo.cocchi@couchbase.com>

MB-33167 [SR]: Persist durability-abort

Here the main characteristics of a durable-abort item on disk:
- it resides into the Durability Prepare key-space
- it has Metadata::Operation

MB-33167 [SR]: Persist durability-abort

Here the main characteristics of a durable-abort item on disk:
- it resides into the Durability Prepare key-space
- it has Metadata::Operation::Abort
- it has the Deleted flag set

Note that persisting an Abort implicitly replaces the previous Prepare.

This patch adds support for both Couchstore and RocksDB.

Change-Id: Ic7d148e2bf44bf3cd339c8afc7189ca0e5729b12
Reviewed-on: http://review.couchbase.org/106446
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# b51e6547 11-Mar-2019 Ben Huddleston <ben.huddleston@couchbase.com>

MB-30040: Remove our folly/lang/Assume.h

Change-Id: Id6ece6feec7e1fa12d84c0ce1cc70ee39cf7db87
Reviewed-on: http://review.couchbase.org/106006
Tested-by: Build Bot <build@couchbase.co

MB-30040: Remove our folly/lang/Assume.h

Change-Id: Id6ece6feec7e1fa12d84c0ce1cc70ee39cf7db87
Reviewed-on: http://review.couchbase.org/106006
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 84251e56 12-Mar-2019 Dave Rigby <daver@couchbase.com>

MB-33186: Warmup: Add new 'loadPreparedSyncWrites' state

Add a new state to the Warmup process - State::LoadPreparedSyncWrites.

This state occurs early in the warmup process, before

MB-33186: Warmup: Add new 'loadPreparedSyncWrites' state

Add a new state to the Warmup process - State::LoadPreparedSyncWrites.

This state occurs early in the warmup process, before the normal bulk
key/value loading phases. It loads from disk all prepared SyncWrites
which were previously persisted, but not /yet/ committed to disk (at
which point the prepared SyncWrite would have been deleted and
replaced with a Committed one).

This patch adds all found prepared SyncWrites are into the
HashTable. Subsequent patches will also add them into the
DurablityMonitor.

Change-Id: Ieb723262421d523308d4788495f1265e1edda89f
Reviewed-on: http://review.couchbase.org/106358
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...


# 4c387968 27-Feb-2019 Dave Rigby <daver@couchbase.com>

MB-33186 [SR]: Add couchstore MetaDataV3

Add a MetaDataV3 which contains metadata fields specific to
synchronous replication: the operation (pending, commit, abort) and
the Level (ma

MB-33186 [SR]: Add couchstore MetaDataV3

Add a MetaDataV3 which contains metadata fields specific to
synchronous replication: the operation (pending, commit, abort) and
the Level (majority, majorityAndPersistActive, persistToMajority).

This inforation is utilised:

a) During warmup to identify pendingSyncWrites which need to be
replayed (re-acknowleged) before being committed.
b) For DCP backfill so a DCP client can correctly receive DCP_PREPARE
messages.

Change-Id: I8210f94d3883a10dc11df489d8fd58111591cf87
Reviewed-on: http://review.couchbase.org/105587
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# dd54a02a 27-Feb-2019 Dave Rigby <daver@couchbase.com>

Reformat couch-kvstore-metadata.h

Run couch-kvstore-metadata.h through clang-format (prior to making
changes for Sync Repl).

Change-Id: I14de8115283e2f80122f388eab08f334696d73ae

Reformat couch-kvstore-metadata.h

Run couch-kvstore-metadata.h through clang-format (prior to making
changes for Sync Repl).

Change-Id: I14de8115283e2f80122f388eab08f334696d73ae
Reviewed-on: http://review.couchbase.org/105586
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v6.0.1, v5.5.3
# af9a9259 23-Oct-2018 Christopher Farman <christopher.farman@couchbase.com>

MB-26907: Add deleteSource to couch-kvstore's metadata

By stealing one bit from flexCode, we can store the deleteSource inside
couch-kvstore-metadata which allows us to retain how a docu

MB-26907: Add deleteSource to couch-kvstore's metadata

By stealing one bit from flexCode, we can store the deleteSource inside
couch-kvstore-metadata which allows us to retain how a document was
deleted on disk without increasing space. This has been achieved by
using a bit mask for flexCode, along with some added bitwise
manipulation for setting and retrieving flexCode and deleteSource.

Change-Id: I50d17796e915524e06b290075dda182697d5c854
Reviewed-on: http://review.couchbase.org/100915
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1, v5.0.1, v5.1.0, v5.0.0
# 45fa1633 21-Aug-2017 Dave Rigby <daver@couchbase.com>

MB-25661: Rename io_read_bytes/num_read to io_bg_fetch_doc_bytes/num_read

To more accuratly reflect what is represents, modify io_read_bytes and
io_num_read:

* Rename io_read_by

MB-25661: Rename io_read_bytes/num_read to io_bg_fetch_doc_bytes/num_read

To more accuratly reflect what is represents, modify io_read_bytes and
io_num_read:

* Rename io_read_bytes to io_bg_fetch_doc_bytes, and io_num_read to
io_bg_fetch_docs_read, as these statistics refer to reads done during
background fetch (and not for example during Flushing).

* Include the size of document metadata in io_bg_fetch_doc_bytes to
more accurately reflect the data read from disk.

Change-Id: I9303fb7bf61a4dfc039bc5db234d47bbaf7da88e
Reviewed-on: http://review.couchbase.org/82568
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# af8487f2 26-Jul-2017 Dave Rigby <daver@couchbase.com>

MB-25398: Allow store_deleted_with_CAS on non-resident items

When attempting to perform a store operation with a deleted body to a
deleted key, with CAS (!), allow it to succeed if the k

MB-25398: Allow store_deleted_with_CAS on non-resident items

When attempting to perform a store operation with a deleted body to a
deleted key, with CAS (!), allow it to succeed if the key is no longer
resident. This is the case if the flusher has completed storing the
delete to disk; and hence the persistence callback has run and removed
the deleted item from the HashTable.

Note this manifested as only an occasional failure when trying to
perform:

1. CAS = delete(key)
2. subdoc_multi_mutate(key, AccessDeleted=True, new_body=...,
in_CAS=cas)

If 1 and 2 executed back-to-back without the flusher running in
between, then the request succeeded, as the deleted item from (1)
would still be resident in the HashTable when we execute (2). However,
if the flusher ran between those two statements, then (2) would fail
with ENOENT.

Change-Id: I4a7e213469a434d437bcab5c78ff34a189ccb5ca
Reviewed-on: http://review.couchbase.org/81479
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>

show more ...


# ef22f9b0 25-May-2017 Dave Rigby <daver@couchbase.com>

Move ep-engine to engines/ep


Revision tags: v4.6.2_ep, v4.6.2_mc, v4.6.1_ep, v4.6.0_ep, v4.5.1-MP1_mc, v4.6.0-DP_mc, v4.6.0-DP_ep
# cb13da85 27-Sep-2016 Trond Norbye <trond.norbye@gmail.com>

Refactor: The core refactored the datatype to be a bitfield

Update the internal in ep-engine to use the new functions
to check for features

Change-Id: Id13ba93bb1bd344655053313d

Refactor: The core refactored the datatype to be a bitfield

Update the internal in ep-engine to use the new functions
to check for features

Change-Id: Id13ba93bb1bd344655053313dbbe77870df43a75
Reviewed-on: http://review.couchbase.org/68680
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v4.5.1-MP1_ep, v4.1.2-MP2_mc, v4.5.1_mc, v4.6.0_mc, v4.1.2-MP1_ep, v3.1.6_ep
# 8350175e 13-Jun-2016 Jim Walker <jim@couchbase.com>

MB-19948: Refactor CouchKVStore meta-data handling

Wrap the ep-engine metadata we store into couchstore in
a class to provide a much cleaner management interface.

This includes

MB-19948: Refactor CouchKVStore meta-data handling

Wrap the ep-engine metadata we store into couchstore in
a class to provide a much cleaner management interface.

This includes a fix for the orginal MB-19948 issue in that
we correctly read V1 metadata (the 18byte version written
by 3.x). The fix is kind of hidden in the refactor, but the
new code explictly looks at the size of metadata and has
construction paths for 16, 18 and 19 bytes.

Tests in cab19c11ea95df30a119196078f7bc8bff683692 lend
some confidence to this refactoring.

Change-Id: I7d2042af519819737a6e4d070fd8185c84778d6a
Reviewed-on: http://review.couchbase.org/64978
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...