History log of /5.5.2/kv_engine/engines/ep/src/storeddockey.h (Results 1 - 14 of 14)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3, v5.5.4, v5.5.5, v5.5.6, v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1
# 12e87fb6 15-May-2018 Dave Rigby <daver@couchbase.com>

MB-29675: Optimize collections lookups when disabled

While collections code is present in Vulcan, it is not enabled in
production configurations - only the default collection is present.

MB-29675: Optimize collections lookups when disabled

While collections code is present in Vulcan, it is not enabled in
production configurations - only the default collection is present.

As such, optimize a number of functions which show up in 'linux perf'
profiles:

- Use std::string[] instead of string::at() do avoid the runtime size
check (and setup for throwing an exception) in StoredDocKey.

- Split Collections::VB::Filter::checkAndUpdate() into a hot and slow
path, and inline the hot path (This is called for every outgoing DCP
mutation).

- Add an explicit case for DefaultCollection in
Collections::DocKey::make() - this avoids an unnecessary call to
findCollection which otherwise needs to scan the document key.

Change-Id: I4c4eeb6ab26dc616728f4dc89ac1fd4243c21e98
Reviewed-on: http://review.couchbase.org/94214
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...


# 2d4b51c0 04-May-2018 Dave Rigby <daver@couchbase.com>

MB-29523: Avoid undefined behaviour upon zero-length SerialisedDocKey

As identified by UBSan, if we try to create a zero-length
SerialisedDocKey (which is valid); the current implementat

MB-29523: Avoid undefined behaviour upon zero-length SerialisedDocKey

As identified by UBSan, if we try to create a zero-length
SerialisedDocKey (which is valid); the current implementation passes a
null pointer to memcpy:

[ RUN ] MutationLogTest.Logging
runtime error: null pointer passed as argument 2, which is declared to never be null
#0 0x11e9309 in SerialisedDocKey::SerialisedDocKey(DocKey) kv_engine/engines/ep/src/storeddockey.h:277
#1 0x11e9309 in MutationLogEntryV2::MutationLogEntryV2(MutationLogType, unsigned short, DocKey const&) kv_engine/engines/ep/src/mutation_log_entry.h:310
#2 0x11e9309 in MutationLogEntryV2::MutationLogEntryV2(MutationLogType, unsigned short) kv_engine/engines/ep/src/mutation_log_entry.h:321
#3 0x11e9309 in MutationLogEntryV2::newEntry(unsigned char*, MutationLogType, unsigned short) kv_engine/engines/ep/src/mutation_log_entry.h:223
#4 0x11e9309 in MutationLog::commit1() kv_engine/engines/ep/src/mutation_log.cc:322

Fix by using std::copy instead.

Change-Id: I0994f1522efeb046c58da375053b6257fdc89a6a
Reviewed-on: http://review.couchbase.org/93762
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...


# ab711953 07-Mar-2018 Tim Bradgate <tim.bradgate@couchbase.com>

MB-27661 [11/n]: Fix MSVC warnings - C4267

This patch addresses the following generated warnings:

C4267 - var : conversion from 'size_t' to 'type', possible loss of data

MB-27661 [11/n]: Fix MSVC warnings - C4267

This patch addresses the following generated warnings:

C4267 - var : conversion from 'size_t' to 'type', possible loss of data
The compiler detected a conversion from size_t to a smaller type.

This is the second and final commit addressing this error.

Change-Id: If23806c43a2024df2d4103f1b40cbd790957f1f0
Reviewed-on: http://review.couchbase.org/90604
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v5.0.1
# dea18910 25-Oct-2017 Jim Walker <jim@couchbase.com>

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from c

MB-26455: Allow correct processing of 'old' keys and separator changes

The collection's separator can only be changed when 0 collections
exist. However nothing stops the separator from changing if there are
deleted collections (and their keys) in play.

Prior to this commit each separator change resulted in a single system
event being mutated, that event had a static key. Thus a VB could have
the following sequence of keys/events.

1 fruit::1
2 fruit::2
<fruit collection is deleted>
<separator is changed from :: to - >
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
<separator is changed from - to # >
9 $collections_separator (the Item value contains the new separator)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

In this sequence, some of the changes are lost historically because a
static key is used for the separator change. Between seqno 2 and 6 the
separator changed from :: to -, but the separator change system event
is currently at seqno 9 with # as its value.

With this setup a problem exists if we were to process the historical
data e.g. whilst backfilling for DCP or performing a compaction
collection erase. The problem is that when we go back to seqno 1 and
2, we have no knowledge of the separator for those items, all we have
is the current # separator. We cannot determine that fruit::1 is a
fruit collection key.

This commit addresses this problem by making each separator change
generate a unique key. The key itself will encode the new separator,
and because the key is unique it will reside at the correct point in
history for each separator change.

The unique key format will be:

$collections_separator:<seqno>:<new_separator>

With this change the above sequence now looks as:

1 fruit::1
2 fruit::2
<fruit collection is deleted>
4 $collections_separator:3:- (change separator to -)
<fruit collection is recreated>
6 fruit-1
7 fruit-2
<fruit collection is deleted>
9 $collections_separator:8:# (change separator to #)
10 $collections::fruit (fruit recreated)
11 fruit#1
12 fruit#2

Now the code which looks at the historical data (e.g. backfill) will
encounter these separator change keys before it encounters collection
keys using that separator. Backfill and collections-erase can just
maintain a current separator and can now correctly split keys to
discover the collection they belong to. The collections eraser and
KVStore scanning code now include a collections context which has data
and associated code for doing this tracking.

A final piece of the commit is the garbage collection of these unique
keys. i.e. if each separator change puts a unique key into the seqno
index, how can we clean these away when they're no longer needed (i.e.
all fruit:: keys purged)?

Whilst the eraser runs it tracks all 'separator change' keys, because
a separator change can only happen when 0 collections exist, it can
assume that all but the latest separator change key are redundant once
the erase has completed. This list of keys are simply deleted in the
normal way by pushing a deleted Item into the checkpoint once
compaction is complete.

Change-Id: I4b38b04ed72d6b39ceded4a860c15260fd394118
Reviewed-on: http://review.couchbase.org/84801
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 130763a5 29-Nov-2017 Dave Rigby <daver@couchbase.com>

MB-26021: Optimize sorting of queued items

When sorting queued items in the flusher (before writing to disk), we
spend a non-trivial amount of time comparing Items by key & seqno.

MB-26021: Optimize sorting of queued items

When sorting queued items in the flusher (before writing to disk), we
spend a non-trivial amount of time comparing Items by key & seqno.

This patch optimizes it by combining the less-than and equality check
into a single call to compare(). This essentially halves the amount
time the actual compare takes.

Add a benchmark to measure the performance of
CompareQueuedItemsBySeqnoAndKey to measure this. This shows a 25%
reduction in runtime for the sorting process:

Before:

Run on (8 X 2300 MHz CPU s)
2017-11-28 15:41:11
--------------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------------
BM_CompareQueuedItemsBySeqnoAndKey 4541594 ns 4540907 ns 140

After:

Run on (8 X 2300 MHz CPU s)
2017-11-28 15:48:54
--------------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------------
BM_CompareQueuedItemsBySeqnoAndKey 3368195 ns 3365729 ns 221

Change-Id: I428179c648b8bb311215b22def5520dcb5f7983a
Reviewed-on: http://review.couchbase.org/86123
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>

show more ...


Revision tags: v5.1.0, v5.0.0
# 56a86674 18-May-2017 Dave Rigby <daver@couchbase.com>

Minimise inclusion of 'item.h'

Remove unnecessary inclusions of item.h, instead use forward decls or
the more specific headers which a file actually needs.

To allow some removal

Minimise inclusion of 'item.h'

Remove unnecessary inclusions of item.h, instead use forward decls or
the more specific headers which a file actually needs.

To allow some removals to occur, un-inline some methods as to remove
the need for the definition of Item from other header files.

Change-Id: Idbcaebce036c3ca4e2fb3d9df14ffa579951a2e9
Reviewed-on: http://review.couchbase.org/78635
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# ef22f9b0 25-May-2017 Dave Rigby <daver@couchbase.com>

Move ep-engine to engines/ep


Revision tags: v4.6.2_ep
# b762539b 14-Mar-2017 Dave Rigby <daver@couchbase.com>

Debug: Add dump() methods to VBucket/SeqList

Add a number of dump() and operator<< methods to dump textual
representation of the class, to aid in debugging.

Change-Id: I4cb3f63f

Debug: Add dump() methods to VBucket/SeqList

Add a number of dump() and operator<< methods to dump textual
representation of the class, to aid in debugging.

Change-Id: I4cb3f63f69d85cab11e1682d82a5ebc562843047
Reviewed-on: http://review.couchbase.org/75148
Reviewed-by: David Haikney <david.haikney@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v4.6.2_mc, v4.6.1_ep
# 9c1a803f 02-Mar-2017 Dave Rigby <daver@couchbase.com>

MB-23109: Fix StoredValue size calculations

A number of the methods to calculate the sizes of StoredValue objects
are incorrect - they over-report the size needed. These issues appear

MB-23109: Fix StoredValue size calculations

A number of the methods to calculate the sizes of StoredValue objects
are incorrect - they over-report the size needed. These issues appear
to be introduced in Spock development, during changes to add
SerializedDocKey (for collections).

Fix them, and enable unit tests to defend against any future issues.

Change-Id: I92f1eefaeb74cb14f83c432563a670caf7c8723b
Reviewed-on: http://review.couchbase.org/74523
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 324c43fc 23-Feb-2017 Jim Walker <jim@couchbase.com>

MB-22896: Add upgrade support for MutationLog.

5d97bde6 broke upgrades in that access log files are no longer
readable and will trigger asserts. The size of each MutationLogEntry
was

MB-22896: Add upgrade support for MutationLog.

5d97bde6 broke upgrades in that access log files are no longer
readable and will trigger asserts. The size of each MutationLogEntry
was increased by 1 byte and thus a access log from 4.6 will no longer
match codes assumptions.

To fix we now increase the version number of the MutationLog from 1
to 2. Next the code adds back the old MutationLogEntry structure so
that we can read a V1 file and convert the data to version 2.

The upgrade uses a serial upgrade technique where we will sequentially
upgrade from n to n + 1 and never skip versions. The intent is that
this will give simpler upgrade steps, but possibly higher cost if
there were many steps.

However in this code there can only be a V1 to V2 upgrade, but the
sturcture of the code is hopefully easily extensible for a few more
versions if we ever need to. After that a table driven approach maybe
better or something more exotic.

A test has been added which manually writes a V1 log file, that is
because MutationLog would need extensive work to make that fully
configurable to generate V1 files, so we just write the bytes into
a vector and flush to disk, then confirm we can now read the data
back.

Manual upgrade testing has also been performed.

Change-Id: Idceec80527ffecdccf2a077a6e00f5e7cb6762ce
Reviewed-on: http://review.couchbase.org/74156
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 9b1d3292 03-Jan-2017 Jim Walker <jim@couchbase.com>

Coverity issue (CID-159031) Uninitialised data-member

docNamespace isn't initialised in the private, default constructor.

Change-Id: I75e8dfc869358d894215c04cce9b373d1c3c930c
Re

Coverity issue (CID-159031) Uninitialised data-member

docNamespace isn't initialised in the private, default constructor.

Change-Id: I75e8dfc869358d894215c04cce9b373d1c3c930c
Reviewed-on: http://review.couchbase.org/71486
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


Revision tags: v4.6.0_ep
# 8ca2bbe4 05-Dec-2016 Jim Walker <jim@couchbase.com>

MB-21917: SerialisedDocKey and test code

Some parts of ep-engine store a document key as part of a larger
memory allocation, e.g. MutationLog has the key inline with other
metadata t

MB-21917: SerialisedDocKey and test code

Some parts of ep-engine store a document key as part of a larger
memory allocation, e.g. MutationLog has the key inline with other
metadata that it wishes to put to file in a single write.

SerialisedDocKey serves this purpose, it can store all data that
DocKey/StoredDocKey carries, but exists to be allocated into a
pre-allocated buffer.

StoredValue and MutationLog make use of this key object.

Change-Id: I21c8f4052b0899e753809d0cd93eb6f2cae0f963
Reviewed-on: http://review.couchbase.org/70722
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


# f734f13f 05-Dec-2016 Jim Walker <jim@couchbase.com>

MB-21916: Make use of StoredDocKey

Where std::string stored a key, now we use StoredDocKey
When we passed std::string& through interfaces we now
pass DocKey as much as possible, thus

MB-21916: Make use of StoredDocKey

Where std::string stored a key, now we use StoredDocKey
When we passed std::string& through interfaces we now
pass DocKey as much as possible, thus delaying the
heap alloc and memcpy to the places we really need
to store a key.

This patch does not store the namespace in StoredValue
This patch does not store the namespace in couchstore/fdb

Thus any keys created in say Collections namespace will
not work. This is correct as this patch does not make
any assertions about the support of non-DefaultCollection

Change-Id: Ibc7f59183e59f43d55fad5e582232e2891231179
Reviewed-on: http://review.couchbase.org/70698
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


# 07df3775 05-Dec-2016 Jim Walker <jim@couchbase.com>

MB-21916: StoredDocKey - A DocKey container and test code

This commit introduces StoredDocKey - a container to replace
our use of std::string for document keys. The container can be used

MB-21916: StoredDocKey - A DocKey container and test code

This commit introduces StoredDocKey - a container to replace
our use of std::string for document keys. The container can be used
in std::map/set etc... and allows for us to store all items that are
needed to reference a document.

- the key (and the key size)
- the document namespace (a critical piece for Collections)

StoredDocKey will store a key of n-bytes in an n+1 std::string.
The extra byte is where we store the DocNamespace.

Change-Id: I1221e55d0081427c5f477773234ebbab482a6226
Reviewed-on: http://review.couchbase.org/70680
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...