History log of /6.0.3/kv_engine/engines/ep/tests/ep_testsuite.cc (Results 1 - 25 of 1213)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3, v5.5.4, v5.5.5, v5.5.6
# 90bd9f55 19-Mar-2019 sduvuru <sduvuru@gmail.com>

MB-35058: Couchstore-Trace operations on a file

-Tracing support to couchstore and dump trace on detection of corruption
-mprotect of iobuffer
-verify write to buffer cache by readin

MB-35058: Couchstore-Trace operations on a file

-Tracing support to couchstore and dump trace on detection of corruption
-mprotect of iobuffer
-verify write to buffer cache by reading back after write

To compile a kvengine test, need to modify the CMakeLists.txt.

Dynamically turn on/off tracing, write verification and mprotect for
couchstore with cbepctl

For example,
./cbepctl localhost:12000 -u Administrator -p ### -b Test
set flush_param couchstore_tracing true
./cbepctl localhost:12000 -u Administrator -p ### -b Test
set flush_param couchstore_write_validation true
./cbepctl localhost:12000 -u Administrator -p ### -b Test
set flush_param couchstore_mprotect true

Change-Id: I7b6f29f493ecbad377df966d3004d2fbbb32e8e1
Reviewed-on: http://review.couchbase.org/113539
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Well-Formed: Build Bot <build@couchbase.com>

show more ...


# 9ea0523f 10-Apr-2019 Daniel Owen <owend@couchbase.com>

MB-34438: Return NOT_MY_VBUCKET for getReplica on pending vbucket

[Backport of MB-33683.]

Currently when a GET_REPLICA operation is sent to a vbucket in a pending
state it is pl

MB-34438: Return NOT_MY_VBUCKET for getReplica on pending vbucket

[Backport of MB-33683.]

Currently when a GET_REPLICA operation is sent to a vbucket in a pending
state it is placed in the pendingOps list and is processed when the
vbucket is moved to the active state.

However given that is is not valid for a GET_REPLICA to be applied to
an active vbucket, when it gets to execute it will just NOT_MY_VBUCKET.

Another issue with placing a GET_REPLICA operation into the pendingOps
list is that if a rebalance fails and the vbucket move from a pending
state back into a replica state, the operation will be held in the
pendingOps list potentially indefinately.

This patch ensures that if a GET_REPLICA operation is sent to a vbucket
in a pending state we immediately return NOT_MY_VBUCKET.

Change-Id: Ifeab25a06ddb823c3e7285733d317d2623c4876e
Reviewed-on: http://review.couchbase.org/110223
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...


# 5e9fca9f 16-May-2019 Jim Walker <jim@couchbase.com>

MB-34173: 2/2 Check for illegal snapshot in vbstate document

Prior to the fix for MB-34173 the persisted snapshot can be
illegal, we should check for obvious illegal state and take
a

MB-34173: 2/2 Check for illegal snapshot in vbstate document

Prior to the fix for MB-34173 the persisted snapshot can be
illegal, we should check for obvious illegal state and take
action.

Active VBs with invalid state gets reset to valid state, e.g.
start=end=high-seqno

ReplicaVBs will be ignored, we will warmup without them and
the janitor will correct KV and set a new replica which rebuilds
from 0.

Change-Id: I1a322a251363fec67bf7e37ca7b0dcff5ea96905
Reviewed-on: http://review.couchbase.org/109270
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 55dedbb3 29-Apr-2019 Jim Walker <jim@couchbase.com>

MB-33919: Generate delete time for newly deleted items

Rather than allowing the existing expiry time to become the delete-time,
which is flawed as per the MB, allow the VBQueueItemCtx t

MB-33919: Generate delete time for newly deleted items

Rather than allowing the existing expiry time to become the delete-time,
which is flawed as per the MB, allow the VBQueueItemCtx to determine
how the delete-time should be set.

DCP consumer and the with-meta interface should not generate a
delete-time, the incoming 'replicated' value should be accepted. However
a value of zero is not allowed, and will be ignored/regenerated
regardless of the VBQueueItemCtx setting.

Genuine deletes/expirys should generate a new delete-time in-line with
the issues highlighted in the MB.

Change-Id: I7ea6ed8575eaa510a02cceb37a86628a28405fb5
Reviewed-on: http://review.couchbase.org/108393
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# bfb2058b 10-Apr-2019 Daniel Owen <owend@couchbase.com>

[BP] MB-33906: Return NOT_MY_VBUCKET for getReplica on pending vbucket

[Backport of MB-33683.]

Currently when a GET_REPLICA operation is sent to a vbucket in a pending
state it

[BP] MB-33906: Return NOT_MY_VBUCKET for getReplica on pending vbucket

[Backport of MB-33683.]

Currently when a GET_REPLICA operation is sent to a vbucket in a pending
state it is placed in the pendingOps list and is processed when the
vbucket is moved to the active state.

However given that is is not valid for a GET_REPLICA to be applied to
an active vbucket, when it gets to execute it will just NOT_MY_VBUCKET.

Another issue with placing a GET_REPLICA operation into the pendingOps
list is that if a rebalance fails and the vbucket move from a pending
state back into a replica state, the operation will be held in the
pendingOps list potentially indefinately.

This patch ensures that if a GET_REPLICA operation is sent to a vbucket
in a pending state we immediately return NOT_MY_VBUCKET.

Change-Id: Ifeab25a06ddb823c3e7285733d317d2623c4876e
Reviewed-on: http://review.couchbase.org/108159
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 37cf721a 26-Mar-2019 Jim Walker <jim@couchbase.com>

MB-15009: 3/3 Defragment HashTable - StoredValue

Building on the existing value defragmenter, we now
defragment StoredValues for persistent buckets.

The StoredValue carries an a

MB-15009: 3/3 Defragment HashTable - StoredValue

Building on the existing value defragmenter, we now
defragment StoredValues for persistent buckets.

The StoredValue carries an age (hidden in the value tag)
which when it reaches a new threshold, triggers the
defragger to ask the HashTable to reallocate the stored
value.

Change-Id: I4b50ebe4a7abc50e15389bc1d7dea4301543a174
Reviewed-on: http://review.couchbase.org/106450
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2
# 2e3b59f6 12-Jun-2018 Sriram Ganesan <sriram@couchbase.com>

MB-28685: Expand ep_testsuite tests for compression active mode

As part of expanding ep_testsuite to work with all compression
modes, run the ep_testsuite tests for active compression mo

MB-28685: Expand ep_testsuite tests for compression active mode

As part of expanding ep_testsuite to work with all compression
modes, run the ep_testsuite tests for active compression mode

Change-Id: Ia61de4deefaa4d69abbbb18769ffb300c0abadb2
Reviewed-on: http://review.couchbase.org/95505
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# e499bd30 11-Jun-2018 Sriram Ganesan <sriram@couchbase.com>

MB-28685: Expand ep_testsuite value_eviction tests to passive mode

As part of expanding the different ep_testsuite tests to work in
different compression modes, this change will support

MB-28685: Expand ep_testsuite value_eviction tests to passive mode

As part of expanding the different ep_testsuite tests to work in
different compression modes, this change will support the different
ep_testsuite value_eviction tests to passive compression mode

Change-Id: I35960c4a36b1c323764c43be87c561d843831a6c
Reviewed-on: http://review.couchbase.org/95449
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 6edbdbfb 24-May-2018 Daniel Owen <owend@couchbase.com>

Add setParam ability for ItemPager configuration parameters

There is now the ability to choose between the original "2-bit_lru" hash
table eviction policy and the new "hifi_mfu" eviction

Add setParam ability for ItemPager configuration parameters

There is now the ability to choose between the original "2-bit_lru" hash
table eviction policy and the new "hifi_mfu" eviction policy. The
"hifi_mfu" will be the default.

There are a number of configuration paramters associated with the
"hifi_mfu" eviction policy:
- item_eviction_age_percentage
- item_eviction_freq_counter_age_threshold
- item_freq_decayer_chunk_duration
- item_freq_decayer_percent

It is possible that we might need to tune these parameters in the field
and therefore we want to be able to set them at runtime using the
cbepctl tool.

In addition if we find a serious problem with the new "hifi_mfu"
eviction policy, we might want to swap at runtime back to the
"2-bit_lru" policy.

Finally both eviction policies also make use of the following two
configuration parameters:
- pager_active_vb_pcnt
- pager_sleep_time_ms

Therefore it is useful to also ensure that these can be set at runtime.

Change-Id: Ia164ecdfe09e4cef59190b2bf41e1e2bf08aec3b
Reviewed-on: http://review.couchbase.org/94698
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...


Revision tags: v5.1.1
# 233d7048 16-May-2018 Daniel Owen <owend@couchbase.com>

MB-29013: Add temporal behaviour to hifi_mfu eviction policy

The YSCB workload D is a highly temporal workload. The current
hfi_mfu eviction policy only relies on execution frequency to

MB-29013: Add temporal behaviour to hifi_mfu eviction policy

The YSCB workload D is a highly temporal workload. The current
hfi_mfu eviction policy only relies on execution frequency to make
eviction decisions.

We are therefore introducing a temporal element into the eviction
decision. The rule is that if an item has an execution frequency
below the freq_counter_age_threshold (defaulted to 1) then its
eviction decision will be based solely on being below the execution
frequency (and if its eligible for eviction).

If an item has an execution counter greater than or equal to
freq_counter_age_threshold then we also take age into account. We only
evict if the item frequency count is less than or equal to the frequency
threshold and the age is greater than or equal to the ageThreshold.

The ageThreshold is generated using an approach similar to the frequency
threshold. We maintain an ageHistogram, and set the ageThreshold from
reading out a value from the histogram at a given percentage. The
percentage is a configuration parameter (defaulted to 20%).

Change-Id: I21378d6a8260c4e9d8908712fcbd4d763794c054
Reviewed-on: http://review.couchbase.org/94286
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 01101706 16-May-2018 Tim Bradgate <tim.bradgate@couchbase.com>

MB-29707: Add checkpoint memory overhead stats

Change-Id: If1e8666d043d76a1fae64f2e7909d36be24790e9
Reviewed-on: http://review.couchbase.org/94307
Tested-by: Build Bot <build@couchba

MB-29707: Add checkpoint memory overhead stats

Change-Id: If1e8666d043d76a1fae64f2e7909d36be24790e9
Reviewed-on: http://review.couchbase.org/94307
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...


# 6aadc6d8 12-Apr-2018 Tim Bradgate <tim.bradgate@couchbase.com>

MB-29240: Add checkpoint memory stats to 'stats all'

Change-Id: Idb9d80b23b34ad22114d0c52f62b7966bd97ed8d
Reviewed-on: http://review.couchbase.org/92619
Tested-by: Build Bot <build@c

MB-29240: Add checkpoint memory stats to 'stats all'

Change-Id: Idb9d80b23b34ad22114d0c52f62b7966bd97ed8d
Reviewed-on: http://review.couchbase.org/92619
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# b33e0670 12-Apr-2018 Tim Bradgate <tim.bradgate@couchbase.com>

MB-29227: Add second trigger path for cursor dropping

Add a second trigger condition for dropping checkpoint cursors. This
trigger is if the total memory used for checkpoints is greater

MB-29227: Add second trigger path for cursor dropping

Add a second trigger condition for dropping checkpoint cursors. This
trigger is if the total memory used for checkpoints is greater than
a % of the bucket quota (initially 50%). If this condition is met,
then we try to free as much memory as we can to take the usage back
down to the lower threshold.

The reason for this change is that we are able to get into a
live-lock like state under the following conditions:

1) All items have been evicted from memory (including meta-data)
2) Checkpoint memory usage is high && overall memory usage >
high water mark
3) Overall memory usage is < 95%

In these conditions, cursors are not being dropped to free up
memory, but the server is not accepting any more writes to
increase memory usage above the point we start dropping cursors.
The server enters a state where it cannot do anything whilst it
waits for the DCP streams to catch up and free checkpoint memory
through normal operation.

Change-Id: Ia7bc2d9a6a840dfed5972c31308f1942c0e92e30
Reviewed-on: http://review.couchbase.org/92559
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...


# 988ab556 05-Apr-2018 Jim Walker <jim@couchbase.com>

prepare_full_evict is skipping full-eviction tests

The prepare_full_evict function should be running full-eviction tests
and skipping value-eviction, however it is skipping full eviction

prepare_full_evict is skipping full-eviction tests

The prepare_full_evict function should be running full-eviction tests
and skipping value-eviction, however it is skipping full eviction
tests.

Change-Id: I11de469f3cc31b08c5d469b40c763d8dd4916cc6
Reviewed-on: http://review.couchbase.org/92187
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# a52ce0a4 10-Apr-2018 Jim Walker <jim@couchbase.com>

MB-29119: Replace revSeqno with a 48-bit counter

Prevent a value too large to be stored in couchstore
from being placed into Item/StoredValue and also the
_local document (via vbucke

MB-29119: Replace revSeqno with a 48-bit counter

Prevent a value too large to be stored in couchstore
from being placed into Item/StoredValue and also the
_local document (via vbucket_state).

Change-Id: I8613c0c51388e91612cde6216445a38c3351c190
Reviewed-on: http://review.couchbase.org/92485
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# c40c7a43 05-Apr-2018 Sriram Ganesan <sriram@couchbase.com>

MB-28631: Add statistic to compute uncompressed item size

Adding this statistic to the hash table would allow us to
compare item memory size due to compression and the memory
size if

MB-28631: Add statistic to compute uncompressed item size

Adding this statistic to the hash table would allow us to
compare item memory size due to compression and the memory
size if the data was stored as uncompressed.

Change-Id: I85f10e6eef5d3ffd434b214c27dc2293f509dac2
Reviewed-on: http://review.couchbase.org/91947
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# d92cc586 27-Mar-2018 Sriram Ganesan <sriram@couchbase.com>

MB-28777: Provide a min_compression_ratio configuration parameter

If there isn't much difference between sizes of compressed and
uncompressed data, then decompressing the data every time

MB-28777: Provide a min_compression_ratio configuration parameter

If there isn't much difference between sizes of compressed and
uncompressed data, then decompressing the data every time for
a client would outweigh the cost of storing the data as
compressed as there is no perceivable benefit in space savings

Change-Id: Ic6d735213faa00dfc2331d577f4cee80b46e715e
Reviewed-on: http://review.couchbase.org/91769
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# 66e17110 27-Mar-2018 Sriram Ganesan <sriram@couchbase.com>

MB-28230: Move active compression to a separate ItemCompressorTask

The DefragmenterTask operates only on a smaller chunk of items at
a time, so that the hash table lock isn't held for to

MB-28230: Move active compression to a separate ItemCompressorTask

The DefragmenterTask operates only on a smaller chunk of items at
a time, so that the hash table lock isn't held for too long that
will block front-end threads. Also, the defragmenter operates on a
10 second interval which is not active enough for active compression.
Moving it to a separate task would also allow for further customizations
and optimizations on when the task can get scheduled.

Run on (8 X 2500 MHz CPU s)
2018-03-23 08:58:47
———————————————————————————————————————————————————————————————————————————————---
Benchmark Time CPU Iterations
UserCounters…——————————————————————————————————————————————————————————————————----
ItemCompressorBench/Visit/0 6941595 ns 6937531 ns 96 6.8733M items/s ValueOnly
ItemCompressorBench/Visit/1 6652186 ns 6651641 ns 103 7.16871M items/s FullEviction
—————————————————————————————————————————————————————————————————————————————————--

Change-Id: Ieed20fc4985936663595c4116504fc8e44c4e043
Reviewed-on: http://review.couchbase.org/90996
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# f985563a 15-Mar-2018 Daniel Owen <owend@couchbase.com>

MB-28721: Add stat for number of times freqDecayerTask is run

To help with debugging any performance issues with the hifi_mfu
eviction policy we want to record how many times the freqDec

MB-28721: Add stat for number of times freqDecayerTask is run

To help with debugging any performance issues with the hifi_mfu
eviction policy we want to record how many times the freqDecayerTask is
run.

Change-Id: I6c45d8f9c6e721ce7fb24307f02a67ac1939c4ee
Reviewed-on: http://review.couchbase.org/91003
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# e759667e 05-Mar-2018 Jim Walker <jim@couchbase.com>

MB-28429: Test for snappy correctly from notify_expired_item

The code was incorrectly using a logical or and must also now check
the document datatype because a compressed document writt

MB-28429: Test for snappy correctly from notify_expired_item

The code was incorrectly using a logical or and must also now check
the document datatype because a compressed document written as
snappy won't have the content_meta flag, but will have datatype snappy

Change-Id: I84a70ad8d95ac629910707bf05e3ad29e23a0d52
Reviewed-on: http://review.couchbase.org/90435
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# c20a676b 05-Mar-2018 Jim Walker <jim@couchbase.com>

MB-28453: [3/n] Add decompress support to xattr::Blob

A number of places use the xattr::Blob on data which has the xattr
datatype, these places need to account for datatype snappy too.

MB-28453: [3/n] Add decompress support to xattr::Blob

A number of places use the xattr::Blob on data which has the xattr
datatype, these places need to account for datatype snappy too.

Enhancing xattr::Blob so it can handle the decompression is the
simplest function fix.

A number of commands are updated:

* dcp_mutation - which was the opcode that triggered the MB
* append/prepend/arithmetic - likely a redundant is_snappy test
because they have decompression support in the getItem part of the
executor
* remove
* subdoc processing
* doc_pre_expiry

Change-Id: Ib0e2442f4e5fcb70bb000b7e0673515a197776d0
Reviewed-on: http://review.couchbase.org/90425
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# d6affa8d 02-Mar-2018 Jim Walker <jim@couchbase.com>

MB-28453: [2/n] Switch xattrs from byte_buffer to char_buffer

Long-term we wish to remove the use of byte_buffer as we frequently
end up casting it to char (and the nonstop reinterpret_c

MB-28453: [2/n] Switch xattrs from byte_buffer to char_buffer

Long-term we wish to remove the use of byte_buffer as we frequently
end up casting it to char (and the nonstop reinterpret_cast is
tedious). This commit just switches xattr code from using a
byte_buffer to char_buffer, making it easier to add decompression
support to xattr::Blob which is required to cleanly address the MB
this commit is linked with.

A couchstore change is linked to this as dbdump.cc requires an update
to keep working with the Blob changes.

Change-Id: I197723539ea1f7c799ccf3dcddfa1757531e36ba
Reviewed-on: http://review.couchbase.org/90363
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 175713a0 19-Feb-2018 Paolo Cocchi <paolo.cocchi@couchbase.com>

CouchRocks: Control space amplification under Universal Compaction

Under Universal Compaction we can control (relax/narrow) the space
amplification constraint using an internal RocksDB o

CouchRocks: Control space amplification under Universal Compaction

Under Universal Compaction we can control (relax/narrow) the space
amplification constraint using an internal RocksDB option. From RocksDB
documentation:

"The size amplification is defined as the amount (in percentage) of
additional storage needed to store a single byte of data in the
database.
For example, a size amplification of 2% means that a database
that contains 100 bytes of user-data may occupy upto 102 bytes of
physical storage. By this definition, a fully compacted database
has a size amplification of 0%.
Default: 200, which means that a 100 byte database could require
upto 300 bytes of storage."

With this change we introduce the new
'rocksdb_uc_max_size_amplification_percent' parameter in configuration.
We keep the RocksDB default as baseline value.

Change-Id: I9d318041d78fe9eb6923f1f2ebd13fa0c5496c4f
Reviewed-on: http://review.couchbase.org/90133
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# f8bc40cf 21-Feb-2018 Dave Rigby <daver@couchbase.com>

Remove remnants of SQLite WAL tests

These 'MULTI_DISPATCHER_CONFIG' tests appear to date to before
couchstore; and are additonal variants on existing tests (note the
'(wal)' suffix).

Remove remnants of SQLite WAL tests

These 'MULTI_DISPATCHER_CONFIG' tests appear to date to before
couchstore; and are additonal variants on existing tests (note the
'(wal)' suffix).

Given this configuration is long dead, remove these tests as they are
essentially just duplicates of the non-wal suffixed ones.

Change-Id: I2b9c62eee28957da58f239473fe142282049634a
Reviewed-on: http://review.couchbase.org/89826
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...


# 90c76d4f 16-Feb-2018 Dave Rigby <daver@couchbase.com>

MB-26021 [5/6]: Set max_checkpoints=100 & chk_max_items=10000

Modify the checkpoint configuration to permit more granular
checkpoints: increase the maximum number of items in a checkpoin

MB-26021 [5/6]: Set max_checkpoints=100 & chk_max_items=10000

Modify the checkpoint configuration to permit more granular
checkpoints: increase the maximum number of items in a checkpoint
before we can create a new checkpoint from 500 to 10,000 items; and
increase the maximum number of checkpoints from 2 to 10.

Previously max_ckpt=2 meant that you would have at most 1 closed
checkpoint; and 1 open checkpoint. If there was a large DWQ (e.g. 100,000
items) then the majority of items would be inside a single large, open
checkpoint:

CKPT[0]: 500 items (max_items=500)
CKPT[1]: 99,500 items

As we have to flush complete checkpoints for consistency, this forces
us to flush 99,500 items in a single checkpoint.

With this change, we allow a larger number of checkpoints, and
additionlly each checkpoint can hold more items before it is
closed. The same 100K items should now look like:

CKPT[0]: 10,000 items (max_items=10,000)
CKPT[1]: 10,000 items
...
CKPT[9]: 10,000 items

This opens up the possibility[*] to flush in smaller chunks - we can
now flush just 10,000 items at once.

Additionally this allows us to close checkpoints at a smaller
granularity - previously if there was a DCP cursor inside CKPT[1]
(99,500 items) then until that cursor moves to the end of that
checkpoint we cannot release any of the memory associated with the
checkpoint. with a larger number of checkpoints we can essentially
free in 10K chunks.

[*] Note: It isn't yet possible to flush less than all outstanding
items - as flushVB() always fetches *all* checkpoint items for the
persistence cursor; however the next patch in the sequence will add
support for this.

Change-Id: I794a3e4b7d65bf69429f3517645080f15d0b4f65
Reviewed-on: http://review.couchbase.org/89586
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


12345678910>>...49