MB-13792: Acquire lock before changing checkpointManager's lastBySeqno

Change-Id: I55e381418b0a5b89704f2e9912caabfa4df8d15c
Reviewed-on: http://review.couchbase.org/47665
Reviewed-by

MB-13792: Acquire lock before changing checkpointManager's lastBySeqno

Change-Id: I55e381418b0a5b89704f2e9912caabfa4df8d15c
Reviewed-on: http://review.couchbase.org/47665
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

Rename all classes/variables with upr to dcp

Change-Id: Iaf644f5034fdb30a14c400ec55def22d8bb0ba87
Reviewed-on: http://review.couchbase.org/40942
Reviewed-by: Chiyoung Seo <chiyoung@c

Rename all classes/variables with upr to dcp

Change-Id: Iaf644f5034fdb30a14c400ec55def22d8bb0ba87
Reviewed-on: http://review.couchbase.org/40942
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11973: Only grab items from the chk manager one chk at a time

We should only grab a single checkpoint at a time from the checkpoint
manager in order to make sure that the replica vbuc

MB-11973: Only grab items from the chk manager one chk at a time

We should only grab a single checkpoint at a time from the checkpoint
manager in order to make sure that the replica vbuckets cannot move
into the open checkpoint. This will also help make sure we don't
copy too make items into an upr stream.

Change-Id: I519353df69cf389e6f32f77ffc6afe97dfba7b36
Reviewed-on: http://review.couchbase.org/40765
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11299: Don't let replica vbuckets send partial snapshots

Replica vbuckets cannot create their own snapshots and must
rely on the snapshots it received from the active vbucket
when

MB-11299: Don't let replica vbuckets send partial snapshots

Replica vbuckets cannot create their own snapshots and must
rely on the snapshots it received from the active vbucket
when forwarding the snapshots to a downstream consumer. In
order to make sure that this happens properly we will only
stream mutations from the closed checkpoint for non-active
vbuckets. This will guarantee that the above requirement is
met.

Change-Id: I58151ce96868e34083f87dca0ce1d18b6a12aedf
Reviewed-on: http://review.couchbase.org/40346
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

Minor refactoring on checkpoint cursor registration function.

Change-Id: I7fdffc233664e26a434e0c757e72416e567d1eb2
Reviewed-on: http://review.couchbase.org/40192
Reviewed-by: abhinav

Minor refactoring on checkpoint cursor registration function.

Change-Id: I7fdffc233664e26a434e0c757e72416e567d1eb2
Reviewed-on: http://review.couchbase.org/40192
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Don't schedule backfill if cursor isn't registered at first chk item

If the cursor is not registered at the first item in the checkpoint
then we do not need to schedule backfill. Since i

Don't schedule backfill if cursor isn't registered at first chk item

If the cursor is not registered at the first item in the checkpoint
then we do not need to schedule backfill. Since items get
de-duplicated in the checkpoint queues the current logic might
choose to schedule a backfill unnecessarily. This change stops the
scheduling of unnecessary backfills.

Change-Id: I11188d302be57035cfac5f1c105cbc6d7179bde6
Reviewed-on: http://review.couchbase.org/40145
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Don't collapse checkpoints when registering an upr cursor

Change-Id: Ic0d64518454a7357871362d7045926ddb708e3b4
Reviewed-on: http://review.couchbase.org/39556
Reviewed-by: Chiyoung Se

Don't collapse checkpoints when registering an upr cursor

Change-Id: Ic0d64518454a7357871362d7045926ddb708e3b4
Reviewed-on: http://review.couchbase.org/39556
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11592: Create snapshots when getting items from the chk queues

This change introduces a new flag to indicate whether or not
ep-engine should create a new checkpoint for this snapshot.

MB-11592: Create snapshots when getting items from the chk queues

This change introduces a new flag to indicate whether or not
ep-engine should create a new checkpoint for this snapshot. This
flag is used internally by ep-engine and only ep-engine. We are
also changing the flags values for snapshot markers since they
weren't really flags previously. Below are the new values.

Disk 0x01
Memory 0x02
Checkpoint 0x04

Change-Id: I8a262eb7856c97e11e95570fbdaef82f3f79054a
Reviewed-on: http://review.couchbase.org/38779
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11661 Disable checkpoint merge support by default.

Merging closed checkpoints in a replica vbucket was mainly
supported to reduce the memory overhead in the replica vbucket,
which

MB-11661 Disable checkpoint merge support by default.

Merging closed checkpoints in a replica vbucket was mainly
supported to reduce the memory overhead in the replica vbucket,
which performs the replication to another replica vbucket.
For example, in the replication chain A->B->C, merging closed
checkpoints can be performed on the replica B if the replica C
is overloaded and constantly backs off the replication from the
replica B.

As of 3.0, we switch to the star replication, which doesn't
require supporting checkpoint merge operation. Thus, this change
disables the checkpoint merge operation by default, but still
allows us to enable / disable it dynamically at runtime.

Change-Id: I46b17030f35a5dad9575a6ecd2fde33f70b497bc
Reviewed-on: http://review.couchbase.org/39342
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11553 Adjust the checkpoint creation interval.

This change reduces the checkpoint creation interval to
mitigate the memory overhead caused by having blob instances
inside the open

MB-11553 Adjust the checkpoint creation interval.

This change reduces the checkpoint creation interval to
mitigate the memory overhead caused by having blob instances
inside the open checkpoint. If a checkpoint creation interval
is large, it can cause significant memory overhead due to blob
instances inside the current open checkpoint.

Note that we also limit the number of checkpoints in memory, so
this change won't allow us to keep creating a new checkpoint if
the limit is reached.

Change-Id: I33119ef8b539d5f0209b64bcfcfcbd82632ae7e1
Reviewed-on: http://review.couchbase.org/39333
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Maintain a cursor's offset in a consistent way.

This change makes sure that a cursor's offset includes checkpoint
meta items (e.g., checkpoint_start, checkpoint_end).
However, when w

Maintain a cursor's offset in a consistent way.

This change makes sure that a cursor's offset includes checkpoint
meta items (e.g., checkpoint_start, checkpoint_end).
However, when we calculate the number of items remaining for
persistence or replication, we should not consider those meta
items.

Change-Id: I8a90c347c163674decdb7ed1b5eb824a5b9d4f86
Reviewed-on: http://review.couchbase.org/39298
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11553 Reduce the min/default/max number of checkpoint items

This reverts commit af0f22545d9fd5f2ab675329de5d540ff5990c01.

It turns out that this change has nothing to do with vie

MB-11553 Reduce the min/default/max number of checkpoint items

This reverts commit af0f22545d9fd5f2ab675329de5d540ff5990c01.

It turns out that this change has nothing to do with view index
performance.

Change-Id: I416de3fbb125579cded468584e6ace98e1f34207
Reviewed-on: http://review.couchbase.org/39092
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Revert "MB-11553 Reduce the min/default/max number of checkpoint items."

This reverts commit a72bea627fe0b9ca2d8f1e53947c9e554a8c67d5.

We observed some performance regression in vie

Revert "MB-11553 Reduce the min/default/max number of checkpoint items."

This reverts commit a72bea627fe0b9ca2d8f1e53947c9e554a8c67d5.

We observed some performance regression in view indexing building.

Change-Id: Ie7fb174c8d21353f21bc787574876f0445c485f5
Reviewed-on: http://review.couchbase.org/39090
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11553 Reduce the min/default/max number of checkpoint items.

As UPR can efficiently avoid the full re-materialization in
replication, we should reduce the min/default/max number of it

MB-11553 Reduce the min/default/max number of checkpoint items.

As UPR can efficiently avoid the full re-materialization in
replication, we should reduce the min/default/max number of items
maintained per checkpoint, which were determined before based on
TAP-based replication.

Change-Id: If57ea1c2ca66b9c7e8a76a99767a66af49db0d9b
Reviewed-on: http://review.couchbase.org/39040
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

MB-11325: Fix race condition in updating persisted seqno

We cannot rely on setting the last persisted seqno by checking
where the cursor/backfill queue is after doing persistence. The

MB-11325: Fix race condition in updating persisted seqno

We cannot rely on setting the last persisted seqno by checking
where the cursor/backfill queue is after doing persistence. The
reason is that we might switch from backfill phase to checkpoint
phase in the middle of a commit which will cause the last persisted
seqno to be set to higher than what it actually is. This can cause
backfills to be scheduled pre-maturely as a result and cause us not
to read enough items off of disk.

The only place we can really trust to get the persistence seqno is
from the actual file. So when we do persistence for a particular
vbucket we should cache the last persisted seqno. This will allow us
to get this value without having to do a disk read.

Change-Id: Ie975dae49d5271f31ba7ee48fdb9e06e30eeb8ae
Reviewed-on: http://review.couchbase.org/38243
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11104 Synchronize reading the last seq num of each vbucket.

Reading the last seq num of each vbucket should be guarded
through a lock.

Change-Id: I63cf73b831a9479da39b27a0c03

MB-11104 Synchronize reading the last seq num of each vbucket.

Reading the last seq num of each vbucket should be guarded
through a lock.

Change-Id: I63cf73b831a9479da39b27a0c03c4c0a8ea50ba1
Reviewed-on: http://review.couchbase.org/37464
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11104 Don't return -1 as high seq num for the empty checkpoint

Change-Id: If6abc17a42e0403b90cfb6774c87f7bc37ecc9f5
Reviewed-on: http://review.couchbase.org/37463
Reviewed-by: Mic

MB-11104 Don't return -1 as high seq num for the empty checkpoint

Change-Id: If6abc17a42e0403b90cfb6774c87f7bc37ecc9f5
Reviewed-on: http://review.couchbase.org/37463
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-10941: When backfill is neccessary always do a full backfill

When sending backfill we need to send the entire thing as a single
snapshot in order to avoid potentially missing de-dupli

MB-10941: When backfill is neccessary always do a full backfill

When sending backfill we need to send the entire thing as a single
snapshot in order to avoid potentially missing de-duplicated items
when then checkpoint ends.

Change-Id: If898be3414bb51b4248dad03176255e5934ac6a2
Reviewed-on: http://review.couchbase.org/37295
Tested-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Properly check the perisistence seqno

If we are in the backfill phase then we need to make sure that we
get the persistence seqno based on where we left off in the backfill
queue. If

Properly check the perisistence seqno

If we are in the backfill phase then we need to make sure that we
get the persistence seqno based on where we left off in the backfill
queue. If it is grabbed from the checkpoint queues then it will be
incorrect.

Change-Id: Ife46f8725a7e8f2c2e7298feee88abc7a663f649
Reviewed-on: http://review.couchbase.org/37351
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Fix build breakage due to missing function definition

Change-Id: If0448fd7a60bec99e591b7de85173f2a641540cf
Reviewed-on: http://review.couchbase.org/37359
Reviewed-by: abhinav dangeti

Fix build breakage due to missing function definition

Change-Id: If0448fd7a60bec99e591b7de85173f2a641540cf
Reviewed-on: http://review.couchbase.org/37359
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

Set the current byseqno in the checkpoint manager for backfilled items

If we don't do this then it can cause a sequence number to be re-used
once new items arrive after a backfill.

Set the current byseqno in the checkpoint manager for backfilled items

If we don't do this then it can cause a sequence number to be re-used
once new items arrive after a backfill.

Change-Id: I2bdcde7bf64c4280ea1457e7ab2c6572d07132c0
Reviewed-on: http://review.couchbase.org/37350
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-10640 Improve UPR rollback during failover.

Stream request now has snap_start_seqno and snap_end_seqno.
(high_seqno is removed) which can be used to deal with
partial snapshot. fa

MB-10640 Improve UPR rollback during failover.

Stream request now has snap_start_seqno and snap_end_seqno.
(high_seqno is removed) which can be used to deal with
partial snapshot. failover check is based on start_seqno,
snap_start_seqno and snap_end_seqno.

Snapshot Marker now has start_seqno, end_seqno and flag.
start_seqno/end_seqno indicates sequence number ranges
of the stream.

The flag indicates if a stream is from disk or memory.
The current value are:
0: memory
1: disk

Change-Id: I4f7528f453e1f0c3021ab51f2173e111ed4a17e9
Reviewed-on: http://review.couchbase.org/35198
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-10856 Position the cursors correctly in collapsed checkpoints

Change-Id: I362f232640458420122f477943c6a6fe9da9ef0f
Reviewed-on: http://review.couchbase.org/36821
Tested-by: Sundar

MB-10856 Position the cursors correctly in collapsed checkpoints

Change-Id: I362f232640458420122f477943c6a6fe9da9ef0f
Reviewed-on: http://review.couchbase.org/36821
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...

MB-10514: The chk low seqno should be set to the chk start item

We use the sequence number of the last item in the previous
checkpoint + 1 as the sequence number of the checkpoint start

MB-10514: The chk low seqno should be set to the chk start item

We use the sequence number of the last item in the previous
checkpoint + 1 as the sequence number of the checkpoint start
item. As a result we should consider this number as the low
sequence number for the checkpoint we are in. This fixes an error
which cause a cursor to not be registered improperly due to the
start seqno being set to -1.

Change-Id: Idd53c8a951f5844b6ac0aa1f53a78d76b9fd7e69
Reviewed-on: http://review.couchbase.org/35243
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-10641 Starts from the first item on a collapsed checkpoint

If a given TAP replication cursor was sitting on an old closed
checkpoint and moved over to the collapsed checkpoint, then

MB-10641 Starts from the first item on a collapsed checkpoint

If a given TAP replication cursor was sitting on an old closed
checkpoint and moved over to the collapsed checkpoint, then
the cursor should start from the first item on the collapsed
checkpoint, so that CHECKPOINT_START message with a new
checkpoint id should be sent to the destination node.

Change-Id: I62314ea402a0d5d7a525445b9e21f6227c9975ef
Reviewed-on: http://review.couchbase.org/34924
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...