Fix build breakage on ubuntu

Change-Id: Ic8d7b9c17086a612b55e2c84a26259081d29c8d4
Reviewed-on: http://review.couchbase.org/47107
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tes

Fix build breakage on ubuntu

Change-Id: Ic8d7b9c17086a612b55e2c84a26259081d29c8d4
Reviewed-on: http://review.couchbase.org/47107
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

MB-13479: Indicate rollback to DCP clients if purge_seqno > snap_start_seqno

The replica may not get all the items if there are purged items in the active
node. Hence this solution propo

MB-13479: Indicate rollback to DCP clients if purge_seqno > snap_start_seqno

The replica may not get all the items if there are purged items in the active
node. Hence this solution proposes to indicate a rollback to seqno 0 in case
the purge_seqno > snap_start_seqno.
Note: Since the occurrence of this scenario is pretty rare, rolling back to 0
may not be a bad thing.

Change-Id: I5c8403115110be136df5d4cb4e2704edc2a4c9e4
Reviewed-on: http://review.couchbase.org/47094
Tested-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-13386: Do not purge the item with highest sequence number from db.

When the highest seq number that DCP is supposed to read from
the db is the last seq number in the db and is purged,

MB-13386: Do not purge the item with highest sequence number from db.

When the highest seq number that DCP is supposed to read from
the db is the last seq number in the db and is purged,
DCP backfill does not know about it and hence waits for it.
This results in DCP connection hang.
To solve this problem, during compaction we do not purge the
item with last(highest) sequence number in the db.

Change-Id: Ib83c335da3f7c0a952e4b760309276f73bff4ccf
Reviewed-on: http://review.couchbase.org/47035
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>

show more ...

MB-11527: Allow tuning of compaction_write_queue_cap

Configuration parameter: compaction_write_queue_cap
Can be set in runtime through cbepctl

Change-Id: Id35899865509f5de13a956

MB-11527: Allow tuning of compaction_write_queue_cap

Configuration parameter: compaction_write_queue_cap
Can be set in runtime through cbepctl

Change-Id: Id35899865509f5de13a9565802aeec7f84a71f3d
Reviewed-on: http://review.couchbase.org/46642
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

MB-12647: Ensure CAS value will always be unique

In windows, the gethrtime() returns the same timestamp when 2 requests
are made in a very short interval, thus causing the same CAS to re

MB-12647: Ensure CAS value will always be unique

In windows, the gethrtime() returns the same timestamp when 2 requests
are made in a very short interval, thus causing the same CAS to return.
This can race conditions with 2 consecutive requests, resulting in
data corruption.

Change-Id: I4f396de3f14129504ca406ebb8d4c7a9f3a89bd8
Reviewed-on: http://review.couchbase.org/43271
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-12576: Limit writer threads to 4

Having more than 4 writers increases bgfetch latencies in DGM
This change selectively reverts 32a166c511d7b242433011a875402e1278300add
Change-Id:

MB-12576: Limit writer threads to 4

Having more than 4 writers increases bgfetch latencies in DGM
This change selectively reverts 32a166c511d7b242433011a875402e1278300add
Change-Id: Icdb996622237747e759c52751f2c8e613c9ba262
Reviewed-on: http://review.couchbase.org/43060
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-12279: Incrementally spawn writer threads for buckets

This change is needed to mitigate the high bgfetch latency
observed in heavy Data-Greater-than-Memory scenarios because
havin

MB-12279: Incrementally spawn writer threads for buckets

This change is needed to mitigate the high bgfetch latency
observed in heavy Data-Greater-than-Memory scenarios because
having high number of writing threads slows down disk read
performance.
Also mitigates MB-11143 slowdown in single HDD case

NOTE: This may slow down disk persistence in fast SSDs by default
NOTE: cbepctl can still be used to dynamically tune writers at runtime

Change-Id: Iddf0d3094f38b66ba8c0e09d6d6a307d15b38d56
Reviewed-on: http://review.couchbase.org/42552
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-12271: Set noop interval individually for dcp connections

We need to do this because it might be the case that two servers
have their respective noop intervals set to different values

MB-12271: Set noop interval individually for dcp connections

We need to do this because it might be the case that two servers
have their respective noop intervals set to different values. If
they are this can cause the connections to be disconnected because
each side is expecting to see a noop at different times.

Change-Id: I6ff475ccba407547e7285fa431b86ad9bf9cdc24
Reviewed-on: http://review.couchbase.org/42049
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-12160: Use appropriate CAS for comparison with locked items

In the case of setWithMeta/deleteWithMeta commands, the locked item's
CAS value needs to be compared with the incoming muta

MB-12160: Use appropriate CAS for comparison with locked items

In the case of setWithMeta/deleteWithMeta commands, the locked item's
CAS value needs to be compared with the incoming mutation's CAS.

Change-Id: Id12a3c4717b18bc41c3d4b7dded99ea215179e9d
Reviewed-on: http://review.couchbase.org/41594
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-12063 Don't delete expired items during warmup.

When a node is added back to the cluster, its vbucket states
can be different from its previous states (e.g., active -> replica).
T

MB-12063 Don't delete expired items during warmup.

When a node is added back to the cluster, its vbucket states
can be different from its previous states (e.g., active -> replica).
Therefore, we should not remove expired items from active vbuckets
during warmup because the states of those vbuckets can be changed
after the warmup.

Change-Id: Ifec4fda80e424a8afa74bae66e17de77b2e0c36b
Reviewed-on: http://review.couchbase.org/41219
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Rename all classes/variables with upr to dcp

Change-Id: Iaf644f5034fdb30a14c400ec55def22d8bb0ba87
Reviewed-on: http://review.couchbase.org/40942
Reviewed-by: Chiyoung Seo <chiyoung@c

Rename all classes/variables with upr to dcp

Change-Id: Iaf644f5034fdb30a14c400ec55def22d8bb0ba87
Reviewed-on: http://review.couchbase.org/40942
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

Replace upr with dcp in all file names

Change-Id: Ia418e96b5bcebf40ef99e3c27ce3fb8f20f68f0e
Reviewed-on: http://review.couchbase.org/40941
Reviewed-by: Chiyoung Seo <chiyoung@couchba

Replace upr with dcp in all file names

Change-Id: Ia418e96b5bcebf40ef99e3c27ce3fb8f20f68f0e
Reviewed-on: http://review.couchbase.org/40941
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11722: Remove the mcCouch dependency

Now,
- unlinking stale files at the end of compaction
- unlinking files in couchkvstore reset
- unlink files in delVbucket

Change

MB-11722: Remove the mcCouch dependency

Now,
- unlinking stale files at the end of compaction
- unlinking files in couchkvstore reset
- unlink files in delVbucket

Change-Id: I000fbd076856b8de1a0dcb82aa673b3316dc745f
Reviewed-on: http://review.couchbase.org/40195
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-12057: warmup must not block bucket creation

Bucket creation is a front end thread and warmup is done by
engine background threads. So if bucket creation waits for
warmup then a d

MB-12057: warmup must not block bucket creation

Bucket creation is a front end thread and warmup is done by
engine background threads. So if bucket creation waits for
warmup then a deadlock happens when there are few reader threads
Test Infrastructure expects vb0 created by ep-engine init
or warmup. This is not necessary for production, so moving
vb0 creation to test setup. Minor test code bug fixes.
Change-Id: I5eae9c8f36fe39d2a18cb1800dc48a29162eab28
Reviewed-on: http://review.couchbase.org/40892
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11890: Active vb shouldn't initiate add Stream req

Change-Id: Ic902e632d52f426a73d7708f9c69854dc7fab874
Reviewed-on: http://review.couchbase.org/40641
Reviewed-by: Michael Wiederh

MB-11890: Active vb shouldn't initiate add Stream req

Change-Id: Ic902e632d52f426a73d7708f9c69854dc7fab874
Reviewed-on: http://review.couchbase.org/40641
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

MB-11299: Don't let replica vbuckets send partial snapshots

Replica vbuckets cannot create their own snapshots and must
rely on the snapshots it received from the active vbucket
when

MB-11299: Don't let replica vbuckets send partial snapshots

Replica vbuckets cannot create their own snapshots and must
rely on the snapshots it received from the active vbucket
when forwarding the snapshots to a downstream consumer. In
order to make sure that this happens properly we will only
stream mutations from the closed checkpoint for non-active
vbuckets. This will guarantee that the above requirement is
met.

Change-Id: I58151ce96868e34083f87dca0ce1d18b6a12aedf
Reviewed-on: http://review.couchbase.org/40346
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11912 Monitor workload pattern for dynamic compaction scheduling.

This change allows us to monitor workload pattern (i.e.,
read_heavy, write_heavy, mixed) at runtime and use the monit

MB-11912 Monitor workload pattern for dynamic compaction scheduling.

This change allows us to monitor workload pattern (i.e.,
read_heavy, write_heavy, mixed) at runtime and use the monitored
workload pattern to adjust the number of concurrent compaction
tasks dynamically at runtime. If the workload pattern is read-heavy,
then only one compaction task is executed at a time.

Note that we plan to keep improving this area for better I/O
scheduling.

Change-Id: Ibbb27da0a7e9f3951571ad8068570ebbf9abb318
Reviewed-on: http://review.couchbase.org/40520
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11887: Properly send the snapshot start and end seqnos

Currently we always set the snapshot start and end seqno to be the
same as the start seqno. This is incorrect if we recently wer

MB-11887: Properly send the snapshot start and end seqnos

Currently we always set the snapshot start and end seqno to be the
same as the start seqno. This is incorrect if we recently were
disconnected and need to resume from a partial snapshot.

Change-Id: I7c9430e0b24ef01cc427cb03b518f4c41a55d875
Reviewed-on: http://review.couchbase.org/40276
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

After APPEND/PREPEND to json, set datatype to binary

This is because appending or prepending anything
to a JSON document breaks the JSON structure.

Change-Id: I0acd99b0c43a54e0b

After APPEND/PREPEND to json, set datatype to binary

This is because appending or prepending anything
to a JSON document breaks the JSON structure.

Change-Id: I0acd99b0c43a54e0b34ce21f3a69c41709e3817e
Reviewed-on: http://review.couchbase.org/40390
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

MB-11884: Consider min_item_threshold in full eviction

In this case, as warmed up keys always matches warmed
up values, we can mark the min item threshold using
warmup's estimated it

MB-11884: Consider min_item_threshold in full eviction

In this case, as warmed up keys always matches warmed
up values, we can mark the min item threshold using
warmup's estimated item count.

Change-Id: Ibd1d58edd15be877c24ef794615fe0d56f9fc675
Reviewed-on: http://review.couchbase.org/40308
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

Have at least 1 unit test with high priority bucket

Change-Id: Icc92c9455c13ca1af21eb285c72b8a8f3555cf6b
Reviewed-on: http://review.couchbase.org/40313
Reviewed-by: Chiyoung Seo <chi

Have at least 1 unit test with high priority bucket

Change-Id: Icc92c9455c13ca1af21eb285c72b8a8f3555cf6b
Reviewed-on: http://review.couchbase.org/40313
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...

MB-11405 Spawn at least 10 global threads

Make the number of threads per task group configurable both
at start time as well as runtime for performance tuning
Change-Id: I4dcbb4de79cf

MB-11405 Spawn at least 10 global threads

Make the number of threads per task group configurable both
at start time as well as runtime for performance tuning
Change-Id: I4dcbb4de79cfa39375602e498f321d596145b81d
Reviewed-on: http://review.couchbase.org/40084
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11821 rename upr to dcp in stats and logs

Change-Id: Ib2cf48190b2bf274f953b8da1b1cd1373de190c2
Reviewed-on: http://review.couchbase.org/39898
Reviewed-by: Michael Wiederhold <mike

MB-11821 rename upr to dcp in stats and logs

Change-Id: Ib2cf48190b2bf274f953b8da1b1cd1373de190c2
Reviewed-on: http://review.couchbase.org/39898
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

Fix to unit test failure: test_item_pager

Change-Id: I186e6d631ff6723fbb2833cff450e9fceaa69037
Reviewed-on: http://review.couchbase.org/40081
Reviewed-by: Sundararaman Sridharan <sun

Fix to unit test failure: test_item_pager

Change-Id: I186e6d631ff6723fbb2833cff450e9fceaa69037
Reviewed-on: http://review.couchbase.org/40081
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

MB-11237: Add backoff stats to dcp and dcpagg stats

This stat will be incremented each time the consumer is unable
to process items due to memory usage being too high.

Change-Id

MB-11237: Add backoff stats to dcp and dcpagg stats

This stat will be incremented each time the consumer is unable
to process items due to memory usage being too high.

Change-Id: Ie79c4c57f8b0c364f6075f9ba1be776f5bcc490a
Reviewed-on: http://review.couchbase.org/40061
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...