Fix deadlock in checkpoint persistence command

We need to release the hpChkMutex before notifying memcached in
order to prevent a deadlock. The deadlock occurs when the flusher
trys

Fix deadlock in checkpoint persistence command

We need to release the hpChkMutex before notifying memcached in
order to prevent a deadlock. The deadlock occurs when the flusher
trys to notify memcached of checkpoint persistence at the same
time a memcached worker thread is trying to add a new checkpoint
persistence request.

Change-Id: Ida313f5b39ef0e063dee9882410cd0a19ce55292
Reviewed-on: http://review.couchbase.org/41363
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

Add id to the logs of checkpoint persistence

Add the id to the logs of checkpoint persistence so that we will have the
sequence number/checkpoint id in case a timeout occurs on checkpoin

Add id to the logs of checkpoint persistence

Add the id to the logs of checkpoint persistence so that we will have the
sequence number/checkpoint id in case a timeout occurs on checkpoint
persistence

Change-Id: Ifd3c70c9d40de43f0b3f1fdc1cf495ee0dc95d60
Reviewed-on: http://review.couchbase.org/40513
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>

show more ...

MB-11887: Properly send the snapshot start and end seqnos

Currently we always set the snapshot start and end seqno to be the
same as the start seqno. This is incorrect if we recently wer

MB-11887: Properly send the snapshot start and end seqnos

Currently we always set the snapshot start and end seqno to be the
same as the start seqno. This is incorrect if we recently were
disconnected and need to resume from a partial snapshot.

Change-Id: I7c9430e0b24ef01cc427cb03b518f4c41a55d875
Reviewed-on: http://review.couchbase.org/40276
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

fix compiler error in vbucket.cc metaDataDisk +=

Change-Id: I5b2992d3298d5b65f959f48cb078ed1c1f7a01b1
Reviewed-on: http://review.couchbase.org/40150
Reviewed-by: abhinav dangeti <abh

fix compiler error in vbucket.cc metaDataDisk +=

Change-Id: I5b2992d3298d5b65f959f48cb078ed1c1f7a01b1
Reviewed-on: http://review.couchbase.org/40150
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...

MB-11689 Added a stat ep_meta_data_disk

This gives approx disk space used to store item meta data by couchstore.

Change-Id: I7f1642279c9aca84b02c885fd621cfdf50c20e98
Reviewed-on

MB-11689 Added a stat ep_meta_data_disk

This gives approx disk space used to store item meta data by couchstore.

Change-Id: I7f1642279c9aca84b02c885fd621cfdf50c20e98
Reviewed-on: http://review.couchbase.org/40038
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-11629: Set engineSpecific to NULL when notify_failed_conns

Context: While notifying pending connections failed

Change-Id: I5e4085794a70a068f7a95735242b1e841cc6cf8f
Reviewed-o

MB-11629: Set engineSpecific to NULL when notify_failed_conns

Context: While notifying pending connections failed

Change-Id: I5e4085794a70a068f7a95735242b1e841cc6cf8f
Reviewed-on: http://review.couchbase.org/39982
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

Add vbucket-seqno stats to vbucket-details

We should include these stats in the vbucket details stats as well
since vbucket-seqno is really only mean to be used internally for
client

Add vbucket-seqno stats to vbucket-details

We should include these stats in the vbucket details stats as well
since vbucket-seqno is really only mean to be used internally for
clients that only want to get the bare minimum needed to make dcp
connections.

Change-Id: Ib95b80f8ec7afb37b146d6932e7c722da9d917c4
Reviewed-on: http://review.couchbase.org/39968
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-11026 Don't allow the underflow on the dirty queue size stat.

We will find the root cause of the underflow issue separately
soon.

Change-Id: Id5459fe3ac14f850159e932179113786

MB-11026 Don't allow the underflow on the dirty queue size stat.

We will find the root cause of the underflow issue separately
soon.

Change-Id: Id5459fe3ac14f850159e9321791137860f265f62
Reviewed-on: http://review.couchbase.org/37153
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...

MB-10476: Use cb_assert instead of assert

If compiled with -DNDEBUG assert is an empty macro

Change-Id: I266ddf8a8fe3f1a388a9d8ce3d21f28614c35315
Reviewed-on: http://review.couc

MB-10476: Use cb_assert instead of assert

If compiled with -DNDEBUG assert is an empty macro

Change-Id: I266ddf8a8fe3f1a388a9d8ce3d21f28614c35315
Reviewed-on: http://review.couchbase.org/34579
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Reduce the amount of headers that are imported in .h files

This change removes imports of certain header files from all
other header files and into .cc files. The problem with
import

Reduce the amount of headers that are imported in .h files

This change removes imports of certain header files from all
other header files and into .cc files. The problem with
importing a header in another header file is that any change
causes all headers used in that header file to be recompiled.
In some cases changing one header file causes us to have to
recompile everyhting. This change moves some of these headers
to only .cc files and should reduce compile time.

File no longer imported from other headers:

conflict_resolution.h
failover-table.h
flusher.h
mutation_log.h
tap-throttle.h
upr-consumer.h
upr-consumer.h

Change-Id: Idd44bcaac874e294c1085da46760badf37a1ea05
Reviewed-on: http://review.couchbase.org/34395
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Track the number of items per vbucket with the full eviction.

As the full eviction ejects an item from cache completely, we
need a new way of tracking the number of item per vbucket when

Track the number of items per vbucket with the full eviction.

As the full eviction ejects an item from cache completely, we
need a new way of tracking the number of item per vbucket when
the full eviction is chosen.

This change basically adds a new flag to the hash table item
(i.e., StoredValue) to indicate whether an item is newly added
to the cache or not, and then increments the item counter. Then,
the flusher figures out whether the counter should be decremented
because the item already exists on the DB file.

Change-Id: I492c03053238a4baa155eb76575893da323d26c4
Reviewed-on: http://review.couchbase.org/32985
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-9919: Provide DB data size and file size stats

This change provides DB data size and file size per vbucket
and their aggregated values through stats call.
stats "vbucket-details v

MB-9919: Provide DB data size and file size stats

This change provides DB data size and file size per vbucket
and their aggregated values through stats call.
stats "vbucket-details vbucket_id" returns the list of a given
vbucket's stats including its DB data size and file size.
For example, "vbucket-details 0" includes the following stats
for vbucket 0:
1) vb_0:db_data_size --> DB data size for vbucket 0
2) vb_0:db_file_size --> DB file size for vbucket 0

On the other hand, stats "all" includes the aggregated DB data
size and file size:
1) ep_db_data_size --> DB data size aggreaged across
all vbuckets.
2) ep_db_file_size --> DB file size aggregated across
all vbuckets.

A test case is added to ensure the value ticks up correctly.

Change-Id: Id8ae17ae7726f9a003a3245defc2ccf4ee89b77c
Reviewed-on: http://review.couchbase.org/32533
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

coding style fixes for gerrit compatibility

gerrit does not handle lines more than 80 characters well
Also coding style dictates fixed line sizes which are
mostly followed but left o

coding style fixes for gerrit compatibility

gerrit does not handle lines more than 80 characters well
Also coding style dictates fixed line sizes which are
mostly followed but left out in a few places..
- remove trailing whitespaces from all files
- limit lines to 80 chars
- try to ensure comments fit into one line so that they can
be traced back from an error log
Change-Id: Id8f1c34c3c7cb698a1b50f5f225b4516667f5e42
Reviewed-on: http://review.couchbase.org/31639
Tested-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

Move towards using C++11 atomics

Start by renamimg all use of Atomics that doesn't "break" by
switching to std::atomic to AtomicValue (when the transition
is done we'll rename everyt

Move towards using C++11 atomics

Start by renamimg all use of Atomics that doesn't "break" by
switching to std::atomic to AtomicValue (when the transition
is done we'll rename everything back).

Change-Id: I3f694856dab498c565806c9ac1ee5baf3bc07b17
Reviewed-on: http://review.couchbase.org/31333
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-9741 Replace usage of QueuedItem with Item class.

This is the first phase of reducing memory used by keys/metadata of items.
This checkin now replace QueuedItem with Item used by Chec

MB-9741 Replace usage of QueuedItem with Item class.

This is the first phase of reducing memory used by keys/metadata of items.
This checkin now replace QueuedItem with Item used by Checkpoing Manager
and prepares for using Item class in StoredValue. QueuedItem is removed
from the code base.

Change-Id: I6afecac51410f209155cf357f6a276c20670b026
Reviewed-on: http://review.couchbase.org/31050
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-9638 UPR persistence sequence number command

This command is similar to the Tap checkpoint persistence command.
But we use sequence number instead of checkpoint id for UPR.
Refact

MB-9638 UPR persistence sequence number command

This command is similar to the Tap checkpoint persistence command.
But we use sequence number instead of checkpoint id for UPR.
Refactored code to support both checkpoint id and seqno persistence.
The new memcached command opcode is CMD_SEQNO_PERSISTENCE 0xb7.
Added a stat "persistence_seqno".
Added a test case "upr_persistence_seqno".

Change-Id: Iec5db260e05cfa8889f253c33bb08209eb87ca2c
Reviewed-on: http://review.couchbase.org/30899
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-9161 Reset engine_specific to NULL when chk_persistence fails

If a checkpoint_persistence request fails due to the timeout,
we should then reset the engine_specific value to NULL, so

MB-9161 Reset engine_specific to NULL when chk_persistence fails

If a checkpoint_persistence request fails due to the timeout,
we should then reset the engine_specific value to NULL, so that
the next checkpoint persistence request should work correctly.

Change-Id: Iab0ab3c41fe39dc7060c3d094ec04cb9cfadcb73
Reviewed-on: http://review.couchbase.org/30677
Reviewed-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
Tested-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>

show more ...

Start migrate the API for our Atomics to look like std::atomic

This patch change the Atomic class CouchbaseAtomic, and change
the API so that:
get() => load()
set() => store(

Start migrate the API for our Atomics to look like std::atomic

This patch change the Atomic class CouchbaseAtomic, and change
the API so that:
get() => load()
set() => store()
cas() => compare_exchange_strong()

and move some of methods that used to be members as
template functions.

Change-Id: I48d873fbf04f40d446444232db44f565ee4fed54
Reviewed-on: http://review.couchbase.org/30135
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-9311: Notify all the pending connections on vbucket delete

If a given vbucket is deleted, we should notify it to all of its
pending connections, so that the connections are released

MB-9311: Notify all the pending connections on vbucket delete

If a given vbucket is deleted, we should notify it to all of its
pending connections, so that the connections are released
immediately and the error messages should be sent to the clients.

Change-Id: Ic230a803f0b26944f98e84ed4d21f8d834e3a561
Reviewed-on: http://review.couchbase.org/30353
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>

show more ...

MB-8988: Sequence numbers are assigned by the CheckpointManager

Change-Id: If3fdf08e1091ad36ea8bcd30b10cd3803db009ae
Reviewed-on: http://review.couchbase.org/29226
Reviewed-by: Chiyo

MB-8988: Sequence numbers are assigned by the CheckpointManager

Change-Id: If3fdf08e1091ad36ea8bcd30b10cd3803db009ae
Reviewed-on: http://review.couchbase.org/29226
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

CBD-1034 Support the full eviction as an optional eviction policy

When an item is evicted from the cache, its value is only
evicted from memory by default. To further reduce the memory

CBD-1034 Support the full eviction as an optional eviction policy

When an item is evicted from the cache, its value is only
evicted from memory by default. To further reduce the memory
overhead, this change supports the full eviction as an additional
option, which removes an item's key, metadata, and value from the
cache. The eviction policy can be chosen when a bucket is
instantiated.

If the full eviction option is chosen, this results in affecting
lots of engine APIs and also causes more disk IO overhead.
For example, CAS for a non-resident item will require a backgroud
fetch because it should know the item's CAS value.

Change-Id: I2edbe2b12527ef1c1fb31992cb04afbb7c078dac
Reviewed-on: http://review.couchbase.org/29155
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-9159 Batch read uses a key instead of a sequence number.

When an item is evicted from memory, we plan to evict all the
metadata fields including the item's sequence number in addiiton

MB-9159 Batch read uses a key instead of a sequence number.

When an item is evicted from memory, we plan to evict all the
metadata fields including the item's sequence number in addiiton
to the item's value. This new eviction approach requires a
batch read to use a key instead of a sequence number when it
reads items from disk.

This change adapts the batch read, so that it can use a key to
fetch an item's metadata or value from disk.

Change-Id: Ief14fd0ef4f42b316e75c6719349cdb1c63fac80
Reviewed-on: http://review.couchbase.org/29181
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-9434: prevent deadlock between flusher & stats

If the memcached stats thread which is holding the thread
lock tries to acquire the high priority checkpoint list
lock just to read

MB-9434: prevent deadlock between flusher & stats

If the memcached stats thread which is holding the thread
lock tries to acquire the high priority checkpoint list
lock just to read its size, it can deadlock on the
flusher thread which may be holding the list lock & trying
to notify the same memcached thread doing the stats

To avoid this deadlock maintain the size of the high
priority checkpoints in a separate variable and allow for
dirty read on the size

Change-Id: I6f0991c22deb2cc7762e5967496b99c3baed2c12
Reviewed-on: http://review.couchbase.org/29900
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...

MB-9098: Acquire lock to retrieve list size

- Need for synchronizing access to shared data
to avoid segmentation faults and possible
memory corruption.

- Backporting from ma

MB-9098: Acquire lock to retrieve list size

- Need for synchronizing access to shared data
to avoid segmentation faults and possible
memory corruption.

- Backporting from master:
http://review.couchbase.org/#/c/29477/

Change-Id: Id24770dbc72c082967959aee30c8e21b9556989e
Reviewed-on: http://review.couchbase.org/29687
Reviewed-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
Tested-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>

show more ...

MB-9098: Acquire lock to retrieve list size

- Need for synchronizing access to shared data
to avoid segmentation faults and possible
memory corruption.

Change-Id: If02dcf0d6

MB-9098: Acquire lock to retrieve list size

- Need for synchronizing access to shared data
to avoid segmentation faults and possible
memory corruption.

Change-Id: If02dcf0d61a397960f0e7bd0b738fe1bb5f93cbb
Reviewed-on: http://review.couchbase.org/29477
Reviewed-by: abhinav dangeti <abhinav.dangeti@gmail.com>
Tested-by: abhinav dangeti <abhinav.dangeti@gmail.com>

show more ...