Revert "MB-36249: Replace notification pipe FDs with libevent event_active()"

Numbers from daily performance runs (KV : Pillowfight, 20/80 R/W, 256B
binary items) shows that this actually _reduces_

Revert "MB-36249: Replace notification pipe FDs with libevent event_active()"

Numbers from daily performance runs (KV : Pillowfight, 20/80 R/W, 256B
binary items) shows that this actually _reduces_ performance by ~2%
(see build 4631 vs 4632).

A brief look at the libevent code _suggests_ this is due to
event_active() needing to acquire the per-event base mutex, which
wasn't the case with the previous implementation which just wrote to
an FD.

Reverting to restore previous performance.

This reverts commit 4e021f71f7bbc2475657135b374c5fa5b3a37726.

Change-Id: Iedf416eb79a81c9b83d23e0654be431ec4aa16ca
Reviewed-on: http://review.couchbase.org/116850
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-36249: Replace notification pipe FDs with libevent event_active()

Currently memcached uses a pair of sockets connected to each other to
perform notification between threads. Benchmarking shows th

MB-36249: Replace notification pipe FDs with libevent event_active()

Currently memcached uses a pair of sockets connected to each other to
perform notification between threads. Benchmarking shows this can be
costly - when running a pillowfight test with
durability_level=persist_majority, profiling shows 15% of the NonIO
thread runtime is spent sending notifications (via libc send()) to
front-end threads to inform them to wake up.

Libevent however already provides a mechanism to wake up an event
directly from a different thread via event_active(). This internally
uses the most efficient mechanism available on the platform - which
for Linux is eventfd(), available since 2.6.22 (2007).

Profling with using eventfd (via libevent's event_active()) reduces
the cost of notifying other threads from NonIO thread to 9% of total
runtime (down from 15%).

Change-Id: I8a1201cba521ce3759cc355be5845b389ae0b8ac
Reviewed-on: http://review.couchbase.org/116587
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Slow down the logging during shutdown

In MB-36467 we've got a problem deleting a bucket, but due to
ns_server giving up waiting for memcached we end up in a
situation where memcached enters a tight

Slow down the logging during shutdown

In MB-36467 we've got a problem deleting a bucket, but due to
ns_server giving up waiting for memcached we end up in a
situation where memcached enters a tight loop trying to shut
down the connection and log every time. Given that we cycle
the log files and ns_server only keeps 20 of them (~400MB)
we'll only have a few seconds left.

Change-Id: I5320947a11321b9b5dcf886bf8fb1bb948e2bc13
Reviewed-on: http://review.couchbase.org/116503
Tested-by: Trond Norbye <trond.norbye@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Settings: Avoid static initialization fiasco

When building on macOS with clang 8 and ThreadSanitizer enabled,
memcached_testapp crashes when destructing the Settings object:

libc++abi.dylib: te

Settings: Avoid static initialization fiasco

When building on macOS with clang 8 and ThreadSanitizer enabled,
memcached_testapp crashes when destructing the Settings object:

libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

With the following backtrace from where the system_error exception is thrown:

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x00007fff7b6411f6 libc++abi.dylib` __cxa_throw
frame #1: 0x00007fff7b6147af libc++.1.dylib` std::__1::__throw_system_error(int, char const*) + 77
frame #2: 0x00007fff7b606c93 libc++.1.dylib` std::__1::mutex::lock() + 29
frame #3: 0x00000001009c27ff memcached_testapp` folly::SharedMutexImpl<...>::annotateLazyCreate() [inlined] std::__1::unique_lock<std::__1::mutex>::unique_lock(__m=<unavailable>) + 191 at __mutex_base:131
frame #4: 0x00000001009c27f7 memcached_testapp` folly::SharedMutexImpl<...>::annotateLazyCreate() [inlined] std::__1::unique_lock<std::__1::mutex>::unique_lock(__m=<unavailable>) at __mutex_base:131
* frame #5: 0x00000001009c27f7 memcached_testapp` folly::SharedMutexImpl<...>::annotateLazyCreate() + 140 at SharedMutex.cpp:33
frame #6: 0x00000001009c276b memcached_testapp` folly::SharedMutexImpl<...>::annotateLazyCreate(this=0x0000000100c7ddc8) + 43 at SharedMutex.h:705
frame #7: 0x00000001009c191a memcached_testapp` folly::SharedMutexImpl<...>::~SharedMutexImpl() [inlined] folly::SharedMutexImpl<...>::annotateDestroy(this=<unavailable>) + 8 at SharedMutex.h:718
frame #8: 0x00000001009c1912 memcached_testapp` folly::SharedMutexImpl<...>::~SharedMutexImpl() + 24 at SharedMutex.h:324
frame #9: 0x00000001009c18fa memcached_testapp` folly::SharedMutexImpl<...>::~SharedMutexImpl(this=0x0000000100c7ddc8) + 26 at SharedMutex.h:300
frame #10: 0x000000010076b837 memcached_testapp` folly::Synchronized<...>::~Synchronized(this=0x0000000100c7ddb0) + 55 at Synchronized.h:484
frame #11: 0x000000010075d1d9 memcached_testapp` folly::Synchronized<...>::~Synchronized(this=0x0000000100c7ddb0) + 41 at Synchronized.h:484
frame #12: 0x000000010075e8fc memcached_testapp` Settings::~Settings(this=0x0000000100c7db10) + 76 at settings.h:51
frame #13: 0x000000010075c8b9 memcached_testapp` Settings::~Settings(this=0x0000000100c7db10) + 41 at settings.h:51
frame #14: 0x0000000102b7d5c1 libclang_rt.tsan_osx_dynamic.dylib` cxa_at_exit_wrapper(void*) + 33
frame #15: 0x00007fff7d72beed libsystem_c.dylib` __cxa_finalize_ranges + 351
frame #16: 0x00007fff7d72c1fe libsystem_c.dylib` exit + 55
frame #17: 0x00007fff7d67f01c libdyld.dylib` start + 8

The problem is at frame 5, where as part of the destruction of
SharedMutex (as used by a member variable of Settings) we are
attempting to acquire a mutex which has already been destructed (as it
is itself a function-local static) - i.e. we have encountered the
static initialization order fiasco :(

Address this by changing `settings` to no longer be a plain static
(global) variable, and instead be created on first use via a new
Settings::instance() static method.

Change-Id: I40bd44ed0ae32eb55271ce739fdf990d9869c32f
Reviewed-on: http://review.couchbase.org/113640
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Remove duplicate log entry for conn alloc failures

dispatch_new_connections added a "generic" log entry if
conn_new returned nullptr, but conn_new had already
logged the reason why it failed to crea

Remove duplicate log entry for conn alloc failures

dispatch_new_connections added a "generic" log entry if
conn_new returned nullptr, but conn_new had already
logged the reason why it failed to create the new
connection.

Change-Id: I54bcbfd28c4307f00f407c5fba80fc525281861e
Reviewed-on: http://review.couchbase.org/113629
Tested-by: Trond Norbye <trond.norbye@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Refactor: Make thread a reference in Connection

The connection object is _ALWAYS_ bound to a thread and it
does NEVER change so we should use a reference to make it
easier for the reader to know tha

Refactor: Make thread a reference in Connection

The connection object is _ALWAYS_ bound to a thread and it
does NEVER change so we should use a reference to make it
easier for the reader to know that.

Change-Id: I5b6f239777753ee3693eb0f00911fa61d0e2efe3
Reviewed-on: http://review.couchbase.org/110588
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Richard de Mellow <richard.demellow@couchbase.com>

show more ...

MB-22005: Move from Histogram<T> to data size HdrHistograms

Replace the usages of the template class Histogram<T> from
platform/histogram.h with specialised versions of HdrHistogram.
This will allow

MB-22005: Move from Histogram<T> to data size HdrHistograms

Replace the usages of the template class Histogram<T> from
platform/histogram.h with specialised versions of HdrHistogram.
This will allow us to display data with dynamic bucketing
what can be based on percentiles or a logarithmic scale. Unlike
Histogram<T> which uses fixed width buckets. Which makes it hard
to view if all the values fall into the same bucket. With
HdrHistogram we can display the information around the data,
making it easier to understand.

Refactor HdrMicroSecHistogram, to reduce it's memory foot print
by reducing its accuracy from 3 sig fig to 1 sig fig. Taking the
histograms memory foot print from 139472 bytes to 3152 bytes. Also
we've renamed it to Hdr1sfMicroSecHistogram, to help reflect this.

This refactor and the histograms added in this patch will take the
memory used by for an empty bucket to 22.2MB from 21.7MB at the point
before MB-22005 code was added.

Add code to get the memory foot print of structures that use
HdrHistograms. As just using sizeof(struct x) will not give an
accuracy value as HdrHistograms are allocated to the heap.

ep-engine_sizes output showing the relative sizes of histograms:
GIGANTOR 9223372036854775808
StoredValue 56
StoredValue with 15 byte key 72
Ordered Stored Value 72
Blob 12
value_t 8
HashTable 352
Item 104
VBucket 1392
VBucketMap 40
Stats 1640
CheckpointManager 384
Checkpoint 312
CheckpointConfig 32
Histogram<whatever> 1048
HistogramBin<size_t> 24
HistogramBin<int> 16
HistogramBin<microseconds> 24
MicrosecondHistogram 1048
EPStats 466080
FileStats 29056
KVStoreStats 99776
Histogram<size_t>{ExponentialGenerator<size_t>(1, 2), 50} 1688
HdrHistogram frequency histo 16592
Hdr1sfMicroSecHistogram 3152
Hdr2sfMicroSecHistogram 20688
HdrUint8Histogram 16592
Hdr1sfInt32Histogram 3920
HdrHistogram(0, std::numeric_limits<int32_t>::max(), 2) 25808
HdrHistogram(0, std::numeric_limits<int32_t>::max(), 1) 3792
HdrHistogram(0, std::numeric_limits<int64_t>::max(), 2) 58576
HdrHistogram(0, std::numeric_limits<int64_t>::max(), 1) 7888
IORequest 112
CouchRequest 240
PersistenceCallback 16
AtomicUnorderedMap<uint32_t, SingleThreadedRCPtr<Stream>> 480
ProbabilisticCounter<uint8_t> 24
DcpResponse 16
MutationResponse 32
queued_item 8

Histogram Ranges

Default Histo 1048
Commit Histo 1048
Hash table depth histo 3920

Change-Id: I6e5e0d299b47b9b46e33f86201b6f34f3a0f93dc
Reviewed-on: http://review.couchbase.org/107284
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-33971: System connections is bound to system port

There was a misunderstanding when the task to account for system
connections. I thought we wanted to account for all "couchbase
internal componen

MB-33971: System connections is bound to system port

There was a misunderstanding when the task to account for system
connections. I thought we wanted to account for all "couchbase
internal components", but ns_server have no control over the
connection usage for components like eventing, indexing, projector
query, fts etc, and we don't want to end up in a situation where
ns_server can't connect to the cluster because the pool of system
connections is consumed.

To work around this ns_server will mark an interface as "system"
and all connections towards that port is system connections.

Change-Id: I65384db1411475c7c76f7a7182931dbe9e0e680f
Reviewed-on: http://review.couchbase.org/108527
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-32687: Backport signalIfIdle refactor

signalIfIdle used to update the event mask in libevent to add
a write event to make the socket available. Libevent would then
pick up that socket next time i

MB-32687: Backport signalIfIdle refactor

signalIfIdle used to update the event mask in libevent to add
a write event to make the socket available. Libevent would then
pick up that socket next time it went down to the OS to do a poll
on the sockets. The problem is that when we try to update the
event mask for a socket libevent will try to acquire the mutex
used by the event base in order to do that. If the code was
called from another worker thread it would hold it's own lock
on the event base (and we're subject for a deadlock).

The new implementation of signalIfIdle push the connection
onto a list of connections to signal for the worker thread,
before it notifies the worker thread by writing to the
notification pipe for the worker thread. The callback for
the worker thread swaps out the list of connections, and
runs the state machine for each of the connections in there.

Change-Id: Ic0ba5de83b8f924a8b7457c65d0e91fbf7927f7d
Reviewed-on: http://review.couchbase.org/105714
Tested-by: Build Bot <build@couchbase.com>
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Cleanup: Remove OpenSSL < 1.1 code

Change-Id: I000889e8ce34aab537b2b4d477f20eed341e2a8b
Reviewed-on: http://review.couchbase.org/107609
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Wa

Cleanup: Remove OpenSSL < 1.1 code

Change-Id: I000889e8ce34aab537b2b4d477f20eed341e2a8b
Reviewed-on: http://review.couchbase.org/107609
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-33599: Allow dynamic reconfig of interfaces

Add support for adding / deleting or changing network
interface descriptions. To simplify the implementation
all reconfiguration happens in the dispatc

MB-33599: Allow dynamic reconfig of interfaces

Add support for adding / deleting or changing network
interface descriptions. To simplify the implementation
all reconfiguration happens in the dispatcher thread
and not as part of reloading the config file.

To avoid ending up in a situation where we no longer
can connect to memcached changing the interfaces
works by first enabling all new interface descriptions,
and if no error happens it'll shut down the ones
no longer in the configuration.

Change-Id: Id036b22bcfdc9780f1a653d9099d9750b354bdd6
Reviewed-on: http://review.couchbase.org/107291
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-22005 Move from TimingHistogram to HdrHistogram (recommit)

This is a recommit of commit 3aa986fb2a41e8f12480e899b6900a87cec28475
as it was reverted due to a performance regression, caused by cach

MB-22005 Move from TimingHistogram to HdrHistogram (recommit)

This is a recommit of commit 3aa986fb2a41e8f12480e899b6900a87cec28475
as it was reverted due to a performance regression, caused by cache
line contention from memory accesses in hdr_record_values() from
HdrHistogram_c.

This commit also contains an additional fix to remove a race
condition when creating a HdrMicroSecHistogram object in
Timings::get_or_create_timing_histogram. Where two threads could try
and create two HdrMicroSecHistogram objects. This would then cause
one to be destoryed while it was being created. Meaning it would
try and free its counts pointer even though it has not been
allocated yet.

Change-Id: Iccb5115f7bf92995fbfbffbfb3470a49b4a5a775
Reviewed-on: http://review.couchbase.org/106001
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchba

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-33226: Push trace from a dedicated thread

To avoid OpenTracing to potentially block the front-end
threads while the OpenTracing module performs the operations
we'll push the context to a "list" a

MB-33226: Push trace from a dedicated thread

To avoid OpenTracing to potentially block the front-end
threads while the OpenTracing module performs the operations
we'll push the context to a "list" and process those entries
in a dedicated thread.

NOTE: The code assumes that only a subset of the commands
requests OpenTracing otherwise we'll probably get lock
contention (and should migrate over to a "per thread"
list).

Change-Id: Ib82e07bbd77a3e1fea8e4368839319ce81bb51e9
Reviewed-on: http://review.couchbase.org/106116
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Include what you use

Include what you use, and remove the C linkage
in cbsasl/util.h (we're all C++)

(prepare to get rid of platform/platform.h)

Change-Id: I6be2912761a34318163f05a1e3be38cf0090932

Include what you use

Include what you use, and remove the C linkage
in cbsasl/util.h (we're all C++)

(prepare to get rid of platform/platform.h)

Change-Id: I6be2912761a34318163f05a1e3be38cf00909320
Reviewed-on: http://review.couchbase.org/105904
Reviewed-by: Daniel Owen <owend@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Revert "MB-22005 Move from TimingHistogram to HdrHistogram"

This reverts commit 3aa986fb2a41e8f12480e899b6900a87cec28475. Due to
performance regression with ephemeral buckets (MB-33273).

Change-Id

Revert "MB-22005 Move from TimingHistogram to HdrHistogram"

This reverts commit 3aa986fb2a41e8f12480e899b6900a87cec28475. Due to
performance regression with ephemeral buckets (MB-33273).

Change-Id: I959b2c261d224bf145f7a11b32c557d1fd1f585a
Reviewed-on: http://review.couchbase.org/105919
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-22005 Move from TimingHistogram to HdrHistogram

Move kv_engine to use HdrHistogram class, where we currently use
TimingHistogram class. This will allow the use of dynamic bucket
ranges and thus a

MB-22005 Move from TimingHistogram to HdrHistogram

Move kv_engine to use HdrHistogram class, where we currently use
TimingHistogram class. This will allow the use of dynamic bucket
ranges and thus allows us to iterate over the data in multiple
different ways.

This patch also changes mctimings to no longer print out buckets
of a fixed width and will now display bucket data based on
percentiles.

Change-Id: I238b326d5468a247b06ef790d88f90bf4720ae4c
Reviewed-on: http://review.couchbase.org/103400
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

OpenSSL: Mute deprecated warning when building on OpenSSL 1.1

Change-Id: I22642b7eb6c8cb2d60dced455fd9c55cf844f8f4
Reviewed-on: http://review.couchbase.org/105333
Reviewed-by: Dave Rigby <daver@couc

OpenSSL: Mute deprecated warning when building on OpenSSL 1.1

Change-Id: I22642b7eb6c8cb2d60dced455fd9c55cf844f8f4
Reviewed-on: http://review.couchbase.org/105333
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: migrate off cb_mutex/cond in core

Use std::mutex and std::condition_variable instead

Change-Id: Iace666ca47153769c6efd21c88c1d3563777b6d3
Reviewed-on: http://review.couchbase.org/105179
T

Refactor: migrate off cb_mutex/cond in core

Use std::mutex and std::condition_variable instead

Change-Id: Iace666ca47153769c6efd21c88c1d3563777b6d3
Reviewed-on: http://review.couchbase.org/105179
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-31402: Back off logging of stuck connections

Change logging of stuck connections during deletion to:

1. Don't log anything the first 30 seconds. Then
dump the state of all stuck connect

MB-31402: Back off logging of stuck connections

Change logging of stuck connections during deletion to:

1. Don't log anything the first 30 seconds. Then
dump the state of all stuck connections.
2. Don't og anything for the next 30 seconds. Then
dump the state of all stuck connections which
changed since the previous dump.
3. goto 2.

Change-Id: Ia0520ae28bb0f6b799f52c9e8902607c95d9485b
Reviewed-on: http://review.couchbase.org/103254
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

clang-tidy: Clean warnings in daemon/thread.cc

Change-Id: If9c6b6343511137ffe894a56eb854889b77081fe
Reviewed-on: http://review.couchbase.org/103324
Reviewed-by: Dave Rigby <daver@couchbase.com>
Test

clang-tidy: Clean warnings in daemon/thread.cc

Change-Id: If9c6b6343511137ffe894a56eb854889b77081fe
Reviewed-on: http://review.couchbase.org/103324
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-32497: Remove potential deadlock with signalIfIdle

signalIfIdle used to update the event mask in libevent to add
a write event to make the socket available. Libevent would then
pick up that socke

MB-32497: Remove potential deadlock with signalIfIdle

signalIfIdle used to update the event mask in libevent to add
a write event to make the socket available. Libevent would then
pick up that socket next time it went down to the OS to do a poll
on the sockets. The problem is that when we try to update the
event mask for a socket libevent will try to acquire the mutex
used by the event base in order to do that. If the code was
called from another worker thread it would hold it's own lock
on the event base (and we're subject for a deadlock).

The new implementation of signalIfIdle push the connection
onto a list of connections to signal for the worker thread,
before it notifies the worker thread by writing to the
notification pipe for the worker thread. The callback for
the worker thread swaps out the list of connections, and
runs the state machine for each of the connections in there.

Change-Id: Ic0ba5de83b8f924a8b7457c65d0e91fbf7927f7d
Reviewed-on: http://review.couchbase.org/103285
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Move new connection queue to standard libs

Use std::pair and std::vector instead of our own classes
with unique-ptrs etc.

Change-Id: Ic8a6676992925e76b4fbea642598432c6d9a62ad
Reviewed-on:

Refactor: Move new connection queue to standard libs

Use std::pair and std::vector instead of our own classes
with unique-ptrs etc.

Change-Id: Ic8a6676992925e76b4fbea642598432c6d9a62ad
Reviewed-on: http://review.couchbase.org/103328
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Refactor: Notification list is on a per-cookie base

When OOO is enabled a connection may have multiple cookies
which needs to be flagged.

Change-Id: I9daec331b37242db23994529e5785fb765f77055
Review

Refactor: Notification list is on a per-cookie base

When OOO is enabled a connection may have multiple cookies
which needs to be flagged.

Change-Id: I9daec331b37242db23994529e5785fb765f77055
Reviewed-on: http://review.couchbase.org/101011
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Retry notification of worker threads during shutdown

When moving over to bufferevents calling signalIfIdle
the actual connection may be run at a later time and
not necessarily immediate. This could

Retry notification of worker threads during shutdown

When moving over to bufferevents calling signalIfIdle
the actual connection may be run at a later time and
not necessarily immediate. This could cause the shutdown
to hang as we've notified the thread, but that notification
what swallowed as part of another "notify_io_complete"
routine causing us to bump back to libevent to wait for
the next event.

Loop and signal the notification pipe until the thread
stops

Change-Id: I170cb39fa91dc47bd3fbe0530a3474a8f5622857
Reviewed-on: http://review.couchbase.org/102719
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...