MB-37374: Producer sends user-xattrs within normal delete

Producer must send user-xattrs when IncludeDeleteUserXattrs::Yes.
Differently, in the next patch we will ensure that user-xattrs

MB-37374: Producer sends user-xattrs within normal delete

Producer must send user-xattrs when IncludeDeleteUserXattrs::Yes.
Differently, in the next patch we will ensure that user-xattrs are
pruned before streaming when IncludeDeleteUserXattrs::No.

Note: Sync Delete will be addressed in follow-up patches.

Change-Id: Ia5d858edda5684f057c156f161377795d56fd725
Reviewed-on: http://review.couchbase.org/c/kv_engine/+/126708
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-37680: Add support for sequential DCP Backfill

Add a new DCP_CONTROL option to allow DCP consumers to request that
backfills are scheduled in sequential order, instead of the default

MB-37680: Add support for sequential DCP Backfill

Add a new DCP_CONTROL option to allow DCP consumers to request that
backfills are scheduled in sequential order, instead of the default
round-robin order.

The new DCP_CONTROL message 'backfill_order' allows the DCP client to
choose from the following values:

* `round-robin` - vBuckets should be backfilled in round-robin order,
reading chunk of data from each in turn. This is the default value
(and the behaviour of DCP BackfillManager before this change).

* `sequential` - vBuckets should be backfilled sequentially - _all_
data from the first vBucket should be read from disk before
advancing to the next vBucket.

Change-Id: I15ae56a6c917960f0efb9ac2de9b0eec49745612
Reviewed-on: http://review.couchbase.org/c/kv_engine/+/128375
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-37680: MockActiveStream: default common arguments

Add default values for typical values used for seqnos, to reduce
boilerplate when using MockActiveStream.

Change-Id: I5197c9

MB-37680: MockActiveStream: default common arguments

Add default values for typical values used for seqnos, to reduce
boilerplate when using MockActiveStream.

Change-Id: I5197c976f4aca9c58e793166e66aacaccc9a653d
Reviewed-on: http://review.couchbase.org/c/kv_engine/+/128109
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Well-Formed: Build Bot <build@couchbase.com>

show more ...

MB-37468: Force notification of stream completing backfill

A stream completing a backfill may race with a stepping producer.
The stream may have just completed the backfill (scanned all

MB-37468: Force notification of stream completing backfill

A stream completing a backfill may race with a stepping producer.
The stream may have just completed the backfill (scanned all items
but not yet completed execution of the completeBackfill function)
whilst the producer steps and finds nothing in the readyQueue (already
processed all items). If the completing backfill then does not notify
the stream as the itemsReady flag has not yet been reset (by the
stepping producer) then the producer will remove the stream from its
VBReadyQueue and not process it again. Nothing else would notify
the stream causing us to fail to call endStream or transitionState.

Change-Id: Ib430ef78261b910616576572bf76cd97c1cf47be
Reviewed-on: http://review.couchbase.org/120292
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Well-Formed: Build Bot <build@couchbase.com>

show more ...

MB-37161: Check if SyncRepl enabled after DCP Control negoatiation complete

When processing a SnapshotMarker from a Producer which doesn't support
SyncReplication, we must set the HCS to

MB-37161: Check if SyncRepl enabled after DCP Control negoatiation complete

When processing a SnapshotMarker from a Producer which doesn't support
SyncReplication, we must set the HCS to zero. To determine if SyncRepl
is enabled, the PassiveStream checks the Consumer's SyncRepl
negotiation status in the PassiveStreeam ctor.

However, as MB-37204 highlighted, this incorrectly assumes that DCP
control negotiation has been completed by the time the PassiveStream
object is created. This isn't the case, as ns_server asynchronously
sends AddStream requests to the Consumer, potentially before the
Consumer has completed feature negotiation with the Producer. As a
result, PassiveStream objects can be created before SyncReplicaiton
has been negotiated, and PassiveStream::supportsSyncReplication is
incorrectly false.

Fix this by deferring setting PassiveStream::supportsSyncReplication
until acceptStream() - which can only be called /after/ DCP Control
has been completed - due to the Consumer step() order:

1. Negotiate DCP control features with Producer.
2. Send StreamRequest message(s) to Producer

Only after (2) are any StreamRequestResponse message received by the
Consumer, and it is those which result in acceptStream() being called.

Change-Id: Ib7256cd299b9c84ef1adf199e6b1c1b527f45edd
Reviewed-on: http://review.couchbase.org/119142
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-37205: Allow duplicate prepares on Disk snapshots

If a replica gets disconnected then it may have an oustanding prepare
when it reconnects to the active. The active node may have move

MB-37205: Allow duplicate prepares on Disk snapshots

If a replica gets disconnected then it may have an oustanding prepare
when it reconnects to the active. The active node may have moved on
and may send a disk snapshot with a different (newer) prepare for the
same key. This is valid and the newer prepare needs to overwrite the
older one.

To do this we currently have an AwaitingFirstSnapshotMarker state in
PassiveStream. We process the first snapshot marker and set a seqno
on the replica for which we allow prepares that already exist to be
overwritten by newer ones. This works fine, but we do not deal with
the case where the active node drops the cursor of the replica. If
this happens then the replica will not get disconnected, it will just
receive a disk snapshot in the middle of its stream. This disk snapshot
may have a prepare that should overwrite one that already exists on the
replica.

Remove the AwaitingFirstSnapshotMarker state and replace it with a
simple check that we are about to process a Disk snapshot when
considering allowing prepares to overwrite ones that already exist on
a replica.

Change-Id: I5b0d78e838151fe0c7db13120fc351bdf08066ea
Reviewed-on: http://review.couchbase.org/119097
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-37161: Set HCS for Disk Snapshots received from non-MH Producers

When a replica vBucket on a MH node is receiving a disk snapshot from
a pre-MH node, no high completed seqno will be s

MB-37161: Set HCS for Disk Snapshots received from non-MH Producers

When a replica vBucket on a MH node is receiving a disk snapshot from
a pre-MH node, no high completed seqno will be sent. This is expected
(given pre-MH knows nothing of HCS), but the replica should treat this
as a HCS of zero (i.e. it's not possible for anything to have been
committed if we don't yet have SyncWrite support).

As part of this, add an additonal check in
CheckpointManager::createSnapshot() that disk snapshots always have a
HCS set.

Change-Id: I37eb82091385dcf55abee47039411689e0d87a06
Reviewed-on: http://review.couchbase.org/119024
Reviewed-by: Daniel Owen <owend@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

[BP] MB-36802: Don't return estimate=0 for dcp-takeover stats before backfill

[Backport of MB-35594.]

The 'dcp-takeover' stats are used by ns_server to estimate how many
mutatio

[BP] MB-36802: Don't return estimate=0 for dcp-takeover stats before backfill

[Backport of MB-35594.]

The 'dcp-takeover' stats are used by ns_server to estimate how many
mutations are remaining on a DCP stream. However, the estimate value
is not updated until the backfill task has run once (and scanned the
disk file). As such, if 'dcp-takeover' stats are requested before that
first backfil task has run, then they can incorrectly report '0'
backfill items.

To address this, change backfillRemaining to be of type
boost::optional, initialized to an empty optional.

Only when the backfill scan has completed (when the number of items
remaining is determined) is the optional populated.

Then, when stats are requested use a new status value
"calculating-item-count" if the optional is empty (i.e. before scan).

Change-Id: Ia11dfe830ebd690bb40884594992acbcd21c104d
Reviewed-on: http://review.couchbase.org/118410
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-36765: Revert "MB-35308: Move HPS on non durable writes"

This reverts commit 1dcdf61e860ea8c18cb75144b049e0104ef31ef9.

Moving the HighPreparedSeqno (HPS) on non-durable writes in

MB-36765: Revert "MB-35308: Move HPS on non durable writes"

This reverts commit 1dcdf61e860ea8c18cb75144b049e0104ef31ef9.

Moving the HighPreparedSeqno (HPS) on non-durable writes introduced a
perf regression in non-durable workloads.

Reverting this patch restores the previous behaviour of moving the HPS
only when a Prepare becomes locally satisfied (e.g., a
PersistToMajority prepare is persisted).

ns_server will instead check both the HPS and HighSeqno to determine
which replica to promote as the HPS alone will no longer be sufficient.

This does reintroduce MB-36133, which was previously fixed as part of
the reverted patch. This will be resolved in a different way in a
following patch.

Change-Id: Iaabc514304997bbb39e406cfa159ed9e2ed5b859
Reviewed-on: http://review.couchbase.org/117869
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-35308: Move HPS on non durable writes

To allow ns_server to only check the HPS when deciding which replica
should be promoted, make the HPS move on non-durable writes too. They
wo

MB-35308: Move HPS on non durable writes

To allow ns_server to only check the HPS when deciding which replica
should be promoted, make the HPS move on non-durable writes too. They
would otherwise have to consider both the HPS and the high seqno when
deciding which replica ought to be promoted.

Change-Id: I851f079c07983b9a67ed87d7a8735abe8d4a779a
Reviewed-on: http://review.couchbase.org/116974
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Use NonNegativeCounter for ActiveStream::backfillRemaining

This counter _shouldn't_ go negative, so use the appropriate type to
implement it instead of manually checking for being > 0.

Use NonNegativeCounter for ActiveStream::backfillRemaining

This counter _shouldn't_ go negative, so use the appropriate type to
implement it instead of manually checking for being > 0.

Change-Id: I53acc53d960f28bafa17f251c892fabed94e807e
Reviewed-on: http://review.couchbase.org/114895
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-35594: Don't return estimate=0 for dcp-takeover stats before backfill

The 'dcp-takeover' stats are used by ns_server to estimate how many
mutations are remaining on a DCP stream. Howe

MB-35594: Don't return estimate=0 for dcp-takeover stats before backfill

The 'dcp-takeover' stats are used by ns_server to estimate how many
mutations are remaining on a DCP stream. However, the estimate value
is not updated until the backfill task has run once (and scanned the
disk file). As such, if 'dcp-takeover' stats are requested before that
first backfil task has run, then they can incorrectly report '0'
backfill items.

To address this, change backfillRemaining to be of type
boost::optional, initialized to an empty optional.

Only when the backfill scan has completed (when the number of items
remaining is determined) is the optional populated.

Then, when stats are requested use a new status value
"calculating-item-count" if the optional is empty (i.e. before scan).

Change-Id: Id7049a0c13a8aab429f137d2f4b293567e360638
Reviewed-on: http://review.couchbase.org/114894
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-35499: Only seqnoAck over an fully established stream

Only seqnoAck as a replica after we have had acknowledgement of the
stream by the producer for a given vbucket, otherwise the pro

MB-35499: Only seqnoAck over an fully established stream

Only seqnoAck as a replica after we have had acknowledgement of the
stream by the producer for a given vbucket, otherwise the producer may
not have context for why the seqnoAck is being sent and will log
an error that it does not have a StreamContainer for the vbucket. This
will also close the connection with the replica, due to the exception
being thrown in the packet execution runloop.

Update dcp_stream_test.cc to correctly update the DCP
stream's state so that we enable the ability to seqnoAck when testing
when seqnoAcks should be sent and add helper methods to perform
state transitions.

Also remove transition from StreamState::Reading to
StreamState::Pending as being valid for a PassiveStream as we never
do this and allows us to treat the Pending state as an unacknowledged
stream by the producer.

Change-Id: I67d43fa8d1cf2bbf3b7c0c14febf4ab8a21925d7
Reviewed-on: http://review.couchbase.org/113946
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Minimise includes of dcp/ headers

Change-Id: Ic9a737cfd50b8f665b52f90dc59f38b71eeaac99
Reviewed-on: http://review.couchbase.org/113439
Tested-by: Build Bot <build@couchbase.com>

Minimise includes of dcp/ headers

Change-Id: Ic9a737cfd50b8f665b52f90dc59f38b71eeaac99
Reviewed-on: http://review.couchbase.org/113439
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-35245: Return CheckpointType from getAllItemsForCursor

The current return type just returns a snapshot range. In a
future patch ActiveStream will need to be aware of the
Checkpoin

MB-35245: Return CheckpointType from getAllItemsForCursor

The current return type just returns a snapshot range. In a
future patch ActiveStream will need to be aware of the
CheckpointType from which the returned items came.

Change-Id: I11a3cb3c95edcc3cc8dcd2fbeae55162ae405789
Reviewed-on: http://review.couchbase.org/112468
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-34789: Send mutation instead of commit for backfilled commits

Send a DCP Mutation instead of a DCP Commit when sending a commit
with prepare seqno > requested stream start seqno if th

MB-34789: Send mutation instead of commit for backfilled commits

Send a DCP Mutation instead of a DCP Commit when sending a commit
with prepare seqno > requested stream start seqno if the commit
has been backfilled from disk. We need to do this because the
de-duplication of keys on disk can result in a commit being the
first seqno a replica will see. In this case we must send a mutation
because the consumer needs to know what to commit. Currently, the
code does this if the prepare seqno is less than the requested
start seqno. This does not solve the case where we de-dupe a prepare
within the disk snapshot though. Instead, send the mutation instead
of the commit if the replica did not receive the prepare before it
connected whilst we are backfilling from disk.

*** EXAMPLE ***

XXXX = Nothing sent/received

Active: P:k1 C:k1 M:k2 P:k1 C:k1 P:k1
Replica: P:k1 Disconnect XXXX M:k2 XXXX C:k1 P:k1

Replica disconnects and then streams from disk.

For the first commit the replica receives we must send a
mutation so that we can commit the correct value. The previous
code would send a commit as the prepare seqno was greater than the
first seqno sent.

Change-Id: I25bb8a11a920c035bfd25f656320642d4aba751e
Reviewed-on: http://review.couchbase.org/111435
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-34585: Handle snap end at abort and commit

Currently we do not call PassiveStream::handleSnapshotEnd at abort
and commit. This leads to us incorrectly not resetting the
receivingI

MB-34585: Handle snap end at abort and commit

Currently we do not call PassiveStream::handleSnapshotEnd at abort
and commit. This leads to us incorrectly not resetting the
receivingInitialDiskSnapshot/backfillingPhase flag in VBucket and
not acking for any prepares in the snapshot if the abort or commit
is the last item received.

Change-Id: I4a2098f9fa0646e1232c75dd4e1a9aff50875b95
Reviewed-on: http://review.couchbase.org/110651
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Minimise #include of dcp/response.h

Remove unnecessary #includes of dcp/response.h:

- replace with forward declarations

- Move some inlined code to .cc to remove need for r

Minimise #include of dcp/response.h

Remove unnecessary #includes of dcp/response.h:

- replace with forward declarations

- Move some inlined code to .cc to remove need for response.h in
header files.

Change-Id: Ie882294a686e9a27ebe2d42adf9b6c99434408f4
Reviewed-on: http://review.couchbase.org/107953
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-33157: Use folly AtomicHashMap in DcpProducer

This change reduces cache line contention due to the acquisition of
the RWLock in our AtomicUnorderedMap.

Perf stats (ops/s):

MB-33157: Use folly AtomicHashMap in DcpProducer

This change reduces cache line contention due to the acquisition of
the RWLock in our AtomicUnorderedMap.

Perf stats (ops/s):
Triton Cluster - Pillowfight, 20/80 R/W, 256B binary items. 2 Nodes.

Current master - ~1,850,000 ops/s
With this patch (map size 512) - 2,020,250 ops/s (10 run average)
With map size 128 - 1,915,520 ops/s (3 run average)
With map size 256 - 1,984,949 ops/s (3 run average)

Change-Id: I2159ff6a346c1a277b6199f1bce01fb5a9c86cec
Reviewed-on: http://review.couchbase.org/107434
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-33167 [SR]: DCP Producer sends DCP_ABORT

With this patch a queue_op::abort_sync_write item is streamed to a DCP
Consumer as a DCP_ABORT message.
Note that Abort items (as Prepare)

MB-33167 [SR]: DCP Producer sends DCP_ABORT

With this patch a queue_op::abort_sync_write item is streamed to a DCP
Consumer as a DCP_ABORT message.
Note that Abort items (as Prepare) are ignored for pre-MH replica
connections.

Note: Currently we send the Key in the DPC_ABORT payload. That is just
temporary and will be removed as soon as we have implemented the
DurabilityMonitor at Replica.

Change-Id: I265093bf130a633f2a58bf59f8da6c4eb8fe8d2b
Reviewed-on: http://review.couchbase.org/107089
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-32785 [SR]: Replica acks memory-seqno at Prepare received

It is enough that the Replica acks the memory-seqno at Prepare received
for achieving the consistency model that we aim to fo

MB-32785 [SR]: Replica acks memory-seqno at Prepare received

It is enough that the Replica acks the memory-seqno at Prepare received
for achieving the consistency model that we aim to for Durability
(consistency model at failure scenario, details in DesignDoc).

This logically reverts http://review.couchbase.org/#/c/103964, where we
enforced memory-seqno acks only at snapshot-end.

Change-Id: Ia72b307967a4296371132996644c7b0e2b5fdb26
Reviewed-on: http://review.couchbase.org/104452
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-31438: Send vbucket manifest UID in resuming stream requests

Update the DCP Consumer to create PassiveStreams with a vbucket
manifest uid json string that will be sent to the correspo

MB-31438: Send vbucket manifest UID in resuming stream requests

Update the DCP Consumer to create PassiveStreams with a vbucket
manifest uid json string that will be sent to the corresponding DCP
Producer. This string will be parsed by the pre-existing vbucket
filter code.

See MB-31326 for further details.

Change-Id: I02344f22431801f19356126cb912463db67de5af
Reviewed-on: http://review.couchbase.org/103580
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

De-inline DCP stream-related functions

De-inline a number of DCP stream / mock stream functions to minimise
the coupling between different files. Note all de-inlined functions
are ei

De-inline DCP stream-related functions

De-inline a number of DCP stream / mock stream functions to minimise
the coupling between different files. Note all de-inlined functions
are either test code (where we don't care about perfomance), or are
not (believed to be) hot code.

Where possible remove now-unnecessary header #includes or replace with
forward-decls.

Measuring the build times for this complete sequence of patches:

* 364c8a9c0 2019-01-25 | De-inline DCP stream-related functions (HEAD -> build_optimise) [Dave Rigby]
* 97096b098 2019-01-25 | Move VBucket-related declarations to vbucket_fwd.h [Dave Rigby]
* 5e58cfb02 2019-01-25 | Move PermittedVBStates to its own file [Dave Rigby]
* 94abc0620 2019-01-25 | De-inline MockDcpProducer methods [Dave Rigby]
* f2287529f 2019-01-25 | Move Collections json-related types to .cc [Dave Rigby]
* 38ba2f323 2019-01-25 | Move vbucket_state to its own file [Dave Rigby]
* 563074cd3 2019-01-25 | Minimise where statwriter.h is #include'd [Dave Rigby]
* cd60141f7 2019-01-25 | Replace <?stream> includes with <iosfwd> where possible (m/master, couchbase/master) [Dave Rigby]

Gives the following results - user+sys time for clean build (best of 3
runs) on MacBook Pro 2018 (2.9GHz Core i9).

Before: 2271s, after: 2153s
119s reduction in time.

Not measured, but I believe incremental builds should show a greater
speedup due to the reductions in header dependancies.

Change-Id: Ie5f46be1e34ce912ef9ca6312cadf13d98967ab4
Reviewed-on: http://review.couchbase.org/104093
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

Move PermittedVBStates to its own file

PermittedVBStates is a simple helper class to represent a set of
allowed vbucket states. It is used in a few places inside KV-Engine,
but is cu

Move PermittedVBStates to its own file

PermittedVBStates is a simple helper class to represent a set of
allowed vbucket states. It is used in a few places inside KV-Engine,
but is currently declared inside <memcached/vbucket.h>. This requires
that <memcached/vbucket.h> includes the headers required by
PermittedVBStates - one of which (<platform/bitset.h>) is relatively
heavyweight.

Additionally PermittedVBStates is only used within ep-engine, so it
probably shouldn't live in a memcached/ header.

Move PermittedVBStates into it's own header file and update various
uses to minise the necessary #includes.

Change-Id: I40b3b591204989fe133769d73d27137df333ebc9
Reviewed-on: http://review.couchbase.org/104089
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-32785: Replica acks memory-seqno at snapshot-end

Currently a Replica sends a SeqnoAck with the memory-seqno as soon as a
Prepare is received.
But, for achieving the consistency mo

MB-32785: Replica acks memory-seqno at snapshot-end

Currently a Replica sends a SeqnoAck with the memory-seqno as soon as a
Prepare is received.
But, for achieving the consistency model that we aim to (consistency at
failure scenario, details in DesignDoc) the Replica cannot acknowledge
mutations in the middle of a snapshot.

With this patch, a Replica sends an aggregated SeqnoAck only when it
receives the snapshot-end mutation, and only if the snapshot contains at
least one durable write.

Change-Id: I9d433940d5ffdc8facddec48604eae4a8ea31ad1
Reviewed-on: http://review.couchbase.org/103964
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...