MB-28777: Provide a min_compression_ratio configuration parameter

If there isn't much difference between sizes of compressed and
uncompressed data, then decompressing the data every time

MB-28777: Provide a min_compression_ratio configuration parameter

If there isn't much difference between sizes of compressed and
uncompressed data, then decompressing the data every time for
a client would outweigh the cost of storing the data as
compressed as there is no perceivable benefit in space savings

Change-Id: Ic6d735213faa00dfc2331d577f4cee80b46e715e
Reviewed-on: http://review.couchbase.org/91769
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Remove engine feature information

The feature information was never used and is currently
generating warnings from gcc 7. Given that we don't use
it we should nuke it.

Chang

Remove engine feature information

The feature information was never used and is currently
generating warnings from gcc 7. Given that we don't use
it we should nuke it.

Change-Id: Ica99af5e9dda549b090b0a4033b0f577e08a5438
Reviewed-on: http://review.couchbase.org/88494
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-27543: [Restore] Decompress the value before storing in 'off' mode

This reverts commit abff81203272095e247926346c5c720b1210385e.
Restoring the commit as it wasn't the root cause of th

MB-27543: [Restore] Decompress the value before storing in 'off' mode

This reverts commit abff81203272095e247926346c5c720b1210385e.
Restoring the commit as it wasn't the root cause of the post commit
failures.

Change-Id: I7f8ad00cc4ddc2760140bcece8168caaef0e86b0
Reviewed-on: http://review.couchbase.org/88193
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>

show more ...

Revert "MB-27543: Decompress the value before storing in 'off' mode"

This reverts commit 268ecbd4b8715333475c833e12efee16a6d0e759. Commit
has resulted in post commit failures. Hence, rev

Revert "MB-27543: Decompress the value before storing in 'off' mode"

This reverts commit 268ecbd4b8715333475c833e12efee16a6d0e759. Commit
has resulted in post commit failures. Hence, reverting.

Change-Id: Ib5355170dd5e0e5aec0a39943f837bb0d90a09ea
Reviewed-on: http://review.couchbase.org/88185
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-27543: Decompress the value before storing in 'off' mode

If the bucket compression mode is 'off' and the value is
snappy compressed, then the value needs to be decompressed before

MB-27543: Decompress the value before storing in 'off' mode

If the bucket compression mode is 'off' and the value is
snappy compressed, then the value needs to be decompressed before
storing it in the hash table

Change-Id: I6c32215ea49aebbd0f71993bb4579ac275acdf6d
Reviewed-on: http://review.couchbase.org/87808
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Use gsl::not_null in engine API

This is an initial refactor. At some point we want to refactor
the cookie argument from a const void pointer to Cookie&

Change-Id: Ie7d

Refactor: Use gsl::not_null in engine API

This is an initial refactor. At some point we want to refactor
the cookie argument from a const void pointer to Cookie&

Change-Id: Ie7d5b15f5d19725c4da2e3cbf760ced319026c0b
Reviewed-on: http://review.couchbase.org/85960
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove unused "cookie" parameter in engine API

release(), get_item_info(), set_item_info() and set_item_cas()
all ignored the cookie being passed in.

Remove the parameter as we

Remove unused "cookie" parameter in engine API

release(), get_item_info(), set_item_info() and set_item_cas()
all ignored the cookie being passed in.

Remove the parameter as we don't use it!

Change-Id: I6c78d734172925d50b69471d95c9aaefc42f99ed
Reviewed-on: http://review.couchbase.org/85956
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Refactor: Use global_assoc directly

There is only a single assoc table used by all engines and it
makes the code easier to test with a mock client if we don't
require a real engine h

Refactor: Use global_assoc directly

There is only a single assoc table used by all engines and it
makes the code easier to test with a mock client if we don't
require a real engine handle to be passed to the assoc
submodule.

Change-Id: I4d728ca041aa633f5ba179e65d831bf535c028f8
Reviewed-on: http://review.couchbase.org/80560
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-25011: Allow buckets to disable/enable xattrs

With full-eviction buckets the set/replace path may see a performance
hit because the read/modify/write semantics required to support xat

MB-25011: Allow buckets to disable/enable xattrs

With full-eviction buckets the set/replace path may see a performance
hit because the read/modify/write semantics required to support xattrs
will see an increase in bg-fetches (where we first see if a key
exists to determine if the new value must inherit existing xattrs).

If a user has pre-spock full-eviction buckets this performance hit
could be an issue for them, if there application is as well pre-spock
then they have 0 xattrs and could get the performance back by
turning xattrs off.

This commit introduces for ep-engine a config switch and cbepctl
option to disable xattrs per bucket.

This works by the engine API gaining a new function "isXattrEnabled"
so that memcached can query the bucket for xattr yes/no.

With isXattrEnabled() returning false the following happens:

* SUBDOC commands with xattr paths will fail validation and see
PROTOCOL_BINARY_RESPONSE_NOT_SUPPORTED
* SET/REPLACE will short-cut the read/mod/write logic and perform only
a write (store_if with a alway true predicate).
* Any command with an xattr dataype will be invalid (e.g. set_w_meta)
* DCP will ignore request to open with xattrs, even replication, thus
if xattrs did exist in the database, they won't be seen on DCP.

Testing:

This patch extends testapp so that it can toggle xattrs using the
set_param command and then adjusts the expected return codes when
xattrs are being used. Some xattr specific tests are adjusted to cope
with when getXattr triggers connection closing (hence they now
reconnect for each mutate/replace)

This also means that default engine now accepts
set_param for xattr_enabled only, this was the simplest way to get
the test harness working for all modes without further churn.

Change-Id: I37c3663de36a9c0068101e0287f53f047b64c98b
Reviewed-on: http://review.couchbase.org/79817
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Add more docs to hash_item

Also release 1 byte of "internal flags" as we only use 3
bits out of the 16 bits we've reserved.

In addition I've sorted the variabels with the widest

Add more docs to hash_item

Also release 1 byte of "internal flags" as we only use 3
bits out of the 16 bits we've reserved.

In addition I've sorted the variabels with the widest
datatypes first (and added a comment in the end with the
number of bytes availabe in the struct).

Change-Id: Ia20967534223bb3fda32768947dbac557e1759a7
Reviewed-on: http://review.couchbase.org/71758
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Refactor: Remove function to get struct member hash_item->cas

Change-Id: I0e6abb870ce864115a0172f775d8a20c18443ae6
Reviewed-on: http://review.couchbase.org/71757
Tested-by: buildbot

Refactor: Remove function to get struct member hash_item->cas

Change-Id: I0e6abb870ce864115a0172f775d8a20c18443ae6
Reviewed-on: http://review.couchbase.org/71757
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Remove support for running without cas

(As part of the refactor we've done in the core by implementing
various commands by using CAS we cannot run without CAS anymore.
It has never b

Remove support for running without cas

(As part of the refactor we've done in the core by implementing
various commands by using CAS we cannot run without CAS anymore.
It has never been a public API to disable CAS so nuking it should
be safe)

Change-Id: I26a33805877e932c2274b3c5d019420c3fa82934
Reviewed-on: http://review.couchbase.org/71646
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>

show more ...

Add support for keeping deleted items in memcached buckets

We need this for unit testing of subdoc access to operate
on deleted items.

It is enabled by setting the configuration

Add support for keeping deleted items in memcached buckets

We need this for unit testing of subdoc access to operate
on deleted items.

It is enabled by setting the configuration parameter

keep_deleted=true

With that option enabled they're treated just as a normal
entry in the cache with respect to the LRU, but not
returned unless explicitly asked for.

Change-Id: I3dff83c200f96b48fa40a1aa0337419b0adc52a5
Reviewed-on: http://review.couchbase.org/70857
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Remove unused function item_get_clsid

Change-Id: I81a49b833f3a718ff8a5c15c5804be7e235fbb53
Reviewed-on: http://review.couchbase.org/70855
Tested-by: buildbot <build@couchbase.com>

Remove unused function item_get_clsid

Change-Id: I81a49b833f3a718ff8a5c15c5804be7e235fbb53
Reviewed-on: http://review.couchbase.org/70855
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-21763: Move final C modules to C++

Remove all of the C modules from memcached. Fairly simple and requires
some changes mostly for const and void* usage.

Change-Id: Ib2a741944

MB-21763: Move final C modules to C++

Remove all of the C modules from memcached. Fairly simple and requires
some changes mostly for const and void* usage.

Change-Id: Ib2a7419448448102fe1c6bd61d4b03596166ad1b
Reviewed-on: http://review.couchbase.org/70261
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...

MB-18786: Fix race condition

Initialize the engine handle before inserting into the list of
created engines.

Change-Id: Ib0ef7ccc42730af329a78e6e031f0fe254f40ab1
Reviewed-on

MB-18786: Fix race condition

Initialize the engine handle before inserting into the list of
created engines.

Change-Id: Ib0ef7ccc42730af329a78e6e031f0fe254f40ab1
Reviewed-on: http://review.couchbase.org/61766
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Will Gardner <will.gardner@couchbase.com>

show more ...

Ensure bucket destroy removes hash-table items belonging
to the destroyed bucket.

Change-Id: I03ac48ca96568af965a2942637af94e830ec0b53
Reviewed-on: http://review.couchbase.org/52532

Ensure bucket destroy removes hash-table items belonging
to the destroyed bucket.

Change-Id: I03ac48ca96568af965a2942637af94e830ec0b53
Reviewed-on: http://review.couchbase.org/52532
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...

Tynset - A multi-bucket default_engine

1. All buckets store their items in a single hash_table (assoc).
1.1 Each item key now has a bucket_id prepended to the key string.
1.2 The eng

Tynset - A multi-bucket default_engine

1. All buckets store their items in a single hash_table (assoc).
1.1 Each item key now has a bucket_id prepended to the key string.
1.2 The engine->assoc points to the single table and all code still
manipulates the table via the engine handle.

2. The bucket_id is managed by memcached 'core'
2.1 The bucket_id is a 32-bit unsigned integer which increments for
every bucket creation the process receives.
2.2 The bucket_id is not persisted, the same bucket after a restart is not
guaranteed the same bucket_id (this is subject to future change).
2.3 The bucket_id must not wrap (i.e. a bucket_id can't be reused)

3. Bucket deletion is performed by the item scrubber.
3.1 A single scrubber thread has a queue of engine's to scrub and delete.
3.2 Global cleanup/shutdown collects this single thread so that the DLL
can be safely unloaded.

Change-Id: Idc2bb6748dc75a227c97ad9c44445758b88015b0
Reviewed-on: http://review.couchbase.org/50122
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

Make memcached multi-bucket-aware

This is step one.

With this patch all engines will be loaded during startup
so that we know that all symbols will be resolved in the
engine

Make memcached multi-bucket-aware

This is step one.

With this patch all engines will be loaded during startup
so that we know that all symbols will be resolved in the
engine (so that we know that during startup rather than
after running for 3 weeks when someone decides to create
a bucket of type x). If you want to test out an experimental
engine you have to do that through the ewouldblock engine

Change-Id: Idb4ef895dacf2d916b07af4aebca79cba7fda6ba
Reviewed-on: http://review.couchbase.org/50053
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Standardise on using engine_loader functions and introduce destroy_engine.

This change primarily introduces a destroy_engine method so that
engines can clean up before unloading. Using e

Standardise on using engine_loader functions and introduce destroy_engine.

This change primarily introduces a destroy_engine method so that
engines can clean up before unloading. Using engine_loader as the
single API for loading ensures all engine define the method.

ep.so currently does not unload the malloc hooks and we segfault
if memory alloc/dealloc occurs post dlclose of ep.so. Adding the
destroy_engine path allows ep.so to clean up the hooks.

Change-Id: I4ead9f87388110078e495f951ae22cb06c089527
Reviewed-on: http://review.couchbase.org/49978
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

Remove unused tap and dcp code in default_engine

Change-Id: Ife41e5d0bd38d8721c234e01d42fb9eecf0b918c
Reviewed-on: http://review.couchbase.org/43174
Tested-by: buildbot <build@couchb

Remove unused tap and dcp code in default_engine

Change-Id: Ife41e5d0bd38d8721c234e01d42fb9eecf0b918c
Reviewed-on: http://review.couchbase.org/43174
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>

show more ...

Rename create_instance()

This allows for explicit calling the function to create an
instance instead of having to run dlopen.

Change-Id: I79910e29825e976b7fa889a43283a06c6f1f40b

Rename create_instance()

This allows for explicit calling the function to create an
instance instead of having to run dlopen.

Change-Id: I79910e29825e976b7fa889a43283a06c6f1f40bb
Reviewed-on: http://review.couchbase.org/41189
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-8578: Add stats uuid

Change-Id: I18bf15ad81b0911710220abaf19f89ed71598bc5
Reviewed-on: http://review.couchbase.org/28672
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
T

MB-8578: Add stats uuid

Change-Id: I18bf15ad81b0911710220abaf19f89ed71598bc5
Reviewed-on: http://review.couchbase.org/28672
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

Revamp build process

* Build using CMake (http://www.cmake.org)
* Link with tcmalloc instead of dlopen
* Build on Windows by using MSVC
* Platform library hiding threads and mute

Revamp build process

* Build using CMake (http://www.cmake.org)
* Link with tcmalloc instead of dlopen
* Build on Windows by using MSVC
* Platform library hiding threads and mutexes
* Dropped support for UDP
* Ensure that the test applications run on windows!

Change-Id: Iecf1aaa81e95564d3c0918a58d041e074e756ab0
Reviewed-on: http://review.couchbase.org/28671
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

Refactor: move memcached core to a separate directory