MB-37096: Use setupWindowsDebugCRTAssertHandling in main()s

Make use of the new function setupWindowsDebugCRTAssertHandling() in the
"main" main() functions in kv_engine.

This provides details at t

MB-37096: Use setupWindowsDebugCRTAssertHandling in main()s

Make use of the new function setupWindowsDebugCRTAssertHandling() in the
"main" main() functions in kv_engine.

This provides details at the console if any errors / aborts are reported
by the Debug CRT.

Note that for production builds (using RelWithDebInfo) this is a no-op.

Change-Id: I876670c3f2465954323c53f927d21575efb01735
Reviewed-on: http://review.couchbase.org/121354
Well-Formed: Build Bot <build@couchbase.com>
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-35604: Add ssl_cipher_suite for TLS>=1.3

OpenSSL use a separate API to configure ciphers for TLS 1.3.
For TLS < 1.3 you may use SSL_CTX_set_cipher_list which scans
the list of provided ciphers an

MB-35604: Add ssl_cipher_suite for TLS>=1.3

OpenSSL use a separate API to configure ciphers for TLS 1.3.
For TLS < 1.3 you may use SSL_CTX_set_cipher_list which scans
the list of provided ciphers and selects the ones it supports
and ignores unknown ciphers. The method is considered as
successful if at least one cipher is selected. For TLS 1.3 one
needs to use SSL_CTX_set_ciphersuites and it will _fail_ if
it encounters unknown ciphers.

To work around this ns_server will provide two list of chiphers:

"ssl_cipher_list" : {
"tls 1.2" : "list of ciphers to use for TLS1, 1.1 and 1.2",
"tls 1.3" : "ciphersuites to use for TLS 1.3"
}

If OpenSSL decides they want to change the API for 1.4 we're ready
for that by adding another field.

This patch also change the logic for what an empty string means.
Previously an empty string indicated that you wanted to use
whatever OpenSSL provided as default. With this patch an empty
string (or not present in the object) means: no ciphers.

Change-Id: If426ed8b02ecb7820851f795e2e2593fa9591a36
Reviewed-on: http://review.couchbase.org/113881
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-35410: Update datatype on wholedoc operation

Wholedoc ops (as part of a subdoc multimutation) can replace the entire
document - potentially with a new value which does not match the
current datat

MB-35410: Update datatype on wholedoc operation

Wholedoc ops (as part of a subdoc multimutation) can replace the entire
document - potentially with a new value which does not match the
current datatype. E.g., existing json document replaced with non-json
raw bytes.

The datatype should be updated in this case.

Change-Id: I481209ac5c29f6f60a4b0f0ec93a3d3b2acd09e5
Reviewed-on: http://review.couchbase.org/113067
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-34367 [SR]: Correctly propogate IO-complete status from subdoc mutation

If the bucket_store() step of a sub-doc update operation returns
EWOULDBLOCK and then the subsequent notify_IO_complete has

MB-34367 [SR]: Correctly propogate IO-complete status from subdoc mutation

If the bucket_store() step of a sub-doc update operation returns
EWOULDBLOCK and then the subsequent notify_IO_complete has a
non-success status (for example a SyncWrite which timed out and notify
returns sync_write_ambiguous), the non-zero status is ignored.

Change-Id: I22dc3e5ea7dadc48e5cffead3aac01466b5f236f
Reviewed-on: http://review.couchbase.org/109911
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

EwbEngine: Allow arbitrary statuses for Sequence mode

Expand the Sequence mode to allow an arbitrary sequence of status
codes to be injected - previously only the original engine status code
(bit cl

EwbEngine: Allow arbitrary statuses for Sequence mode

Expand the Sequence mode to allow an arbitrary sequence of status
codes to be injected - previously only the original engine status code
(bit cleared) or a single status code (bit set) could be injected.

This allows more advanced error injection to be performed by allowing
different injected codes for the same setup of ewouldblock_engine.

Additionally, for Sequence mode enhance EWOULDBLOCK injection - the
actual result of the IO operation can now be specified (as the status
code following EWOULDBLOCK in the sequence) instead of always
returning SUCCESS.

Change-Id: I61968fb186108d42130c91f43038e7dfc83a3134
Reviewed-on: http://review.couchbase.org/109910
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

Testapp: Use execute() instead of executeCommand()

Refactor out all use of executeCommand to make sure execute
is used everywhere (as it may be configured to deal with
ETMPFAIL)

Change-Id: I29a5a69

Testapp: Use execute() instead of executeCommand()

Refactor out all use of executeCommand to make sure execute
is used everywhere (as it may be configured to deal with
ETMPFAIL)

Change-Id: I29a5a69cd902c96175061bcde4b6cb14f6f16509
Reviewed-on: http://review.couchbase.org/109425
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Cleanup: remove use of snappy-c.h

This is provided by cbcompress

Change-Id: I34fbdef4dfc3ebf9069fc34a45d3592c0c373597
Reviewed-on: http://review.couchbase.org/108646
Tested-by: Build Bot <build@cou

Cleanup: remove use of snappy-c.h

This is provided by cbcompress

Change-Id: I34fbdef4dfc3ebf9069fc34a45d3592c0c373597
Reviewed-on: http://review.couchbase.org/108646
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-33971: System connections is bound to system port

There was a misunderstanding when the task to account for system
connections. I thought we wanted to account for all "couchbase
internal componen

MB-33971: System connections is bound to system port

There was a misunderstanding when the task to account for system
connections. I thought we wanted to account for all "couchbase
internal components", but ns_server have no control over the
connection usage for components like eventing, indexing, projector
query, fts etc, and we don't want to end up in a situation where
ns_server can't connect to the cluster because the pool of system
connections is consumed.

To work around this ns_server will mark an interface as "system"
and all connections towards that port is system connections.

Change-Id: I65384db1411475c7c76f7a7182931dbe9e0e680f
Reviewed-on: http://review.couchbase.org/108527
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...

MB-33699: Detect if we try to connect to port -1

From looking at the test failure it seems like we've got
an occurrence of port being set to -1.

Change the code to:

1. Detect that the portnumber

MB-33699: Detect if we try to connect to port -1

From looking at the test failure it seems like we've got
an occurrence of port being set to -1.

Change the code to:

1. Detect that the portnumber file don't contain an entry
with the port being set to -1

2. Whenever we try to connect that the port is set to
something else than -1.

Change-Id: Ie7b0ddc4b675c68a76fa7b7adc7336d32f510ef2
Reviewed-on: http://review.couchbase.org/107453
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>

show more ...

MB-33599: Allow dynamic reconfig of interfaces

Add support for adding / deleting or changing network
interface descriptions. To simplify the implementation
all reconfiguration happens in the dispatc

MB-33599: Allow dynamic reconfig of interfaces

Add support for adding / deleting or changing network
interface descriptions. To simplify the implementation
all reconfiguration happens in the dispatcher thread
and not as part of reloading the config file.

To avoid ending up in a situation where we no longer
can connect to memcached changing the interfaces
works by first enabling all new interface descriptions,
and if no error happens it'll shut down the ones
no longer in the configuration.

Change-Id: Id036b22bcfdc9780f1a653d9099d9750b354bdd6
Reviewed-on: http://review.couchbase.org/107291
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove unused method: generate_config

Change-Id: I60411ae82e4c31d5c0cd3abc84ebbbec12e64f45
Reviewed-on: http://review.couchbase.org/107349
Reviewed-by: Richard de Mellow <richard.demellow@couchbase.

Remove unused method: generate_config

Change-Id: I60411ae82e4c31d5c0cd3abc84ebbbec12e64f45
Reviewed-on: http://review.couchbase.org/107349
Reviewed-by: Richard de Mellow <richard.demellow@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

All ephemeral server ports must be tagged

This is a preparation for the upcoming refactor of interface
handling so that we may dynamically change all properties of
the interface description (includi

All ephemeral server ports must be tagged

This is a preparation for the upcoming refactor of interface
handling so that we may dynamically change all properties of
the interface description (including adding / removing
interfaces)

Change-Id: If2c64b2e01b2de0eb235f67181145fdfe4ee092c
Reviewed-on: http://review.couchbase.org/107281
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-30040: Update kv_engine to use Folly

Change-Id: Id2eeb0205ff3c1ddf4269876366613b4860c6c2a
Reviewed-on: http://review.couchbase.org/105225
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: T

MB-30040: Update kv_engine to use Folly

Change-Id: Id2eeb0205ff3c1ddf4269876366613b4860c6c2a
Reviewed-on: http://review.couchbase.org/105225
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Clang-format testapp.cc and testapp.h

Change-Id: I9c7965c0e76d9bac35a114882cbdbf9d70238e2b
Reviewed-on: http://review.couchbase.org/106198
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dav

Clang-format testapp.cc and testapp.h

Change-Id: I9c7965c0e76d9bac35a114882cbdbf9d70238e2b
Reviewed-on: http://review.couchbase.org/106198
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Include what you use

Include what you use, and remove the C linkage
in cbsasl/util.h (we're all C++)

(prepare to get rid of platform/platform.h)

Change-Id: I6be2912761a34318163f05a1e3be38cf0090932

Include what you use

Include what you use, and remove the C linkage
in cbsasl/util.h (we're all C++)

(prepare to get rid of platform/platform.h)

Change-Id: I6be2912761a34318163f05a1e3be38cf00909320
Reviewed-on: http://review.couchbase.org/105904
Reviewed-by: Daniel Owen <owend@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Remove most usages of cb_getpid()

We don't really need this function, the majority of the usages are
just to create unique file names.

Change-Id: Ieba43018fc247b0b4d7e8143a97ab61bc9f6c81b
Reviewed-

Remove most usages of cb_getpid()

We don't really need this function, the majority of the usages are
just to create unique file names.

Change-Id: Ieba43018fc247b0b4d7e8143a97ab61bc9f6c81b
Reviewed-on: http://review.couchbase.org/105464
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-32147: Interlock "last_modified" vattr test with persistence

The last_modified field can only be returned by KV after the first
flush, as that's when KV begins tracking the hlc_epoch

Change-Id:

MB-32147: Interlock "last_modified" vattr test with persistence

The last_modified field can only be returned by KV after the first
flush, as that's when KV begins tracking the hlc_epoch

Change-Id: If874d8c0597b7ae77d84f9c6f2b56a4fc26de352
Reviewed-on: http://review.couchbase.org/105377
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Refactor: Use the callback method for stats

And use nlohmann to parse the value. This means
that if we'll successfully add JSON values reported
in the value into the returned JSON object instead
of

Refactor: Use the callback method for stats

And use nlohmann to parse the value. This means
that if we'll successfully add JSON values reported
in the value into the returned JSON object instead
of returning a JSON object encoded as a string.

Old: { "foo" : "{\"bar\": true}" }
New: { "foo" : { "bar" : true" } }

Change-Id: If0c6971240a131a2685f6ca0012a94e3f961075f
Reviewed-on: http://review.couchbase.org/104253
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-32514: Remove use of cJSON in testapp

This patch replace the last bits of use of cJSON in
memcached_testapp with nlohmann::json

Change-Id: Id8deea2d25157094b2a0d4415548007db870819e
Reviewed-on:

MB-32514: Remove use of cJSON in testapp

This patch replace the last bits of use of cJSON in
memcached_testapp with nlohmann::json

Change-Id: Id8deea2d25157094b2a0d4415548007db870819e
Reviewed-on: http://review.couchbase.org/104517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>

show more ...

MB-30041: Migrate client_connection_map to nlohmann::json

Change-Id: I17beac8275c29c6a6c250465c744735ba95d524e
Reviewed-on: http://review.couchbase.org/104254
Tested-by: Build Bot <build@couchbase.c

MB-30041: Migrate client_connection_map to nlohmann::json

Change-Id: I17beac8275c29c6a6c250465c744735ba95d524e
Reviewed-on: http://review.couchbase.org/104254
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-30041: Complete client connection stats migration

Complete the migration of the client connection stats function
to nlohmann::json by renaming the "overload" to the old name.

Change-Id: Ia6dbb1a

MB-30041: Complete client connection stats migration

Complete the migration of the client connection stats function
to nlohmann::json by renaming the "overload" to the old name.

Change-Id: Ia6dbb1ab162946936ca70f2c89db55adf8f6a0de
Reviewed-on: http://review.couchbase.org/104148
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-30041: Migrate final usages of client connection stats

Migrate the final usages of the client connection stats function
to nlohmann::json

Change-Id: Ic73c6cd7be8a412d7bb610fe33040c2b6630dff3
Rev

MB-30041: Migrate final usages of client connection stats

Migrate the final usages of the client connection stats function
to nlohmann::json

Change-Id: Ic73c6cd7be8a412d7bb610fe33040c2b6630dff3
Reviewed-on: http://review.couchbase.org/104147
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

Require IPv4 and IPv6 ports for testapp

We want to ensure that we may connect to the server
and talk over IPv4 and IPv6 so we should make it a
fatal error if we fail to get one of them.

Change-Id:

Require IPv4 and IPv6 ports for testapp

We want to ensure that we may connect to the server
and talk over IPv4 and IPv6 so we should make it a
fatal error if we fail to get one of them.

Change-Id: I99a6e0b52c32c2de86da0de704732342ab0004f9
Reviewed-on: http://review.couchbase.org/104015
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-32704: Remove per-port setting of max connections

*Problem*

Memcached used to have a “per port” setting of the maximum number
of client which may be connected to the port. The motivation
behind

MB-32704: Remove per-port setting of max connections

*Problem*

Memcached used to have a “per port” setting of the maximum number
of client which may be connected to the port. The motivation
behind that was that we wanted to keep a pool of connections
available to make sure that ns_server could connect to the
system (via the 11209 port). Later on when we added support for
SSL we didn’t have time to look at the overall model, we just
copied the “per port” setting into the new SSL connection. This
leads into the following “problem” with the current
configuration:

I can have 5000 connections to 11209 (plain, ipv4/6)
I can have 30 000 connections to 11210 (plain, ipv4/6)
I can have 30 000 connections to 11207 (SSL, ipv4/6)

In a deployment which use a mix of SSL and plain clients one may
use 60k clients connected to the system (30k of each type), but
deployments which only use PLAIN connections may only use 30k
connections (trying to use 30 001 would fail, even if we don’t
use a single SSL connection).

*Solution*

The configuration file provided to memcached contains two new
toplevel keys in the configuration:

max_connections - A number containing the maximum number
of connections allowed to memcached
(65k if we use the values in the example
above)

system_connections - A number reserved for users authenticated
as system users. (5k if we use the values
in the example above)

memcached continues to accept all sockets until we reach
max_connections, at that time it'll accept the socket and
immediately close the socket.

To make sure that the normal client's don't occupy all of the
connections to the server, memcached performs checks in the
validation phase to determine if connections needs to be
closed by using the following logic:

1. If the connection is represent a system-internal user,
leave the connection alone and continue to execute the
command.

2. If we've exceeded the number of "user" connections (60k
in the example above) and used more than half of the system
connections the connection is disconnected if it is
authenticated or the commad being executed isn't one of
Hello, SaslListMech, SaslAuth or SaslStep.

Change-Id: I3aec178b48f45fa055efb465ca9cea59fd71a895
Reviewed-on: http://review.couchbase.org/103761
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove unused tag "management" for interface definitions

The support for setting the tag was never used by ns_server

Change-Id: I6b6888b9a3b79a3183beb2b55e16aeba7db99ed8
Reviewed-on: http://review.

Remove unused tag "management" for interface definitions

The support for setting the tag was never used by ns_server

Change-Id: I6b6888b9a3b79a3183beb2b55e16aeba7db99ed8
Reviewed-on: http://review.couchbase.org/103402
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...