MB-32688: Final log entries missing from crash

Fix a race condition in the shutdown of the logger where we
post a flush message to the async logger then abort the process
without waiting for the flu

MB-32688: Final log entries missing from crash

Fix a race condition in the shutdown of the logger where we
post a flush message to the async logger then abort the process
without waiting for the flush to complete.

Change-Id: I69f812d3726f44ccb374bfd109656d5261bb64a2
Reviewed-on: http://review.couchbase.org/103767
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor SpdloggerTest: Move logger::Config to member var

To allow further customization of the logger::Config parameters (apart
from cycle size & log level), move the Config object to a
SpdloggerTe

Refactor SpdloggerTest: Move logger::Config to member var

To allow further customization of the logger::Config parameters (apart
from cycle size & log level), move the Config object to a
SpdloggerTest member variable and remove the arguments to
setUpLogger(). Tests can now change the config member before calling
setUpLogger() if they want to customize anything.

Change-Id: I893eb613c13d49c6dd465ed6ff0d034ef949ec69
Reviewed-on: http://review.couchbase.org/103779
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-30552: Add test for correct log formatting of Vbid

Added a test to ensure that Vbid logs correctly, along with a fix
to make that actually happen. <spdlog/fmt/ostr.h> is needed to
output custom t

MB-30552: Add test for correct log formatting of Vbid

Added a test to ensure that Vbid logs correctly, along with a fix
to make that actually happen. <spdlog/fmt/ostr.h> is needed to
output custom types, however having this in vbucket.cc allows the
implicit conversion of Vbid to uint16_t to take precedence and
ignore the formatting. By including it in the appropriate
locations, this issue can be avoided. This also removes the need
to have a separate vbucket.cc file.

Change-Id: I7cff526218c8700f5e86cf95676e4ab67fc8bdbf
Reviewed-on: http://review.couchbase.org/99439
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Get rid of extra indirection in ServerLogIface

There isn't much value in calling a function which returns a struct
with another function pointer in the server log interface instead
of lett

Refactor: Get rid of extra indirection in ServerLogIface

There isn't much value in calling a function which returns a struct
with another function pointer in the server log interface instead
of letting the method return the actual pointer directly.

In addition move the small struct over to server_api.h which
contains the rest of the server API.

Change-Id: I11d88d1473782a8bd2cd8216d2de38b9da68f91a
Reviewed-on: http://review.couchbase.org/99324
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Remove leftover of old logger: EXTENSION_LOGGER_DESCRIPTOR

Change-Id: I698070a26b93f04bbc0774be9c82d5045a063a6a
Reviewed-on: http://review.couchbase.org/99315
Tested-by: Build Bot <build@couchbase.c

Remove leftover of old logger: EXTENSION_LOGGER_DESCRIPTOR

Change-Id: I698070a26b93f04bbc0774be9c82d5045a063a6a
Reviewed-on: http://review.couchbase.org/99315
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-30302: Spdlog style formatting for EP Engine

Modification of SERVER_LOG_API and EP Engine to allow for spdlog style
formatting in EP Engine. Requires passing a pointer to the spdlog logger
object

MB-30302: Spdlog style formatting for EP Engine

Modification of SERVER_LOG_API and EP Engine to allow for spdlog style
formatting in EP Engine. Requires passing a pointer to the spdlog logger
object to create a new logger to format the log messages and prepend with
the engine name before using the original spdlog logger to print them.

Change-Id: I3711229fba45002d07bf40e2c6cb7f796e91a7bf
Reviewed-on: http://review.couchbase.org/96605
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: use cb::io::loadFile instead of MemoryMappedFile

cb::io::loadFile is a helper function which performs the
mapping and copying the data to a string buffer (which
was how it was used in thes

Refactor: use cb::io::loadFile instead of MemoryMappedFile

cb::io::loadFile is a helper function which performs the
mapping and copying the data to a string buffer (which
was how it was used in these cases)

Change-Id: Ie8a067ef4855e6d9c0521a90776087aef4df4431
Reviewed-on: http://review.couchbase.org/96105
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...

Minimise config.h includes

Remove includes from config.h which arn't really
configuration-related, nor handling platform abstraction. If source
files need libevent, or socket functionality they shou

Minimise config.h includes

Remove includes from config.h which arn't really
configuration-related, nor handling platform abstraction. If source
files need libevent, or socket functionality they should include it
themselves.

Change-Id: Icabac2122d97519ab7586a78fbd7df9eaaceb37b
Reviewed-on: http://review.couchbase.org/94060
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-28735: Skip HandleOpenFileErrors under UBSan

This test fails under UBSan, when spdlog fails to open a new file (in
custom_rotating_file_sink::_sink_it):

common.h:139:9: runtime error: member

MB-28735: Skip HandleOpenFileErrors under UBSan

This test fails under UBSan, when spdlog fails to open a new file (in
custom_rotating_file_sink::_sink_it):

common.h:139:9: runtime error: member access within address <ADDR> which does not point to an object of type 'spdlog::spdlog_ex'
<ADDR>: note: object has invalid vptr

Examining <ADDR> in a debugger indicates a valid object, and I cannot
identify any issue here. Additinally, this code is only called in an
error path if we have run out of file descriptors (so likely Bad
Things will already be happing).

As such, suppress this warning by skipping the test under UBSan.

Change-Id: Ieebe7992bbd1a0c038c2db99536032e38e74423f
Reviewed-on: http://review.couchbase.org/91097
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-27986,MB-28225,MB-27882: Remove log dedupe and flush at 200ms

This patch removes the deduplication logic from the logger as
we want to make sure that we dump (and flush) our log "immediately"
to

MB-27986,MB-28225,MB-27882: Remove log dedupe and flush at 200ms

This patch removes the deduplication logic from the logger as
we want to make sure that we dump (and flush) our log "immediately"
to the files. All of the log is written in a dedicated thread
which use a buffered FILE stream. We don't want to flush this
stream for _every_ log message (as that is typically 100 bytes
or so) which would cause a significant increse of file IO.

Instead we want the logger to keep writing data to the FILE
pointer, but if no log message is provided within 200ms we'll
cause FILE stream to be flushed to disk. This means that we'll
try to fill up the FILE streams buffer issue fewer actual
filesystem calls, but at the same time we won't keep the data
in those buffers hanging around for a long time.

Change-Id: I1f320dc540d39d9890c8ba84e89b0ce1f203aaa1
Reviewed-on: http://review.couchbase.org/90146
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Simplify logger tests

Increase the file rotation size to 20MB for the "basic" tests
to avoid having to deal with multiple files. Create a separate
test batch for all of the tests which expects file

Simplify logger tests

Increase the file rotation size to 20MB for the "basic" tests
to avoid having to deal with multiple files. Create a separate
test batch for all of the tests which expects file rotation
with a 2k file rotation threshold.

Disable time-based flushing by setting sleeptime to 0

Dump the log contents if the deduplication tests fail (to ease
debugging from the CV machines)

Change-Id: Iec7f9ae2e0aabacf60008f140e71ee967cb3904c
Reviewed-on: http://review.couchbase.org/89320
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-27882: Tighten up logger tests

The deduplication tests was a bit flaky as they would potentially
fail under load as the logger could flush the data to the logfiles
due to the 1 sec threshold for

MB-27882: Tighten up logger tests

The deduplication tests was a bit flaky as they would potentially
fail under load as the logger could flush the data to the logfiles
due to the 1 sec threshold for the automatic flushing

Change-Id: If71307e68acd2f4ec2b43c4a3f75b4ff3a45bb8e
Reviewed-on: http://review.couchbase.org/89246
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Create unit test to verify that the logger formats correctly

We want to move the unit tests to use the spdlogging api directly.
Create a new unit tests which use the "old" API, and one which
use the

Create unit test to verify that the logger formats correctly

We want to move the unit tests to use the spdlogging api directly.
Create a new unit tests which use the "old" API, and one which
use the spdlogger API.

Change-Id: Ib49adbca8aa05e639fb27cb5f1cf67c860e88082
Reviewed-on: http://review.couchbase.org/89244
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Simplify helper function to search a file

Just memorymap the entire file and use std::search instead
of parsing the file line by line.

Change-Id: I94137cb84bb9e4368c11a58ffef83666a1b4e8dd
Reviewed-

Simplify helper function to search a file

Just memorymap the entire file and use std::search instead
of parsing the file line by line.

Change-Id: I94137cb84bb9e4368c11a58ffef83666a1b4e8dd
Reviewed-on: http://review.couchbase.org/89243
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...

Don't require the server API when creating logger

It's only used for setting the log level, and we may do that
later on if we really want to change it after we've created
the logger. (If verbosity i

Don't require the server API when creating logger

It's only used for setting the log level, and we may do that
later on if we really want to change it after we've created
the logger. (If verbosity is being used it is automatically
updated anyway).

Change-Id: Ided21e53e016980e04e3acb9d601de8dd282b708
Reviewed-on: http://review.couchbase.org/89169
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove unused part of the old logger API

Change-Id: I111500132370add975c23b6d69a1358816006fb0
Reviewed-on: http://review.couchbase.org/89067
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Te

Remove unused part of the old logger API

Change-Id: I111500132370add975c23b6d69a1358816006fb0
Reviewed-on: http://review.couchbase.org/89067
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Remove unused argument to shutdown

Change-Id: I42034b142e8b1e574db61d18e6368012afb6f090
Reviewed-on: http://review.couchbase.org/89064
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com

Refactor: Remove unused argument to shutdown

Change-Id: I42034b142e8b1e574db61d18e6368012afb6f090
Reviewed-on: http://review.couchbase.org/89064
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Remove support for extensions in configuration

It was only used by the stdin_listener. Drop that part of the
API, and make the stdin_listener a part of the core.

Change-Id: I32fa173814b18

Refactor: Remove support for extensions in configuration

It was only used by the stdin_listener. Drop that part of the
API, and make the stdin_listener a part of the core.

Change-Id: I32fa173814b18e0018dfa2979ded853ce4112076
Reviewed-on: http://review.couchbase.org/89020
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Use spdlogger for everything

Remove the old stderr and stdout loggers and use the spdlogger
everywhere (and remove the ability to install custom loggers).

Change-Id: Ibd5a5d6775970e33664341f2476dd2

Use spdlogger for everything

Remove the old stderr and stdout loggers and use the spdlogger
everywhere (and remove the ability to install custom loggers).

Change-Id: Ibd5a5d6775970e33664341f2476dd2dee709abff
Reviewed-on: http://review.couchbase.org/88781
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-25624[2]: Implement custom sink which deduplicates log messages

Plug dedupe_sink into spdlogger and remove redundant code.

Change-Id: I81b0487754a90e57a7135684adc262be54fdb90f
Reviewed-on: http:

MB-25624[2]: Implement custom sink which deduplicates log messages

Plug dedupe_sink into spdlogger and remove redundant code.

Change-Id: I81b0487754a90e57a7135684adc262be54fdb90f
Reviewed-on: http://review.couchbase.org/86828
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Add a method to fetch the logger

A number of cmake targets had to be updated to reference
the memcached_logger library to have the correct include
path being propagated.

Change-Id: I36e90

Refactor: Add a method to fetch the logger

A number of cmake targets had to be updated to reference
the memcached_logger library to have the correct include
path being propagated.

Change-Id: I36e905b57f86e714ec0b05c36497162b3b5804aa
Reviewed-on: http://review.couchbase.org/88365
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove (almost) unused DETAIL log level

The detail level was only used to print out when we moved to
closing state (or all state transitions in the state machinery).

It is always disabled in produc

Remove (almost) unused DETAIL log level

The detail level was only used to print out when we moved to
closing state (or all state transitions in the state machinery).

It is always disabled in production, and if enabled it'll generate
so much log data that it is pretty much useless ;)

Change-Id: Ie487674274bb270998afb99e105607a164717175
Reviewed-on: http://review.couchbase.org/88233
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-27665: Fix problems with EMFILE for spdlogger

Continue to log to the same file if we encounter problems opening
the next log file.

Change-Id: Ia35a5aa4f6f320704725ae1c987bd581d605a637
Reviewed-o

MB-27665: Fix problems with EMFILE for spdlogger

Continue to log to the same file if we encounter problems opening
the next log file.

Change-Id: Ia35a5aa4f6f320704725ae1c987bd581d605a637
Reviewed-on: http://review.couchbase.org/88134
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...