MB-37096: Debug CRT: Fix Windows Cmd Prompt formatting

Windows Command Prompt doesn't support ANSI formatting, so spdlog
console output just displays garbage at the start / end of each

MB-37096: Debug CRT: Fix Windows Cmd Prompt formatting

Windows Command Prompt doesn't support ANSI formatting, so spdlog
console output just displays garbage at the start / end of each
line.

Change to use the higher-level 'stderr_color_sink_mt' which selects the
correct sink based on the platform (ANSI or Windows console sink).

Change-Id: I377bc9051b8e491c59d507a0c026db8b453a3ede
Reviewed-on: http://review.couchbase.org/121818
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-33901: Fix incorrect uses of spdlog and tidy related code

Change-Id: Ife6cfadb94ebe60a18f9f4ff21bc0ae25b36716d
Reviewed-on: http://review.couchbase.org/110538
Reviewed-by: Ben Hud

MB-33901: Fix incorrect uses of spdlog and tidy related code

Change-Id: Ife6cfadb94ebe60a18f9f4ff21bc0ae25b36716d
Reviewed-on: http://review.couchbase.org/110538
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave

Remove config.h

Change-Id: I79eb8c762971255db9d36a5f6461a8a6d0f29249
Reviewed-on: http://review.couchbase.org/106517
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-32688: Final log entries missing from crash

Fix a race condition in the shutdown of the logger where we
post a flush message to the async logger then abort the process
without wai

MB-32688: Final log entries missing from crash

Fix a race condition in the shutdown of the logger where we
post a flush message to the async logger then abort the process
without waiting for the flush to complete.

Change-Id: I69f812d3726f44ccb374bfd109656d5261bb64a2
Reviewed-on: http://review.couchbase.org/103767
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-32061: Lower spdlog buffer size

Between spdlog version 0.14.0 and 1.1.0 the async pool buffer
was changed from being specified in MB to number of items. As we
were passing the sam

MB-32061: Lower spdlog buffer size

Between spdlog version 0.14.0 and 1.1.0 the async pool buffer
was changed from being specified in MB to number of items. As we
were passing the same default parameters we were constructing far
larger buffers than necessary (512MB - 2 million items vs 2MB by
default).

Update the default buffersize to 8192 items (2MB).

Set the default cyclesize in unit tests to the default (100MB).
Unit tests that rely on it already modify it beforehand.

Remove the CB_MAXIMIZE_LOGGER_BUFFER_SIZE environment variable.
Given that we now flush the async logger every second, 8192 items
should be plenty.

Change-Id: I8a0b88faa19df571aa768253aebac80c6108cf9b
Reviewed-on: http://review.couchbase.org/101953
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

show more ...

MB-30302: Register/Unregister BucketLoggers

Register BucketLoggers on creation in the spdlog registry
within the memcached_logger library. Unregister them on
destruction.

Le

MB-30302: Register/Unregister BucketLoggers

Register BucketLoggers on creation in the spdlog registry
within the memcached_logger library. Unregister them on
destruction.

Leverages changes made to spdlog/details/registry.h
wherein it now stores weak_ptr's to ensure that a BucketLogger
can be deleted when it is no longer needed.

This will allow us to change verbosity of all loggers at
runtime in a future patch. Loggers that set a custom verbosity
(views & fts) are unregistered on creation of the DcpProducer
object preventing their verbosity from being changed. This is
in keeping with previous behaviour and ensure we don't end up
in a situation where we cannot revert verbosity settings.

Change-Id: I118fed644418b538dcef54e46c123fef0be6ffc7
Reviewed-on: http://review.couchbase.org/100363
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-31636: Print correct timezone in log messages

(Backport of MB-31619 to vulcan branch).

As part of the migration to spdlog as our logger, the timezone offset
was incorrectly o

MB-31636: Print correct timezone in log messages

(Backport of MB-31619 to vulcan branch).

As part of the migration to spdlog as our logger, the timezone offset
was incorrectly omitted from the log timestamp - e.g.

2018-10-10T15:23:49.565757Z INFO Pushing new cluster config for bucket:[testB] revision:[4179]

Note the actual timestamp (15:23:49) /is/ in the local timezone (not
UTC); so this is potentially very misleading it's *not* a Zulu (UTC+0)
timestamp.

Compare this to ns_server which correctly prints the offset (-10:00 here):

[ns_server:debug,2018-10-11T01:16:47.682-10:00,ns_1@127.0.0.1:compaction_new_daemon<0.419.0>:compaction_new_daemon:process_compactors_exit:1352]Finished compaction iteration.

Fix by inserting the missing timezone offset formatter.

(cherry picked from commit d0c17cc8a803812c2d2a304479cc3a0b200c9aba)

Change-Id: I4a55396a2b73532b4cf0a2e05d0a5d0af942f45c
Reviewed-on: http://review.couchbase.org/100576
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Paolo Cocchi <paolo.cocchi@couchbase.com>

show more ...

MB-31619: Print correct timezone in log messages

As part of the migration to spdlog as our logger, the timezone offset
was incorrectly omitted from the log timestamp - e.g.

MB-31619: Print correct timezone in log messages

As part of the migration to spdlog as our logger, the timezone offset
was incorrectly omitted from the log timestamp - e.g.

2018-10-10T15:23:49.565757Z INFO Pushing new cluster config for bucket:[testB] revision:[4179]

Note the actual timestamp (15:23:49) /is/ in the local timezone (not
UTC); so this is potentially very misleading it's *not* a Zulu (UTC+0)
timestamp.

Compare this to ns_server which correctly prints the offset (-10:00 here):

[ns_server:debug,2018-10-11T01:16:47.682-10:00,ns_1@127.0.0.1:compaction_new_daemon<0.419.0>:compaction_new_daemon:process_compactors_exit:1352]Finished compaction iteration.

Fix by inserting the missing timezone offset formatter.

Change-Id: I4a55396a2b73532b4cf0a2e05d0a5d0af942f45c
Reviewed-on: http://review.couchbase.org/100506
Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Well-Formed: Build Bot <build@couchbase.com>

show more ...

MB-30798: Update spdlog to version 1.1

As we need to make changes to spdlog we should first update
our version of it to 1.1. Make the necessary changes to
our code to use the new spd

MB-30798: Update spdlog to version 1.1

As we need to make changes to spdlog we should first update
our version of it to 1.1. Make the necessary changes to
our code to use the new spdlog api.

Use the new buffer/size style string returned by fmtlib
where we perform custom formatting.

Change-Id: I616628d50008758e21a54003cf676c1eb3109cc9
Reviewed-on: http://review.couchbase.org/100224
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Optimise logger.h header

- Move logger::Config into it's own file (logger_config.h), changing
all users which just need the logger Config to #include
logger_config.h instead of l

Optimise logger.h header

- Move logger::Config into it's own file (logger_config.h), changing
all users which just need the logger Config to #include
logger_config.h instead of logger.h

- Minimise includes of logger.h; removing where no longer needed in
header files and moving instead to necessary .cc files.

Clean build times:

$ ninja clean
$ CCACHE_DISABLE=true time ninja kv_engine/all -j6

Before (6a3ec65a7):
254.75 real 1426.10 user 78.62 sys
user+sys: 1504

After (d8e692c45):
243.33 real 1366.29 user 76.51 sys
user+sys: 1442

Results in 4% speedup in user+sys build (62s reduction).

Change-Id: I81625a830c3bd7fb7bd0a62a98138334fc3887f5
Reviewed-on: http://review.couchbase.org/99954
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Refactor: Get rid of extra indirection in ServerLogIface

There isn't much value in calling a function which returns a struct
with another function pointer in the server log interface ins

Refactor: Get rid of extra indirection in ServerLogIface

There isn't much value in calling a function which returns a struct
with another function pointer in the server log interface instead
of letting the method return the actual pointer directly.

In addition move the small struct over to server_api.h which
contains the rest of the server API.

Change-Id: I11d88d1473782a8bd2cd8216d2de38b9da68f91a
Reviewed-on: http://review.couchbase.org/99324
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Remove the old log levels (currently unused)

Change-Id: I55358a9fc60750e60cf56c8e9321dbbb27d17876
Reviewed-on: http://review.couchbase.org/99320
Tested-by: Build Bot <build@couchbase

Remove the old log levels (currently unused)

Change-Id: I55358a9fc60750e60cf56c8e9321dbbb27d17876
Reviewed-on: http://review.couchbase.org/99320
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Remove leftover of old logger: EXTENSION_LOGGER_DESCRIPTOR

Change-Id: I698070a26b93f04bbc0774be9c82d5045a063a6a
Reviewed-on: http://review.couchbase.org/99315
Tested-by: Build Bot <b

Remove leftover of old logger: EXTENSION_LOGGER_DESCRIPTOR

Change-Id: I698070a26b93f04bbc0774be9c82d5045a063a6a
Reviewed-on: http://review.couchbase.org/99315
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Add missing newline at the end of the files

Change-Id: Ice8288295622cde9f1f9473e8419d0795c2e19ba
Reviewed-on: http://review.couchbase.org/97282
Tested-by: Build Bot <build@couchbase.

Add missing newline at the end of the files

Change-Id: Ice8288295622cde9f1f9473e8419d0795c2e19ba
Reviewed-on: http://review.couchbase.org/97282
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-30302: Spdlog style formatting for EP Engine

Modification of SERVER_LOG_API and EP Engine to allow for spdlog style
formatting in EP Engine. Requires passing a pointer to the spdlog l

MB-30302: Spdlog style formatting for EP Engine

Modification of SERVER_LOG_API and EP Engine to allow for spdlog style
formatting in EP Engine. Requires passing a pointer to the spdlog logger
object to create a new logger to format the log messages and prepend with
the engine name before using the original spdlog logger to print them.

Change-Id: I3711229fba45002d07bf40e2c6cb7f796e91a7bf
Reviewed-on: http://review.couchbase.org/96605
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

spdlogger: use sync logger for unit tests

When running in unit-test mode; use a synchronous spdlog logger - this
means that log messages are logger immediately and are seen in the
sa

spdlogger: use sync logger for unit tests

When running in unit-test mode; use a synchronous spdlog logger - this
means that log messages are logger immediately and are seen in the
same order as other events (for example stopping in GDB will show
up-to-date messages).

Change-Id: Ic4ce03a400492367f5683c611b39abb145953e34
Reviewed-on: http://review.couchbase.org/96777
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Add EXTENSION_LOG_TRACE level

Add a new log level 'trace' which is more detailed than the current
finest level 'debug'.

Change a couple of the existing DEBUG log messages which

Add EXTENSION_LOG_TRACE level

Add a new log level 'trace' which is more detailed than the current
finest level 'debug'.

Change a couple of the existing DEBUG log messages which are very
verbose to be at level TRACE.

Change-Id: Iad9c0a9e685828e54d594730b4c9f2d2d0f23d6e
Reviewed-on: http://review.couchbase.org/96261
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-29377: Change return signature of cb::logger::get()

Change the return signature of getting the logger object
to return a direct pointer to the object. This has a notable
affect on

MB-29377: Change return signature of cb::logger::get()

Change the return signature of getting the logger object
to return a direct pointer to the object. This has a notable
affect on the logger benchmark results, in the case where we are
A) getting the logger or B) logging at a level which is disabled.

Before change:

Run on (8 X 2300 MHz CPU s)
2018-04-25 15:37:36
----------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------------
LoggerBench_Blackhole/GetLogger/threads:1 13 ns 13 ns 50902427
LoggerBench_Blackhole/GetLogger/threads:16 38 ns 320 ns 2124144
LoggerBench/LogToLoggerWithDisabledLogLevel/threads:1 13 ns 13 ns 50811533
LoggerBench/LogToLoggerWithDisabledLogLevel/threads:16 54 ns 483 ns 1284736
LoggerBench/LogToLoggerWithEnabledLogLevel/threads:1 475 ns 473 ns 1485288
LoggerBench/LogToLoggerWithEnabledLogLevel/threads:16 606 ns 1065 ns 550688

After change:

Run on (8 X 2300 MHz CPU s)
2018-04-25 15:37:36
----------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------------
LoggerBench_Blackhole/GetLogger/threads:1 2 ns 2 ns 335836112
LoggerBench_Blackhole/GetLogger/threads:16 0 ns 3 ns 220889872
LoggerBench/LogToLoggerWithDisabledLogLevel/threads:1 2 ns 2 ns 334633624
LoggerBench/LogToLoggerWithDisabledLogLevel/threads:16 0 ns 4 ns 156165040
LoggerBench/LogToLoggerWithEnabledLogLevel/threads:1 472 ns 471 ns 1490142
LoggerBench/LogToLoggerWithEnabledLogLevel/threads:16 423 ns 1075 ns 590576

Change-Id: I82ebd5c1776ff0efad6b9316e1ff35a20887892f
Reviewed-on: http://review.couchbase.org/93286
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-29377: Improve spdlog usage within kv_engine

[1]: Add isInitialized method to cb::logger and use this in
auditd.cc instead of cb::logger::get
[2]: Replace all usages of cb::l

MB-29377: Improve spdlog usage within kv_engine

[1]: Add isInitialized method to cb::logger and use this in
auditd.cc instead of cb::logger::get
[2]: Replace all usages of cb::logger::get in daemon with
the LOG_* macros
[3]: Add default log level to the logger config object and
use this to set the log level during initialization

Change-Id: I3728ec786eef7729609f65a36a35267a3b7e2b78
Reviewed-on: http://review.couchbase.org/93201
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

spdlogger: Remove unused headers

This also avoids a Windows linker error when trying to simplify the
usage of phosphor:

spdlogger.cc.obj : error LNK2019: unresolved external

spdlogger: Remove unused headers

This also avoids a Windows linker error when trying to simplify the
usage of phosphor:

spdlogger.cc.obj : error LNK2019: unresolved external symbol "public: __cdecl phosphor::ChunkLock::ChunkLock ..."

Change-Id: I87163c6e9663910a31a8aab77040c43aeea388ff
Reviewed-on: http://review.couchbase.org/90744
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...

MB-28492: Allow running verbose mode from testapp tests

Due to the layered architecture of spdlogger where we build
up a chain of sinks where we send the log message, each of
these s

MB-28492: Allow running verbose mode from testapp tests

Due to the layered architecture of spdlogger where we build
up a chain of sinks where we send the log message, each of
these sinks may filter only a subset of the messages.

When running in production we don't want the console logger
to log below the warning level, as ns_server is the consumer
and it can't keep up with the amount of data we would otherwise
log. ns_server's solution to that is to "drop" the log, so
we'd rather just log at a rate it can consume.

Running in unit-test-mode we do however want to be able to
send more output to the console so we need to set the filter
level to trace (note that the filter set by the logger
on top would cause the events to be filtered out unless
verbosity is bumped)

Change-Id: I71e4bfce2fbb258cad9432a0ec90c012c39524bf
Reviewed-on: http://review.couchbase.org/90496
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-27986,MB-28225,MB-27882: Remove log dedupe and flush at 200ms

This patch removes the deduplication logic from the logger as
we want to make sure that we dump (and flush) our log "imme

MB-27986,MB-28225,MB-27882: Remove log dedupe and flush at 200ms

This patch removes the deduplication logic from the logger as
we want to make sure that we dump (and flush) our log "immediately"
to the files. All of the log is written in a dedicated thread
which use a buffered FILE stream. We don't want to flush this
stream for _every_ log message (as that is typically 100 bytes
or so) which would cause a significant increse of file IO.

Instead we want the logger to keep writing data to the FILE
pointer, but if no log message is provided within 200ms we'll
cause FILE stream to be flushed to disk. This means that we'll
try to fill up the FILE streams buffer issue fewer actual
filesystem calls, but at the same time we won't keep the data
in those buffers hanging around for a long time.

Change-Id: I1f320dc540d39d9890c8ba84e89b0ce1f203aaa1
Reviewed-on: http://review.couchbase.org/90146
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

It is up to the app to make sure they set up the logger

There is no point of running this test the entire lifefime
of memcached. It is up to the user of the library to make sure
that

It is up to the app to make sure they set up the logger

There is no point of running this test the entire lifefime
of memcached. It is up to the user of the library to make sure
that they create a logger before using it.

Change-Id: I21210c4308c349d252c0be86660e62c52990dba3
Reviewed-on: http://review.couchbase.org/89180
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>

show more ...

Remove create-lock in the logger

The lock is pretty useless as we don't support dynamic changes
to the logger.

If we want to use a lock for this we would need to improve the

Remove create-lock in the logger

The lock is pretty useless as we don't support dynamic changes
to the logger.

If we want to use a lock for this we would need to improve the
get() method so that it'll tries to aqcuire the lock if
it thinks it needs to create the blackhole logger and
rerun the check with the lock being held to avoid racing with
other threads creating the file logger.

This is just adding complexity to the code, as the "normal"
use of the logger in production is:

Create console logger
do stuff
read configuration
replace the console logger with the file logger
do stuff
shut down the file logger
exit

The blackhole logger is only used by our unit tests.

Change-Id: If11af8685f96939a7064aa9d516340997745a9f6
Reviewed-on: http://review.couchbase.org/89211
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

Don't require the server API when creating logger

It's only used for setting the log level, and we may do that
later on if we really want to change it after we've created
the logger.

Don't require the server API when creating logger

It's only used for setting the log level, and we may do that
later on if we really want to change it after we've created
the logger. (If verbosity is being used it is automatically
updated anyway).

Change-Id: Ided21e53e016980e04e3acb9d601de8dd282b708
Reviewed-on: http://review.couchbase.org/89169
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...