History log of /6.0.3/kv_engine/engines/ep/src/executorpool.h (Results 1 - 25 of 69)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3, v5.5.4, v5.5.5, v5.5.6, v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1
# d3d41268 14-May-2018 Dave Rigby <daver@couchbase.com>

MB-29675: Remove ExecutorThread taskLog & slowLog

ExecutorThread contains two ring buffers detailing the name, type and
runtime of:

a) The last 80 tasks executed.

b) th

MB-29675: Remove ExecutorThread taskLog & slowLog

ExecutorThread contains two ring buffers detailing the name, type and
runtime of:

a) The last 80 tasks executed.

b) the last 80 "slow" tasks (ones which exceed their expected
runtime).

However these logs are now of limited value:

- They only record a very small number of elements (particulary the
'taskLog'), the information is typically of little use by the time
it's captured in stats.

- The information is presented in a separate stats block (with a
different timestamp format), making it hard to correlate back to the
main memcached log (which is typically needed to actually make use
of the information).

- The cost of logging is non-zero - for each recorded log element,
there is one memory allocation for std::string, potentially an
additional allocation when appending the description to the task
name.

Now we have Phosphor tracing in KV-Engine, the taskLog/slowLog don't
really add anything - Phosphor provides ~400,000 events in the default
20MB trace buffer; and has an extremly low cost - no per-event memory
allocation.

As such; remove the taskLogs.

Change-Id: I2389b10ad3a1ad89c32db2037673f6c45107ebb3
Reviewed-on: http://review.couchbase.org/94162
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 37b6ac98 08-Feb-2018 Trond Norbye <trond.norbye@gmail.com>

Clean up memcached/extension.h

The file includes a number of other (now) obsolete header
files (in addition to that we had a weird intentation).

Fix the missing includes in all

Clean up memcached/extension.h

The file includes a number of other (now) obsolete header
files (in addition to that we had a weird intentation).

Fix the missing includes in all of the other files caused
by removal of these files.

Change-Id: I1cebe8a1d956c161e0d9f218e478c5d1e9663ff1
Reviewed-on: http://review.couchbase.org/89063
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v5.0.1, v5.1.0, v5.0.0
# d36551bc 06-Sep-2017 Eugen-Alexandru Virtan <eugen.virtan@couchbase.com>

MB-25884: Audit and remove, where possible, uses of array new[]

Replace several dynamic arrays with safer hadlers, such
as std::vector, std::string or std::unique_ptr. These prevent

MB-25884: Audit and remove, where possible, uses of array new[]

Replace several dynamic arrays with safer hadlers, such
as std::vector, std::string or std::unique_ptr. These prevent
memory leaks caused by array resources not being freed up, and
conform to Couchbase's and C++'s best practices.

Change-Id: Iad0686e3749120203d9636b0a2a2622ee7db514d
Reviewed-on: http://review.couchbase.org/83147
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


# bf84c61b 16-Jun-2017 Dave Rigby <daver@couchbase.com>

Remove extraneous log messages ep-engine

Based on looking at the content of memcached.log on a two-node system,
remove log messages which give little or no useful information, and
si

Remove extraneous log messages ep-engine

Based on looking at the content of memcached.log on a two-node system,
remove log messages which give little or no useful information, and
simplify other messages where possible.

Change-Id: Id4a128a5fd98e2ce13a5a270e789cfd0deb33853
Reviewed-on: http://review.couchbase.org/79613
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# ef22f9b0 25-May-2017 Dave Rigby <daver@couchbase.com>

Move ep-engine to engines/ep


# 74551d34 28-Apr-2017 Dave Rigby <daver@couchbase.com>

Remove unnecessary #include "tasks.h"

tasks.h is a relatively expensive header to include, as it contains
the definitions of various disjoint Task types, so in turn it includes
a lar

Remove unnecessary #include "tasks.h"

tasks.h is a relatively expensive header to include, as it contains
the definitions of various disjoint Task types, so in turn it includes
a large proportion of ep-engine headers.

However, it is unneessarily included in a large number of places. This
is probably at last in part due to the fact that globaltask.h didn't
used to be it's own header, and hence to get the GlobalTask definition
one needed to include tasks.h

Change-Id: I0971eab4808ce51d470fe061c13796d18b444234
Reviewed-on: http://review.couchbase.org/77498
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 68841d59 07-Apr-2017 James Harrison <00jamesh@gmail.com>

Streamline configuration.h

Move more out of configuration.h into configuration_impl.h as
configuration.h is large and included in a number of places.

Change-Id: I099ab7925337267

Streamline configuration.h

Move more out of configuration.h into configuration_impl.h as
configuration.h is large and included in a number of places.

Change-Id: I099ab79253372675f70d677c13e73a2d9c59d74f
Reviewed-on: http://review.couchbase.org/76475
Reviewed-by: David Haikney <david.haikney@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v4.6.2_ep, v4.6.2_mc, v4.6.1_ep
# 7dfe4a57 24-Feb-2017 James Harrison <00jamesh@gmail.com>

MB-22018: Add tasks stats to cbstats

Initial work on adding a "top" like tool showing tasks running and
sleeping in a cluster.

This can be viewed by running

cbstats

MB-22018: Add tasks stats to cbstats

Initial work on adding a "top" like tool showing tasks running and
sleeping in a cluster.

This can be viewed by running

cbstats host:port tasks

This shows the information about the tasks such as the state, next
runtime, description, type etc.

Future additions to the tool are intended to add features such as load
averages for task types (of the style shown by `uptime`) and percentage
utilisation over a period of time per thread.

Change-Id: I005a2589a35b4c4364ba46cf2b44b71061ba06b3
Reviewed-on: http://review.couchbase.org/74510
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 3ca9fd41 27-Mar-2017 James Harrison <00jamesh@gmail.com>

MB-22018: Simplify task scheduling

Tasks are now associated with a particular type, so the queue type no
longer needs to be specified when scheduling.

Change-Id: I53a4d76a6b0395

MB-22018: Simplify task scheduling

Tasks are now associated with a particular type, so the queue type no
longer needs to be specified when scheduling.

Change-Id: I53a4d76a6b0395957abd5391780aa1cf32a59444
Reviewed-on: http://review.couchbase.org/75776
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# b20b9600 06-Mar-2017 James Harrison <00jamesh@gmail.com>

MB-22041 [12/N]: Rename thread count parameters

rename

max_num_{readers,writers,auxio,nonio}

to

num_{reader,writer,auxio,nonio}_threads

in config and {cb,

MB-22041 [12/N]: Rename thread count parameters

rename

max_num_{readers,writers,auxio,nonio}

to

num_{reader,writer,auxio,nonio}_threads

in config and {cb,mc}epctl to better reflect that changing these
properties alters the number of threads. Backward compatability is
achieved by adding aliases to the previous names.

Change-Id: I727c61f836f1e2262f4710098d9aa0521a5a8e15
Reviewed-on: http://review.couchbase.org/74696
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# 0b95e08b 09-Feb-2017 Dave Rigby <daver@couchbase.com>

Cleanup #include usage in headers (2)

Further cleanup of unnecessary #includes in headers

Change-Id: I31c74ffc2ed477f66c0ff955e706b069d06b99a8
Reviewed-on: http://review.couchba

Cleanup #include usage in headers (2)

Further cleanup of unnecessary #includes in headers

Change-Id: I31c74ffc2ed477f66c0ff955e706b069d06b99a8
Reviewed-on: http://review.couchbase.org/73460
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


# e7ce1c4f 08-Feb-2017 James Harrison <00jamesh@gmail.com>

MB-22041 [3/N]: Allow dynamic changes to number of threads

Reader, Writer, AuxIO and NonIO threads can now be dynamically stopped
and started using the interface previously used to set t

MB-22041 [3/N]: Allow dynamic changes to number of threads

Reader, Writer, AuxIO and NonIO threads can now be dynamically stopped
and started using the interface previously used to set the maximum
number of these threads - i.e., setMaxReaders.

The behaviour is outwardly the same - the max controls how many tasks of
a given type can be run concurrently, but this can now be dynamically
tuned both upwards and downwards.

Change-Id: I8e10487e5b57466fbe299e7e043a90dd6c8d5aa8
Reviewed-on: http://review.couchbase.org/73304
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>

show more ...


# 9a55aa87 14-Dec-2016 WillGardner <willg@rdner.io>

Replace RingBuffer with cb::RingBuffer from platform

To reduce the amount of code we keep around. Also cb::RingBuffer
behaves more like an STL container.

Change-Id: Iba06562c9b4

Replace RingBuffer with cb::RingBuffer from platform

To reduce the amount of code we keep around. Also cb::RingBuffer
behaves more like an STL container.

Change-Id: Iba06562c9b409944310792469c8a1b1fc42afa77
Reviewed-on: http://review.couchbase.org/70960
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v4.6.0_ep, v4.5.1-MP1_mc, v4.6.0-DP_mc, v4.6.0-DP_ep, v4.5.1-MP1_ep, v4.1.2-MP2_mc
# 455fe779 08-Sep-2016 Dave Rigby <daver@couchbase.com>

MB-20834: Use get_available_cpu_count() for executorpool #threads calculation

Use the new Couchbase::get_available_cpu_count() when calculating how
many executor pool threads to create.

MB-20834: Use get_available_cpu_count() for executorpool #threads calculation

Use the new Couchbase::get_available_cpu_count() when calculating how
many executor pool threads to create. get_available_cpu_count()
accounts for the number of logical cores a process is allowed to run
on, which is the mechanism cgroups (and hence Docker) uses to limit
the CPUs available to a container.

This fixes the problem of us creating "too many" executorpool threads
under Docker containers which have the --cpuset-cpus option set.

Change-Id: I3e3b91eecc51aea298ae9aceb9e8c6d3f16b7612
Reviewed-on: http://review.couchbase.org/67493
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Well-Formed: buildbot <build@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


Revision tags: v4.5.1_mc, v4.6.0_mc, v4.1.2-MP1_ep, v3.1.6_ep
# 0009ef75 30-Jun-2016 Jim Walker <jim@couchbase.com>

MB-18453: Make task scheduling fairer

The MB identified that we can starve tasks by scheduling
a higher priority task via ExecutorPool::wake().

This occurs because ExecutorPool:

MB-18453: Make task scheduling fairer

The MB identified that we can starve tasks by scheduling
a higher priority task via ExecutorPool::wake().

This occurs because ExecutorPool::wake() pushes tasks
into the readyQueue enabling frequent wakes to trigger
the starvation bug.

The fix is to remove readyQueue.push from wake, so that we only
push to the readyQueue. The fetch side of scheduling only looks at
the futureQueue once the readyQueue is empty, thus the identified
starvation won't happen.

A unit-test demonstrates the fix using the single-threaded harness and
expects that two tasks of differing priorities get executed, rather
than the wake() starving the low-priority task.

This test drives:
- ExecutorPool::schedule
- ExecutorPool::reschedule
- ExecutorPool::wake

These are all the methods which can add tasks into the scheduler
queue.

The fetch side is also covered:
- ExecutorPool::fetchNextTask

This commit is an update to a previous commit that was reverted due
to performance issues. The original commit was reverted to minimise
disruption.

- original commit is e22c9ebeda1aac
- revert is 27cb1120e3e37

Change-Id: I70a4dcf7cd1c3a6f04548e9bbc3f95e24cdf50ad
Reviewed-on: http://review.couchbase.org/65929
Well-Formed: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


# 27cb1120 14-Jul-2016 Jim Walker <jim@couchbase.com>

Revert "MB-18453: Make task scheduling fairer"

When running in a >1 node cluster memcached CPU is running
very high. The original fix has introduced a problem which
needs further inv

Revert "MB-18453: Make task scheduling fairer"

When running in a >1 node cluster memcached CPU is running
very high. The original fix has introduced a problem which
needs further investigation (fetchTask is very very cpu hot).

This reverts commit e22c9ebeda1aac2fc8f4325cc39a93c3bcefffab.

Change-Id: If53a3a60692fbaaef4e54462f99284a8044cd899
Reviewed-on: http://review.couchbase.org/65780
Well-Formed: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


# b77fd62a 16-Jun-2016 Dave Rigby <daver@couchbase.com>

MB-20054: Backport ep-engine_unit_tests from watson to 3.0.x

In Watson we have created a set of 'unit' (i.e. class-level) tests for
ep-engine. To assist in backporting bug fixes, and spe

MB-20054: Backport ep-engine_unit_tests from watson to 3.0.x

In Watson we have created a set of 'unit' (i.e. class-level) tests for
ep-engine. To assist in backporting bug fixes, and specifically their
unit tests (to demonstrate they are correct), this patch backports the
test infrastructure itself.

Note these tests require GTest, so the CMake changes necessary for it
have also been included.

Tests are a backport from couchbase/watson as of commit feda304.
Modified to handle changes in APIs etc, and to remove tests
which fail on 3.0.x as we never chose to fix them in the 3.0.x
branch.

Change-Id: Iaaf59b0d8d6ba0a2211b630ba00fd837ca01614a
Reviewed-on: http://review.couchbase.org/64979
Well-Formed: buildbot <build@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...


# e22c9ebe 30-Jun-2016 Jim Walker <jim@couchbase.com>

MB-18453: Make task scheduling fairer

The MB identified that we can starve tasks by scheduling
a higher priority task via ExecutorPool::wake().

This occurs because ExecutorPool:

MB-18453: Make task scheduling fairer

The MB identified that we can starve tasks by scheduling
a higher priority task via ExecutorPool::wake().

This occurs because ExecutorPool::wake() pushes tasks
into the readyQueue enabling frequent wakes to trigger
the starvation bug.

The fix is to remove readyQueue.push from wake, so that we only
push to the readyQueue. The fetch side of scheduling only looks at
the futureQueue once the readyQueue is empty, thus the identified
starvation won't happen.

A unit-test demonstrates the fix using the single-threaded harness and
expects that two tasks of differing priorities get executed, rather
than the wake() starving the low-priority task.

This test drives:
- ExecutorPool::schedule
- ExecutorPool::reschedule
- ExecutorPool::wake

These are all the methods which can add tasks into the scheduler
queue.

The fetch side is also covered:
- ExecutorPool::fetchNextTask

Change-Id: Ie797a637ce4e7066e3155751ff467bc65d083646
Reviewed-on: http://review.couchbase.org/65385
Well-Formed: buildbot <build@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v4.5.0_mc, v4.5.0_ep, v4.1.1_ep, v3.1.5_ep, v4.1.1_mc
# 66bb41ad 04-Mar-2016 Dave Rigby <daver@couchbase.com>

Remove AtomicValue and Mutex defines

We already removed our own custom AtomicValue and Mutex classes when
we moved to C++11; replacing them with #defines to std::atomic and
std::mute

Remove AtomicValue and Mutex defines

We already removed our own custom AtomicValue and Mutex classes when
we moved to C++11; replacing them with #defines to std::atomic and
std::mutex respectively. This patch completes that changover, using
the underlying types directly so it's easier to see what is actually
being used.

Change-Id: If3f717f766727042cd81ae7b8c65f9a19934f2fa
Reviewed-on: http://review.couchbase.org/60925
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Will Gardner <will.gardner@couchbase.com>

show more ...


# c9a6de87 20-May-2016 Dave Rigby <daver@couchbase.com>

Regression / unit test for MB-19695

Regression / unit test for MB-19695. See MB for full background of the
problem.

Given the issue is triggered by an unfortunate Task execution

Regression / unit test for MB-19695

Regression / unit test for MB-19695. See MB for full background of the
problem.

Given the issue is triggered by an unfortunate Task execution order,
creating a unit test for it requires that we can explicitly control
when Tasks and scheduled and executed.

To do this a FakeExecutorPool has been created. It maintains the same
set of Tasks on taskqueues, but no ExecutorThreads exist. Tasks can be
manually fetched from the queues, and then manually run on a
FakeExecutorThread (just runs the Task on the main thread). This
allows unit tests to have complete control on when Tasks are woken up
and run.

Change-Id: I6ef6e051a633c799740aa948eb7531974f94a949
Reviewed-on: http://review.couchbase.org/64253
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...


# bd1c6438 01-Apr-2016 Dave Rigby <daver@couchbase.com>

MB-18940: Make ExecutorPool::get() thread-safe

ExecutorPool::get() has a similar problem as
MemoryTracker::getInstance() (although not as bad) - there is a
potential data-race which

MB-18940: Make ExecutorPool::get() thread-safe

ExecutorPool::get() has a similar problem as
MemoryTracker::getInstance() (although not as bad) - there is a
potential data-race which could result in multiple ExecutorPools being
created.

The issue is that the 'instance' pointer is not atomic - this means
that the compiler /could/ reorder the assignment and object creation
so the object is assigned /before/ it is created. Paraphrasing from a
Dr Dobbs article[1] about double-checked locking:

Singleton* Singleton::instance() {
if (pInstance == 0) {
Lock lock;
if (pInstance == 0) {
pInstance = // Step 3
operator new(sizeof(Singleton)); // Step 1
new (pInstance) Singleton; // Step 2
}
}
return pInstance;
}

... consider the following sequence of events:

* Thread A enters instance, performs the first test of pInstance,
acquires the lock, and executes the statement made up of Steps 1
and 3. It is then suspended. At this point, pInstance is not
null, but no Singleton object has yet been constructed in the
memory pInstance points to.

* Thread B enters instance, determines that pInstance is not null,
and returns it to instance's caller. The caller then
dereferences the pointer to access the Singleton that, oops, has
not yet been constructed.

Fix in the same way as MemoryTracker::getInstance() - make `instance`
be an atomic variable and perform the object creation using a
temporary variable; which ensures that the ordering is correct.

[1]: http://www.drdobbs.com/cpp/c-and-the-perils-of-double-checked-locki/184405726

Change-Id: Ic8b5c29e0c404e2a4ce0b1e62545776f97754acc
Reviewed-on: http://review.couchbase.org/62299
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>

show more ...


Revision tags: v3.1.4_ep, v3.1.4_mc, v3.1.5_mc, v3.1.3_ep, v4.1.0_ep
# e7754280 05-Nov-2015 Dave Rigby <daver@couchbase.com>

Simplify common.h: Remove unnecessary headers [3/3]

Remove all #includes which arn't actually needed by the file including
them (aka "include what you use").

Reduces build time

Simplify common.h: Remove unnecessary headers [3/3]

Remove all #includes which arn't actually needed by the file including
them (aka "include what you use").

Reduces build time down from before this patch series from:

31.82 real 222.13 user 16.09 sys

down to:

26.95 real 188.54 user 13.29 sys

giving a 15% speedup in (user+sys).

(Built using `make clean && CCACHE_DISABLE=1 time make -j8`)

Change-Id: Ia47f08487d20c94167d93de8dd6590f7703dd898
Reviewed-on: http://review.couchbase.org/56705
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


Revision tags: v3.1.2_ep, v4.1.0_mc
# e06c9f8a 06-Oct-2015 Dave Rigby <daver@couchbase.com>

Change GlobalTask::taskable from ptr to reference

GlobalTask::taskable is a pointer which is always set at object
construction and never changed. Therefore simplify code by converting

Change GlobalTask::taskable from ptr to reference

GlobalTask::taskable is a pointer which is always set at object
construction and never changed. Therefore simplify code by converting
to a reference - no longer need to perform any NULL checks on use.

Change-Id: I37299b171329c0c5b176026a167d22faefcf6270
Reviewed-on: http://review.couchbase.org/55828
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


Revision tags: v3.1.2_mc, v3.1.1_mc, v3.1.1_ep
# c366cba1 21-Aug-2015 abhinavdangeti <abhinav@couchbase.com>

MB-16155: [BP] MB-15374 Cancel all tasks if force flag set during destroy

Override the blockShutdown flag for tasks if force is set
during shutdown/destroy.

Reviewed-on: http://

MB-16155: [BP] MB-15374 Cancel all tasks if force flag set during destroy

Override the blockShutdown flag for tasks if force is set
during shutdown/destroy.

Reviewed-on: http://review.couchbase.org/54746

Change-Id: Ic95807a5d8781455a6d56c61ae053bda47dbf4de
Reviewed-on: http://review.couchbase.org/54828
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


# 42a540c9 21-Aug-2015 abhinavdangeti <abhinav@couchbase.com>

MB-15374: Cancel all tasks if force flag set during destroy

Override the blockShutdown flag for tasks if force is set
during shutdown/destroy.

Change-Id: Ic95807a5d8781455a6d56c

MB-15374: Cancel all tasks if force flag set during destroy

Override the blockShutdown flag for tasks if force is set
during shutdown/destroy.

Change-Id: Ic95807a5d8781455a6d56c61ae053bda47dbf4de
Reviewed-on: http://review.couchbase.org/54746
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


123