History log of /6.0.3/kv_engine/engines/ep/src/taskqueue.h (Results 1 - 25 of 25)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
Revision tags: v7.0.2, v6.6.3, v7.0.1, v7.0.0, v6.6.2, v6.5.2, v6.6.1, v6.0.5, v6.6.0, v6.5.1, v6.0.4, v6.5.0, v6.0.3
# e7ed2862 27-Aug-2019 Dave Rigby <daver@couchbase.com>

MB-35649: Remove ExecutorThread::waketime

Since the fix for MB-35649, this member variable is no longer used for
thread scheduling (wakeup); sleep time is now directly calculated from

MB-35649: Remove ExecutorThread::waketime

Since the fix for MB-35649, this member variable is no longer used for
thread scheduling (wakeup); sleep time is now directly calculated from
the futureQueue contents.

As such remove it.

Change-Id: Iaff9f9e7d19f12416000dd3a9b31bb41d5e9e12d
Reviewed-on: http://review.couchbase.org/114231
Well-Formed: Build Bot <build@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: James Harrison <james.harrison@couchbase.com>

show more ...


Revision tags: v5.5.4, v5.5.5, v5.5.6, v6.0.1, v5.5.3, v6.0.0, v5.1.3, v5.5.2, v5.5.1, v5.1.2, v5.1.1, v5.0.1, v5.1.0, v5.0.0
# ef22f9b0 25-May-2017 Dave Rigby <daver@couchbase.com>

Move ep-engine to engines/ep


# 74551d34 28-Apr-2017 Dave Rigby <daver@couchbase.com>

Remove unnecessary #include "tasks.h"

tasks.h is a relatively expensive header to include, as it contains
the definitions of various disjoint Task types, so in turn it includes
a lar

Remove unnecessary #include "tasks.h"

tasks.h is a relatively expensive header to include, as it contains
the definitions of various disjoint Task types, so in turn it includes
a large proportion of ep-engine headers.

However, it is unneessarily included in a large number of places. This
is probably at last in part due to the fact that globaltask.h didn't
used to be it's own header, and hence to get the GlobalTask definition
one needed to include tasks.h

Change-Id: I0971eab4808ce51d470fe061c13796d18b444234
Reviewed-on: http://review.couchbase.org/77498
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...


Revision tags: v4.6.2_ep, v4.6.2_mc, v4.6.1_ep
# 9a55aa87 14-Dec-2016 WillGardner <willg@rdner.io>

Replace RingBuffer with cb::RingBuffer from platform

To reduce the amount of code we keep around. Also cb::RingBuffer
behaves more like an STL container.

Change-Id: Iba06562c9b4

Replace RingBuffer with cb::RingBuffer from platform

To reduce the amount of code we keep around. Also cb::RingBuffer
behaves more like an STL container.

Change-Id: Iba06562c9b409944310792469c8a1b1fc42afa77
Reviewed-on: http://review.couchbase.org/70960
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...


Revision tags: v4.6.0_ep
# d8577c54 15-Nov-2016 Daniel Owen <owend@couchbase.com>

MB-20079: Use std::chrono::steady_clock (ProcessClock)

Change task scheduling to use ProcessClock which is not
affected by changes to wall clock time.

Change-Id: I2fc9688abb782f

MB-20079: Use std::chrono::steady_clock (ProcessClock)

Change task scheduling to use ProcessClock which is not
affected by changes to wall clock time.

Change-Id: I2fc9688abb782fe2c9e80efb6da840be3643d4a5
Reviewed-on: http://review.couchbase.org/69899
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

show more ...


Revision tags: v4.5.1-MP1_mc, v4.6.0-DP_mc, v4.6.0-DP_ep, v4.5.1-MP1_ep, v4.1.2-MP2_mc, v4.5.1_mc
# 63e4ba7a 27-Aug-2016 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-20735: Fix incorrect waketime for executor threads

When a thread is done with a task from a particular queue,
it calls doneWork() which was resetting its current Queue type
result

MB-20735: Fix incorrect waketime for executor threads

When a thread is done with a task from a particular queue,
it calls doneWork() which was resetting its current Queue type
resulting in subsequent reschedule to pick infinity as
earliest thread waketime and end up sleeping longer when the
readyQueue is empty.
Simplify the reschedule logic as part of this fix.
+Add unit test for validation.

Change-Id: Ifdad1d21e624982460046393d4c377247da90735
Reviewed-on: http://review.couchbase.org/67112
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


Revision tags: v4.6.0_mc, v4.1.2-MP1_ep, v3.1.6_ep
# 24c408e6 30-Jun-2016 Jim Walker <jim@couchbase.com>

MB-20061: Improved futureQueue heap property fix

MB-9986 found that we break the heap property of the underlying
container of the std::priority_queue used in scheduling. This is
beca

MB-20061: Improved futureQueue heap property fix

MB-9986 found that we break the heap property of the underlying
container of the std::priority_queue used in scheduling. This is
because we use the wakeTime to sort the queue, but then later
change the value of a task which is still in the queue, thus
order is broken. MB-9986 emptied the futureQueue and rebuilt it.

In this fix we utilise the C++ standard which shows that we can
access the container of the priority_queue. Thus we can sub-class
it and create a method that allows us to update a tasks time and
maintain the heap order.

Change-Id: I238d36ea684d59ef06326183fa1f16c04f8d29ad
Reviewed-on: http://review.couchbase.org/65394
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...


Revision tags: v4.5.0_mc, v4.5.0_ep, v4.1.1_ep, v3.1.5_ep, v4.1.1_mc, v3.1.4_ep, v3.1.4_mc, v3.1.5_mc, v3.1.3_ep, v4.1.0_ep, v3.1.2_ep, v4.1.0_mc, v3.1.2_mc, v3.1.1_mc, v3.1.1_ep, v4.0.0_ep, v4.0.0_mc, v3.1.0_ep, v3.1.0_mc, v3.1.6_mc, v3.0.2-MP2_mc, v3.0.2_ep, v3.0.2_mc
# c8b050ee 12-Nov-2014 Dave Rigby <daver@couchbase.com>

MB-19248: Fix race in TaskQueue.{ready,future,pending}Queue access

Fix race as identified by ThreadSanitizer:

WARNING: ThreadSanitizer: data race (pid=4243)
Read of si

MB-19248: Fix race in TaskQueue.{ready,future,pending}Queue access

Fix race as identified by ThreadSanitizer:

WARNING: ThreadSanitizer: data race (pid=4243)
Read of size 8 at 0x7d04000fde60 by main thread (mutexes: write M1367):
#0 std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >::operator++() /usr/include/c++/4.8/bits/stl_list.h:235 (ep.so+0x00000013a129)
#1 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::__distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::input_iterator_tag) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:82 (ep.so+0x000000138b67)
#2 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:118 (ep.so+0x000000136b63)
#3 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::size() const /usr/include/c++/4.8/bits/stl_list.h:874 (ep.so+0x000000135538)
#4 ExecutorPool::doTaskQStat(EventuallyPersistentEngine*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/executorpool.cc:654 (ep.so+0x0000001331b6)
#5 EventuallyPersistentEngine::doWorkloadStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4198 (ep.so+0x000000112ed3)
#6 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4454 (ep.so+0x000000114cb4)
#7 EvpGetStats ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
#8 mock_get_stats memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
#9 test_workload_stats ep-engine/tests/ep_testsuite.cc:7094 (ep_testsuite.so+0x00000004e931)
#10 execute_test memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc)
#11 main memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)

Previous write of size 8 at 0x7d04000fde60 by thread T5 (mutexes: write M45):
#0 RCValue::_rc_decref() const ep-engine/src/atomic.h:293 (ep.so+0x000000096797)
#1 __gnu_cxx::new_allocator<std::_List_node<SingleThreadedRCPtr<GlobalTask> > >::allocate(unsigned long, void const*) /usr/include/c++/4.8/ext/new_allocator.h:104 (ep.so+0x00000017b42b)
#2 std::_List_base<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_get_node() /usr/include/c++/4.8/bits/stl_list.h:334 (ep.so+0x00000017b0dc)
#3 std::_List_node<SingleThreadedRCPtr<GlobalTask> >* std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_create_node<SingleThreadedRCPtr<GlobalTask> const&>(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:502 (ep.so+0x00000017a25a)
#4 void std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_insert<SingleThreadedRCPtr<GlobalTask> const&>(std::_List_iterator<SingleThreadedRCPtr<GlobalTask> >, SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1561 (ep.so+0x000000178e7b)
#5 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::push_back(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1016 (ep.so+0x00000017817b)
#6 TaskQueue::_fetchNextTask(ExecutorThread&, bool) ep-engine/src/taskqueue.cc:126 (ep.so+0x000000176932)
#7 TaskQueue::fetchNextTask(ExecutorThread&, bool) ep-engine/src/taskqueue.cc:142 (ep.so+0x000000176acf)
#8 ExecutorPool::_nextTask(ExecutorThread&, unsigned char) ep-engine/src/executorpool.cc:214 (ep.so+0x000000130707)
#9 ExecutorPool::nextTask(ExecutorThread&, unsigned char) ep-engine/src/executorpool.cc:229 (ep.so+0x0000001307a5)
#10 ExecutorThread::run() ep-engine/src/executorthread.cc:78 (ep.so+0x000000149da0)
#11 launch_executor_thread ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a)
#12 platform_thread_wrap platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
#13 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)

Fix by adding new helper methods which will return the size of the
various queues (while holding the TaskQueue's mutex).

Change-Id: If5e8d357e45803d78c4ba6ed1475e6e1a90e1c89
Reviewed-on: http://review.couchbase.org/62969
Well-Formed: buildbot <build@couchbase.com>
Reviewed-by: Will Gardner <will.gardner@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

show more ...


# 5dbbb7c2 23-Jul-2015 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-19223: Switch to hrtime from timeval in Global Thread Pool

This has small improvements in memory and cpu usage.
Also fixes several ThreadSanitizer races from unit tests - for example:

MB-19223: Switch to hrtime from timeval in Global Thread Pool

This has small improvements in memory and cpu usage.
Also fixes several ThreadSanitizer races from unit tests - for example:

WARNING: ThreadSanitizer: data race (pid=21672)
Write of size 8 at 0x7d140000e7b8 by main thread (mutexes: write M14972, write M14985):
#0 memcpy <null> (engine_testapp+0x000000453040)
#1 TaskQueue::_wake(SingleThreadedRCPtr<GlobalTask>&) /home/daver/repos/couchbase/server/ep-engine/src/taskqueue.cc:255 (ep.so+0x0000002577f6)
#2 TaskQueue::wake(SingleThreadedRCPtr<GlobalTask>&) /home/daver/repos/couchbase/server/ep-engine/src/taskqueue.cc:282 (ep.so+0x000000257c73)
#3 ExecutorPool::_wake(unsigned long) /home/daver/repos/couchbase/server/ep-engine/src/executorpool.cc:320 (ep.so+0x0000001acc76)
#4 ExecutorPool::wake(unsigned long) /home/daver/repos/couchbase/server/ep-engine/src/executorpool.cc:328 (ep.so+0x0000001ace13)
#5 Flusher::wait() /home/daver/repos/couchbase/server/ep-engine/src/flusher.cc:41 (ep.so+0x0000001cc4ff)
#6 EventuallyPersistentStore::stopFlusher() /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:402 (ep.so+0x0000000d54d5)
#7 ~EventuallyPersistentStore /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:364 (ep.so+0x0000000d49cb)
#8 ~EventuallyPersistentEngine /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:5778 (ep.so+0x000000161043)
#9 EvpDestroy(engine_interface*, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:143 (ep.so+0x000000135efa)
#10 mock_destroy /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:61 (engine_testapp+0x0000004bb9d6)
#11 destroy_engine /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:998 (engine_testapp+0x0000004bb646)
#12 execute_test /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1048 (engine_testapp+0x0000004baa11)
#13 main /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000004b8861)

Previous read of size 8 at 0x7d140000e7b8 by thread T14:
#0 ExecutorThread::run() /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:106 (ep.so+0x0000001e3488)
#1 launch_executor_thread(void*) /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2a5a)
#2 platform_thread_wrap /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000035dc)

Change-Id: I78fdddb832251fc062058c04f75f8d22c4c2f68d
Reviewed-on: http://review.couchbase.org/62912
Well-Formed: buildbot <build@couchbase.com>
Reviewed-by: Will Gardner <will.gardner@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


# b9334207 30-Nov-2015 Dave Rigby <daver@couchbase.com>

Mutex modernization [2/2]: Use std::condition_variable for SyncObject

Change Mutex to a typedef to std::mutex, and use
std::condition_variable for the implementation of SyncObject.

Mutex modernization [2/2]: Use std::condition_variable for SyncObject

Change Mutex to a typedef to std::mutex, and use
std::condition_variable for the implementation of SyncObject.

Note these two changes are mutually dependent hence being done in the
same patch.

Change-Id: Ife74822e9c4e04efefff446e964a952b672a1f91
Reviewed-on: http://review.couchbase.org/57328
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


# 5f3eb257 23-Jul-2015 Sundar Sridharan <sundar.sridharan@gmail.com>

Switch to hrtime from timeval in Global Thread Pool

This has small improvements in memory and cpu usage.
Also fixes several ThreadSanitizer races from unit tests.

Change-Id: I78

Switch to hrtime from timeval in Global Thread Pool

This has small improvements in memory and cpu usage.
Also fixes several ThreadSanitizer races from unit tests.

Change-Id: I78fdddb832251fc062058c04f75f8d22c4c2f68d
Reviewed-on: http://review.couchbase.org/53381
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


# f1770e55 12-Nov-2014 Dave Rigby <daver@couchbase.com>

Fix race in TaskQueue.{ready,future,pending}Queue access

Fix race as identified by ThreadSanitizer:

WARNING: ThreadSanitizer: data race (pid=4243)
Read of size 8 at 0x

Fix race in TaskQueue.{ready,future,pending}Queue access

Fix race as identified by ThreadSanitizer:

WARNING: ThreadSanitizer: data race (pid=4243)
Read of size 8 at 0x7d04000fde60 by main thread (mutexes: write M1367):
#0 std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >::operator++() /usr/include/c++/4.8/bits/stl_list.h:235 (ep.so+0x00000013a129)
#1 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::__distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::input_iterator_tag) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:82 (ep.so+0x000000138b67)
#2 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:118 (ep.so+0x000000136b63)
#3 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::size() const /usr/include/c++/4.8/bits/stl_list.h:874 (ep.so+0x000000135538)
#4 ExecutorPool::doTaskQStat(EventuallyPersistentEngine*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:654 (ep.so+0x0000001331b6)
#5 EventuallyPersistentEngine::doWorkloadStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4198 (ep.so+0x000000112ed3)
#6 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4454 (ep.so+0x000000114cb4)
#7 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
#8 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
#9 test_workload_stats /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:7094 (ep_testsuite.so+0x00000004e931)
#10 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc)
#11 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)

Previous write of size 8 at 0x7d04000fde60 by thread T5 (mutexes: write M45):
#0 RCValue::_rc_decref() const /repos/couchbase/server/source/ep-engine/src/atomic.h:293 (ep.so+0x000000096797)
#1 __gnu_cxx::new_allocator<std::_List_node<SingleThreadedRCPtr<GlobalTask> > >::allocate(unsigned long, void const*) /usr/include/c++/4.8/ext/new_allocator.h:104 (ep.so+0x00000017b42b)
#2 std::_List_base<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_get_node() /usr/include/c++/4.8/bits/stl_list.h:334 (ep.so+0x00000017b0dc)
#3 std::_List_node<SingleThreadedRCPtr<GlobalTask> >* std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_create_node<SingleThreadedRCPtr<GlobalTask> const&>(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:502 (ep.so+0x00000017a25a)
#4 void std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_insert<SingleThreadedRCPtr<GlobalTask> const&>(std::_List_iterator<SingleThreadedRCPtr<GlobalTask> >, SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1561 (ep.so+0x000000178e7b)
#5 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::push_back(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1016 (ep.so+0x00000017817b)
#6 TaskQueue::_fetchNextTask(ExecutorThread&, bool) /repos/couchbase/server/source/ep-engine/src/taskqueue.cc:126 (ep.so+0x000000176932)
#7 TaskQueue::fetchNextTask(ExecutorThread&, bool) /repos/couchbase/server/source/ep-engine/src/taskqueue.cc:142 (ep.so+0x000000176acf)
#8 ExecutorPool::_nextTask(ExecutorThread&, unsigned char) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:214 (ep.so+0x000000130707)
#9 ExecutorPool::nextTask(ExecutorThread&, unsigned char) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:229 (ep.so+0x0000001307a5)
#10 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:78 (ep.so+0x000000149da0)
#11 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a)
#12 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
#13 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)

Fix by adding ew helper methods which will return the size of the
various queues (while holding the TaskQueue's mutex).

Change-Id: If5e8d357e45803d78c4ba6ed1475e6e1a90e1c89
Reviewed-on: http://review.couchbase.org/43313
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


Revision tags: v3.0.1_mc, v3.0.0_ep, v3.0.0-beta3_mc, v3.0.0_mc, v3.0.0-beta2_mc
# e616ac04 17-Jul-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-11731 remove shard level locking from ExecutorPool

we no longer need to block threads at shard level since
locks are acquired at the vbucket level

Change-Id: Iebc368d243499b7

MB-11731 remove shard level locking from ExecutorPool

we no longer need to block threads at shard level since
locks are acquired at the vbucket level

Change-Id: Iebc368d243499b76b5cf1f4f0d50c487866960dc
Reviewed-on: http://review.couchbase.org/39502
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...


# b6534b8a 14-Jul-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-11434 distributed sleep on taskQ

Make shared thread pool workers sleep inside their respective
TaskQueue mutexes as opposed to a single global mutex to reduce
the CPU overhead of

MB-11434 distributed sleep on taskQ

Make shared thread pool workers sleep inside their respective
TaskQueue mutexes as opposed to a single global mutex to reduce
the CPU overhead of putting threads to sleep and waking them up.
We still try to wake only as many threads as ready tasks.
First try waking local threads, if still more tasks are ready,
then visit other TaskQueues to wake their sleeping threads

Results in 15% improvement in view operations local testing
Lowered scheduling overhead with larger number of threads

Change-Id: I8ca746a696b4b2166f327385125b8510c0567407
Reviewed-on: http://review.couchbase.org/39210
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...


Revision tags: v3.0.0-beta1_mc
# 2900340f 30-Jun-2014 Chiyoung Seo <chiyoung.seo@gmail.com>

MB-11488 Set the thread local to NULL before calling thread pool API

Memory allocations and de-allocations inside a global thread pool
API should not be accounted in a bucket memory quot

MB-11488 Set the thread local to NULL before calling thread pool API

Memory allocations and de-allocations inside a global thread pool
API should not be accounted in a bucket memory quota because they
might not be executed in the same bucket context. As a todo task,
we need to keep track of the memory overhead of the global thread
pool separately.

Change-Id: I328dc39c98a23c2e397ccae08b0dc76936d6d2f6
Reviewed-on: http://review.couchbase.org/38942
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


# d197ec73 24-Jun-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-11405 Wake only as many threads as ready tasks

+ If a thread has uncovered only one ready task then do not
wake more threads, since current thread will suffice.
+ Only signal th

MB-11405 Wake only as many threads as ready tasks

+ If a thread has uncovered only one ready task then do not
wake more threads, since current thread will suffice.
+ Only signal threads if they are sleeping.
+ cond_broadcast and wake all threads if we have more ready tasks
than sleeping threads
+ Aggregate and signal threads to avoid repeated mutex contentions
Change-Id: Ic130b87c19a2c5cd9eb971078c9a40b3612b5da0
Reviewed-on: http://review.couchbase.org/38756
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


# 495aaee4 26-Jun-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-11524 Check pendingQueue on poll to avoid starvation

We limit number of threads that can work on a task queue by
blocking excess tasks in a pendingQueue.
However in mixed priority

MB-11524 Check pendingQueue on poll to avoid starvation

We limit number of threads that can work on a task queue by
blocking excess tasks in a pendingQueue.
However in mixed priority scenarios, we can end up with
starvation if tasks are blocked in low priority pendingQueue
while capacity is released to a high priority task queue.

Fix is to release capacity when task finishes execution, and
have the task queues poll the pendingQueue.

This avoids starvation and also reduces mutex contentions.

Change-Id: Iaeb08a53d31d21b463502085198d07923c0731c7
Reviewed-on: http://review.couchbase.org/38825
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...


# 3af1e552 10-Jun-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-11287 Lock TaskQ selectively, avoid busy loops

Global Thread Pool offers 2 serialization features:
1. lockless shard serialization (Flusher, VBDelete, Snapshot etc)
2. task-queue

MB-11287 Lock TaskQ selectively, avoid busy loops

Global Thread Pool offers 2 serialization features:
1. lockless shard serialization (Flusher, VBDelete, Snapshot etc)
2. task-queue based thread control (NonIO and AuxIO)
If thread is working on tasks that need neither of the above two,
then avoid any additional locking overhead.

Also isLock variable ends up causing more overhead, removing same

Change-Id: I38b060573975786973aa608277d40851adb64e45
Reviewed-on: http://review.couchbase.org/38102
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...


# 82430da3 04-Jun-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-11287 move taskLogs to threads, lessen locking

This fix is meant to both reduce CPU and memory usage when
there are a large number of buckets by moving non-essential
task logging

MB-11287 move taskLogs to threads, lessen locking

This fix is meant to both reduce CPU and memory usage when
there are a large number of buckets by moving non-essential
task logging to ExecutorThread to avoid lock contention in
TaskQueue mutex

Change-Id: I8c82db1fb333febcead7e981e1dcffef3ce0b0a2
Reviewed-on: http://review.couchbase.org/37881
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


# 1172eb5d 21-May-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-10383 AuxIO, NonIO taskQ serialization must not affect CPU

If a thread cannot pick up an AuxIO task or a NonIO task, then stash
it away in a pendingQueue to avoid busy looping and hig

MB-10383 AuxIO, NonIO taskQ serialization must not affect CPU

If a thread cannot pick up an AuxIO task or a NonIO task, then stash
it away in a pendingQueue to avoid busy looping and high cpu usage,
performance problems.
added stats for the new pendingQueue
Change-Id: I00c40bab8bd8d4dd8e691d3e532aa1c427066f6a
Reviewed-on: http://review.couchbase.org/37367
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>

show more ...


# 8ee8b461 14-May-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-11083: block shard serial runnable tasks to save CPU

Ready tasks in the task queue that cannot be picked up by
a thread because of shard serialization are now put in
a shard-local

MB-11083: block shard serial runnable tasks to save CPU

Ready tasks in the task queue that cannot be picked up by
a thread because of shard serialization are now put in
a shard-local pending list to prevent threads from busy looping
Change-Id: I84b999df2da6e769f0da49bfda0409cae564792a
Reviewed-on: http://review.couchbase.org/37056
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


# 527009f4 25-Apr-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

Serialize Shard Level Operations from ExecutorPool

Add a flag in every shard to indicate that a thread is
currently working on it.
In TaskQueue, threads check this flag first to ensu

Serialize Shard Level Operations from ExecutorPool

Add a flag in every shard to indicate that a thread is
currently working on it.
In TaskQueue, threads check this flag first to ensure that
only one thread picks up task from this shard
Mark certain tasks like VBDeleteTask and VBSnapshotTask
as needing serialization

Change-Id: Ib4f0ae14d90a3d95ea0a59531197da4f87f388d3
Reviewed-on: http://review.couchbase.org/36286
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>

show more ...


# 29849d03 11-Mar-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-9986 ensure futureQueue's heap sort

Windows ep-engine unit tests enabling fix..
Microsoft Visual Studio has a debug heap checker which
requires that STL container priority_queue a

MB-9986 ensure futureQueue's heap sort

Windows ep-engine unit tests enabling fix..
Microsoft Visual Studio has a debug heap checker which
requires that STL container priority_queue always satisfies
the HEAPIFY property. In GIO's implementation it is possible
that futureQueue can disobey the heap sort property temporarily
when we wake and cancel tasks.
To fix, always re-sort queue when task is woken or cancelled
Change-Id: I5631a4ec7e609953f037cd5a7c689ed876b40c12
Reviewed-on: http://review.couchbase.org/34264
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

show more ...


Revision tags: v2.5.1_ep, v2.5.1-MP1-A_mc
# ddb60c3d 28-Jan-2014 Sundar Sridharan <sundar.sridharan@gmail.com>

MB-9939: worker stats fix, debug for MB-9675

Move tasklog and slow log to the taskQueue so that worker
stats are consistent and allows debugging of low disk utilization
seen in MB-96

MB-9939: worker stats fix, debug for MB-9675

Move tasklog and slow log to the taskQueue so that worker
stats are consistent and allows debugging of low disk utilization
seen in MB-9675
Change-Id: I9184a5b87ea91eebc3db0e7c14b7e406245ea468
Reviewed-on: http://review.couchbase.org/32887
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>

show more ...


Revision tags: v2.5.0_mc
# e32900fb 08-Jan-2014 Trond Norbye <trond.norbye@gmail.com>

Extract TaskQueue out of scheduler.h

Change-Id: If46478214e81defa42d17f3e7c63f639ac266756
Reviewed-on: http://review.couchbase.org/31720
Reviewed-by: Sundararaman Sridharan <sundar.s

Extract TaskQueue out of scheduler.h

Change-Id: If46478214e81defa42d17f3e7c63f639ac266756
Reviewed-on: http://review.couchbase.org/31720
Reviewed-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
Tested-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>

show more ...