Name Date Size

..11-Feb-20204 KiB

.gitignoreH A D11-Feb-20201.1 KiB

.mailmapH A D11-Feb-2020435

benchmarks/H11-Feb-20204 KiB

CMakeLists.txtH A D11-Feb-202030.4 KiB

configuration.jsonH A D11-Feb-202036.3 KiB

docs/H11-Feb-20204 KiB

DoxyfileH A D11-Feb-202063.9 KiB

LICENSEH A D11-Feb-202011.1 KiB

management/H11-Feb-20204 KiB

README.mdH A D11-Feb-20208.6 KiB

scripts/H11-Feb-20204 KiB

src/H11-Feb-202012 KiB

tests/H11-Feb-20204 KiB

tools/H11-Feb-20204 KiB

wrapper/H11-Feb-20204 KiB

README.md

1# Eventually Persistent Engine
2## Threads
3Code in ep-engine is executing in a multithreaded environment, two classes of
4thread exist.
5
61. memcached's threads, for servicing a client and calling in via the
7[engine API] (https://github.com/couchbase/memcached/blob/master/include/memcached/engine.h)
82. ep-engine's threads, for running tasks such as the document expiry pager
9(see subclasses of `GlobalTasks`).
10
11## Synchronisation Primitives
12
13There are two mutual-exclusion primitives available in ep-engine (in
14addition to those provided by the C++ standard library):
15
161. `RWLock` shared, reader/writer lock - [rwlock.h](./src/rwlock.h)
172. `SpinLock` 1-byte exclusive lock - [atomix.h](./src/atomic.h)
18
19A condition-variable is also available called `SyncObject`
20[syncobject.h](./src/syncobject.h). `SyncObject` glues a `std::mutex` and
21`std::condition_variable` together in one object.
22
23These primitives are managed via RAII wrappers - [locks.h](./src/locks.h).
24
251. `LockHolder` - a deprecated alias for std::lock_guard
262. `MultiLockHolder` - for acquiring a reference to a vector of `std::mutex`
27                       or `SyncObject`.
28
29### Mutex
30The general style is to create a `std::lock_guard` when you need to acquire a
31`std::mutex`, the constructor will acquire and when the `lock_guard` goes out of
32scope, the destructor will release the `std::mutex`. For certain use-cases the
33caller can explicitly lock/unlock a `std::mutex` via the `std::unique_lock`
34class.
35
36```c++
37std::mutex mutex;
38void example1() {
39    std::lock_guard<std::mutex> lockHolder(mutex);
40    ...
41    return;
42}
43
44void example2() {
45    std::unique_lock<std::mutex> lockHolder(mutex);
46    ...
47    lockHolder.unlock();
48    ...
49    lockHolder.lock();
50    ...
51    return;
52}
53```
54
55A `MultiLockHolder` allows a vector of locks to be conveniently acquired and
56released, and similarly to `LockHolder` the caller can choose to manually
57lock/unlock at any time (with all locks locked/unlocked via one call).
58
59```c++
60std::mutex mutexes[10];
61Object objects[10];
62void foo() {
63    MultiLockHolder lockHolder(&mutexes, 10);
64    for (int ii = 0; ii < 10; ii++) {
65        objects[ii].doStuff();
66    }
67    return;
68}
69```
70
71### RWLock
72
73`RWLock` allows many readers to acquire it and exclusive access for a writer.
74Like a std::mutex `RWLock` can be used with a std::lock_guard. The RWLock can
75either be explicitly casted to a `ReaderLock` / `WriterLock` through its
76`reader()` and `writer()` member functions or you can rely on the implicit
77conversions used by the `lock_guard` constructor.
78
79```c++
80RWLock rwLock;
81Object thing;
82
83void foo1() {
84    std::lock_guard<ReaderLock> rlh(rwLock);
85    if (thing.getData()) {
86    ...
87    }
88}
89
90void foo2() {
91    std::lock_guard<WriterLock> wlh(rwLock);
92    thing.setData(...);
93}
94```
95
96### SyncObject
97
98`SyncObject` inherits from `std::mutex` and is thus managed via a `LockHolder` or
99`MultiLockHolder`. The `SyncObject` provides the conditional-variable
100synchronisation primitive enabling threads to block and be woken.
101
102The wait/wakeOne/wake method is provided by the `SyncObject`.
103
104Note that `wake` will wake up a single blocking thread, `wakeOne` will wake up
105every thread that is blocking on the `SyncObject`.
106
107```c++
108SyncObject syncObject;
109bool sleeping = false;
110void foo1() {
111    LockHolder lockHolder(&syncObject);
112    sleeping = true;
113    syncObject.wait(); // the mutex is released and the thread put to sleep
114    // when wait returns the mutex is reacquired
115    sleeping = false;
116}
117
118void foo2() {
119    LockHolder lockHolder(&syncObject);
120    if (sleeping) {
121        syncObject.notifyOne();
122    }
123}
124```
125
126### SpinLock
127
128A `SpinLock` uses a single byte for the lock and our own code to spin until the
129lock is acquired. The intention for this lock is for low contention locks.
130
131The RAII pattern is just like for a mutex.
132
133
134```c++
135SpinLock spinLock;
136void example1() {
137    std::lock_guard<SpinLock> lockHolder(&spinLock);
138    ...
139    return;
140}
141```
142
143### _UNLOCKED convention
144
145ep-engine has a function naming convention that indicates the function should
146be called with a lock acquired.
147
148For example the following `doStuff_UNLOCKED` method indicates that it expect a
149lock to be held before the function is called. What lock should be acquired
150before calling is not defined by the convention.
151
152```c++
153void Object::doStuff_UNLOCKED() {
154}
155
156void Object::run() {
157    LockHolder lockHolder(&mutex);
158    doStuff_UNLOCKED();
159    return;
160}
161```
162
163## Atomic / thread-safe data structures
164
165In addition to the basic synchronization primitives described above,
166there are also the following higher-level data structures which
167support atomic / thread-safe access from multiple threads:
168
1691. `AtomicQueue`: thread-safe, approximate-FIFO queue, optimized for
170   multiple-writers, one reader - [atomicqueue.h](./src/atomicqueue.h)
1712. `AtomicUnorderedMap` : thread-safe unordered map -
172   [atomic_unordered_map.h](./src/atomic_unordered_map.h)
173
174## Thread Local Storage (ObjectRegistry).
175
176Threads in ep-engine are servicing buckets and when a thread is dispatched to
177serve a bucket, the pointer to the `EventuallyPersistentEngine` representing
178the bucket is placed into thread local storage, this avoids the need for the
179pointer to be passed along the chain of execution as a formal parameter.
180
181Both threads servicing frontend operations (memcached's threads) and ep-engine's
182own task threads will save the bucket's engine pointer before calling down into
183engine code.
184
185Calling `ObjectRegistry::onSwitchThread(enginePtr)` will save the `enginePtr`
186in thread-local-storage so that subsequent task code can retrieve the pointer
187with `ObjectRegistry::getCurrentEngine()`.
188
189## Tasks
190
191A task is created by creating a sub-class (the `run()` method is the entry point
192of the task) of the `GlobalTask` class and it is scheduled onto one of 4 task
193queue types. Each task should be declared in `src/tasks.defs.h` using the TASK
194macro. Using this macro ensures correct generation of a task-type ID, priority,
195task name and ultimately ensures each task gets its own scheduling statistics.
196
197The recipe is simple.
198
199### Add your task's class name with its priority into `src/tasks.defs.h`
200 * A lower value priority is 'higher'.
201```
202TASK(MyNewTask, 1) // MyNewTask has priority 1.
203```
204
205### Create your class and set its ID using `MY_TASK_ID`.
206
207```
208class MyNewTask : public GlobalTask {
209public:
210    MyNewTask(EventuallyPersistentEngine* e)
211        : GlobalTask(e/*engine/,
212                     MY_TASK_ID(MyNewTask),
213                     0.0/*snooze*/){}
214...
215```
216
217### Define pure-virtual methods in `MyNewTask`
218* run method
219
220The run method is invoked when the task is executed. The method should return
221true if it should be scheduled again. If false is returned, the instance of the
222task is never re-scheduled and will deleted once all references to the instance are
223gone.
224
225```
226bool run() {
227   // Task code here
228   return schedule again?;
229}
230```
231
232* Define the `getDescription` method to aid debugging and statistics.
233```
234std::string getDescription() {
235    return "A brief description of what MyNewTask does";
236}
237```
238
239### Schedule your task to the desired queue.
240```
241ExTask myNewTask = new MyNewTask(&engine);
242myNewTaskId = ExecutorPool::get()->schedule(myNewTask, NONIO_TASK_IDX);
243```
244
245The 4 task queue types are:
246* Readers -  `READER_TASK_IDX`
247 * Tasks that should primarily only read from 'disk'. They generally read from
248the vbucket database files, for example background fetch of a non-resident document.
249* Writers (they are allowed to read too) `WRITER_TASK_IDX`
250 * Tasks that should primarily only write to 'disk'. They generally write to
251the vbucket database files, for example when flushing the write queue.
252* Auxilliary IO `AUXIO_TASK_IDX`
253 * Tasks that read and write 'disk', but not necessarily the vbucket data files.
254* Non IO `NONIO_TASK_IDX`
255 * Tasks that do not perform 'disk' I/O.
256
257### Utilise `snooze`
258
259The snooze value of the task sets when the task should be executed. The initial snooze
260value is set when constructing `GlobalTask`. A value of 0.0 means attempt to execute
261the task as soon as scheduled and 5.0 would be 5 seconds from being scheduled
262(scheduled meaning when `ExecutorPool::get()->schedule(...)` is called).
263
264The `run()` function can also call `snooze(double snoozeAmount)` to set how long
265before the task is rescheduled.
266
267It is **best practice** for most tasks to actually do a sleep forever from their run function:
268
269```
270  snooze(INT_MAX);
271```
272
273Using `INT_MAX` means sleep forever and tasks should always sleep until they have
274real work todo. Tasks **should not periodically poll for work** with a snooze of
275n seconds.
276
277### Utilise `wake()`
278When a task has work todo, some other function should be waking the task using the wake method.
279
280```
281ExecutorPool::get()->wake(myNewTaskId)`
282```
283