Name Date Size #Lines LOC

..09-Aug-2022-

benchmarks/H09-Aug-2022-2,1461,285

docs/H09-Aug-2022-3,1132,609

management/H09-Aug-2022-3,5662,769

scripts/H09-Aug-2022-14075

src/H09-Aug-2022-105,26265,056

tests/H09-Aug-2022-108,03874,154

tools/H09-Aug-2022-5,1653,476

.gitignoreH A D09-Aug-20221.1 KiB7877

.mailmapH A D09-Aug-2022435 87

CMakeLists.txtH A D09-Aug-202218 KiB459409

DoxyfileH A D09-Aug-202263.9 KiB1,5521,110

LICENSEH A D09-Aug-202211.1 KiB203169

README.mdH A D09-Aug-20228.8 KiB291229

configuration.jsonH A D09-Aug-202246.1 KiB1,2881,286

README.md

1# Eventually Persistent Engine
2## Threads
3Code in ep-engine is executing in a multithreaded environment, two classes of
4thread exist.
5
61. memcached's threads, for servicing a client and calling in via the
7[engine API] (https://github.com/couchbase/memcached/blob/master/include/memcached/engine.h)
82. ep-engine's threads, for running tasks such as the document expiry pager
9(see subclasses of `GlobalTasks`).
10
11## Synchronisation Primitives
12
13There are two mutual-exclusion primitives available in ep-engine (in
14addition to those provided by the C++ standard library):
15
161. `RWLock` shared, reader/writer lock - [rwlock.h](./src/rwlock.h)
172. `SpinLock` 1-byte exclusive lock - [atomix.h](./src/atomic.h)
18
19A condition-variable is also available called `SyncObject`
20[syncobject.h](./src/syncobject.h). `SyncObject` glues a `std::mutex` and
21`std::condition_variable` together in one object.
22
23These primitives are managed via RAII wrappers - [locks.h](./src/locks.h).
24
251. `LockHolder` - a deprecated alias for std::lock_guard
262. `MultiLockHolder` - for acquiring a reference to a vector of `std::mutex`
27                       or `SyncObject`.
28
29### Mutex
30The general style is to create a `std::lock_guard` when you need to acquire a
31`std::mutex`, the constructor will acquire and when the `lock_guard` goes out of
32scope, the destructor will release the `std::mutex`. For certain use-cases the
33caller can explicitly lock/unlock a `std::mutex` via the `std::unique_lock`
34class.
35
36```c++
37std::mutex mutex;
38void example1() {
39    std::lock_guard<std::mutex> lockHolder(mutex);
40    ...
41    return;
42}
43
44void example2() {
45    std::unique_lock<std::mutex> lockHolder(mutex);
46    ...
47    lockHolder.unlock();
48    ...
49    lockHolder.lock();
50    ...
51    return;
52}
53```
54
55A `MultiLockHolder` allows a vector of locks to be conveniently acquired and
56released, and similarly to `LockHolder` the caller can choose to manually
57lock/unlock at any time (with all locks locked/unlocked via one call).
58
59```c++
60std::mutex mutexes[10];
61Object objects[10];
62void foo() {
63    MultiLockHolder lockHolder(&mutexes, 10);
64    for (int ii = 0; ii < 10; ii++) {
65        objects[ii].doStuff();
66    }
67    return;
68}
69```
70
71### RWLock
72
73`RWLock` allows many readers to acquire it and exclusive access for a writer.
74Like a std::mutex `RWLock` can be used with a std::lock_guard. The RWLock can
75either be explicitly casted to a `ReaderLock` / `WriterLock` through its
76`reader()` and `writer()` member functions or you can rely on the implicit
77conversions used by the `lock_guard` constructor.
78
79```c++
80RWLock rwLock;
81Object thing;
82
83void foo1() {
84    std::lock_guard<ReaderLock> rlh(rwLock);
85    if (thing.getData()) {
86    ...
87    }
88}
89
90void foo2() {
91    std::lock_guard<WriterLock> wlh(rwLock);
92    thing.setData(...);
93}
94```
95
96### SyncObject
97
98`SyncObject` inherits from `std::mutex` and is thus managed via a `LockHolder` or
99`MultiLockHolder`. The `SyncObject` provides the conditional-variable
100synchronisation primitive enabling threads to block and be woken.
101
102The wait/wakeOne/wake method is provided by the `SyncObject`.
103
104Note that `wake` will wake up a single blocking thread, `wakeOne` will wake up
105every thread that is blocking on the `SyncObject`.
106
107```c++
108SyncObject syncObject;
109T data;
110bool notified = false;
111
112void consumer() {
113    std::unique_lock<std::mutex> lockHolder(syncObject);
114    // Upon call to wait() the mutex is released and the
115    // thread put to sleep
116    syncObject.wait(lockHolder,
117                    [&notified](){ return notified; });
118    // when wait returns the mutex is reacquired
119    ... do something with data ...
120    notified = false;
121}
122
123void producer() {
124    // Produce something for consumer to do...
125    {
126        std::unique_lock<std::mutex> lockHolder(syncObject);
127        data = ...;
128        notified = true;
129    }
130    syncObject.notifyOne();
131}
132```
133
134### SpinLock
135
136A `SpinLock` uses a single byte for the lock and our own code to spin until the
137lock is acquired. The intention for this lock is for low contention locks.
138
139The RAII pattern is just like for a mutex.
140
141
142```c++
143SpinLock spinLock;
144void example1() {
145    std::lock_guard<SpinLock> lockHolder(&spinLock);
146    ...
147    return;
148}
149```
150
151### _UNLOCKED convention
152
153ep-engine has a function naming convention that indicates the function should
154be called with a lock acquired.
155
156For example the following `doStuff_UNLOCKED` method indicates that it expect a
157lock to be held before the function is called. What lock should be acquired
158before calling is not defined by the convention.
159
160```c++
161void Object::doStuff_UNLOCKED() {
162}
163
164void Object::run() {
165    LockHolder lockHolder(&mutex);
166    doStuff_UNLOCKED();
167    return;
168}
169```
170
171## Atomic / thread-safe data structures
172
173In addition to the basic synchronization primitives described above,
174there are also the following higher-level data structures which
175support atomic / thread-safe access from multiple threads:
176
1771. `AtomicQueue`: thread-safe, approximate-FIFO queue, optimized for
178   multiple-writers, one reader - [atomicqueue.h](./src/atomicqueue.h)
1792. `AtomicUnorderedMap` : thread-safe unordered map -
180   [atomic_unordered_map.h](./src/atomic_unordered_map.h)
181
182## Thread Local Storage (ObjectRegistry).
183
184Threads in ep-engine are servicing buckets and when a thread is dispatched to
185serve a bucket, the pointer to the `EventuallyPersistentEngine` representing
186the bucket is placed into thread local storage, this avoids the need for the
187pointer to be passed along the chain of execution as a formal parameter.
188
189Both threads servicing frontend operations (memcached's threads) and ep-engine's
190own task threads will save the bucket's engine pointer before calling down into
191engine code.
192
193Calling `ObjectRegistry::onSwitchThread(enginePtr)` will save the `enginePtr`
194in thread-local-storage so that subsequent task code can retrieve the pointer
195with `ObjectRegistry::getCurrentEngine()`.
196
197## Tasks
198
199A task is created by creating a sub-class (the `run()` method is the entry point
200of the task) of the `GlobalTask` class and it is scheduled onto one of 4 task
201queue types. Each task should be declared in `src/tasks.defs.h` using the TASK
202macro. Using this macro ensures correct generation of a task-type ID, priority,
203task name and ultimately ensures each task gets its own scheduling statistics.
204
205The recipe is simple.
206
207### Add your task's class name with its priority into `src/tasks.defs.h`
208 * A lower value priority is 'higher'.
209```
210TASK(MyNewTask, 1) // MyNewTask has priority 1.
211```
212
213### Create your class and set its ID using `MY_TASK_ID`.
214
215```
216class MyNewTask : public GlobalTask {
217public:
218    MyNewTask(EventuallyPersistentEngine* e)
219        : GlobalTask(e/*engine/,
220                     MY_TASK_ID(MyNewTask),
221                     0.0/*snooze*/){}
222...
223```
224
225### Define pure-virtual methods in `MyNewTask`
226* run method
227
228The run method is invoked when the task is executed. The method should return
229true if it should be scheduled again. If false is returned, the instance of the
230task is never re-scheduled and will deleted once all references to the instance are
231gone.
232
233```
234bool run() {
235   // Task code here
236   return schedule again?;
237}
238```
239
240* Define the `getDescription` method to aid debugging and statistics.
241```
242std::string getDescription() {
243    return "A brief description of what MyNewTask does";
244}
245```
246
247### Schedule your task to the desired queue.
248```
249ExTask myNewTask = new MyNewTask(&engine);
250myNewTaskId = ExecutorPool::get()->schedule(myNewTask, NONIO_TASK_IDX);
251```
252
253The 4 task queue types are:
254* Readers -  `READER_TASK_IDX`
255 * Tasks that should primarily only read from 'disk'. They generally read from
256the vbucket database files, for example background fetch of a non-resident document.
257* Writers (they are allowed to read too) `WRITER_TASK_IDX`
258 * Tasks that should primarily only write to 'disk'. They generally write to
259the vbucket database files, for example when flushing the write queue.
260* Auxilliary IO `AUXIO_TASK_IDX`
261 * Tasks that read and write 'disk', but not necessarily the vbucket data files.
262* Non IO `NONIO_TASK_IDX`
263 * Tasks that do not perform 'disk' I/O.
264
265### Utilise `snooze`
266
267The snooze value of the task sets when the task should be executed. The initial snooze
268value is set when constructing `GlobalTask`. A value of 0.0 means attempt to execute
269the task as soon as scheduled and 5.0 would be 5 seconds from being scheduled
270(scheduled meaning when `ExecutorPool::get()->schedule(...)` is called).
271
272The `run()` function can also call `snooze(double snoozeAmount)` to set how long
273before the task is rescheduled.
274
275It is **best practice** for most tasks to actually do a sleep forever from their run function:
276
277```
278  snooze(INT_MAX);
279```
280
281Using `INT_MAX` means sleep forever and tasks should always sleep until they have
282real work todo. Tasks **should not periodically poll for work** with a snooze of
283n seconds.
284
285### Utilise `wake()`
286When a task has work todo, some other function should be waking the task using the wake method.
287
288```
289ExecutorPool::get()->wake(myNewTaskId)`
290```
291