Name Date Size

..11-Jan-20224 KiB

__init__.pyH A D11-Jan-20220

cbtop.pyH A D11-Jan-20221.9 KiB

do_cluster.pyH A D11-Jan-20222.2 KiB

eperf.pyH A D11-Jan-202253.1 KiB

eperf_mongo.pyH A D11-Jan-20223.5 KiB

Graph.mdH A D11-Jan-20222.4 KiB

iperf.pyH A D11-Jan-202220.7 KiB

nru_moninor.pyH A D11-Jan-20223.7 KiB

perf.pyH A D11-Jan-202247.6 KiB

perf_defaults.pyH A D11-Jan-20227 KiB

README.mdH A D11-Jan-20224.7 KiB

viewgen.pyH A D11-Jan-20226.3 KiB

README.md

1Performance Test Phases
2=======================
3
4A performance test run has several phases...
5
6Server machine preparation
7--------------------------
8
9This step is usually done manually, where a cluster of machines is
10prepared, O/S installed, disks configured, etc.  The user might be
11doing this on EC2, leveraging chef recipes, etc.  At the end, a list
12of IP addresses is needed, in a testrunner *.ini file.  In this
13document, our example is called "CLUSTER.ini".
14
15Example *.ini files are in testrunner/resources/perf/*.ini directory.
16
17Running testrunner across several phases
18----------------------------------------
19
20Next, various scripts are run in several phases.  All these phases
21comprise a complete "run".  The "inputs" to a run are...
22
23* BUILD - version of software that's the target of testing.  Example: 2.0.0r-1131)
24* TESTNAME - the test configuration (*.conf).  Example: write-20M.conf
25* CLUSTER - the cluster to test against.  Example: CLUSTER.ini, from above.
26
27## install phase
28
29In the install phase, the software-to-be-tested is downloaded and
30installed on the CLUSTER.ini server machines, using this script...
31
32    testrunner/scripts/install.py -i CLUSTER.ini -p product=cb,version=$BUILD
33
34## cluster phase
35
36In this phase, the cluster is configured, nodes joined and rebalanced,
37but the cluster is still empty.  The script which does this is...
38
39    testrunner/pytests/performance/do_cluster.py
40
41## load phase
42
43In this phase, data is initially loaded, using...
44
45    cd testrunenr
46    ./testrunner -i resources/perf/$CLUSTER.ini -c conf/perf/$TESTNAME.conf \
47        -p load_phase=1,access_phase=0,...
48
49This is usually done with multiple, concurrent clients to speed data
50loading.  Each client is given a different partition of the dataset to
51load.
52
53## reload phase
54
55The reload phase "touches" a specified set of data, so that the
56cluster will have that data warm and cached in-memory.  The goal of
57the reload phase is to get the cluster to a point of simulated actual
58usage of user deployments.
59
60The reload phase is currently done by a single client, but it can be
61one-day be partitioned for higher performance.
62
63## indexing phase
64
65In this phase, which only applies for Couchbase 2.0 and those systems
66that support indexing, indexes are defined/created on the cluster and
67the data in the cluster is indexed.  The actual index definitions are
68defined by the test spec.
69
70## access phase
71
72The access phase is the meat of the performance test.
73
74In this phase, one or more concurrent clients access the database
75using a defined mix of commands.  The number of clients and actual mix
76of command types are defined by a test spec.
77
78In the access phase, testrunner invokes mcsoda with the appropriate,
79test-specific parameters.
80
81    ./testrunner -i resources/perf/$CLUSTER.ini -c conf/perf/$TESTNAME.conf \
82        -p load_phase=0,access_phase=1,...
83
84An explanation of mcsoda is here...
85
86    testrunner/pytests/performance/README.md
87
88## drain phase (optional)
89
90In this phase, we wait for any dirty write queues to drain to disk, so
91that all data is fully persisted.
92
93## last compact phase (optional)
94
95A last compaction phase might be run, to measure the final on-disk
96data usage of the system.
97
98## last stats collection phase
99
100The system has been collecting stats all along across the phases on
101each node.  But, finally, there's a last stats "collect" phase that
102aggregates data, lightly cleanses it, and pushes data to a data
103archive where it becomes immutable.
104
105The entire system, roughly, is almost a giant function with inputs of
106BUILD, CLUSTER, TESTNAME, START_TIME and outputs in the data archive.
107
108We currently use CouchDB as the data archive, but may also introduce
109other file stores like S3.
110
111## report generation phase
112
113This phases uses information from the data archive to generate PDF's
114or other reports.  In this phase, results from various runs can be
115compared.  The compared runs, naturally, need to have the same CLUSTER
116and TESTNAME, but may have differing BUILD and START_TIME.
117
118# During Development
119
120During development, you can run load and access phase all in one shot,
121for example, by having all XXX_phase parameter flags set to 1.  For
122example...
123
124    ./testrunner -i resources/perf/$CLUSTER.ini -c conf/perf/$TESTNAME.conf \
125        -p load_phase=1,access_phase=1,...
126
127# Automated Performance Runs
128
129We use a continuous integraton system, Jenkins, to schedule and
130execute the above phases for each run.  Input parameters to the
131Jenkins job include the BUILD, CLUSTER, TESTNAME, and Jenkins computes
132the START_TIME.
133
134# Immutability Conventions
135
136The conf/perf/$TESTNAME.conf files should be treated as immutable.  If
137you want to change a test, don't change an existing file.  Instead,
138create a new copy and tweak the copy.
139
140The resources/perf/$CLUSTER.ini files should be similarly considered
141as immutable, by convention.
142