Name Date Size

..11-Feb-20204 KiB

.gitignoreH A D11-Feb-2020156

b/resources/H11-Feb-20204 KiB

buildout.cfgH A D11-Feb-2020407

conf/H11-Feb-20204 KiB

doc/H11-Feb-20204 KiB

enginetests/H11-Feb-20204 KiB

lib/H11-Feb-20204 KiB

logging.conf.sampleH A D11-Feb-2020579

longevity/H11-Feb-20204 KiB

MakefileH A D11-Feb-20202.2 KiB

mcsoda-setup.pyH A D11-Feb-20201.1 KiB

mcsoda.logging.confH A D11-Feb-2020444

pysystests/H11-Feb-20204 KiB

pytests/H11-Feb-20204 KiB

README.mdH A D11-Feb-20207.4 KiB

resources/H11-Feb-20204 KiB

scripts/H11-Feb-20204 KiB

scripts.logging.confH A D11-Feb-2020552

setup.pyH A D11-Feb-20201.5 KiB

TestInput.pyH A D11-Feb-202017.8 KiB

testrunnerH A D11-Feb-202038

testrunner.pyH A D11-Feb-202017.5 KiB

unittests/H11-Feb-20204 KiB

README.md

1Prerequisites
2-------------
3
4* Python 2.6 or 2.7
5* pip or easy_install
6
7Dependencies
8------------
9
10General:
11
12    pip install paramiko
13
14    pip install boto
15
16Performance tests:
17
18    pip install btrc
19
20PDF reports:
21
22    pip install couchdbkit
23
24Documentation:
25
26    pip install sphinx
27
28    pip install sphinx-pypi-upload
29
30Buildout:
31
32    pip install zc.buildout
33
34Usage
35-----
36
37    $ ./testrunner  -h
38    Usage: testrunner [options]
39
40    Options:
41      -h, --help            show this help message and exit
42      -q
43      -p PARAMS, --params=PARAMS
44                            Optional key=value parameters, comma-separated -p
45                            k=v,k2=v2,...
46      -n, --noop            NO-OP - emit test names, but don't actually run them
47                            e.g -n true
48      -l LOGLEVEL, --log-level=LOGLEVEL
49                            e.g -l info,warning,error
50
51      TestCase/Runlist Options:
52        -i INI, --ini=INI   Path to .ini file containing server information,e.g -i
53                            tmp/local.ini
54        -c RUNLIST, --config=RUNLIST
55                            Config file name (located in the conf subdirectory),
56                            e.g -c py-view.conf
57        -t TESTCASE, --test=TESTCASE
58                            Test name (multiple -t options add more tests) e.g -t
59                            performance.perf.DiskDrainRate
60
61Buildout
62--------
63
64Initiate buildout directory structure, create sandbox, build packages and scripts, fetch dependencies, and etc.:
65
66    buildout
67
68You can execute testrunner now:
69
70    ./bin/testrunner -h
71
72Resource Files
73--------------
74
75Ini files represents the ns_server information which is accessible to the tests.
76
77**[global]** section defines the rest username, password that tests use to login to
78ns_server.
79
80Example:
81
82    [global]
83    username:Administrator
84    password:membase
85
86    [membase]
87    rest_username:Administrator
88    rest_password:asdasd
89
90**[servers]** section lists port and ssh related information. ssh connection
91information is required for small subset of tests where test needs to perform
92installation,backup or restore. If ns_server instances are started using
93ns_server/cluster_run script then you only need to define ip and port for those
94nodes.
95
96Example:
97
98    [servers]
99    1:10.1.6.104_1
100    2:10.1.6.104_2
101    3:10.1.6.104_3
102    4:10.1.6.104_4
103
104    [10.1.6.104_1]
105    ip:10.1.6.104
106    port:9000
107
108    [10.1.6.104_2]
109    ip:10.1.6.104
110    port:9001
111
112    [10.1.6.104_3]
113    ip:10.1.6.104
114    port:9002
115
116    [10.1.6.104_4]
117    ip:10.1.6.104
118    port:9003
119
120Test Execution and Reporting
121----------------------------
122
123For every test run testrunner creates a temp folder and dumps the logs and
124xunit reports in the newly generated folder.
125
126for instance if you run
127
128    $ ./testrunner -i resources/jenkins/single-node-centos-32.ini -t setgettests.MembaseBucket.value_100b
129
130you will see this summary after each test is ran:
131
132    summary so far suite setgettests.MembaseBucket , pass 1 , fail 0
133    logs and results are available under tmp-12-11-47
134
135and logs:
136
137    $ ls  tmp-12-11-47/
138    report-12-11-47.xml-setgettests.MembaseBucket.xml	value_100b.log
139
140Development
141-----------
142
143When using git on Linux/OSX systems, you might run into issues where git
144incorrectly believes Windows-related files have been modified. In reality, git
145is merely mis-treating CRLF line endings.
146Try the following...
147
148    $ cd testrunner
149    $ git config core.autocrlf false
150
151Running End to End Tests Before Submitting a Patch to Gerrit
152------------------------------------------------------------
153
154Testrunner project has different test suites which can be run priori to
155submitting the code to gerrit. There are test suites that can be run against a
156single node which validates basic database operations such as persistence and
157bucket management. There are also key-value clustering related test cases which
158can be run against Membase/Couchbase 1.8 multiple nodes. Recently we have also
159been adding more tests which validates basic view functionalities on a cluster
160and on a single.
161
162### "make e2e-kv-single-node"
163
164This make target will start ns_server using ```cluster_run -n1``` and run all the
165test cases listed in conf/py-all-dev.conf. The test runtime can vary between
16615-30 minutes depending on your machine.
167
168Testrunner prints out a human readable pass/fail of the tests. Please refer to
169the "rerunning test" section for more information of how to re-run one single
170test against cluster_run.
171
172### "make test-views"
173
174This make target will start four ns_server(s) using ```cluster_run -n4``` and
175runs all the test cases listed in conf/py-view.conf. The test runtime can vary
176between 30-45 minutes. py-view.conf contains basic test cases which validates
177clustering operations such as rebalancing.Each test is also parametrized so you
178can easily modify this run list and change the number of docs or the
179load_duration.
180
181For instance, ```test_get_view_during_x_min_load_y_working_set,num-docs=10000,load-time=1,run-view-time=1```
182test will create a view, inserts 1000 documents, mutate those documents for 1
183minute and run view queries in parallel to the load for 1 min. You can easily
184change the parameters there to insert 1M items and keep the load running for 10
185mins for example.
186
187Using mcsoda to load json documents on to a node/cluster
188--------------------------------------------------------
189
190mcsoda is a load generator tool that randomly generates json
191documents. It is available under testrunner/lib/perf_engines/ .
192
193Mcsoda uses moxi port to distribute load in a cluster.
194The moxi can be set up on the server side or on a client side as follows:
195
196### "server-side moxi"
197
198From a client, load can be run on a cluster represented by: xx.xx.xx.xxx as:
199
200	lib/perf_engines/mcsoda.py xx.xx.xx.xxx:11211 vbuckets=1024 doc-gen=0 doc-cache=0
201	ratio-creates=0.5 ratio-sets=0.2 ratio-deletes=0.02  ratio-expirations=0.03
202	expiration=3600 min-value-size=2,4 threads=100 max-items=18000000
203	exit-after-creates=1 prefix=KEY_ max-creates=18000000
204
205This is going to ensure that there will be 20% sets against 80% gets.
206Of the 20% sets, 50% will be creates and the rest updates.
207There shall be 2% deletes, and minimum item size varies from 2 to 4 Bytes.
2083% of the items set will be marked as expired after a duration of 1 hour.
209The tool will use 100 threads to generate the json load, and every item will be
210prefixed by "KEY_". max_creates limits the no. of items created. All the load
211generated is going to get loaded on bucket "default". Prefix the IP by
212"bucket_name:password@", to load on to standard/sasl buckets.
213
214For more options in mcsoda, type in:
215```lib/perf_engines/mcsoda.py -h```
216
217### "client-side moxi"
218
219Set up the moxi for a specific bucket on the client side as follows:
220(Make sure couchbase-server is installed on the client)
221
222	/opt/couchbase/bin/moxi -Z usr=Administrator,pwd=password,port_listen=11611,
223	concurrency=1024,wait_queue_timeout=200,connect_timeout=400,connect_max_errors=3,
224	connect_retry_interval=30000,auth_timeout=100,downstream_conn_max=128,downstream_timeout=5000,
225	cycle=200 http://xx.xx.xx.xxx:8091/pools/default/bucketsStreaming/[bucket_name] -d
226
227This command sets up moxi for the cluster on the client side at port 11611.
228For more information on the moxi command, type in:
229```/opt/couchbase/bin/moxi -h```
230
231Once the moxi is all set up, run mcsoda against the moxi just set up:
232
233	lib/perf_engines/mcsoda.py localhost:11611 vbuckets=1024 doc-gen=0 doc-cache=0
234	ratio-creates=0.5 ratio-sets=0.2 ratio-deletes=0.02  min-value-size=2,4 threads=100
235	max-items=18000000 exit-after-creates=1 prefix=KEY_ max-creates=18000000
236