Name Date Size

..19-Aug-20204 KiB

.gitignoreH A D19-Aug-2020156

array_simple_table.txtH A D19-Aug-20200

b/resources/H19-Aug-20204 KiB

buildout.cfgH A D19-Aug-2020407

cloudtest/H19-Aug-20204 KiB

conf/H19-Aug-202012 KiB

doc/H19-Aug-20204 KiB

enginetests/H19-Aug-20204 KiB

lib/H19-Aug-20204 KiB

LICENSEH A D19-Aug-202010 KiB

logging.conf.sampleH A D19-Aug-2020579

longevity/H19-Aug-20204 KiB

MakefileH A D19-Aug-20203 KiB

mcsoda-setup.pyH A D19-Aug-20201.1 KiB

mcsoda.logging.confH A D19-Aug-2020444

pysystests/H19-Aug-20204 KiB

pytests/H19-Aug-20204 KiB

README.mdH A D19-Aug-20207.5 KiB

resources/H19-Aug-20204 KiB

scripts/H19-Aug-20204 KiB

scripts.logging.confH A D19-Aug-2020552

setup.pyH A D19-Aug-20201.5 KiB

TestInput.pyH A D19-Aug-202023.8 KiB

testrunnerH A D19-Aug-202038

testrunner.pyH A D19-Aug-202040 KiB

unittests/H19-Aug-20204 KiB

README.md

1*** Test Runner: Readme ***
2
3Prerequisites
4-------------
5
6* Python 2.6 or 2.7
7* pip or easy_install
8
9Dependencies
10------------
11
12General:
13
14    pip install paramiko
15
16    pip install boto
17
18Performance tests:
19
20    pip install btrc
21
22PDF reports:
23
24    pip install couchdbkit
25
26Documentation:
27
28    pip install sphinx
29
30    pip install sphinx-pypi-upload
31
32Buildout:
33
34    pip install zc.buildout
35
36Usage
37-----
38
39    $ ./testrunner  -h
40    Usage: testrunner [options]
41
42    Options:
43      -h, --help            show this help message and exit
44      -q
45      -p PARAMS, --params=PARAMS
46                            Optional key=value parameters, comma-separated -p
47                            k=v,k2=v2,...
48      -n, --noop            NO-OP - emit test names, but don't actually run them
49                            e.g -n true
50      -l LOGLEVEL, --log-level=LOGLEVEL
51                            e.g -l info,warning,error
52
53      TestCase/Runlist Options:
54        -i INI, --ini=INI   Path to .ini file containing server information,e.g -i
55                            tmp/local.ini
56        -c RUNLIST, --config=RUNLIST
57                            Config file name (located in the conf subdirectory),
58                            e.g -c py-view.conf
59        -t TESTCASE, --test=TESTCASE
60                            Test name (multiple -t options add more tests) e.g -t
61                            performance.perf.DiskDrainRate
62
63Buildout
64--------
65
66Initiate buildout directory structure, create sandbox, build packages and scripts, fetch dependencies, and etc.:
67
68    buildout
69
70You can execute testrunner now:
71
72    ./bin/testrunner -h
73
74Resource Files
75--------------
76
77Ini files represents the ns_server information which is accessible to the tests.
78
79**[global]** section defines the rest username, password that tests use to login to
80ns_server.
81
82Example:
83
84    [global]
85    username:Administrator
86    password:membase
87
88    [membase]
89    rest_username:Administrator
90    rest_password:asdasd
91
92**[servers]** section lists port and ssh related information. ssh connection
93information is required for small subset of tests where test needs to perform
94installation,backup or restore. If ns_server instances are started using
95ns_server/cluster_run script then you only need to define ip and port for those
96nodes.
97
98Example:
99
100    [servers]
101    1:10.1.6.104_1
102    2:10.1.6.104_2
103    3:10.1.6.104_3
104    4:10.1.6.104_4
105
106    [10.1.6.104_1]
107    ip:10.1.6.104
108    port:9000
109
110    [10.1.6.104_2]
111    ip:10.1.6.104
112    port:9001
113
114    [10.1.6.104_3]
115    ip:10.1.6.104
116    port:9002
117
118    [10.1.6.104_4]
119    ip:10.1.6.104
120    port:9003
121
122Test Execution and Reporting
123----------------------------
124
125For every test run testrunner creates a temp folder and dumps the logs and
126xunit reports in the newly generated folder.
127
128for instance if you run
129
130    $ python testrunner.py -i resources/jenkins/single-node-centos-32.ini -t setgettests.MembaseBucket.value_100b -p your_first_parameter=x,your_second_parameter=y
131
132you will see this summary after each test is ran:
133
134    summary so far suite setgettests.MembaseBucket , pass 1 , fail 0
135    logs and results are available under tmp-12-11-47
136
137and logs:
138
139    $ ls  tmp-12-11-47/
140    report-12-11-47.xml-setgettests.MembaseBucket.xml	value_100b.log
141
142Development
143-----------
144
145When using git on Linux/OSX systems, you might run into issues where git
146incorrectly believes Windows-related files have been modified. In reality, git
147is merely mis-treating CRLF line endings.
148Try the following...
149
150    $ cd testrunner
151    $ git config core.autocrlf false
152
153Running End to End Tests Before Submitting a Patch to Gerrit
154------------------------------------------------------------
155
156Testrunner project has different test suites which can be run priori to
157submitting the code to gerrit. There are test suites that can be run against a
158single node which validates basic database operations such as persistence and
159bucket management. There are also key-value clustering related test cases which
160can be run against Membase/Couchbase 1.8 multiple nodes. Recently we have also
161been adding more tests which validates basic view functionalities on a cluster
162and on a single.
163
164### "make e2e-kv-single-node"
165
166This make target will start ns_server using ```cluster_run -n1``` and run all the
167test cases listed in conf/py-all-dev.conf. The test runtime can vary between
16815-30 minutes depending on your machine.
169
170Testrunner prints out a human readable pass/fail of the tests. Please refer to
171the "rerunning test" section for more information of how to re-run one single
172test against cluster_run.
173
174### "make test-views"
175
176This make target will start four ns_server(s) using ```cluster_run -n4``` and
177runs all the test cases listed in conf/py-view.conf. The test runtime can vary
178between 30-45 minutes. py-view.conf contains basic test cases which validates
179clustering operations such as rebalancing.Each test is also parametrized so you
180can easily modify this run list and change the number of docs or the
181load_duration.
182
183For instance, ```test_get_view_during_x_min_load_y_working_set,num-docs=10000,load-time=1,run-view-time=1```
184test will create a view, inserts 1000 documents, mutate those documents for 1
185minute and run view queries in parallel to the load for 1 min. You can easily
186change the parameters there to insert 1M items and keep the load running for 10
187mins for example.
188
189Using mcsoda to load json documents on to a node/cluster
190--------------------------------------------------------
191
192mcsoda is a load generator tool that randomly generates json
193documents. It is available under testrunner/lib/perf_engines/ .
194
195Mcsoda uses moxi port to distribute load in a cluster.
196The moxi can be set up on the server side or on a client side as follows:
197
198### "server-side moxi"
199
200From a client, load can be run on a cluster represented by: xx.xx.xx.xxx as:
201
202	lib/perf_engines/mcsoda.py xx.xx.xx.xxx:11211 vbuckets=1024 doc-gen=0 doc-cache=0
203	ratio-creates=0.5 ratio-sets=0.2 ratio-deletes=0.02  ratio-expirations=0.03
204	expiration=3600 min-value-size=2,4 threads=100 max-items=18000000
205	exit-after-creates=1 prefix=KEY_ max-creates=18000000
206
207This is going to ensure that there will be 20% sets against 80% gets.
208Of the 20% sets, 50% will be creates and the rest updates.
209There shall be 2% deletes, and minimum item size varies from 2 to 4 Bytes.
2103% of the items set will be marked as expired after a duration of 1 hour.
211The tool will use 100 threads to generate the json load, and every item will be
212prefixed by "KEY_". max_creates limits the no. of items created. All the load
213generated is going to get loaded on bucket "default". Prefix the IP by
214"bucket_name:password@", to load on to standard/sasl buckets.
215
216For more options in mcsoda, type in:
217```lib/perf_engines/mcsoda.py -h```
218
219### "client-side moxi"
220
221Set up the moxi for a specific bucket on the client side as follows:
222(Make sure couchbase-server is installed on the client)
223
224	/opt/couchbase/bin/moxi -Z usr=Administrator,pwd=password,port_listen=11611,
225	concurrency=1024,wait_queue_timeout=200,connect_timeout=400,connect_max_errors=3,
226	connect_retry_interval=30000,auth_timeout=100,downstream_conn_max=128,downstream_timeout=5000,
227	cycle=200 http://xx.xx.xx.xxx:8091/pools/default/bucketsStreaming/[bucket_name] -d
228
229This command sets up moxi for the cluster on the client side at port 11611.
230For more information on the moxi command, type in:
231```/opt/couchbase/bin/moxi -h```
232
233Once the moxi is all set up, run mcsoda against the moxi just set up:
234
235	lib/perf_engines/mcsoda.py localhost:11611 vbuckets=1024 doc-gen=0 doc-cache=0
236	ratio-creates=0.5 ratio-sets=0.2 ratio-deletes=0.02  min-value-size=2,4 threads=100
237	max-items=18000000 exit-after-creates=1 prefix=KEY_ max-creates=18000000
238