MB-9133 Use native index builder for index builds

Just use the new external program in couchstore to do the
initial index builds.

Sample test:

Building an index for a d

MB-9133 Use native index builder for index builds

Just use the new external program in couchstore to do the
initial index builds.

Sample test:

Building an index for a design document with 4 views, from
one of the evperf tests, for a dataset of 6.5M documents:

Before this change:

Total indexing time: 10m58.060s
Btree build phase time: 127.85283s

After this change:

Indexing time: 9m1.305s
Btree build phase time: 22.018846s

Change-Id: Ibec96b74ca219483694ebf71323af921c23dc58e
Reviewed-on: http://review.couchbase.org/29033
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

MB-7882 Optimize btree node decoding

Using the +bin_opt_info flag, the Erlang compiler told the
decode function was not binary optimized:

/opt/r14b04/bin/erlc -I../../src +bin_o

MB-7882 Optimize btree node decoding

Using the +bin_opt_info flag, the Erlang compiler told the
decode function was not binary optimized:

/opt/r14b04/bin/erlc -I../../src +bin_opt_info couch_btree.erl;
./couch_btree.erl:70: Warning: NOT OPTIMIZED: sub binary used by binary:copy/1
./couch_btree.erl:433: Warning: NOT OPTIMIZED: called function decode_node/3 does not begin with a suitable binary matching instruction
./couch_btree.erl:442: Warning: NOT OPTIMIZED: called function decode_node/3 does not begin with a suitable binary matching instruction

After this change, the compiler tells it is now optimized:

/opt/r14b04/bin/erlc -I../../src +bin_opt_info couch_btree.erl;
./couch_btree.erl:73: Warning: NOT OPTIMIZED: sub binary used by binary:copy/1
./couch_btree.erl:436: Warning: OPTIMIZED: creation of sub binary delayed
./couch_btree.erl:447: Warning: OPTIMIZED: creation of sub binary delayed

Change-Id: Ie14a85a76a4e00f1f6aac99e19218fbd9841022c
Reviewed-on: http://review.couchbase.org/25394
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

MB-7882 Simple btree optimizations

This change removes some inefficiencies in the btree module:

1) Building unnecessary tuples when the extract and assemble
functions are the

MB-7882 Simple btree optimizations

This change removes some inefficiencies in the btree module:

1) Building unnecessary tuples when the extract and assemble
functions are the default ones (as is the case for views).
For this case, just don't build any tuples nor do any
assemble/extract function calls;

2) Traversing a list of KVs and applying the assemble function
to build a new list - this is waste of time and cpu when
the assemble function is the default one (as in the case for
views);

3) Traversing the same list 2 times and reversing an accumulator
when encoding a kp node. This can be done in a single pass
with the list comprehension expression and avoiding the need
to keep a list accumulator and reverse it at the end.

This was found it profiling data from fprof. Nearly no impact on
view updates (due to existing pipelining), however view compaction
had gains of about 6% regarding time. Very small impact as well
on query response time.

Change-Id: I0b9b8e013deede1bdba86f4f1ba2d4cc052bbbb4
Reviewed-on: http://review.couchbase.org/25353
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

MB-7413 More efficient and faster incremental updates

At any time, insert larger batches of kv pairs into the
btrees. These batches are on disk sorted files, and while
they're applie

MB-7413 More efficient and faster incremental updates

At any time, insert larger batches of kv pairs into the
btrees. These batches are on disk sorted files, and while
they're applied to the btrees, the next batch is prepared
in parallel (maps, conversion to binaries, sorting, etc).
This reduces index update time, slows down fragmentation
and requires less disk space.

This has a very significant positive impact if the number
of changes to apply to an index are large, such as the case
when adding a group of new partitions (active, passive of
replica) to an index.

Small/medium scale performance test descriptions follows.

Test 1:

Initially 3.2M documents, design document with 4 views (both
from evperf lucky8 tests), index was built and later more
3.2M documents were added (new 512 partitions added to the
index as active).

Initial build took: 11m17.911s
Index file size after initial build: 380Mb

** Before this change

Incremental update took: 75m43.575s
Index file size after incremental update: 30.0Gb

** After this change

Incremental update took: 18m36.762s
Index file size after incremental update: 4.0Gb

Test 2:

Initially 3.2M documents (from lucky8 evperf test), design
document with a single view (emits meta.id key and null value
for each document), index was built and later more 3.2M documents
were added (new 512 partitions added to the index as active).

Initial build took: 6m23.199s
Index file size after initial build: 134Mb

** Before this change

Incremental update took: 21m31.615s
Index file size after incremental update: 8.6Gb

** After this change

Incremental update took: 7m13.394s
Index file size after incremental update: 1.8Gb

Change-Id: I66fde0adeb0f7a810c125bf7cb42cdd17dd03d9d
Reviewed-on: http://review.couchbase.org/23674
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>

show more ...

MB-7131 Tweak btree node splits

Instead of creating several full nodes and one not
completely full, make all nodes not more than half
full.

Experiments (including evperf dat

MB-7131 Tweak btree node splits

Instead of creating several full nodes and one not
completely full, make all nodes not more than half
full.

Experiments (including evperf datasets) showed that
this is benefical for incremental index updates
(random inserts/updates), as it makes fragmentation
grows more slowly and speedups a little inserts/updates
and btree lookups.

For an evperf dataset, incrementally updating an index (ddoc
with 4 views) that had 1 million items (per view) and a data
size of 167Mb, with 1 million more new items (like it happens
after a rebalance-out):

** Before this change

Update duration: 26m50.270s
Final file size: 12 513 Mb

** After this change

Update duration: 25m43.893s
Final file size: 11 640 Mb

With a more comprehensive test, results follow.

Before

1> btree_bench:test(2000000, 500000, 1000, "/home/fdmanana/tmp/foo.couch").
Creating btree with initial number of items 2000000, batch size 5000, random batches
Btree created in 276553.336 ms
Btree stats after creation:
btree_size 16684244
file_size 1317839559
fragmentation 98.73397001280942
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2000000
kp_nodes 628
kv_nodes 32865
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 107
min_elements_per_kp_node 1
avg_elements_per_kp_node 53.3312101910828
elements_per_kp_node_90_percentile 96
elements_per_kp_node_95_percentile 99
elements_per_kp_node_99_percentile 106
max_elements_per_kv_node 162
min_elements_per_kv_node 2
avg_elements_per_kv_node 60.85501293169025
elements_per_kv_node_90_percentile 131
elements_per_kv_node_95_percentile 145
elements_per_kv_node_99_percentile 159
max_kp_node_size 4602
min_kp_node_size 44
avg_kp_node_size 2294.2420382165606
kp_node_size_90_percentile 4129
kp_node_size_95_percentile 4258
kp_node_size_99_percentile 4559
max_compressed_kp_node_size 2038
min_compressed_kp_node_size 43
avg_compressed_kp_node_size 1034.7420382165606
compressed_kp_node_size_90_percentile 1825
compressed_kp_node_size_95_percentile 1882
compressed_kp_node_size_99_percentile 2010
max_kv_node_size 4375
min_kv_node_size 55
avg_kv_node_size 1644.0853491556368
kv_node_size_90_percentile 3538
kv_node_size_95_percentile 3916
kv_node_size_99_percentile 4294
max_compressed_kv_node_size 1239
min_compressed_kv_node_size 37
avg_compressed_kv_node_size 479.60921953445916
compressed_kv_node_size_90_percentile 991
compressed_kv_node_size_95_percentile 1094
compressed_kv_node_size_99_percentile 1193
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2000 keys)
Percentiles for single key lookups, 90% -> 1.539ms, 95% -> 6.763ms, 99% -> 12.805ms

Compacting btree
Btree compaction took 57355.094 ms
Btree stats after compaction:
btree_size 15269217
file_size 15269217
fragmentation 0.0
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2000000
kp_nodes 123
kv_nodes 12500
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 105
min_elements_per_kp_node 2
avg_elements_per_kp_node 102.6178861788618
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 160
min_elements_per_kv_node 160
avg_elements_per_kv_node 160.0
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4516
min_kp_node_size 87
avg_kp_node_size 4413.569105691057
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 1686
min_compressed_kp_node_size 72
avg_compressed_kp_node_size 1347.2113821138212
compressed_kp_node_size_90_percentile 1394
compressed_kp_node_size_95_percentile 1401
compressed_kp_node_size_99_percentile 1417
max_kv_node_size 4321
min_kv_node_size 4321
avg_kv_node_size 4321.0
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1228
min_compressed_kv_node_size 1183
avg_compressed_kv_node_size 1199.90384
compressed_kv_node_size_90_percentile 1210
compressed_kv_node_size_95_percentile 1211
compressed_kv_node_size_99_percentile 1214
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2000 keys)
Percentiles for single key lookups, 90% -> 1.395ms, 95% -> 1.567ms, 99% -> 1.88ms

Starting incremental inserts of 500000 new items, in random batches of size 5000
Incremental inserts took 36789.824 ms
Btree stats after inserts:
btree_size 19447730
file_size 180187724
fragmentation 89.20696173508468
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2500000
kp_nodes 320
kv_nodes 20791
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 106
min_elements_per_kp_node 1
avg_elements_per_kp_node 65.96875
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 162
min_elements_per_kv_node 2
avg_elements_per_kv_node 120.24433649175124
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4559
min_kp_node_size 44
avg_kp_node_size 2837.65625
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 1995
min_compressed_kp_node_size 44
avg_compressed_kp_node_size 1029.378125
compressed_kp_node_size_90_percentile 1394
compressed_kp_node_size_95_percentile 1457
compressed_kp_node_size_99_percentile 1551
max_kv_node_size 4375
min_kv_node_size 55
avg_kv_node_size 3247.5970852772834
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1228
min_compressed_kv_node_size 39
avg_compressed_kv_node_size 911.1964792458275
compressed_kv_node_size_90_percentile 1209
compressed_kv_node_size_95_percentile 1210
compressed_kv_node_size_99_percentile 1213
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2500 keys)
Percentiles for single key lookups, 90% -> 1.47ms, 95% -> 1.603ms, 99% -> 4.404ms

Starting incremental updates of every 5th item in the btree, in batches of size 5000 (500000 items), random order
Incremental updates took 61793.899 ms
Btree stats after updates:
btree_size 19447730
file_size 180187724
fragmentation 89.20696173508468
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2500000
kp_nodes 320
kv_nodes 20791
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 106
min_elements_per_kp_node 1
avg_elements_per_kp_node 65.96875
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 162
min_elements_per_kv_node 2
avg_elements_per_kv_node 120.24433649175124
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4559
min_kp_node_size 44
avg_kp_node_size 2837.65625
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 1995
min_compressed_kp_node_size 44
avg_compressed_kp_node_size 1029.378125
compressed_kp_node_size_90_percentile 1394
compressed_kp_node_size_95_percentile 1457
compressed_kp_node_size_99_percentile 1551
max_kv_node_size 4375
min_kv_node_size 55
avg_kv_node_size 3247.5970852772834
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1228
min_compressed_kv_node_size 39
avg_compressed_kv_node_size 911.1964792458275
compressed_kv_node_size_90_percentile 1209
compressed_kv_node_size_95_percentile 1210
compressed_kv_node_size_99_percentile 1213
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2500 keys)
Percentiles for single key lookups, 90% -> 1.053ms, 95% -> 1.232ms, 99% -> 4.041ms

Compacting btree
Btree compaction took 12494.651 ms
Btree stats after compaction:
btree_size 19088176
file_size 19088176
fragmentation 0.0
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2500000
kp_nodes 152
kv_nodes 15625
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 105
min_elements_per_kp_node 2
avg_elements_per_kp_node 103.78947368421052
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 160
min_elements_per_kv_node 160
avg_elements_per_kv_node 160.0
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4516
min_kp_node_size 87
avg_kp_node_size 4463.9473684210525
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 1686
min_compressed_kp_node_size 75
avg_compressed_kp_node_size 1361.5394736842106
compressed_kp_node_size_90_percentile 1392
compressed_kp_node_size_95_percentile 1400
compressed_kp_node_size_99_percentile 1417
max_kv_node_size 4321
min_kv_node_size 4321
avg_kv_node_size 4321.0
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1228
min_compressed_kv_node_size 1183
avg_compressed_kv_node_size 1200.02208
compressed_kv_node_size_90_percentile 1210
compressed_kv_node_size_95_percentile 1211
compressed_kv_node_size_99_percentile 1214
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2500 keys)
Percentiles for single key lookups, 90% -> 1.018ms, 95% -> 1.169ms, 99% -> 1.498ms

ok
2>

After

1> btree_bench:test(2000000, 500000, 1000, "/home/fdmanana/tmp/foo.couch").
Creating btree with initial number of items 2000000, batch size 5000, random batches
Btree created in 268182.733 ms
Btree stats after creation:
btree_size 16422278
file_size 1218231774
fragmentation 98.65195783343606
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2000000
kp_nodes 662
kv_nodes 28530
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 107
min_elements_per_kp_node 1
avg_elements_per_kp_node 44.095166163142
elements_per_kp_node_90_percentile 78
elements_per_kp_node_95_percentile 93
elements_per_kp_node_99_percentile 102
max_elements_per_kv_node 162
min_elements_per_kv_node 1
avg_elements_per_kv_node 70.10164738871363
elements_per_kv_node_90_percentile 136
elements_per_kv_node_95_percentile 146
elements_per_kv_node_99_percentile 158
max_kp_node_size 4602
min_kp_node_size 44
avg_kp_node_size 1897.0921450151056
kp_node_size_90_percentile 3355
kp_node_size_95_percentile 4000
kp_node_size_99_percentile 4387
max_compressed_kp_node_size 2225
min_compressed_kp_node_size 43
avg_compressed_kp_node_size 882.0135951661631
compressed_kp_node_size_90_percentile 1529
compressed_kp_node_size_95_percentile 1776
compressed_kp_node_size_99_percentile 1977
max_kv_node_size 4375
min_kv_node_size 28
avg_kv_node_size 1893.7444794952683
kv_node_size_90_percentile 3673
kv_node_size_95_percentile 3943
kv_node_size_99_percentile 4267
max_compressed_kv_node_size 1234
min_compressed_kv_node_size 25
avg_compressed_kv_node_size 546.8209954433929
compressed_kv_node_size_90_percentile 1028
compressed_kv_node_size_95_percentile 1103
compressed_kv_node_size_99_percentile 1189
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2000 keys)
Percentiles for single key lookups, 90% -> 1.301ms, 95% -> 4.603ms, 99% -> 12.706ms

Compacting btree
Btree compaction took 62111.127 ms
Btree stats after compaction:
btree_size 15269217
file_size 15269217
fragmentation 0.0
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2000000
kp_nodes 123
kv_nodes 12500
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 105
min_elements_per_kp_node 2
avg_elements_per_kp_node 102.6178861788618
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 160
min_elements_per_kv_node 160
avg_elements_per_kv_node 160.0
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4516
min_kp_node_size 87
avg_kp_node_size 4413.569105691057
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 1686
min_compressed_kp_node_size 72
avg_compressed_kp_node_size 1347.2113821138212
compressed_kp_node_size_90_percentile 1394
compressed_kp_node_size_95_percentile 1401
compressed_kp_node_size_99_percentile 1417
max_kv_node_size 4321
min_kv_node_size 4321
avg_kv_node_size 4321.0
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1228
min_compressed_kv_node_size 1183
avg_compressed_kv_node_size 1199.90384
compressed_kv_node_size_90_percentile 1210
compressed_kv_node_size_95_percentile 1211
compressed_kv_node_size_99_percentile 1214
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2000 keys)
Percentiles for single key lookups, 90% -> 1.324ms, 95% -> 1.45ms, 99% -> 1.849ms

Starting incremental inserts of 500000 new items, in random batches of size 5000
Incremental inserts took 36604.463 ms
Btree stats after inserts:
btree_size 19431715
file_size 171302728
fragmentation 88.65650580882752
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2500000
kp_nodes 279
kv_nodes 20419
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 106
min_elements_per_kp_node 1
avg_elements_per_kp_node 74.18279569892474
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 162
min_elements_per_kv_node 1
avg_elements_per_kv_node 122.43498702189137
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4559
min_kp_node_size 44
avg_kp_node_size 3190.8602150537636
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 2018
min_compressed_kp_node_size 45
avg_compressed_kp_node_size 1159.6810035842293
compressed_kp_node_size_90_percentile 1592
compressed_kp_node_size_95_percentile 1790
compressed_kp_node_size_99_percentile 1923
max_kv_node_size 4375
min_kv_node_size 28
avg_kv_node_size 3306.744649591067
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1230
min_compressed_kv_node_size 29
avg_compressed_kv_node_size 927.4621675890103
compressed_kv_node_size_90_percentile 1209
compressed_kv_node_size_95_percentile 1210
compressed_kv_node_size_99_percentile 1213
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2500 keys)
Percentiles for single key lookups, 90% -> 1.115ms, 95% -> 1.264ms, 99% -> 3.103ms

Starting incremental updates of every 5th item in the btree, in batches of size 5000 (500000 items), random order
Incremental updates took 58408.701 ms
Btree stats after updates:
btree_size 19431715
file_size 171302728
fragmentation 88.65650580882752
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2500000
kp_nodes 279
kv_nodes 20419
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 106
min_elements_per_kp_node 1
avg_elements_per_kp_node 74.18279569892474
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 162
min_elements_per_kv_node 1
avg_elements_per_kv_node 122.43498702189137
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4559
min_kp_node_size 44
avg_kp_node_size 3190.8602150537636
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 2018
min_compressed_kp_node_size 45
avg_compressed_kp_node_size 1159.6810035842293
compressed_kp_node_size_90_percentile 1592
compressed_kp_node_size_95_percentile 1790
compressed_kp_node_size_99_percentile 1923
max_kv_node_size 4375
min_kv_node_size 28
avg_kv_node_size 3306.744649591067
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1230
min_compressed_kv_node_size 29
avg_compressed_kv_node_size 927.4621675890103
compressed_kv_node_size_90_percentile 1209
compressed_kv_node_size_95_percentile 1210
compressed_kv_node_size_99_percentile 1213
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2500 keys)
Percentiles for single key lookups, 90% -> 1.433ms, 95% -> 1.599ms, 99% -> 3.0ms

Compacting btree
Btree compaction took 12399.026 ms
Btree stats after compaction:
btree_size 19088176
file_size 19088176
fragmentation 0.0
kv_chunk_threshold 7168
kp_chunk_threshold 6144
kv_count 2500000
kp_nodes 152
kv_nodes 15625
max_depth 4
min_depth 4
avg_depth 4.0
depth_90_percentile 4
depth_95_percentile 4
depth_99_percentile 4
max_reduction_size 8
min_reduction_size 8
avg_reduction_size 8.0
reduction_size_90_percentile 8
reduction_size_95_percentile 8
reduction_size_99_percentile 8
max_elements_per_kp_node 105
min_elements_per_kp_node 2
avg_elements_per_kp_node 103.78947368421052
elements_per_kp_node_90_percentile 105
elements_per_kp_node_95_percentile 105
elements_per_kp_node_99_percentile 105
max_elements_per_kv_node 160
min_elements_per_kv_node 160
avg_elements_per_kv_node 160.0
elements_per_kv_node_90_percentile 160
elements_per_kv_node_95_percentile 160
elements_per_kv_node_99_percentile 160
max_kp_node_size 4516
min_kp_node_size 87
avg_kp_node_size 4463.9473684210525
kp_node_size_90_percentile 4516
kp_node_size_95_percentile 4516
kp_node_size_99_percentile 4516
max_compressed_kp_node_size 1686
min_compressed_kp_node_size 75
avg_compressed_kp_node_size 1361.5394736842106
compressed_kp_node_size_90_percentile 1392
compressed_kp_node_size_95_percentile 1400
compressed_kp_node_size_99_percentile 1417
max_kv_node_size 4321
min_kv_node_size 4321
avg_kv_node_size 4321.0
kv_node_size_90_percentile 4321
kv_node_size_95_percentile 4321
kv_node_size_99_percentile 4321
max_compressed_kv_node_size 1228
min_compressed_kv_node_size 1183
avg_compressed_kv_node_size 1200.02208
compressed_kv_node_size_90_percentile 1210
compressed_kv_node_size_95_percentile 1211
compressed_kv_node_size_99_percentile 1214
max_key_size 16
min_key_size 16
avg_key_size 16.0
key_size_90_percentile 16
key_size_95_percentile 16
key_size_99_percentile 16
max_value_size 6
min_value_size 6
avg_value_size 6.0
value_size_90_percentile 6
value_size_95_percentile 6
value_size_99_percentile 6

Doing key lookups for every 1000th key (2500 keys)
Percentiles for single key lookups, 90% -> 0.943ms, 95% -> 1.106ms, 99% -> 1.396ms

ok

Change-Id: Iee6f4b9a3974e7bca1adf37ff2f0efc384bf4b1c
Reviewed-on: http://review.couchbase.org/22369
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>

show more ...

MB-7130 Refactor btree stats

Move the code that computes the statistics into a
separate module and make it easier to add new metrics.
Also add 90%, 95% and 99% percentiles to all met

MB-7130 Refactor btree stats

Move the code that computes the statistics into a
separate module and make it easier to add new metrics.
Also add 90%, 95% and 99% percentiles to all metrics.

Example:

$ curl -s http://localhost:9500/_set_view/default/_design/C/_btree_stats | json
{
"id_btree": {
"btree_size": 73351175,
"file_size": 13003147447,
"fragmentation": 99.43589676807885,
"kv_chunk_threshold": 7168,
"kp_chunk_threshold": 6144,
"kv_count": 1986453,
"kp_nodes": 2589,
"kv_nodes": 48685,
"max_depth": 5,
"min_depth": 5,
"avg_depth": 5,
"depth_90_percentile": 5,
"depth_95_percentile": 5,
"depth_99_percentile": 5,
"max_reduction_size": 133,
"min_reduction_size": 133,
"avg_reduction_size": 133,
"reduction_size_90_percentile": 133,
"reduction_size_95_percentile": 133,
"reduction_size_99_percentile": 133,
"max_elements_per_kp_node": 32,
"min_elements_per_kp_node": 2,
"avg_elements_per_kp_node": 19.80417149478563,
"elements_per_kp_node_90_percentile": 31,
"elements_per_kp_node_95_percentile": 32,
"elements_per_kp_node_99_percentile": 32,
"max_elements_per_kv_node": 68,
"min_elements_per_kv_node": 32,
"avg_elements_per_kv_node": 40.80215672178289,
"elements_per_kv_node_90_percentile": 48,
"elements_per_kv_node_95_percentile": 51,
"elements_per_kv_node_99_percentile": 62,
"max_kp_node_size": 5377,
"min_kp_node_size": 337,
"avg_kp_node_size": 3328.100811123986,
"kp_node_size_90_percentile": 5209,
"kp_node_size_95_percentile": 5377,
"kp_node_size_99_percentile": 5377,
"max_compressed_kp_node_size": 4039,
"min_compressed_kp_node_size": 264,
"avg_compressed_kp_node_size": 2116.863267670915,
"compressed_kp_node_size_90_percentile": 3207,
"compressed_kp_node_size_95_percentile": 3264,
"compressed_kp_node_size_99_percentile": 3396,
"max_kv_node_size": 6055,
"min_kv_node_size": 2931,
"avg_kv_node_size": 3789.253918044572,
"kv_node_size_90_percentile": 4419,
"kv_node_size_95_percentile": 4701,
"kv_node_size_99_percentile": 5829,
"max_compressed_kv_node_size": 2312,
"min_compressed_kv_node_size": 1047,
"avg_compressed_kv_node_size": 1385.282078668994,
"compressed_kv_node_size_90_percentile": 1608,
"compressed_kv_node_size_95_percentile": 1711,
"compressed_kv_node_size_99_percentile": 2076,
"max_key_size": 16,
"min_key_size": 16,
"avg_key_size": 16,
"key_size_90_percentile": 16,
"key_size_95_percentile": 16,
"key_size_99_percentile": 16,
"max_value_size": 73,
"min_value_size": 53,
"avg_value_size": 71.84445290172987,
"value_size_90_percentile": 73,
"value_size_95_percentile": 73,
"value_size_99_percentile": 73
},
"experts2": {
"btree_size": 41358068,
"file_size": 13003147447,
"fragmentation": 99.68193802178608,
"kv_chunk_threshold": 7168,
"kp_chunk_threshold": 6144,
"kv_count": 1986453,
"kp_nodes": 1576,
"kv_nodes": 32524,
"max_depth": 5,
"min_depth": 5,
"avg_depth": 5,
"depth_90_percentile": 5,
"depth_95_percentile": 5,
"depth_99_percentile": 5,
"max_reduction_size": 133,
"min_reduction_size": 133,
"avg_reduction_size": 133,
"reduction_size_90_percentile": 133,
"reduction_size_95_percentile": 133,
"reduction_size_99_percentile": 133,
"max_elements_per_kp_node": 31,
"min_elements_per_kp_node": 4,
"avg_elements_per_kp_node": 21.63642131979696,
"elements_per_kp_node_90_percentile": 30,
"elements_per_kp_node_95_percentile": 31,
"elements_per_kp_node_99_percentile": 31,
"max_elements_per_kv_node": 123,
"min_elements_per_kv_node": 1,
"avg_elements_per_kv_node": 61.07652810232444,
"elements_per_kv_node_90_percentile": 100,
"elements_per_kv_node_95_percentile": 104,
"elements_per_kv_node_99_percentile": 111,
"max_kp_node_size": 5581,
"min_kp_node_size": 719,
"avg_kp_node_size": 3889.072335025381,
"kp_node_size_90_percentile": 5368,
"kp_node_size_95_percentile": 5546,
"kp_node_size_99_percentile": 5580,
"max_compressed_kp_node_size": 4954,
"min_compressed_kp_node_size": 225,
"avg_compressed_kp_node_size": 2479.73540609137,
"compressed_kp_node_size_90_percentile": 3377,
"compressed_kp_node_size_95_percentile": 3576,
"compressed_kp_node_size_99_percentile": 4395,
"max_kv_node_size": 5088,
"min_kv_node_size": 38,
"avg_kv_node_size": 2548.569979092363,
"kv_node_size_90_percentile": 4171,
"kv_node_size_95_percentile": 4324,
"kv_node_size_99_percentile": 4639,
"max_compressed_kv_node_size": 2275,
"min_compressed_kv_node_size": 40,
"avg_compressed_kv_node_size": 1142.756887221744,
"compressed_kv_node_size_90_percentile": 1851,
"compressed_kv_node_size_95_percentile": 1922,
"compressed_kv_node_size_99_percentile": 2057,
"max_key_size": 28,
"min_key_size": 23,
"avg_key_size": 27.71111322543247,
"key_size_90_percentile": 28,
"key_size_95_percentile": 28,
"key_size_99_percentile": 28,
"max_value_size": 9,
"min_value_size": 9,
"avg_value_size": 9,
"value_size_90_percentile": 9,
"value_size_95_percentile": 9,
"value_size_99_percentile": 9
},
"category": {
"btree_size": 97917630,
"file_size": 13003147447,
"fragmentation": 99.24696977867008,
"kv_chunk_threshold": 7168,
"kp_chunk_threshold": 6144,
"kv_count": 1986453,
"kp_nodes": 2345,
"kv_nodes": 46057,
"max_depth": 5,
"min_depth": 5,
"avg_depth": 5,
"depth_90_percentile": 5,
"depth_95_percentile": 5,
"depth_99_percentile": 5,
"max_reduction_size": 133,
"min_reduction_size": 133,
"avg_reduction_size": 133,
"reduction_size_90_percentile": 133,
"reduction_size_95_percentile": 133,
"reduction_size_99_percentile": 133,
"max_elements_per_kp_node": 30,
"min_elements_per_kp_node": 4,
"avg_elements_per_kp_node": 20.64008528784648,
"elements_per_kp_node_90_percentile": 30,
"elements_per_kp_node_95_percentile": 30,
"elements_per_kp_node_99_percentile": 30,
"max_elements_per_kv_node": 69,
"min_elements_per_kv_node": 34,
"avg_elements_per_kv_node": 43.1303167813796,
"elements_per_kv_node_90_percentile": 50,
"elements_per_kv_node_95_percentile": 53,
"elements_per_kv_node_99_percentile": 64,
"max_kp_node_size": 5551,
"min_kp_node_size": 740,
"avg_kp_node_size": 3813.549680170576,
"kp_node_size_90_percentile": 5541,
"kp_node_size_95_percentile": 5546,
"kp_node_size_99_percentile": 5549,
"max_compressed_kp_node_size": 5063,
"min_compressed_kp_node_size": 246,
"avg_compressed_kp_node_size": 2451.555223880597,
"compressed_kp_node_size_90_percentile": 3443,
"compressed_kp_node_size_95_percentile": 3487,
"compressed_kp_node_size_99_percentile": 4355,
"max_kv_node_size": 5989,
"min_kv_node_size": 2948,
"avg_kv_node_size": 3784.008098660356,
"kv_node_size_90_percentile": 4394,
"kv_node_size_95_percentile": 4653,
"kv_node_size_99_percentile": 5629,
"max_compressed_kv_node_size": 3182,
"min_compressed_kv_node_size": 1526,
"avg_compressed_kv_node_size": 1992.260590138307,
"compressed_kv_node_size_90_percentile": 2299,
"compressed_kv_node_size_95_percentile": 2425,
"compressed_kv_node_size_99_percentile": 2945,
"max_key_size": 33,
"min_key_size": 28,
"avg_key_size": 32.71111322543247,
"key_size_90_percentile": 33,
"key_size_95_percentile": 33,
"key_size_99_percentile": 33,
"max_value_size": 50,
"min_value_size": 50,
"avg_value_size": 50,
"value_size_90_percentile": 50,
"value_size_95_percentile": 50,
"value_size_99_percentile": 50
},
"realm3": {
"btree_size": 95096840,
"file_size": 13003147447,
"fragmentation": 99.2686629111328,
"kv_chunk_threshold": 7168,
"kp_chunk_threshold": 6144,
"kv_count": 1986453,
"kp_nodes": 2401,
"kv_nodes": 45082,
"max_depth": 5,
"min_depth": 5,
"avg_depth": 5,
"depth_90_percentile": 5,
"depth_95_percentile": 5,
"depth_99_percentile": 5,
"max_reduction_size": 133,
"min_reduction_size": 133,
"avg_reduction_size": 133,
"reduction_size_90_percentile": 133,
"reduction_size_95_percentile": 133,
"reduction_size_99_percentile": 133,
"max_elements_per_kp_node": 30,
"min_elements_per_kp_node": 4,
"avg_elements_per_kp_node": 19.7759266972095,
"elements_per_kp_node_90_percentile": 30,
"elements_per_kp_node_95_percentile": 30,
"elements_per_kp_node_99_percentile": 30,
"max_elements_per_kv_node": 70,
"min_elements_per_kv_node": 35,
"avg_elements_per_kv_node": 44.06310722683111,
"elements_per_kv_node_90_percentile": 51,
"elements_per_kv_node_95_percentile": 54,
"elements_per_kv_node_99_percentile": 65,
"max_kp_node_size": 5491,
"min_kp_node_size": 732,
"avg_kp_node_size": 3614.363182007497,
"kp_node_size_90_percentile": 5468,
"kp_node_size_95_percentile": 5486,
"kp_node_size_99_percentile": 5490,
"max_compressed_kp_node_size": 4847,
"min_compressed_kp_node_size": 239,
"avg_compressed_kp_node_size": 2353.94543940025,
"compressed_kp_node_size_90_percentile": 3425,
"compressed_kp_node_size_95_percentile": 3476,
"compressed_kp_node_size_99_percentile": 4476,
"max_kv_node_size": 5951,
"min_kv_node_size": 2963,
"avg_kv_node_size": 3777.697972583293,
"kv_node_size_90_percentile": 4380,
"kv_node_size_95_percentile": 4634,
"kv_node_size_99_percentile": 5578,
"max_compressed_kv_node_size": 3169,
"min_compressed_kv_node_size": 1513,
"avg_compressed_kv_node_size": 1975.109467193115,
"compressed_kv_node_size_90_percentile": 2280,
"compressed_kv_node_size_95_percentile": 2401,
"compressed_kv_node_size_99_percentile": 2882,
"max_key_size": 31,
"min_key_size": 26,
"avg_key_size": 30.71111322543247,
"key_size_90_percentile": 31,
"key_size_95_percentile": 31,
"key_size_99_percentile": 31,
"max_value_size": 50,
"min_value_size": 50,
"avg_value_size": 50,
"value_size_90_percentile": 50,
"value_size_95_percentile": 50,
"value_size_99_percentile": 50
},
"realm2": {
"btree_size": 52542690,
"file_size": 13003147447,
"fragmentation": 99.59592329307839,
"kv_chunk_threshold": 7168,
"kp_chunk_threshold": 6144,
"kv_count": 1986453,
"kp_nodes": 1545,
"kv_nodes": 30876,
"max_depth": 5,
"min_depth": 5,
"avg_depth": 5,
"depth_90_percentile": 5,
"depth_95_percentile": 5,
"depth_99_percentile": 5,
"max_reduction_size": 133,
"min_reduction_size": 133,
"avg_reduction_size": 133,
"reduction_size_90_percentile": 133,
"reduction_size_95_percentile": 133,
"reduction_size_99_percentile": 133,
"max_elements_per_kp_node": 30,
"min_elements_per_kp_node": 4,
"avg_elements_per_kp_node": 20.98381877022654,
"elements_per_kp_node_90_percentile": 26,
"elements_per_kp_node_95_percentile": 28,
"elements_per_kp_node_99_percentile": 30,
"max_elements_per_kv_node": 118,
"min_elements_per_kv_node": 1,
"avg_elements_per_kv_node": 64.336474931986,
"elements_per_kv_node_90_percentile": 94,
"elements_per_kv_node_95_percentile": 98,
"elements_per_kv_node_99_percentile": 105,
"max_kp_node_size": 5489,
"min_kp_node_size": 732,
"avg_kp_node_size": 3834.906796116505,
"kp_node_size_90_percentile": 4759,
"kp_node_size_95_percentile": 5119,
"kp_node_size_99_percentile": 5477,
"max_compressed_kp_node_size": 5220,
"min_compressed_kp_node_size": 241,
"avg_compressed_kp_node_size": 2607.493851132686,
"compressed_kp_node_size_90_percentile": 3218,
"compressed_kp_node_size_95_percentile": 3375,
"compressed_kp_node_size_99_percentile": 4833,
"max_kv_node_size": 5169,
"min_kv_node_size": 42,
"avg_kv_node_size": 2877.555415209224,
"kv_node_size_90_percentile": 4218,
"kv_node_size_95_percentile": 4372,
"kv_node_size_99_percentile": 4675,
"max_compressed_kv_node_size": 2834,
"min_compressed_kv_node_size": 44,
"avg_compressed_kv_node_size": 1562.440082912294,
"compressed_kv_node_size_90_percentile": 2275,
"compressed_kv_node_size_95_percentile": 2363,
"compressed_kv_node_size_99_percentile": 2525,
"max_key_size": 31,
"min_key_size": 26,
"avg_key_size": 30.71111322543247,
"key_size_90_percentile": 31,
"key_size_95_percentile": 31,
"key_size_99_percentile": 31,
"max_value_size": 9,
"min_value_size": 9,
"avg_value_size": 9,
"value_size_90_percentile": 9,
"value_size_95_percentile": 9,
"value_size_99_percentile": 9
}
}

Change-Id: I353f68901475eb07a390e59793e7af0b191a50b7
Reviewed-on: http://review.couchbase.org/22368
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>

show more ...

MB-6591: Use raw collation for _all_docs query params

_all_docs are always sorted in raw order, not Unicode. Hence
the query parameters (startkey and endkey) also need to
take this r

MB-6591: Use raw collation for _all_docs query params

_all_docs are always sorted in raw order, not Unicode. Hence
the query parameters (startkey and endkey) also need to
take this raw collation into account. The query validation checks
whether startkey is smaller than endkey, hence we need to take
the collation type (Unicode or raw) into account.

An example for raw collation is:

Foo < bar < foo < zoo

With Unicode collation it would be:

bar < foo < Foo < zoo

This patch is based on a patch for Apache CouchDB [1].

I've kept the original commit message below.

[1] https://github.com/apache/couchdb/commit/46d2dc647f786fa1920f365af89ce4d0070ef280

Fix empty range check for raw collation.

The check for empty ranges was not taking into account the
view option for raw collation. This fixes that by passing
the couch_btree:less/2 function into the check.

Patch by: Jason Smith
Re: COUCHDB-1228 4/4

Change-Id: I4569683ff0253221f2f265e287b7a52d2ad32ac1
git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1156509 13f79535-47bb-0310-9956-ffa450edef68
Reviewed-on: http://review.couchbase.org/21359
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>

show more ...

MB-6866 Use different thresholds for kp and kv nodes

For views, since values are mostly raw JSON, compression
gains are high, therefore chunk threshold (uncompressed
size) can be rai

MB-6866 Use different thresholds for kp and kv nodes

For views, since values are mostly raw JSON, compression
gains are high, therefore chunk threshold (uncompressed
size) can be raised, which decreases the number of kv nodes
in the trees. For kp nodes, because they all have a 1024
bits bitmask in their reductions, their branching factor was
a bit low, which made the trees deeper.

These changes made queries and compaction faster both for
generic btree tests and evperf lucky8 views (lucky8.conf and
lucky8-3d.conf). Inserts and updates also get a small speedup.

Change-Id: I2a94ec6005b881c81fd5295a2c89c7657cc6b19f
Reviewed-on: http://review.couchbase.org/21484
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>

show more ...

MB-6489 Add couch_btree:stats/1 for debugging

Traverses the entire tree to collect several useful stats
for developers, meant for debugging only. Example of stats
computed:

MB-6489 Add couch_btree:stats/1 for debugging

Traverses the entire tree to collect several useful stats
for developers, meant for debugging only. Example of stats
computed:

btree_size 117741553
file_size 208359643
fragmentation 43.49119085407532
chunk_threshold 5120
kp_nodes 3155
kv_nodes 188596
max_depth 7
min_depth 3
max_reduction_size 8
min_reduction_size 8
max_elements_per_kp_node 172
min_elements_per_kp_node 1
avg_elements_per_kp_node 60.776545166402535
max_elements_per_kv_node 102
min_elements_per_kv_node 38
avg_elements_per_kv_node 58.32573331353793
kv_count 11000000
min_kp_node_size 44
min_kp_node_size_compressed 46
max_kp_node_size 7397
max_kp_node_size_compressed 2755
avg_kp_node_size 2614.391442155309
avg_kp_node_size_compressed 971.2161648177496
min_kv_node_size 1045
min_kv_node_size_compressed 389
max_kv_node_size 2805
max_kv_node_size_compressed 1011
avg_kv_node_size 1604.9576661222932
avg_kv_node_size_compressed 599.7721160576046

Change-Id: Ic10067f339f88c04738e65af4ef480b79a02a23c
Reviewed-on: http://review.couchbase.org/20350
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

MB-6096 Fix too high memory consumption during indexing

Several things where causing the indexer to consume too much
memory:

1) The use of binary mode btrees, introduced by CBD-

MB-6096 Fix too high memory consumption during indexing

Several things where causing the indexer to consume too much
memory:

1) The use of binary mode btrees, introduced by CBD-426, was
creating #doc_info records with IDs and Revs as sub-binary
references, which were preventing the binaries read from disk
to be released soon, as these records in turn were passed
to several view processes (work queues, etc). The solution
here is to copy those sub-binary using binary:copy/1.

2) Reduce the size of the map queue. Doing it helps reducing the
memory consumption and doesn't seem to affect the index build
time;

3) Use enif_alloc() and enif_free() in the mapreduce NIF. Using
C++'s new operator or C's malloc/calloc/etc is perfectly fine
but less efficient. Using enif_alloc() overall memory consumptiom
is significantly lower;

4) Some minor optimizations regarding handling and matching of
binaries, as explained by:
http://www.erlang.org/doc/man/binary.html

Change-Id: I49cbfbd4c4be06cc6de1451c18aabcdb5b100581
Reviewed-on: http://review.couchbase.org/19430
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

CBD-426 Reject writes if keys/values/reductions are too large

Rather than accept the accept the writes, which will overflow the
size fields, and then find later hard to debug failures at

CBD-426 Reject writes if keys/values/reductions are too large

Rather than accept the accept the writes, which will overflow the
size fields, and then find later hard to debug failures at read
time - which can be very similar to file corruption issues and
therefore difficult to distinguish.

Change-Id: Ia2f54c185ceff1b1ca7078aaabdd979bb2dea52c
Reviewed-on: http://review.couchbase.org/19014
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

CBD-417 Ability to skip btree branches when folding it

Based on the reduction value of a non-leaf node and a user
callback, it's now possible to skip entire btree branches
efficientl

CBD-417 Ability to skip btree branches when folding it

Based on the reduction value of a non-leaf node and a user
callback, it's now possible to skip entire btree branches
efficiently.

Change-Id: Icb8cfae75b6c8e6dc07cdf8c8fc41fb04f87e135
Reviewed-on: http://review.couchbase.org/17812
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>

show more ...

New storage file format that uses no erlang terms.

For better integration with Couchstore, the core storage engine
no longer uses Erlang terms for persisting data to disk, but instead

New storage file format that uses no erlang terms.

For better integration with Couchstore, the core storage engine
no longer uses Erlang terms for persisting data to disk, but instead
uses a pure binary format that is more easily used non-erlang storage
libraries, so that they no longer have any dependencies on Erlang term
conversion.

Change-Id: I121885b487d71966672a8f2a1e86875effcd3842
Reviewed-on: http://review.couchbase.org/13557
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>

show more ...

Fix fold reduce with pagination

Paginated fold reduce, with inclusive_end=false and one
of startkey_docid or endkey_docid was producing wrong
reduce values.
For views, it ignored

Fix fold reduce with pagination

Paginated fold reduce, with inclusive_end=false and one
of startkey_docid or endkey_docid was producing wrong
reduce values.
For views, it ignored completely the doc ID component of the
keys.

Change-Id: Ifd6796f251caef38acda8a218757865aea5c8004
Reviewed-on: http://review.couchbase.org/13368
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

Making doc_info and body info simpler, for use by couchstore

Simplifying the storage to only store a single body, with content_meta
type to determine if json or raw binary (and why it's

Making doc_info and body info simpler, for use by couchstore

Simplifying the storage to only store a single body, with content_meta
type to determine if json or raw binary (and why it's not json), and if compressed.

We are now snappy compressing all persisted erlang terms, doc bodies must be
compressed in a separate step.

Removing couch_btree_nif, as we will be using a native updater in ep-engine.

Change-Id: I6822bbc271861748049bca5ebac7ca0f5fe816d4
Reviewed-on: http://review.couchbase.org/13090
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>

show more ...

Remove unncessary backwards compatibility code

Change-Id: I960e1b4c6a833e9c245e50ad1c816b586b9322f2
Reviewed-on: http://review.couchbase.org/13033
Reviewed-by: Damien Katz <damien@co

Remove unncessary backwards compatibility code

Change-Id: I960e1b4c6a833e9c245e50ad1c816b586b9322f2
Reviewed-on: http://review.couchbase.org/13033
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

Fix btree guided purge function

When the guide function asked to stop and it was in
the middle of a kp node, it would return a new kp
node with unsorted keys. This made subsequent bt

Fix btree guided purge function

When the guide function asked to stop and it was in
the middle of a kp node, it would return a new kp
node with unsorted keys. This made subsequent btree
operations to fail or have unpredictable behaviour.

This relates to MB-4518

Change-Id: I76ce4b8541bf333dbdaf6413e06cdd01e84dc1b7
Reviewed-on: http://review.couchbase.org/12767
Tested-by: buildbot <qe@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>

show more ...

Allow btree chunk threshold to be configurable

Change-Id: Ic8ed424daa4aff5ae08903ac793a3ab3e1f00123
Reviewed-on: http://review.couchbase.org/12164
Reviewed-by: Damien Katz <damien@co

Allow btree chunk threshold to be configurable

Change-Id: Ic8ed424daa4aff5ae08903ac793a3ab3e1f00123
Reviewed-on: http://review.couchbase.org/12164
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

Add couch_btree functions add_remove/5 query_modify/6

These functions combine updates, insertions and deletions (by key)
with deletions by value. This reuses the helper function of the

Add couch_btree functions add_remove/5 query_modify/6

These functions combine updates, insertions and deletions (by key)
with deletions by value. This reuses the helper function of the
already existing function guided_purge/3.

The caller specifies 2 new extra arguments:
1) A purge function;
2) An accumulator for the purge function

The purge function is invoked against Key/Value pairs or against
the reduction value of intermediate (non-leaf) nodes to hint the
tree traversal code whether to go explore or not tree branches.

Change-Id: If17f47a4c18716587068fa26dc769b080d51a844
Reviewed-on: http://review.couchbase.org/10356
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

Add "set view" modules

These are the base for adding the superstart btree indexer and
querying capabilities.

Change-Id: Ie0b3cc5aa7012efaa3e61456bff84479822d3519
Reviewed-on

Add "set view" modules

These are the base for adding the superstart btree indexer and
querying capabilities.

Change-Id: Ie0b3cc5aa7012efaa3e61456bff84479822d3519
Reviewed-on: http://review.couchbase.org/9368
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

Make btree guided purge more responsive to stop order

If the purge callback tells us to stop, we shouldn't traverse
all the remaining child nodes of a branch node but instead
really

Make btree guided purge more responsive to stop order

If the purge callback tells us to stop, we shouldn't traverse
all the remaining child nodes of a branch node but instead
really stop the recursion immediately.

Change-Id: Ib5912b078005bc907e90ebd5cc37d11f4f8bfe32
Reviewed-on: http://review.couchbase.org/9745
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

Add couch_btree:guided_purge/3

The purpose of this function is remove values from
a btree with the guidance of a user supplied function.
This function can take decisions to skip tree

Add couch_btree:guided_purge/3

The purpose of this function is remove values from
a btree with the guidance of a user supplied function.
This function can take decisions to skip tree branches
based on a branch node's reduction value.

Change-Id: I18b7970c79a5e25159e203be36c40a974749ef35
Reviewed-on: http://review.couchbase.org/9680
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>

show more ...

More efficient term size calculation

Unlike byte_size(term_to_binary(Term)), the BIF erlang:external_size/1 doesn't
do the serialization step, it only calculates the maximum external siz

More efficient term size calculation

Unlike byte_size(term_to_binary(Term)), the BIF erlang:external_size/1 doesn't
do the serialization step, it only calculates the maximum external size for
any term, which is more efficient (faster and avoids the garbage generation).

With the test couch_http_bulk_writes.sh at [1], using 20 writers and batches
of 100 1Kb documents, it's possible to write about 1 400 000 documents with
this patch instead of about 1 300 000.

[1] https://github.com/fdmanana/basho_bench_couch

Change-Id: I7cbfe1a97c6247ac0188019d31af7603ff11a3da
git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1129597 13f79535-47bb-0310-9956-ffa450edef68
Reviewed-on: http://review.couchbase.org/6689
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Jan Lehnardt <jan@apache.org>
Tested-by: Jan Lehnardt <jan@apache.org>

show more ...

Add configurable file compression (snappy, deflate or none)

Not only this makes database and view index files smaller it also increases
database read/write performance, view index genera

Add configurable file compression (snappy, deflate or none)

Not only this makes database and view index files smaller it also increases
database read/write performance, view index generation (specially for large
documents and/or documents with nested JSON structures) and compaction.
Closes COUCHDB-1120.




git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1098558 13f79535-47bb-0310-9956-ffa450edef68

show more ...

Minor refactoring regarding compression option handling