Fix include-what-you-use issues

Build with CB_ENABLE_HEADER_INCLUDE_CHECK enabled and fix various
header issues which have crept in since we last ran.

Change-Id: I310df28c6684cf

Fix include-what-you-use issues

Build with CB_ENABLE_HEADER_INCLUDE_CHECK enabled and fix various
header issues which have crept in since we last ran.

Change-Id: I310df28c6684cf0d8800312bc9c711b6418c373e
Reviewed-on: http://review.couchbase.org/115593
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Richard de Mellow <richard.demellow@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@couchbase.com>

show more ...

MB-33151: Remove hifi mfu config param

2 bit LRU support was left as a dynamic config option to allow
easy testing and for customers to revert the eviction algorithm
changes if the h

MB-33151: Remove hifi mfu config param

2 bit LRU support was left as a dynamic config option to allow
easy testing and for customers to revert the eviction algorithm
changes if the hifi MFU changes caused issues. Hifi MFU has been
around long enough that we want to remove some complexity and get
rid of 2 bit LRU support. Remove the hifi_mfu config parameter
entirely and the 2 bit LRU code. The DcpProducer can still talk to
older nodes and send 2 bit LRU hotness values instead of hifi MFU
hotness values if the DcpConsumer does not hello with supports hifi
MFU.

Change-Id: Ibdbd39a070205911514d189eb00f2a8190e97373
Reviewed-on: http://review.couchbase.org/105338
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>

show more ...

MB-29915: Provide the ability to switch HT eviction policy at runtime

The hifi_mfu eviction policy is new and has yet to be vigously tested in
"real-life" customer environments. Therefo

MB-29915: Provide the ability to switch HT eviction policy at runtime

The hifi_mfu eviction policy is new and has yet to be vigously tested in
"real-life" customer environments. Therefore we may want to revert to
the original 2-bit_lru policy if a serious issue is found with it.

Given that we currently maintain the counters for the 2-bit_lru policy
and the hfifi_mfu policy, we can switch policies at runtime. Although
this will not be encouraged (or documented for the end-user) it provides
a fail-safe for Support if the new policy is found to have a severe
adverse effect.

The long term plan is that once we have gained confidence in the new
hifi_mfu eviction policy we will remove the old 2-bit_lru policy and the
ability to switch between the two policies.

Change-Id: Ie6b72ce98ddb05a1aef38468286b7cf9336172d5
Reviewed-on: http://review.couchbase.org/94842
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-29013: Add temporal behaviour to hifi_mfu eviction policy

The YSCB workload D is a highly temporal workload. The current
hfi_mfu eviction policy only relies on execution frequency to

MB-29013: Add temporal behaviour to hifi_mfu eviction policy

The YSCB workload D is a highly temporal workload. The current
hfi_mfu eviction policy only relies on execution frequency to make
eviction decisions.

We are therefore introducing a temporal element into the eviction
decision. The rule is that if an item has an execution frequency
below the freq_counter_age_threshold (defaulted to 1) then its
eviction decision will be based solely on being below the execution
frequency (and if its eligible for eviction).

If an item has an execution counter greater than or equal to
freq_counter_age_threshold then we also take age into account. We only
evict if the item frequency count is less than or equal to the frequency
threshold and the age is greater than or equal to the ageThreshold.

The ageThreshold is generated using an approach similar to the frequency
threshold. We maintain an ageHistogram, and set the ageThreshold from
reading out a value from the histogram at a given percentage. The
percentage is a configuration parameter (defaulted to 20%).

Change-Id: I21378d6a8260c4e9d8908712fcbd4d763794c054
Reviewed-on: http://review.couchbase.org/94286
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

MB-29333: Only decay frequency for items that are eligible for eviction

The original patch did not address the issue correctly.
(see Change-Id: I95d328238176a97d7ac9e80c268c639bbb647953)

MB-29333: Only decay frequency for items that are eligible for eviction

The original patch did not address the issue correctly.
(see Change-Id: I95d328238176a97d7ac9e80c268c639bbb647953)

It only decrements the frequency count if the item is NOT elgible for
eviction (from the max u8int_t value of 255). This behaviour is
wrong for two reasons:

1) We want to decrement the frequency count of items that were eligible
for eviction except that their frequency count is above the threshold.

2) We don't want to decrement the frequency count that are not eligible
for eviction (regardless of what their frequency count is).

This patch corrects the behaviour to only decrement the fequency count
of items that are eligible for eviction except that their frequency
count is above the threshold.

The original associated test was also found to not exercise the newly
added code. This has been addressed and an additional test added.

Change-Id: I508964dbf9aefd8055ee7e654caca10d1450aa5b
Reviewed-on: http://review.couchbase.org/93338
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-29333: Decay frequency for items considered for eviction

The issue is we are unable to evict when there is only one document in
a vbucket.

The hifi_mfu algorithm will current

MB-29333: Decay frequency for items considered for eviction

The issue is we are unable to evict when there is only one document in
a vbucket.

The hifi_mfu algorithm will currently only evict the first document it
visits if its execution frequency count is zero. But documents are
initially added with an initialFreqCount of 4.

The solution is to decrement a document's frequncy count when it visited
but not evicted. Therefore after sufficient multiple passes of
running the hifi_mfu algorithm the document will be evicted (if it is
eligible).

Change-Id: I95d328238176a97d7ac9e80c268c639bbb647953
Reviewed-on: http://review.couchbase.org/93038
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>

show more ...

Allow replica vbuckets to be visited before active vbuckets

This functionality is to be used by the hifi_mfu hash table eviction
algorithm, allowing items to be evicted from all the repl

Allow replica vbuckets to be visited before active vbuckets

This functionality is to be used by the hifi_mfu hash table eviction
algorithm, allowing items to be evicted from all the replica buckets
before evicting from the active buckets.

Change-Id: If5f3f33845f9accf8938fe492d43d2f65ade792c
Reviewed-on: http://review.couchbase.org/92048
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Tim Bradgate <tim.bradgate@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...

MB-29236: Add max value to ItemEviction histogram if not eligible

We currently are adding the max value (255) to the ItemEviction
histogram if a storedValue is eligible to be evicted, bu

MB-29236: Add max value to ItemEviction histogram if not eligible

We currently are adding the max value (255) to the ItemEviction
histogram if a storedValue is eligible to be evicted, but just exceeds
the threshold frequency.

We should only be adding the max value if the storedValue is not
eligible to be evicted.

Change-Id: I4d3768cc8678f03aeb5f152dd5439ebccde0f4ec
Reviewed-on: http://review.couchbase.org/92306
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

show more ...