Commit graph

98 commits

Author SHA1 Message Date
Timo Tijhof
4938038175 tests: Remove use of wfRandomString() for test fixtures
This reduces confidence in the test. There is no guruantee that
it won't return the same value twice during the duration of a full
PHPUnit run of all test suites, whether twice in a row or 20 minutes
apart.

For a test that needs a string of any kind, use an explicit, consinstent
and cheap literal value.

For a test that specifically needs some kind of uniqueness compared to
something else within the same test case, do so explicitly.

Tests that require something globally unique (for some undefined/vague
definition of "global") were not found, and should not exist anyway.

Also, in libs/objectcache tests, fix order of parameters in some
assertions (expected first, then actual), and use assertFalse/assertSame
instead of assertEqual for cases where false is expected to remove
tolerance of other loosely equal values.

Change-Id: Ifc60e88178da471330b94bfbf12e2731d2efc77d
2019-05-06 17:22:03 +00:00
Aaron Schulz
7ff911ba11 objectcache: Remove hacky testMerge_fork() method in BagOStuffTest
This method easily fails due to use of the same resource in each
thread after calling pcntl_fork(). This can cause thread-safety
problems like mysqli errors, which in turn make tests fail.

Change-Id: I4ff3657940c1d1cdad58bafb7449bdd0182cbe56
2019-03-28 19:39:05 -07:00
Aaron Schulz
962495b17a objectcache: merge getWithToken() into doGet() for simplicity
Change-Id: I581f866521e1086ca350973d9cdeff6656f48fe8
2019-03-27 13:03:25 -07:00
jenkins-bot
0fd9e748cc Merge "objectcache: defined some missing methods in MultiWriteBagOStuff" 2019-03-27 18:34:30 +00:00
Aaron Schulz
25fd7dbc7b objectcache: defined some missing methods in MultiWriteBagOStuff
Also, reorder some method definitions to match the base class.
This makes it easier to compare the two. In addition, refactor
the doWrite() method to make the calls clearer.

Change-Id: I795b395b9c54645f62962461a28067ae38b291ed
2019-03-27 17:26:43 +00:00
Aaron Schulz
861683b722 objectcache: fix race condition in SqlBagOStuff::changeTTL
Also fix --use-bagostuff parameter when given 0

Change-Id: I64971d4ccad2c856cc7688577ae564c8db068819
2019-03-26 17:06:21 -07:00
Aaron Schulz
e626d014ac objectcache: remove BagOStuff::mergeViaLock() and update RESTBagOStuff
All subclasses are now using the CAS variant, which respects $attempts.

Change-Id: Ia7ec6a7f3337cabe95c54c1142c3c5464c1794e7
2019-03-24 00:35:23 -07:00
Aaron Schulz
e2b03d5d30 objectcache: add BagOStuff::deleteMulti() method for consistency
Also:
* Make the BagOStuff tests actually pass for memcached and sql
* Add more unit tests for BagOStuff
* Make SqlBagOStuff::add() more atomic

Change-Id: Ic1eec0990a66b595b57c646498c3bd229442230c
2019-03-14 23:12:30 +00:00
Aaron Schulz
bcf563ef38 objectcache: add metrics for WAN cache deletes and check key touches/resets
Change-Id: I3dc707af53e480b27b7349aca53292f3bb26c45a
2019-03-06 08:48:47 -08:00
Aaron Schulz
b707afa7f4 objectcache: optimize WAN cache key updates during HOLDOFF_TTL
Avoid the ADD operation spam from all threads trying to access
a tombstoned key by checking the interim value cache timestamp.
This also avoids the GET/CAS spam from threads that manage to
get the mutex. If a single thread repeatedly accesses the same
tombstoned value in rapid succession, there will significantly
less cache operation spam.

Do the same for cache updates to keys in the holdoff state
due to "check keys" or the "touchedCallback" function.

Relatedly, fix getWithSetCallback() to disregard interim values
set prior to or at the same time as the latest delete() call.
This can slightly reduce the chance of the cache being behind
replica DBs for a second. It also avoids unit test failures
were a series of deletes and cache access happen at the same
timestamp (via time injection or regular system time calls).

In addition:
* Add PASS_BY_REF flag with backwards compatibility to avoid
  bloating the signature of get()/getMulti() with the new
  tombstone information needed for the above changes.
* Avoid confusing pass-by-reference in getInterimValue() and
  fix use of incorrect $asOf parameter.
* Move some more logic into setInterimValue().
* Update some comments regarding broadcasted operations that
  were not true for the currently assumed mcrouter setup.
* Rename $cValue => $curValue and $versioned => $needsVersion
  for better readability.

Bug: T203786
Change-Id: I0eb3f9b697193d39a70dd3c0967311ad7e194f20
2019-03-04 10:00:29 +00:00
Aaron Schulz
f50ea02994 objectcache: simplify WAN cache unwrap() method by removing FLG_STALE
Instead, make the "high read lag with lockTSE" case just lower FLD_TTL.
This also avoids constant regenerations by threads getting the mutex.

Added logging for TTL adjustments in set() for the lockTSE case.

Also remove some delete() calls from tests that were not needed.

Change-Id: Id7695f0377235e4a2f6e0efc88e870c8a990c3b0
2019-02-27 18:57:45 +00:00
Aaron Schulz
5414aee495 objectcache: remove dangling WAN cache EventRelayer references
* Remove 'channels' field references from config/setup
* Remove 'relayer'/'channels' field reference in unit tests
* Remove unused DEFAULT_PURGE_CHANNEL class constant
* Also remove long-since bogus 'pool' field references

Follow-up to 4753b0a4ed

Change-Id: If6670ff4e1dccc8ae253a08b46d205601da10024
2019-02-16 20:22:32 -08:00
Aaron Schulz
e4930d255b objectcache: avoid occasional test flakiness due to microtime()
* Fix the timestamps to static hard-coded values
* Force the timestamps before various get/set tests so they do not
  use the microtime() value either.
* Remove the direct and duplicated (causing further risk) microtime()
  calls from testGetWithSeveralCheckKeys().

Bug: T207247
Change-Id: Id30a8127f11501dbe54e075b6e9d18490353f4a5
2019-02-04 21:35:42 -08:00
Umherirrender
7c4ceae4e7 Add missing @covers to backend related tests
Change-Id: I6ecbedf7d4f6c48b4f17331c31588f2c8b772054
2019-02-01 22:08:32 +01:00
Aaron Schulz
2190b78987 objectcache: add expiration check callback to WANObjectCache::getWithSetCallback
This is useful when the timestamps to be checked depend on the value or are stored
in the database rather than as check keys.

Change-Id: I81ab08a943ee7d2f96a132d371965501941ed37f
2018-12-21 19:25:26 +00:00
James D. Forrester
903e8b63de doc: Modernise parameter names and documentation for 'replica' DBs
Non-breaking change. Remaining uses are public interfaces (a constant, two
globals, a config sub-parameter, SQL queries, storage function names), one i18n
message key, and a whole lot of maintenance scripts with calls to the deprecated
function wfWaitForSlaves().

Change-Id: I6ee5ca92ccf6a80c08f53d9efe38ebb4b05064d7
2018-10-31 10:36:48 -07:00
jenkins-bot
55ffac2024 Merge "objectcache: add "epoch" parameter to WANObjectCache" 2018-08-08 23:16:40 +00:00
Aaron Schulz
4af7fbd76a objectcache: add "epoch" parameter to WANObjectCache
This can be used to discard all values before a certain point in
time, such as periods of severe network problems or misconfiguration
that resulted in missed purges.

Change-Id: I612db8f91a5960b912e9f35645a3d3872df47460
2018-08-06 16:43:50 -07:00
Aaron Schulz
b8779134ed objectcache: make BagOStuff::mergeViaLock() timeout more sensible
Interpret "1 attempt" as non-blocking and use 3 seconds otherwise
(instead of 6 seconds). This means that the WAN cache merge call
does not block.

Bug: T198239
Change-Id: Ieb194a949f18785c2cc9dd20fdc4d2e3fed51abf
2018-07-12 16:02:38 +01:00
Aaron Schulz
6612407f9c objectcache: make MultiWriteBagOStuff handle duplicate add() operations
Bug: T198280
Change-Id: Ib1bcde2b3fbfb452f80d8d840c494be2eb70eb87
2018-06-28 16:04:35 +00:00
Aaron Schulz
7a730a6f9e objectcache: update MemcachedPeclBagOStuff for pecl memcached 3.0.0
The get() $cas_token parameter was changed into $flags, which can act
as a switch to make the return value an associative array of details.

pecl and PHP-based memcached BagOStuff class both pass all tests now.

Bug: T196125
Change-Id: I4678250331a48db4d50d1fc6c489c991a4dee920
2018-06-02 17:25:22 -07:00
jenkins-bot
285080c311 Merge "objectcache: make RedisBagOStuff pass all tests" 2018-06-02 01:35:26 +00:00
Aaron Schulz
acace9a049 objectcache: add setMockTime() method to BagOStuff/WANObjectCache
Change-Id: I3e5760814fb7dbe628eb0d979d690c3275fc3c15
2018-06-01 03:46:58 +00:00
Aaron Schulz
13f7232bf4 objectcache: make RedisBagOStuff pass all tests
* Provide a default lock-based BagOStuff::cas implementation
* Make RedisBagOStuff::merge() use mergeViaCas()
* Use the raw unserialized string as the redis CAS token to
  avoid any bad interaction with __wakeup() methods changing
  field values every time
* Make RedisBagOStuff::incr() return false when there is no
  such key, not null
* Rewrite merge() test to make the order of write/cas phase
  of the parent and child merge() calls well defined instead
  of arbitrary usleep() calls
* Avoid cache key reuse in test runs

Change-Id: I388ec173cf3858bb2fc7a8c8a00cda68703074ce
2018-05-31 20:43:10 -07:00
Kunal Mehta
b4925e34d0 tests: Enable PHPUnit 4/6 compat layer in some tests that need it
Change-Id: I27a21fa9e97414fae02acbefb28011f0275cba63
2018-04-07 19:31:24 -07:00
Umherirrender
63d96c15fd build: Updating mediawiki/mediawiki-codesniffer to 16.0.0
Change-Id: I59b59f79bbf3ce4feff3b3a20c1c31bc16370531
2018-02-17 13:29:13 +01:00
Aaron Schulz
7206bd3468 objectcache: use region prefixes for mcrouter-backed WAN cache
This allows for mcrouter to have proper cross-DC and intra-DC timeouts.

Change-Id: If48f740f435d266a2050839f34611c0c8f36b3a7
2018-02-01 10:09:44 -08:00
Umherirrender
45da581551 Use ::class to resolve class names in tests
This helps to find renamed or misspelled classes earlier.
Phan will check the class names

Change-Id: Ie541a7baae10ab6f5c13f95ac2ff6598b8f8950c
2018-01-26 22:49:13 +01:00
Kunal Mehta
75160bdd3b Use MediaWikiCoversValidator for tests that don't use MediaWikiTestCase
Change-Id: I8c4de7e9c72c9969088666007b54c6fd23f6cc13
2018-01-01 08:28:02 +00:00
Kunal Mehta
cf0ebed65a Add @covers tags to objectcache tests
Change-Id: I5db623a582a0424f50d939aaa14fe29da9358b4f
2017-12-24 23:29:00 -08:00
jenkins-bot
d965b71fbf Merge "objectcache: Fix HashBagOStuffTest test in PHP 7.1" 2017-12-01 22:30:15 +00:00
Aaron Schulz
b29ee7a960 objectcache: add WANObjectCache::getMultiCheckKeyTime method
Change-Id: I5675fabc5aa70f72659ce02d68caae88be20e06d
2017-12-01 21:45:32 +00:00
Timo Tijhof
7dbc5b0fe6 objectcache: Fix HashBagOStuffTest test in PHP 7.1
> There was 1 error:
>
> 1) HashBagOStuffTest::testEvictionAdd
> A non-numeric value encountered
>
> tests/phpunit/includes/libs/objectcache/HashBagOStuffTest.php:106

This warning is new in PHP 7.1, however the behaviour is not new.
This code was already not behaving as it should have. Concatenation
preceeds addition/multiplication. As such, this was producing
(int)-10 each time instead of (str)"key0" through (str)"key10".

Change-Id: Ibb1a59e373740772f02dfec77ee7ebd9d181d852
2017-12-01 13:28:36 -08:00
Aaron Schulz
73b928750c objectcache: Run preemptive WAN cache refreshes post-send
This keeps HTTP request time consistent in case of expensive keys

Change-Id: I0746fde29a6e2f27d1b92f1af599c741d5972f46
2017-12-01 08:15:26 +00:00
Aaron Schulz
e90eafdf61 objectcache: Make WANObjectCache interim caching not interfere with ChronologyProtector
Also removed useless line from testLockTSE(). That would have needed
to be using $this->internalCache and those locks are freed immediately.

Bug: T180035
Change-Id: Ida1a923f779aaf8410da76643457d2200da6cb20
2017-11-30 23:54:22 +00:00
Aaron Schulz
ea22e3d1f6 objectcache: only give current format keys getWithSetCallback() callbacks
Callback code that happens to make use of $oldValue might not be able
to handle missing, new, or changed fields due to key version changes.

Overhaul testGetWithSetCallback_versions() to be cleaner and cover the
case of unversioned => versioned keys.

Change-Id: If108a73078c530c985d30bdadcbfa9ddd53dc2be
2017-11-28 21:26:06 +00:00
Aaron Schulz
300c655377 Use time forcing methods to avoid WANObjectCacheTest flakeiness
Use of microtime() is now just for baselines, and it is no longer
assumed to be increasing with each call. Such an assumption is
particuliarly bad on Windows.

I've done 100X run rounds with now failures on Windows.

Change-Id: Ica2a47982495bc95b10ca507414972744ea9507e
2017-11-28 06:50:44 +00:00
Aaron Schulz
0d7137099a objectcache: add "graceTTL" option to WANObjectCache::getWithSetCallback()
Also made worthRefreshExpiring() fully match the method documentation.

Change-Id: I48a4b1b9d006de100389b47c03c1185a9316abfc
2017-11-27 17:11:13 -08:00
Aaron Schulz
fce6862e46 objectcache: add "staleTTL" into WANObjectCache::getWithSetCallback()
This simply involves passing it through to the set() call

Also added some related commons to adaptiveTTL() involving
usage of this option.

Change-Id: Id5833a5d4efb6cad2eb646832e5b0188e86e12fc
2017-11-26 21:49:47 +00:00
Aaron Schulz
1772c4fe34 objectcache: add some WAN cache preemptive refresh tests
Added some extra sanity checks to WANObjectCache

Change-Id: Iac511b0cc1fc8d57ac98e9d7f2cacbcddc1c6db9
2017-11-22 18:38:53 -08:00
Aaron Schulz
4432e898be Add statsd metric support to WANObjectCache
Bug: T178531
Change-Id: I3037281d09cd5195347789f544deae89711f128b
2017-10-25 01:41:31 +00:00
Aaron Schulz
6b2f13b055 objectcache: Always use interim values on WAN cache tombstones
This stores values for very short times while the main
value is a tombstone in case of particularly high traffic.

Also make mutex keys expire immediately on unlock.

Change-Id: I4ec5cf7f8b49239fdd2518e5d955534877a0f7ee
2017-10-24 00:32:06 +00:00
Aaron Schulz
0fe50ad759 objectcache: Split off some code in WANObjectCache::getWithSetCallback
This makes it a bit easier to follow

Change-Id: I67968814ab046473eb8eca4086a8600c77417b82
2017-10-20 17:12:21 +00:00
Timo Tijhof
6c8f3b9683 objectcache: Improve WANObjectCache test coverage
class WANObjectCache (methods: 40% -> 67%, lines: 65% -> 88%)

* Allow indirect coverage of protected/private utility methods from
  existing tests.
* Add basic test for setLogger().
* Add basic test for newEmpty().
* Add basic test for getQoS().

Change-Id: Ifb79ed2ff1febbd2f5477b8ed6319992ce88eb29
2017-09-28 18:59:09 +01:00
Timo Tijhof
71f7b56520 objectcache: Improve WANObjectCache test coverage
class WANObjectCache (methods: 30% -> 40%, lines: 62% -> 65%)

* doGetWithSetCallback: 98% -> 100%
 - Add case: Throw for invalid callback.
* getWithSetCallback: 86% -> 100%
 - Cover case: Else branch for `$cur[self::VFLD_VERSION] === $version`.
* getMulti: 91% -> 98%
 - Cover case: If branch for `$this->warmupCache`.
* set: 93% -> 100%
 - Cover case: If branch for `lockTSE >= 0`.
* reap: 84% -> 100%
 - Add case: Error when changeTTL() fails.

Change-Id: I4844ab0ebdd4d3ec9acc9cd4500721b04cfda317
2017-09-26 00:22:23 +01:00
Umherirrender
86463f3df6 build: Updating mediawiki/mediawiki-codesniffer to 0.12.0
Change-Id: Iefaae5043fa77d5d556c31079549dab8f61bd3ef
2017-09-10 21:11:37 +02:00
Umherirrender
3f1a52805e Use short type bool/int in param documentation
Enable the phpcs sniffs for this and used phpcbf

Change-Id: Iaa36687154ddd2bf663b9dd519f5c99409d37925
2017-08-20 13:20:59 +02:00
Umherirrender
5544cef16b Add missing type to @param documentation
Change-Id: I6b2c9c7af9a281fe457099cc3a336a60a25e74aa
2017-08-11 20:37:35 +02:00
Umherirrender
43e8cd0579 Make phpcs pass
Reenable MediaWiki.WhiteSpace.SpaceBeforeClassBrace.NoSpaceBeforeBrace,
because the mentioned bug is fixed

Bug: T172933
Change-Id: I1593bdba2295ebed401b921f2beabed69dba7638
2017-08-11 16:22:42 +02:00
Brad Jorsch
edf69e62d8 MultiWriteBagOStuff: Fix async writes of mutable objects
If someone writes an object into a BagOStuff, they typically expect that
later changes to the object will not affect the value stored.
MultiWriteBagOStuff's async write handling was violating this
expectation, which is potentially causing T168040.

Bug: T168040
Change-Id: Ie897b900befdc8998614af06f9339cd07665703e
2017-07-25 02:08:00 +00:00