Commit graph

83 commits

Author SHA1 Message Date
James D. Forrester
903e8b63de doc: Modernise parameter names and documentation for 'replica' DBs
Non-breaking change. Remaining uses are public interfaces (a constant, two
globals, a config sub-parameter, SQL queries, storage function names), one i18n
message key, and a whole lot of maintenance scripts with calls to the deprecated
function wfWaitForSlaves().

Change-Id: I6ee5ca92ccf6a80c08f53d9efe38ebb4b05064d7
2018-10-31 10:36:48 -07:00
jenkins-bot
55ffac2024 Merge "objectcache: add "epoch" parameter to WANObjectCache" 2018-08-08 23:16:40 +00:00
Aaron Schulz
4af7fbd76a objectcache: add "epoch" parameter to WANObjectCache
This can be used to discard all values before a certain point in
time, such as periods of severe network problems or misconfiguration
that resulted in missed purges.

Change-Id: I612db8f91a5960b912e9f35645a3d3872df47460
2018-08-06 16:43:50 -07:00
Aaron Schulz
b8779134ed objectcache: make BagOStuff::mergeViaLock() timeout more sensible
Interpret "1 attempt" as non-blocking and use 3 seconds otherwise
(instead of 6 seconds). This means that the WAN cache merge call
does not block.

Bug: T198239
Change-Id: Ieb194a949f18785c2cc9dd20fdc4d2e3fed51abf
2018-07-12 16:02:38 +01:00
Aaron Schulz
6612407f9c objectcache: make MultiWriteBagOStuff handle duplicate add() operations
Bug: T198280
Change-Id: Ib1bcde2b3fbfb452f80d8d840c494be2eb70eb87
2018-06-28 16:04:35 +00:00
Aaron Schulz
7a730a6f9e objectcache: update MemcachedPeclBagOStuff for pecl memcached 3.0.0
The get() $cas_token parameter was changed into $flags, which can act
as a switch to make the return value an associative array of details.

pecl and PHP-based memcached BagOStuff class both pass all tests now.

Bug: T196125
Change-Id: I4678250331a48db4d50d1fc6c489c991a4dee920
2018-06-02 17:25:22 -07:00
jenkins-bot
285080c311 Merge "objectcache: make RedisBagOStuff pass all tests" 2018-06-02 01:35:26 +00:00
Aaron Schulz
acace9a049 objectcache: add setMockTime() method to BagOStuff/WANObjectCache
Change-Id: I3e5760814fb7dbe628eb0d979d690c3275fc3c15
2018-06-01 03:46:58 +00:00
Aaron Schulz
13f7232bf4 objectcache: make RedisBagOStuff pass all tests
* Provide a default lock-based BagOStuff::cas implementation
* Make RedisBagOStuff::merge() use mergeViaCas()
* Use the raw unserialized string as the redis CAS token to
  avoid any bad interaction with __wakeup() methods changing
  field values every time
* Make RedisBagOStuff::incr() return false when there is no
  such key, not null
* Rewrite merge() test to make the order of write/cas phase
  of the parent and child merge() calls well defined instead
  of arbitrary usleep() calls
* Avoid cache key reuse in test runs

Change-Id: I388ec173cf3858bb2fc7a8c8a00cda68703074ce
2018-05-31 20:43:10 -07:00
Kunal Mehta
b4925e34d0 tests: Enable PHPUnit 4/6 compat layer in some tests that need it
Change-Id: I27a21fa9e97414fae02acbefb28011f0275cba63
2018-04-07 19:31:24 -07:00
Umherirrender
63d96c15fd build: Updating mediawiki/mediawiki-codesniffer to 16.0.0
Change-Id: I59b59f79bbf3ce4feff3b3a20c1c31bc16370531
2018-02-17 13:29:13 +01:00
Aaron Schulz
7206bd3468 objectcache: use region prefixes for mcrouter-backed WAN cache
This allows for mcrouter to have proper cross-DC and intra-DC timeouts.

Change-Id: If48f740f435d266a2050839f34611c0c8f36b3a7
2018-02-01 10:09:44 -08:00
Umherirrender
45da581551 Use ::class to resolve class names in tests
This helps to find renamed or misspelled classes earlier.
Phan will check the class names

Change-Id: Ie541a7baae10ab6f5c13f95ac2ff6598b8f8950c
2018-01-26 22:49:13 +01:00
Kunal Mehta
75160bdd3b Use MediaWikiCoversValidator for tests that don't use MediaWikiTestCase
Change-Id: I8c4de7e9c72c9969088666007b54c6fd23f6cc13
2018-01-01 08:28:02 +00:00
Kunal Mehta
cf0ebed65a Add @covers tags to objectcache tests
Change-Id: I5db623a582a0424f50d939aaa14fe29da9358b4f
2017-12-24 23:29:00 -08:00
jenkins-bot
d965b71fbf Merge "objectcache: Fix HashBagOStuffTest test in PHP 7.1" 2017-12-01 22:30:15 +00:00
Aaron Schulz
b29ee7a960 objectcache: add WANObjectCache::getMultiCheckKeyTime method
Change-Id: I5675fabc5aa70f72659ce02d68caae88be20e06d
2017-12-01 21:45:32 +00:00
Timo Tijhof
7dbc5b0fe6 objectcache: Fix HashBagOStuffTest test in PHP 7.1
> There was 1 error:
>
> 1) HashBagOStuffTest::testEvictionAdd
> A non-numeric value encountered
>
> tests/phpunit/includes/libs/objectcache/HashBagOStuffTest.php:106

This warning is new in PHP 7.1, however the behaviour is not new.
This code was already not behaving as it should have. Concatenation
preceeds addition/multiplication. As such, this was producing
(int)-10 each time instead of (str)"key0" through (str)"key10".

Change-Id: Ibb1a59e373740772f02dfec77ee7ebd9d181d852
2017-12-01 13:28:36 -08:00
Aaron Schulz
73b928750c objectcache: Run preemptive WAN cache refreshes post-send
This keeps HTTP request time consistent in case of expensive keys

Change-Id: I0746fde29a6e2f27d1b92f1af599c741d5972f46
2017-12-01 08:15:26 +00:00
Aaron Schulz
e90eafdf61 objectcache: Make WANObjectCache interim caching not interfere with ChronologyProtector
Also removed useless line from testLockTSE(). That would have needed
to be using $this->internalCache and those locks are freed immediately.

Bug: T180035
Change-Id: Ida1a923f779aaf8410da76643457d2200da6cb20
2017-11-30 23:54:22 +00:00
Aaron Schulz
ea22e3d1f6 objectcache: only give current format keys getWithSetCallback() callbacks
Callback code that happens to make use of $oldValue might not be able
to handle missing, new, or changed fields due to key version changes.

Overhaul testGetWithSetCallback_versions() to be cleaner and cover the
case of unversioned => versioned keys.

Change-Id: If108a73078c530c985d30bdadcbfa9ddd53dc2be
2017-11-28 21:26:06 +00:00
Aaron Schulz
300c655377 Use time forcing methods to avoid WANObjectCacheTest flakeiness
Use of microtime() is now just for baselines, and it is no longer
assumed to be increasing with each call. Such an assumption is
particuliarly bad on Windows.

I've done 100X run rounds with now failures on Windows.

Change-Id: Ica2a47982495bc95b10ca507414972744ea9507e
2017-11-28 06:50:44 +00:00
Aaron Schulz
0d7137099a objectcache: add "graceTTL" option to WANObjectCache::getWithSetCallback()
Also made worthRefreshExpiring() fully match the method documentation.

Change-Id: I48a4b1b9d006de100389b47c03c1185a9316abfc
2017-11-27 17:11:13 -08:00
Aaron Schulz
fce6862e46 objectcache: add "staleTTL" into WANObjectCache::getWithSetCallback()
This simply involves passing it through to the set() call

Also added some related commons to adaptiveTTL() involving
usage of this option.

Change-Id: Id5833a5d4efb6cad2eb646832e5b0188e86e12fc
2017-11-26 21:49:47 +00:00
Aaron Schulz
1772c4fe34 objectcache: add some WAN cache preemptive refresh tests
Added some extra sanity checks to WANObjectCache

Change-Id: Iac511b0cc1fc8d57ac98e9d7f2cacbcddc1c6db9
2017-11-22 18:38:53 -08:00
Aaron Schulz
4432e898be Add statsd metric support to WANObjectCache
Bug: T178531
Change-Id: I3037281d09cd5195347789f544deae89711f128b
2017-10-25 01:41:31 +00:00
Aaron Schulz
6b2f13b055 objectcache: Always use interim values on WAN cache tombstones
This stores values for very short times while the main
value is a tombstone in case of particularly high traffic.

Also make mutex keys expire immediately on unlock.

Change-Id: I4ec5cf7f8b49239fdd2518e5d955534877a0f7ee
2017-10-24 00:32:06 +00:00
Aaron Schulz
0fe50ad759 objectcache: Split off some code in WANObjectCache::getWithSetCallback
This makes it a bit easier to follow

Change-Id: I67968814ab046473eb8eca4086a8600c77417b82
2017-10-20 17:12:21 +00:00
Timo Tijhof
6c8f3b9683 objectcache: Improve WANObjectCache test coverage
class WANObjectCache (methods: 40% -> 67%, lines: 65% -> 88%)

* Allow indirect coverage of protected/private utility methods from
  existing tests.
* Add basic test for setLogger().
* Add basic test for newEmpty().
* Add basic test for getQoS().

Change-Id: Ifb79ed2ff1febbd2f5477b8ed6319992ce88eb29
2017-09-28 18:59:09 +01:00
Timo Tijhof
71f7b56520 objectcache: Improve WANObjectCache test coverage
class WANObjectCache (methods: 30% -> 40%, lines: 62% -> 65%)

* doGetWithSetCallback: 98% -> 100%
 - Add case: Throw for invalid callback.
* getWithSetCallback: 86% -> 100%
 - Cover case: Else branch for `$cur[self::VFLD_VERSION] === $version`.
* getMulti: 91% -> 98%
 - Cover case: If branch for `$this->warmupCache`.
* set: 93% -> 100%
 - Cover case: If branch for `lockTSE >= 0`.
* reap: 84% -> 100%
 - Add case: Error when changeTTL() fails.

Change-Id: I4844ab0ebdd4d3ec9acc9cd4500721b04cfda317
2017-09-26 00:22:23 +01:00
Umherirrender
86463f3df6 build: Updating mediawiki/mediawiki-codesniffer to 0.12.0
Change-Id: Iefaae5043fa77d5d556c31079549dab8f61bd3ef
2017-09-10 21:11:37 +02:00
Umherirrender
3f1a52805e Use short type bool/int in param documentation
Enable the phpcs sniffs for this and used phpcbf

Change-Id: Iaa36687154ddd2bf663b9dd519f5c99409d37925
2017-08-20 13:20:59 +02:00
Umherirrender
5544cef16b Add missing type to @param documentation
Change-Id: I6b2c9c7af9a281fe457099cc3a336a60a25e74aa
2017-08-11 20:37:35 +02:00
Umherirrender
43e8cd0579 Make phpcs pass
Reenable MediaWiki.WhiteSpace.SpaceBeforeClassBrace.NoSpaceBeforeBrace,
because the mentioned bug is fixed

Bug: T172933
Change-Id: I1593bdba2295ebed401b921f2beabed69dba7638
2017-08-11 16:22:42 +02:00
Brad Jorsch
edf69e62d8 MultiWriteBagOStuff: Fix async writes of mutable objects
If someone writes an object into a BagOStuff, they typically expect that
later changes to the object will not affect the value stored.
MultiWriteBagOStuff's async write handling was violating this
expectation, which is potentially causing T168040.

Bug: T168040
Change-Id: Ie897b900befdc8998614af06f9339cd07665703e
2017-07-25 02:08:00 +00:00
Umherirrender
be42e09aa8 build: Prepare for mediawiki/mediawiki-codesniffer to 0.9.0
The used phpcs has a bug, so the version 0.9.0 could not be enforced at the moment.
Will be fixed in next version, see T167168

Changed:
- Remove duplicate newline at end of file
- Add space between function and ( for closures
- and -> &&, or -> ||

Change-Id: I4172fb08861729bccd55aecbd07e029e2638d311
2017-06-26 17:14:31 +00:00
Timo Tijhof
dd629564df objectcache: Forward MultiWriteBagOStuff::makeKey to primary backend
Similar to what WANObjectCache and CachedBagOStuff are already doing.

Also add missing tests for WANObjectCache (similar to those for CachedBagOStuff).

Bug: T167465
Change-Id: I1a0c9324726aa6a1b221def985773b1b819181fd
2017-06-14 18:11:55 +01:00
Aaron Schulz
83fcb86dfa objectcache: add getMultiWithUnionSetCallback() method
This supports callbacks that fetch all the missing values at once.

Change-Id: I74747cc06f97edc9163178180597e6651743b048
2017-05-30 23:34:28 +00:00
Aaron Schulz
6ddaf52d32 objectcache: respect process cache in getMultiWithSetCallback()
Backend cache queries are now avoided if they are fresh in memory.

Also factor out some code into private methods for clarity.

Change-Id: Ib343fde3dbf63f39e8cb09eca6278811a7d9738b
2017-05-26 15:34:45 -07:00
Aaron Schulz
0c24cefa12 objectcache: fix cache warmup bug in getMultiWithSetCallback()
The warmup cache was not properly prefixed and was also using the entity
IDs instead of the cache keys. Thus, it effectively just wasted a
getMulti() query and resulted in the usual separate GETs anyway.

Added some unit tests for this.

Change-Id: I75b7a31214b515511856f9d95db32e8881d80ccc
2017-05-26 04:28:13 +00:00
Gergő Tisza
525bfbc8df Switch to librarized version of TestingAccessWrapper
Replaces \TestingAccessWrapper (defined in core) with
\Wikimedia\TestingAccessWrapper (defined in the composer package
wikimedia/testing-access-wrapper).

See https://gerrit.wikimedia.org/r/#/q/topic:librarize-testing-access-wrapper
for downstream patches.

The core version of the class is kept around for a while to avoid
circular dependency problems.

Bug: T163434
Change-Id: I52cc257e593da3d6c3b01a909e554a950225aec8
2017-04-20 14:15:57 +00:00
Timo Tijhof
8e81621dcf objectcache: Complete code coverage for CachedBagOStuff
Change-Id: I8a228d68701f1ad4d37f60de53d105c32898dc8b
2017-04-07 23:31:41 +00:00
Timo Tijhof
a116bc58b5 objectcache: Fix CachedBagOStuff to use backend makeKey()
Follows-up 25dbd91513.

Change-Id: Ib727c57cb27f05c0462bfdfee89a185ef6603ddd
2017-04-07 23:31:25 +00:00
Timo Tijhof
447ce7e39a phpunit: Avoid use of deprecated getMock for PHPUnit 5 compat
The default will remain PHPUnit 4.x due to PHP 5.5 support.

But, we should allow developers to run tests with newer PHPUnit
versions which are noticably faster (especially for code coverage
reports).

* <https://github.com/sebastianbergmann/phpunit/wiki/Release-Announcement-for-PHPUnit-5.4.0>
  PHPUnit 5 deprecates the getMock() shortcut for getMockBuilder()->getMock().
  It instead introduces the shortcut createMock() which has better defaults
  than getMockBuilder(). For example, it sets 'disableArgumentCloning' and
  other things by default.

  Going forward, code should either use getMockBuilder directly and configure
  it using the setter methods (instead of the confusing variadic arguments
  of getMock) or simply use the new minimalistic createMock method. This patch
  backports the createMock method to MediaWikiTestCase so that we can start
  using it.

Change-Id: I091c0289b21d2b1c876adba89529dc3e72b99af2
2017-04-06 00:44:32 +00:00
Timo Tijhof
751f2b9d25 objectcache: Complete code coverage for HashBagOStuff
Change-Id: I06cb9778df8239706bf82c0c39cccb419eade4b2
2017-03-31 18:03:38 -07:00
Aaron Schulz
2e5eb693de objectcache: add WANObjectCacheReaper for assuring purges
* This fixes keys based on some sort of change log.
  Updates are wrapped in a mutex and keep track of the
  last known good position.
* Make WANObjectReapUpdate class that cleans up title
  related keys using the recentchanges table. This triggers
  as a deferred updates on RC view.

Change-Id: I7f14b9ca2533032147e62b1a3cc004a23da86579
2017-01-27 02:21:06 +00:00
Hashar
4d487d1942 Revert "objectcache: detect default getWithSetCallback() set options"
Causes HHVM 3.12.7 to segfault and or:
Fatal error: Stack overflow in includes/libs/objectcache/BagOStuff.php on line 754

This reverts commit b47ce21cec.

Bug: T150833
Change-Id: Ia52fdf65f5e386c0eaa1046328680fb5c8a9081a
2016-11-16 10:35:27 +00:00
Aaron Schulz
b47ce21cec objectcache: detect default getWithSetCallback() set options
This works by setting a callback to return the cache set
options. The callback will watch DB reads and create a
merged result from said usage.

This handles callers that are missing getCacheSetOptions().

Change-Id: Ia264f011e45e8cf105480955dad7e2c4c2357b73
2016-11-16 04:53:53 +00:00
Aaron Schulz
43ff2a83b5 objectcache: avoid using process cache in nested callbacks
Because the process cache can be lagged by virtue of blind TTL,
the HOLDOFF_TTL might not be enough to account for it, so avoid
using it when already inside a callback.

Also split of the tests from the MediaWiki test class, so this
does not require DB access anymore.

Change-Id: I743a1233a5efc7f036fad140a9ff8a30b32f8f27
2016-10-20 22:13:29 +00:00
Timo Tijhof
1cdecd581d objectcache: Fix test coverage
Follows-up 24200e8, 37df4f8. Test coverage still broken.

> Trying to @cover not existing method
> "WANObjectCache::getMultitWithSetCallback".

Change-Id: Ie4c9a998ea352f93f3c7dcdea25ea5c017ffbab5
2016-10-11 21:48:44 +02:00