Commit graph

508 commits

Author SHA1 Message Date
Aaron Schulz
85bc62c5a8 objectcache: add "coalesceKeys" option to WANObjectCache for key grouping
This is useful for grouping related keys on the same servers to reduce
the need for cache server connections and availability. A cache key that
uses "lockTSE" can already involve accessing several keys during the
read/write cache-aside paths:
a) The value key itself
b) The check key (named after the main key, a common pattern)
c) The mutex key (used if the value looks stale)
d) The cool-off key (used if regeneration took a while)

Any problems accessing the first two could cause extra value regenerations.
Problems with the mutex key could lead to stampedes due to threads assuming
another thread was regerating a soon-to-expire value when, in fact, none was.
A similar problem could happen with cool-off keys, with threads assuming
that another saved the newly regenerated value when, in fact, none did.

The use of hash stops puts the tiny related keys on the same server as the
main cache key that they serve. This is only for hash-based routing, and not
route prefix routing (e.g. All*Route still sends the key to multiple child
routes, but the PoolRoute/HashRoute function will hash differently).

The option is not enabled by default yet.

Change-Id: I37e92a88f356ef1e2a2b7728263040e2f6f09a13
2020-02-06 20:27:08 +00:00
jenkins-bot
51280df815 Merge "objectcache: fix cache pollution in WANObectCache Multi* methods" 2020-01-30 18:17:03 +00:00
Aaron Schulz
527fd0109f objectcache: fix cache pollution in WANObectCache Multi* methods
This was triggered by bad reference handling during preemptive refreshes

Bug: T235188
Change-Id: I239a3e1922f478c74c94d8d2debff28f525c7c31
2020-01-29 21:08:39 +00:00
Aaron Schulz
758e980427 objectcache: remove APCBagOStuff class
apcu comes with PHP 5.5+ and the minimum version for MediaWiki is 7.2

Change-Id: I69a988c6c2999766a3c7e56b841cd4f4091b4d95
2020-01-29 17:27:47 +00:00
jenkins-bot
0a46bef2db Merge "objectcache: fix storage of null values in WANObjectCache" 2020-01-29 04:14:20 +00:00
Aaron Schulz
4fb5210b62 objectcache: fix storage of null values in WANObjectCache
Bug: T234583
Change-Id: I38a531b9a0acb95d7884519f3381b48cd9d8faa0
2020-01-24 22:49:26 +00:00
jenkins-bot
1ae315f699 Merge "objectcache: rename $ttl => $exptime in BagOStuff for consistency" 2020-01-10 03:25:34 +00:00
Aaron Schulz
73ba4c6a41 objectcache: rename $ttl => $exptime in BagOStuff for consistency
Change-Id: Ie09176eb591faa525169de3dc1e861b9610d90e5
2020-01-09 12:39:00 -08:00
Aaron Schulz
379c839cf5 objectcache: simplify some of the WAN cache logging added in 24812e4096
Change-Id: Id6f61ed5fe01d7bba6c27cd2e4a2ddeca764dbdc
2020-01-06 13:31:40 -08:00
jenkins-bot
f73d62d47d Merge "Add more logging to getWithSetCallback()" 2019-12-20 00:36:31 +00:00
Daimona Eaytoy
25ba3b3794 Document more scalar types for BagOStuff::makeKey and Language::formatNum
These were spotted by phan's stricter type checks. All make*Key methods
can also take integers, and that's pretty common across core and
extensions.
Language::formatNum can also take numbers (and tests cover this case);
in this case, move the cast to the top of the method, so that we're
passing a string to strtr, too.

The methods touched in this patch are the main source of false positives
in both core and extensions.

Change-Id: I6d538ce3ca7fd2d495c2bafbab7cc279da69db1c
2019-12-15 00:02:04 +00:00
Daimona Eaytoy
598c4d7fcb build: Upgrade phan to 0.9.0
Scalar casts are still allowed (for now), because there's a huge amount
of false positives. Ditto for invalid array offsets.

Thoughts about the rest: luckily, many false positives with array offsets
have gone. Moreover, since *Internal issues are suppressed in the base
config, we can remove inline suppressions.

Unfortunately, there are a couple of new issues about array additions
with only false positives, because apparently they don't take
branches into account.

Change-Id: I5a3913c6e762f77bfdae55051a395fae95d1f841
2019-12-07 20:16:19 +00:00
Umherirrender
0688dd7c6d Set method visibility for various constructors
Change-Id: Id3c88257e866923b06e878ccdeddded7f08f2c98
2019-12-03 20:17:30 +01:00
Tim Starling
24812e4096 Add more logging to getWithSetCallback()
To help isolate the referenced bug, which is a sporadic test failure
in testGetWithSetCallback().

Bug: T238197
Change-Id: If35d60340c804b6bfe1e9ddfcf53c76373c794b1
2019-11-18 16:32:47 +11:00
Daimona Eaytoy
ae424ce5da build: Upgrade mediawiki-phan-config to 0.8.0
This is to ensure that the CI job is working with the new version.

Note: redundant_condition_detection should have worked as expected by
this version, but unfortunately it still has false positives.

Bug: T235049
Bug: T231636
Change-Id: Idaba6584cb5b2ff19b6455c7bbec6b89619ddbff
2019-10-22 09:16:45 +00:00
jenkins-bot
35b29d132a Merge "objectcache: make incr() and incrWithInit() atomic in APCUBagOStuff" 2019-10-22 03:38:58 +00:00
jenkins-bot
c3c45ff689 Merge "Fix new phan errors, part 6" 2019-10-20 18:05:18 +00:00
Daimona Eaytoy
114ee6e412 Fix new phan errors, part 6
Bug: T231636
Change-Id: I1870b6cbeb31e54fde5e675fec51446b330e06c5
2019-10-20 17:53:48 +00:00
daniel
e1f8a8148d SqlBlobStore HOT FIX: remove caching from getBlobBatch
This is a TEMPORARY HACK to fix a problem with cached values
getting mixed up when using getBlobBatch(). As of the creation of this
patch, the actual cause is unknown. Caching should be put back once the
cause of the problem has been found.

Note that getBlobBatch is only used in maintenance scripts.
Removal of caching in that method has no effect on web requests.

getBlobBatch() was introduced in I56306c50a661 and is used by
RevisionStore::getContentBlobsForBatch(), which in turn is needed
by the Translate extension to remove its dependency on the deprecated
rev_text_id field.

Bug: T235188
Change-Id: I94c6f9ba7b9caeebaa9b055916f15f7bbdcd8fb6
2019-10-16 23:02:52 +00:00
Nikki Nikkhoui
3cda7b7846 Fix example Kask configuration in RESTBagOStuff class comment
Fix error in RESTBagOStuff class comment, so that the example Kask
configuration is properly documented

Bug: T235559

Change-Id: If664913a4e61013dba070153b10def5ecd6caeb3
2019-10-15 22:43:42 +00:00
Aaron Schulz
f565c012d7 objectcache: make incr() and incrWithInit() atomic in APCUBagOStuff
Change-Id: I8f2007fd8a94f288cc57e71ac2da7997d56842cc
2019-10-14 09:43:37 +00:00
Bill Pirkle
2ed69af15c Add optional serialization_type and hmac_key param values to RESTBagOStuff
T233537 made RESTBagOStuff work with the Kask external session
storage service, but broke backward compatibility. Add optional
values to the RESTBagOStuff $params constructor parameter to
support communicating with Kask, and to allow using HMAC. The
new values are:
  serialization_type: legacy (default), PHP, or JSON
  hmac_key: HMAC key to use for protecting the serialized blob
If these new values are not specified, behavior remains
unchanged (PHP serialization with no HMAC protection).

Bug: T233963
Change-Id: Ia2625c04e08cfe9616569500f1d613be73c170a2
2019-10-06 15:52:01 -05:00
Max Semenik
06a275f48e Remove more HHVM hacks
Change-Id: I6bd298ef3b887173b87004ee055be2a4f6ea5f11
2019-10-06 10:16:09 +00:00
Aaron Schulz
223b7a3717 objectcache: fully respect "pcTTL" in WANObjectCache instead of using INF when >= 0
This was broken since 611e2d5596

Change-Id: I612eaf211ff698d5ab1c911aa58195b7bc44f00c
2019-09-24 02:56:39 -07:00
Timo Tijhof
9244cdb977 objectcache: Escape dots from cache keys in StatsD metrics
Bug: T232907
Change-Id: Ia61256e24fe35803e69a83a1ca235e18297c75cd
2019-09-14 18:16:55 +00:00
DannyS712
4dfae3bcc7 Fix a typo (yeild -> yield)
Bug: T201491
Change-Id: I70443514d55c681e51233b6e80ac7911c5e12d5a
2019-09-13 03:38:14 +00:00
Timo Tijhof
21ecb77750 docs: Fix Doxygen warning for invalid @bar in WANObjectCache
Also change a `@private` tag to `@internal` in ResourceLoaderFileModule
for consistency with other RL code.

Change-Id: I8c3a5aa36b643083c0b6d2f3c8d623f344b7c0be
2019-09-05 16:23:36 +00:00
Daimona Eaytoy
b5cbb5ab3f Upgrade phan config to 0.7.1
This allows us to remove many suppressions for phan false positives.

Bug: T231636
Depends-On: I82a279e1f7b0fdefd3bb712e46c7d0665429d065
Change-Id: I5c251e9584a1ae9fb1577afcafb5001e0dcd41c7
2019-09-04 08:20:53 +00:00
jenkins-bot
3587a9427d Merge "objectcache: Optimise array_map in MemcachedBagOStuff::makeKey()" 2019-09-02 17:00:21 +00:00
Aaron Schulz
9a9944e225 objectcache: add "sessionConsistencyWindow" option to ReplicatedBagOStuff
Change-Id: I25af780f063879eda2de1b9a1168e37115f823ed
2019-09-01 21:55:25 +00:00
Daimona Eaytoy
327e8ea416 Unsuppress phan issues part 6
Bug: T231636
Depends-On: I50377746f01749b058c39fd8229f9d566224cc43
Change-Id: I2cd24e73726394e3200a570c45d5e86b6849bfa9
2019-09-01 09:48:45 +00:00
Daimona Eaytoy
e2e543f7c2 Unsuppress more phan issues (part 5)
Bug: T231636
Depends-On: I6e5fba7bd273219b1206559420b5bdb78734aa84
Change-Id: I50377746f01749b058c39fd8229f9d566224cc43
2019-09-01 09:48:31 +00:00
Timo Tijhof
e0d817b7f6 objectcache: Optimise array_map in MemcachedBagOStuff::makeKey()
This can get called a lot on an average page load, optimise
a bit by using a referenced foreach iteration instead.

Using a simplified test case, I found this saves about 70% on PHP 7.2.
For 100 iterations, from ~1.5ms to ~0.4 ms.

```lang=php
$args = [ 'FooBar', 'thisthat', 4, 'foo', 12 ];
$left = 100;
foreach ($args as &$arg) {
    $arg = strtr( $arg, ' ', '_' );
    if ( strlen( $arg ) < $left || true ) {
        $arg = '#' . $arg;
    }
    $left--;
}
```

Change-Id: Ie779c4661306a6d3dc08c08671f1a36682ca1afb
2019-08-31 22:43:21 +01:00
Daimona Eaytoy
5eac6d131c Unsuppress more phan issues (part 3)
Bug: T231636
Depends-On: I78354bf5f0c831108c8f606e50c87cf6bc00d8bd
Change-Id: I58e67c2b38389df874438deada4239510d21654f
2019-08-31 16:38:55 +00:00
jenkins-bot
4334e1cc02 Merge "objectcache: make MediumSpecificBagOStuff::mergeViaCas() handle negative TTLs" 2019-08-25 16:57:41 +00:00
Aaron Schulz
bee2a20903 objectcache: make WANObjectCache::relayPurge() actually use $holdoff
Fixes a regression introduced by 70547f3fa3 in 2016.

Change-Id: I5996b63c31ac3b382e838a6858b8585bdc8c585c
2019-08-24 22:41:39 +00:00
Aaron Schulz
94eb0c946f objectcache: improve BagOStuff arithmetic method signatures
Make the default $init value for incrWithInit() be $value.
This is far less suprising and also makes the operation
easier to replicate without conflicts.

Make decr() definitions more explicit since various cache
drivers do not handle negative incr() values (e.g. memcached).

Change-Id: I2b8d642656cc91c841abbd7a55d97eba101b027a
2019-08-24 12:50:37 -07:00
jenkins-bot
2b5f468efa Merge "objectcache: add BagOStuff::fieldHasFlags() convenience method" 2019-08-23 19:14:59 +00:00
Aaron Schulz
b70e54e986 objectcache: add BagOStuff::fieldHasFlags() convenience method
Change-Id: Id2bf4ec0dd0999f988b70a895003c4b1aaae51a2
2019-08-22 15:07:14 -07:00
Aaron Schulz
44e2ab3560 objectcache: move MemcachedClient class to /utils subdir
Change-Id: I2774d3e7a3c96ef691106b21908a42bfbbf25956
2019-08-21 16:17:20 -07:00
jenkins-bot
b43543acc9 Merge "objectcache: Remove lock()/unlock() stubs from MemcachedClient" 2019-08-21 18:58:58 +00:00
Aaron Schulz
942f708b61 objectcache: fix WRITE_ALLOW_SEGMENTS in BagOStuff cas() and add() methods
Add MediumSpecificBagOStuff::getValueOrSegmentList() helper method.

Also:
* Use $keysMissing variable correctly in CachedBagOStuff::getMulti()
  to avoid extra overhead.
* Optimize mergeViaCas() when the current value matches the new one.

Change-Id: I5c4bd74379bc459216ac0278150ce3aecff3b851
2019-08-20 10:55:42 -04:00
Aaron Schulz
5e7829e08f objectcache: Remove lock()/unlock() stubs from MemcachedClient
Change-Id: I56628957b16d91cfa9b79b07abdece17fb396577
2019-08-20 13:05:49 +00:00
Aaron Schulz
49025f52b2 objectcache: clean up MemcachedBagOStuff expiry handling
Partly a follow-up to 88640fd902.

Use real time in changeTTL() tests to fix all remaining
failures for BagOStuff sub-classes.

Change-Id: I537d665d6c8770a68a5a79233a913f1714881dfb
2019-08-10 12:12:12 -07:00
Aaron Schulz
7f25cf3031 objectcache: make MediumSpecificBagOStuff::mergeViaCas() handle negative TTLs
Change-Id: Ieb503cdb5d0849be7b7be21780ae552ab937ae80
2019-08-09 15:35:28 -07:00
jenkins-bot
c19031b21f Merge "objectcache: fix failing tests for non-HashBagOStuff backends" 2019-08-07 17:22:57 +00:00
jenkins-bot
8e338a10c0 Merge "objectcache: avoid using deprecated phpredis::delete() alias" 2019-08-07 15:14:08 +00:00
Aaron Schulz
88640fd902 objectcache: fix failing tests for non-HashBagOStuff backends
Use real time for testing absolute expirations with changeTTL().
Otherwise, backends like memcached or redis will fail since
they do not use the mock time.

Also:
* Make SqlBagOStuff actually override changeTTLMulti() by
  using the right method name
* Check TTL_INDEFINITE more explicitly for clarity
* Rename TTL conversion methods for clarity
* Use isRelativeExpiration() in MemcachedBagOStuff

Change-Id: I9365ceb31d4e7bef65906363d42b8c3020a66346
2019-08-07 14:07:22 +00:00
Aaron Schulz
f445700ccc objectcache: avoid using deprecated phpredis::delete() alias
Change-Id: I3ca8bd9160eefff6590228082f030a32d0edb511
2019-08-07 12:58:55 +00:00
Aaron Schulz
0bee81df81 Move WANObjectCache to wancache/ instead of WinCache
Follow up 6ca3de5fa0.

Change-Id: I7021dfe5c900f810ab3b6dc701a04e2c3e1c4c66
2019-08-02 22:07:47 +00:00