Track the serialized value bytes/second spent on cache backfills for
each key class. The combination of a Count and Timing metrics within
checkAndSetCooloff() permits calculation of bytes/call as well.
Bug: T248962
Change-Id: I324e29e85bc4df7689bd2d5fb45cf8750b92a8d9
If a key always takes a very long time to regenerate, is popular,
and does not use lockTSE, it still needs to be cacheable. Since a
value cannot be more up-to-date than the time it takes to regenerate
it, take the "lower the TTL" approach for these cases. Use "walltime"
to narrow down the "reject the set()" case based on regeneration time.
This is already provided by getWithSetCallback() automatically.
Bug: T244877
Change-Id: Id43fb02738b28dad3bc922057efb7eee0272d0e1
* Document how wgWANObjectCaches and wgMainWANCache are
used / should be used.
Remove outdated mention of event relayers.
* The easiest way I could document the 'other' options that
wgWANObjectCaches entries can have was by saying which keys
can't be set and that aside from these and class/cacheId
they are all passed on.
However, that wasn't (yet) technically right given we do pass
class/cacheId on blindly, which might be unexpected for
subclasses in theory. Explicitly unset those in ServiceWiring
to avoid any chance of confusion. Also simplify that code
slightly overall in terms of var names and var re-use,
and improve error messages a litle bit.
* Test plan for the wanobjectcache-deployment doc ref:
$ php maintenance/mwdocgen.php \
--file includes/DefaultSettings.php,includes/libs/objectcache/
Then open path/to/mediawiki/docs/html/, click on the docs for
DefaultSettings, and find-in-page to "wgMainWANCache".
Change-Id: I1cfc65c2cc4dbceed6b9777c2b808527a58daeb9
This can be used by WANObjectCache or statsd metrics to estimate
the size of objects and network commands without having to do an
extra serialization round.
Change-Id: I7b1b787bba39a21e2e04262e77ee7ae24e88f8ca
Rewrite checkAndSetCooloff() to have a clearer intent and a
broader scope. It now applies any time a key is evicted and
seems "risky" (based on value size and regeneration time).
Previously, only keys that used "lockTSE" were considered
for such throttling.
For scalar values, estimating the size is easy enough, though
the final object size might be smaller due to compression.
A pessimistic size is assumed for other variable types.
Bug: T239983
Change-Id: Ib233a650112a9d574a6aa744f6af23bc16d4f5d7
This better matches the BagOStuff interface. If a value is saved with
a TTL of X, then it should take ~X seconds for it to expire. A key may
need to use low, often randomized, TTLs, and this behavior will break
such use cases.
The default apc.use_request_time value (1) assigns writes a timestamp
equal to the web request timestamp. If a request has been going for Y
seconds, then a TTL of X becomes (X-Y) as far as new requests can see,
due to the "back-dating". If Y > X, then the value was effectively
purged for all new requests. This can easily be simulated via sleep().
Change-Id: I3d0abdc3a1870cb76386c6db2db27ad1c9b5885b
* Avoid direct $GLOBALS lookups.
* Avoid MediaWikiServices singleton for Config object.
Also given that newFromParams() constructs its Logger object
unconditionally as of I2aa835c5ec79, stop creating it ahead of
time in ServiceWiring.
This code, including the default loggroup value, was duplicated by
me from ObjectCache.php to ServiceWiring.php in commits 3828558140
and bd16c5eb34 because I needed to obtain the Logger object
in order to send a message to it about how it was created.
Solve this debt by letting ServiceWiring access the actual logger
that was created by newFromParams() through a new getLogger()
object.
Change-Id: Ib2de7b22f6c79db0c700fae06269d04fbe27831d
This is useful for grouping related keys on the same servers to reduce
the need for cache server connections and availability. A cache key that
uses "lockTSE" can already involve accessing several keys during the
read/write cache-aside paths:
a) The value key itself
b) The check key (named after the main key, a common pattern)
c) The mutex key (used if the value looks stale)
d) The cool-off key (used if regeneration took a while)
Any problems accessing the first two could cause extra value regenerations.
Problems with the mutex key could lead to stampedes due to threads assuming
another thread was regerating a soon-to-expire value when, in fact, none was.
A similar problem could happen with cool-off keys, with threads assuming
that another saved the newly regenerated value when, in fact, none did.
The use of hash stops puts the tiny related keys on the same server as the
main cache key that they serve. This is only for hash-based routing, and not
route prefix routing (e.g. All*Route still sends the key to multiple child
routes, but the PoolRoute/HashRoute function will hash differently).
The option is not enabled by default yet.
Change-Id: I37e92a88f356ef1e2a2b7728263040e2f6f09a13
These were spotted by phan's stricter type checks. All make*Key methods
can also take integers, and that's pretty common across core and
extensions.
Language::formatNum can also take numbers (and tests cover this case);
in this case, move the cast to the top of the method, so that we're
passing a string to strtr, too.
The methods touched in this patch are the main source of false positives
in both core and extensions.
Change-Id: I6d538ce3ca7fd2d495c2bafbab7cc279da69db1c
Scalar casts are still allowed (for now), because there's a huge amount
of false positives. Ditto for invalid array offsets.
Thoughts about the rest: luckily, many false positives with array offsets
have gone. Moreover, since *Internal issues are suppressed in the base
config, we can remove inline suppressions.
Unfortunately, there are a couple of new issues about array additions
with only false positives, because apparently they don't take
branches into account.
Change-Id: I5a3913c6e762f77bfdae55051a395fae95d1f841
To help isolate the referenced bug, which is a sporadic test failure
in testGetWithSetCallback().
Bug: T238197
Change-Id: If35d60340c804b6bfe1e9ddfcf53c76373c794b1
This is to ensure that the CI job is working with the new version.
Note: redundant_condition_detection should have worked as expected by
this version, but unfortunately it still has false positives.
Bug: T235049
Bug: T231636
Change-Id: Idaba6584cb5b2ff19b6455c7bbec6b89619ddbff
This is a TEMPORARY HACK to fix a problem with cached values
getting mixed up when using getBlobBatch(). As of the creation of this
patch, the actual cause is unknown. Caching should be put back once the
cause of the problem has been found.
Note that getBlobBatch is only used in maintenance scripts.
Removal of caching in that method has no effect on web requests.
getBlobBatch() was introduced in I56306c50a661 and is used by
RevisionStore::getContentBlobsForBatch(), which in turn is needed
by the Translate extension to remove its dependency on the deprecated
rev_text_id field.
Bug: T235188
Change-Id: I94c6f9ba7b9caeebaa9b055916f15f7bbdcd8fb6
Fix error in RESTBagOStuff class comment, so that the example Kask
configuration is properly documented
Bug: T235559
Change-Id: If664913a4e61013dba070153b10def5ecd6caeb3
T233537 made RESTBagOStuff work with the Kask external session
storage service, but broke backward compatibility. Add optional
values to the RESTBagOStuff $params constructor parameter to
support communicating with Kask, and to allow using HMAC. The
new values are:
serialization_type: legacy (default), PHP, or JSON
hmac_key: HMAC key to use for protecting the serialized blob
If these new values are not specified, behavior remains
unchanged (PHP serialization with no HMAC protection).
Bug: T233963
Change-Id: Ia2625c04e08cfe9616569500f1d613be73c170a2
Also change a `@private` tag to `@internal` in ResourceLoaderFileModule
for consistency with other RL code.
Change-Id: I8c3a5aa36b643083c0b6d2f3c8d623f344b7c0be
This allows us to remove many suppressions for phan false positives.
Bug: T231636
Depends-On: I82a279e1f7b0fdefd3bb712e46c7d0665429d065
Change-Id: I5c251e9584a1ae9fb1577afcafb5001e0dcd41c7
This can get called a lot on an average page load, optimise
a bit by using a referenced foreach iteration instead.
Using a simplified test case, I found this saves about 70% on PHP 7.2.
For 100 iterations, from ~1.5ms to ~0.4 ms.
```lang=php
$args = [ 'FooBar', 'thisthat', 4, 'foo', 12 ];
$left = 100;
foreach ($args as &$arg) {
$arg = strtr( $arg, ' ', '_' );
if ( strlen( $arg ) < $left || true ) {
$arg = '#' . $arg;
}
$left--;
}
```
Change-Id: Ie779c4661306a6d3dc08c08671f1a36682ca1afb
