Commit graph

540 commits

Author SHA1 Message Date
Aaron Schulz
7523716ebd objectcache: only use memcached "gets" when tokens are needed
This saves a few bytes in the response size and make it easy
for memcached proxies to distinguish key fetches that are part
of check-and-set cycles from those that are not.

MediumSpecificBagOStuff now requires PASS_BY_REF to fetch CAS
tokens. BagOStuff::merge() and WinCacheBagOStuff::doCas() are
the only callers that need this mode.

Bug: T257003
Change-Id: If91963f58adc4cda94f6d634ee0252a479a0fc5e
2020-07-15 00:16:46 +00:00
Bill Pirkle
8e9cba8bf0 Remove deprecated "legacy" serialization type from RESTBagOStuff
T233963 added serizalization type to RESTBagOStuff. Although
there were no known callers that were not refactored to be
aware of this, T233963 also added a deprecated "legacy" type,
marked for removal in MW 1.35, just in case. There are still no
known callers using this, so remove "legacy" as planned.

Bug: T234779
Change-Id: I0c7707692aa1d0c75e262c914e064bddc10897c7
2020-07-13 10:29:47 -05:00
jenkins-bot
39705eb311 Merge "Replace "@stable for subclassing" with "@stable to extend"" 2020-07-13 09:31:38 +00:00
daniel
3c50afa46b Replace "@stable for subclassing" with "@stable to extend"
For compliance with the new version of the table interface policy
(T255803).

This patch was created by an automated search & replace operation
on the includes/ directory.

Bug: T257789
Change-Id: Ie32c1b11b3d16ddfc0c83a757327d449ff80b2e4
2020-07-13 11:00:30 +02:00
daniel
272db6afde Replace "@stable for calling" by "@stable to call"
For compliance with the new version of the table interface policy
(T255803).

This patch was created by an automated search & replace operation
on the includes/ directory.

Bug: T257789
Change-Id: If560596f5e1e0a3da91afc36e656e7c27f040968
2020-07-13 08:55:28 +00:00
daniel
7572a152fc Annotate extensible classes.
This marks some base classes as stable for subclassing.

Bug: T247862
Change-Id: I7b77075849aa40960f2486b463e498a3ea8058e9
2020-07-09 11:07:38 +02:00
daniel
128565505c Mark additional classes as newable for now.
The following glasses are marked as newable per the
Stable Interface Policy, even though logically, they should not be
newable. This is done for classes that are currently instantiated
by extensions, and lack an alternative.

A better way for obtaining an instance of these classes should be
created in the future. At that point, direct instantiation should
be deprecated and replaced.

- includes/ApiMain.php - needs factory
- includes/media/BitmapMetadataHandler.php - should become a stateless service or use handler pattern
- includes/GitInfo.php - should become a stateless service
- includes/logging/LogPage.php - should become a stateless service or use command pattern
- includes/logging/ManualLogEntry.php - should become a stateless service or use command pattern
- includes/poolcounter/PoolCounterWorkViaCallback.php - needs a factory
- includes/context/RequestContext.php - needs to be narrowed down, and should use a factory
- includes/search/SearchHighlighter.php - should have a factory
- includes/TitleArrayFromResult.php - should perhaps be part of TitleFactory
- includes/user/User.php - should at least get a factory method for anons
- includes/diff/Diff.php: needs a TextDiffGenerator service or a factory
- includes/EditPage.php: needs a factory

Bug: T247862
Change-Id: I033158e693c98630ee167d9528fc8c9936f978d4
2020-07-08 18:06:12 +02:00
daniel
bd5c3d06a2 Annotate newable classes
This annotates classes that can safely be instantiated by
extensions, per the Stable Interface Policy.

Bug: T247862
Change-Id: Ia280f559874fc0750265ddeb7f831e65fd7d7d6a
2020-07-07 15:38:57 +02:00
Umherirrender
bc5cb7ae64 phan: Enable redundant_condition_detection
Remove duplicate casts
Suppress false positives

Bug: T248438
Change-Id: I2f89664a4bcd3b39b15e7cf850adda2f0c90ae6f
2020-07-01 20:13:07 +00:00
Aaron Schulz
55d147e4bb objectcache: Split out BagOStuffTestBase class for easier subclassing
Create a simple APCUBagOStuff subclass and also fix
APCUBagOStuff::incrWithInit() default $init value.

Change-Id: If84963fe7dcfedd6edebfb8785235263e0868ece
2020-06-13 20:15:23 +00:00
Reedy
07faedc8c6 Fix more libs PSR12.Properties.ConstantVisibility.NotFound
Change-Id: If3bac6b0ff6fbb89bfa2b9fa91809135a76c610b
2020-05-16 20:13:22 +00:00
Reedy
40e522747c Fix some includes/libs PSR12.Properties.ConstantVisibility.NotFound
Change-Id: I92e2cbf39d2851d215e8be456e86bb99524dea5f
2020-05-16 02:32:53 +01:00
Reedy
12a3883a7b Fix SingleSpaceBeforeSingleLineComment
Change-Id: I285af438ce484af40741489797f20455726ec110
2020-05-11 00:57:11 +00:00
jenkins-bot
20bfc01395 Merge "objectcache: add "non-global" mode to WANObjectCache "coalesceKeys"" 2020-05-08 17:15:07 +00:00
Aaron Schulz
c95c6f7470 objectcache: add "non-global" mode to WANObjectCache "coalesceKeys"
This makes it easier to rollout one keyspace/project at a time even
if some keys are shared and receive purges. The shared keys can all
be done as the last step.

Also, simplify getMulti() to no longer need extractBaseKey().
Make the "warm up cache" logic a bit easier to follow and less
likely to copy values around.

Change-Id: I8b602ddf5dd1feaada45fb0af202c5603836a8dd
2020-05-08 17:01:31 +00:00
Ed Sanders
387f3c2a20 Fix typo: 'the the' -> 'the'
Change-Id: Ia57fb787b43c8e49da4f33a65c265cbc37ee1f77
2020-05-03 20:45:36 +01:00
Aaron Schulz
2e26a4dfbb objectcache: improve documentation for BagOStuff::setNewPreparedValues()
Change-Id: I0a30da88e693a3a0ad43b70ab5724e361a07149f
2020-04-18 21:47:41 +00:00
Aaron Schulz
f7141fa09f objectcache: add size metrics to WANObjectCache::getWithSetCallback()
Track the serialized value bytes/second spent on cache backfills for
each key class. The combination of a Count and Timing metrics within
checkAndSetCooloff() permits calculation of bytes/call as well.

Bug: T248962
Change-Id: I324e29e85bc4df7689bd2d5fb45cf8750b92a8d9
2020-04-18 20:59:44 +00:00
jenkins-bot
8e3297246d Merge "objectcache: make WANObjectCache::set() handle very slow regeneration" 2020-04-15 01:46:35 +00:00
Aaron Schulz
9ec57d7e5b objectcache: make WANObjectCache::set() handle very slow regeneration
If a key always takes a very long time to regenerate, is popular,
and does not use lockTSE, it still needs to be cacheable. Since a
value cannot be more up-to-date than the time it takes to regenerate
it, take the "lower the TTL" approach for these cases. Use "walltime"
to narrow down the "reject the set()" case based on regeneration time.
This is already provided by getWithSetCallback() automatically.

Bug: T244877
Change-Id: Id43fb02738b28dad3bc922057efb7eee0272d0e1
2020-04-14 22:53:38 +00:00
jenkins-bot
0e74c1b031 Merge "objectcache: Improve wgMainWANCache docs" 2020-04-03 16:08:36 +00:00
Timo Tijhof
c7330ebabc objectcache: Improve wgMainWANCache docs
* Document how wgWANObjectCaches and wgMainWANCache are
  used / should be used.
  Remove outdated mention of event relayers.

* The easiest way I could document the 'other' options that
  wgWANObjectCaches entries can have was by saying which keys
  can't be set and that aside from these and class/cacheId
  they are all passed on.

  However, that wasn't (yet) technically right given we do pass
  class/cacheId on blindly, which might be unexpected for
  subclasses in theory. Explicitly unset those in ServiceWiring
  to avoid any chance of confusion. Also simplify that code
  slightly overall in terms of var names and var re-use,
  and improve error messages a litle bit.

* Test plan for the wanobjectcache-deployment doc ref:

  $ php maintenance/mwdocgen.php \
   --file includes/DefaultSettings.php,includes/libs/objectcache/

  Then open path/to/mediawiki/docs/html/, click on the docs for
  DefaultSettings, and find-in-page to "wgMainWANCache".

Change-Id: I1cfc65c2cc4dbceed6b9777c2b808527a58daeb9
2020-04-02 20:11:20 +00:00
Aaron Schulz
e865062d02 objectcache: set QOS_DURABILITY_NONE for EmptyBagOStuff
This allows for class-agnostic detection with BagOStuff
as well as WANObjectCache instances.

Change-Id: Ib48cbf3d93a49587f87fa05e86b65f727e63bc34
2020-03-30 10:14:20 -07:00
Aaron Schulz
bfef92cf98 objectcache: add BagOStuff::setNewPreparedValues()
This can be used by WANObjectCache or statsd metrics to estimate
the size of objects and network commands without having to do an
extra serialization round.

Change-Id: I7b1b787bba39a21e2e04262e77ee7ae24e88f8ca
2020-03-26 20:11:26 +00:00
Aaron Schulz
3f0ee2ec6f objectcache: reduce the number of return points in APCUBagOStuff
Change-Id: I24e01b14ae440c6cb43782d61747f8dfea5c0184
2020-03-18 18:01:37 +00:00
Aaron Schulz
0edd560c95 objectcache: improve WANObjectCache I/O spike avoidance
Rewrite checkAndSetCooloff() to have a clearer intent and a
broader scope. It now applies any time a key is evicted and
seems "risky" (based on value size and regeneration time).
Previously, only keys that used "lockTSE" were considered
for such throttling.

For scalar values, estimating the size is easy enough, though
the final object size might be smaller due to compression.
A pessimistic size is assumed for other variable types.

Bug: T239983
Change-Id: Ib233a650112a9d574a6aa744f6af23bc16d4f5d7
2020-03-17 19:18:40 +00:00
jenkins-bot
e53a82d22b Merge "objectcache: make decr() atomic in APCUBagOStuff" 2020-03-10 23:37:47 +00:00
Aaron Schulz
dda34f24db objectcache: make decr() atomic in APCUBagOStuff
This avoids negative counter initialization contrary to the interface
of the base class.

Change-Id: Ib54258e82b8d134f02da98bad06ae2fdc6c77166
2020-03-09 19:42:43 -07:00
Aaron Schulz
1d750641ca objectcache: set apc.use_request_time to 0 during APCUBagOStuff writes
This better matches the BagOStuff interface. If a value is saved with
a TTL of X, then it should take ~X seconds for it to expire. A key may
need to use low, often randomized, TTLs, and this behavior will break
such use cases.

The default apc.use_request_time value (1) assigns writes a timestamp
equal to the web request timestamp. If a request has been going for Y
seconds, then a TTL of X becomes (X-Y) as far as new requests can see,
due to the "back-dating". If Y > X, then the value was effectively
purged for all new requests. This can easily be simulated via sleep().

Change-Id: I3d0abdc3a1870cb76386c6db2db27ad1c9b5885b
2020-03-10 02:33:49 +00:00
Timo Tijhof
8d4aaa43cb objectcache: Inject Config object to ObjectCache::newFromParams
* Avoid direct $GLOBALS lookups.
* Avoid MediaWikiServices singleton for Config object.

Also given that newFromParams() constructs its Logger object
unconditionally as of I2aa835c5ec79, stop creating it ahead of
time in ServiceWiring.

This code, including the default loggroup value, was duplicated by
me from ObjectCache.php to ServiceWiring.php in commits 3828558140
and bd16c5eb34 because I needed to obtain the Logger object
in order to send a message to it about how it was created.

Solve this debt by letting ServiceWiring access the actual logger
that was created by newFromParams() through a new getLogger()
object.

Change-Id: Ib2de7b22f6c79db0c700fae06269d04fbe27831d
2020-03-04 22:34:30 +00:00
Aaron Schulz
69950da666 objectcache: split out StorageAwareness/ExpirationAwareness from IExpiringStore
Add more detailed QOS_* constants while at it

Bug: T236412
Change-Id: Ia862c5111a3daf10a34fc78163301629228efa6b
2020-03-02 20:01:24 +00:00
Aaron Schulz
b9b3f366bb objectcache: fix "coalesceKeys" option name in WANObjectCache
Follow-up to 85bc62c5a8

Fix related unit tests that otherwise break as a result

Change-Id: I28b3a1537d319c68a7c12c578e1acfb916f3ec99
2020-02-13 03:17:34 +00:00
Aaron Schulz
85bc62c5a8 objectcache: add "coalesceKeys" option to WANObjectCache for key grouping
This is useful for grouping related keys on the same servers to reduce
the need for cache server connections and availability. A cache key that
uses "lockTSE" can already involve accessing several keys during the
read/write cache-aside paths:
a) The value key itself
b) The check key (named after the main key, a common pattern)
c) The mutex key (used if the value looks stale)
d) The cool-off key (used if regeneration took a while)

Any problems accessing the first two could cause extra value regenerations.
Problems with the mutex key could lead to stampedes due to threads assuming
another thread was regerating a soon-to-expire value when, in fact, none was.
A similar problem could happen with cool-off keys, with threads assuming
that another saved the newly regenerated value when, in fact, none did.

The use of hash stops puts the tiny related keys on the same server as the
main cache key that they serve. This is only for hash-based routing, and not
route prefix routing (e.g. All*Route still sends the key to multiple child
routes, but the PoolRoute/HashRoute function will hash differently).

The option is not enabled by default yet.

Change-Id: I37e92a88f356ef1e2a2b7728263040e2f6f09a13
2020-02-06 20:27:08 +00:00
jenkins-bot
51280df815 Merge "objectcache: fix cache pollution in WANObectCache Multi* methods" 2020-01-30 18:17:03 +00:00
Aaron Schulz
527fd0109f objectcache: fix cache pollution in WANObectCache Multi* methods
This was triggered by bad reference handling during preemptive refreshes

Bug: T235188
Change-Id: I239a3e1922f478c74c94d8d2debff28f525c7c31
2020-01-29 21:08:39 +00:00
Aaron Schulz
758e980427 objectcache: remove APCBagOStuff class
apcu comes with PHP 5.5+ and the minimum version for MediaWiki is 7.2

Change-Id: I69a988c6c2999766a3c7e56b841cd4f4091b4d95
2020-01-29 17:27:47 +00:00
jenkins-bot
0a46bef2db Merge "objectcache: fix storage of null values in WANObjectCache" 2020-01-29 04:14:20 +00:00
Aaron Schulz
4fb5210b62 objectcache: fix storage of null values in WANObjectCache
Bug: T234583
Change-Id: I38a531b9a0acb95d7884519f3381b48cd9d8faa0
2020-01-24 22:49:26 +00:00
jenkins-bot
1ae315f699 Merge "objectcache: rename $ttl => $exptime in BagOStuff for consistency" 2020-01-10 03:25:34 +00:00
Aaron Schulz
73ba4c6a41 objectcache: rename $ttl => $exptime in BagOStuff for consistency
Change-Id: Ie09176eb591faa525169de3dc1e861b9610d90e5
2020-01-09 12:39:00 -08:00
Aaron Schulz
379c839cf5 objectcache: simplify some of the WAN cache logging added in 24812e4096
Change-Id: Id6f61ed5fe01d7bba6c27cd2e4a2ddeca764dbdc
2020-01-06 13:31:40 -08:00
jenkins-bot
f73d62d47d Merge "Add more logging to getWithSetCallback()" 2019-12-20 00:36:31 +00:00
Daimona Eaytoy
25ba3b3794 Document more scalar types for BagOStuff::makeKey and Language::formatNum
These were spotted by phan's stricter type checks. All make*Key methods
can also take integers, and that's pretty common across core and
extensions.
Language::formatNum can also take numbers (and tests cover this case);
in this case, move the cast to the top of the method, so that we're
passing a string to strtr, too.

The methods touched in this patch are the main source of false positives
in both core and extensions.

Change-Id: I6d538ce3ca7fd2d495c2bafbab7cc279da69db1c
2019-12-15 00:02:04 +00:00
Daimona Eaytoy
598c4d7fcb build: Upgrade phan to 0.9.0
Scalar casts are still allowed (for now), because there's a huge amount
of false positives. Ditto for invalid array offsets.

Thoughts about the rest: luckily, many false positives with array offsets
have gone. Moreover, since *Internal issues are suppressed in the base
config, we can remove inline suppressions.

Unfortunately, there are a couple of new issues about array additions
with only false positives, because apparently they don't take
branches into account.

Change-Id: I5a3913c6e762f77bfdae55051a395fae95d1f841
2019-12-07 20:16:19 +00:00
Umherirrender
0688dd7c6d Set method visibility for various constructors
Change-Id: Id3c88257e866923b06e878ccdeddded7f08f2c98
2019-12-03 20:17:30 +01:00
Tim Starling
24812e4096 Add more logging to getWithSetCallback()
To help isolate the referenced bug, which is a sporadic test failure
in testGetWithSetCallback().

Bug: T238197
Change-Id: If35d60340c804b6bfe1e9ddfcf53c76373c794b1
2019-11-18 16:32:47 +11:00
Daimona Eaytoy
ae424ce5da build: Upgrade mediawiki-phan-config to 0.8.0
This is to ensure that the CI job is working with the new version.

Note: redundant_condition_detection should have worked as expected by
this version, but unfortunately it still has false positives.

Bug: T235049
Bug: T231636
Change-Id: Idaba6584cb5b2ff19b6455c7bbec6b89619ddbff
2019-10-22 09:16:45 +00:00
jenkins-bot
35b29d132a Merge "objectcache: make incr() and incrWithInit() atomic in APCUBagOStuff" 2019-10-22 03:38:58 +00:00
jenkins-bot
c3c45ff689 Merge "Fix new phan errors, part 6" 2019-10-20 18:05:18 +00:00
Daimona Eaytoy
114ee6e412 Fix new phan errors, part 6
Bug: T231636
Change-Id: I1870b6cbeb31e54fde5e675fec51446b330e06c5
2019-10-20 17:53:48 +00:00