A single test function can have multiple @dataProviders. No problem.
A @dataProvider can also be used by multiple test functions. No
problem. No need to duplicate such large pieces of code when it's
identical anyway.
Change-Id: I5aea36304ec2d1666ff2334ba883df07a70c15c5
And deprecated aliases for the the no namespaced classes.
ReplicatedBagOStuff that already is deprecated isn't moved.
Bug: T353458
Change-Id: Ie01962517e5b53e59b9721e9996d4f1ea95abb51
This patch migrates BagOStuff metrics capture from StatsdD to
StatsLib. `updateOpStats()` definition significantly changed with
this migration thereby affecting the changes in the test case.
What we tried to do in the test is to assert that the metrics
match the StatsLib equivalent ones which is the new format we're
using and that the values corresponding to the samples are also
in the range of values written to the cache.
Notes
=====
* And the cache key also uses underscore (_) instead of (.) because
of StatsLib as well.
* The most important thing which is tested is that the keys inserted
into the cache are all there and correspond to their values. The way
we do this is observe the metrics stream after `->flush()`.
Bug: T356062
Change-Id: Ia6c14746de5bddeaca7917c76f1c9d1e100ae2b2
When getWithSetCallback() is called with check keys, if the keys are
missing, a check key is inserted with the current time, as if
touchCheckKey() were called. This causes cache misses for
HOLDOFF_TTL = 11 seconds. This seems unnecessary since in the case of
an empty cache, there is no expectation of replication delay.
However, it's reasonable for it to be a cache miss when the check key is
missing, and a cache hit subsequently, so we do need to add a purge
value.
So, in getWithSetCallback(), set the holdoff to zero when inserting a
purge value.
Also, use a holdoff of zero when initialising a missing touch key in
getCheckKeyTime().
Bug: T344191
Change-Id: Ib3ae4b963816e5b090e87e4cb93624afefbf8058
== Background
Most of this was introduced in commit 5c335f9d77 (I1eb897c2cea3f5b7).
The original motivation was:
* Ensure wrappers like MultiWriteBagOStuff naturally do the right
thing. In practice, makeKey() results are interchangeable, with
the most contrained one (Memcached) also generally used as the first
tier. However, this is not intuitive and may change in the future.
To make it more intuitive, the default implemention became known
as "generic", with proxyCall() responsible for decoding these,
and then re-encoding them with makeKey() from the respective
underlying BagOStuff. This meant that MultiWriteBag would no longer
use the result of the Memcached-formatted cache key and pass it
to SqlBagOStuff.
* Allow extraction of the key group from a given key cache,
for use in statistics.
Both motivations remains valid and addressed after this refactor.
== Change
* Remove boilerplate and indirection around makeKey from a dozen
classes. E.g. copy-paste stubs for makeKey, makeKeyInternal, and
convertGenericKey.
Instead, let BagOStuff::makeKey and ::makeKeyInternal hold the
defaults. I believe this makes the logic easier to find, understand,
and refer to.
The three non-default implementations (Memcached, WinCache, Sql)
now naturally reflect what they are in terms of business logic,
they are a method override.
Introduce a single boolean requireConvertGenericKey() to let the
three non-default implementations signal their need to convert
keys before use.
* Further improve internal consistently of BagOStuff::makeKeyInternal.
The logic of genericKeyFromComponents() was moved up into
BagOStuff::makeKeyInternal. As a result of caling this directly
from BagOStuff::makeKey(), this code now sees $keyspace and $components
as separate arguments. To keep the behaviour the same, we would
have to either unshift $keyspace into $components, or duplicate
the strtr() call to escape it.
Instead, excempt keyspace from escaping. This matches how the most
commonly used BagOStuff implementations (MemcachedBag, and SqlBag)
already worked for 10+ years, thus this does not introduce any new
responsibility on callers. In particular, keyspace (not key group)
is set by MediaWiki core in service wiring to the wiki ID, and so
is not the concern of individual callers anyway.
* Docs: Explain in proxyCall() why this indirection and complexity
exists. It lets wrapping classes decode and re-encode keys.
* Docs: Explain the cross-wiki and local-wiki semantics of makeKey
and makeKeyGlobal, and centralise this and other important docs
about this method in the place with the most eye balls where it is
most likely seen and discovered, namely BagOStuff::makeKey.
Remove partial docs from other places in favour of references to this one.
Previously, there was no particular reason to follow `@see IStoreKeyEncoder`
much less to know that it holds critical that communicate the
responsibility to limit the key group to 48 chars.
* Docs: Consistently refer to the first component as the "key group",
thus unifying what was known as "key class", "collection",
"key collection name", or "collection name".
The term "key group" seems to be what is used by developers in
conversations for this concept, matching WMF on-boarding docs and
WMF's Grafana dashboard for WANObjectCache.
Change-Id: I6b3167cac824d8bd8773bc66c386f41e4d380021
This was introduced for use by WANObjectCache, however, this never
worked in practice.
The feature only influences determineKeyPrefixForStats(), which in
turn is only called by MediumSpecificBagOStuff::updateOpStats, which
in turn is exclusively called within BagOStuff subclasses that aren't
used with WANObjectCache (RESTBag, RedisBag, SqlBag).
The next commit simplifies BagOStuff::makeKey, which can make a bigger
impact if this feature doesn't need to be supported.
During drafting of the next commit, I found that the feature doesn't
appear to work well in practice, because getCollectionFromSisterKey
did not account for mcrouter-style route prefixes, as inserted by
makeSisterKey().
Change-Id: Iecea959c88abbd7a0a17b92c1c4d854602748280
When the value of the associated array is not used, there is no need to
use array_keys and create internal a copy of the array, just omit the
value with $_ to tell static analyzer that the var is unused.
Change-Id: I5c6d667e98c0167c0573c683f0d617bc1d4ca619
Initally used a new sniff with autofix (T333745),
but some provide are defined non-static in TestBase class
and need more work to make them static in a compatible way
Bug: T332865
Change-Id: I889d33424f0c01fb26f2d86f8d4fc3de3e568843
* Triple backslash in regex should really be quadruple backslash
* Using the returned value of a void method
* Immediately overwritten array keys
* Duplicate array keys
* Foreach variable reuse
* sprintf() with too many params
* Incorrect reference usage
Change-Id: I3c649b543c9561a1614058c50f3847f663ff04df
We'd lose more useful coverage and spend more effort keeping these
accurate through refactors etc than is worth the theoretically "bad"
accidental coverage. Plus, even then we still very often forget to
update these and waste time digging into seemingly uncovered code
and either write duplicate test or discover they are already covered.
Especially for private methods this can be a flawed endavour as
we tend to trust tests when changing those and thus feel we shouldn't
update tests, except it's riddled with private method mentions for
coverage purposes (not in terms of actual called code).
I think the best practice is to write good narrow unit tests,
not to write bad tests and then hide their coverage. Generally
that's what we do. Maybe these are useful when invoking a huge
legacy class, but generally we don't need these I think, at least
in the libs our team maintains.
Change-Id: I4c7d826c7ec654b9efa20778c46c498784661f1c
I noticed this in CI during development for Ia893ddb36427eb5e9bff,
however, it is reproducible on latest master as well. Locally, it
happens about 1/30 times:
```
$ php phpunit.php --filter testLockTSESlow unit/includes/libs/objectcache/
Using PHP 8.1.13
PHPUnit 9.5.27 by Sebastian Bergmann and contributors.
There was 1 failure:
1) WANObjectCacheTest::testLockTSESlow with data set #0 (array('hash_tag'), '{')
Callback not used (mutex not acquired, stale value used)
Failed asserting that 3 is identical to 2.
/w/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php:1311
/w/tests/phpunit/MediaWikiUnitTestCase.php:11
```
* Remove randomisation from test. The HashBag is empty at this point,
and a dedicated instance is created for each test case. Remove doubt
over conflicts or other uncertainty, and make it easier to debug at
the same time.
* Increase mocked time gap from 2s to 32s similar to what the
previous assertion in the same test does already. 2s does not actually
go past the reduced logical TTL which is 30s at this point.
It seems to still pass on master in most cases, but it makes sense
that it can fail. I could not reproduce failures after this change.
Change-Id: I6e491ae3c4f41413e7a952aa0796a787d559ac48
Loosen up the code coverage a bit so that we can more clearly identity
what logic truly isn't excercised by tests. It also restores some
gaps that were left behind by various refactors when code moved around
or out into various private methods.
Change-Id: I4e65f92ea1eb3f0e781ada4fb2ced0f8ef9d3cc4
When the cache key doesn't contain a prefix indentifying the collection,
don't use the entire cache key as the metrics name. Doing so may explode
the number of metrics.
Bug: T323357
Change-Id: Ia7ff3f060a36f848c7a74e39070bd62a932120e7
This avoids the latency of waiting for a response.
Change yieldStampedeLock() to use delete() instead of changeTTL() since
the PECL memcached client does not support WRITE_BACKGROUND with TOUCH.
Also, simply relayVolatilePurges() into relayVolatilePurge() and tweak
some related documentation for WANObjectCache
Bug: T302623
Change-Id: I16f011b730f34332aa54a48c579de3c6606dafda
This allows for large string value storage similar to that
of direct BagOStuff callers.
Bug: T309126
Change-Id: I22d2d8bb520ef2ea6ccd28016d34db36fe27c3df
Make set() and getWithSetCallback() wrap setMainValue(), avoiding some
duplicate option array resolution logic.
Track set() calls via statsd, now that it can be distinguished from
getWithSetCallback() invocations.
Change-Id: Ia1e7b244b1db3b76314d718b3aea9e6e708235c0
Introduced in 2017 with I7f14b9ca2533032 (2e5eb693) but remains
unused at WMF, and disabled by default.
Follows-up I62107789fa (9e49260fc9) which added reap to LinkCache
test cases in 2021.
Change-Id: I0654c29a671467dd6b366f462d1c09b90a273413
Follows-up Ia550ef7fe3 (4d3549ad71) which added reap to NameTableStore
in 2018, which was then (mostly) removed again with I63e61e1ab (2c7e4adcea)
for T198561, except one call was left behind in reloadMap(). I've
changed this to match the existing delete call in acquireId().
To be removed in I0654c29a671467dd6.
Change-Id: I7541c9c6d1ef70d552944aca94cbc4d0f7ec0d5b
And also assertNotRegExp -> assertDoesNotMatchRegularExpression. The
methods were renamed in PHPUnit 9.
Done automatically with:
grep -rl assertRegExp tests/ | xargs sed -r -i "s/>assertRegExp\(/>assertMatchesRegularExpression\(/"
grep -rl assertNotRegExp tests/ | xargs sed -r -i "s/>assertNotRegExp\(/>assertDoesNotMatchRegularExpression\(/"
Split out from Ifdba0f9e98eb6bce4590b7eb73170c51a697d7c6 so that it
remains smaller and easier to review.
Also make a test use MediaWikiUnitTestCase (it's already in the unit/
dir) so that it can access the forward-compat method.
Bug: T243600
Change-Id: Ifa279d5f201d7abeebece292141ebface8278046
It took me a good half hour to understand what these tests are doing,
and another half hour to build up a modicum of confidence that the
current behaviour is the indeed the desired behaviour.
* Remove random strings.
* Remove indirection and computed expected values as much as possible
in favour of inlined literals that explicitly expect outcomes.
* Remove confusing second argument from the data provider
getMultiWithUnionSetCallback_provider, which wasn't used, and
add labels to it.
* Remove some of the existing inline comments.
* Add some inline comments with what I've understood.
* Fix bug in "C: Repeat case with recently created check keys",
which claimed to be observing that caches were ignored due to the
check key being too new, when in fact it was a cache miss simply
because the key didn't exist yet upto this point. Make it actually
repeating by adding a second call afterwards to observe the
stated behaviour.
Bug: T303092
Change-Id: I874a20ecd49464f7b0e6a0b579c000dc0e804c0a
The examples and comments were misleading with regard to the handling
of missing entities.
Bug: T303092
Change-Id: Ib672fd18d1270c35253683eb54b48be056e6dc3c
The new style of checking for the last error during a section of
calls is more robust since it allows nesting of callers. Typically,
an external caller will want to watch a section of code that will
involve zero or more internally watched sections. Errors that are
seen internally (leading to a failing response) should also be
visible externally.
Replace internal BagOStuff clearLastError() calls.
Replace WANObjectCache clearLastError() calls. Such a class should not
clear the error codes since the class is effectively "internal". Callers
that are more meaningfully "external" might want to check the errors.
Cleanup "last" error handling for proxy backends.
Change-Id: I281817a85602967c0ec2bdd23a5d8be101680b64
Tweak testLockTSESlow() tests to reduce mt_rand() dependency.
Also, rename last few $now variables to $mockWallClock and use
integral floats to make it easier to find raw microtime() values
via debugging statements.
Change-Id: Ie2b125e42d3ac37104f5b87b75a9a81558956518
Handle edge case of fast callbacks that happen to run just before
the transaction age reaches MAX_READ_LAG. Such callbacks will no
longer be considered "slow".
Distinguish slow and fast regenerations for the case when TTL
mitigation is triggered by the combination of snapshot and
replication lag.
Reduce the cases where TTL_UNCACHEABLE is used as the mitigation
TTL, to avoid operational risk.
Simplify the mitigation TTL code by reducing needless variation
between the "lockTSE" and non-"lockTSE" cases.
Track the key and time of the last 10 get() calls that returned
either a stale value or no value at all. Use this this estimate
the walltime in set(), if it was not provided.
Bug: T285070
Change-Id: Ie6c88fbf7dbfc831d67bfb6cde082c123555a8cf
… including PHPDoc tags like `@return <type> $variableName`.
A return value doesn't have a variable name. I can see that
some people do this intentionally, repeating the variable
name that was used in the final `return $var;` at the end
of a method. This can indeed be helpful. I leave a lot of
these untouched and removed them only when it's obviously
wrong, or does not provide any additional information in
addition to what the code already says.
Change-Id: Ia18cd9f25ef658b08ad25b97a744897e2a8deffc
It is not entirely meaningless. It might be an indicator that
the number of calls to a method is intentionally unlimited.
This is similar to e.g. an @inheritDoc PHPDoc comment that
marks a method as being "intentionally undocumented".
However, what's the meaning of being "intentionally
unconstrained"? Let's just not have any constraint then.
I feel all these ->expects( $this->any() ) bloat the test
code so much that it's never worth it.
Change-Id: I9925e7706bd03e1666f6eb0b284cb42b0dd3be23
Ended up using
grep -Prl '\->setMethods\(' . | xargs sed -r -i 's/setMethods\(/onlyMethods\(/g'
special-casing setMethods( null ) -> onlyMethods( [] )
and then manual fix of failing test (from PS2 onwards).
Bug: T278010
Change-Id: I012dca7ae774bb430c1c44d50991ba0b633353f1
Also remove the unused 'region' option.
Added release notes, and updated the notes for coalesceKeys as well
to reside in the configuration section instead.
Change-Id: I219ea67416c6d121ce1488d99dab9a3a089f27d9
Renamed several methods for clarity and updated various comments
Use explicit number_format() instead of casting floats to strings
when making purge values
Fix LocalRepoTest mocking of WANObjectCache::makeGlobalKey()
Bug: T264604
Change-Id: Ib03b97fcab8a34d38d3a8a12da6e28e12feef4ad
* Split makePurgeValue() into one for "check" keys and one for
tombstones.
* Simplify tombstones by ommitting the zeroed-out holdoff component.
* Remove unused and deprecated HOLDOFF_NONE constant.
* Clarify various comments.
Change-Id: If6cc34538b1ea30764836b6870ba8648424ce9d7
Key coalescing hints are now unconditionally enabled, using
the mcrouter style by default. This can be changed by setting
"coalesceScheme" (e.g. in case of experimental dynomite use).
Change-Id: I4866ed66d42d7d7da2b3edc2f99b8ba35fe3d511
This reverts commit 4e94fa8812.
The extra serialization call here is not needed since storing
a value naturally results in serialization and thus exception
throwing already (hence the fatal error at T273242).
Performing serialization twice comes with a significant cost,
which has in past led to notable regressions.
Adding a try-catch makes sense, but this can be placed at a
higher level instead, as is already done by Ibfcbf0eceb4b7a36
which was already merged and back-ported, making this obsolete.
Other use cases:
- Continuous integration where HashBag is used, which doesn't
need serialize and thus doesn't cover the Closure issue.
Cost of serialize was removed there for performance reasons,
but if we want to add it back, let's do so there in the natural
place where it once was (or in MediumBag, which might be preferable
so that we can use that same code path to gradually switch to JSON
encoding and warn/throw for non-jsonic values).
- Regular getWithSet calls (rather than the async refresh from which
the fatal is currently seen). For regular getWithSet calls we already
get a useful stack trace that points directly to the line of code
where the value and cache key are computed, so additional logging
does not seem that valuable. But for consistency we could move the
try-catch from Ibfcbf0eceb4b7a36e33a8 a little lower to cover all
fetchOrRegenerate() calls, not just those from scheduleAsyncRefresh.
Bug: T273242
Change-Id: Ic649016dde7b58323a028ce6d4fe617ef62a1ff8
