Add WANObjectCache::multiRemap() as an array_combine() wrapper for
easily working with IDs after getMultiWith(Union)SetCallback() calls.
Make the enforcement of uniqueness in makeMultiKeys() stricter and
discourage poor key design in comments. Add WANObjectCache::hash256()
method for getting good key component hashes.
Also avoid pointless use of ArrayIterator::getArrayCopy().
Change-Id: I61ffdbf4af4374864bac180df590b4dddc8da56b
Add FLD_VALUE_VERSION key to the value wrapper array to hold the version
number used in getWithSetCallback(). Remove the VFLD_* wrapper array from
FLD_VALUE for versioned values.
Keys stored with the old VFLD_VERSION and VFLD_DATA fields will be seen as
having the wrong version. The previous WAN cache code will see the new keys
that use FLD_VALUE_VERSION as having the wrong version too. In either case,
the usual variant key logic applies, so there should not be any issues.
This means that moving from a non-versioned to a versioned cache key is no
longer a breaking change when, for the same key, some code passes a version
number to getWithSetCallback() while other code does not.
Also:
* Make "pcTTL" respect the version number for sanity
* Make sure set() respects TTL_UNCACHEABLE for completeness
* Track slow regeneration callback runtime in FLD_GENERATION_TIME
* Remove is_callable() check overhead and rely on PHP Error instances
* Refactor unwrap() to return a more immediately useful value
* Simplify getNonProcessCachedKeys() signature by using $opts
* Split out PURGE_* constants for purge entries since those keys are
never stored in any serialize value but are only in PHP arrays
* Rename doGetWithSetCallback() to be more succinct
* Rename and reorganize some variables for clarity
Change-Id: I4060b19583cdfd9fa36c91d7014441eeef4b3609
This changeset implements T89432 and related tickets and is based on exploration
done at the Prague Hackathon. The goal is to identify tests in MediaWiki core
that can be run without having to install & configure MediaWiki and its dependencies,
and provide a way to execute these tests via the standard phpunit entry point,
allowing for faster development and integration with existing tooling like IDEs.
The initial set of tests that met these criteria were identified using the work Amir did in
I88822667693d9e00ac3d4639c87bc24e5083e5e8. These tests were then moved into a new subdirectory
under phpunit/ and organized into a separate test suite. The environment for this suite
is set up via a PHPUnit bootstrap file without a custom entry point.
You can execute these tests by running:
$ vendor/bin/phpunit -d memory_limit=512M -c tests/phpunit/unit-tests.xml
Bug: T89432
Bug: T87781
Bug: T84948
Change-Id: Iad01033a0548afd4d2a6f2c1ef6fcc9debf72c0d
Avoid the ADD operation spam from all threads trying to access
a tombstoned key by checking the interim value cache timestamp.
This also avoids the GET/CAS spam from threads that manage to
get the mutex. If a single thread repeatedly accesses the same
tombstoned value in rapid succession, there will significantly
less cache operation spam.
Do the same for cache updates to keys in the holdoff state
due to "check keys" or the "touchedCallback" function.
Relatedly, fix getWithSetCallback() to disregard interim values
set prior to or at the same time as the latest delete() call.
This can slightly reduce the chance of the cache being behind
replica DBs for a second. It also avoids unit test failures
were a series of deletes and cache access happen at the same
timestamp (via time injection or regular system time calls).
In addition:
* Add PASS_BY_REF flag with backwards compatibility to avoid
bloating the signature of get()/getMulti() with the new
tombstone information needed for the above changes.
* Avoid confusing pass-by-reference in getInterimValue() and
fix use of incorrect $asOf parameter.
* Move some more logic into setInterimValue().
* Update some comments regarding broadcasted operations that
were not true for the currently assumed mcrouter setup.
* Rename $cValue => $curValue and $versioned => $needsVersion
for better readability.
Bug: T203786
Change-Id: I0eb3f9b697193d39a70dd3c0967311ad7e194f20
Instead, make the "high read lag with lockTSE" case just lower FLD_TTL.
This also avoids constant regenerations by threads getting the mutex.
Added logging for TTL adjustments in set() for the lockTSE case.
Also remove some delete() calls from tests that were not needed.
Change-Id: Id7695f0377235e4a2f6e0efc88e870c8a990c3b0
* Remove 'channels' field references from config/setup
* Remove 'relayer'/'channels' field reference in unit tests
* Remove unused DEFAULT_PURGE_CHANNEL class constant
* Also remove long-since bogus 'pool' field references
Follow-up to 4753b0a4ed
Change-Id: If6670ff4e1dccc8ae253a08b46d205601da10024
* Fix the timestamps to static hard-coded values
* Force the timestamps before various get/set tests so they do not
use the microtime() value either.
* Remove the direct and duplicated (causing further risk) microtime()
calls from testGetWithSeveralCheckKeys().
Bug: T207247
Change-Id: Id30a8127f11501dbe54e075b6e9d18490353f4a5
This is useful when the timestamps to be checked depend on the value or are stored
in the database rather than as check keys.
Change-Id: I81ab08a943ee7d2f96a132d371965501941ed37f
This can be used to discard all values before a certain point in
time, such as periods of severe network problems or misconfiguration
that resulted in missed purges.
Change-Id: I612db8f91a5960b912e9f35645a3d3872df47460
Also removed useless line from testLockTSE(). That would have needed
to be using $this->internalCache and those locks are freed immediately.
Bug: T180035
Change-Id: Ida1a923f779aaf8410da76643457d2200da6cb20
Callback code that happens to make use of $oldValue might not be able
to handle missing, new, or changed fields due to key version changes.
Overhaul testGetWithSetCallback_versions() to be cleaner and cover the
case of unversioned => versioned keys.
Change-Id: If108a73078c530c985d30bdadcbfa9ddd53dc2be
Use of microtime() is now just for baselines, and it is no longer
assumed to be increasing with each call. Such an assumption is
particuliarly bad on Windows.
I've done 100X run rounds with now failures on Windows.
Change-Id: Ica2a47982495bc95b10ca507414972744ea9507e
This simply involves passing it through to the set() call
Also added some related commons to adaptiveTTL() involving
usage of this option.
Change-Id: Id5833a5d4efb6cad2eb646832e5b0188e86e12fc
This stores values for very short times while the main
value is a tombstone in case of particularly high traffic.
Also make mutex keys expire immediately on unlock.
Change-Id: I4ec5cf7f8b49239fdd2518e5d955534877a0f7ee
class WANObjectCache (methods: 40% -> 67%, lines: 65% -> 88%)
* Allow indirect coverage of protected/private utility methods from
existing tests.
* Add basic test for setLogger().
* Add basic test for newEmpty().
* Add basic test for getQoS().
Change-Id: Ifb79ed2ff1febbd2f5477b8ed6319992ce88eb29
Reenable MediaWiki.WhiteSpace.SpaceBeforeClassBrace.NoSpaceBeforeBrace,
because the mentioned bug is fixed
Bug: T172933
Change-Id: I1593bdba2295ebed401b921f2beabed69dba7638
The used phpcs has a bug, so the version 0.9.0 could not be enforced at the moment.
Will be fixed in next version, see T167168
Changed:
- Remove duplicate newline at end of file
- Add space between function and ( for closures
- and -> &&, or -> ||
Change-Id: I4172fb08861729bccd55aecbd07e029e2638d311
Similar to what WANObjectCache and CachedBagOStuff are already doing.
Also add missing tests for WANObjectCache (similar to those for CachedBagOStuff).
Bug: T167465
Change-Id: I1a0c9324726aa6a1b221def985773b1b819181fd
Backend cache queries are now avoided if they are fresh in memory.
Also factor out some code into private methods for clarity.
Change-Id: Ib343fde3dbf63f39e8cb09eca6278811a7d9738b
The warmup cache was not properly prefixed and was also using the entity
IDs instead of the cache keys. Thus, it effectively just wasted a
getMulti() query and resulted in the usual separate GETs anyway.
Added some unit tests for this.
Change-Id: I75b7a31214b515511856f9d95db32e8881d80ccc
Replaces \TestingAccessWrapper (defined in core) with
\Wikimedia\TestingAccessWrapper (defined in the composer package
wikimedia/testing-access-wrapper).
See https://gerrit.wikimedia.org/r/#/q/topic:librarize-testing-access-wrapper
for downstream patches.
The core version of the class is kept around for a while to avoid
circular dependency problems.
Bug: T163434
Change-Id: I52cc257e593da3d6c3b01a909e554a950225aec8
The default will remain PHPUnit 4.x due to PHP 5.5 support.
But, we should allow developers to run tests with newer PHPUnit
versions which are noticably faster (especially for code coverage
reports).
* <https://github.com/sebastianbergmann/phpunit/wiki/Release-Announcement-for-PHPUnit-5.4.0>
PHPUnit 5 deprecates the getMock() shortcut for getMockBuilder()->getMock().
It instead introduces the shortcut createMock() which has better defaults
than getMockBuilder(). For example, it sets 'disableArgumentCloning' and
other things by default.
Going forward, code should either use getMockBuilder directly and configure
it using the setter methods (instead of the confusing variadic arguments
of getMock) or simply use the new minimalistic createMock method. This patch
backports the createMock method to MediaWikiTestCase so that we can start
using it.
Change-Id: I091c0289b21d2b1c876adba89529dc3e72b99af2
* This fixes keys based on some sort of change log.
Updates are wrapped in a mutex and keep track of the
last known good position.
* Make WANObjectReapUpdate class that cleans up title
related keys using the recentchanges table. This triggers
as a deferred updates on RC view.
Change-Id: I7f14b9ca2533032147e62b1a3cc004a23da86579
Causes HHVM 3.12.7 to segfault and or:
Fatal error: Stack overflow in includes/libs/objectcache/BagOStuff.php on line 754
This reverts commit b47ce21cec.
Bug: T150833
Change-Id: Ia52fdf65f5e386c0eaa1046328680fb5c8a9081a
This works by setting a callback to return the cache set
options. The callback will watch DB reads and create a
merged result from said usage.
This handles callers that are missing getCacheSetOptions().
Change-Id: Ia264f011e45e8cf105480955dad7e2c4c2357b73
Because the process cache can be lagged by virtue of blind TTL,
the HOLDOFF_TTL might not be enough to account for it, so avoid
using it when already inside a callback.
Also split of the tests from the MediaWiki test class, so this
does not require DB access anymore.
Change-Id: I743a1233a5efc7f036fad140a9ff8a30b32f8f27
Follows-up 24200e8, 37df4f8. Test coverage still broken.
> Trying to @cover not existing method
> "WANObjectCache::getMultitWithSetCallback".
Change-Id: Ie4c9a998ea352f93f3c7dcdea25ea5c017ffbab5
This does what it says on the tin, e.g. a way to fetch
multiple keys at once. Callbacks are still called in a
serial manner when needed.
Change-Id: I8e24a6de7f46499a53ec41636c5a4f106b9b3d09
This lets callers use adaptive TTLs on the near-expiration
preemptive refreshes if the new and current values match,
using the as-of time as $mtime.
Change-Id: Ie541c35f890c9f789d1accf9f2a43506daaf31f0
This lets callers have their own process cache but keep it
managed in a central location, making it easier to reset
when DB transaction snapshot are flushed or when unit tests
want to clear caches between runs.
Use this to replace the Revision text process cache.
Change-Id: Ic61ee9140d4ce9836cc4650adb5bb75a291fea18
