All callers in Wikimedia gerrit repos have since been migrated to
incrWithInit(), which is usually what is desired and is easier to
implement in various backends. Newer memcached versions of the
memcached protocol even support auto-initializing increments
(including an initial TTL) via the 'ma' command. The incr()/decr()
methods currently bloat the interface and subclasses (often with
slow and buggy implementations). These methods are also hard to
implement in a multi-DC store (either extremely slow or racey)
even though callers might assume they handle high concurrency.
Also, force each MediumSpecificBagOStuff subclass to implement
incrWithInit() instead of having some rely on the base method
In mcc.php, add incrWithInit() and simplify parameters, making
it possible to test non-memcached backends.
Change-Id: I53c9c2c839a1e71d5c104913fea0680c30d11108
The new style of checking for the last error during a section of
calls is more robust since it allows nesting of callers. Typically,
an external caller will want to watch a section of code that will
involve zero or more internally watched sections. Errors that are
seen internally (leading to a failing response) should also be
visible externally.
Replace internal BagOStuff clearLastError() calls.
Replace WANObjectCache clearLastError() calls. Such a class should not
clear the error codes since the class is effectively "internal". Callers
that are more meaningfully "external" might want to check the errors.
Cleanup "last" error handling for proxy backends.
Change-Id: I281817a85602967c0ec2bdd23a5d8be101680b64
Avoid the duplication of the reentrant lock logic in subclasses.
Move "lock already expired" logic from getScopedLock() to unlock()
so that it applies more generally. Warn when unlock() is called on
a key that is not even locked as well.
Also:
* Rename lock() $expiry argument to $exptime for consistency
* Fix return types for serialize()/unserialize()
Bug: T274174
Change-Id: I211536e616cf7f1cc60181c378bbf9b35ffa40a4
Large wiki farms may have to purge servers concurrently (instead of
one at a time) in order to keep up with new writes and delete expired
rows faster than new rows are written.
The parameter for this uses server tags for three reasons:
* Maintenance risk and complexity.
This requires the least information about MW configuration to be
hardcoded in the scheduled maintenance cronjob (compared to: server
indexes which are a runtime concept, or specific hostnames/IP/
tableprefixes which may change and should not require
coordinating changes elsewhere).
* Operational convenience.
By using server tags, the parameters don't have to vary between
data centers.
* Code complexity.
The current code for obtaining connections is based on server indexes,
which are easy to mapped at runtime from server tags. Other ways
of identifying shard like hostnames are either an awkward fit (as
they don't uniquely identify a shard per-se, with multiple instances
on the same hardware at WMF), or require SqlBagOStuff to store and
maintain more information about connections than it currently has
readily available.
Bug: T282761
Change-Id: I618bc1e8ca3008a4dd54778ac24aa5948f27c52e
This is micro-optimization of closure code to avoid binding the closure
to $this where it is not needed.
Created by I25a17fb22b6b669e817317a0f45051ae9c608208
Change-Id: I0ffc6200f6c6693d78a3151cb8cea7dce7c21653
BagOStuff has a debug mode because MemcachedClient has a debug mode,
because Ryan T. Dean put one in there in 2003. It's not strictly
necessary now that we have log levels.
This means that a few existing $this->logger->debug() calls are now on
the same effective level as $this->debug() calls, they can't be enabled
separately. But that seems inconsequential.
Change-Id: I4d1cb9153559b29468b6ca530bbc56e2b35d1721
Update SQL, REST, and redis subclasses to emit call count and
payload size metrics for cache key operations. These metrics
are bucketed by cache key collection (similar to WANCache).
Bug: T235705
Change-Id: Icaa3fa1ae9c8b0f664c26ce70b7e1c4fc5f92767
Add "generic" key methods for quickly deriving keys from
key component lists in a bijective manor. This is useful
for BagOStuff classes that wrap other BagOStuff instances
or for parsing keys to get stats.
Make the proxy BagOStuff classes (ReplicatedBagOStuff,
MultiWriteBagOStuff, CachedBagOStuff) use "generic" keys
so that they can convert to appropriate keys when making
backing cache instance method calls.
Make EmptyBagOStuff, HashBagOStuff, APCUBagOStuff,
RedisBagOStuff, and RESTBagOStuff use "generic" keys rather
than those of MediumSpecificBagOStuff::makeKeyInternal().
This lets proxy BagOStuff classes bypass key conversions
when used with instances of these classes as backing stores.
Also:
* Fix missing incr(), incrWithInit(), and decr() return
values in MultiWriteBagOStuff.
* Make MultiWriteBagOfStuff, ReplicatedBagOStuff, and
CachedBagOStuff use similar backend method forwarding
styles by using a new BagOStuff method.
* Improved various related bits of documentation.
Bug: T250239
Bug: T235705
Change-Id: I1eb897c2cea3f5b756dd1e3c457b7cbd817599f5
For compliance with the new version of the table interface policy
(T255803).
This patch was created by an automated search & replace operation
on the includes/ directory.
Bug: T257789
Change-Id: If560596f5e1e0a3da91afc36e656e7c27f040968
The following glasses are marked as newable per the
Stable Interface Policy, even though logically, they should not be
newable. This is done for classes that are currently instantiated
by extensions, and lack an alternative.
A better way for obtaining an instance of these classes should be
created in the future. At that point, direct instantiation should
be deprecated and replaced.
- includes/ApiMain.php - needs factory
- includes/media/BitmapMetadataHandler.php - should become a stateless service or use handler pattern
- includes/GitInfo.php - should become a stateless service
- includes/logging/LogPage.php - should become a stateless service or use command pattern
- includes/logging/ManualLogEntry.php - should become a stateless service or use command pattern
- includes/poolcounter/PoolCounterWorkViaCallback.php - needs a factory
- includes/context/RequestContext.php - needs to be narrowed down, and should use a factory
- includes/search/SearchHighlighter.php - should have a factory
- includes/TitleArrayFromResult.php - should perhaps be part of TitleFactory
- includes/user/User.php - should at least get a factory method for anons
- includes/diff/Diff.php: needs a TextDiffGenerator service or a factory
- includes/EditPage.php: needs a factory
Bug: T247862
Change-Id: I033158e693c98630ee167d9528fc8c9936f978d4
This can be used by WANObjectCache or statsd metrics to estimate
the size of objects and network commands without having to do an
extra serialization round.
Change-Id: I7b1b787bba39a21e2e04262e77ee7ae24e88f8ca
Make the default $init value for incrWithInit() be $value.
This is far less suprising and also makes the operation
easier to replicate without conflicts.
Make decr() definitions more explicit since various cache
drivers do not handle negative incr() values (e.g. memcached).
Change-Id: I2b8d642656cc91c841abbd7a55d97eba101b027a
This should help fix the following issues that various repos
are getting from Phan as of late:
> Call with 5 arg(s) to \BagOStuff::makeKey() which only takes 2 arg(s)
> defined at ../../includes/libs/objectcache/BagOStuff.php:456
> <source="PhanParamTooMany"/>
Bug: T228563
Depends-On: I5cfba063821101325a5a7359e6b8ad71a0fb1b2f
Depends-On: Ifa5b96735376f2fbe3680799f960616ba8d357ff
Change-Id: Ic9df7f3ad7f356c7cbdfe1edfbe35821b931dda6
In SqlBagOStuff:
* Add modifyMulti() helper method to reduce code duplication
* Improve atomicity of add(), cas(), and changeTTL() queries
* Avoid integer serialization and improve atomicity of incr()
* Optimize new BagOStuff::changeTTLMulti() method
In BagOStuff:
* Add changeTTLMulti() method for subclasses to optimize
* Make set() ignore WRITE_ALLOW_SEGMENTS for integers so incr() works
* Strip WRITE_ALLOW_SEGMENTS flag from the setMulti() call in set() to
avoid triggering bogus sanity check exceptions
* Fix BagOStuffTest::testSetSegmentable failures via the above changes
* Enforce WRITE_ALLOW_SEGMENTS sanity check in setMulti() for all the
subclasses by using a final wrapper method
* Add WRITE_ALLOW_SEGMENTS sanity check to deleteMulti()
Bug: T113916
Change-Id: I25d1790fa9b0d1837643efccfa94a12043cfbf42
Make the method use a callable type hint for $progressCallback.
Also make the SQLBagOStuff subclass shuffle() the array of server and table
indexes in case the limit keeps getting applied. Only garbage collect on DBs
that were going to be written to anyway. Also use DeferredUpdates if possible.
Change-Id: I723e6377c26750ff98e33f7ab103c6162ae65f43
Also adjust the frequency default and put in a 100 row limit for
randomized purges. This adds a $limit parameter to the BagOStuff
method deleteObjectsExpiringBefore().
Change-Id: I66bfcc67e8e118f8c659dbbee13d54bf2cd937f5
Use it for ApiStashEdit so that large PaserOutput can be stored.
Add flag to allow for value segmentation on set() in BagOStuff.
Also add a flag for immediate deletion of segments on delete().
BagOStuff now has base serialize()/unserialize() methods.
Bug: T204742
Change-Id: I0667a02612526d8ddfd91d5de48b6faa78bd1ab5
Also, reorder some method definitions to match the base class.
This makes it easier to compare the two. In addition, refactor
the doWrite() method to make the calls clearer.
Change-Id: I795b395b9c54645f62962461a28067ae38b291ed
This makes it consistent with set() and merge(). Also, one subclass
was already using the field in this manner.
Clean up the code related to WRITE_SYNC in SqlBagOStuff.
Change-Id: I0fb84f4475311889507d3ef98afd4476fb81174f
This lets backends that support CAS take advantage of it.
Also fix race condition with merge() in asyncWrites mode where two
threads might add to a list/set but the first request finishes after
the second, which would erase an item during the set().
Bug: T198239
Change-Id: Ibd2539429746f12b4de74d0f031089cb80aa8b4b
Deprecate the unnamespaced version and move it to includes/compat.
Bug: T147167
Depends-On: I39c805bfb98b32f32f3d0dc1eee9e823afe1c21a
Change-Id: I3780c7adf51683f3f7adb35a88f9a25a0a2e2530
For some varargs a variable name is added with suffix ,... as seen for
many other varargs
Some @param are swapped, because there are in the wrong order
Enable Sniff MediaWiki.Commenting.FunctionComment.ParamNameNoMatch
Change-Id: I60fec6025bce824d5c67563ab7b65ad6cd628ad8
If someone writes an object into a BagOStuff, they typically expect that
later changes to the object will not affect the value stored.
MultiWriteBagOStuff's async write handling was violating this
expectation, which is potentially causing T168040.
Bug: T168040
Change-Id: Ie897b900befdc8998614af06f9339cd07665703e
Similar to what WANObjectCache and CachedBagOStuff are already doing.
Also add missing tests for WANObjectCache (similar to those for CachedBagOStuff).
Bug: T167465
Change-Id: I1a0c9324726aa6a1b221def985773b1b819181fd
* Add the ability to expose key BagOStuff attributes like whether the
emulation SQL cache is being used. Several callers use hacks to detect
this and can be updated later.
* Fallback to a safe empty WAN cache in the CACHE_DB case. This fixes
a regression from f4bf52e843.
* Also add this protection to the server cache, which could break
similarly before too.
* Also fix CACHE_ANYTHING by avoid the recursion risk from
fc1d4d7960 by just checking the disabled service map.
Bug: T123829
Bug: T141804
Change-Id: I17ee26138f69e01ec1aaddb55ab27caa4d542193
