Commit graph

440 commits

Author SHA1 Message Date
jenkins-bot
a93b8e7d0b Merge "objectcache: Remove deprecated WAN-methods in ObjectCache" 2019-10-11 18:24:49 +00:00
jenkins-bot
3300dc7b17 Merge "objectcache: Log debug message for backend of MainObjectStash" 2019-10-11 15:43:53 +00:00
Timo Tijhof
0cfc89dbc3 objectcache: Remove deprecated WAN-methods in ObjectCache
Deprecated as of 1.34, not used anywhere.

Change-Id: I4974f55a415115d2cbeb8be944e82c949e7b8c36
2019-10-11 02:26:05 +01:00
Timo Tijhof
bd16c5eb34 objectcache: Log debug message for backend of MainObjectStash
Bug: T234361
Change-Id: I8e33591dc92198269f45a25d14014af633a351f2
2019-10-10 21:30:47 +01:00
Timo Tijhof
b10b56dc69 objectcache: Deprecate ObjectCache::getWANInstance
This is not used anywhere. Once removed we can remove a bunch
more protected/private methods in this class that are all only
here still due to this one public method.

It was obsolete when the creation of WANObjectCache was
moved to ServiceWiring. In theory multiple instances can be
created but in the future that should be done by either constructing
the WANObjectCache directly where needed, or by implementing
it as a custom service.

Also deprecate ObjectCache::newWANCacheFromParams. This should
have been `@internal`, but oh well. Deprecate as well since
only needed by getWANInstance(). No longer used internally
either as of I118b6f01e4940.

Change-Id: Icc500e76c860c35ff40fdee47142c63091829dfe
2019-10-10 20:23:36 +00:00
Aaron Schulz
990e9268e3 objectcache: fix DatabaseSqlite::initSqliteDatabase() call location
This should be called in the non-LB cause (e.g. custom DBs).

Follow-up mishandled merge conflict in 3298a9db90.

Bug: T233117
Change-Id: I71a5c28db05a3c294e5c244037be2ee0209d61ad
2019-09-17 19:23:44 -07:00
Aaron Schulz
7c2d1be7d8 objectcache: remove references to recently removed methods
Follow-up to 9d5e3f56d5

Change-Id: Idca42b5ca405c31953a516aa3f96cf64cc0e0a12
2019-09-15 13:52:07 -07:00
Zoranzoki21
9d5e3f56d5 Remove getMainWANInstance and getMainStashInstance functions
from ObjectCache as they were deprecated in 1.28

Change-Id: I133470a1c69c836f38b1ae5fecc05e50b70f4457
2019-09-15 15:18:51 +00:00
Daimona Eaytoy
e70b5b3309 Unsuppress other phan issues (part 4)
Bug: T231636
Depends-On: I58e67c2b38389df874438deada4239510d21654f
Change-Id: I6e5fba7bd273219b1206559420b5bdb78734aa84
2019-08-31 17:13:39 +00:00
Daimona Eaytoy
fb3428eb8f Unsuppress other phan issues with low count
And also update approximated counts, which for the most part are lower
than reported (hooray!)

Bug: T231636
Depends-On: Ica50297ec7c71a81ba2204f9763499da925067bd
Change-Id: I78354bf5f0c831108c8f606e50c87cf6bc00d8bd
2019-08-30 09:42:15 +00:00
jenkins-bot
6ba38fa942 Merge "Remove $wgMemCachedDebug" 2019-08-29 18:06:20 +00:00
Aaron Schulz
f78e39d3d7 Remove $wgMemCachedDebug
Change-Id: I98380ac1cb66e33719b4836410311aea3513115c
2019-08-29 17:16:31 +00:00
Aaron Schulz
1b57f6a81b Make ObjectCache check the value of apc.enable_cli in CLI mode
Add HashBagOStuff fallback for APC in MWLBFactory::injectObjectCaches.

Also fix APC-cache variable typo in MWLBFactory.

Bug: T227838
Change-Id: I71cb2ca58972ea09ab2f64f7e47bda7a5096c19b
2019-08-29 09:04:18 -07:00
Aaron Schulz
3298a9db90 objectcache: SqlBagOStuff cleanups for sqlite and postgres
Make createTables() work for Postgres. Also add and use new
initSqliteDatabase() method for SQLite.

Also use ConvertibleTimestamp instead of wfTimestamp

Change-Id: I8e4505775e57fc6cd7380afa015fc0901a1bfc1b
2019-08-26 16:26:26 +00:00
jenkins-bot
7414a955f8 Merge "objectcache: rename getDB() to getConnection() in SqlBagOStuff" 2019-08-24 20:57:17 +00:00
Aaron Schulz
6bb0f4fb95 objectcache: rename getDB() to getConnection() in SqlBagOStuff
Change-Id: I7ec33e2622f5b418712d76bedcbb4be65376cf91
2019-08-24 13:18:50 -07:00
Aaron Schulz
94eb0c946f objectcache: improve BagOStuff arithmetic method signatures
Make the default $init value for incrWithInit() be $value.
This is far less suprising and also makes the operation
easier to replicate without conflicts.

Make decr() definitions more explicit since various cache
drivers do not handle negative incr() values (e.g. memcached).

Change-Id: I2b8d642656cc91c841abbd7a55d97eba101b027a
2019-08-24 12:50:37 -07:00
jenkins-bot
a158a4fb32 Merge "objectcache: move lag waiting to SqlBagOStuff::doCas() instead of overriding merge()" 2019-08-23 19:19:51 +00:00
jenkins-bot
2b5f468efa Merge "objectcache: add BagOStuff::fieldHasFlags() convenience method" 2019-08-23 19:14:59 +00:00
Aaron Schulz
7ff1a16397 objectcache: make more SqlBagOStuff methods private and rename shard variables
Also optimize silenceTransactionProfiler() when LoadBalancer is not in use

Change-Id: I83fd5c17058ba082a13e957e4b4590e2d1d5b581
2019-08-22 17:55:15 -07:00
Aaron Schulz
cdc0565705 objectcache: move lag waiting to SqlBagOStuff::doCas() instead of overriding merge()
Change-Id: I61b49bc7b00ffa8308d93e76f4bb94cf09ba9936
2019-08-22 15:07:14 -07:00
Aaron Schulz
b70e54e986 objectcache: add BagOStuff::fieldHasFlags() convenience method
Change-Id: Id2bf4ec0dd0999f988b70a895003c4b1aaae51a2
2019-08-22 15:07:14 -07:00
jenkins-bot
90d6f495bd Merge "objectcache: fix WRITE_ALLOW_SEGMENTS in BagOStuff cas() and add() methods" 2019-08-20 15:50:58 +00:00
Aaron Schulz
942f708b61 objectcache: fix WRITE_ALLOW_SEGMENTS in BagOStuff cas() and add() methods
Add MediumSpecificBagOStuff::getValueOrSegmentList() helper method.

Also:
* Use $keysMissing variable correctly in CachedBagOStuff::getMulti()
  to avoid extra overhead.
* Optimize mergeViaCas() when the current value matches the new one.

Change-Id: I5c4bd74379bc459216ac0278150ce3aecff3b851
2019-08-20 10:55:42 -04:00
Aaron Schulz
d421f22eb6 objectcache: make newFromId()/newWANcacheFromId()/getDefaultKeyspace() private
Change-Id: I30b99f23dbd9637ca8178d3a3650b4c38ec43e7d
2019-08-20 13:11:27 +00:00
Aaron Schulz
49025f52b2 objectcache: clean up MemcachedBagOStuff expiry handling
Partly a follow-up to 88640fd902.

Use real time in changeTTL() tests to fix all remaining
failures for BagOStuff sub-classes.

Change-Id: I537d665d6c8770a68a5a79233a913f1714881dfb
2019-08-10 12:12:12 -07:00
Aaron Schulz
88640fd902 objectcache: fix failing tests for non-HashBagOStuff backends
Use real time for testing absolute expirations with changeTTL().
Otherwise, backends like memcached or redis will fail since
they do not use the mock time.

Also:
* Make SqlBagOStuff actually override changeTTLMulti() by
  using the right method name
* Check TTL_INDEFINITE more explicitly for clarity
* Rename TTL conversion methods for clarity
* Use isRelativeExpiration() in MemcachedBagOStuff

Change-Id: I9365ceb31d4e7bef65906363d42b8c3020a66346
2019-08-07 14:07:22 +00:00
Aaron Schulz
83e294a892 Convert LoadBalancer::getConnection() callers to LoadBalancer::getConnectionRef()
Change-Id: Ie1ae322202105ce093779e6d0b00f335c71be31f
2019-07-21 01:02:49 -07:00
Aaron Schulz
bebe30333d objectcache: add MediumSpecificBagOStuff base class for non-proxy subclasses
This make it much clearer what needs to be overridden

Change-Id: I3073f8a0605f557c6a3a93d0d8401cddd0fb8dbe
2019-07-19 13:01:46 -07:00
Aaron Schulz
12f4ce87e9 objectcache: make getMultiWith(Union)SetCallback() usage easier
Add WANObjectCache::multiRemap() as an array_combine() wrapper for
easily working with IDs after getMultiWith(Union)SetCallback() calls.
Make the enforcement of uniqueness in makeMultiKeys() stricter and
discourage poor key design in comments. Add WANObjectCache::hash256()
method for getting good key component hashes.

Also avoid pointless use of ArrayIterator::getArrayCopy().

Change-Id: I61ffdbf4af4374864bac180df590b4dddc8da56b
2019-07-14 14:51:51 +00:00
Aaron Schulz
c50202a9e4 objectcache: only process cache non-LB connections in SqlBagOStuff
LB should manage any such process caching otherwise as it owns the handles

Change-Id: Ic9cd84c7c3cba790eb127e70f6506270e5bec2a1
2019-07-13 14:52:15 -07:00
jenkins-bot
b5f06bbd12 Merge "objectcache: Hard deprecate ObjectCache::getMainWANInstance()" 2019-07-13 21:34:28 +00:00
Aaron Schulz
383460afa3 objectcache: normalize BagOStuff method overriding pattern for *Multi() methods
Change-Id: I1bebb60307b1a166461cb5f9a55a79194cc0e363
2019-07-11 04:32:35 -07:00
jenkins-bot
12e0f910b4 Merge "Update $wgMainStash comments and merge in those from a deprecated ObjectCache method" 2019-07-11 02:10:14 +00:00
Aaron Schulz
d3c2e4d385 Update $wgMainStash comments and merge in those from a deprecated ObjectCache method
Change-Id: I807f36d9c51476a969d7046d57bfc32cf37c2745
2019-07-10 14:02:41 -07:00
Aaron Schulz
65b4b61284 objectcache: rename "slaveOnly" parameter to "replicaOnly" in SQLBagOStuff
Also removed redundant check

Change-Id: I9f454eb15f223f5a9816dca4510c1a9add540b49
2019-07-10 19:14:51 +00:00
Aaron Schulz
20a9c8d24e bagostuff: optimize SqlBagOStuff and fix failing segmentation tests
In SqlBagOStuff:
* Add modifyMulti() helper method to reduce code duplication
* Improve atomicity of add(), cas(), and changeTTL() queries
* Avoid integer serialization and improve atomicity of incr()
* Optimize new BagOStuff::changeTTLMulti() method

In BagOStuff:
* Add changeTTLMulti() method for subclasses to optimize
* Make set() ignore WRITE_ALLOW_SEGMENTS for integers so incr() works
* Strip WRITE_ALLOW_SEGMENTS flag from the setMulti() call in set() to
  avoid triggering bogus sanity check exceptions
* Fix BagOStuffTest::testSetSegmentable failures via the above changes
* Enforce WRITE_ALLOW_SEGMENTS sanity check in setMulti() for all the
  subclasses by using a final wrapper method
* Add WRITE_ALLOW_SEGMENTS sanity check to deleteMulti()

Bug: T113916
Change-Id: I25d1790fa9b0d1837643efccfa94a12043cfbf42
2019-07-09 22:53:18 -07:00
Aaron Schulz
4245a2c200 objectcache: deleteObjectsExpiringBefore() signature and code improvements
Make the method use a callable type hint for $progressCallback.

Also make the SQLBagOStuff subclass shuffle() the array of server and table
indexes in case the limit keeps getting applied. Only garbage collect on DBs
that were going to be written to anyway. Also use DeferredUpdates if possible.

Change-Id: I723e6377c26750ff98e33f7ab103c6162ae65f43
2019-07-09 02:32:06 -07:00
Derick Alangi
ff320d9e49 objectcache: Hard deprecate ObjectCache::getMainWANInstance()
This method was soft-deprecated in 1.28 and this patch cleans up
usage (via object cache) in core and uses services. So, this is now
fit for hard deprecation.

Usage
=====

https://codesearch.wmflabs.org/search/?q=getMainWANInstance&i=nope&files=&repos=

Change-Id: I081a979ed6ed7acf37168a6d81ca4048aae5dd9c
2019-07-09 09:29:57 +00:00
Aaron Schulz
c860f99a6e objectcache: tweak SqlBagOStuff purging to happen on write only
Also adjust the frequency default and put in a 100 row limit for
randomized purges. This adds a $limit parameter to the BagOStuff
method deleteObjectsExpiringBefore().

Change-Id: I66bfcc67e8e118f8c659dbbee13d54bf2cd937f5
2019-07-06 00:04:13 -07:00
Aaron Schulz
e2e2eb49d6 objectcache: optimize lock() and unlock() methods in SqlBagOStuff
Also clean up base method in BagOStuff.

Bug: T113916
Change-Id: I3a1c6afe531a8eae34608bc7fe0aa6f9f4d439fe
2019-07-04 14:00:02 +00:00
Derick Alangi
f415b57749 objectcache: Hard deprecate ObjectCache::getMainStashInstance()
This method was soft deprecated in 1.28 and usage is no longer in core
except for FlaggedRevs ext (here in removed in patch I3ada0b53e8b2a6f118f).

Usage
=====

https://codesearch.wmflabs.org/search/?q=getMainStashInstance&i=nope&files=&repos=

Depends-On: I3ada0b53e8b2a6f118fae37e7fc644b187bf95f8
Change-Id: I3f06b3732f67b703cdc12ec5849b67067475cdda
2019-07-02 18:27:24 +01:00
jenkins-bot
699920cc66 Merge "objectcache: add object segmentation support to BagOStuff" 2019-06-13 18:19:17 +00:00
Aaron Schulz
b09b3980f9 objectcache: add object segmentation support to BagOStuff
Use it for ApiStashEdit so that large PaserOutput can be stored.

Add flag to allow for value segmentation on set() in BagOStuff.
Also add a flag for immediate deletion of segments on delete().

BagOStuff now has base serialize()/unserialize() methods.

Bug: T204742
Change-Id: I0667a02612526d8ddfd91d5de48b6faa78bd1ab5
2019-06-11 16:14:17 +01:00
Umherirrender
5f43b1706c Pass options as array to IDatabase::insert
The documentation only allows arrays there

Change-Id: I00c6e47a817a70bed9a443aebc675ef4c3d6b1e5
2019-06-07 19:12:35 +02:00
Aaron Schulz
e742701091 objectcache: make detectLocalServerCache() prefer apcu over apc
Bug: T220470
Change-Id: I8b85859b3991d2360a74107299d44da2c60d0214
2019-05-22 22:23:22 +00:00
Aaron Schulz
384a27e7ac Remove unused EventRelayer code from ObjectCache::newWANCacheFromParams()
Change-Id: I6c0f274e3c49312b63446732f22123af9dac8a2b
2019-05-09 16:13:15 -07:00
Aaron Schulz
962495b17a objectcache: merge getWithToken() into doGet() for simplicity
Change-Id: I581f866521e1086ca350973d9cdeff6656f48fe8
2019-03-27 13:03:25 -07:00
jenkins-bot
8818ee3a8d Merge "objectcache: fix return value type in SqlBagOStuff::incr" 2019-03-27 18:10:56 +00:00
Aaron Schulz
f80671a784 objectcache: fix return value type in SqlBagOStuff::incr
Change-Id: I14a70956f77f13346857236eda5c4b6fb80c6223
2019-03-26 17:08:37 -07:00