Commit graph

407 commits

Author SHA1 Message Date
Aaron Schulz
383460afa3 objectcache: normalize BagOStuff method overriding pattern for *Multi() methods
Change-Id: I1bebb60307b1a166461cb5f9a55a79194cc0e363
2019-07-11 04:32:35 -07:00
jenkins-bot
12e0f910b4 Merge "Update $wgMainStash comments and merge in those from a deprecated ObjectCache method" 2019-07-11 02:10:14 +00:00
Aaron Schulz
d3c2e4d385 Update $wgMainStash comments and merge in those from a deprecated ObjectCache method
Change-Id: I807f36d9c51476a969d7046d57bfc32cf37c2745
2019-07-10 14:02:41 -07:00
Aaron Schulz
65b4b61284 objectcache: rename "slaveOnly" parameter to "replicaOnly" in SQLBagOStuff
Also removed redundant check

Change-Id: I9f454eb15f223f5a9816dca4510c1a9add540b49
2019-07-10 19:14:51 +00:00
Aaron Schulz
20a9c8d24e bagostuff: optimize SqlBagOStuff and fix failing segmentation tests
In SqlBagOStuff:
* Add modifyMulti() helper method to reduce code duplication
* Improve atomicity of add(), cas(), and changeTTL() queries
* Avoid integer serialization and improve atomicity of incr()
* Optimize new BagOStuff::changeTTLMulti() method

In BagOStuff:
* Add changeTTLMulti() method for subclasses to optimize
* Make set() ignore WRITE_ALLOW_SEGMENTS for integers so incr() works
* Strip WRITE_ALLOW_SEGMENTS flag from the setMulti() call in set() to
  avoid triggering bogus sanity check exceptions
* Fix BagOStuffTest::testSetSegmentable failures via the above changes
* Enforce WRITE_ALLOW_SEGMENTS sanity check in setMulti() for all the
  subclasses by using a final wrapper method
* Add WRITE_ALLOW_SEGMENTS sanity check to deleteMulti()

Bug: T113916
Change-Id: I25d1790fa9b0d1837643efccfa94a12043cfbf42
2019-07-09 22:53:18 -07:00
Aaron Schulz
4245a2c200 objectcache: deleteObjectsExpiringBefore() signature and code improvements
Make the method use a callable type hint for $progressCallback.

Also make the SQLBagOStuff subclass shuffle() the array of server and table
indexes in case the limit keeps getting applied. Only garbage collect on DBs
that were going to be written to anyway. Also use DeferredUpdates if possible.

Change-Id: I723e6377c26750ff98e33f7ab103c6162ae65f43
2019-07-09 02:32:06 -07:00
Aaron Schulz
c860f99a6e objectcache: tweak SqlBagOStuff purging to happen on write only
Also adjust the frequency default and put in a 100 row limit for
randomized purges. This adds a $limit parameter to the BagOStuff
method deleteObjectsExpiringBefore().

Change-Id: I66bfcc67e8e118f8c659dbbee13d54bf2cd937f5
2019-07-06 00:04:13 -07:00
Aaron Schulz
e2e2eb49d6 objectcache: optimize lock() and unlock() methods in SqlBagOStuff
Also clean up base method in BagOStuff.

Bug: T113916
Change-Id: I3a1c6afe531a8eae34608bc7fe0aa6f9f4d439fe
2019-07-04 14:00:02 +00:00
Derick Alangi
f415b57749 objectcache: Hard deprecate ObjectCache::getMainStashInstance()
This method was soft deprecated in 1.28 and usage is no longer in core
except for FlaggedRevs ext (here in removed in patch I3ada0b53e8b2a6f118f).

Usage
=====

https://codesearch.wmflabs.org/search/?q=getMainStashInstance&i=nope&files=&repos=

Depends-On: I3ada0b53e8b2a6f118fae37e7fc644b187bf95f8
Change-Id: I3f06b3732f67b703cdc12ec5849b67067475cdda
2019-07-02 18:27:24 +01:00
jenkins-bot
699920cc66 Merge "objectcache: add object segmentation support to BagOStuff" 2019-06-13 18:19:17 +00:00
Aaron Schulz
b09b3980f9 objectcache: add object segmentation support to BagOStuff
Use it for ApiStashEdit so that large PaserOutput can be stored.

Add flag to allow for value segmentation on set() in BagOStuff.
Also add a flag for immediate deletion of segments on delete().

BagOStuff now has base serialize()/unserialize() methods.

Bug: T204742
Change-Id: I0667a02612526d8ddfd91d5de48b6faa78bd1ab5
2019-06-11 16:14:17 +01:00
Umherirrender
5f43b1706c Pass options as array to IDatabase::insert
The documentation only allows arrays there

Change-Id: I00c6e47a817a70bed9a443aebc675ef4c3d6b1e5
2019-06-07 19:12:35 +02:00
Aaron Schulz
e742701091 objectcache: make detectLocalServerCache() prefer apcu over apc
Bug: T220470
Change-Id: I8b85859b3991d2360a74107299d44da2c60d0214
2019-05-22 22:23:22 +00:00
Aaron Schulz
384a27e7ac Remove unused EventRelayer code from ObjectCache::newWANCacheFromParams()
Change-Id: I6c0f274e3c49312b63446732f22123af9dac8a2b
2019-05-09 16:13:15 -07:00
Aaron Schulz
962495b17a objectcache: merge getWithToken() into doGet() for simplicity
Change-Id: I581f866521e1086ca350973d9cdeff6656f48fe8
2019-03-27 13:03:25 -07:00
jenkins-bot
8818ee3a8d Merge "objectcache: fix return value type in SqlBagOStuff::incr" 2019-03-27 18:10:56 +00:00
Aaron Schulz
f80671a784 objectcache: fix return value type in SqlBagOStuff::incr
Change-Id: I14a70956f77f13346857236eda5c4b6fb80c6223
2019-03-26 17:08:37 -07:00
Aaron Schulz
861683b722 objectcache: fix race condition in SqlBagOStuff::changeTTL
Also fix --use-bagostuff parameter when given 0

Change-Id: I64971d4ccad2c856cc7688577ae564c8db068819
2019-03-26 17:06:21 -07:00
Aaron Schulz
d5afd5f909 objectcache: minor refactoring to BagOStuff
Split up expiry functions and moved add() near other primitive writes.

Change-Id: I97bb91e648b797b1ab5a6934d212689b1e67f7c6
2019-03-26 15:14:42 -07:00
Derick Alangi
47e7f59682 objectcache: Remove unused variable $cache which is overriden
This variable was defined to false and immediately overriden in the
if condition below it which makes the assignment above useless.

Change-Id: Ie72ec7a8860d2a38f59f38b3d861217d39da3e29
2019-03-25 15:20:19 +01:00
Aaron Schulz
e2b03d5d30 objectcache: add BagOStuff::deleteMulti() method for consistency
Also:
* Make the BagOStuff tests actually pass for memcached and sql
* Add more unit tests for BagOStuff
* Make SqlBagOStuff::add() more atomic

Change-Id: Ic1eec0990a66b595b57c646498c3bd229442230c
2019-03-14 23:12:30 +00:00
Aaron Schulz
547f207fa6 objectcache: make the $flags argument appear more consistently in BagOStuff
Change-Id: I08879ede5e9f0ab227497bb1dab89ea61b65abce
2019-03-12 22:55:18 +00:00
jenkins-bot
db1f41172f Merge "objectcache: add $flags argument to BagOStuff::delete()" 2019-03-08 00:51:25 +00:00
Aaron Schulz
ebd1343860 objectcache: remove obsolete code from SqlBagOStuff::handleWriteError()
Given that the DB classes already check read-only mode and bad
transaction states are already handled by 3975e04cf4, this
code was not useful.

Change-Id: I4b6e8909196448a12cf74598de0272fa43bdfc62
2019-03-07 13:21:41 -08:00
Aaron Schulz
5dd409cd38 objectcache: add $flags argument to BagOStuff::delete()
This makes it consistent with set() and merge(). Also, one subclass
was already using the field in this manner.

Clean up the code related to WRITE_SYNC in SqlBagOStuff.

Change-Id: I0fb84f4475311889507d3ef98afd4476fb81174f
2019-03-07 13:15:29 -08:00
Fomafix
4b46994799 Fix usage of MediaWikiServices in comments and documentation
MediaWikiServices::getFoo()

is wrong. Right is:

MediaWikiServices::getInstance()->getFoo()

Change-Id: Ib6d844ddfe5bd6ccd72b887a63d9ad476c8d196f
2019-03-03 14:09:33 +00:00
Aaron Schulz
5196ac32c6 Rename WikiMap DB domain ID methods to reduce confusion with web domains
Those added in this same release do not need alias methods.

Change-Id: I05feeb9b0b13afe43aea1f95551965489cdbe094
2019-02-06 12:28:45 -08:00
jenkins-bot
bd78869618 Merge "No yoda conditions" 2018-12-09 01:34:23 +00:00
Timo Tijhof
6be5e0a92c objectcache: Add urls for mysql bug refs in documentation
Change-Id: Ie5fe5dacae364b2b0c07f2ab0aa9b6777277c10f
2018-12-05 09:20:42 -05:00
Fomafix
3ee1560232 No yoda conditions
Replace
  if ( 42 === $foo )
by
  if ( $foo === 42 )

Change-Id: Ice320ef1ae64a59ed035c20134326b35d454f943
2018-11-21 17:54:39 +01:00
jenkins-bot
988af574cb Merge "Use new WikiMap::getCurrentWikiDomain() for objectcache and profiler keyspace" 2018-10-30 07:08:27 +00:00
Aaron Schulz
42a6f991e0 Use new WikiMap::getCurrentWikiDomain() for objectcache and profiler keyspace
Change-Id: I0e77c58a55b7ce908ec1d6399ec6629d00a48953
2018-10-30 06:41:07 +00:00
Fomafix
43244db9a2 Use PHP 7 '??' operator instead of if-then-else
Change-Id: If9d4be5d88c8927f63cbb84dfc8181baf62ea3eb
2018-10-21 21:46:46 +02:00
Umherirrender
719bfa8977 Use a ScopedCallback to silence transaction profiler in SqlBagOStuff
This affects most LCStoreDB which is the default cache
and spamming the log.

Bug: T154424
Bug: T97693
Change-Id: I68af177c7d0d88ab266eb3fefb4336eff86d861c
2018-10-12 20:04:31 +02:00
Bartosz Dziewoński
485f66f174 Use PHP 7 '??' operator instead of '?:' with 'isset()' where convenient
Find: /isset\(\s*([^()]+?)\s*\)\s*\?\s*\1\s*:\s*/
Replace with: '\1 ?? '

(Everywhere except includes/PHPVersionCheck.php)
(Then, manually fix some line length and indentation issues)

Then manually reviewed the replacements for cases where confusing
operator precedence would result in incorrect results
(fixing those in I478db046a1cc162c6767003ce45c9b56270f3372).

Change-Id: I33b421c8cb11cdd4ce896488c9ff5313f03a38cf
2018-05-30 18:06:13 -07:00
Max Semenik
69aecc2eea Don't initialize MediaWikiServices before extensions have been loaded
Bug: T153256
Bug: T190425
Change-Id: I749f66d13a8c8a8ae4a83661b83c56f0db74a187
2018-05-03 23:36:10 -07:00
Aaron Schulz
8350329922 Avoid warnings in ObjectCache::newWANCacheFromParams if "channels" is not set"
Bug: T192473
Change-Id: I150eef62285e99fc1269a58fee3827e5fbb0537b
2018-04-19 07:53:13 +00:00
Aaron Schulz
f8c2486d15 rdbms: rename CONN_TRX_AUTO constant to CONN_TRX_AUTOCOMMIT
The "AUTO" means AUTOCOMMIT, not "automatic transactions"/DBO_TRX,
which is basically the opposite concept. The new name does not
suffer from that ambiguity.

Keep the old constant as an alias for backwards compatibility.

Also remove LoadBalancer comment about non-existing field

Change-Id: I63beeb061fc9be73f320308e4d6393b58628b8c8
2018-04-12 13:01:52 -07:00
Reedy
39f0f919c5 Update suppressWarning()/restoreWarning() calls
Bug: T182273
Change-Id: I9e1b628fe5949ca54258424c2e45b2fb6d491d0f
2018-02-10 08:50:12 +00:00
Aaron Schulz
8f4ae0d51c Fix LoadBalancer::getMasterPos() callers to handle a non-object being returned
Change-Id: I3356057f27a051835734329afa339ea76842a389
2018-01-12 15:18:14 -08:00
Thiemo Mättig
409da2d8b3 Remove leading backslashes from "use \…" tags
Change-Id: I494b029de089a07e3b946ee78293a12d5036f63e
2017-12-28 16:30:05 +01:00
Aaron Schulz
8ed0b8db05 Only send statsd data for WAN cache in non-CLI mode
Bug: T181385
Change-Id: Ic51a084206cea93621393359ded4ff77beef5f13
2017-12-02 12:34:22 -08:00
Aaron Schulz
73b928750c objectcache: Run preemptive WAN cache refreshes post-send
This keeps HTTP request time consistent in case of expensive keys

Change-Id: I0746fde29a6e2f27d1b92f1af599c741d5972f46
2017-12-01 08:15:26 +00:00
Aaron Schulz
4432e898be Add statsd metric support to WANObjectCache
Bug: T178531
Change-Id: I3037281d09cd5195347789f544deae89711f128b
2017-10-25 01:41:31 +00:00
Aaron Schulz
99c80a8fc7 rdbms: Support secondary autocommit connections in LoadBalancer
This is useful for things like SqlBagOStuff and JobQueueDB
being able to write in auto-commit mode while the main
transaction round still goes on.

SqlBagOStuff already had this sort of logic.

JobQueueDB now also takes this approach so it does not have to
defer insertion except for sqlite. This makes callers more likely
to know when insertion failed or not.

Make sure LoadBalancer sets "master"/"replica" LB info itself;
this fixes a bug found in the new tests.

Bug: T140338
Bug: T42451
Change-Id: I4199a9598d7afbf976a6efa8ed84b85b56da02bd
2017-08-18 01:28:34 +00:00
Umherirrender
a9007e8baf Add missing & to @param documentation to match functon call
Change-Id: I81e68310abcbc59964b22e0e74842d509f6b1fb9
2017-08-11 18:47:46 +02:00
Aaron Schulz
f24b3899e3 objectcache: Use a separate postgres connection in SqlBagOStuff
The flags to the driver use new connections for new LBs since
fda4d46fc4. This makes it consistent with what we do for
MySQL already.

This should fix warnings about TransactionProfiler in objectcache,
as well as warnings about "Pending writes" in WANObjectCache.

Bug: T167946
Bug: T154424
Change-Id: I0b0d9a7210b6a3270d32df778fcc4b9918d3dcd1
2017-07-06 23:08:06 +00:00
Tim Starling
fb91d409d6 Fix SqlBagOStuff exptime uniqueness assumption
The WMF parser cache tables have some 30000 rows with an identical, old
expiry time. So each time deleteObjectsExpiringBefore() is run, the
first batch on each table is correct, but the keys for the second batch
are selected with exptime > X, where X is shared exptime of the many old
rows.

So, use exptime >= X instead.

Change-Id: I6853f64f88f65d4529be2a42c9ed70cfa62cf653
2017-06-10 20:54:23 +10:00
Aaron Schulz
dd359741cc Move DB errors to Rdbms namespace
Change-Id: I463bd86123501abc68fdb78b4cda6110f7af2549
2017-04-15 10:47:41 -07:00
Aaron Schulz
d735dc562d Move Database and subclasses to Rdbms namespace
Change-Id: I52bef87512f9ddd155d1f4cc0052f6b7a0db5b42
2017-04-12 10:43:57 -07:00