Commit graph

150 commits

Author SHA1 Message Date
Aaron Schulz
990e9268e3 objectcache: fix DatabaseSqlite::initSqliteDatabase() call location
This should be called in the non-LB cause (e.g. custom DBs).

Follow-up mishandled merge conflict in 3298a9db90.

Bug: T233117
Change-Id: I71a5c28db05a3c294e5c244037be2ee0209d61ad
2019-09-17 19:23:44 -07:00
Daimona Eaytoy
e70b5b3309 Unsuppress other phan issues (part 4)
Bug: T231636
Depends-On: I58e67c2b38389df874438deada4239510d21654f
Change-Id: I6e5fba7bd273219b1206559420b5bdb78734aa84
2019-08-31 17:13:39 +00:00
Aaron Schulz
3298a9db90 objectcache: SqlBagOStuff cleanups for sqlite and postgres
Make createTables() work for Postgres. Also add and use new
initSqliteDatabase() method for SQLite.

Also use ConvertibleTimestamp instead of wfTimestamp

Change-Id: I8e4505775e57fc6cd7380afa015fc0901a1bfc1b
2019-08-26 16:26:26 +00:00
jenkins-bot
7414a955f8 Merge "objectcache: rename getDB() to getConnection() in SqlBagOStuff" 2019-08-24 20:57:17 +00:00
Aaron Schulz
6bb0f4fb95 objectcache: rename getDB() to getConnection() in SqlBagOStuff
Change-Id: I7ec33e2622f5b418712d76bedcbb4be65376cf91
2019-08-24 13:18:50 -07:00
Aaron Schulz
94eb0c946f objectcache: improve BagOStuff arithmetic method signatures
Make the default $init value for incrWithInit() be $value.
This is far less suprising and also makes the operation
easier to replicate without conflicts.

Make decr() definitions more explicit since various cache
drivers do not handle negative incr() values (e.g. memcached).

Change-Id: I2b8d642656cc91c841abbd7a55d97eba101b027a
2019-08-24 12:50:37 -07:00
jenkins-bot
a158a4fb32 Merge "objectcache: move lag waiting to SqlBagOStuff::doCas() instead of overriding merge()" 2019-08-23 19:19:51 +00:00
jenkins-bot
2b5f468efa Merge "objectcache: add BagOStuff::fieldHasFlags() convenience method" 2019-08-23 19:14:59 +00:00
Aaron Schulz
7ff1a16397 objectcache: make more SqlBagOStuff methods private and rename shard variables
Also optimize silenceTransactionProfiler() when LoadBalancer is not in use

Change-Id: I83fd5c17058ba082a13e957e4b4590e2d1d5b581
2019-08-22 17:55:15 -07:00
Aaron Schulz
cdc0565705 objectcache: move lag waiting to SqlBagOStuff::doCas() instead of overriding merge()
Change-Id: I61b49bc7b00ffa8308d93e76f4bb94cf09ba9936
2019-08-22 15:07:14 -07:00
Aaron Schulz
b70e54e986 objectcache: add BagOStuff::fieldHasFlags() convenience method
Change-Id: Id2bf4ec0dd0999f988b70a895003c4b1aaae51a2
2019-08-22 15:07:14 -07:00
Aaron Schulz
942f708b61 objectcache: fix WRITE_ALLOW_SEGMENTS in BagOStuff cas() and add() methods
Add MediumSpecificBagOStuff::getValueOrSegmentList() helper method.

Also:
* Use $keysMissing variable correctly in CachedBagOStuff::getMulti()
  to avoid extra overhead.
* Optimize mergeViaCas() when the current value matches the new one.

Change-Id: I5c4bd74379bc459216ac0278150ce3aecff3b851
2019-08-20 10:55:42 -04:00
Aaron Schulz
88640fd902 objectcache: fix failing tests for non-HashBagOStuff backends
Use real time for testing absolute expirations with changeTTL().
Otherwise, backends like memcached or redis will fail since
they do not use the mock time.

Also:
* Make SqlBagOStuff actually override changeTTLMulti() by
  using the right method name
* Check TTL_INDEFINITE more explicitly for clarity
* Rename TTL conversion methods for clarity
* Use isRelativeExpiration() in MemcachedBagOStuff

Change-Id: I9365ceb31d4e7bef65906363d42b8c3020a66346
2019-08-07 14:07:22 +00:00
Aaron Schulz
83e294a892 Convert LoadBalancer::getConnection() callers to LoadBalancer::getConnectionRef()
Change-Id: Ie1ae322202105ce093779e6d0b00f335c71be31f
2019-07-21 01:02:49 -07:00
Aaron Schulz
bebe30333d objectcache: add MediumSpecificBagOStuff base class for non-proxy subclasses
This make it much clearer what needs to be overridden

Change-Id: I3073f8a0605f557c6a3a93d0d8401cddd0fb8dbe
2019-07-19 13:01:46 -07:00
Aaron Schulz
c50202a9e4 objectcache: only process cache non-LB connections in SqlBagOStuff
LB should manage any such process caching otherwise as it owns the handles

Change-Id: Ic9cd84c7c3cba790eb127e70f6506270e5bec2a1
2019-07-13 14:52:15 -07:00
Aaron Schulz
383460afa3 objectcache: normalize BagOStuff method overriding pattern for *Multi() methods
Change-Id: I1bebb60307b1a166461cb5f9a55a79194cc0e363
2019-07-11 04:32:35 -07:00
Aaron Schulz
65b4b61284 objectcache: rename "slaveOnly" parameter to "replicaOnly" in SQLBagOStuff
Also removed redundant check

Change-Id: I9f454eb15f223f5a9816dca4510c1a9add540b49
2019-07-10 19:14:51 +00:00
Aaron Schulz
20a9c8d24e bagostuff: optimize SqlBagOStuff and fix failing segmentation tests
In SqlBagOStuff:
* Add modifyMulti() helper method to reduce code duplication
* Improve atomicity of add(), cas(), and changeTTL() queries
* Avoid integer serialization and improve atomicity of incr()
* Optimize new BagOStuff::changeTTLMulti() method

In BagOStuff:
* Add changeTTLMulti() method for subclasses to optimize
* Make set() ignore WRITE_ALLOW_SEGMENTS for integers so incr() works
* Strip WRITE_ALLOW_SEGMENTS flag from the setMulti() call in set() to
  avoid triggering bogus sanity check exceptions
* Fix BagOStuffTest::testSetSegmentable failures via the above changes
* Enforce WRITE_ALLOW_SEGMENTS sanity check in setMulti() for all the
  subclasses by using a final wrapper method
* Add WRITE_ALLOW_SEGMENTS sanity check to deleteMulti()

Bug: T113916
Change-Id: I25d1790fa9b0d1837643efccfa94a12043cfbf42
2019-07-09 22:53:18 -07:00
Aaron Schulz
4245a2c200 objectcache: deleteObjectsExpiringBefore() signature and code improvements
Make the method use a callable type hint for $progressCallback.

Also make the SQLBagOStuff subclass shuffle() the array of server and table
indexes in case the limit keeps getting applied. Only garbage collect on DBs
that were going to be written to anyway. Also use DeferredUpdates if possible.

Change-Id: I723e6377c26750ff98e33f7ab103c6162ae65f43
2019-07-09 02:32:06 -07:00
Aaron Schulz
c860f99a6e objectcache: tweak SqlBagOStuff purging to happen on write only
Also adjust the frequency default and put in a 100 row limit for
randomized purges. This adds a $limit parameter to the BagOStuff
method deleteObjectsExpiringBefore().

Change-Id: I66bfcc67e8e118f8c659dbbee13d54bf2cd937f5
2019-07-06 00:04:13 -07:00
Aaron Schulz
e2e2eb49d6 objectcache: optimize lock() and unlock() methods in SqlBagOStuff
Also clean up base method in BagOStuff.

Bug: T113916
Change-Id: I3a1c6afe531a8eae34608bc7fe0aa6f9f4d439fe
2019-07-04 14:00:02 +00:00
jenkins-bot
699920cc66 Merge "objectcache: add object segmentation support to BagOStuff" 2019-06-13 18:19:17 +00:00
Aaron Schulz
b09b3980f9 objectcache: add object segmentation support to BagOStuff
Use it for ApiStashEdit so that large PaserOutput can be stored.

Add flag to allow for value segmentation on set() in BagOStuff.
Also add a flag for immediate deletion of segments on delete().

BagOStuff now has base serialize()/unserialize() methods.

Bug: T204742
Change-Id: I0667a02612526d8ddfd91d5de48b6faa78bd1ab5
2019-06-11 16:14:17 +01:00
Umherirrender
5f43b1706c Pass options as array to IDatabase::insert
The documentation only allows arrays there

Change-Id: I00c6e47a817a70bed9a443aebc675ef4c3d6b1e5
2019-06-07 19:12:35 +02:00
Aaron Schulz
962495b17a objectcache: merge getWithToken() into doGet() for simplicity
Change-Id: I581f866521e1086ca350973d9cdeff6656f48fe8
2019-03-27 13:03:25 -07:00
jenkins-bot
8818ee3a8d Merge "objectcache: fix return value type in SqlBagOStuff::incr" 2019-03-27 18:10:56 +00:00
Aaron Schulz
f80671a784 objectcache: fix return value type in SqlBagOStuff::incr
Change-Id: I14a70956f77f13346857236eda5c4b6fb80c6223
2019-03-26 17:08:37 -07:00
Aaron Schulz
861683b722 objectcache: fix race condition in SqlBagOStuff::changeTTL
Also fix --use-bagostuff parameter when given 0

Change-Id: I64971d4ccad2c856cc7688577ae564c8db068819
2019-03-26 17:06:21 -07:00
Aaron Schulz
d5afd5f909 objectcache: minor refactoring to BagOStuff
Split up expiry functions and moved add() near other primitive writes.

Change-Id: I97bb91e648b797b1ab5a6934d212689b1e67f7c6
2019-03-26 15:14:42 -07:00
Aaron Schulz
e2b03d5d30 objectcache: add BagOStuff::deleteMulti() method for consistency
Also:
* Make the BagOStuff tests actually pass for memcached and sql
* Add more unit tests for BagOStuff
* Make SqlBagOStuff::add() more atomic

Change-Id: Ic1eec0990a66b595b57c646498c3bd229442230c
2019-03-14 23:12:30 +00:00
Aaron Schulz
547f207fa6 objectcache: make the $flags argument appear more consistently in BagOStuff
Change-Id: I08879ede5e9f0ab227497bb1dab89ea61b65abce
2019-03-12 22:55:18 +00:00
jenkins-bot
db1f41172f Merge "objectcache: add $flags argument to BagOStuff::delete()" 2019-03-08 00:51:25 +00:00
Aaron Schulz
ebd1343860 objectcache: remove obsolete code from SqlBagOStuff::handleWriteError()
Given that the DB classes already check read-only mode and bad
transaction states are already handled by 3975e04cf4, this
code was not useful.

Change-Id: I4b6e8909196448a12cf74598de0272fa43bdfc62
2019-03-07 13:21:41 -08:00
Aaron Schulz
5dd409cd38 objectcache: add $flags argument to BagOStuff::delete()
This makes it consistent with set() and merge(). Also, one subclass
was already using the field in this manner.

Clean up the code related to WRITE_SYNC in SqlBagOStuff.

Change-Id: I0fb84f4475311889507d3ef98afd4476fb81174f
2019-03-07 13:15:29 -08:00
jenkins-bot
bd78869618 Merge "No yoda conditions" 2018-12-09 01:34:23 +00:00
Timo Tijhof
6be5e0a92c objectcache: Add urls for mysql bug refs in documentation
Change-Id: Ie5fe5dacae364b2b0c07f2ab0aa9b6777277c10f
2018-12-05 09:20:42 -05:00
Fomafix
3ee1560232 No yoda conditions
Replace
  if ( 42 === $foo )
by
  if ( $foo === 42 )

Change-Id: Ice320ef1ae64a59ed035c20134326b35d454f943
2018-11-21 17:54:39 +01:00
Umherirrender
719bfa8977 Use a ScopedCallback to silence transaction profiler in SqlBagOStuff
This affects most LCStoreDB which is the default cache
and spamming the log.

Bug: T154424
Bug: T97693
Change-Id: I68af177c7d0d88ab266eb3fefb4336eff86d861c
2018-10-12 20:04:31 +02:00
Bartosz Dziewoński
485f66f174 Use PHP 7 '??' operator instead of '?:' with 'isset()' where convenient
Find: /isset\(\s*([^()]+?)\s*\)\s*\?\s*\1\s*:\s*/
Replace with: '\1 ?? '

(Everywhere except includes/PHPVersionCheck.php)
(Then, manually fix some line length and indentation issues)

Then manually reviewed the replacements for cases where confusing
operator precedence would result in incorrect results
(fixing those in I478db046a1cc162c6767003ce45c9b56270f3372).

Change-Id: I33b421c8cb11cdd4ce896488c9ff5313f03a38cf
2018-05-30 18:06:13 -07:00
Aaron Schulz
f8c2486d15 rdbms: rename CONN_TRX_AUTO constant to CONN_TRX_AUTOCOMMIT
The "AUTO" means AUTOCOMMIT, not "automatic transactions"/DBO_TRX,
which is basically the opposite concept. The new name does not
suffer from that ambiguity.

Keep the old constant as an alias for backwards compatibility.

Also remove LoadBalancer comment about non-existing field

Change-Id: I63beeb061fc9be73f320308e4d6393b58628b8c8
2018-04-12 13:01:52 -07:00
Reedy
39f0f919c5 Update suppressWarning()/restoreWarning() calls
Bug: T182273
Change-Id: I9e1b628fe5949ca54258424c2e45b2fb6d491d0f
2018-02-10 08:50:12 +00:00
Aaron Schulz
8f4ae0d51c Fix LoadBalancer::getMasterPos() callers to handle a non-object being returned
Change-Id: I3356057f27a051835734329afa339ea76842a389
2018-01-12 15:18:14 -08:00
Thiemo Mättig
409da2d8b3 Remove leading backslashes from "use \…" tags
Change-Id: I494b029de089a07e3b946ee78293a12d5036f63e
2017-12-28 16:30:05 +01:00
Aaron Schulz
99c80a8fc7 rdbms: Support secondary autocommit connections in LoadBalancer
This is useful for things like SqlBagOStuff and JobQueueDB
being able to write in auto-commit mode while the main
transaction round still goes on.

SqlBagOStuff already had this sort of logic.

JobQueueDB now also takes this approach so it does not have to
defer insertion except for sqlite. This makes callers more likely
to know when insertion failed or not.

Make sure LoadBalancer sets "master"/"replica" LB info itself;
this fixes a bug found in the new tests.

Bug: T140338
Bug: T42451
Change-Id: I4199a9598d7afbf976a6efa8ed84b85b56da02bd
2017-08-18 01:28:34 +00:00
Umherirrender
a9007e8baf Add missing & to @param documentation to match functon call
Change-Id: I81e68310abcbc59964b22e0e74842d509f6b1fb9
2017-08-11 18:47:46 +02:00
Aaron Schulz
f24b3899e3 objectcache: Use a separate postgres connection in SqlBagOStuff
The flags to the driver use new connections for new LBs since
fda4d46fc4. This makes it consistent with what we do for
MySQL already.

This should fix warnings about TransactionProfiler in objectcache,
as well as warnings about "Pending writes" in WANObjectCache.

Bug: T167946
Bug: T154424
Change-Id: I0b0d9a7210b6a3270d32df778fcc4b9918d3dcd1
2017-07-06 23:08:06 +00:00
Tim Starling
fb91d409d6 Fix SqlBagOStuff exptime uniqueness assumption
The WMF parser cache tables have some 30000 rows with an identical, old
expiry time. So each time deleteObjectsExpiringBefore() is run, the
first batch on each table is correct, but the keys for the second batch
are selected with exptime > X, where X is shared exptime of the many old
rows.

So, use exptime >= X instead.

Change-Id: I6853f64f88f65d4529be2a42c9ed70cfa62cf653
2017-06-10 20:54:23 +10:00
Aaron Schulz
dd359741cc Move DB errors to Rdbms namespace
Change-Id: I463bd86123501abc68fdb78b4cda6110f7af2549
2017-04-15 10:47:41 -07:00
Aaron Schulz
d735dc562d Move Database and subclasses to Rdbms namespace
Change-Id: I52bef87512f9ddd155d1f4cc0052f6b7a0db5b42
2017-04-12 10:43:57 -07:00