Commit graph

112 commits

Author SHA1 Message Date
Umherirrender
719bfa8977 Use a ScopedCallback to silence transaction profiler in SqlBagOStuff
This affects most LCStoreDB which is the default cache
and spamming the log.

Bug: T154424
Bug: T97693
Change-Id: I68af177c7d0d88ab266eb3fefb4336eff86d861c
2018-10-12 20:04:31 +02:00
Bartosz Dziewoński
485f66f174 Use PHP 7 '??' operator instead of '?:' with 'isset()' where convenient
Find: /isset\(\s*([^()]+?)\s*\)\s*\?\s*\1\s*:\s*/
Replace with: '\1 ?? '

(Everywhere except includes/PHPVersionCheck.php)
(Then, manually fix some line length and indentation issues)

Then manually reviewed the replacements for cases where confusing
operator precedence would result in incorrect results
(fixing those in I478db046a1cc162c6767003ce45c9b56270f3372).

Change-Id: I33b421c8cb11cdd4ce896488c9ff5313f03a38cf
2018-05-30 18:06:13 -07:00
Aaron Schulz
f8c2486d15 rdbms: rename CONN_TRX_AUTO constant to CONN_TRX_AUTOCOMMIT
The "AUTO" means AUTOCOMMIT, not "automatic transactions"/DBO_TRX,
which is basically the opposite concept. The new name does not
suffer from that ambiguity.

Keep the old constant as an alias for backwards compatibility.

Also remove LoadBalancer comment about non-existing field

Change-Id: I63beeb061fc9be73f320308e4d6393b58628b8c8
2018-04-12 13:01:52 -07:00
Reedy
39f0f919c5 Update suppressWarning()/restoreWarning() calls
Bug: T182273
Change-Id: I9e1b628fe5949ca54258424c2e45b2fb6d491d0f
2018-02-10 08:50:12 +00:00
Aaron Schulz
8f4ae0d51c Fix LoadBalancer::getMasterPos() callers to handle a non-object being returned
Change-Id: I3356057f27a051835734329afa339ea76842a389
2018-01-12 15:18:14 -08:00
Thiemo Mättig
409da2d8b3 Remove leading backslashes from "use \…" tags
Change-Id: I494b029de089a07e3b946ee78293a12d5036f63e
2017-12-28 16:30:05 +01:00
Aaron Schulz
99c80a8fc7 rdbms: Support secondary autocommit connections in LoadBalancer
This is useful for things like SqlBagOStuff and JobQueueDB
being able to write in auto-commit mode while the main
transaction round still goes on.

SqlBagOStuff already had this sort of logic.

JobQueueDB now also takes this approach so it does not have to
defer insertion except for sqlite. This makes callers more likely
to know when insertion failed or not.

Make sure LoadBalancer sets "master"/"replica" LB info itself;
this fixes a bug found in the new tests.

Bug: T140338
Bug: T42451
Change-Id: I4199a9598d7afbf976a6efa8ed84b85b56da02bd
2017-08-18 01:28:34 +00:00
Umherirrender
a9007e8baf Add missing & to @param documentation to match functon call
Change-Id: I81e68310abcbc59964b22e0e74842d509f6b1fb9
2017-08-11 18:47:46 +02:00
Aaron Schulz
f24b3899e3 objectcache: Use a separate postgres connection in SqlBagOStuff
The flags to the driver use new connections for new LBs since
fda4d46fc4. This makes it consistent with what we do for
MySQL already.

This should fix warnings about TransactionProfiler in objectcache,
as well as warnings about "Pending writes" in WANObjectCache.

Bug: T167946
Bug: T154424
Change-Id: I0b0d9a7210b6a3270d32df778fcc4b9918d3dcd1
2017-07-06 23:08:06 +00:00
Tim Starling
fb91d409d6 Fix SqlBagOStuff exptime uniqueness assumption
The WMF parser cache tables have some 30000 rows with an identical, old
expiry time. So each time deleteObjectsExpiringBefore() is run, the
first batch on each table is correct, but the keys for the second batch
are selected with exptime > X, where X is shared exptime of the many old
rows.

So, use exptime >= X instead.

Change-Id: I6853f64f88f65d4529be2a42c9ed70cfa62cf653
2017-06-10 20:54:23 +10:00
Aaron Schulz
dd359741cc Move DB errors to Rdbms namespace
Change-Id: I463bd86123501abc68fdb78b4cda6110f7af2549
2017-04-15 10:47:41 -07:00
Aaron Schulz
d735dc562d Move Database and subclasses to Rdbms namespace
Change-Id: I52bef87512f9ddd155d1f4cc0052f6b7a0db5b42
2017-04-12 10:43:57 -07:00
Aaron Schulz
488a647831 Move IDatabase/IMaintainableDatabase to Rdbms namespace
Change-Id: If7e8a8ff574661fd827de8bcec11d2c39a687300
2017-03-28 15:32:38 -07:00
jenkins-bot
17eda64357 Merge "includes: Replace implicit Bugzilla bug numbers with Phab ones" 2017-02-28 00:51:57 +00:00
Aaron Schulz
64df456b39 Move remaining LoadBalancer classes to Rdbms
The old names are left as aliases.

Change-Id: I52a327f2463a2ba7437324047b5b00d28cd1d758
2017-02-23 20:38:31 -08:00
James D. Forrester
9635dda73a includes: Replace implicit Bugzilla bug numbers with Phab ones
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.

Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
2017-02-21 18:13:24 +00:00
Aaron Schulz
59053a0716 Move ChronologyProtector/TransactionProfiler to Rdbms namespace
Change-Id: I37a655bd8bd267c9bc32028b55925b2dce527d33
2017-01-26 10:30:57 -08:00
Erik Bernhardson
9053890020 Cleanup static analysis errors
While prepping the code base for using etsy/phan i found a few problems.
A couple are real problems, while others are just places etsy couldn't
figure out the types.

* AuthPlugin referenced UserLoginTemplate which doesn't exist. Tracing
  back usages this accepts classes that extend from BaseTemplate.
* DatabaseSqlite had an invalid @var annotation.
* LoadBalancer had some missing or under-defined annotations. Make them
  more specific where possible.
* SqlBagOStuff didn't have the appropriate use statement for
  WaitConditionLoop and probably doesn't work. Also updated annotations
  that point to non-existent DatabaseBase to use IDatabase.

Change-Id: Iff2270b418ad2f8f97cfdb6df646c435d3283536
2016-12-06 16:10:38 -08:00
Aaron Schulz
5bbac35ed1 Replace DatabaseBase:: with Database:: and update type hints
Change-Id: I3919b04eb2de4fa0bf8a02239fb5bbf17d347511
2016-09-27 04:20:03 +00:00
Aaron Schulz
a3dacac90f Support masking the WRITE_SYNC latency from ChronologyProtector
* Use OutputPage::output() as the method to mask latency, since it
  takes a good while to run. By the time it runs, cache replication
  should have caught up, so the reap call will likely not block.
* For redirects emitted after changes in POST, instead of masking
  with OutputPage, add a parameter to the redirect and block on
  the positions appearing. This uses the redirection RTT to mask
  the replication latency.

Change-Id: Ib23690c302e8033610fef9a0ef451dafe8a5803e
2016-09-12 23:58:49 +00:00
Aaron Schulz
57e19b610d Renamed some variables from "slave" to "replica"
Change-Id: I455278294cd7ea344d14a76ac5957ece2e07fbf3
2016-09-05 23:03:01 -07:00
Aaron Schulz
950cf6016c Rename DB_SLAVE constant to DB_REPLICA
This is more consistent with LoadBalancer, modern, and inclusive
of master/master mysql, NDB cluster, and MariaDB galera cluster.

The old constant is an alias now.

Change-Id: I0b37299ecb439cc446ffbe8c341365d1eef45849
2016-09-05 22:55:53 -07:00
Aaron Schulz
16266edff3 Change "slave" => "replica DB" in /includes
Change-Id: Icb716219c9335ff8fa447b1733d04b71d9712bf9
2016-09-05 21:01:01 +00:00
Aaron Schulz
e99bb1b7dc objectcache: allow for callbacks to mask SYNC_WRITE latency
Change-Id: I908222ad3788ebe330aa58831cda139da32becd8
2016-09-01 13:44:18 -07:00
Aaron Schulz
e92cefafdf objectcache: Make SqlBagOStuff::waitForSlaves() no-op without slaves
Change-Id: Ibaa4745c18c6f4f66edb4c5f190196b575d3b738
2016-08-30 22:47:04 +00:00
Aaron Schulz
c387e5cd84 Cleanups to SqlBagOStuff
* Keep track of the custom LoadBalancer when it makes one.
* Use the custom LoadBalancer to wait for slaves if one
  was used, rather than the main singleton.
* Only wait on the slaves in the LoadBalancer if the
  main DBs are being used.

Change-Id: I11de814306c44f27e0c33b08b5921c0fd4cdc24f
2016-08-24 19:38:14 -07:00
Aaron Schulz
21abf22680 Code cleanups to SqlBagOStuff
* Refactor local DB usage check into usesMainDB() method.
* Avoid using the db member of DBError instances.

Change-Id: I7350f5a471c551492094bfaf545ebc222eb6f7dd
2016-08-23 05:10:05 +00:00
Aaron Schulz
1f313389e2 Remove direct rollback() calls from some places
Rely on the mass-rollback logic in MWExceptionHandler instead.
This results in a better chance of atomicity.

Change-Id: I2eb5661d4acc105a1323d69c5463268c234bd745
2016-08-23 01:19:21 +00:00
Aaron Schulz
ab0aaef7ad objectcache: Optimize changeTTL() for SqlBagOStuff
Change-Id: I539dd83175a8ea922221f194e23ca47aea7539cf
2016-08-17 19:14:09 +00:00
Aaron Schulz
5f921702d2 Avoid stack overflow in LoadBalancer with CACHE_DB WAN/server cache
* Add the ability to expose key BagOStuff attributes like whether the
  emulation SQL cache is being used. Several callers use hacks to detect
  this and can be updated later.
* Fallback to a safe empty WAN cache in the CACHE_DB case. This fixes
  a regression from f4bf52e843.
* Also add this protection to the server cache, which could break
  similarly before too.
* Also fix CACHE_ANYTHING by avoid the recursion risk from
  fc1d4d7960 by just checking the disabled service map.

Bug: T123829
Bug: T141804
Change-Id: I17ee26138f69e01ec1aaddb55ab27caa4d542193
2016-08-02 13:06:49 +00:00
Kunal Mehta
4a8ae0a0b4 Use callable type-hint in a few places
Mostly places which immediately had a:
 if ( !is_callable( $callback ) ) {
  throw new Exception(...);
 }
check at the beginning of the function.

Change-Id: Ia78663b2231629010816bd1cda8814b996968d1d
2016-06-07 16:39:22 -07:00
Aaron Schulz
d5c4451380 Allow tag names for SqlBagOStuff consistent hashing
This is needed to avoid poor hit rates in multi-DC setups

Change-Id: I8830bd6bffa4619e31609676527282ef9af0c76c
2016-04-18 22:21:22 +00:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
Aaron Schulz
3a15ecebea Migrate callers to waitForReplication()
Change-Id: I7b2b13b9315891561d2d8cc04a12ecad2dc73d70
2016-01-28 03:00:08 +00:00
Aaron Schulz
5b77995417 Removed useless field in SqlBagOStuff
Change-Id: I27c0b35769667ce48ba761ff4bbc828033c6849d
2015-11-10 01:05:03 +00:00
Aaron Schulz
cd09f0d0fb Make SqlBagOStuff respect WRITE_SYNC flag
Also remove some pointless newlines in class.

Change-Id: I75880c56b1eac63c9a0616ca10890904ce823a99
2015-11-02 19:56:03 -08:00
Aaron Schulz
5c8ef13306 Add WRITE_SYNC flag to BagOStuff::set()/merge()
* This blocks on writing to all replicas
  and returns false if any failed.
* This is useful if ChronologyProtector is to work across
  domains by having the writes go everywhere so that later
  reads will see them (and be local at the same time).
* Redundant doc comments were also removed.

Change-Id: I9ed098d563c64dba605e7809bc96731da3b3e79d
2015-10-22 01:44:09 +00:00
Aaron Schulz
1fa1235d32 Clean up BagOStuff::get() interface
* Callers of get() no longer have to contend with
  the annoying $casToken parameter, which is there
  but totally unusable to non-BagOStuff code.
* The default get() now delegates to doGet(),
  which callers must implement instead. They can
  ignore the overhead of generating $casToken if
  they do not implement cas(), which applies to
  callers that use the stock merge(). If cas() is
  used for merge(), then getWithToken() must be
  implemented.
* Also add BagOStuff::READ_LATEST to mergeViaCas()
  for sanity, as that missing before.
  Likewise with mergeViaLock().

Change-Id: I4efce6a9ab4b1eadd2f161dff641004a7239c516
2015-10-07 02:54:57 +00:00
Aaron Schulz
7c07943eab Update more docs and type hints to use IDatabase
Change-Id: I8c8d85b32a8aba21e14d2a2dde4c25eb509186c1
2015-10-06 18:49:52 -07:00
Aaron Schulz
a88df43d3f Database debug log cleanup (remove wgDebugDumpSqlLength/wgDebugDBTransactions)
* Simplify the debug log call and use queries group
* Remove $wgDebugDumpSqlLength, as profiler output
  already has shortened query strings (one can use
  profiling without DBO_DEBUG)
* Removed $wgDebugDBTransactions as BEGIN/COMMIT already show
* Removed PostgresTransactionState as it was only used for
  $wgDebugDBTransactions handling
* This cuts down on lots of global variable usage

Change-Id: I185adb1694441d074dea965960429b4910727620
2015-10-05 12:47:09 -07:00
Aaron Schulz
c3dec1193c Added BagOStuff READ_LATEST flag for replicated stores
Bug: T88493
Change-Id: I7ea050a2eabba635f2aadb4e33b6f8fbfb1b01a8
2015-08-04 16:03:37 -07:00
jenkins-bot
5de749d207 Merge "Made SqlBagOStuff set() wrap setMulti()" 2015-07-02 08:14:36 +00:00
jenkins-bot
7bc770bbd8 Merge "Made SqlBagOStuff unset DB handles on connection errors" 2015-06-28 00:28:30 +00:00
Aaron Schulz
29dbdc3a64 Made SqlBagOStuff set() wrap setMulti()
* This also makes sure garbageCollect() is called on writes from
  time to time, since set() is used much more often

Change-Id: I5b6cc686bd6204fc3d6ff8b003be8b8d6a493e24
2015-06-24 13:16:50 -07:00
jenkins-bot
799ca9856b Merge "Made ReplicatedBagOStuff wrapping the SQL class the default stash" 2015-06-24 20:09:53 +00:00
Aaron Schulz
4825b4c08e Made ReplicatedBagOStuff wrapping the SQL class the default stash
* To make this work, a slaveOnly flag was added to SqlBagOStuff
  and it no longer prunes expired items on get(), it just treats
  them as missing. The use of garbageCollect() for the writeFactory
  cache and the use of replace() on updates already makes this
  a non-issue.
* This also makes the default $wgMainStash non-bogus.

Bug: T88493
Change-Id: I9d1f31305e08430de29a3cd521cdb10a82dffd10
2015-06-24 19:37:03 +00:00
Aaron Schulz
674858617a Made SqlBagOStuff unset DB handles on connection errors
* A "gone away" error can trigger ping(), which can fail,
  triggering a full open(). If this fails (as it often will),
  mConn is false and the DatabaseBase object is not usable.

Bug: T103435
Change-Id: Id2aaed87d2642e73c7811076da908180ee5ab5a4
2015-06-23 12:28:03 -07:00
umherirrender
d8821f2b0b Fixed spacing
- Removed space after casts
- Removed spaces in array index
- Added spaces around string concat
- Added space after words: switch, foreach
- else if -> elseif
- Removed parentheses around require_once, because it is not a function
- Added newline at end of file
- Removed double spaces
- Added spaces around operations
- Removed repeated newlines

Bug: T102609
Change-Id: Ib860222b24f8ad8e9062cd4dc42ec88dc63fb49e
2015-06-17 20:22:32 +00:00
Kunal Mehta
f6e5079a69 Use mediawiki/at-ease library for suppressing warnings
wfSuppressWarnings() and wfRestoreWarnings() were split out into a
separate library. All usages in core were replaced with the new
functions, and the wf* global functions are marked as deprecated.

Additionally, some uses of @ were replaced due to composer's autoloader
being loaded even earlier.

Ie1234f8c12693408de9b94bf6f84480a90bd4f8e adds the library to
mediawiki/vendor.

Bug: T100923
Change-Id: I5c35079a0a656180852be0ae6b1262d40f6534c4
2015-06-11 18:49:29 +00:00
Aaron Schulz
61df5e785f Made SqlBagOStuff avoid tripping TransactionProfiler
* Set a custom profiler with no expectations to avoid tripping
  the "0 write" expection. This avoids useless log entries.

Change-Id: Iac849a729eb36b1a8affb0dbc8b8c195fab4b03a
2015-02-27 17:45:48 +00:00