Commit graph

105 commits

Author SHA1 Message Date
Umherirrender
a9007e8baf Add missing & to @param documentation to match functon call
Change-Id: I81e68310abcbc59964b22e0e74842d509f6b1fb9
2017-08-11 18:47:46 +02:00
Aaron Schulz
f24b3899e3 objectcache: Use a separate postgres connection in SqlBagOStuff
The flags to the driver use new connections for new LBs since
fda4d46fc4. This makes it consistent with what we do for
MySQL already.

This should fix warnings about TransactionProfiler in objectcache,
as well as warnings about "Pending writes" in WANObjectCache.

Bug: T167946
Bug: T154424
Change-Id: I0b0d9a7210b6a3270d32df778fcc4b9918d3dcd1
2017-07-06 23:08:06 +00:00
Tim Starling
fb91d409d6 Fix SqlBagOStuff exptime uniqueness assumption
The WMF parser cache tables have some 30000 rows with an identical, old
expiry time. So each time deleteObjectsExpiringBefore() is run, the
first batch on each table is correct, but the keys for the second batch
are selected with exptime > X, where X is shared exptime of the many old
rows.

So, use exptime >= X instead.

Change-Id: I6853f64f88f65d4529be2a42c9ed70cfa62cf653
2017-06-10 20:54:23 +10:00
Aaron Schulz
dd359741cc Move DB errors to Rdbms namespace
Change-Id: I463bd86123501abc68fdb78b4cda6110f7af2549
2017-04-15 10:47:41 -07:00
Aaron Schulz
d735dc562d Move Database and subclasses to Rdbms namespace
Change-Id: I52bef87512f9ddd155d1f4cc0052f6b7a0db5b42
2017-04-12 10:43:57 -07:00
Aaron Schulz
488a647831 Move IDatabase/IMaintainableDatabase to Rdbms namespace
Change-Id: If7e8a8ff574661fd827de8bcec11d2c39a687300
2017-03-28 15:32:38 -07:00
jenkins-bot
17eda64357 Merge "includes: Replace implicit Bugzilla bug numbers with Phab ones" 2017-02-28 00:51:57 +00:00
Aaron Schulz
64df456b39 Move remaining LoadBalancer classes to Rdbms
The old names are left as aliases.

Change-Id: I52a327f2463a2ba7437324047b5b00d28cd1d758
2017-02-23 20:38:31 -08:00
James D. Forrester
9635dda73a includes: Replace implicit Bugzilla bug numbers with Phab ones
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.

Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
2017-02-21 18:13:24 +00:00
Aaron Schulz
59053a0716 Move ChronologyProtector/TransactionProfiler to Rdbms namespace
Change-Id: I37a655bd8bd267c9bc32028b55925b2dce527d33
2017-01-26 10:30:57 -08:00
Erik Bernhardson
9053890020 Cleanup static analysis errors
While prepping the code base for using etsy/phan i found a few problems.
A couple are real problems, while others are just places etsy couldn't
figure out the types.

* AuthPlugin referenced UserLoginTemplate which doesn't exist. Tracing
  back usages this accepts classes that extend from BaseTemplate.
* DatabaseSqlite had an invalid @var annotation.
* LoadBalancer had some missing or under-defined annotations. Make them
  more specific where possible.
* SqlBagOStuff didn't have the appropriate use statement for
  WaitConditionLoop and probably doesn't work. Also updated annotations
  that point to non-existent DatabaseBase to use IDatabase.

Change-Id: Iff2270b418ad2f8f97cfdb6df646c435d3283536
2016-12-06 16:10:38 -08:00
Aaron Schulz
5bbac35ed1 Replace DatabaseBase:: with Database:: and update type hints
Change-Id: I3919b04eb2de4fa0bf8a02239fb5bbf17d347511
2016-09-27 04:20:03 +00:00
Aaron Schulz
a3dacac90f Support masking the WRITE_SYNC latency from ChronologyProtector
* Use OutputPage::output() as the method to mask latency, since it
  takes a good while to run. By the time it runs, cache replication
  should have caught up, so the reap call will likely not block.
* For redirects emitted after changes in POST, instead of masking
  with OutputPage, add a parameter to the redirect and block on
  the positions appearing. This uses the redirection RTT to mask
  the replication latency.

Change-Id: Ib23690c302e8033610fef9a0ef451dafe8a5803e
2016-09-12 23:58:49 +00:00
Aaron Schulz
57e19b610d Renamed some variables from "slave" to "replica"
Change-Id: I455278294cd7ea344d14a76ac5957ece2e07fbf3
2016-09-05 23:03:01 -07:00
Aaron Schulz
950cf6016c Rename DB_SLAVE constant to DB_REPLICA
This is more consistent with LoadBalancer, modern, and inclusive
of master/master mysql, NDB cluster, and MariaDB galera cluster.

The old constant is an alias now.

Change-Id: I0b37299ecb439cc446ffbe8c341365d1eef45849
2016-09-05 22:55:53 -07:00
Aaron Schulz
16266edff3 Change "slave" => "replica DB" in /includes
Change-Id: Icb716219c9335ff8fa447b1733d04b71d9712bf9
2016-09-05 21:01:01 +00:00
Aaron Schulz
e99bb1b7dc objectcache: allow for callbacks to mask SYNC_WRITE latency
Change-Id: I908222ad3788ebe330aa58831cda139da32becd8
2016-09-01 13:44:18 -07:00
Aaron Schulz
e92cefafdf objectcache: Make SqlBagOStuff::waitForSlaves() no-op without slaves
Change-Id: Ibaa4745c18c6f4f66edb4c5f190196b575d3b738
2016-08-30 22:47:04 +00:00
Aaron Schulz
c387e5cd84 Cleanups to SqlBagOStuff
* Keep track of the custom LoadBalancer when it makes one.
* Use the custom LoadBalancer to wait for slaves if one
  was used, rather than the main singleton.
* Only wait on the slaves in the LoadBalancer if the
  main DBs are being used.

Change-Id: I11de814306c44f27e0c33b08b5921c0fd4cdc24f
2016-08-24 19:38:14 -07:00
Aaron Schulz
21abf22680 Code cleanups to SqlBagOStuff
* Refactor local DB usage check into usesMainDB() method.
* Avoid using the db member of DBError instances.

Change-Id: I7350f5a471c551492094bfaf545ebc222eb6f7dd
2016-08-23 05:10:05 +00:00
Aaron Schulz
1f313389e2 Remove direct rollback() calls from some places
Rely on the mass-rollback logic in MWExceptionHandler instead.
This results in a better chance of atomicity.

Change-Id: I2eb5661d4acc105a1323d69c5463268c234bd745
2016-08-23 01:19:21 +00:00
Aaron Schulz
ab0aaef7ad objectcache: Optimize changeTTL() for SqlBagOStuff
Change-Id: I539dd83175a8ea922221f194e23ca47aea7539cf
2016-08-17 19:14:09 +00:00
Aaron Schulz
5f921702d2 Avoid stack overflow in LoadBalancer with CACHE_DB WAN/server cache
* Add the ability to expose key BagOStuff attributes like whether the
  emulation SQL cache is being used. Several callers use hacks to detect
  this and can be updated later.
* Fallback to a safe empty WAN cache in the CACHE_DB case. This fixes
  a regression from f4bf52e843.
* Also add this protection to the server cache, which could break
  similarly before too.
* Also fix CACHE_ANYTHING by avoid the recursion risk from
  fc1d4d7960 by just checking the disabled service map.

Bug: T123829
Bug: T141804
Change-Id: I17ee26138f69e01ec1aaddb55ab27caa4d542193
2016-08-02 13:06:49 +00:00
Kunal Mehta
4a8ae0a0b4 Use callable type-hint in a few places
Mostly places which immediately had a:
 if ( !is_callable( $callback ) ) {
  throw new Exception(...);
 }
check at the beginning of the function.

Change-Id: Ia78663b2231629010816bd1cda8814b996968d1d
2016-06-07 16:39:22 -07:00
Aaron Schulz
d5c4451380 Allow tag names for SqlBagOStuff consistent hashing
This is needed to avoid poor hit rates in multi-DC setups

Change-Id: I8830bd6bffa4619e31609676527282ef9af0c76c
2016-04-18 22:21:22 +00:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
Aaron Schulz
3a15ecebea Migrate callers to waitForReplication()
Change-Id: I7b2b13b9315891561d2d8cc04a12ecad2dc73d70
2016-01-28 03:00:08 +00:00
Aaron Schulz
5b77995417 Removed useless field in SqlBagOStuff
Change-Id: I27c0b35769667ce48ba761ff4bbc828033c6849d
2015-11-10 01:05:03 +00:00
Aaron Schulz
cd09f0d0fb Make SqlBagOStuff respect WRITE_SYNC flag
Also remove some pointless newlines in class.

Change-Id: I75880c56b1eac63c9a0616ca10890904ce823a99
2015-11-02 19:56:03 -08:00
Aaron Schulz
5c8ef13306 Add WRITE_SYNC flag to BagOStuff::set()/merge()
* This blocks on writing to all replicas
  and returns false if any failed.
* This is useful if ChronologyProtector is to work across
  domains by having the writes go everywhere so that later
  reads will see them (and be local at the same time).
* Redundant doc comments were also removed.

Change-Id: I9ed098d563c64dba605e7809bc96731da3b3e79d
2015-10-22 01:44:09 +00:00
Aaron Schulz
1fa1235d32 Clean up BagOStuff::get() interface
* Callers of get() no longer have to contend with
  the annoying $casToken parameter, which is there
  but totally unusable to non-BagOStuff code.
* The default get() now delegates to doGet(),
  which callers must implement instead. They can
  ignore the overhead of generating $casToken if
  they do not implement cas(), which applies to
  callers that use the stock merge(). If cas() is
  used for merge(), then getWithToken() must be
  implemented.
* Also add BagOStuff::READ_LATEST to mergeViaCas()
  for sanity, as that missing before.
  Likewise with mergeViaLock().

Change-Id: I4efce6a9ab4b1eadd2f161dff641004a7239c516
2015-10-07 02:54:57 +00:00
Aaron Schulz
7c07943eab Update more docs and type hints to use IDatabase
Change-Id: I8c8d85b32a8aba21e14d2a2dde4c25eb509186c1
2015-10-06 18:49:52 -07:00
Aaron Schulz
a88df43d3f Database debug log cleanup (remove wgDebugDumpSqlLength/wgDebugDBTransactions)
* Simplify the debug log call and use queries group
* Remove $wgDebugDumpSqlLength, as profiler output
  already has shortened query strings (one can use
  profiling without DBO_DEBUG)
* Removed $wgDebugDBTransactions as BEGIN/COMMIT already show
* Removed PostgresTransactionState as it was only used for
  $wgDebugDBTransactions handling
* This cuts down on lots of global variable usage

Change-Id: I185adb1694441d074dea965960429b4910727620
2015-10-05 12:47:09 -07:00
Aaron Schulz
c3dec1193c Added BagOStuff READ_LATEST flag for replicated stores
Bug: T88493
Change-Id: I7ea050a2eabba635f2aadb4e33b6f8fbfb1b01a8
2015-08-04 16:03:37 -07:00
jenkins-bot
5de749d207 Merge "Made SqlBagOStuff set() wrap setMulti()" 2015-07-02 08:14:36 +00:00
jenkins-bot
7bc770bbd8 Merge "Made SqlBagOStuff unset DB handles on connection errors" 2015-06-28 00:28:30 +00:00
Aaron Schulz
29dbdc3a64 Made SqlBagOStuff set() wrap setMulti()
* This also makes sure garbageCollect() is called on writes from
  time to time, since set() is used much more often

Change-Id: I5b6cc686bd6204fc3d6ff8b003be8b8d6a493e24
2015-06-24 13:16:50 -07:00
jenkins-bot
799ca9856b Merge "Made ReplicatedBagOStuff wrapping the SQL class the default stash" 2015-06-24 20:09:53 +00:00
Aaron Schulz
4825b4c08e Made ReplicatedBagOStuff wrapping the SQL class the default stash
* To make this work, a slaveOnly flag was added to SqlBagOStuff
  and it no longer prunes expired items on get(), it just treats
  them as missing. The use of garbageCollect() for the writeFactory
  cache and the use of replace() on updates already makes this
  a non-issue.
* This also makes the default $wgMainStash non-bogus.

Bug: T88493
Change-Id: I9d1f31305e08430de29a3cd521cdb10a82dffd10
2015-06-24 19:37:03 +00:00
Aaron Schulz
674858617a Made SqlBagOStuff unset DB handles on connection errors
* A "gone away" error can trigger ping(), which can fail,
  triggering a full open(). If this fails (as it often will),
  mConn is false and the DatabaseBase object is not usable.

Bug: T103435
Change-Id: Id2aaed87d2642e73c7811076da908180ee5ab5a4
2015-06-23 12:28:03 -07:00
umherirrender
d8821f2b0b Fixed spacing
- Removed space after casts
- Removed spaces in array index
- Added spaces around string concat
- Added space after words: switch, foreach
- else if -> elseif
- Removed parentheses around require_once, because it is not a function
- Added newline at end of file
- Removed double spaces
- Added spaces around operations
- Removed repeated newlines

Bug: T102609
Change-Id: Ib860222b24f8ad8e9062cd4dc42ec88dc63fb49e
2015-06-17 20:22:32 +00:00
Kunal Mehta
f6e5079a69 Use mediawiki/at-ease library for suppressing warnings
wfSuppressWarnings() and wfRestoreWarnings() were split out into a
separate library. All usages in core were replaced with the new
functions, and the wf* global functions are marked as deprecated.

Additionally, some uses of @ were replaced due to composer's autoloader
being loaded even earlier.

Ie1234f8c12693408de9b94bf6f84480a90bd4f8e adds the library to
mediawiki/vendor.

Bug: T100923
Change-Id: I5c35079a0a656180852be0ae6b1262d40f6534c4
2015-06-11 18:49:29 +00:00
Aaron Schulz
61df5e785f Made SqlBagOStuff avoid tripping TransactionProfiler
* Set a custom profiler with no expectations to avoid tripping
  the "0 write" expection. This avoids useless log entries.

Change-Id: Iac849a729eb36b1a8affb0dbc8b8c195fab4b03a
2015-02-27 17:45:48 +00:00
Aaron Schulz
0a873a3c28 Avoid calling rollback() in handleWriteError() if there is no trx
Change-Id: Ic3be89151160ad57acda5d95e3995a77f3835717
2015-02-24 11:34:23 -08:00
Aaron Schulz
250f2b9e5a Made BagOStuff::cas properly optional
* merge() using the locking version by default. The callers that implement cas()
  override merge() to use the cas-based version. Those that do not no longer
  need to define the unused dummy method.
* Removed some redundant doc blocks.

Change-Id: I49f7dd5432efa8d76f4758c273f0859376ddafb7
2015-02-23 05:15:24 +00:00
Aaron Schulz
61697bab1e Made BagOStuff::cas protected, which is just one merge() implementation
Change-Id: I3ef82226231f7e03f7493ae042cad22339f4c869
2015-02-10 10:38:41 -08:00
Aaron Schulz
ac8d6b3538 Removed old SqlBagOStuff alias
Change-Id: I6fdd8791713ce9ed67473065332266c217284974
2015-02-06 02:03:04 -08:00
jenkins-bot
2da34d0a9c Merge "Removed unused and poorly supported time argument to BagOStuff::delete" 2015-02-05 02:28:01 +00:00
Kunal Mehta
ed90263776 objectcache: Use Psr\Log\LoggerInterface instead of wfDebug
ObjectCache::newFromParams() will handle a 'loggroup' parameter
specially, getting an instance of \Psr\Log\LoggerInterface to
pass to the BagsOStuff.

BagOStuff implementations can use $this->logger which will be
an implementation of \Psr\Log\LoggerInterface. As this is set in
BagOStuff::__construct(), all subclasses must now call the
parent constructor.

The goal of this is to make the logging in BagOStuff non-MediaWiki
specific, in the hopes of separating it out into a separate library
in the future.

Change-Id: I8a8e278e6f028814499d8457d6d5341d03eabc7a
2015-01-30 22:03:23 +00:00
Aaron Schulz
13d1a86eef Removed unused and poorly supported time argument to BagOStuff::delete
Change-Id: I8f90f480ec8bc70605c85c39f73f29df21d1927d
2015-01-26 17:14:33 -08:00