Commit graph

95 commits

Author SHA1 Message Date
jenkins-bot
fcc3cd5341 Merge "Call ssl_set() in DatabaseMysqli if DBO_SSL is set" 2016-08-24 19:46:02 +00:00
Aaron Schulz
8359993708 Various database class cleanups
* Refactor out some code duplication in query() into a
  separate private method.
* Remove the total master/slave query profiling, which is not
  necessary and redundant.
* Provide a default implementation for reconnect().
* Make reconnect() catch errors so it can match the docs that say
  it returns true/false to indicate failure. Likewise for ping().
* Optimize ping() to no-op if there was obvious recent activity.
* Move the ping() round in JobRunner to approveMasterChanges.
  This way, all commit rounds benefit from this logic.
* Add more doc comments for DatabaseBase fields.

Change-Id: Ic90ce2be4187244a0e8d44854c39d4b78be8e642
2016-08-22 20:15:41 -07:00
Aaron Schulz
b815622c94 Call ssl_set() in DatabaseMysqli if DBO_SSL is set
Bug: T136218
Change-Id: I849b14c58d3f3c12853b95e218517288d822b107
2016-08-22 10:37:31 -07:00
Aaron Schulz
4bcfe7d0ad Database transaction flushing cleanups
* Do not commit() inside masterPosWait(). This could happen
  inside JobRunner::commitMasterChanges, resulting in
  one DB committing while the others may or may not later commit.
* Migrate some commit() callers to commitMasterChanges().
* Removed unsafe upload class commit() which could break
  outer transactions.
* Also cleaned up the "flush" flag to make it harder to misuse.

Change-Id: I75193baaed979100c5338abe0c0899abba3444cb
2016-08-16 20:34:54 +00:00
Aaron Schulz
3675f1d447 Make Database disconnect and error suppression more robust
* Disallow $ignoreErrors in query() on deadlocks, since that would otherwise
  silently rollback all changes from any other callers.
* Move recoverability checks for disconnects to canRecoverFromDisconnect().
* The first write of a DBO_TRX transaction is now considered recoverable.
* Run onTransactionResolution() callbacks on disconnect/deadlock rollback.
  Some DeferrableUpdate need this to know to abort.
* Disallow $ignoreErrors on disconnects considered unrecoverable. This
  makes it so that query() callers cannot cause writes from other callers
  to be silently lost, which is hard to reason about.
* Moved ping() logic to simple reconnect() method and ping() simply do
  a dummy SELECT, which triggeres reconnection if safe. Previously,
  ping() might cause subtle partial transaction loss.
* Remove ping() from strencode(), which would cause partial transaction
  loss where it actually reached.
* Remove mysqlPing() per https://bugs.php.net/bug.php?id=52561.

Bug: T142079
Change-Id: Ifb7f772ae849d67c0d92240a115c3f392e252937
2016-08-11 07:26:33 +00:00
Aaron Schulz
3155b5850f Clean up some IDatabase comments
Change-Id: Ia4b344a412e78185b6b2fe584540a358c35651b8
2016-08-09 16:45:37 -07:00
Aaron Schulz
e543a0d47f Add missing addQuotes() call to masterPosWait()
Change-Id: Ic7f35920fe10d99b63fdff23dc6e3f7cd4c7ca8b
2016-08-08 12:36:18 -07:00
Aaron Schulz
f4bf52e843 Automatically detect READ_ONLY mode for MySQL/MariaDB
This avoids having users think they can make edits when an
exception will just be thrown when they try to save. Likewise
for other write actions.

Bug: T24923
Change-Id: I49c4057b672875ec6f34681a5668a509cec05677
2016-07-30 02:32:06 +00:00
Aaron Schulz
d1337eadda Added GTID support to slave lag methods
The IDs will be included in MySQLMasterPos objects and,
if specified by config, in slave lag wait methods.

Bug: T135027
Change-Id: I1dfc0210b715b449ec07760c712d0267763f2697
2016-07-19 02:41:31 +00:00
Aaron Schulz
dad87efa5a Support non-replicating DB clusters for static datasets
* Make the slave wait methods work properly in this case if
  the config flag is set.
* Also fixed a small IDEA error.

Bug: T135690
Change-Id: I9a12f1fc54c0b426f64c19ac2bf720e683fcf2fe
2016-05-18 16:03:27 -07:00
Aaron Schulz
6e38db159c Support custom query in pt-heartbeat lag detection
Bug: T111266
Change-Id: Ib0f9ca8072d9becebacfa8c9afe9d61567f73912
2016-04-01 23:41:14 +00:00
Aaron Schulz
b64a178a21 Fixes to masterPosWait() for master switchovers
* Clean up return value types and docs.
* Handle master switch-over better w.r.t the job queue due
  to binlog name changes (the host portion). Previously the method
  would fail and trigger read-only mode when waiting on former
  master positions. Assume the the switch-over was done properly
  and thus return immediately.

Bug: T126436
Change-Id: Ib8c05a5c72d03a5c98e41b23c5653fc194b6d130
2016-03-08 03:37:44 -08:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
Ricordisamoa
02818c441c Stop doing $that = $this in includes/db
Closures support $this as of PHP 5.4

Also made DatabaseBase::indexName() protected

Change-Id: Iff31e7d9186832a855a953b923ac182f1c66ffa0
2016-02-10 18:08:17 +01:00
Aaron Schulz
79d12a284b Do not auto-reconnect to DBs if named locks where lost
Otherwise, transactions might be committed without cooperative
locks that were supposed to be present. This is similar to the
logic of not auto-reconnecting if a transaction was started to
avoid committing incomplete changes.

Change-Id: Ia7bc6b188bb5ee53a5bf7c5a30718bc7c4dd0ba9
2016-01-30 09:17:17 -08:00
Aaron Schulz
fef85e5d92 Add better error logging for DB getLag() calls
Bug: T32257
Change-Id: I4ea5db670fe96d20b1d593cc2d759f9c3f570790
2016-01-14 01:34:56 +00:00
jenkins-bot
7028f7e31c Merge "Make getLagFromPtHeartbeat() always use the LB cluster master entry" 2015-12-23 23:27:30 +00:00
Aaron Schulz
7643d7d885 Avoid "Incorrect user-level lock name " errors in Database lock() method
Use a sha1 hash if the lock name exceeds the MySQL limit. The class
for Postgres already avoids this problem with number hashes.

Change-Id: If587c672b4ff210ee3b512bbe2910ae247079d3f
2015-12-16 16:00:46 +00:00
Aaron Schulz
9c0b55d926 Make getLagFromPtHeartbeat() always use the LB cluster master entry
Before, it just used the immediate master entry, which could be another
slave. In that case, it may not even exists at all.

Bug: T119648
Change-Id: Iea970b81ad2c9855aafcccf0bb0662fc0b3a8d4d
2015-12-09 22:00:19 -08:00
jenkins-bot
bd3e6861c6 Merge "Improve docs for lag related DB functions" 2015-11-19 07:33:19 +00:00
Aaron Schulz
36f025f8c9 Improve docs for lag related DB functions
* They can return false if replication is stopped.
* Also remove redundant related doc blocks.

Change-Id: I40a1cb2da2f23815b9da8f79195df45c5e510585
2015-11-18 18:23:08 -08:00
umherirrender
5311afb6a4 phpcs: Assignment expression not allowed
Fix some "Assignment expression not allowed"

Found by tests:
https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs/2736/consoleFull

Change-Id: Ibfc62b0aaa5c7fa63081edea3ef2b4d0dc984f85
2015-11-18 19:39:19 +01:00
Timo Tijhof
bf331d7c79 Use makeGlobalKey() directly instead of wfGlobalCacheKey()
Saves a small trip through ObjectCache::getLocalClusterInstance(),
call_user_func_array, and func_get_args.

Change-Id: If5d2170accaecb6cc8c0520ab9de9241b42a4727
2015-11-18 00:21:02 +00:00
Aaron Schulz
da4a378e74 Improve LoadBalancer::doWait() performance via APC
* This avoids bothering with MASTER_POS_WAIT() if an equal/higher
  position was already successfully waited on by another process.
* Add DBMasterPos toString() and hasReached() methods, which
  MySQLMasterPos already implemented and used.
* Moved more wfDebug() statements to the 'replication' log.

Change-Id: I423b5fe2da8d97889a6d204a635e351342de7649
2015-11-10 05:07:39 +00:00
jenkins-bot
4f7d15021d Merge "Make WANObjectCache sets account for slave lag" 2015-10-06 00:11:22 +00:00
jenkins-bot
c5ac11e2eb Merge "Start rename of DatabaseBase => Database" 2015-10-05 23:45:36 +00:00
Aaron Schulz
db0b9ef264 Make WANObjectCache sets account for slave lag
* This gets lag information that is useful when
  the calling code is about to run queries that
  will have their results cached.
* This is now used in place of trxTimestamp() for
  WANObjectCache set() and getWithSetCallback().
* The WAN cache will use a low TTL if the lag is
  too high to avoid caching stale data for weeks.
* Bumped MAX_COMMIT_DELAY as nothing enforces it.

Bug: T113204
Change-Id: I2a95b4088cff42d6e980351555f81a4b13519e24
2015-10-05 16:45:13 -07:00
Aaron Schulz
1b67a929ab Start rename of DatabaseBase => Database
This gives static method callers the option
to use methods like Database::factory() instead
of having to use the uglier DatabaseBase::factory().

Change-Id: I61800626b71ad2803a897df060059dbaf8778679
2015-10-04 12:36:56 -07:00
Aaron Schulz
b11abb540f Fix DatabaseMysqlBase.php IDEA errors
* Declare base mysqlRealEscapeString()
* Fix MySQLField::tableName() return value

Change-Id: I1277166e741e69cb1c045b3baa6ed8de664e9255
2015-10-03 18:28:17 -07:00
Aaron Schulz
ff34e3d464 Added pt-heartbeat support to DatabaseMysqlBase
Bug: T111266
Bug: T19179
Change-Id: I04c93d5c7389fbb8fba633d955591b30a9504f31
2015-10-02 18:34:36 +00:00
Aaron Schulz
c3740286ac Removed old fake slave hacks used for debug log testing
* Any modern automatic tests would use mocking methods

Change-Id: I5fbd8ebe7090f75bdcccc7a2848ae88f34a64512
2015-09-29 00:44:48 -07:00
Amir E. Aharoni
6b7c6e9be9 Make lines short to pass phpcs in files under includes/db/
Bug: T102614
Change-Id: I95ec0f9bdfc23e0d10d24534801fe5d6609ed0cb
2015-09-27 15:59:23 +00:00
jeroendedauw
795bd0e7db Remove unused argument
Change-Id: I7e7cf2b1f99834a140d0213c39cc67cfbc481ec9
2015-09-24 19:10:45 +02:00
umherirrender
271da88127 Some bugzilla.wikimedia.org -> phabricator.wikimedia.org changes
Changed some old bugzilla links to new phabricator links in comments,
test data and error message. This reduces the need for redirects from
old bugzilla to new phabricator from our source code.

Change-Id: Id98278e26ce31656295a23f3cadb536859c4caa5
2015-09-24 17:17:25 +02:00
Aaron Schulz
51be75ba00 Avoid update.php suggestion on DB connection errors
Bug: T98788
Change-Id: I21974d6b775f13c33055f76c30142e937eb0e965
2015-08-07 00:31:46 -07:00
umherirrender
974608d9fc Pass function name in DatabaseMysqlBase::setBigSelects
This allow to see the correct function name in the debug logs

Change-Id: I913acbc39e8e7b21f647b544ef161574c5f720f8
2015-07-10 23:03:52 +02:00
jenkins-bot
0bcd8347e5 Merge "database: Small DB class cleanups" 2015-06-28 00:26:07 +00:00
Aaron Schulz
359b1b7cbe database: Small DB class cleanups
* Moved some duplicated logic to assertOpen() method
* Override doc type for mConn for the mysqli subclass
* Fixed a few code comments

Change-Id: I78d595554ed51f64ca7cf7bd7ce369a492a59145
2015-06-28 00:15:54 +00:00
Aaron Schulz
7bea0f628d database: Make LoadBalancer not yield DB objects that hopelessly lost the connection
* This is useful if the first slave picked went down and others
  can be used, especially in longer running scripts.
* A possible improvement to this would be to eventually allow
  retries by removing the bad handles from the load balancer,
  since isOpen() will never change from false. This would only
  be useful for very very long running CLI scripts and is
  probably an edge case for now.

Change-Id: Iecfc4004b4b2289907a4645b431de19198790d6c
2015-06-26 07:32:36 +00:00
Aaron Schulz
c63fde6797 database: Throw exceptions when dead mysql DB handles are used instead of fatals
Bug: T103435
Change-Id: I75c4f3a950b3b333a289d0a6a41eb4f00c292121
2015-06-26 07:27:07 +00:00
Kunal Mehta
f6e5079a69 Use mediawiki/at-ease library for suppressing warnings
wfSuppressWarnings() and wfRestoreWarnings() were split out into a
separate library. All usages in core were replaced with the new
functions, and the wf* global functions are marked as deprecated.

Additionally, some uses of @ were replaced due to composer's autoloader
being loaded even earlier.

Ie1234f8c12693408de9b94bf6f84480a90bd4f8e adds the library to
mediawiki/vendor.

Bug: T100923
Change-Id: I5c35079a0a656180852be0ae6b1262d40f6534c4
2015-06-11 18:49:29 +00:00
jenkins-bot
661f171575 Merge "Added $wgJobSerialCommitThreshold setting" 2015-04-27 08:29:46 +00:00
Aaron Schulz
7c821caef5 Added $wgJobSerialCommitThreshold setting
* This is used to avoid lag by certain jobs

Bug: T95501
Change-Id: Id707c9a840fa23d56407e03aaae4e25149a1f906
2015-04-24 11:38:16 -07:00
Aaron Schulz
fe72859716 Allow setting MySQL session variables via config
Change-Id: I39c7e0f5ffe53a3fb15da489f4dddc36af99a4b3
2015-04-24 11:07:02 -07:00
umherirrender
b21bf82bb1 Fixed detection of unsigned mysql column in updater
MediaWiki running with an old mysql or php without mysqli cannot
use the newly added flags property of Database::fieldInfo.
For backward compatibility add the missing properties using the flags
property to the result of Database::fieldInfo.

Added missing unsigned, numeric, blob and zerofill properties to the
MysqlField using flags in mysqli.
The missing fields in mysqli are taken from
http://php.net/manual/en/function.mysql-fetch-field.php

Follow-Up: I641dc1fad0a5bb14815f55e0357243a562fd672b
Change-Id: I1da904df72f8af7725fc7ff94121ef85b18a36f8
2015-03-05 18:25:50 +00:00
umherirrender
a086557ab8 [schema] Make column user_newtalk.user_id an unsigned int
The primary key of the user table is an unsigned int, the column
user_newtalk.user_id refer the id and should have the same typ to store
the same values as the user table does.

No pre-deployment activity necessary; no need to sync deployment
and merge; can be done at any time (now or later)

Bug: T77005
Change-Id: I641dc1fad0a5bb14815f55e0357243a562fd672b
2015-02-17 17:27:54 +00:00
Kevin Israel
5f3957498b Database: Cast to int in estimateRowCount(), selectRowCount()
Doc comments state that these methods return ints. In order to ensure
that, values must be cast to int before they are returned.

With respect to selectRowCount(), follows-up 65f81d2843.

Change-Id: I108221ce4ad1b5b103b015fe875de54e04781741
2015-02-06 05:03:45 -05:00
Aaron Schulz
c73cd2ef7c Always set group_concat_max_len to avoid awful truncation gotcha
Change-Id: I1000412a0c58a9d76ea25f6ecc37a3a464aa430a
2015-01-13 17:24:16 -08:00
Ricordisamoa
2ae155da52 Fix phpcs errors in includes/
Mostly Squiz.WhiteSpace.SuperfluousWhitespace.EmptyLines

Change-Id: I678b2f0902f11cd1dfa1611b9da24e7237df9122
2015-01-08 20:15:07 +01:00
Aaron Schulz
4ff8136807 Removed remaining profile calls
Change-Id: I31c81c78715048004fc8fca0f27d09c1fa71c118
2015-01-08 02:49:33 -08:00