This was mentioned in the constructor comments but was not actually
implemented. It is now possible to defined what section is the catch-all
section, instead of it always being named "DEFAULT". Configuration of the
sections themselves can then be simplified by only using the real section
name for all sections.
For example, at Wikimedia, the catch-all section is "s3". The etcd config
uses "s3". The pt-heartbeat service uses "s3", thus "lagDetectionOptions"
must also use "s3". There is no reason that "templateOverridesBySection"
or "readOnlyBySection" should have to use "DEFAULT" as the key for "s3".
Change-Id: I098afd5f6108f5d7099a4cd6a611c5647ae96879
In LoadBalancer:
* Make the "chronologyCallback" return the DBPrimaryPos and make
loadSessionPrimaryPos() set the "waitForPos" more directly by
calling setSessionPrimaryPosIfHigher(). Previously, it relied
on the callback calling waitFor() to set the position as a side
effect.
* Remove redundant debug log entry in loadSessionPrimaryPos().
* Use type hints for waitFor()/waitForAll(). All callers already check
this for before invocation.
* Mark getReplicaResumePos() as @internal.
In ChronologyProtector:
* Update applySessionReplicationPosition() to return the position.
* Rename applySessionReplicationPosition() to yieldSessionPrimaryPos()
and stageSessionReplicationPosition() to stageSessionPrimaryPos() for
for consistency LoadBalancer/DBPrimaryPos.
Bug: T314434
Change-Id: I32aa784b424e7534047c9240e32fa5e0a2ac90b0
These options only support one value in the array: maxWriteDuration
Instead of turning this into a full array and pass it around, just pass
the integer to simplify the logic, avoid mistakes by typos and more.
Bug: T326274
Depends-On: Ib5c76346d0a61c3a6906365b3ced9fca2d43e4d2
Change-Id: Ib60f25ba4a7ca1d14d062d9121fe34e94ccc3b70
Factor out normalizeServerMaps() method to simplify getServerName()
by pre-setting "serverName". Add uniqueness safety check, since we
rely on this property in cache keys and in reconfigure().
Change-Id: I06672885c79611e1257adb5f3dec88194e71b705
Notable changes:
* In SqlBagOStuff::getConnectionFromServerInfo, only two loggers were
injected. The rest implicitly got a NullLogger due to being absent.
These are now effectively unsilenced.
* Database::__construct() required almost all parameters, even the
loggers. I've wanted to move some of DatabaseFactory into the ctor
here for a while. In order to make this change not a breaking
change, the new 'logger' parameter is optional with NullLogger as
default. This allowed some of the test cases, which were simply
passing NullLogger, to be fixed by passing nothing instead of
passing the new option name.
The Database class is behind a dozen layers of indirection for
real use, so this will still be injected just fine (DBF, LB, LBF,
MWLBF, etc.).
* In LegacyLogger, the handling for $wgDBerrorLog was previously
limited to DBConnection and DBQuery. This now includes errors
from other (generally, newer) parts of Rdbms as well, which were
previously missing.
This only affects sites (typically CI and dev setup) where
$wgDBerrorLog is used, as opposed to the more common
$wgDebugLogGroups by-channel configuration.
* TransactionProfiler gets its logger injected in a rather odd way,
via entrypoints (MediaWiki.php, ApiMain.php, and MaintenanceRunner)
as opposed to service wiring. This is kept as-is for now.
* In LBFactoryTest, in particular testInvalidSelectDBIndependent2,
there are cases that intentionally produce failures of which
the result is then observed. In CI we assert that dberror.log is
empty so instead of adding the missing logger fields to that
LBFactory instance, the only one set (replLogger) is removed.
The alternative is to set 'logger' now, which would naturally
cause CI failures due to unexpected entries coming through to
non-mocked error log.
Bug: T320873
Change-Id: I7ca996618e41b93f488cb5c4de82000bb36e0dd3
Rename lazyLoadReplicationPositions() and doWait(). The later now
handles the logic of checking if a position is set or if the reader
index is that of the primary DB.
Narrow the job of reallyOpenConnection() by removing "waitForPos"
loading logic. It now only deals with initializing a connection and
not the logic that goes along with tracking/pooled connection handles.
Note that getReaderIndex() still triggers primary position loading and
was the only method that automatically triggered logic to actually wait
on the primary position.
Add more comments to getReaderIndex() and optimize call order slightly.
Minor cleanups to related code comments.
Change-Id: I685b1f7946aadb5463e8870edda4340be3fc4ae2
Remove 'insertSelectIsSafe' option, unused.
Remove 'topologicalPrimaryConnRef' option, no longer used as of two
months ago with I41a57247503 (8c9398f7f9).
Remove unneeded DatabaseSqlite::getTopologyBasedServerId
implementation which can inherit null instead of overriding with string
of "0". Only caller is SqlBagOStuff::makeTimestampedModificationToken
which can be used as MainStash DB, where its important that a given
server always has the same unique name within a set of db hosts that
may replicate to each other. By inheriting null as topology server ID,
it SqlBagOStuff will use IDatabase::getServerName instead. That in turn
uses the 'host' connection parameter, which defaults to null in
DatabaseFactory, and then falls back to the string "unknown" which is
as good as "0" for this purpose.
Bug: T299691
Change-Id: Iceb65c28cdd3c4a89b3c8b34c3f95d3285718ec0
Make DBConnRef enforce the DB domain selected during its lifetime
and allow more nested and successive use of the same connection handle
via DBConnRef. This can avoid extra connections in some cases where
getConnection()/getConnectionRef() is used.
Also:
* Reduce the number of connection pools arrays from six to two
* Merge getLocalConnection()/getForeignConnection() into one method
* Expand various related code comments
Since LoadBalancer::getReadOnlyReason() no longer user the local domain
but rather DOMAIN_ANY, it should not result in "USE" errors if the local
domain does not have a database on the server.
This version of the patch removes the unused reuseConnectionInternal()
method (the method was previously added back to the patch by mistake).
Bug: T226595
Change-Id: I62502f4de4f86a54f25be1699c4d1a1c1baee60b
Make DBConnRef enforce the DB domain selected during its lifetime
and allow more nested and successive use of the same connection handle
via DBConnRef. This can avoid extra connections in some cases where
getConnection()/getConnectionRef() is used.
Also:
* Reduce the number of connection pools arrays from six to two
* Merge getLocalConnection()/getForeignConnection() into one method
* Expand various related code comments
Bug: T226595
Depends-On: If808cbab429d41e1f2289683533e4a781a4bdf5e
Change-Id: I540b08920997c57cad6445ddb09d8e663eaf4714
This method is unused and the naming is not very clear in terms of what
it returns (server name vs config map). Removing it reduces externally
exposed complexity.
Optimize DBConnRef::getServerName() when the server index is known.
Use this to eliminate the "topologicalMaster" parameter from Database.
Rename internal fields and paramters in Database to use "primary"
instead of "master", for consistency.
Add some additional clarifying comments.
Bug: T299691
Change-Id: I98515fa02a58a4c72a06f1ff283b249b1617c886
This is really hard to read. What is code, what is string? These
places are so simple, they really don't need the "{$var}" syntax.
Change-Id: I589dedb8c0193eec4eef500bbb896b5b790b727b
This was written for the scenerio where we would support writes
on a primary DC request, immediately followed by a secondary DC
request, and we'd then first use the CP cookie to wait-for Redis
replication to communicate the MySQL position data, and then wait-for
a MySQL replica to have caught up to that point.
In the revised 2020 plan for Multi-DC (T254634), CP data is kept
dc-local only, and we use a sticky-dc cookie and other traffic
routing to remain in the primary DC for certain scenarios.
CP remains as playing a rule to ensure consistency within the primary
DC between the write request and the next few seconds of GET requests
to ensure a replica is picked or waited-for to have caught up with
prior writes in the same primary DC.
One test needed to be updated as this commit exposed a happy accident
where previously one key ('writeIndex') was unset in the mocked
array, which was tolerated with the null-ish operator. Now that we
use is_array only and then access the key, this undefined offset
got exposed. Fix by removing the hardcoded logic and instead
use $cp->key and $cp-mergePositions(), which also helps gain more
confidence in the test by relying on their side-effects.
Bug: T314434
Change-Id: I712e5aeb6712f4038011c3c48799e859aa40009d
The 'watchlist' query group no longer exists in WMF production since
T263127, and it's not included in the list of supported query groups
in docs/database.md.
Change-Id: I6ea5c65921891ac6a705a6ff7e79b08fa5a9bf42
This introduces $wgLBFactoryConf['configCallback'] which can be set to a
function that returns updates to be applied to $wgLBFactoryConf. The new
method LBFactory::autoreConfigure() can be called to check the callabck
and, if the config changed, reconfigure all existing LoadBalancers.
Reconfiguring the LoadBalancers causes all open connections to be
invalidated; however, any DBConnRef instances will remain valid and will
acquire a fresh connection from the LoadBalancer automatically when
appropriate.
As a proof of concept, this patch adds support for config reloding
into WikiExporter.
Bug: T298485
Change-Id: I6c3ffde62f6e038730736abe980befd90ec43e1a
createMock() does the same, but is much easier to read.
A small difference is that some of the replacements made in this
patch didn't use disableOriginalConstructor() before. In case this
was relevant we should see the respective test fail. If not we can
save some CPU cycles and skip these constructors.
Change-Id: Ib98fb06e0fe753b7a53cb087a47e1159515a8ad5
Use a single callback for both mocks. This is needed because using
PHP 8.1 results in:
> 1) LBFactoryTest::testChronologyProtector
> Implicit conversion from float 1657063075.187724
> to int loses precision
>
> tests/phpunit/includes/db/LBFactoryTest.php:313
> includes/libs/rdbms/ChronologyProtector.php:290
Bug: T312183
Change-Id: I2764cf9b81b08832c904bf7245bd54ca0f0eddad
Add Database::queryMulti(), which will execute an array of
queries as a batch with minimal roundtrips.
SQLite fallbacks to looping through each statement and
invoking doQuery().
Add QueryStatus class to reduce complexity in Database.
Rewrite doQuery() as doSingleStatementQuery().
Change-Id: I3d51083e36ab06fcc1d94558e51b38e106f71bb9
This would simplify the code for its users a lot.
Bug: T255493
Depends-On: I6ab4440a13f4682dd3a6e9cfc6c629222fbeab8a
Change-Id: I6e7544763bde56fc1e19de0358b71ba984a978ca
The first of many changes decoupling SQL building blocks from Database
class.
Bug: T299691
Depends-On: I5d1d5b9b875bced7bda234f45d6d22ed59db4871
Change-Id: I784e78361f5ee629d31c68629d669ee0ddddf929
Callback style iteration made sense before generators existed, but
generators make for simpler code. The "call method" variants made
sense before closures existed but defeat static analysis.
So, in LBFactory:
* Add ILBFactory::getAllLBs()
* Deprecate ILBFactory::forEachLB()
* Remove LBFactory::forEachLBCallMethod(), was protected.
* Add LBFactory::getLBsForOwner(), which is protected and has the
internal interface in @return. Adding a new abstract method breaks
Wikibase tests despite LBFactory not being stable to extend.
* Migrate callers. Generators allow you to return/break from the middle
of the loop, which implies a little rearrangement for some callers.
In LoadBalancer, connections supposedly of type IDatabase were
routinely type-hinted as Database in closure parameters so that methods
could be called that were not in the interface. So it's convenient to
get rid of public iteration methods entirely in favour of private
methods returning Database[].
* Hard-deprecate ILoadBalancerForOwner::forEachOpenConnection()
and replace it with a private generator method since nothing called
it externally except for core tests.
* Hard-deprecate ILoadBalancerForOwner::forEachOpenPrimaryConnection()
and replace it with a private generator. DeferredUpdates needed it for
iterating over IDatabase::explicitTrxActive(), so add
ILoadBalancer::explicitTrxActive() as a replacement.
* Replace private method LoadBalancer::forEachOpenReplicaConnection()
with a generator.
Depends-On: If0b382231e27d6d1197fb7b6aef6ab50335df4e5
Change-Id: I64514e77b9bfe737be5b12e1d3c9c49976bb522f
Rename canRecoverFromDisconnect() in order to better describe
its function. Make it use the transaction ID and query walltime
as arguments and return an ERR_* class constant instead of a bool.
Avoid retries of slow queries that yield lost connection errors.
Track session state errors caused by the loss of named locks or
temp tables (e.g. during connection loss). Such errors will prevent
further queries except for rollback() and flushSession(), which must
be issued to resolve the error.
Add flushPrimarySessions() methods to LBFactory/LoadBalancer
and use it in places where session state loss is meant to be
safely aknowledged.
Change-Id: I60532f86e629c83b357d4832d1963eca17752944
Pass the error number as an argument, similar to isConnectionError()
Fix related mysql documentation links
Change-Id: Id32ef2fd27de65376960de3f5138ffdf7654ff71
This reverts commit eed58f2f61.
Reason for revert: Blocking the train, going to revert the whole chain
Change-Id: I64f9e5a9dde106671783f958a686ca697182077b
Use the LoadBalancer id in flushPrimarySessions(), not the LBFactory one,
and use assertOwnership() to check $owner, similar to other methods.
In DatabaseMysqlBase::doFlushSession(), change RELEASE_ALL_LOCKS() query
to use RELEASE_LOCK(), since only newer MariaDB versions (>=10.5.2) support
it. No errors were thrown in the method since they are suppressed, but the
syntax error would cause the transaction to be placed in an error state.
Add assertion to testTransactionCallbackChains() that would otherwise fail.
Randomize lock names in lock() tests to avoid contention.
Bug: T292239
Bug: T303887
Follow-Up: ee3c65d541
Follow-Up: 4cac31de4e
Change-Id: I414d737028338cfd5369eee24576df4aa26a2f6f
Rename canRecoverFromDisconnect() in order to better describe
its function. Make it use the transaction ID and query walltime
as arguments and return an ERR_* class constant instead of a bool.
Avoid retries of slow queries that yield lost connection errors.
Add methods and class constants to track session state errors
caused by the loss of named locks or temp tables. Such errors can
be resolved by a "session flush" method.
Make assertQueryIsCurrentlyAllowed() better distinguish ROLLBACK
queries from ROLLBACK TO SAVEPOINT queries. For some scenarios,
only full tranasction ROLLBACK queries should be allowed.
Add flushSession() method to Database and flushPrimarySessions()
methods to LBFactory/LoadBalancer.
Also:
* Rename wasKnownStatementRollbackError() and make it take the
error number as an argument, similar to wasConnectionError().
Add mysql error codes for query timeouts since they only cause
statement rollbacks.
* Rename wasConnectionError() and mark it as protected. This is an
internal method with no outside callers.
* Rename wasQueryTimeout(), remove some HHVM-specific code, and
simplify the arguments.
* Make executeQuery() use a for loop for the query retry logic
to reduce code duplication.
* Move the error state setting logic in executeQueryAttempt() up
in order to reduce code duplication.
* Move the beginIfImplied() call in executeQueryAttempt() up to the
retry loop in executeQuery(). This narrows the executeQueryAttempt()
concerns to sending a single query and updating tracking fields.
* Make closeConnection() and doHandleSessionLossPreconnect() in
DatabaseSqlite more consistent with the base class by releasing named locks.
* Mark trxStatus() as @internal.
Bug: T281451
Bug: T293859
Change-Id: I200f90e413b8a725828745f81925b54985c72180
