This is marked @internal and thus can be dropped without notice.
To avoid CP opening a connection, simply do the logic of checking in LBF
where it has access to better tooling.
Bug: T325389
Change-Id: I4b30c2fb2158a5ef0588b585366dc9411a08dc12
Now that we merged DatabaseMysqli with DatabaseMysqlBase, there is no
base anymore.
I could have gone with DatabaseMysql, and the naming is not that
consistent in rdbms:
amir@amir-ThinkPad-P1-Gen-3:~/core/includes/libs/rdbms$ find . | grep -i mysql
./platform/MySQLPlatform.php
./field/MySQLField.php
./dbal/MWMySQLPlatform.php
./database/DatabaseMysqlBase.php
./database/DatabaseMysqli.php
./database/position/MySQLPrimaryPos.php
./database/resultwrapper/MysqliResultWrapper.php
./database/replication/MysqlReplicationReporter.php
The majority is MySQL and since it's the correct form, I went with that
instead.
Change-Id: I3ee792f357dda974c855ba24b9b35e72fc73db06
This patch makes two major changes:
- In the PoolCounter chain, we simply inject CP and call it directly
and as result, there is no need for ILBF::getChronologyProtectorTouched
- Instead of injecting CP callback to LB, just pass the object down the
chain which leads to simpler and more stable code.
Bug: T275713
Change-Id: If78f4498d98e256015e54cc46561cb11b2947058
LBF carries a several attributes that it only need to init CP object,
including $secret that might be confusing to reader what the secret is
about.
Also, LBF should not be really coupled to CP, at least not this much.
Most stuff happen via calling CP functions passed by $lb object.
Some callers of LBF methods also don't need LBF really, they need the CP
object (e.g. ::disableChronologyProtection()) which in turn we can
make ChronologyProtector a dedicated service (it only needs BagOStuff and secret
from the LBF configuration, the rest are CLI mode or logger) and call
those separately and slowly just inject it to LBF instead.
This patch has already reduced the size of LBF by 10%.
Bug: T275713
Change-Id: If5e34a372030238093b66c292a02d11e5933ff88
Mostly things related to the test database. Also adjust visibility and
remove getTestPrefixFor(), which is only used in core according to
codesearch.
Making a method static is a backwards compatible change, as invoking the
method non-statically is valid.
Bug: T342259
Change-Id: I6111fb5ff5f3c87d5d3f9188b3f50351391a29c3
This clearly has nothing to do with LBF but is more related to CP. It's
not used outside of core anywhere, all usages fixed.
Bug: T326274
Change-Id: I6d07337fc2a9144c960073100d6078001283ace3
These methods are either:
- Completely unused, removed
- Used only by the class itself, made private
- Used by LBF (ILB methods) which they were moved to
ILoadBalancerForOwner
In the meantime, completely deprecating and removing per-domain and
per-cluster waitForReplication. Simply wait for all LBs to catch up:
* In reality, the replication lag is so small that it doesn't matter.
It's probably caught up already by that point anyway.
* If you don't make a db call, you don't init a new LBF to be waited on
so it's already quite small.
Bug: T326274
Change-Id: I2c0a89d70152de56d35290f2666b6699822ed330
It's only used in CP, CP doesn't really need replica positions. It only
needs the master position, if there is no connection to master, it means
either nothing happened on that section or connection got dropped
(extremely rare) and regardless, it's not worth the complexity and
specially a new method in LB that now almost every code piece can
access.
Bug: T342564
Bug: T326274
Change-Id: I0579e688a44d7a13d6b42622f7e88608f88e9841
See https://phabricator.wikimedia.org/T314434#9039400
This is making one memcached call for every request while it doesn't
need basically any of it.
Bug: T314434
Co-Authored-by: Máté Szabó <mszabo@fandom.com>
Change-Id: If7082f62f2d6cdedb7a505ac68ba79f08634034d
Track the insert ID value in Database, similar to the affected rows.
This makes it possible for subclasses to stash or override the value,
which is useful when emulating a write operation using multiple queries.
This includes the case of internal use of atomic sections, where the
COMMIT/RELEASE can reset the last_insert_id tracked in the PECL driver
itself.
Use separate methods and fields for "last query statement" information
and "last query method" information.
Make insertId() for SQLite and Postgres better match MySQL:
* Return 0 if the last query statement did not change any rows.
This helps protect against callers that fail to check affectedRows().
* Make it return the existing ROWID/SERIAL column when upsert() updates
an existing row. This adds a new getInsertIdColumnForUpsert() helper
function.
Directly use query() in doReplace() and doInsertSelectGeneric() to make
the affected row/ID logic easier to follow.
Improve insertId() and affectedRows() documentation.
Add more integration tests of row insertion methods.
Bug: T314100
Change-Id: I7d43a2e52260e66acb713554bb883f5f4a14d010
Use str_starts_with, str_ends_with or string offset where appropriate.
This fixes a bug in MimeAnalyzer where the "UTF-16LE" header could not
be identified because of wrong constant. This is the exact type of bug
that the new functions can avoid.
Change-Id: I9f30881e7e895f011db29cf5dcbe43bc4f341062
Subclasses no longer have to implement fetchAffectedRowCount()
and affectedRows() no longer depends on the driver connection
handle.
Set "port" to $wgDBport in LBFactoryTest/LoadBalancerTest to
avoid postgres failures.
Bug: T314100
Change-Id: Ib31a9d2db18d7ba7dcf61fb110d0fef53f455464
There are many, many more places that can benefit from these new
interfaces. I like to go step by step. This makes patches like this
much easier to review.
Change-Id: I461064c1a7f70b3ba3b2a6d47a67cb6e8a54528d
This is part of LB doing wearing way too many hats. LB's job is to take
an index out of a hat, not to init the db object. We have a factory for
that already.
Also completely get rid of injecting DBFactory as a parameter in config
of LB and LBF, It doesn't make any sense to pass this around as a
configuration option, it's a php class, so can't even be properly set in
many systems (code should not be a configuration). On top of that it's
making multiple ways to override configurations with non-obvious
priority that can easily lead to outages.
Bug: T326274
Change-Id: I1e0c38cd3b378669d0940b9f243b61cb64c193b7
Follows-up I4c7d826c7ec654b, I1287f3979aba1bf1.
We lose useful coverage and spend valuable time keeping these accurate
through refactors (or worse, forget to do so). The theoretically "bad"
accidental coverage is almost never actually bad.
Having said that, I'm not removing them wholesale (yet). I've audited
each of these specific files to confirm it is a general test of the
specified subject class, and also kept it limited to those specified
classes. That's imho more than 100% of the benefit for less than 1%
of the cost (more because `@covers` is more valuable than the fragile
and corrosive individual private method tracking in tests that
inevitably get out of date with no local incentive to keep them up to
date).
Cases like structure tests keep `@coversNothing` etc and we still don't
count coverage of other classes. There may be a handful of large
legacy classes where some methods are effectively class-like in
complexity and that's why it's good for PHPUnit to offer the precision
instrument but that doesn't meant we have to use that by-default for
everything.
I think best practice is to write good narrow unit tests, that reflect
how the code should be used in practice. Not to write bad tests and
hide part of its coverage within the same class or even namespace.
Fortunately, that's generally what we do already it's just that we
also kept these annotations still in many cases.
This wastes time to keep methods in sync, time to realize (and fix)
when other people inevitably didn't keep them in sync, time to find
uncovered code only to realize it is already covered, time for a less
experienced engineer to feel obligate to and do write a low quality
test to cover the "missing" branch in an unrealistic way, time wasted
in on-boarding by using such "bad" tests as example for how to use
the code and then having to unlearn it months/years later, loss of
telemetry in knowing what code actually isn't propertly tested due to
being masked by a bad test, and lost oppertunities to find actually
ununused/unreachable code and to think about how to instead structure
the code such that maybe that code can be removed.
------
Especially cases like LBFactoryTest.php were getting out of hand,
and in GlobalIdGeneratorTest.php we even resorted to reminding people
with inline comments to keep tags in sync.
Change-Id: I69b5385868cc6b451e5f2ebec9539694968bf58c
Use server names to handle the case where server indexes get
shifted around due to the depooling or provisioning of a server.
Previously, loads could be assigned to a wrong server, assigned
to a server that was depooled, or assigned to a server that is
to new to appear in the current "servers" array. This would mean
that getReaderIndex() or getLagTimes() could trigger exceptions.
Only change loads for depooled servers. Update the loads of other
servers to the current loads only makes sense in the context of
using all the current servers. This does not apply to reconfigure()
since it does not see newly pooled servers (for simplicity).
Handle servers depooled only from custom query groups.
Bug: T322156
Change-Id: I9f710aa32f5d5b74796bb80a8426a5f653b8e4d3
This was mentioned in the constructor comments but was not actually
implemented. It is now possible to defined what section is the catch-all
section, instead of it always being named "DEFAULT". Configuration of the
sections themselves can then be simplified by only using the real section
name for all sections.
For example, at Wikimedia, the catch-all section is "s3". The etcd config
uses "s3". The pt-heartbeat service uses "s3", thus "lagDetectionOptions"
must also use "s3". There is no reason that "templateOverridesBySection"
or "readOnlyBySection" should have to use "DEFAULT" as the key for "s3".
Change-Id: I098afd5f6108f5d7099a4cd6a611c5647ae96879
In LoadBalancer:
* Make the "chronologyCallback" return the DBPrimaryPos and make
loadSessionPrimaryPos() set the "waitForPos" more directly by
calling setSessionPrimaryPosIfHigher(). Previously, it relied
on the callback calling waitFor() to set the position as a side
effect.
* Remove redundant debug log entry in loadSessionPrimaryPos().
* Use type hints for waitFor()/waitForAll(). All callers already check
this for before invocation.
* Mark getReplicaResumePos() as @internal.
In ChronologyProtector:
* Update applySessionReplicationPosition() to return the position.
* Rename applySessionReplicationPosition() to yieldSessionPrimaryPos()
and stageSessionReplicationPosition() to stageSessionPrimaryPos() for
for consistency LoadBalancer/DBPrimaryPos.
Bug: T314434
Change-Id: I32aa784b424e7534047c9240e32fa5e0a2ac90b0
These options only support one value in the array: maxWriteDuration
Instead of turning this into a full array and pass it around, just pass
the integer to simplify the logic, avoid mistakes by typos and more.
Bug: T326274
Depends-On: Ib5c76346d0a61c3a6906365b3ced9fca2d43e4d2
Change-Id: Ib60f25ba4a7ca1d14d062d9121fe34e94ccc3b70
Factor out normalizeServerMaps() method to simplify getServerName()
by pre-setting "serverName". Add uniqueness safety check, since we
rely on this property in cache keys and in reconfigure().
Change-Id: I06672885c79611e1257adb5f3dec88194e71b705
Notable changes:
* In SqlBagOStuff::getConnectionFromServerInfo, only two loggers were
injected. The rest implicitly got a NullLogger due to being absent.
These are now effectively unsilenced.
* Database::__construct() required almost all parameters, even the
loggers. I've wanted to move some of DatabaseFactory into the ctor
here for a while. In order to make this change not a breaking
change, the new 'logger' parameter is optional with NullLogger as
default. This allowed some of the test cases, which were simply
passing NullLogger, to be fixed by passing nothing instead of
passing the new option name.
The Database class is behind a dozen layers of indirection for
real use, so this will still be injected just fine (DBF, LB, LBF,
MWLBF, etc.).
* In LegacyLogger, the handling for $wgDBerrorLog was previously
limited to DBConnection and DBQuery. This now includes errors
from other (generally, newer) parts of Rdbms as well, which were
previously missing.
This only affects sites (typically CI and dev setup) where
$wgDBerrorLog is used, as opposed to the more common
$wgDebugLogGroups by-channel configuration.
* TransactionProfiler gets its logger injected in a rather odd way,
via entrypoints (MediaWiki.php, ApiMain.php, and MaintenanceRunner)
as opposed to service wiring. This is kept as-is for now.
* In LBFactoryTest, in particular testInvalidSelectDBIndependent2,
there are cases that intentionally produce failures of which
the result is then observed. In CI we assert that dberror.log is
empty so instead of adding the missing logger fields to that
LBFactory instance, the only one set (replLogger) is removed.
The alternative is to set 'logger' now, which would naturally
cause CI failures due to unexpected entries coming through to
non-mocked error log.
Bug: T320873
Change-Id: I7ca996618e41b93f488cb5c4de82000bb36e0dd3
Rename lazyLoadReplicationPositions() and doWait(). The later now
handles the logic of checking if a position is set or if the reader
index is that of the primary DB.
Narrow the job of reallyOpenConnection() by removing "waitForPos"
loading logic. It now only deals with initializing a connection and
not the logic that goes along with tracking/pooled connection handles.
Note that getReaderIndex() still triggers primary position loading and
was the only method that automatically triggered logic to actually wait
on the primary position.
Add more comments to getReaderIndex() and optimize call order slightly.
Minor cleanups to related code comments.
Change-Id: I685b1f7946aadb5463e8870edda4340be3fc4ae2
Remove 'insertSelectIsSafe' option, unused.
Remove 'topologicalPrimaryConnRef' option, no longer used as of two
months ago with I41a57247503 (8c9398f7f9).
Remove unneeded DatabaseSqlite::getTopologyBasedServerId
implementation which can inherit null instead of overriding with string
of "0". Only caller is SqlBagOStuff::makeTimestampedModificationToken
which can be used as MainStash DB, where its important that a given
server always has the same unique name within a set of db hosts that
may replicate to each other. By inheriting null as topology server ID,
it SqlBagOStuff will use IDatabase::getServerName instead. That in turn
uses the 'host' connection parameter, which defaults to null in
DatabaseFactory, and then falls back to the string "unknown" which is
as good as "0" for this purpose.
Bug: T299691
Change-Id: Iceb65c28cdd3c4a89b3c8b34c3f95d3285718ec0
Make DBConnRef enforce the DB domain selected during its lifetime
and allow more nested and successive use of the same connection handle
via DBConnRef. This can avoid extra connections in some cases where
getConnection()/getConnectionRef() is used.
Also:
* Reduce the number of connection pools arrays from six to two
* Merge getLocalConnection()/getForeignConnection() into one method
* Expand various related code comments
Since LoadBalancer::getReadOnlyReason() no longer user the local domain
but rather DOMAIN_ANY, it should not result in "USE" errors if the local
domain does not have a database on the server.
This version of the patch removes the unused reuseConnectionInternal()
method (the method was previously added back to the patch by mistake).
Bug: T226595
Change-Id: I62502f4de4f86a54f25be1699c4d1a1c1baee60b
Make DBConnRef enforce the DB domain selected during its lifetime
and allow more nested and successive use of the same connection handle
via DBConnRef. This can avoid extra connections in some cases where
getConnection()/getConnectionRef() is used.
Also:
* Reduce the number of connection pools arrays from six to two
* Merge getLocalConnection()/getForeignConnection() into one method
* Expand various related code comments
Bug: T226595
Depends-On: If808cbab429d41e1f2289683533e4a781a4bdf5e
Change-Id: I540b08920997c57cad6445ddb09d8e663eaf4714
This method is unused and the naming is not very clear in terms of what
it returns (server name vs config map). Removing it reduces externally
exposed complexity.
Optimize DBConnRef::getServerName() when the server index is known.
Use this to eliminate the "topologicalMaster" parameter from Database.
Rename internal fields and paramters in Database to use "primary"
instead of "master", for consistency.
Add some additional clarifying comments.
Bug: T299691
Change-Id: I98515fa02a58a4c72a06f1ff283b249b1617c886
This is really hard to read. What is code, what is string? These
places are so simple, they really don't need the "{$var}" syntax.
Change-Id: I589dedb8c0193eec4eef500bbb896b5b790b727b
This was written for the scenerio where we would support writes
on a primary DC request, immediately followed by a secondary DC
request, and we'd then first use the CP cookie to wait-for Redis
replication to communicate the MySQL position data, and then wait-for
a MySQL replica to have caught up to that point.
In the revised 2020 plan for Multi-DC (T254634), CP data is kept
dc-local only, and we use a sticky-dc cookie and other traffic
routing to remain in the primary DC for certain scenarios.
CP remains as playing a rule to ensure consistency within the primary
DC between the write request and the next few seconds of GET requests
to ensure a replica is picked or waited-for to have caught up with
prior writes in the same primary DC.
One test needed to be updated as this commit exposed a happy accident
where previously one key ('writeIndex') was unset in the mocked
array, which was tolerated with the null-ish operator. Now that we
use is_array only and then access the key, this undefined offset
got exposed. Fix by removing the hardcoded logic and instead
use $cp->key and $cp-mergePositions(), which also helps gain more
confidence in the test by relying on their side-effects.
Bug: T314434
Change-Id: I712e5aeb6712f4038011c3c48799e859aa40009d
The 'watchlist' query group no longer exists in WMF production since
T263127, and it's not included in the list of supported query groups
in docs/database.md.
Change-Id: I6ea5c65921891ac6a705a6ff7e79b08fa5a9bf42
This introduces $wgLBFactoryConf['configCallback'] which can be set to a
function that returns updates to be applied to $wgLBFactoryConf. The new
method LBFactory::autoreConfigure() can be called to check the callabck
and, if the config changed, reconfigure all existing LoadBalancers.
Reconfiguring the LoadBalancers causes all open connections to be
invalidated; however, any DBConnRef instances will remain valid and will
acquire a fresh connection from the LoadBalancer automatically when
appropriate.
As a proof of concept, this patch adds support for config reloding
into WikiExporter.
Bug: T298485
Change-Id: I6c3ffde62f6e038730736abe980befd90ec43e1a
createMock() does the same, but is much easier to read.
A small difference is that some of the replacements made in this
patch didn't use disableOriginalConstructor() before. In case this
was relevant we should see the respective test fail. If not we can
save some CPU cycles and skip these constructors.
Change-Id: Ib98fb06e0fe753b7a53cb087a47e1159515a8ad5
