* Create task-specific methods with simple defaults that require no
mocking or stubbing of any kind, as used by the pure unit tests
where service container (and by extent, storage services) are
disabled.
* Remove all use of global variables, LBFactory, JobQueue,
StatsdFactory, and RequestContext.
Bug: T265749
Change-Id: If85c448d2d1b806e70f641f06263680d49c6eeec
In prep for the next decoupling commit:
* Rephrase introduction to state what its actually for in practice.
* Remove mentions of implementation details in favour of inline
comments, especially stuff relating to databases.
Except for 1 thing, every integration with databases is already
fully decoupled.
Automatic cancellation on rollback works by having the caller pass
on an IDatabase object to addCallableUpdate() for that specific
update, which works even if it came from an unknown LBFactory
or unknown service containter.
Oppertunistic execution is triggered by service wiring, where
MWLBFactory takes responsibility for having LBFactory notify the
DeferredUpdates singleton; not the other way around.
Bug: T265749
Change-Id: I048d22ffe2fa3838d9a5f4aa4128c756185a6b2e
DU is a PHP process level mechanism to finish things related to a MW
request (which is possible to context changes). By request, meaning it's
stateful and a lot of things can be altered before the response gets to
the client or completes.
Services on the other hand are by design stateless meaning that making
DU stateless is approaching things from a contrary perspective. This means
when an update is deferred, nothing can change anymore as it's now done
so via a DUManager but that's the opposite of what we want becuase an
update (whether presend or postsend) should be able to do other things
in the background (perform changes to the DB, change an article's rendering
etc).
This patch is an attempt to move DU away from being a service class.
Bug: T265749
Depends-On: I2b0e80bdd92a5fd4d7c325fc8792df45e0776c30
Depends-On: I8126cc76440724753c356c48ba4e0fcc9be5b41a
Depends-On: Ia2fa060237c4e9d328d1b2b9622a65fcca97b854
Change-Id: Ifbfb0442bf6264d0bf786334416c5d99c0dd4699
This patch doesn't deal with the injection of dependencies
and removal of the global state, but rather moves the code
from DeferredUpdates to the new service essentially as-is,
to simplify review. The changes to inject the various
services needed and make DeferredUpdatesManager a proper
service will be done in follow-ups, to make them easier
to follow.
While almost everything is changed from static to non-static,
DeferredUpdates::$scopeStack remains static as
DeferredUpdatesManager::$scopeStack, just in case multiple
versions of the service are created, to ensure that no
updates are missed.
Bug: T265749
Change-Id: I7f07eddf2fc399b15db4fe9be4c792ef8eb0747b
This is really hard to read. What is code, what is string? These
places are so simple, they really don't need the "{$var}" syntax.
Change-Id: I589dedb8c0193eec4eef500bbb896b5b790b727b
Keep the $unused for now since, while it is internal, there are
actually callers in test suites of various extensions.
Change-Id: Ib56298c98b26b22568abcd71bc844651b120fe01
Callback style iteration made sense before generators existed, but
generators make for simpler code. The "call method" variants made
sense before closures existed but defeat static analysis.
So, in LBFactory:
* Add ILBFactory::getAllLBs()
* Deprecate ILBFactory::forEachLB()
* Remove LBFactory::forEachLBCallMethod(), was protected.
* Add LBFactory::getLBsForOwner(), which is protected and has the
internal interface in @return. Adding a new abstract method breaks
Wikibase tests despite LBFactory not being stable to extend.
* Migrate callers. Generators allow you to return/break from the middle
of the loop, which implies a little rearrangement for some callers.
In LoadBalancer, connections supposedly of type IDatabase were
routinely type-hinted as Database in closure parameters so that methods
could be called that were not in the interface. So it's convenient to
get rid of public iteration methods entirely in favour of private
methods returning Database[].
* Hard-deprecate ILoadBalancerForOwner::forEachOpenConnection()
and replace it with a private generator method since nothing called
it externally except for core tests.
* Hard-deprecate ILoadBalancerForOwner::forEachOpenPrimaryConnection()
and replace it with a private generator. DeferredUpdates needed it for
iterating over IDatabase::explicitTrxActive(), so add
ILoadBalancer::explicitTrxActive() as a replacement.
* Replace private method LoadBalancer::forEachOpenReplicaConnection()
with a generator.
Depends-On: If0b382231e27d6d1197fb7b6aef6ab50335df4e5
Change-Id: I64514e77b9bfe737be5b12e1d3c9c49976bb522f
Doesn't cost anything to compute and include in logs, and would be
useful for understanding serial processing of DeferredUpdates
(basically, non-deferred deferred updates) in CI are slowing down
browser tests.
Bug: T225968
Change-Id: Id55bde5d8f4e465723d8db6df29b4c554fc1bc9c
This logic is not needed to run on every PHP process and was making
it difficult to run offline maintenance scripts without additional
complexity based on Maintenance::getDbType and DB_NONE.
Instead of skipping this only for DB_NONE, and establishing a pattern
that may spread to other ad-hoc places throughout the codebase, instead
remove this entirely from the eager set up code for all PHP processes
and move it to the service wiring and dependency injection.
That way, it naturally doesn't happen until and unless the DB service
is actually called upon. Scripts and entry point that need to disable
the DB service, can continue to use
MediaWikiServices::disableStorageBackend.
== Impact on SiteStatsUpdate ==
With wgCommandLineMode no longer being read at run-time from a global,
but in service wiring, this means SiteStatsUpdateTest can't change
the behaviour between CLI-like and Web-like, unless it e.g. resets
the 'DBLoadBalancerFactory' service first. Unfortunately, while most
any reset is supported, a reset of the 'DBLoadBalancerFactory' would
be unsupported as that would lose the temporary db clone context and
such, bringing us to either the developer's live db, or a broken set
up altogether. If there is a strong need for toggling oppertunistic
updates off and on at run-time, this could be supported in the
DeferredUpdates class perhaps, but we already have numerous methods
there (incl db begin/commit being a good proxy already), which this
test already used, so for now I've just removed the extra assertion
for this as it wasn't essential to that test.
Bug: T228895
Bug: T238436
Change-Id: Icf29bc484c155f52b6d8f61e5902233a15ba0c6d
This is micro-optimization of closure code to avoid binding the closure
to $this where it is not needed.
Created by I25a17fb22b6b669e817317a0f45051ae9c608208
Change-Id: I0ffc6200f6c6693d78a3151cb8cea7dce7c21653
Follow-up ba6490aa1e.
That patch worked around $executeContext iteration errors due to the
lack of recursion support in doUpdates()/handleUpdateQueue(). Errors
were triggered by MediaWiki::schedulePostSendJobs() enqueueing deferred
updates that invoke JobRunner::run(), which, in turn, invoke doUpdates()
for each job via JobRunner:: doExecuteJob(). The doUpdates() method was
changed to operate on the sub-queue for the in-progress DeferrableUpdate.
Further improve the recursion logic:
* Use classes for the scope stack and scope queues.
* Put *all* updates added during a DeferrableUpdate::doUpdate() call into
the subqueue, regardless of "defer until" stage or whether it implements
MergeableUpdate. Now, doUpdate() can run *everything* it enqueued instead
of a non-obvious subset. Note that the TransactionRoundDefiningUpdate
that invokes JobRunner was already a POSTSEND update before ba6490aa1eb;
the only effect of this change is that MergeableUpdate instances from jobs
will once again run in doExecuteJob().
* Make recursive DeferredUpdates::doUpdate() calls error out immediately
unless the DeferrableUpdate responsible is a TransactionRoundAwareUpdate
with the TRX_ROUND_ABSENT flag. This covers the schedulePostSendJobs()
scenario and only prohibits insane call patterns. Failing early avoids
the risk of handleUpdateQueue() dropping all the updates due to the same
DBTransactionError error in DeferredUpdates::attemptUpdate().
* Avoid recursion loop in tryOpportunisticExecute() when JobQueueDB is in
use and a large number of tasks are pending. This happened due to methods
like onTransactionPreCommitOrIdle() being used within JobQueueDB.
Mark DeferredUpdates::doUpdates()/tryOpportunisticExecute() as @internal
and create a Maintenance::shutdown() method to avoid a direct call in the
doMaintenance.php file.
Bug: T249069
Bug: T268840
Change-Id: Ib369f0e74243a48ababdb9cd83b155c9a0f5e741
PHP 7.0 makes many error conditions throw instances of the new Error class
which does not extend the known Exception.
The Throwable interface provides a concise and type-safe way of handling
either, e.g. for logging purposes, but HHVM did not support it, requiring
tedious fallback checks.
This commit replaces occurrences of Exception in code paths equally
covered by Throwable, like Exception|Throwable parameter and return types
(also nullable), instanceof guards, duplicated `catch` blocks, as well as
related comments and documentation blocks, with the exception of $previous
parameter descriptions consistent with the manual at
https://www.php.net/manual/en/exception.construct.php
Proper type declarations have been added or reinstated where possible.
Change-Id: I5d3920d3cc66936a350314e2f19c4f6faeffd7c0
This way most existing Logstash dashboards looking for traces
can pick this up without requiring the filter to be duplicated
or written as a custom Query DSL entry.
Also:
* Fix the broken logger from the jobify() method which did not
define the "{type}" or "{exception_message}" keys at all.
* Fix the over-normalized logging of all fatal errors from deferred
updates as "Error from {type}: {exception}" which isn't useful to
aggregate (we can already get an overall count for
channel:DeferredUpdates, level:error). The normalized aggregate
is meant to uniquely identify specific errors that have a common
cause. For other fatal exceptions we leave the keep the cause
and exception message in the raw message and only add the request
context and trace into placeholder fields (MWExceptionHandler.php).
Apply the same principal here as well.
* Field 'exception.class' will now be available. This was previously
missing (e.g. what exception is being thrown). It would log
"The given Title does not belong" instead of
"InvalidArgumentException: The given Title…".
Bug: T233342
Change-Id: I9b15658c97e3bfcc2ce8b234d1c0d47c9c294fb7
Bail out in attemptUpdate() if the transaction state is dirty rather
that failing at some later point. Also, flush implicit transaction
rounds before calling DeferrableUpdate::doUpdate() for fresher data.
Note that only instances of EnqueueableDataUpdate can become jobs.
Make handleUpdateQueue() defer throwing the exception until every task
was attempted for the special unit test logic.
Clean up some of the logging from 34427e7d7b.
Bug: T206283
Change-Id: I84ba1f2f8c4bf7c8ef21a907f73ad1065dd8f330
Some of the errors are suppressed because they're phan false positives.
The idea behind this is that they'll be fixed in a future version of
phan, and we'll just have to remove the suppressions.
Note: I'm disabling UnusedPluginSuppression so that we can start suppressing
issues even if they're still disabled. The sniff should be re-enabled
as soon as we upgrade phan.
Bug: T231636
Change-Id: I0f7fa06a9e03fbb86c7a5eb6e50a850bb258a7f7
Bail out in attemptUpdate() if the transaction state is dirty rather
that failing at some later point. Also, flush implicit transaction
rounds before calling DeferrableUpdate::doUpdate() for fresher data.
Bug: T225103
Change-Id: I4f5d2f9814a562069619f05e003663fcedbd3f64
Other cleanups and fixes:
* Split up handleUpdate() method into run() and jobify()
* Handle Throwable errors
* Use 'cli' in stats keys in CLI mode instead of "get"
* Tweak some code comments
Change-Id: I7749465df2d7b58e66ee5ebdd3c3d25aea52eeb3
