After the roll-out of $wgResourceLoaderUseObjectCacheForDeps on
WMF wikis, there was an unrelated database spike that caused some
error messages:
> DependencyStoreException:
> Cannot access the database: Too many connections (db1132)
I found this confusing, because servers shouldn't be using old
DepStore class any more (ref T311788?). And because the new store
is x2.mainstash, whereas the reported hostname is an s1.enwiki host.
I wasted some time not trusting the code path as there was no Rdbms
trace available to confirm for sure that this isn't an unrelated query
that happens to be caught during the DepStore interaction (e.g. some
generic MW code running from a hook, or Rdbms internal query from
LoadMonitor etc).
Improve telemetry by preserving the original trace.
I considered passing `$e` as third parameter to
DependencyStoreException, but since the new implementation doesn't
actually use this class, it's effectively going to remain unused in the
future and would not reliably indicate anything in particular to callers
unaware of which implementation is in use. There's also some benefit
to being able to aggregate and filter out specific db issues, which
is made harder by the same issue being reported multiple different
ways through wrapped errors.
The old implementation will remain for one release as default, and
probably one release after that as option for any third parties that
encounter an issue during upgrade so as to not block their upgrade
while we find/address the issue in question.
Bug: T113916
Change-Id: Iaa3907fc3aa0622daa9648eabfdd7efabdd4f2a9
== Background
When file dependency information is lost, the startup module computes
a hash that is based on an incomplete summary of bundled resources.
This means it arrives at a "wrong" hash. Once a browser actually asks
for that version of the module, though, we rediscover the dependency
information, and subsequent startup responses will include arrive once
again at the same correct hash. These 5-minute windows of time where
the browser cache of anyone visiting is churned over are not great,
and so we try to avoid them.
The status quo is the dedicated module_deps table in core with no
expiry. This means a potential concern is building up gargage over
time for modules and extensions that no longer exist or are no longer
deployed on that wiki. In practice this has not been much of an issue,
we haven't run the cleanupRemovedModules.php or purgeModuleDeps.php
scripts in years. Once in 2017 to fix corrupt rows (T158105), and
once in 2020 to estimate needed space if we had expiries
<https://phabricator.wikimedia.org/T113916#6142457>.
Hence we're moving to mainstash via KeyValueDepStore, and not to
memcached. But for that we might as well start using experies.
To not compromise on losing dep info regularly and causing avoidable
browser cache for modules that are hot and very much still existing,
we adopted `renew()` in 5282a0296 when drafting KeyValueDepStore, so that
we keep moving the TTL of active rows forward and let the rest naturally
expire.
== Problem
The changeTTL writes are so heavy and undebounced, that it fully
saturates the hardware disk, unable to keep up simply with the amount
of streaming append-only writes to disk.
https://phabricator.wikimedia.org/T312902
== Future
Perhaps we can make this work if SqlBagOStuff in "MainStash" mode
was more efficient and lenient around changeTTL. E.g. rather than
simultanously ensure presence of the row itself for perfect eventual
consistency, maybe it could just be a light "touch" to ensure the
TTL of any such row has a given minimum TTL.
Alternatively, if we don't make it part of the generalised
SqlBag/MainStash interface but something speciifc to KeyValueDepStore,
we could also do something several orders of magnitudes more efficient,
such as only touching it once a day or once a week, instead of several
hundred times a second after every read performing a write that
amplifies the read back into a full row write, with thus a very large
and repetative binlog.
== This change
As interim measure, I propose we remove renew() and instead increase
the TTL from 1 week to 1 year. This is still shorter than "indefinite"
which is what the module_deps table does in the status quo, and that
was never an issue in practice in terms of space. This is because
the list of modules modules is quite stable. It's limited to modules
that are both file-backed (so no gadgets) and also have non-trivial
file dependencies (such as styles.less -> foo.css -> bar.svg).
== Impact
The installer and update.php (DatabaseUpdater) already clear
`module_deps` and `objectcache` so this is a non-issue for third
parties.
For WMF, it means that the maintenance script we never ran, can
be removed as it will now automatically clean up this stuff after
a year of inactivity, with a small cache churn cost to pay at that
time.
Bug: T113916
Bug: T312902
Change-Id: Ie11bdfdcf5e6724bc19ac24e4353aaea316029fd
Move ResourceLoader classes to their own namespace. Strip the
"ResourceLoader" prefix from all except ResourceLoader itself.
Move the tests by analogy.
I used a namespace alias "RL" in some callers since RL\Module is less
ambiguous at the call site than just "Module".
I did not address DependencyStore which continues to have a non-standard
location and namespace.
Revert of a241d83e0a.
Bug: T308718
Change-Id: Id08a220e1d6085e2b33f3f6c9d0e3935a4204659
This reverts commit e08ea8ccb9.
Reason for revert: Breaks Phan in extensions, and as far as I’m aware,
this change isn’t urgently needed for anything, so the simplest fix is
to revert it again for now. After PHP 7.4 it should be safer to try this
again (we hopefully won’t need the two “hack” classes by then).
Bug: T308443
Change-Id: Iff3318cbf97a67f821f78e60da62a583f63e389e
Move ResourceLoader classes to their own namespace. Strip the
"ResourceLoader" prefix from all except ResourceLoader and
ResourceLoaderContext.
Move the tests by analogy.
I used a namespace alias "RL" in some callers since RL\Module is less
ambiguous at the call site than just "Module".
I did not address DependencyStore which continues to have a non-standard
location and namespace.
Change-Id: I92998ae6a82e0b935c13e02a183e7c324fa410a3
2022-05-16 14:41:27 +10:00
Renamed from includes/resourceloader/dependencystore/SqlModuleDependencyStore.php (Browse further)
