Current configuration on the wiki
Find a file
Timo Tijhof 1d66a22805 ResourceLoader: Remove DependencyStore::renew
== Background

When file dependency information is lost, the startup module computes
a hash that is based on an incomplete summary of bundled resources.
This means it arrives at a "wrong" hash. Once a browser actually asks
for that version of the module, though, we rediscover the dependency
information, and subsequent startup responses will include arrive once
again at the same correct hash. These 5-minute windows of time where
the browser cache of anyone visiting is churned over are not great,
and so we try to avoid them.

The status quo is the dedicated module_deps table in core with no
expiry. This means a potential concern is building up gargage over
time for modules and extensions that no longer exist or are no longer
deployed on that wiki. In practice this has not been much of an issue,
we haven't run the cleanupRemovedModules.php or purgeModuleDeps.php
scripts in years. Once in 2017 to fix corrupt rows (T158105), and
once in 2020 to estimate needed space if we had expiries
<https://phabricator.wikimedia.org/T113916#6142457>.

Hence we're moving to mainstash via KeyValueDepStore, and not to
memcached. But for that we might as well start using experies.

To not compromise on losing dep info regularly and causing avoidable
browser cache for modules that are hot and very much still existing,
we adopted `renew()` in 5282a0296 when drafting KeyValueDepStore, so that
we keep moving the TTL of active rows forward and let the rest naturally
expire.

== Problem

The changeTTL writes are so heavy and undebounced, that it fully
saturates the hardware disk, unable to keep up simply with the amount
of streaming append-only writes to disk.

https://phabricator.wikimedia.org/T312902

== Future

Perhaps we can make this work if SqlBagOStuff in "MainStash" mode
was more efficient and lenient around changeTTL. E.g. rather than
simultanously ensure presence of the row itself for perfect eventual
consistency, maybe it could just be a light "touch" to ensure the
TTL of any such row has a given minimum TTL.

Alternatively, if we don't make it part of the generalised
SqlBag/MainStash interface but something speciifc to KeyValueDepStore,
we could also do something several orders of magnitudes more efficient,
such as only touching it once a day or once a week, instead of several
hundred times a second after every read performing a write that
amplifies the read back into a full row write, with thus a very large
and repetative binlog.

== This change

As interim measure, I propose we remove renew() and instead increase
the TTL from 1 week to 1 year. This is still shorter than "indefinite"
which is what the module_deps table does in the status quo, and that
was never an issue in practice in terms of space. This is because
the list of modules modules is quite stable. It's limited to modules
that are both file-backed (so no gadgets) and also have non-trivial
file dependencies (such as styles.less -> foo.css -> bar.svg).

== Impact

The installer and update.php (DatabaseUpdater) already clear
`module_deps` and `objectcache` so this is a non-issue for third
parties.

For WMF, it means that the maintenance script we never ran, can
be removed as it will now automatically clean up this stuff after
a year of inactivity, with a small cache churn cost to pay at that
time.

Bug: T113916
Bug: T312902
Change-Id: Ie11bdfdcf5e6724bc19ac24e4353aaea316029fd
2022-07-12 15:25:39 -07:00
.phan tests/parser: Run phan 2022-06-02 00:13:30 +02:00
.pipeline
cache
docs registration: Add suggested dependencies to extension.json schema 2022-07-10 00:47:20 +00:00
extensions
images
includes ResourceLoader: Remove DependencyStore::renew 2022-07-12 15:25:39 -07:00
languages Localisation updates from https://translatewiki.net. 2022-07-11 09:35:03 +02:00
maintenance Merge "Use GrantsInfo and GrantsLocalization instead of MWGrants" 2022-07-10 18:19:01 +00:00
mw-config build: Update stylelint-config-wikimedia to 0.13.0 2022-05-04 23:17:03 +01:00
resources Merge "Merge away mediawiki.ForeignStructuredUpload module" 2022-07-06 18:49:47 +00:00
skins
tests Merge "Hard deprecated the MWGrants class" 2022-07-11 19:31:09 +00:00
.dockerignore
.editorconfig build: Exclude .git/** in .editorconfig 2022-06-22 01:53:33 +01:00
.eslintignore
.eslintrc.json
.fresnel.yml
.gitattributes
.gitignore
.gitreview
.mailmap mailmap: Update Zabe's entry 2022-05-29 23:52:32 +02:00
.phpcs.xml ResourceLoader namespace (attempt 2) 2022-05-24 15:41:46 +00:00
.stylelintrc.json build: Update stylelint-config-wikimedia to 0.13.0 2022-05-04 23:17:03 +01:00
.svgo.config.js build: Update 'svgo' to latest v2.8.0 2022-02-27 23:54:58 +00:00
.vsls.json
api.php
autoload.php language: Move l10n-related classes from /cache to /language 2022-06-28 16:27:20 -07:00
CODE_OF_CONDUCT.md
composer.json Upgrading wikimedia/xmp-reader (0.8.2 => 0.8.3) 2022-07-10 03:26:14 +01:00
composer.local.json-sample
COPYING
CREDITS Prepare active branch following REL1_38 cut, labelling as 1.39-alpha 2022-03-14 21:50:17 -04:00
DEVELOPERS.md
docker-compose.yml
FAQ
Gruntfile.js
HISTORY HISTORY: Add 1.37.4 2022-07-08 18:43:28 +01:00
img_auth.php
index.php index.php: Remove global $mediaWiki variable 2022-03-26 21:55:56 +00:00
INSTALL installer: Require MySQL 5.7+ or MariaDB 10.3+, and distinguish in docs 2022-05-15 18:59:13 -04:00
jsduck.json Add codex-search v0.1.0-alpha.6 2022-05-24 17:11:48 -04:00
load.php resourceloader: Minor doc changes following PSR-4 namepace change 2022-05-29 16:41:19 +00:00
opensearch_desc.php
package-lock.json package-lock: run npm audit fix 2022-06-04 15:45:34 +01:00
package.json Rest: Move TransformHandler to core (part 1) 2022-06-02 15:55:06 +01:00
phpunit.xml.dist Revert "phpunit: Default to vendor/bin/phpunit, remove suites.xml" 2022-06-09 14:15:54 +00:00
README.md
RELEASE-NOTES-1.39 Merge "Hard deprecate HTMLCacheUpdate" 2022-07-11 19:51:34 +00:00
rest.php
SECURITY
thumb.php maintenance: Replace some usages of Wikimedia\(suppress|restore)Warnings() 2022-02-24 12:52:05 +00:00
thumb_handler.php
UPGRADE Bump minimum required version for upgrade to 1.31 2022-03-15 22:12:59 +01:00

MediaWiki

MediaWiki is a free and open-source wiki software package written in PHP. It serves as the platform for Wikipedia and the other Wikimedia projects, used by hundreds of millions of people each month. MediaWiki is localised in over 350 languages and its reliability and robust feature set have earned it a large and vibrant community of third-party users and developers.

MediaWiki is:

  • feature-rich and extensible, both on-wiki and with hundreds of extensions;
  • scalable and suitable for both small and large sites;
  • simple to install, working on most hardware/software combinations; and
  • available in your language.

For system requirements, installation, and upgrade details, see the files RELEASE-NOTES, INSTALL, and UPGRADE.

MediaWiki is the result of global collaboration and cooperation. The CREDITS file lists technical contributors to the project. The COPYING file explains MediaWiki's copyright and license (GNU General Public License, version 2 or later). Many thanks to the Wikimedia community for testing and suggestions.