MWCryptHKDF was added ten years ago (in af66c04d39), and as far as
I can tell, it was never used anywhere. It seems unlikely that CryptHKDF
will be used in the future, at least in its current form, for several
reasons:
* PHP 7.1.2+ has hash_hkdf(), so HKDF() would not be needed.
* At the time MWCryptHKDF was created, access to a CSPRNG was dependent
on server configuration: operating system, enabled PHP extensions,
open_basedir, etc. The "clock drift" RNG used as a last resort was not
considered to be secure or fast enough for generating large amounts of
output.[1] random_bytes(), added in PHP 7, changed the situation.
* Depleting the input pool of Linux's RNG is no longer a concern; there
is no more blocking output pool for /dev/random.[2][3] In 2022, this
change and others, including some that improved performance,[4] were
backported to stable kernels as old as 4.9.[5]
* $wgAuthenticationTokenVersion obviated the primary use case of
quickly resetting the user_token field for all users, assuming all
the existing tokens are unique.
* CryptHKDF seems to perform much slower than random_bytes(), at least
on Linux, making it pointless to use given that the other reasons for
its existence no longer apply.
[1]: https://bots.wmflabs.org/logs/%23mediawiki-core/20161004.txt
[2]: https://lwn.net/Articles/808575/
[3]: https://lore.kernel.org/all/cover.1577088521.git.luto@kernel.org/
[4]: https://www.zx2c4.com/projects/linux-rng-5.17-5.18/
[5]: https://lore.kernel.org/all/Yo3pmh9hiUFtQz77@zx2c4.com/T/
Change-Id: I29136fad826341d21728671aa30285d5551f1162
In T340552, the official PHP OpenTelemetry client was effectively
rejected for inclusion in MediaWiki due to its size. Implement a minimal
tracing library instead that eschews conformance with the OTEL client
specification in favor of simplicity, while remaining capable of
emitting trace data in OTLP format and thus retaining compatibility with
any ingestion endpoint capable of handling OTLP.
In its current state, the library supports a basic feature set that
should be sufficient for basic tracing integration:
* Span creation, inclusive span activation and automatic parent span
assignment,
* Span attributes and span kinds,
* Basic resource (process/request)-level metadata generation,
* Data export over OTLP.
Additional functionality, such as trace propagation, can then be
incrementally added to the library.
Bug: T340552
Change-Id: Ibc3910058cd7ed064cad293a3cdc091344e66b86
Replaces 'copyright' with 'copyright-footer' and 'history_copyright'
with 'copyright-footer-history' (the original still takes precedence
if set). Adds SkinCopyrightFooterMessage hook which works the same
way as SkinCopyrightFooter for the new messages. Allows disabling
the old messages by setting $wgAllowRawHtmlCopyrightMessages = false.
Co-Authored-By: Gergő Tisza <tgr.huwiki@gmail.com>
Bug: T45646
Change-Id: I5fd5607f8d43b6e934c8d4d35097cec430c56043
Controlled by $wgParsoidSelectiveUpdateSampleRate (which defaults to off)
randomly sample 1 in N parses to collect statistics to inform the design
of Parsoid selective update:
* For both legacy parses and Parsoid, count how many times a previous
parse is in the cache when a new parse is requested. This needs to
sample the legacy parser as well as Parsoid because Parsoid is not
yet invoked from the RefreshLinksJob. We also count the relative
number of parses from the different
RevisionRenderer::getRenderedRevision() call sites to determine
which pathways might account for the most opportunities for
optimized selective update.
* For sampled parses using the Parsoid parser where a previous parse
result is available, also fetch the previous wikitext source from the
database.
Bug: T371713
Change-Id: I208aeac1b315a96bdb9669427cd03de461b914b4
Depending on configuration, this returns either the interface language
code of the current user or the current page language.
Bug: T4085
Change-Id: Iab7fda272ec81af88c74612727ff6bed014d4a81
Some wikis treat the category list from ParserOutput as a /set/, others
as an /ordered list/. For those who don't care about the order of
categories, provide the option for wikis to sort the categories
in OutputPage.
This can also be activated with a query parameter, `&sortcat=1`, which
is useful to the Parsoid team when doing visual diff testing to avoid
false positives caused by differences in category ordering.
Bug: T373480
Change-Id: Idd14650a1898c6a49c88441ef024ce3012903bbe
Update a few tests that relied on the feature flag to ignore
the 'requireemail' preference on "User1" to instead use "User2",
who doesn't have the preference set.
Bug: T242406
Change-Id: I996d3996272d704a071d1d2094c3568247b80f98
Developers can use this to test their local version of Codex with
MediaWiki by pointing $wgCodexDevelopmentDir to their local clone of the
Codex repo, e.g. $wgCodexDevelopmentDir = '/home/yourname/git/codex';
Setting $wgCodexDevelopmentDir affects where the following things come
from:
- Codex JS/CSS files for the full library
- Codex JS/CSS files for code-split chunks, and the manifest.json file
that points to them
- Icons retrieved by CodexModule::getIcons()
- CSS-only icons imported in Less
- Design tokens imported in Less
Other changes in this patch:
- Add CodexModule::makeFilePath() to centralize the repeated path
concatenation. This makes it easier to switch out the regular path for
the dev mode path.
- Replace all uses of $IP (which is deprecated) and MW_INSTALL_PATH in
CodexModule with the BaseDirectory config setting.
- Make CodexModule::getIcons() reset its static cache if the path to the
icons file changes. Without this, it's impossible to make the unit
tests pass.
- Move the i18n messages code from the CodexModule constructor to
getMessages(). It can't be in the constructor because makeFilePath()
doesn't work there (it fails because the Config object hasn't been set
up yet).
- Add a 'mediawiki.skin.codex' import path so that we can stop
hard-coding the path to the Codex mixins file. Without this, we can't
make the Codex mixins come from the right place in development mode.
- Consider $wgCodexDevelopmentDir in setting the cache key for compiled
Less code, since changing this setting can change the output of Less
compilation (by changing design tokens, icons or mixins).
- Add unit tests for (the non-dev mode behavior of)
CodexModule::getIcons() and the i18n message key handling.
Bug: T314507
Change-Id: I11c6a81a1ba34fe10f4b1c98bf76f0db40c1ce98
== Motivation ==
* Reduce amount of configuration switches relating to MW Stats component,
to reduce noise for MW sysadmins and devs.
* Reduce amount of code in the legacy Statsd implementation.
== Background ==
* commit e56f7b6c63 (2015, MW 1.26): Add statsd sampling.
Refers to T106457 under T106450 ("hook usage counts"), which used it
in https://gerrit.wikimedia.org/r/226640 which was reverted shortly
after in https://gerrit.wikimedia.org/r/233045 due to too high
overhead of stats code for something as hot as hooks, regardless
of actual sampling/stats sending.
* commit b203ec5fcd (2016, MW 1.29): Make statsd sampling configurable.
Use case unknown, but it is not used today in wmf-config, and
appears to have never been set in the past either:
```
operations-mediawiki-config (master)
$ git log -p wmf-config/ | grep wgStatsdSamplingRates
24s ec=1$
```
Looking for unmerged patches, I do find an abandoned patch for
the Wikibase extension that attempted to use to instrument database
load in 2020. It was abandoned in favour of dedicated DB tooling
to inspect database queries, which offer more detail as well.
https://gerrit.wikimedia.org/r/q/message:wgStatsdSamplingRateshttps://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/593335
== Future ==
In 10 years, we've not needed sampling anywhere even at our scale.
StatsD absorbs high traffic very well, with controlled flushing to
Graphite as backend. Prometheus exporters are similarly buffered
close to the producer with controlled scraping intervals. This
should make sampling unlikely to be needed.
We did have one experiment where sampling was attempted, which led
to the developments this commit removes. That instrumentation was
removed because the code was far too frequently called and hot that
it posed a latency problem, even before we factor in whether it
sends the metric. The overhead of the stats code itself added too
much overhead to Hook::run / HookContainer.
The new StatsFactory service class does feature per-metric sampling,
although this is similarly without use case at the moment:
https://codesearch.wmcloud.org/search/?q=%3EsetSampleRate&excludeFiles=test
New instrumentations should use StatsFactory, and if sampling were
needed in the future, it could be experimented by calling setSampleRate
on Metric objects (as part of the instrumentation, rather than in
site config, unless the feature in question makes it configurable).
That seems enough upfront investment for a theoretical future need.
Change-Id: I5f68e48d6d2c880a43f83915234d3c9f32a2d1ef
Adds an experimental configuration to allow extensions to define
OutputPipelineStages to include in the DefaultOutputPipeline.
There are a lot of open questions about this api, like ordering of
execution, but adding it @experimental will help surface the
requirements.
Bug: T370541
Needed-By: I6dc92af0611c680b6e55605a7c9ff8a3fc1dfa26
Change-Id: I64baea40a1687c7a06fbcda9efe9f9a159b0ae8d
Special:RestSandbox presents a Swagger-UI interface for exploring REST APIs. The available APIs can be configured using RestSandboxSpecs.
For now, the default is to support no APIs, so the feature is disabled in production. In the future, it would make sense to expose the wiki's own REST API per default. The corresponding entry in $wgRestSandboxSpecs in LocalSettings.php would look like this:
'mw' => [
'url' => $wgScriptPath . '/rest.php/',
'name' => 'MediaWiki REST API',
]
Note that the spec URL may still change.
To also explore the endpoints exposed through RESTbase, we might add:
'wmf-restbase' => [
'url' => $wgServer . '/api/rest_v1/',
'name' => 'Wikimedia RESTbase API',
]
Similarly, we could expose a spec for endpoints on api.wikimedia.org, which could then be explored using the new special page.
NOTE: This adds a dependency on the swagger-ui npm library. See T325558 for the security review.
Bug: T362006
Change-Id: I1dd5ed82680a28f9c15136b446a2de0398525061
- Add a disabled by default feature flag 'EnableProtectionIndicators'
- When the config flag is enabled, show a lock indicator at the top
of the page.
- The lock icon should be overridable by the content of the page
- The indicator has a predictable ID which could be potentially used
to style the icon using the onwiki Common.css file.
- The lock icon by default links to https://www.mediawiki.org/wiki/Help:Protection. However
this link can be customized per wiki per protection level using a
empty message (for example: `protection-sysop-helppage`)
Bug: T12347
Change-Id: I3e36d98edfe54a9c138b1fe8e5057e107ded281b
Legacy parser can now output headings using a more accessible markup,
which is also identical to the markup used by the Parsoid parser.
Changes to client-side JS and CSS necessary to support the new markup
have already been merged in earlier commits.
includes/skins/Skin.php
includes/ServiceWiring.php
* Define a new skin option, 'supportsMwHeading', which can be used
to toggle the new markup per-skin.
* Update the built-in fallback skin to enable it. This affects the
output in parser tests.
docs/config-schema.yaml
includes/config-schema.php
includes/config-vars.php
includes/MainConfigNames.php
includes/MainConfigSchema.php
* Add a new configuration setting, 'ParserEnableLegacyHeadingDOM',
which can be used to toggle the new markup per-site.
includes/OutputTransform/Stages/HandleSectionLinks.php
* Output new heading HTML for skins that enabled the option.
tests/*
* Duplicate parser tests that cover heading generation to cover both
new and old markup. Update other parser tests to use new markup.
* Add some unit and integration tests for the behavior of the skin
option and some parser tests for edge cases of the new markup.
Bug: T13555
Change-Id: I1180169a8e83af834c2984ba16089e6277f2a8dd
openssl provides the proper password hashing algorithm, so it is needed
in order to have safe and efficent password hashing.
Change-Id: I61498275c7f7cf19787f0aee50dc4884c57b82b2
Why:
- We don't want to allow unlimited acquisition of temp account names.
These should be rate limited in similar way to how we limit the
creation of temp accounts
What:
- Provide a TempAccountNameAcquisitionThrottle, and use it in the
acquireName() method
- Set a default that is 10 times the limit of
TempAccountNameCreationThrottle
Depends-On: If660aad1d0f04f366414084aff3f88484a19d416
Bug: T343101
Change-Id: I99d5973498a89ac227847de5837c0a8e895c28fb
With this change, when async uploads are enabled, upload-by-url
will spawn a job and a form with a button to check the status of the
process is shown to the user.
In the process, add processing of warnings in the remote jobs spawned by
the API or the Special page. This is done by adding checks to
UploadJobTrait::verifyUpload. In order to manage warnings serialized in
the job status, a method to unserialize the result of
UploadBase::makeWarningsSerializable.
Things that we might want to fix:
* The form's UI is abysmal, we should probably use Codex
* While it's not a huge deal, I'd like to figure out why I need to
purge the page cache if I want the file to show up. And more
interestingly, why this doesn't happen when uploading via the API
Bug: T295007
Bug: T118887
Change-Id: I49181d93901f064815808380285fc4abae755341
Described as "Evil hack for mobile :(", this config variable had been
overridden in the MobileFrontend extension for some time, but this has
been removed last year in Icb1711a0cef828dda0d99b8a7c7963bc409816c9.
Since it was introduced, most HTMLForms using the 'table' format have
been reworked to use 'ooui' (or 'codex', these days), so the hack is
no longer needed.
We recently removed $wgUseMediaWikiUIEverywhere for similar reasons
in aa7eeeeef9 / be135c0fc0.
Change-Id: Id507c0dc5a1c1cfb9737073bae97b14badc30a54
Add a temporary feature flag that is going to be covering the work
around implementing a logout confirmation.
Bug: T357484
Change-Id: I3ee8f52976cb956e24f32115c02903c5c4f358c3
Why:
- The pathway to creating a regular account is different from that of a
temporary account. It makes sense to rate limiting creation of these
two types of accounts at different rates.
What:
- Add a TempAccountCreationThrottle config with a restrictive default
that matches the existing production configuration of
AccountCreationThrottle (6 creations per day)
- Update resetAuthenticationThrottle.php to support resetting the temp
account creation throttle
- For now, not adding an equivalent hook for account creation throttle's
ExemptFromAccountCreationThrottle
Bug: T357777
Change-Id: Ibf36ac96f717107bace6f0a3326f79ed129a1dfe
The TranslationAliasesDirs configuration allows defining translatable
aliases in JSON files. The value should be a name or names of folders
that contains files that have localized aliases. Each language should
have a separate file.
Currently, it supports defining special page aliases but in the
future can be extended to support magic words and namespace aliases.
The patch adds a script: ConvertExtensionsMessagesToTranslationAlias
that can be used to convert existing ExtensionMessagesFiles to the new
format.
Bug: T89947
Change-Id: Ief16a48a8dc8742854f67301791aa2a0b0531116
Introduce a temporary feature flag that will be used to display
Special:Block UI in Codex.
Bug: T358153
Change-Id: I74bc90af986ac2f429a952a19b6edf4fb02c5618
After the deployment of the updated mediawiki.cookie module, the
configuration variable can be safely removed.
Bug: T344791
Change-Id: Idf3b3ef521ef2ac368c3f575f99bf4c6da7baa82
Not to be confused with $wgOpenSearchTemplates (plural)
Soft-deprecated since 1.25 and hard-deprecated since at least
I9776d11d4e2d184 (1.39).
Not set in production.
Change-Id: I6a95b9e0e2b74bb44d51c375a62886761ec9ea88
MediaWiki UI has never really made it out of beta before being
deprecated. Legacy code using plain HTML forms doesn't need it;
new code can be written using Codex.
Bug: T182050
Change-Id: Ib9966bc6a4a94f771cb99a5aa52fb6a1dc826ca5
Soft-deprecated since 1.26 and hard-deprecated since at least
I9776d11d4e2d184 (1.39).
Not set in production.
Change-Id: I55e4569c5a40f77a719bc7ab1924eaa706e0da53
Soft-deprecated since 1.35 and hard-deprecated since at least
I9776d11d4e2d184 (1.39).
Not set in production.
Change-Id: I68b68ffcdff22daafad34c186d1cc609dcecbf32
Soft-deprecated since 1.33 and hard-deprecated since at least
I9776d11d4e2d184 (1.39). It's already noop in the code.
Not set in production.
Change-Id: I5895bc4dae9620d53c37188662e21bf01ae7bd85
Soft-deprecated since 1.37 and hard-deprecated since at least
I9776d11d4e2d184 (1.39)
Not set in production.
Change-Id: I0b25e89d1652dfa697ab1176a1e2b6ca3163bdab
It has been deprecated since 1.35, hard deprecated since at least I9776d11d4e2d184
No grep result in wmf-config
Change-Id: I2c67bab3e3212f25a0cbce7301f5e32a082f76da
INSERT IGNORE when inserting NULL into a non-nullable field will succeed
with a warning on MySQL but fail on PostgreSQL. In any case, it's
probably harmful and unintended. But to check the error code of MySQL
warnings, you need to query the server with SHOW WARNINGS, so there is a
performance cost.
So, add a configuration variable which, when enabled, checks warnings
after INSERT to see if there were any null type constraint errors. Set
it to true in DevelopmentSettings.php and TestSetup.php.
Change-Id: I5e47e2d3cc7e0f804036e11b512b1e3b76804432
Why:
Sometimes, it is necessary to have different behavior
for newly registered and existing users. For example,
this happens in the Echo or GrowthExperiments extensions.
As of now, this behavior is implemented by inserting
user_properties rows in onLocalUserCreated.
Over time, this results in a singificant amount of rows
inserted, which contributes to the user_properties table bloat,
which is already overly large (cf. T54777). This patch makes it
possible to remove such rows by supporting conditional defaults
for user properties.
What:
Add support for conditional defaults of user properties. This can be
configured via `ConditionalUserOptions` config option.
Bug: T321527
Change-Id: I1549c3137e66801c85e03e46427e27da333d68e2
Following up Id9539a28f0f143539334002c3:
* Don't run the script twice.
* Wrap the decoded dump in an array with key "data".
* The default assignment for DJVU_DUMP and DJVU_TXT had the effect of
running the scripts anyway if the configuration variables are null.
Don't do that.
* If both $wgDjvuDump and $wgDjvuTxt are null, don't run the shellbox
script.
* Centralise shell location configuration.
* Factor out call to convertDumpToJSON().
* Instead of txt_exit_code, just use existence of the file to
communicate success. This avoids a deprecation warning if
txt_exit_code was not received, due to passing null to trim.
* Check for the existence of the result files instead of just trying to
use them.
* Check the exit status of the overall script.
* Confirm that the BoxedCommand branch is functional and works in CLI
mode by using it in DjvuTest.
* Change the service name from "media" to "djvu". Existing examples are
"pagedtiffhandler" and "pdfhandler", i.e. named after the extension,
there is no other core caller. I think it should be more fine-grained
than "media". The name was possibly a conflation with the
ProductionServices array key.
Also:
* Check the exit status of djvudump and don't use the output file if it
is non-zero.
* Check the return value of convertDumpToJSON().
* Don't use isset() unless error suppression is intended.
Bug: T352515
Change-Id: If41a2baada2e4e2462518c1f437af458feb29632
This change was first introduced in Ic4e3dff and then reverted because
of a regression on CLI. So for now forward-port the code from that patch
and put it under a feature flag we can remove once we've debugged
the current issue.
Bug: T289228
Bug: T352515
Change-Id: Id9539a28f0f143539334002c356026b0d185450b
Pages that are fast to render can be omitted from the parser cache
to preserve disk space and cache write operations.
The threshold is configurable per namespace, so the tradeoff can
be evaluated based on different access patterns. For example, pages
that are accessed rarely, like file description pages on commons,
may have a high threshold configured, while pages that are read
frequently, like wikipedia articles, may be configured to be always
cached, using a 0 threshold.
Filtering is based on a time profile recorded in the ParserOutput.
A generic mechanism for capturing the timing profile is implemented
in the ContentHandler base class. Subclasses may implement a more
rigorous capture mechanism.
Bug: T346765
Change-Id: I38a6f3ef064f98f3ad6a7c60856b0248a94fe9ac
mcrouter-primary-dc was added in 9c14bb5 (I9bc589ec57f4c6), follow
the reverts and reapply too for more context as a way to store tokens
across all servers that run the application, and later purged afterward.
This patch introduces a MediaWiki wrapper service for interfacing
with mcrouter-primary-dc instead of accessing the setting directly
via operations config. This service will be configured by a setting
through $wgMicroStashType.
Testing plan
============
We can begin to directly use this patch as a way to test core rate
limiter via WRStatsFactory. So if that works accurately with this
patch without breaking CI, then that's good news.
In this patch also, MainConfigNames::StatsCacheType setting has been
removed in favor of MicroStashType via a wrapper service. NOTE that
there is already a config patch to take care of the switch and the
former will be removed (on cleanup later).
Bug: T336004
Depends-On: I96f6f7c0240a46e43a785dbd3c0a5a524d4eabc3
Change-Id: I08ebea5e8a601f161fcd678068cecb77f8e15c7c
This config has been set to false in production since 2018 (Ie4ea1dc0d3927).
This doesn't provide much benefit and its impact yet to be proven and
it's blocking removal of ILoadBalancer::getAnyOpenConnection()
That removal makes $dbwSerial unconditionally false which turned the
whole method into a one-liner and since it was used only once,
we just replaced that.
Bug: T325389
Depends-On: I24f33a15d214d49bfb6c6013e7fd64c7d0eb0086
Change-Id: I19043e775fce9f7ddded128cb2c9395b46dd19a3
Add $wgEnableMultiBlocks and $wgBlockTargetMigrationStage vars that will
allow for multi-stage migration.
Bug: T346671
Change-Id: Id356445605870abf279f056490040cb2b5921d7a
