We already had a $filterCacheVersion variable, but it was
only used for the internal cache for JS and CSS minification,
which is not enough. If there is a breaking change in either
of these processes, we also need to invalidate version hashes.
This commit renames ResourceLoader::$filterCacheVersion to
ResourceLoader::CACHE_VERSION and takes it into account in
getVersionHash(). Adding it to getDefinitionSummary() is not
sufficient, because content-hashed modules also need to be
invalidated when there's a breaking change in the minifiers.
This cache version can also be incremented when there's a
breaking change in image embedding or LESS compilation,
although content hashing deals with that already, so we
could also add a separate cache version for those that's
only added to getDefinitionSummary().
Bug: T176884
Change-Id: Ife6efa71f310c90b9951afa02212b2cb6766e76d
This seems quite nuclear. I'd actually like to deprecate the
wgCacheEpoch variable more generally in favour of a handful
of more specific version constants, but as starting point,
remove it from the hash used for load.php urls and localStorage
keys.
The latter is also controlled by wgResourceLoaderStorageVersion
already.
Also ref T32956 about a more standalone ResourceLoader.
Change-Id: I913f846090e82d3d822653b9b7ce22233cdb5e90
This existed for internal use by OutputPage, which is no longer
the case as of I11b390f2e4f5e7db.
Also move the unit tests from OutputPageTest,
to ResourceLoaderSkinModuleTest.
Change-Id: I8b23f976f5f89b1005b387a827f75031f5c96141
This was introduced in OutputPage before support for getPreloadLinks()
was added to ResourceLoader. The introduction in ResourceLoader was
actually inspired by this original implementation.
Now that we have it, we should make use of it for this module
as well. Doing so has several benefits:
* Makes the code cleaner by not requiring every skin to implement
the extra boolean method. Instead, it naturally works. If
the skin loads the SkinModule, it gets the preload as well.
If not (such as Minerva, which has a different logo config),
then it also doesn't get the preload link.
Naturally, automatic.
* Makes code cleaner by not having static methods, and by not
having OutputPage call into a Module class.
* Fixes the problem where, if a site's logo is changed, all cached
HTML is preloading the old logo whilst the stylesheet fetches
the newer one. Causing both to be downloaded.
* Still preloads the logo well before it can render.
Change-Id: I11b390f2e4f5e7db8b4506ab547839152888005c
They were accidentally made non-functional in April 2009 by commit
e4d21170. Subsequent commits 2d045fa1, 9e230f30, e275ea28, ae45908c
removed all callers by October 2011. Needless to say, I found no
callers in core or extensions.
It seems we have the functionality in some other way, probably just by
directly calling setHTMLTitle(), so there's no need to revive this
feature.
Bug: T200643
Change-Id: Ifa2537b97cf19e7e91ee33d4ea4c131f7a38ee89
The primary goal here is a defense in depth measure to
stop an attacker who found a bug in the parser allowing
them to insert malicious attributes.
This wouldn't stop someone who could insert a full
script tag (since at current it can't distinguish between
malicious and legit user js). It also would not prevent
DOM-based or reflected XSS for anons, as the nonce value
is guessable for anons when receiving a response cached
by varnish. However, the limited protection of just stopping
stored XSS where the attacker only has control of attributes,
is still a big win in my opinion. (But it wouldn't prevent
someone who has that type of xss from abusing things like
data-ooui attribute).
This will likely break many gadgets. Its expected that any
sort of rollout on Wikimedia will be done very slowly, with
lots of testing and the report-only option to begin with.
This is behind feature flags that are off by default, so
merging this patch should not cause any change in default
behaviour.
This may break some extensions (The most obvious one
is charinsert (See fe648d41005), but will probably need
some testing in report-only mode to see if anything else breaks)
This uses the unsafe-eval option of CSP, in order to
support RL's local storage thingy. For better security,
we may want to remove some of the sillier uses of eval
(e.g. jquery.ui.datepicker.js).
For more info, see spec: https://www.w3.org/TR/CSP2/
Additionally see:
https://www.mediawiki.org/wiki/Requests_for_comment/Content-Security-Policy
Bug: T135963
Change-Id: I80f6f469ba4c0b608385483457df96ccb7429ae5
There has long been a hack for previewing edits to user JS/CSS, where
OutputPage would pass an 'excludepage' parameter to
ResourceLoaderUserModule to tell it not to load one particular page and
would instead embed that page statically. That's nice, but there are
other places where we could use the same thing.
This patch generalizes it:
* DerivativeResourceLoaderContext may now contain a callback for mapping
titles to replacement Content objects.
* ResourceLoaderWikiModule::getContent() uses the overrides, and
requests embedding when they're used. All subclasses in Gerrit should
pick it up automatically.
* OutputPage gains methods for callers to add to the override mapping,
which it passes on to RL. It loses a bunch of the special casing it
had for the 'user' and 'user.styles' modules.
* EditPage sets the overrides on OutputPage when doing the preview, as
does ApiParse for prop=headhtml. TemplateSandbox does too in I83fa0856.
* OutputPage::userCanPreview() gets less specific to editing user CSS
and JS, since RL now handles the embedding based on the actual
modules' dependencies and EditPage only requests it on preview.
ApiParse also gets a new hook to support TemplateSandbox's API
integration (used in I83fa0856).
Bug: T112474
Change-Id: Ib9d2ce42931c1de8372e231314a1f672d7e2ac0e
The siteinfo API response's 'writeapi' value is now hard-set to true,
as are the ResourceLoader variables wgEnableAPI and wgEnableWriteAPI,
to be deprecated later.
Bug: T115414
Change-Id: I54ff9428b247ba203d67aba079149393f323d5a9
Clean up use of @codingStandardsIgnore
- @codingStandardsIgnoreFile -> phpcs:ignoreFile
- @codingStandardsIgnoreLine -> phpcs:ignore
- @codingStandardsIgnoreStart -> phpcs:disable
- @codingStandardsIgnoreEnd -> phpcs:enable
For phpcs:disable always the necessary sniffs are provided.
Some start/end pairs are changed to line ignore
Change-Id: I92ef235849bcc349c69e53504e664a155dd162c8
SVGs could already be used through $wgLogo. However, if a PNG fallback
is desired for older browsers, using SVGs was previously not possible.
This commit adds support for using an SVG image in $wgLogoHD and,
using $wgLogo as the fallback image.
Usage example:
> $wgLogo = '/path/to/png';
> $wgLogoHD = [
> 'svg' => 'path/to/svg',
> ];
Note: When the 'svg' key is set in $wgLogoHD, any '1.5x' and '2x' keys will
no longer be used because SVGs can render optimally on any screen sizes.
@Reedy, @Krinkle and @Brion VIBBER helped me alot with this.
Bug: T86229
Change-Id: I6197d96ce9110f4711ef2c4b198445bc5c6ae110
This fixes two bugs:
* 1) When two modules are requested, and the first one ends with ";"
inside a comment, the second module might become part of
that comment.
* 2) A request with script=only where the requested module content
ends in a statement without ";", but has a comment after it
that does ends with a semicolon, then in debug=false, mw.loader.state()
would be appended directly after the semicolon-less statement because
the check is performed before minification.
Previously:
script> foo()
script> // bar();
states> mw.loader.state( {} );
Became (minified separately):
script> foo()
states> mw.loader.state({});
Became (concatenated)
> foo()mw.loader.state();
Which is invalid code.
Both of these are now fixed.
Bug: T162719
Change-Id: Ic8114c46ce232f5869400eaa40d3027003550533
Regardless of whether other modules exist with group=user or group=site,
these two modules in particular must always be in their own request for
legacy reasons.
This has already always been the case because even in the few cases where
an extension uses this group (eg. MobileFrontend's custom site module) it
would load it instead of another module in that group, never at the same
time. There is one notable exception, which is GlobalCssJs. However the
ext.globalCssJs.user.styles module is usually served from another wiki
which is why that went unnoticed as well. This commit fixes that so that
even if you're viewing a page on the central wiki, the modules are still
in separate requests.
Aside from this one existing edge case, there is also need to add
group=site to gadgets by default so that they load after the DynamicStyles
marker instead of before, which is currently causing problems with the
cascading order (gadget apply before core and skin styles due to being
in the same request group and alphabetically sorting before them).
Semantically, the appropiate solution is group=site, but this wasn't
possible due to core putting "all" group=site modules in the same request
(under the assumption there is only one such module). This commit removes
that fragile assumption.
Bug: T147667
Change-Id: I9eb725c083124d22a9af3bf3d075ade6f3b970a3
Replaces \TestingAccessWrapper (defined in core) with
\Wikimedia\TestingAccessWrapper (defined in the composer package
wikimedia/testing-access-wrapper).
See https://gerrit.wikimedia.org/r/#/q/topic:librarize-testing-access-wrapper
for downstream patches.
The core version of the class is kept around for a while to avoid
circular dependency problems.
Bug: T163434
Change-Id: I52cc257e593da3d6c3b01a909e554a950225aec8
Follows-up 5f55e9c9c2.
If the logo url is from within /w, then ResourceLoaderSkinModule
will (as it should) apply a file hash query to it.
The preloader didn't do that, so it specified the wrong url.
Refactored SkinModule to make this logic re-usable.
Bug: T100999
Change-Id: I1ba11f7c70d1a725ad72754fee4a3f33c2a4c1be
This greatly increases the priority of loading
the logo on browsers that support rel="preload".
Bug: T100999
Change-Id: I0738fcc0a575153dab65016fa87faaa9b8b97a9d
Updated tests to reflect this use case. Currently we assume all web-accessible
paths within ResourceBasePath (e.g "/w") to exist on disk at the same path
in $IP (e.g. "/var/www/mw").
While in theory any number of web server rewrites or aliases could exist,
there is one case in particular that we should support since the information
is available in the configuration: UploadDir and UploadPath. This path may
be rewritten in a way that varies by wiki in multi-wiki installs that share
the same source code. E.g. a server may rewrite "/w/images" to somewhere
else, which means it will not match the directory on disk that is shared
between wikis.
Bug: T155146
Change-Id: I320478c9c262cc012f08b585b48d290594ec2420
This is a very basic test setup for the core functionality only. All
the special cases in getHeadLinksArray are currently not tested. But
this is a start.
This is relevant for the changes made in Ie8fd697.
Bug: T51859
Change-Id: I61e2da68ae0daea15fab2065a5dd63605d0b837d
transformResourcePath is only supposed to be called with something
that starts with a '/' (absolute path from document root).
While the primary caller (CSSMin) only calls this method if the path
makes sense as a local file path, ResourceLoaderSkinModule did not,
and wgLogo may be set to a full url that includes a domain.
While chance made it so that protocol-including urls were already
discarded, protocol-relative urls were mistaken for being a path
inside the file system root with a duplicate slash (e.g.
'//tmp/foo.txt', as 'foo.txt' in /fmp, instead of '/foo.txt' at
http://tmp).
This should be fixed upstream in the wikimedia/relpath library,
but workaround it for now since it really shouldn't be called
with urls in the first place.
Bug: T155310
Change-Id: I9b063f1219ddeca5cc2c8a48832cdf8c9eaffe58
This is a pure documentation change. It mostly removes empty lines from
comments (and entirely empty comments), as well as adds a few missing
documentation blocks and fixes a minor mistake. I hope it's ok to have
this in one patch. I can split it, please tell me.
Change-Id: I9668338602ac77b903ab6b02ff56bd52743c37c4
There's currently no way to get, e.g., all categories except the
hidden ones just as text. The OutputPage::getCategories() method
always returns all categories as an array of strings (titles) and
the getCategoryLinks() method returns the result of Linker::link
but with the distinction between "normal" and "hidden" categories.
This change adds a new parameter to OutputPage::getCategories(),
$type, which can be used to define, what categories should be
returned. The default value is "all", which means, that all categories
are returned (the current result of the method). With the value
"normal" and "hidden", the method will return the respective values.
This could be used in I97d7de723fe72da26c7dbde0a559a13704c7099a to
remove the stupid Linker::link() and isset workaround.
Change-Id: Iadda9ae362a21fbee770240234b8f55326219932
HTML formatting of the queue was distributed over several OutputPage methods.
Each method demanding a snippet of HTML by calling makeResourceLoaderLink()
with a limited amount of information. As such, makeResourceLoaderLink() was
unable to provide the client with the proper state information.
Centralising it also allows it to better reduce duplication in HTML output
and maintain a more accurate state.
Problems fixed by centralising:
1. The 'user' module is special (due to per-user 'version' and 'user' params).
It is manually requested via script-src. To avoid a separate (and wrong)
request from something that requires it, we set state=loading directly.
However, because the module is in the bottom, the old HTML formatter could
only put state=loading in the bottom also. This sometimes caused a wrong
request to be fired for modules=user if something in the top queue
triggered a requirement for it.
2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose
of allowing dependencies on style modules without risking duplicate loading
on pages where the styles are loaded already. This didn't work, because the
state information about page-style modules is output near the stylesheet,
which is after the script tag with mw.loader.load(). That runs first, and
mw.loader would still make a duplicate request before it learns the state.
Changes:
* Document reasons for style/script tag order in getHeadHtml (per 09537e83).
* Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed
before since a duplicate check in makeResourceLoaderLink() verified the
origin a second time.
* Declare explicit position 'top' on 'user.options' and 'user.tokens' module.
Previously, OutputPage hardcoded them in the top. The new formatter doesn't.
* Remove getHeadScripts().
* Remove getInlineHeadScripts().
* Remove getExternalHeadScripts().
* Remove buildCssLinks().
* Remove getScriptsForBottomQueue().
* Change where Skin::setupSkinUserCss() is called. This methods lets the skin
add modules to the queue. Previously it was called from buildCssLinks(),
via headElement(), via prepareQuickTemplate(), via OutputPage::output().
It's now in OutputPage::output() directly (slightly earlier). This is needed
because prepareQuickTemplate() calls bottomScripts() before headElement().
And bottomScript() would lazy-initialise the queue and lock it before
setupSkinUserCss() is called from headElement().
This makes execution order more predictable instead of being dependent on
the arbitrary order of data extraction in prepareQuickTemplate (which varies
from one skin to another).
* Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early
on so. This avoids wrongful loading and fixes problem 1.
Effective changes in output:
* mw.loader.state() is now before mw.loader.load(). This fixes problem 2.
* mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading".
* mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1.
* The <script async src> tag for 'startup' changed position (slightly).
Previously it was after all inline scripts and stylesheets. It's still after
all inline scripts and after most stylesheets, but before any user styles.
Since the queue is now formatted outside OutputPage, it can't inject the
meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle
of existing output. This shouldn't have any noticable impact.
Bug: T87871
Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
tl;dr: Having unnessary complexity in security critical code is bad.
* Extra options add extra complexity and maintenance burden
** Thus we should only have one html output mode. well formed = false
was already vetoed in T52040, so lets go with WellFormed=true.
* Options which are used by very few people tend to get tested less
* Escaping is an area of code where we should be very conservative
* Having escaping rules depend on making assumptions about which
characters various browsers consider "whitespace" is scary
* $wgWellFormedXml=false has had a negative security impact in the
past (Usually not directly its fault, but has made other bugs
more exploitable)
* Saving a couple bytes (even less bytes after gzip taken into
account) is really not worth it in this context (imho).
Change-Id: I5c922e0980d3f9eb39adb5bb5833e158afda42ed
* Send 'module' and 'require' parameters to module closures.
This depends on Ia925844cc22f143 being deployed one cycle earlier.
* Patch Moment and OOjs to ensure these libraries continue to expose
their module as globals as well. AMD/UMD-compatible libraries
only expose a global *OR* an export, not both. We need both
for back-compat.
* Update pluralRuleParser to make use of module export to allow
usage via require().
To test, check out the patch and run:
> mw.loader.load('moment');
> mw.loader.require('moment')()
> mw.loader.require('moment')('2011-04-01').fromNow()
Bug: T108655
Change-Id: Idbd054880ee70d659ec760aef8fcb38d0704a394
* Remove trailing space in self-closing tag.
Brings parserTest output of Parser and Parsoid closer together.
* Remove various line breaks at begin and end of script contents.
* Remove FILTER_NOMIN from makeConfigSetScript() output.
This isn't part of any user- or page-dependent module and not minified.
And Xml::encodeJsCall already ensures compact output for prod mode.
Bug: T127328
Change-Id: I85a5a59fd0955c1a112e8b24b933f0d9e983a156
startup.js:
* Don't assign to "window.RLQ" and read from "RLQ". Use the same
identifier in both places instead of relying on global scope.
* Create the global window.RLQ only once. The temporary [] fallback
doesn't need to be exposed globally, so use a local variable for
the loop that processes the pre-existing queue built up while
the startup module was loading.
OutputPage:
* Simplify the RLQ wrap to be more idiomatic and less repetitive.
Matches the pattern used in Google and Facebook open-source projects.
Change-Id: I9176377bc05432e51add841696a356428feec8ce
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
- Batch retrieval.
- Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
part of get() instead of within updateMessage().
ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
Timestamps are unreliable and roll over too frequently for message blob store
because there is no authoritative source. The timestamps were inferred based on
when a change is observed. Message overrides from the local wiki have an
explicit update event when the page is edited. All other messages, such as
from MediaWiki core and extensions using LocalisationCache, have a single
timestamp for all messages which rolls over every time the cache is rebuilt.
A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
Instead, detect whether a module's message list is empty at runtime.
ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.
cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.
Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
Ensure that OutputPage::haveVaryCacheCookies() handles falsey values correctly,
and that it requires a cookie to have a nonempty value before it declares a
match.
Change-Id: I2afe54b62c940187a427498cb4037e1dd0e78dd9
MediaWiki currently has support for a header called X-Vary-Options
(XVO), used to communicate to upstream caches more granular cache
variance options than the Vary header can.
The header was envisioned by Tim Starling back in 2008 and implemented
into MediaWiki and Squid 2.0, with those patches submitted to the
squid-dev mailing list at the time:
http://www.squid-cache.org/mail-archive/squid-dev/200802/0085.html
The patches never actually made it into an upstream Squid release,
however, and Squid has since evolved in potentially significant ways.
Wikimedia has since switched to Varnish but XVO was not ported over as
it was deemed too complex at the time; custom VCL was used instead. To
our knowledge, noone else is using XVO in production and certainly not
with recent, up-to-date MediaWiki releases.
There is currently work at IETF's httpbis working group for a "Key"
header that is in concept and implementation very similar to Tim's XVO
header: https://datatracker.ietf.org/doc/draft-fielding-http-key/
Rather than rip XVO out of MediaWiki, replace it with support for the
Key header, as preliminary defined by the draft spec. This is an almost
straight search-and-replace.
No other software (caching proxy or user-agent) currently implements Key
to my knowledge, so this is essentially untested.
Change-Id: I949fc289dd5d48bd34f3b37f7739e2b9cd5db277
Rewrite OutputPage::addVaryHeader which had a very confusing
structure.
There is one breaking change: the $option argument was declared
as array|null, but the function accepted everything and showed
inconsistent behavior; e.g.
$op->addVaryHeader( 'Foo', 'bar' )
resulted in 'X-Vary-Options: Foo;bar' but
$op->addVaryHeader( 'Foo' )
$op->addVaryHeader( 'Foo', 'bar' )
resulted in 'X-Vary-Options: Foo'. With the patch, non-array
arguments (other than null) result in an error.
Change-Id: Id31d95fe27b01b00ec8a1d7a3996275fc0aacf3c
Follows-up r88883 (8dab43f703). The IEUrlExtension library only considers the
query string if the base path has no extension (or if the dot is urlencoded).
Neither of which is the case for load.php urls.
If the server doesn't expose undecoded request URIs (per IEUrlExtension)
then it will look in the query string always. However even then it is
only an issue if the last query parameter in the url contains a dot,
which is very unlikely for load.php (they typically end in 'skin' or
'version' which don't contain plain dots).
Ref T30840.
Change-Id: I4a4ef7ef5ad7d74ecb47a44342e6324fa9844173
