Being a raw module means that when it is requested from load.php with
"only=scripts" set, then the output is *not* wrapped in an
'mw.loader.implement' closure *and* there no 'mw.loader.state()' appendix.
Instead, it is served "raw".
Before 2018, the modules 'mediawiki' and 'jquery' were raw modules.
They were needed before the client could define 'mw.loader.implement', and
could never be valid dependencies. Module 'mediawiki' merged to 'startup',
and 'jquery' became a regular module (T192623). Based on the architecture
of modules being deliverable bundles, it doesn't make sense for there to
ever be raw modules again. Anything that 'startup' needs should be bundled
with it. Anything else is a regular module.
On top of that, we never actually needed this feature because specifying
the 'only=scripts' and 'raw=1' parameters does the same thing.
The only special bit about marking modules (not requests) as "raw" was that
it allowed the client to forget to specify "raw=1" and the server would
automatically omit the 'mw.loader.state()' appendix based on whether the
module is marked as raw. As of Ie4564ec8e26ad53f2, the two remaining use
cases for raw responses now specify the 'raw=1' request parameter, and we
can get rid of the "raw module" feature and all the complexity around it.
== Startup module
In the startup module there was an interesting use of isRaw() that has
little to do with the above. The "ATTENTION" warning there applies to the
startup module only, not raw modules in general. This is now fixed by
explicitly checking for StartupModule.
Above that warning, it talked about saving bytes, which was an optimisation
given that "raw" modules don't communicate with mw.loader, they also don't
need to be registered there because even if mw.loader would try to load
them, the server would never inform mw.loader about the module having
arrived. There are now no longer any such modules.
Bug: T201483
Change-Id: I8839036e7b2b76919b6cd3aa42ccfde4d1247899
Also, for the unit test, disable the two methods we use there
that can get called. The unintended side-effects of these two
methods was the only reason it used `@group Database`.
Removing that makes the test a bit faster as well.
Enforce this via MediaWikiServices for this suite to avoid an
untracked dependency slipping back in in the future.
Bug: T225730
Change-Id: I6c54466e9517d9899bc39f8f9bb946369c0a526d
The use cases we've seen for using computed (or virtual) package files,
involve expensive computations to expand and transform data for the client
that we don't want to evaluate in full just to compute the module's version
hash (e.g. in the StartupModule, where we need to do this for 1000s of
modules).
For such cases, the module can specify a 'versionCallback' of which the
return value will be used to seed the module's version hash. The default
remains the same as before, which is to use the full content to seed the
version hash (via getDefinitionSummary).
Bug: T223260
Change-Id: I76f573239e6bd429287e7adb33a92ffd5e260c20
Was introduced as a transitional measure in Ic566a1cd7efd075c3.
Depends-On: I717f03caf9ea8266e6a4d2b6daf4c543c0815931
Change-Id: I6ae615ea38572042f8ba705338067b393827153a
The module definition format for 'packageFiles', as initially designed,
mixes sequential and associative arrays. This works in PHP, but not in
JSON. To make the format JSON compatible, use a 'name' key instead of
using the key in the main array.
Leave backwards compatibility in place so that extensions using the old
format can be migrated. This will be removed in the next commit.
Before:
'packageFiles' => [
'script.js',
'script2.js',
'config.json' => [ 'config' => [ 'Foo', 'Bar' ] ],
'data.json' => [ 'callback' => function () { ... } ],
],
After:
'packageFiles' => [
'script.js',
'script2.js',
[ 'name' => 'config.json', 'config' => [ 'Foo', 'Bar' ] ],
[ 'name' => 'data.json', 'callback' => function () { ... } ],
],
This can then be written in extension.json as:
"packageFiles": [
"script.js",
"script2.js",
[ "name": "config.json", "config": [ "Foo", "Bar" ] ],
[ "name": "data.json", "callback: [ "MyExtHooks", "getData" ] ]
]
Change-Id: Ic566a1cd7efd075c380bc50ba0cc2c329a2041d7
Deprecated since MW 1.27.
Also update ResourcesTest to use TestingAccessWrapper instead of long-form
object reflection, and also apply it to its call for this method given
its meant to be private.
Change-Id: I9cc1af93730f632e4f8bf3a16d514a51ee73cb03
Package files are files that are part of a module, but are not
immediately executed when the module executes. Instead, they are
lazy-excecuted when require() is called on them. Package files can be
scripts (JS) or data (JSON), and can be real files on the file system,
or virtual files generated by a callback.
Using virtual data files, server-side data and config variables can be
bundled with a module. Support for file-based require() allows us to
import npm modules into ResourceLoader more easily.
The require function passed to each script execution context, which was
previously a reference to the global mw.loader.require() function, is
changed to one that is scoped to the module and the file being executed.
This is needed to support relative paths: require( '../foo.js' ) can
mean a different file depending on the path of the calling file.
The results of require()ing each file (i.e. the value of module.exports
after executing it) are stored, and calling require() on the same file a
second time won't execute it again, but will return the stored value.
Miscellaneous changes:
- Add XmlJsCode::encodeObject(), which combines an associative array of
XmlJsCode objects into one larger XmlJsCode object. This is needed for
encoding the packageFiles parameter in mw.loader.implement() calls.
Bug: T133462
Change-Id: I78cc86e626de0720397718cd2bed8ed279579112
Not used since 2011 (MediaWiki 1.18).
In an early version of ResourceLoader, we ran JSMinPlus syntax
validation on-demand on all served JavaScript content. This
was identified as cause of slowdown and high memory use, and
generally not considered as useful in production.
The reason it was there originally was not for the purpose of
validating static files, but for user-generated content.
So in MediaWiki 1.18, the behaviour of wgResourceLoaderValidateJS
was changed to only apply to user-generated content, and the
rest was disabled behind wgResourceLoaderValidateStaticJS, which
we then never use. Not even in development, given that we now
have superior experience through ESLint, even within IDEs where
supported.
Follows-up 49d3d18033 (r91914).
Change-Id: Ie25109a4fb23ee93fed0db4af5db4b11fe9ffe7f
The use of global variables was deprecated in favour of
ResourceLoaderModule::getLessVars() on a per-module basis.
Also moved testLessFileCompilation case to the appropiate file as it
covers ResourceLoaderFileModule.php, not ResourceLoader.php.
Bug: T140804
Depends-On: Ib1b2808df2384473bfac47f53a5d25d7c9bbca2b
Depends-On: I96047f69d01c4736306df2719267e6347daf556f
Change-Id: If708087c85c80355c7e78f1768529b5f2e16ed07
Add @covers for various helper methods used by public methods, where the helper
methods actually contain most of the logic being tested in FileModuleTest.
I've changed these methods from protected to private (confirmed no usage)
to further pin down that their contract doesn't matter beyond making the
public methods work.
Change-Id: I2aef0d322b38bc3595e7d2c2339112b16fc66b8d
Find: /isset\(\s*([^()]+?)\s*\)\s*\?\s*\1\s*:\s*/
Replace with: '\1 ?? '
(Everywhere except includes/PHPVersionCheck.php)
(Then, manually fix some line length and indentation issues)
Then manually reviewed the replacements for cases where confusing
operator precedence would result in incorrect results
(fixing those in I478db046a1cc162c6767003ce45c9b56270f3372).
Change-Id: I33b421c8cb11cdd4ce896488c9ff5313f03a38cf
This prevents cache churn when the wiki-global LESS variables vary
between wikis because the cache key is used as a "global" instead of
db-local. This is good for the common case, but should still explicitly
vary if the vars differ between wikis.
Bug: T191937
Change-Id: If12fd07a7062792205384150d6f5fd9a83f996cc
I can see that "parent::__construct" literally calls the parent
constructor. I can see that stuff preceeded by the keyword "protected"
is protected. I really (really) don't need comments explaining such.
Change-Id: I7458e714976a6acd3ba6a7c93fdc27d03903df83
This already worked as expected for any module that uses the new
enableModuleContentVersion model, but for the majority of file modules
this is not yet the case for performance reasons. As such, make
sure lessVars are included in our manual tracking.
Include it conditionally to avoid changing the array for other modules,
which would needlessly invalidate their cache.
Bug: T171809
Change-Id: Ib250068e0ecfc29a09ca33c23bef901ee0482bf2
Should be useful for cases where we pull in an external library
that already manages RTL flipping in its styles, and CSSJanus
does the wrong thing without extra markup.
Example:
'ext.tmh.video-js' => $baseExtensionResource + [
'scripts' => 'resources/videojs/video.js',
'styles' => 'resources/videojs/video-js.css',
'noflip' => true,
...
],
Bug: T148572
Bug: T148565
Change-Id: Icbad20d8a6e9a0d354ad159f5816f4fb67cc2775
ResourceLoader modules can now carry a 'deprecated' option which can
be a boolean or an object with message key. This message or a default
deprecation message will be show whenever that module is used in production.
Note: This will not work in debug mode for ResourceLoaderFile modules
and this is deemed acceptable for the time being. We can revisit later.
Bug: T137772
Change-Id: Ib9ebd2d39a59fd41d8537e06884699f77b03580c
This allows dynamically loaded modules to depend on page-style modules
without it causing the page-style module to be loaded a second time.
* New method Module::getType() indicates whether a module is
a page-style module or supposed to be dynamically loaded.
* Emit warning from addModuleStyles() when given a module that is
not a page-style module (to be enforced later)
Bug: T92459
Bug: T87871
Change-Id: I8b6c6a10d965e73965f877c42e995d04202524f3
We read files and concatenate their contents. Files may start with a BOM character.
BOM characters are only allowed at the beginning of a file, not half way.
Stripping it should be safe, since we already assume that everything is UTF-8.
Change-Id: I14ad698a684e78976e873e9ae2c367475550a063
Follows-up 8f5cd11d82.
The old getLocalFileReferences() method is no longer used anywhere.
Remove it and rename getAllLocalFileReferences back to it.
Change-Id: I864258aad128ba9b54464c7bc854543f2937f977
The output of this function is not deterministic. This function
is what tranforms urls to include content hashes and also embeds
content as data URI. The calls to this function are expected to be
up to date and matching the version hash as constructed by the
current web request.
Previously, the content hash as constructed based on the collective
file hash of all references images was accurate but the generated
css associated with it could be stale.
Unless we abuse MemoizedCallable to take a separate cache buster
(with $localFileRefs?), it's best to remove this as it isn't
a deterministic function and shouldn't be memoized.
Bug: T128668
Change-Id: Icb87ddc585d7320ac48619446bb8b9cbe5f4780e
This way, they automatically bypass any stale client or server cache.
Remove an old wfExpandUrl() call that would interfere with
transformResourcePath()'s ability to recognise the path.
Expanding the url there wasn't needed anyway. Whether it's on a separate host
(eg. bits.wikimedia.org), or path-absolute, either way works as-is.
Bug: T90983
Change-Id: I64eb3291adcfc3733ef96690399c09c06e19b3aa
Cache Less variables in the instance instead of statically.
This allows tests to populate their own less variabless via ResourceLoaderLESSVars.
Make getLessCompiler() and getLessVars() regular public methods and
update callers.
Change-Id: I95506b8bb20a4b2b3f82014a7b0fcee5f28973e6
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
- Batch retrieval.
- Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
part of get() instead of within updateMessage().
ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
Timestamps are unreliable and roll over too frequently for message blob store
because there is no authoritative source. The timestamps were inferred based on
when a change is observed. Message overrides from the local wiki have an
explicit update event when the page is edited. All other messages, such as
from MediaWiki core and extensions using LocalisationCache, have a single
timestamp for all messages which rolls over every time the cache is rebuilt.
A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
Instead, detect whether a module's message list is empty at runtime.
ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.
cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.
Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
This effectively reverts d6b4d3c537 and declines T97420.
This was previously attempted in b7c0e537eb.
Drop support for position "bottom" for addModuleStyles().
This feature was only recently introduced with the intent
to optimise page load performance, but had an adverse effect.
It increases chances of FOUC due to late discovery of these styles.
It also caused minor problems for some gadgets and extensions
that did not or were unable to set these flags. Some mobile
code was introduced around the same time as this feature and
was never given position=top.
Stylesheets that don't affect initial render or are only needed
on interaction should be loaded via addModules() instead; which
is handled by the asynchronous loader in JavaScript.
Change-Id: Ib9821b7b87cfc8a59062bb6ca358974fdb01ced1
Use BagOStuff::makeKey() and BagOStuff::makeGlobalKey() instead of
wfMemcKey() and wfGlobalCacheKey().
Change-Id: Id4e58c0e616b74de220faf13ba7c9ea606fef8c1
Using the following command line, I have found doc comments (and
a wfDeprecated() call) mentioning "1.26" when they should mention
"1.27" instead, which I have fixed manually:
git diff -M REL1_26 | grep --color=always -C 10 -iP \
'^\+.*\D1\.26(\D|$)' | aha > oldver.html
Follows-up these commits:
* 047b60b96d
* 169b7b98b5
* 25a44aa3e4
* 2fb2a3f14b
* 3f1e9fa268 [1]
* 8c84af70b6
* c0cb80beac
* d04a92a551
* d3b85592ea
[1] Release notes moved in I195dd1cf.
Because a release note stating that the UserRights hook is deprecated
was added in 37062a0c0d (before the branch cut), this commit does not
change the Hooks::run call that had been changed in 21206c8fbe.
Change-Id: I5a427f003e7e3b4559fe377bcdfdca466a570708
Add a simple class to `libs/` for memoizing functions by caching return values
in APC. I decided not to make this an external library just yet because I see
this as potentially a part of a larger functional programming library.
Doesn't use APCBagOStuff for two reasons: (1) avoid dependency on MediaWiki
code; (2) ability to pass third &$success parameter to apc_store, to
distinguish between cache misses and cached false values.
Use this in ResourceLoaderFileModule to cache CSSMin::remap.
Change-Id: I00a47983a2583655d4631ecc9c6ba17597e36b5f
As Timo noted in a review of I1ff6115, using '+=' to combine file dependencies
is incorrect here. I fixed it in one place but not the other, causing the file
dependencies to vary based on whether a .lesscache file was used.
Change-Id: Ie3f48048febc32c5c2d45acd87e10f946f6dfc94
Re-use the md_skin database field for now as to not need a
schema change, beause this table is going away soon (T113916).
Bug: T113868
Change-Id: I7c7546ec58fd9be0447604989b908dd2084b0fe3
Make paths stored in the module_deps table relative to $IP. This ensures that when
the MediaWiki install path changes and/or if the location of the extension or skins
directory changes, that ResourceLoader's internal model is still accurate.
Previously when these paths change, ResourceLoader would exhibit various bugs.
1. Unable to detect changes in the module (if the directory no longer exists).
2. Point #1 is usually preceeded by one last cache invalidation as the content hash
of the file path changes (from a valid hash to an empty string).
3. Unnecessary cache invalidation (if both old and new directories exist). This
happens when a file is both an entry point (in the 'scripts' or 'styles' array)
and also a file dependency. At first they are de-duplicated by array_unique.
But after the disk path changes, the next check will result in the old path
being fetched from module_deps, and the new path from the live configuration.
This causes two changes that result in needless cache invalidation:
- The hash list contains one more item (T111481).
- The hash list contains both the old and new version of a file.
(or even alternate versions, e.g. when a change is backported to the old
wmf branch; it also affects wikis on the new branch due to the stale
file path still in the database).
It seems unusual to move a MediaWiki install, and usually we recommend third
parties to run update.php if site administrators do move their wiki. However
Wikimedia's deployment system implicitly moves the MediaWiki install continously
from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6".
This caused virtually all ResourceLoader caching layers to get invalidated every
week when another wmf-branch is deployed, thus breaking these file paths, which
changes the version hash, which then invalidates the cache.
Bug: T111481
Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
* Remove ResourceLoaderFileModule::getLessCompiler(). There is no reason for a
module to need to get a compiler in a different manner than
ResourceLoader::getLessCompiler().
* Add ResourceLoaderModule::getLessVars(). This method provides a means for
subclasses to easily inject custom LESS variables. The default implementation
simply returns an empty array.
* Make the $context parameter for ResourceLoaderFileModule::readStyleFiles()
non-optional (via graceful deprecation). The only callers I found were either
already calling it with a ResourceLoader context, or had a perfectly usable
ResourceLoaderContext in local scope.
* Make ResourceLoaderFileModule::{readStyleFile,getLessCompiler} require a
context. These methods are protected, so I didn't bother with a deprecation.
* Call ksort() on the LESS variables array in the only place it matters -- when
hashing its serialized representation to construct a cache lookup key. This
relieves getLessVars() subclasses from having to remember to re-sort the
variables array if they modify it.
* These changes make it possible to substantially simplify
ResourceLoaderEditToolbarModule, because the only thing it needs to do now is
implement its own getLessVars() method.
* This also allows it to be versioned like any other ResourceLoaderFileModule,
rather than having to use enableModuleContentVersion().
Change-Id: Ic3eab71691e502bfe19bdf4eb6f82cc679a7782f
Most of it was already there (SELECT in preloadInfo, and in-class holder via
setFileDependencies) but the logic to write to the database was within
the file module class.
Moving it the other way around may make more sense in the future, but for
the moment ResourceLoader.php makes assumptions about all modules being
able to store setFileDependencies as read from the database. Whether or not
a class currently implements a way to write to that table.
Change-Id: I06b5da4144798197478c66e8b9ccd4cc62cf6fb6
Caching the output of a LESS compiler is tricky, because a LESS file may
include additional LESS files via @imports, in which case the cache needs
to vary as the contents of those files vary (and not just the contents of
the primary LESS file).
To solve this, we first introduce a utility class, FileContentsHasher. This
class is essentially a smart version of md5_file() -- given one or more file
names, it computes a hash digest of their contents. It tries to avoid
re-reading files by caching the hash digest in APC and re-using it as long as
the files' mtimes have not changed. This is the same approach I used in
I5ceb8537c.
Next, we use this class in ResourceLoaderFileModule in the following way:
whenever we compile a LESS file, we cache the result as an associative array
with the following keys:
* `files` : the list of files whose contents influenced the compiled CSS.
* `hash` : a hash digest of the combined contents of those files.
* `css` : the CSS output of the compiler itself.
Before using a cached value, we verify that it is still current by asking
FileContentHasher for a hash of the combined contents of all referenced files,
and we compare that against the value of the `hash` key of the cached entry.
Bug: T112035
Change-Id: I1ff61153ddb95ed17e543bd4af7dd13fa3352861
The way this is implemented is really dirty... but it found us
a few pre-existing bugs already (T111518, T111519, T111771).
I think it might be worth it.
* CSSMin: Add new method getAllLocalFileReferences() which skips the
file_exists() check.
* ResourceLoaderModule: Make use of it to track missing files too.
* ResourcesTest: Verify that the missing files are missing.
Change-Id: I5a3cdeb7d53485f161ccf8133e76850cdf5b4579
