Commit graph

57 commits

Author SHA1 Message Date
Timo Tijhof
ae1db0cff0 resourceloader: Fix reliance on global state in SkinModule
* Remove odd use of ResourceLoaderFileModule::extractBasePaths()
  SkinModule.php which was basically just returning $IP.

* Use getConfig()->get('ResourceBasePath') so that the test does
  not need to override global configuration variables.

Follows-up 6845912bcf and f1457312dc.

Bug: T277728
Change-Id: I64262b48bb5dc71c53b4114eddb2f080a6a076f5
2021-05-14 20:29:53 +00:00
Umherirrender
a120f1dc87 Use @before/@after in MediaWikiIntegrationTestCase/MediaWikiUnitTestCase
Set functions to final.
This avoids that the function is overwritten without parent call

Follows-up I9d4771c28160356ff58.

Change-Id: I40cde489a892b06284692ecbbef14e650afe7c9e
2021-04-27 22:21:59 +02:00
Daimona Eaytoy
535d7abf59 phpunit: Mass-replace setMethods with onlyMethods and adjust
Ended up using
  grep -Prl '\->setMethods\(' . | xargs sed -r -i 's/setMethods\(/onlyMethods\(/g'

special-casing setMethods( null ) -> onlyMethods( [] )

and then manual fix of failing test (from PS2 onwards).

Bug: T278010
Change-Id: I012dca7ae774bb430c1c44d50991ba0b633353f1
2021-04-16 20:15:00 +02:00
Roan Kattouw
b267f7aa90 resourceloader: Allow modules to mark themselves as ES6-only
Modules that set "es6": true in their module definition will error when
a non-ES6 client tries to load them.

To detect ES6 support, this looks for native Promise support,
RegExp.prototype.flags, and non-BMP characters in variable names. All
browsers that lack full ES6 support fail at least one of those checks.

To flag modules as requiring ES6, this adds a ! to the end of their
version string. This takes up much less space than adding another
register() parameter (which would have to be at the end). It's hacky,
but we expect this feature to be relatively temporary, until we require
ES6 for running any JS at all (probably in about a year).

For distinguishing different types of errors thrown from
sortDependencies(), use e.name. We can't subclass Error properly because
that requires ES6.

Bug: T272104
Change-Id: I45670c910ff12eb422ae54c9fcf372e45c7b2bf1
2021-03-04 14:53:36 -05:00
DannyS712
1567f9bafe Fix extraneous indents in ResourceLoaderTestCase
Change-Id: I5a22337fb1a8098bee526453530c85a8bb18e364
2020-12-18 21:49:07 +00:00
addshore
959bc315f2 MediaWikiTestCase to MediaWikiIntegrationTestCase
The name change happened some time ago, and I think its
about time to start using the name name!
(Done with a find and replace)

My personal motivation for doing this is that I have started
trying out vscode as an IDE for mediawiki development, and
right now it doesn't appear to handle php aliases very well
or at all.

Change-Id: I412235d91ae26e4c1c6a62e0dbb7e7cf3c5ed4a6
2020-06-30 17:02:22 +01:00
Reedy
ead6b1e53f Fix some PSR12.Properties.ConstantVisibility.NotFound in tests/phpunit/
Change-Id: I0f678049dc274f0cd29f543bb293c33da51d8529
2020-05-09 23:55:09 +00:00
Timo Tijhof
69e9b31fc5 resourceloader: Export ResourceModuleSkinStyles as extension attribute
Instead of exporting this as a global variable, export it as an extension
attribute.

The $wgResourceModuleSkinStyles configuration variable will continue
to be supported for its main purpose of setting skin styles. However
it may no longer be used to read these settings. The canonical copy
of this is now restricted to the ResourceLoader class and not (yet)
made publicly available (I found no use cases for it).

This opens the door to making it lazy-loaded attribute, which would help
reduce the size of the "main" APCu cache key for ExtensionRegistry.

This is not possible with global variables as those must be exported
unconditionally from Setup.php.

Bug: T32956
Bug: T247265
Change-Id: I4ecf558d9c630c91959786d2573c34e619223cef
2020-03-30 16:08:00 -07:00
Umherirrender
97ed35e2e8 Add missing @param and @return to documentation
Change-Id: I48ee838a7ebf4f25a4883d4c7886b39c2d3916af
2019-11-16 13:45:55 +01:00
Max Semenik
48a323f702 tests: Add explicit return type void to setUp() and tearDown()
Bug: T192167
Depends-On: I581e54278ac5da3f4e399e33f2c7ad468bae6b43
Change-Id: I3a21fb55db76bac51afdd399cf40ed0760e4f343
2019-10-30 14:31:22 -07:00
Timo Tijhof
9f516f1d3b resourceloader: Reduce width of module hash from 7 chars to 5
In a nut shell:

* We very often (52% of modules on enwiki) pad the hash with a zero,
  which means the amount of bits we currently compute already fit in
  6 characters already for most modules. For some modules (3%) we
  even padded two zeroes.

* For the (now documented) use cases, the space of 78 Giga
  (78 billion, or 78 milliard) seems more than we need. The space of
  60 million should be enough.

  This follows-up dfd046412f from 2016, which previously shortened the hash
  down from 8 chars of base 64 (or 12 chars of hex) to 7 chars of base 32.
  Before that change, the space was 281 Tera (64^8, or 16^12).

For more details see the added inline comment for ResourceLoader::makeHash,
and also the data at <https://phabricator.wikimedia.org/T229245>.

Bug: T229245
Change-Id: I9ad11772a33b3a44cb625275b1d7353e1393ee49
2019-09-02 01:25:48 +00:00
Timo Tijhof
d6dd6e4d72 resourceloader: Remove use of object registering in test suites
This was done as a "clever" shortcut to make sure tests a little
but shorter, but also made them less consistent with normal code.

Remove this in favour of 'class' or 'factory' options as needed.
Also remove a bunch of unneeded register() calls.

The tests cover everything affected by this change.

Side fix - isFileModule should reject modules with 'factory'
the same way it rejected raw objects and non-FileModule 'class'
cases already. This is now covered by tests as well.

Bug: T222637
Change-Id: I3996317dbcd780cc6e0f82c84e769c08a3fc42bb
2019-07-12 01:17:44 +00:00
Timo Tijhof
a72b082811 resourceloader: Document which FileModule methods use a DB
Also, for the unit test, disable the two methods we use there
that can get called. The unintended side-effects of these two
methods was the only reason it used `@group Database`.
Removing that makes the test a bit faster as well.

Enforce this via MediaWikiServices for this suite to avoid an
untracked dependency slipping back in in the future.

Bug: T225730
Change-Id: I6c54466e9517d9899bc39f8f9bb946369c0a526d
2019-06-19 23:44:39 +01:00
Timo Tijhof
469d961bb7 resourceloader: Switch Xml::encodeJsCall call in getDeprecationInformation
Also rename one of the test classes to have a more descriptive name.

Bug: T32956
Change-Id: I1a970c198300b7ef0d99c15609f1fb7fa8783b98
2019-06-19 15:54:08 +01:00
Timo Tijhof
d9a5806940 resourceloader: Convert test for FileModule versionHash to data provider
In prep for the next commit.

Change-Id: If08c3d8b769b6ec03faf8ff24f216ce9f670f6ac
2019-06-12 15:11:18 +01:00
Timo Tijhof
c9318edc2d resourceloader: Warn on ResourceLoader::construct without Config
The only remaining use of 'new ResourceLoader' is in tests, which have
been migrated in this commit to either passing the real config explicitly
(for integration tests), or by passing a HashConfig from a new
'getMinimalConfig' method which has only the keys required for the tests
to pass (e.g. avoid any ConfigExeption for unknown keys).

Also clean up some related code quality issues:

* Migrate wfScript() to $conf->get() so that the local Config is used,
  instead of implicitly using global variables. This isn't deprecated for
  MediaWiki generally, but done here to prepare ResourceLoader for becoming
  a standalone library.

* Remove mocking of 'CacheEpoch' config, this is no longer used anywhere
  in ResourceLoader.

* Change EmptyResourceLoader to use the minimal config by default and
  remove code duplication by calling the parent.

  Update the small number of uses that are integration tests, to explicitly
  pass in the live config as needed. And for the one case that tests the
  'startup' module, it no longer needs to register it manually given this
  is part of ResourceLoader::__construct() by default.

Bug: T32956
Change-Id: I127346fd530fa66f205156e545758b1c29d0fac0
2019-06-10 15:24:45 +00:00
Reedy
9f2ffdfbd4 Remove "Squiz.WhiteSpace.FunctionSpacing" from phpcs exclusions
Change-Id: I78b3315f26ab91b6b443f5b028a635552f82f5a3
2019-05-11 02:44:26 +01:00
Timo Tijhof
b54a1e709e resourceloader: Use 'fallback' as default for ResourceLoaderTestCase
Follows-up fa05976f5f. This broke the Travis CI tests because
the Vector skin is not installed there (unlike in Wikimedia CI).

Make the test no longer reliant on that detail.

* Fix ResourceLoaderTestCase to use a default that actually
  works in plain core (previously 'vector' would be rejected
  as being an unknown skin, but then be re-normalised back to
  'vector' thanks to wgDefaultSkin. This was fixed in fa05976f5f,
  and the unknown skin now produces 'fallback' which is better.

* Update ResourceLoaderOOUIImageModuleTest to match this
  new default and also fix the test message while at it
  to mention 'skin', not 'image'.

Change-Id: I5fb69a6a38a42b1a5f325c0134e01ad880f65087
2019-04-15 20:21:00 +01:00
Timo Tijhof
5c90f8d137 resourceloader: Move registering of custom sources to ServiceWiring
Remove the now-redundant mocking of this configuration from various
test classes.

Bug: T32956
Change-Id: If90a10a76b8289c4ba707382ac915441c17d831f
2019-03-29 01:46:34 +00:00
Timo Tijhof
9a4a754231 resourceloader: Replace ResourceLoaderDebug config use with context
Reduce our reliance on static state and configuration, and
propagate more state in explicit ways, through context, and
request parameters.

OutputPage creates ResourceLoaderContext and ResourceLoaderClientHtml
based on the configuration (via ResourceLoader::inDebugMode).

Everything within those classes should not need to check it
again.

* ResourceLoaderClientHtml:
  Already doesn't check MW config, but it's test was still
  mocking it. Removed now, and confirmed that it passes both
  with true and false. The individual test cases set
  debug=true/false as needed already.

  It's sets were previously relying on the accidental behaviour
  that within a unit test, we don't serialise over HTTP, which
  meant that a pure PHP boolean would survive. With the new
  raw `=== 'true'` check, this no longer works. Set it as a
  string explicitly instead, which is the only thing we support
  outside unit tests as well.

* ResourceLoaderContext:
  Remove fallback to MW config when 'debug' is unset.
  This is never unset in practice given that all load.php
  urls have it set by OutputPage based on ResourceLoader::inDebugMode.

  This change means that manually constructed ad-hoc load.php
  urls that are missing 'debug=' parameter, will now always be
  read as debug=false. This was the default already, but could
  previously be changed through wgResourceLoaderDebug.

When changing wgResourceLoaderDebug, everything will still have
debug=true as before. The only change is when constructing load.php
urls manually and explicitly not set it.

Bug: T32956
Change-Id: Ie3424be46e2b8311968f3068ca08ba6a1139224a
2019-03-08 20:33:16 +00:00
Fomafix
6866cfec37 Simplify PHP by using ?? and ?:
Also remove not necessary surrounding parentheses.

Change-Id: I0eb5c9c1bdfb09a800258379cdcefb5fd4d3d21c
2018-07-10 20:03:17 +00:00
Timo Tijhof
ee8a5e488e resourceloader: Apply safemode to startup module registry
This effectively applies safemode to the mw.loader client,
without the client itself needing specific knowledge of safemode.

Test Plan:
* Unchanged: When viewing a page in safemode, the 'user' and
  'site' modules are still not queued by OutputPage.
* New: mw.loader.getState('site'), previously would yield
  'registered', but will now yield null.
* New: mw.loader.load('site'), previously loaded the module,
  it now logs a dependency warning for unknown module, like for
  any other unknown module.
* New: mw.loader.using('site'), previously resolved, it is now
  rejected.

Bug: T185303
Change-Id: I672e3891c8e1b3c2d13655fa134d0f1d031b8247
2018-05-19 22:14:33 +02:00
James D. Forrester
de6dab71e3 Remove $wgScriptExtension (deprecated and ignored since 1.25)
* Remove left-over mention of the .php5 entry points in docs.

* Remove dead logic in NoLocalSettings for php5 entry points.

* Remove dead match in WebRequest for php5 entry points (they'd
  redirect since 1.25, and not seen by PHP).

Change-Id: Ia0ee8588591860b8fe34030c8503f38e9bce31f3
2018-04-19 01:11:23 +01:00
Umherirrender
14d19eb90a resourceloader: Removed tests for deprecated 'position' parameter
This parameter is a no-op for these tests, as such, nothing was being
tested by specifying it in tests. The behaviour 'test' and 'test.top'
was identical.

Bug: T184257
Change-Id: Ia2bb731f00d4b4175f7b75174aeafaca9412329e
2018-02-12 21:39:05 +00:00
Umherirrender
45da581551 Use ::class to resolve class names in tests
This helps to find renamed or misspelled classes earlier.
Phan will check the class names

Change-Id: Ie541a7baae10ab6f5c13f95ac2ff6598b8f8950c
2018-01-26 22:49:13 +01:00
Umherirrender
f739a8f368 Improve some parameter docs
Add missing @return and @param to function docs and fixed some @param

Change-Id: I810727961057cfdcc274428b239af5975c57468d
2017-09-10 20:32:31 +02:00
Timo Tijhof
3f10aa34e6 resourceloader: Include lessVars in FileModule definition summary
This already worked as expected for any module that uses the new
enableModuleContentVersion model, but for the majority of file modules
this is not yet the case for performance reasons. As such, make
sure lessVars are included in our manual tracking.

Include it conditionally to avoid changing the array for other modules,
which would needlessly invalidate their cache.

Bug: T171809
Change-Id: Ib250068e0ecfc29a09ca33c23bef901ee0482bf2
2017-07-27 03:02:09 +00:00
Brad Jorsch
f6466732bb resourceloader: Add ResourceLoaderModule::shouldEmbedModule and use it
Rather than only the 'private' group triggering embedding, allow modules
to explicitly specify if they should be embedded.

The default is still to only embed when the group is 'private', and the
'private' group is still special in that ResourceLoader::respond() will
still refuse to serve it from load.php.

Change-Id: Ib9a043c566822e278baecc15e87f9c5cebc2eb98
2017-07-21 17:57:34 +00:00
Bartosz Dziewoński
b6a238e6bf resourceloader: Add unit tests for ResourceLoaderImage
Follows-up I5b14d65a and I5a563c59.

Change-Id: Id42e1b868c9fe97cdb14b4bc7328947820a7fd94
2017-04-11 21:55:59 +00:00
Timo Tijhof
466939c631 resourceloader: Don't let module exception break startup
When getScript (or some other method used in a module response)
throws an error, only that module fails (by outputting mw.loader.state
instead of mw.loader.implement). Other modules will work.

This has always been the case and is working fine. For example,
"load.php?modules=foo|bar", where 'foo' throws, will return:

```js
/* exception message: .. */
mw.loader.implement('bar', ..)
mw.loader.state('foo', 'error')
```

The problem, however, is that during the generation of the startup
module, we iterate over all other modules. In 2011, the
getVersionHash method (then: getModifiedTime) was fairly simple
and unlikely to throw errors.

Nowadays, some modules use enableModuleContentVersion which will
involve the same code path as for regular module responses.

The try/catch in ResourceLoader::makeModuleResponse() suffices
for the case of loading modules other than startup. But when
loading the startup module, and an exception happens in getVersionHash,
then the entire startup response is replaced with an exception comment.

Example case:
* A file not existing for a FileModule subclass that uses
  enableModuleContentVersion.
* A database error from a data module, like CiteDataModule or
  CNChoiceData.

Changes:
* Ensure E-Tag is still useful while an error happens in production
  because we respond with 200 OK and one error isn't the same as
  another.
  Fixed by try/catch in getCombinedVersion.
* Ensure start manifest isn't disrupted by one broken module.
  Fixed by try/catch in StartupModule::getModuleRegistrations().

Tests:
* testMakeModuleResponseError: The case that already worked fined.
* testMakeModuleResponseStartupError: The case fixed in this commit.
* testGetCombinedVersion: The case fixed in this commit for E-Tag.

Bug: T152266
Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-15 23:25:57 +00:00
Timo Tijhof
917fa4eded phpunit: Make getResourceLoaderContext() more extendable
This makes it easier to add other options in the future,
such as setting 'modules' in the context to something else.

Change-Id: I53c25fa7ad705cc34e44f95e4f87eb53612d800e
2016-12-03 04:24:47 +00:00
addshore
b12086b618 Get ConfigFactory & MainConfig from MediaWikiServices
Change-Id: Iafdd7e00747060572463ffb05aae4543f3a06163
2016-11-23 00:12:38 +00:00
Timo Tijhof
45bec76755 resourceloader: Don't cache stale responses in mw.loader.store
Follows-up 6fa1e56. This is already fixed for http caches by
shortening the Cache-Control max-age in case of a version mismatch.

However the client still cached it blindly in mw.loader.store.
Resolve this by communicating to the client what version of the module
was exported. The client can then compare this version to the version
it originally requested and decide not to cache it.

Adopt the module key format (name@version) from mw.loader.store
in mw.loader.implement() as well.

Bug: T117587
Change-Id: I1a7c44d0222893afefac20bef507bdd1a1a87ecd
2016-10-10 19:48:25 +00:00
Timo Tijhof
dbe592df6d resourceloader: Fix WikiModule preload to support localised titles
Follows-up 6f8dc27ca2 and dbd11e04aa. You'd expect a bug in preloading
to fallback to fetching it on-demand but due to a title format mismatch
the preload method did use the same title format for the cache key, but
not the same title format for discovering the results. As such, it set
the right cache key to an empty array.

* Make relevant methods testable and mockable.
* Add regression test.
* Change call to array_interect_key to use the same format as
  fetchTitleInfo(): Normalised title keys from Title::getPrefixedText.
  Previously, the intersect used the declared titles from getPages() which
  are not localised - causing an empty array to be returned from the
  intersect on wikis where the namespace name is localised.

Bug: T145673
Change-Id: Ibe788157724d73c727b9e2127b6828db32ca9420
2016-09-14 14:54:01 -07:00
Timo Tijhof
7dbad8562c resourceloader: Improve coverage in ResourceLoaderTest.php
* Fix signature of makeLoaderSourcesScript() to match
  the change in behaviour since e103ba265.

* Consistently order providers before the test.

* Simplify testRegisterValid() and remove needless @depends.

* Remove unused private method stripNoflip().

Coverage:

* Expand test coverage for register().

* Add tests for getModuleNames().

* Add tests for getModule().

* Expand test coverage for addSource().
  (case of invalid array)

* Expand test coverage for makeLoaderImplementScript().
  (case of unwrapped user script, and case of invalid scripts)

* Add tests for makeLoaderSourcesScript().

Change-Id: Ibca3e486fcd3664f171f135327a0f340ee6da9ee
2016-08-24 18:56:20 -07:00
Timo Tijhof
80e5b160e0 resourceloader: Move queue formatting out of OutputPage
HTML formatting of the queue was distributed over several OutputPage methods.
Each method demanding a snippet of HTML by calling makeResourceLoaderLink()
with a limited amount of information. As such, makeResourceLoaderLink() was
unable to provide the client with the proper state information.

Centralising it also allows it to better reduce duplication in HTML output
and maintain a more accurate state.

Problems fixed by centralising:

1. The 'user' module is special (due to per-user 'version' and 'user' params).
   It is manually requested via script-src. To avoid a separate (and wrong)
   request from something that requires it, we set state=loading directly.
   However, because the module is in the bottom, the old HTML formatter could
   only put state=loading in the bottom also. This sometimes caused a wrong
   request to be fired for modules=user if something in the top queue
   triggered a requirement for it.

2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose
   of allowing dependencies on style modules without risking duplicate loading
   on pages where the styles are loaded already. This didn't work, because the
   state information about page-style modules is output near the stylesheet,
   which is after the script tag with mw.loader.load(). That runs first, and
   mw.loader would still make a duplicate request before it learns the state.

Changes:

* Document reasons for style/script tag order in getHeadHtml (per 09537e83).

* Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed
  before since a duplicate check in makeResourceLoaderLink() verified the
  origin a second time.

* Declare explicit position 'top' on 'user.options' and 'user.tokens' module.
  Previously, OutputPage hardcoded them in the top. The new formatter doesn't.

* Remove getHeadScripts().
* Remove getInlineHeadScripts().
* Remove getExternalHeadScripts().
* Remove buildCssLinks().
* Remove getScriptsForBottomQueue().

* Change where Skin::setupSkinUserCss() is called. This methods lets the skin
  add modules to the queue. Previously it was called from buildCssLinks(),
  via headElement(), via prepareQuickTemplate(), via OutputPage::output().
  It's now in OutputPage::output() directly (slightly earlier). This is needed
  because prepareQuickTemplate() calls bottomScripts() before headElement().
  And bottomScript() would lazy-initialise the queue and lock it before
  setupSkinUserCss() is called from headElement().
  This makes execution order more predictable instead of being dependent on
  the arbitrary order of data extraction in prepareQuickTemplate (which varies
  from one skin to another).

* Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early
  on so. This avoids wrongful loading and fixes problem 1.

Effective changes in output:

* mw.loader.state() is now before mw.loader.load(). This fixes problem 2.
* mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading".
* mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1.

* The <script async src> tag for 'startup' changed position (slightly).
  Previously it was after all inline scripts and stylesheets. It's still after
  all inline scripts and after most stylesheets, but before any user styles.
  Since the queue is now formatted outside OutputPage, it can't inject the
  meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle
  of existing output. This shouldn't have any noticable impact.

Bug: T87871
Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
2016-08-08 12:23:09 -07:00
Krinkle
daf49115f9 Revert "resourceloader: Add support for variables in WikiModule"
No longer needed per doing Ic137cb494ba23 in a different way.
This may be useful to revisit, but for now preferring to keep
simplicity and removing this unused option.

This reverts commit 9e217bf42d.

Change-Id: I9c0c316a3b58a3d0a3d3282dd74c7fa4eef8e378
2016-06-09 19:27:34 +00:00
Timo Tijhof
9e217bf42d resourceloader: Add support for variables in WikiModule
To be used for the 'site' module. This will make it easier to split up
'site' into 'site' and 'site.styles' (per T92459).

Change-Id: Iaac3e458d5107e4c10c2826bd64608d5c47e8b87
2016-06-06 19:09:32 +01:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
Timo Tijhof
a4fa0229c1 Add unit tests for MessageBlobStore
* Also added getBlob() and getBlobs() methods. These don't
  require passing ResourceLoader as argument.
  In preparation for T113092.

Change-Id: I3239c45f0243eca1eaf85bc53fca736e4c9a209c
2015-11-21 00:09:47 +00:00
Timo Tijhof
f1c3e9d791 resourceloader: Use 'phpunit' instead of 'test' as default target in PHPUnit tests
Follows-up e9b9e4df7a, which started using target=test as reserved for
Special:JavaScriptTest. However that conflicted with this.

Without this change, the unit tests fail if the MediaWiki install has
$wgEnableJavaScriptTest enabled.

Change-Id: I471017e7aedb74dddb4dc18ee4e89c1706ff09b5
2015-11-17 03:59:02 +00:00
Timo Tijhof
2581453fac resourceloader: Enable module content version for data modules
This greatly simplifies logic required to compute module versions.
It also makes it significantly less error-prone.

Since f37cee996e, we support hashes as versions (instead of timestamps).
This means we can build a hash of the content directly, instead of compiling a
large array with all values that may influence the module content somehow.

Benefits:
* Remove all methods and logic related to querying database and disk for
  timestamps, revision numbers, definition summaries, cache epochs, and more.

* No longer needlessly invalidate cache as a result of no-op changes to
  implementation datails. Due to inclusion of absolute file paths in the
  definition summary, cache was always invalidated when moving wikis to newer
  MediaWiki branches; even if the module observed no actual changes.

* When changes are reverted within a certain period of time, old caches can now
  be re-used. The module would produce the same version hash as before.
  Previously when a change was deployed and then reverted, all web clients (even
  those that never saw the bad version) would have re-fetch modules because the
  version increased.

Updated unit tests to account for the change in version. New default version of
empty test modules is: "mvgTPvXh". For the record, this comes from the base64
encoding of the SHA1 digest of the JSON serialised form of the module content:
> $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}';
> echo base64_encode(sha1($str, true));
> FEb3+VuiUm/fOMfod1bjw/te+AQ=

Enabled content versioning for the data modules in MediaWiki core:
* EditToolbarModule
* JqueryMsgModule
* LanguageDataModule
* LanguageNamesModule
* SpecialCharacterDataModule
* UserCSSPrefsModule
* UserDefaultsModule
* UserOptionsModule

The FileModule and base class explicitly disable it for now and keep their
current behaviour of using the definition summary. We may remove it later, but
that requires more performance testing first.

Explicitly disable it in the WikiModule class to avoid breakage when the
default changes.

Ref T98087.

Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-18 20:39:38 +00:00
umherirrender
d8821f2b0b Fixed spacing
- Removed space after casts
- Removed spaces in array index
- Added spaces around string concat
- Added space after words: switch, foreach
- else if -> elseif
- Removed parentheses around require_once, because it is not a function
- Added newline at end of file
- Removed double spaces
- Added spaces around operations
- Removed repeated newlines

Bug: T102609
Change-Id: Ib860222b24f8ad8e9062cd4dc42ec88dc63fb49e
2015-06-17 20:22:32 +00:00
Andrew Green
7e0d3369c2 resourceloader: Add context param to ResourceLoaderModule::getDependencies
By providing context as a parameter in getDependencies, we allow
modules to dyanamically determine dependencies based on context.
Note: To ease rollout, the parameter is optional in this patch. It is expected
that it will be made non-optional in the near future.

The use case is for CentralNotice campaigns to be able to add special
modules ahead of deciding which banner to show a user. The dynamically
chosen RL modules would replace ad-hoc JS currently sent with some banners.
A list of possible campaigns and banners is already sent as a PHP-
implemented RL module; that's the module that will dynamically choose other
modules as dependencies when appropriate. This approach will save a round
trip as compared to dynamically loading the modules client-side.

For compatibility, extensions that override
ResourceLoaderModule::getDependencies() should be updated with the new
method signature. Here are changes for extensions currently deployed on
Wikimedia wikis:

* CentralNotice: I816bffa3815e2eab7e88cb04d1b345070e6aa15f
* Gadgets: I0a10fb0cbf17d095ece493e744296caf13dcee02
* EventLogging: I67e957f74d6ca48cfb9a41fb5144bcc78f885e50
* PageTriage: Ica3ba32aa2fc76d11a44f391b6edfc871e7fbe0d
* UniversalLanguageSelector: Ic63e617f51702c27104e123d4bed91983a726b7f
* VisualEditor: I0ac775ca286e64825e31a9213b94648e41a5bc30

For more on the CentralNotice use case, please see I9f80edcbcacca2.

Bug: T98924
Change-Id: Iee61e5b527321d01287baa03ad9b4d4f526ff3ef
2015-06-09 03:10:20 +01:00
Timo Tijhof
1e4336d435 resourceloader: Add unit test for validateScriptFile()
Follows-up cd0dff5c00.

Change-Id: Ie208e58053048e932ef3f61f849148b1d88bc0be
2015-06-02 23:50:57 +01:00
Timo Tijhof
f37cee996e resourceloader: Replace timestamp system with version hashing
Modules now track their version via getVersionHash() instead of getModifiedTime().

== Background ==

While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.

For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.

Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.

For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.

More scenarios can be found in the associated Phabricator tasks.

== Version hash ==

Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.

All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.

For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.

As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.

* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule

The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).

== ETag ==

Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).

Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).

Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.

While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).

== File cache ==

If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.

For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.

This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.

== Misc ==

* Remove redundant modifiedTime cache in ResourceLoaderFileModule.

* Change bugzilla references to Phabricator.

* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
  logic was duplicated in each place the modified timestamp was used.

* It's easy to forget calling the parent class in getDefinitionSummary().
  Previously this method only tracked 'class' by default. As such, various
  extensions hardcoded that one value instead of calling the parent and extending
  the array. To better prevent this in the future, getVersionHash() now asserts
  that the '_cacheEpoch' property made it through.

* tests: Don't use getDefinitionSummary() as an API.
  Fix ResourceLoaderWikiModuleTest to call getPages properly.

* In tests, the default timestamp used to be 1388534400000 (which is the unix time
  of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
  modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
  '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'

* Add sha1.js library for client-side hash generation.
  Compared various different implementations for code size (after minfication/gzip),
  and speed (when used for short hexidecimal strings).
  https://jsperf.com/sha1-implementations
  - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
    http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
    Chrome: 45k, Firefox: 89k, Safari: 92k
  - jsSHA <https://github.com/Caligatio/jsSHA>
    https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
    Chrome: 65k, Firefox: 53k, Safari: 69k
  - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
    https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
    Chrome: 200k, Firefox: 280k, Safari: 78k

  Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
  only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
  code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
  needed in the module loader, we'd have to load the fallback regardless. Considering
  this is not used in a critical path for performance, it's not worth shipping two
  implementations for this optimisation.

May also resolve:
* T44094
* T90411
* T94810

Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-05-19 22:28:17 +00:00
Kunal Mehta
870f50d45c resourceloader: Implement '$pages' parameter to ResourceLoaderWikiModule constructor
This makes it easier for subclasses to use ResourceLoaderWikiModule. Currently
many subclasses of this simply need to override the getPages() method.

UserModule and SiteModule keep their getPages override due to the set of pages
being dependent on context.

Change-Id: I388531398671afacfec36c6c5746d72267b5bdac
2015-03-03 17:17:02 +00:00
Kunal Mehta
0e05ec5b31 Don't create Language objects during ResourceLoader tests
Mock calls to ResourceLoaderContext::getDirection(), which creates
Language objects to get the directionality of a language.

Change-Id: Ibe6da3013e658aa7cf596c1da2f8ca1314b7cdd3
2014-12-18 16:52:28 -08:00
Timo Tijhof
9d390a09cd resourceloader: Condition-wrap the HTML tag instead of JS response
Follows-up 9272bc6c47, 03c503da22, 1e063f6078.

One can't wrap arbitrary JavaScript in an if-statement and have
its inner-body mean exactly the same.

Certain statements are only allowed in the top of a scope (such
as hoisted function declarations). These are not allowed inside
a block. They're fine in both global scope and local function
scope, but not inside an if-block of any scope.

The ECMAScript spec only describes what is an allowed token.
Any unexpected token should result in a SyntaxError.

Chrome's implementation (V8) allows function declarations in
blocks and hoists them to *outside* the condition. Firefox's
SpiderMonkey silently ignores the statement. Neither throw a
SyntaxError.

Rgular ResourceLoader responses only contain mw.loader.implement()
and mw.loader.state() call which could be wrapped without issues.
However such responses don't need wrapping as they're only made
by mediawiki.js (in which case mw is obviously loaded). The
wrapping is for legacy scripts that execute in the global scope.

For those, let's wrap the script tag itself (instead of the
response). That seems like the most water-tight and semantically
correct solution.

Had to bring in $isRaw from ResourceLoader.php, else the startup
module would have been wrapped as well (added regression test).

Bug: 69924
Change-Id: Iedda0464f734ba5f7a884726487f6c7e07d444f1
2014-09-09 15:54:16 +00:00
Kunal Mehta
8968d8787f Check page_len in ResourceLoaderWikiModule::isKnownEmpty() for 'user' modules
In most cases, we just check whether the pages exist before saying
the module is not empty to avoid generating cached HTML without
the appropriate <script> or <link> tags.

However, for modules in the 'user' group, normal users cannot
delete their personal JavaScript/CSS pages, causing needless
extra requests, even though we know the pages are empty.

ResourceLoader::isKnownEmpty() now checks the page_len field
for modules in the 'user' group to check that there is
some actual content.

Bug: 68488
Change-Id: I0570f62887fd4642fd60367ae0b51d7dc19488ca
2014-08-30 00:24:37 +00:00