Commit graph

137 commits

Author SHA1 Message Date
jenkins-bot
4bbf8a908d Merge "Move some hooks to ResourceLoader\HookRunner" 2023-05-15 18:38:21 +00:00
Umherirrender
d85f9bd2b0 Move some hooks to ResourceLoader\HookRunner
- ResourceLoaderGetConfigVarsHook
- ResourceLoaderJqueryMsgModuleMagicWordsHook

Change-Id: Ifd8fa776655b347cb539ac824426afb12463148a
2023-05-11 20:51:19 +02:00
Timo Tijhof
85cde7fa8f ResourceLoader: Fix "out of sync" message to ignore $errors case
The conditional here was handling two different cases, exceptions
thrown with $errors, and versions out of sync.

Bug: T321394
Change-Id: I0b861dbac4d497c974244da7e9f444e3683e9032
2023-05-11 18:54:05 +01:00
Umherirrender
e04d3a28f6 Replace internal Hooks::runner
The Hooks class contains deprecated functions and the whole class is
going to get removed, so remove the convenience function and inline the
code.

Bug: T335536
Change-Id: I8ef3468a64a0199996f26ef293543fcacdf2797f
2023-05-11 06:17:38 +00:00
Tim Starling
a59c446154 ResourceLoader: Allow FileModule scripts to be generated with a callback
To implement source maps, we want FileModule::getScript() to return
an array with path information, but that would break subclasses of
FileModule which concatenate to the return value of parent::getScript().
So allow scripts to be generated by a callback, eliminating the need for
concatenation in subclasses.

* Factor out most of the loop body of expandPackageFiles() into
  expandFileInfo(). Add 'name' to the return value. Ensure 'filePath'
  is always a FilePath object since we need that to safely return it
  to ResourceLoader later. Document the return value.
* Make sure the base path is always set in FilePath objects returned by
  expandFileInfo().
* Factor out the loop body of the final stage of file info expansion
  into readFileInfo(). Retain filePath, do not unset it.
* Assert that $fileInfo['content'] is definitely set.
* Convert array_map() in getDefinitionSummary() to a loop.
* Migrate LanguageDataModule.

Bug: T47514
Change-Id: I97d61b5793159cea365740e0563f7b733e0f16de
2023-05-09 11:47:40 +10:00
Timo Tijhof
b2bd5d70df ResourceLoader: Log when MAXAGE_RECOVER is detected
Send a message to Logstash when this condition is reached.
It is expected that this happens for a portion of clients during the
5min after a deployment, such such use an informational message,
not a warning.

This level is already enabled in wmf-config for channel=resourceloader.

I'm adding this now as I'd like to rule out or confirm whether the
our own CDN caching plays a rule in the corruptions reported at
T321394. If so, we can dig deeper there. If not, then we're likely
looking at a problem relating to browser extensions, Wikipedia mirrors,
or user scripts; rather than something under our own control.

Bug: T321394
Change-Id: Iad2f5189da33551b59653c2a6783419d6ad955d0
2023-05-05 03:21:12 +01:00
Gergő Tisza
2e19512ca9 foreign-resources.yaml: Add doc-only type
Add a noop entry type for documenting upstream libraries which we
use in some non-standard way (patched, with a build step etc).

Use it to document jquery.ui (heavily patched). Also fix alphabetic
ordering of the entries.

Bug: T335690
Change-Id: Ieb75f23dfd103ebec8bdf7911d8bb6a0c77bb95d
2023-05-03 19:33:04 +00:00
Tim Starling
234f56e07f ResourceLoader: Use JSON_THROW_ON_ERROR when loading data files
Otherwise content ends up being null, and a less specific exception is
thrown.

Depends-On: Id951da0d4ffa31f4593b2396c806dd6081670d36
Change-Id: Ic2ccd76df70ffba0ed25100925db3dd3086c5e8c
2023-04-28 14:20:50 +00:00
Gergő Tisza
9d26c3f446
ForeignResourceManager: Add transform step
Add a transformation step to foreign-resources.yaml that allows
for simple modifications after the files are fetched and hashed.

This is experimental and mainly meant as a short-term solution
for T335326. In the long term, we might decide to handle the
issue of libraries with ES2017 syntax differently, and this
functionality might be removed. Use at your own risk.

Currently the only supported modification is adding a 'nomin'
filter to the beginning of the file, which is useful for files
which are already minified or which the MediaWiki minifier can't
deal with for some reason. In the future, this could be used
e.g. to apply patches.

Change-Id: I4d93bf6d04683cc4072e99f6ab02d0e618d85914
2023-04-26 14:52:12 +02:00
Gergő Tisza
5099f79718 Add ForeignResourceDirs extension/skin attribute
Add an extension/skin attribute and a command line option for
manageForeignResources.php, for management of an extension's or
skin's foreign resources. This is less clumsy and more
introspectable than each extension/skin creating its own copy
of manageForeignResources.php.

Bug: T330508
Change-Id: I5df95ea4799d0efce60b58a35cb916c8e0f555b0
2023-04-25 09:27:35 -04:00
Bartosz Dziewoński
6ba47296d9 Fix Phan suppressions related to Title::castFrom*() and friends
There is no way to express that Title::castFromPageIdentity(),
Title::castFromPageReference() and Title::castFromLinkTarget()
can only return null when the parameter is null. We need to add
Phan suppressions or explicit types almost everywhere that these
methods are used with parameters that are known to not be null.

Instead, introduce new methods Title::newFromPageIdentity() and
Title::newFromPageReference() (Title::newFromLinkTarget() already
exists), without the null-coalescing behavior, and use them when
the parameter is not null. This lets static analysis tools, and
humans, easily understand where nulls can't appear.

Do the same with the corresponding TitleFactory methods.

Change the obvious uses of castFrom*() to newFrom*() (if there is
a Phan suppression, a type check, or a method call on the result).

Change-Id: Ida4da75953cf3bca372a40dc88022443109ca0cb
2023-04-22 16:45:09 +02:00
Timo Tijhof
656ea1b045 ResourceLoader: Remove Module::getName check before $stats->timing()
Follows-up 2c93649672 (I2b2905d7452bd2), which added the conditional
in order to silently accept case where setName was not called.

getName() is not conditinally accessed elsewhere. It being undefined
would be an error condition on the part of the caller. In this case,
Wikibase forgot to call setName, which I1cc5c54db42ca fixes.

See also ResourceLoader::getModule(), which indicates values that
must be injected in production code. In tests we can sometimes be
lazy and skip a required injection, such as Config and HookContainer,
that we only use in certain methods that perhaps your test doesn't
call. However, upon finding that a test starts exposing such need,
the test need to be updated to provide or mock that value, instead of
tolerating the absence in production code.

This change also fixes a bug in the previous change which would have
theoretically emitted 'all' without the per-module metric, which
would invalidate percentage-based calculations in Graphite as the
total would no longer be equal to the sum.

Depends-On: I1cc5c54db42ca3a9da38ab56cf88cd71dc5b8343
Change-Id: I3c9b7a17b1331750c631b8dc3e9e7ee04d79884f
2023-04-14 08:34:07 +00:00
jenkins-bot
5b4f212c5d Merge "ResourceLoader: Add structure test for ResourceLoader::respond()" 2023-04-13 04:15:57 +00:00
jenkins-bot
0856164ba6 Merge "ResourceLoader: Avoid new use of MWException" 2023-04-11 15:13:26 +00:00
jenkins-bot
222bcd8081 Merge "ResourceLoader: Descriptive error for Less imports from codex-design-tokens" 2023-04-11 02:04:01 +00:00
Timo Tijhof
6ef84a7218 ResourceLoader: Avoid new use of MWException
Follows-up I992ccde79a59ad51c.

* Soften the test to only asserting that an exception is thrown,
  the type isn't part of the API.

* Throw RuntimeException instead of MWException per T86704.

Bug: T86704
Bug: T328602
Change-Id: Ifaa5d659941d60789ca14f771648e5830bbef317
2023-04-10 18:47:45 -07:00
Tim Starling
05c54baa36 ResourceLoader: Add structure test for ResourceLoader::respond()
Confirm that load.php can respond for all registered modules without
errors (except private modules, which can't we don't serve there).

Bug: T47514
Change-Id: I44655b2c05529ae719d71622f57bfed8d632550e
2023-04-10 18:18:54 -07:00
Anne Tomasevich
3c2061497e Create separate ResourceLoader modules for Codex styles
This patch removes styles from the `@wikimedia/codex` and
`@wikimedia/codex-search` modules, and moves them into new,
separate modules. This will enable use of CSS-only Codex
components within MediaWiki.

Additionally, this patch adds a check for theme styles in the
CodexModules class to prevent it from breaking if they're
not provided.

Bug: T330466
Change-Id: I259a706641fcd51787d7492e7ada11db78277659
2023-04-05 15:29:39 -04:00
Tim Starling
25d0d37adc ResourceLoader: tweak comments and error messages
Change-Id: Ifcaefedf65b090a87ec1417808277ae6451302dc
2023-04-03 18:21:30 +10:00
Timo Tijhof
3e2d1921aa ResourceLoader: Improve makeLoaderImplementScript() docs
* Document `$scripts` types in a list for clarity, and make the
  end-to-end connection to what these are for in practice (especially
  for debug mode, and for site/user scripts).

* Fix outdated `$styles` docs. This has changed long ago.

* Remove support for $messages array. This existed solely to pass
  an empty value for cases where messages are not needed.
  Change that to null instead.

Change-Id: I0c0ef94d830171a3dd8588de8c4a1f3d67607d41
2023-03-31 21:58:14 -07:00
jenkins-bot
8bb1e60724 Merge "ResourceLoader: Raise MW JavaScript startup requirement to ES6" 2023-03-31 01:18:54 +00:00
Roan Kattouw
aaecb49e9b ResourceLoader: Descriptive error for Less imports from codex-design-tokens
Importing Less files from @wikimedia/codex-design-tokens/ doesn't work,
on purpose. Instead of letting these imports fail with a nondescript
"file not found" error that doesn't help the developer understand why it
didn't work, provide a descriptive error message that points them in the
right direction.

Bug: T328602
Change-Id: I992ccde79a59ad51c7ebfe3ac7548a6e531f4a59
2023-03-30 16:19:38 -07:00
Roan Kattouw
0d96067d46 ResourceLoader: Use str_starts_with() for Less import paths
Follow-up to I2df266cde90d1b8dad8d1d1367b67001e2b2984d

Change-Id: I983df7c558473519297d745694452ffde1a07139
2023-03-30 09:53:24 -07:00
James D. Forrester
6c7177349b ResourceLoader: Raise MW JavaScript startup requirement to ES6
The UA sniffs that overrode the feature tests are no longer needed.

* MSIE 10: Fine, rejected by feature checks.

* UC Mini "Speed Mode": Redundant, the version that this sniff
  matched is pre-ES6. Current versions of UC Mini don't appear to
  support enabling "Speed Mode" on random websites nor does it offer
  it for Wikipedia specifically.
  Details at https://phabricator.wikimedia.org/T178356#8740573.

* Google Web Light: Redundant, shutdown as of 2022.
  Any references or extensions that still reach the proxy, get
  redirected to our online URLs
  https://googleweblight.com/?lite_url=https://en.m.wikipedia.org/wiki/Banana
  https://phabricator.wikimedia.org/T152602
  https://en.wikipedia.org/wiki/Google_Web_Light

* MeeGo: Redundant, discontinued and presumed rejected.
  Either way, unsupported.

* Opera Mini: Fine, rejected by checks.
  Details at https://phabricator.wikimedia.org/T178356#8740573.

* Ovi Browser: Redundant, discontinued and presumed rejected.
  Either way, unsupported.

* Google Glass: Improve UX (since 2013, T58008).

* NetFront: Redundant. Old versions are presumed rejected.
  Current versions are Chromium-based and presumed fine.
  The exclusion was not UX based, but due to jQuery explicitly not
  supporting it in 2013. This is no longer the case, so we can let
  the feature test lead the way here.

* PlayStation: Redundant, same story as NetFront.
  The version that matched the sniff is presumed rejected.
  Current versions probably fine, but even not, don't match
  our sniff so are already enabled today.

Bug: T178356
Change-Id: Ib6263ce3ffd11af5e501de8857f3e48a248c6210
2023-03-30 14:26:05 +01:00
jenkins-bot
6fdeb63eb8 Merge "ResourceLoader: Add path remapping for Less imports" 2023-03-29 23:47:38 +00:00
Timo Tijhof
102f184cc0 ResourceLoader: Strengthen ForeignResourceManager cache key
It is currently possible to update the source URL of a package
in foreign-resources.yaml without updating the integrity hash (
or updating it but forgetting to stage/commit it). And yet CI would
still pass if the URL ends with an unversioned filename, because
the cache key would remain unchanged and thus CI would not download
the URL.

Change-Id: I04d77c438b7b083b1981f92cf6ddeaae0633ec86
2023-03-28 17:28:54 -07:00
Roan Kattouw
7492234847 ResourceLoader: Add path remapping for Less imports
This replaces the hacky wrapper files in
resources/src/mediawiki.less/mediawiki.skin.codex-design-tokens/ and
resources/src/medawiki.less/@wikimedia/codex-icons/dist/ with real
import path aliasing/remapping.

Bug: T328602
Change-Id: I2df266cde90d1b8dad8d1d1367b67001e2b2984d
2023-03-28 16:18:44 -07:00
Tim Starling
be3018b268 Just another 80 or so PHPStorm inspection fixes (#4)
* Unnecessary regex modifier. I agree with this inspection which flags
  /s modifiers on regexes that don't use a dot.
* Property declared dynamically.
* Unused local variable. But it's acceptable for an unused local
  variable to take the return value of a method under test, when it is
  being tested for its side-effects. And it's acceptable for an unused
  local variable to document unused list expansion elements, or the
  nature of array keys in a foreach.

Change-Id: I067b5b45dd1138c00e7269b66d3d1385f202fe7f
2023-03-25 00:39:06 +00:00
Tim Starling
317b460500 Fix even more PHPStorm inspections (#3)
* Inappropriate @inheritDoc usage. Arguably all @inheritDoc is
  inappropriate but these are the ones PHPStorm flags as misleading
  due to the method not being inherited.
* Doc comment type does not match actual argument/return type.
* I replaced "@return void|never" with "@return void" since never means
  never, it doesn't make sense for it to be conditional. If a method
  can return (even if that is unlikely) then @return contains the type
  that it returns. "@return never" means that there is no such type
  because the method never returns.
* Incomplete/partial/broken doc tags

Change-Id: Ide86bd6d2b44387f37d234c2b059d6fbc42ec962
2023-03-25 00:30:15 +00:00
Timo Tijhof
89c17acbf6 ResourceLoader: Silence failure of encodeJson() for $modules param
The $modules param is user controlled. We already catch invalid UTF-8
in server configurations. If something like that makes it here, the
only place it can come from is the client doing it intentionally,
such as the result of scanning tools.

Follows-up I89832142b, which silenced $modules formatting in a CSS
response. This patch also covers the case of a JS response, and adds a
test for both.

Bug: T331641
Change-Id: I03a0ec262203509f069654e7fa6fd8d8debd25d5
2023-03-23 11:20:59 -07:00
Aaron Schulz
29bab859fc profiler: Add ProfilingContext class
Use this class to track the entry point and handler used for requests,
making it available for use in profiling, stats, and logging code.

This makes it possible for periodic and/or shutdown profiling callbacks
to know the basic action handler that applies to the request (if any).
Metric names can easily include this string along with MW_ENTRY_POINT
to create per-action profiling dashboards.

This info cannot otherwise be acquired from things like excimer stack
traces since the router and handler classes do not appear in the stack
during PRESEND deferred updates and variations like ApiMain/SpecialPage
"inclusion mode" would have to be detected somehow.

Bug: T330810
Change-Id: Icca5a7a343faeeb18652994c96752acb61a61fd1
2023-03-23 00:08:49 +00:00
Tim Starling
819efb3234 Don't write to the source tree in ForeignResourceManagerTest
* Wait until we know the action before choosing the temp directory.
  Use the configured temp directory if the action is verify, since the
  arguments for using a subdirectory of the source tree do not apply on
  verify.
* Use the configured cache directory in precedence over the source tree.

Change-Id: I0431db332238c8e2de14ec71fb611e0a8d4e959b
2023-03-17 20:45:05 +11:00
Timo Tijhof
02b4671dbf ResourceLoader: Silence new encodeJson warning for $states array
Follows-up I89832142b55ec0f for T329330. encodeJson() is almost always
called with data that a core component or extension was responsible
for processing or generating. There is one exception that I know of,
namely within ResourceLoader itself, where we report back the name of
a load.php?modules parameter value if it doesn't exist or is invalid.

In that case, silence the newly added warning from encodeJson() since
the audience for that error is not the developers working on the code,
but the user making a badly formed URL. And the warning was coming
from the exact code formulating that same error message.

Bug: T331641
Change-Id: Idcccb979c89ef837546892ab786d803d429ecda5
2023-03-10 19:00:05 +00:00
jenkins-bot
a37c1d0b4a Merge "ResourceLoader: Make internal encodeJsonForScript private" 2023-03-07 20:43:41 +00:00
Timo Tijhof
1ece7ee097 ResourceLoader: Make internal encodeJsonForScript private
Has been marked `@internal` since initial commit, with no references
outside this class in Codesearch Everywhere. The method relies on global
state in the form of $wgRequest, via ::inDebugMode.

New code can (and does) uses RL\Context::encodeJson() instead.

Bug: T32956
Change-Id: Ia5f3da26f4ee6dd5f49eb1869653fa7c3ce90e3a
2023-03-06 23:49:23 +00:00
Timo Tijhof
7ac1439211 ResourceLoader: Remove makeMessageSetScript() method
This was originally an internal method for ResourceLoader to create
the load.php?only=messages response. This was removed in 2015 with
change Ia6c87d687c6 (9b6ee1da59) in MW 1.26. After that it was
briefly used by RCFilters until change I2d58f55701 (2dd6342b64)
in 2018.

It has no known use in Codesearch Everywhere, no test coverage,
and relies on global state (including $wgRequest, via ::inDebugMode).

Bug: T32956
Change-Id: I49d71aa319c571cfefc9bcb459cd97047eb6d839
2023-03-06 01:53:13 +00:00
James D. Forrester
ad06527fb4 Reorg: Namespace the Title class
This is moderately messy.

Process was principally:

* xargs rg --files-with-matches '^use Title;' | grep 'php$' | \
  xargs -P 1 -n 1 sed -i -z 's/use Title;/use MediaWiki\\Title\\Title;/1'
* rg --files-without-match 'MediaWiki\\Title\\Title;' . | grep 'php$' | \
  xargs rg --files-with-matches 'Title\b' | \
  xargs -P 1 -n 1 sed -i -z 's/\nuse /\nuse MediaWiki\\Title\\Title;\nuse /1'
* composer fix

Then manual fix-ups for a few files that don't have any use statements.

Bug: T166010
Follows-Up: Ia5d8cb759dc3bc9e9bbe217d0fb109e2f8c4101a
Change-Id: If8fc9d0d95fc1a114021e282a706fc3e7da3524b
2023-03-02 08:46:53 -05:00
Reedy
cd29104a8b ResourceLoader: Check for valid SPDX license in ForeignResourceManager
Bug: T330428
Change-Id: Iddccef5d3c6a1baf89a6c626671323b6eed49c42
2023-03-01 20:11:31 +00:00
Amir Sarabadani
e8032eb8f7 ResourceLoader: Switch to use select query builder in WikiModule
Bug: T311866
Change-Id: I77d711a10b935680682eb2a91dab635460de3cf6
2023-02-28 17:23:58 +01:00
Amir Sarabadani
4bb2886562 Reorg: Migrate WikiMap to WikiMap/ out of includes
And WikiReference

Bug: T321882
Change-Id: I60cf4b9ef02b9d58118caa39172677ddfe03d787
2023-02-27 05:19:46 +01:00
Timo Tijhof
1f5b4764d5 qunit: Enable debug=2 by default for SpecialJavaScriptTest
Bug: T85805
Bug: T250045
Change-Id: Ic417ac6b5bcfd1a3ca9c842e84335a822e1c0435
2023-02-21 21:59:23 +00:00
Amir Sarabadani
7d8768e931 Reorg: Move HTML-related classes out of includes/ to Html/
Bug: T321882
Change-Id: I5dc1f7e9c303cd3f5b9dd7010d6bb470d8400a18
2023-02-16 20:40:01 +01:00
Timo Tijhof
2ab050f67b ResourceLoader: Permit partial failure of encodeJson() and add logging
Similar to the exception thrown by ResourceLoader::makeConfigSetScript,
but non-fatal so as to generally allow more things to continue working
without it being silent and without giving up visibility through
Logstash.

Bug: T329330
Change-Id: I89832142b55ec0f4d1168314a897624e13ac8a0f
2023-02-14 23:40:20 +00:00
jenkins-bot
6494f7b6dd Merge "Remove unused arguments to private functions" 2023-02-08 22:42:46 +00:00
Umherirrender
ed169d991e Remove unused arguments to private functions
Found by phan dead detection

Change-Id: I93379b7b9a733206d0e53add04fcdb9478c58755
2023-02-08 19:00:47 +00:00
jenkins-bot
b4e039dc60 Merge "ResourceLoader: Remove SVG fallback hack from SkinModule" 2023-02-08 13:14:02 +00:00
jenkins-bot
c3e74c2f99 Merge "ResourceLoader: Remove SVG fallback hack from ImageModule" 2023-02-08 13:13:55 +00:00
Bartosz Dziewoński
d3c64c5ce6 ResourceLoader: Remove SVG fallback hack from SkinModule
Bug: T329127
Change-Id: I28c02efc137c3a9ceacdc07366ac374c70b0fb7e
2023-02-08 00:32:52 +00:00
Bartosz Dziewoński
564d4fe7a5 ResourceLoader: Remove SVG fallback hack from ImageModule
Bug: T329127
Change-Id: I0bf50e2d2e0a3ba0ba4c8fceb19cd2ae96d8d9fb
2023-02-08 01:32:07 +01:00
Roan Kattouw
f326fc7f0d skins: Allow skins to specify their Codex theme
Add a "SkinCodexThemes" key to skin.json, that works just like
"SkinOOUIThemes". Based on the value the skin sets,
ResourceLoader\CodexModule loads the correct Codex theme for the skin.
The default theme is wikimedia-ui, meant for skins with a 16px base font
size. Skins with a 14px base font size (like Vector) should override
this to use wikimedia-ui-legacy.

Bug: T324367
Change-Id: I7435413928f2b8c5d43c34d9e2e7df470d710c93
2023-02-06 10:27:43 -08:00