Commit graph

166 commits

Author SHA1 Message Date
Petr Pchelko
b1a30eb0c4 Hard-deprecate all public property access on CacheTime and ParserOutput.
Bug: T263851
Change-Id: I3d3ff7b5b6899150df836e445b56896dfd5b887e
2020-11-19 10:12:41 -07:00
Petr Pchelko
b956c77d27 Merge CacheTime and ParserOutput accessedOptions properties
Change-Id: I5785596d68e8923f8bcbd182ace0b1991bd75c9a
2020-11-19 10:12:39 -07:00
Petr Pchelko
dbdc2a3cd3 Introduce JsonCodec to help with serialization/deserialization
Change-Id: I5433090ae8e2b3f2a4590cc404baf838025546ce
2020-11-19 08:32:21 -07:00
jenkins-bot
d009d75c73 Merge "Start generateSchemaChangeSql.php" 2020-11-16 08:40:36 +00:00
Petr Pchelko
7c68ae9296 Safe ParserOutput extension data and JsonUnserializable helper.
One major difference with what we've had before is that now we
actually write class names into the serialization - given that
this new mechanism is extencible, we can't establish any kind
of mapping of allowed classes. I do not think it's a problem
though.

Bug: T264394
Change-Id: Ia152f3b76b967aabde2d8a182e3aec7d3002e5ea
2020-11-10 11:21:09 -07:00
Petr Pchelko
017cfcf016 Forward-compat for merging CacheTime and ParserOutput mOptions
CacheTime::mUsedOptions and ParserOutput::mAccessedOptions
do exactly the same thing and has to be merged into a single property.
This patch adds forward-compatibility and needs to be deployed
at least one train before the patch which actually merges the properties.

Change-Id: Ic9d71a443994e2545ebf2a826b9155c82961cb88
2020-11-10 07:09:41 -07:00
daniel
cac89b547c ParserOutput: add support for binary properties in JSON.
This introduces a mechanism for encoding binary data in
strings set via setProperty(). This is needed to accommodate compressed
data as used by TemplateData, which uses gzip compression to make the
data fit into the page_props table.

Bug: T266200
Change-Id: I19fa0dea8c25d93fcdec9dc5ddd6f3c9c162b621
2020-11-04 18:52:09 +01:00
Amir Sarabadani
1125167b8d Start generateSchemaChangeSql.php
Creating schema changes from abstract schema and even abstracting a
schema change.

Bug: T230420
Change-Id: If626e866642af820dd70c5f9b0fe7c6a951e0a25
2020-11-01 21:14:53 +01:00
Ammar Abdulhamid
d095b56e25 Expand DoctrineSchemaBuilderTest
* Test the generated table for each platform
* Use the handy dirname() function instead of full raw string path

Bug: T191231
Depends-On: If344395615087c360597a5b3d66ea03e930b7d9b
Change-Id: I936c5d10924a46743c7f76df8526105eec7ff48d
2020-10-17 21:17:46 +00:00
daniel
600f64029f Use JSON for parser cache
This adds JSON serialization and deserialization capabilities
to CacheTime and ParserOutput.

NOTE: JSON serialization is disabled for now. Merging this patch
should not change behavior in production.

Bug: T263579
Change-Id: I18187e8bce573d21f6f1bd29106e07c63a6d2f4d
2020-10-13 16:28:52 -07:00
Petr Pchelko
8a879605d9 Add deserialization acceptance tests for ParserOutput
Bug: T264397
Change-Id: I6476fd9b8eff0e1b61ce5f43280d1cd9b7aaa77c
2020-10-12 08:55:32 +00:00
daniel
6eea7d7ed5 Add test infra for ParserCache serialization/deserialization
Based on Daniel's work at Ia6e70179b7ee5ce4e93888585ccc30d92da165c3
however was changed enough to move into a separate changeset.

More acceptance tests and data will be added in a followup commit.

Bug: T264397
Change-Id: I135187e83cbfa02b97c5656f0752f8bf1ceb58d0
2020-10-09 08:14:57 -06:00
Timo Tijhof
0c01d8cc52 resourceloader: Add skin-based 'mediawiki.skin.variables.less' import
Add the SkinLessImportPaths attribute for skin-specific LESS import
paths, which skins can use to override the mediawiki.skin.variables.less
file.

As a starting point, add the following 5 variables:

* device widths (3x)
  To help phase out 'mediawiki.ui/variables'. These are
  commonly used by MobileFrontend.

* @font-family-sans
  Recommended by Volker. Used by multiple skins.

* @border-radius-base
  Recommended by Volker as example of something that we currently
  hardcode in MediaWiki core for Vector and OOUI/WikimediaUI
  in 'mediawiki.widgets.datetime' but should instead be allowed
  to vary by skin and OOUI theme.

  Remove the hardcoded value for '@border-radius-base' in
  various places in favour of importing from mediawiki.skin.
  The default is a bare default of 0 (as border-radius is off
  by default in the browser).

  The value for Vector is restored there by I47da304667811.
  The value for MonoBook is improved by I000f319ab31b.

Bug: T112747
Change-Id: Icf86c930a3b5524254bb549624737d3b9dccb032
2020-09-15 10:05:53 +00:00
David Causse
01eada74ef Write category parent hierarchy when handling categorization
Since we do brute force update (DELETE ALL+INSERT ALL) of the parent
categories we need to always write the parent category hierarchy.

Bug: T246568
Change-Id: Id9c3282a97bf3fc3d5355cc08a0c9d833c72c9b2
2020-07-31 12:14:13 +02:00
David Causse
6cbe99618e categoryChangesAsRdf: Add test for categorization
The test is bit slow as it has to propagate the categorisation
changes through several jobs. Users have suffered multiple regressions
on this script that it might be worth the additional cost.

Change-Id: I666d0039cfe4ba4e2a17210e8070f836767598a1
2020-07-31 12:14:11 +02:00
Cindy Cicalese
c69489f44c Check for unknown slot.
Bug: T220525
Change-Id: I89ac1679ea70cb403433f8b97fe11e8eb6811280
2020-07-20 10:39:28 -04:00
daniel
7ffacacc6b MCR import
Bug: T220525
Change-Id: I7d41a8af12e33aed802526eb39a0880292c4f534
2020-07-18 20:34:16 -04:00
daniel
cac44ca333 Add import/export round trip test.
This creates the infrastructure we want to use later to test
improt and export of multiple slots.

Bug: T220525
Change-Id: I8e02927bd5532fe9cee0694d48a4c7d9ec060ba1
2020-07-14 15:55:55 +02:00
Roan Kattouw
8c6c1c9adc VueComponentParser: Use RemexHtml instead of PHP's HTML parser
This fixes an issue with HTML tags inside the <script> tag.
Remex also doesn't throw errors on attributes like @click, although it
does mangle them when producing DOM. To work around this, don't use DOM
serialization for the template HTML, but parse everything again using a
Remex parse+serialize pipeline that extracts the template and
(optionally) removes comments and strips whitespace.

One important effect of this change is that we'll have to forbid
self-closing tags in Vue templates, because Remex doesn't handle those
correctly (or rather, handles them *too* correctly). But on the up side,
we can now allow shorthands for v-bind/v-on/v-slot again.

Bug: T253334
Bug: T255587
Depends-On: I2253a2317187fe0d781ba5bfefab95e0f97d0a80
Change-Id: Id9a9728b7163601cc60bc587be07b70977d41970
2020-06-22 19:54:15 +00:00
Tim Starling
b9907ca9b6 Remove DatabaseIntegrationTest::testStoredFunctions()
As was pointed out in CR on the original change from December 2011, the
test requires elevated privileges when it is run against a database with
binlogging enabled, which is a nuisance when I'm trying to test replication-
related code. The author commented that the test was to support GeoData,
but that was only true until the "schema revamp" of February 2012.

I considered mocking the database and making it into a test of
streamStatementEnd, since that code was introduced at the same time, but
we already have DatabaseMysqlBaseTest::testStreamStatementEnd().

I also removed the restoreFlags() call since that was apparently left
over from testFlagSetting(), which has been moved to DatabaseTest.

Refs:

* https://www.mediawiki.org/wiki/Special:Code/MediaWiki/107376
* https://www.mediawiki.org/wiki/Special:Code/MediaWiki/108603
* https://www.mediawiki.org/wiki/Special:Code/MediaWiki/110649
* 67f08d6990

Change-Id: I1a6b9d0cd91c8539ac1080a423b519743088733e
2020-06-02 10:47:29 +10:00
Amir Sarabadani
dbbb3af6dc Small improvements to DoctrineSchemaBuilder and generateSchemaSql
Adding scope to const, adding docstring, adding comments to generated
files, fix default path.

Bug: T230428
Change-Id: I955a446421ffe5c8fa42aeedc0f962230e92e02a
2020-05-09 16:08:44 +02:00
Reedy
26cf0fc2e8 tests: Add test file to test SPDX license expressions
Bug: T251642
Change-Id: I6950ab0c9d0ef080dbe247f5b73458aac4d27f8a
2020-05-02 12:47:31 +01:00
jenkins-bot
fbf4346a10 Merge "build: Provide linting for .vue files from upstream" 2020-04-16 01:11:55 +00:00
Roan Kattouw
bed84e3281 build: Provide linting for .vue files from upstream
Was first attempted in f1eaa06d5e,
and reverted in 0b3fe4fe8c. Now this
is done in eslint-config-wikimedia for all repos.

Change-Id: I9ee735eef40085b6373b565838d67e186e438657
2020-04-15 17:40:31 -07:00
daniel
5a58e4421c Make dump tests based on fixture files.
This adds test coverage for dumping non-main slots.

NOTE: This invents a syntax for XML file "templates" that are
expected to "match" the actual output file.

Bug: T240213
Change-Id: I05e46175809c4562f3530377fe37db911bee244a
2020-04-06 15:21:00 -07:00
Timo Tijhof
75ccdc6147 languages: Move default $wgNamespaceAliases to MessagesEn.php
These are not configuration but business logic, similar to the
canonical names that are in NamespaceInfo.php, these must always
exist and cannot be altered or unset.

They were previously unconditionally assigned during all requests
in Setup.php and passed down as "site configuration".

Changes:

* Move them to MessagesEn.php where they can be cached and
  processed the same way as other core-provided aliases.

  Document and confirm with tests that this is a mergeable
  attribute that follows the language chain.

* Remove the duplicated code in a few places that was reading
  this variable + Language::getNamespaceAliases(), to instead
  just call the latter and move the logic there, centralised,
  and tested.

  In doing so I noticed that these were applied in an
  inconsistent order. Sometimes the config won, sometimes not.
  There's no obvious right or wrong way here, but I've chosen
  to standardise on the way that Language::getNamespaceIds() did
  it, which is that config wins. This because that method seems
  to be most widely used of the three (it decides how URLs and
  titles are parsed), and thus the one I least want to change
  the behaviour of.

* Document that $wgNamespaceAliases may only be used to
  define (extra) aliases, it is and never was a way to access
  the complete list of aliases.

Bug: T189966
Change-Id: Ibb14181aba8c1b509264ed40523e9ab4000fd71a
2020-03-14 19:27:40 +00:00
Roan Kattouw
ca46126e98 resourceloader: Support single-file component .vue files
Allows .vue files to be used in package modules as if they were .js
files: they can be added to the 'packageFiles' array in module
definitions, and require()d from JS files.

In the load.php output, each .vue file is transformed to a function that
contains the JS from the <script> tag, then a line that sets
module.exports.template to the contents of the <template> tag (encoded
as a string). The contents of the <style> tag are added to the module's
styles.

Internally, the type of a .vue file is inferred as 'script-vue', and the
file is parsed with VueComponentParser, which extracts the three parts.
After the transformation, the file's type is set to 'script+style', and
files of this type contribute to both getScript() and getStyles().

This change also adds caching to getPackageFiles(), because it now needs
to be called twice (in getScript() and getStyles()).

Change-Id: Ic0a5c771901450a518eb7d24456053956507e1ed
2020-03-12 14:32:41 -07:00
Roan Kattouw
f8861b7622 TemplateParserTest: Add test case with syntax error
Change-Id: Id4860269e55345176ad58fd75dcf3522a8617fa2
2020-01-29 23:15:53 +00:00
jenkins-bot
9318f46267 Merge "Improve tests in DoctrineSchemaBuilder" 2020-01-06 11:00:54 +00:00
Kunal Mehta
d6390ba1ee registration: Implement lazy-loaded attributes
These attributes are only accessed at a time when performance isn't
critical (e.g. tests), so we can skip caching them and read each
extension.json file again when we do need them.

At the same time, we want to reduce the size of the array that we're
caching due to PHP 7 performance regressions with APC (T187154).

'QUnitModules' is the first attribute to be converted into a lazy-loaded
attribute.

Bug: T220994
Change-Id: I224cdace05dbac00ff947c41daecb74b07c967bb
2019-12-15 19:04:50 +00:00
Timo Tijhof
3e6db51725 resourceloader: Add tests for module-level 'noflip' option
Also widen up @covers for the integration tests to let the
coverage percentage increase from those.

Change-Id: Ib9d42b124774b4c968c52f34e587059c63e8ffc2
2019-12-03 15:58:36 +00:00
Amir Sarabadani
6d4dc90831 Improve tests in DoctrineSchemaBuilder
Also changing its structure to be more readable and more like
https://w.wiki/CXo

My plan is to add more tables to this test.

Bug: T230428
Change-Id: Ia19d5c875cc95238f7fd7cd9adbed1ddb92af078
2019-11-22 19:53:10 +01:00
Max Semenik
32df52aa37 ComposerInstalledTest: remove mentions of phpunit-mock-objects
We're not using it anymore, so removing it from unrelated tests to
prevent it from appearing in searches.

Change-Id: I83550477faea36f7d739e91fb449e0ee3f07ea18
2019-11-21 12:59:45 -08:00
Fomafix
d607ef8512 Remove traling double newlines from text files
Performed using:

git ls-files \*.json \*.php \*.xml \*.xsd \*.html \*.txt \*.sql | xargs sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba'

Excluded tests/parser/preprocess.

Change-Id: I513538ff13c942acfe589f15c5128a8df2b3ace9
2019-10-23 21:17:43 +02:00
Max Semenik
dc3fba4619 tests: Nuke everything HHVM-related
Change-Id: Ieccce5ebba25132b2433fc13a575b65a5bd538f2
2019-10-05 12:51:07 -07:00
Derk-Jan Hartman
9ce26a564d Fix XMP parser errors due to trailing nullchar
JPEG files can have trailing \0 chars at the end of the XMP value. Use
trim() to remove these from the string value.

Bug: T118799
Change-Id: Id4ab223ef432e5d2c0dd3b4e332320db02422700
2019-09-11 22:34:33 +00:00
Stanislav Malyshev
af8e2632ad Fix categories detele SPARQL clause
Bug: T228348
Change-Id: I87b3b5831e31dc619b5eefb7854d3223acd605fc
2019-08-27 16:19:31 -07:00
Tim Starling
d8e06a46a8 MimeAnalyzer: fix ZIP parsing failure
unpack() actually returns an array with indexes starting from 1, not
zero, so unpack(...)[0] gives a notice and always returns null. It is
lucky that ZIPs normally have zero-length comments, so this would have
had little impact on file type detection aside from log spam.

Also, add a check to make sure the unpack() will not read beyond
the end of the file. Without this, unpack() could generate a warning.

The bug was introduced by me in f12db38048.

Add tests. The test files were generated by appending an EOCDR signature
and some extra bytes to 1bit-png.png.

Bug: T223728
Change-Id: I6fab63102d1d8eea92cdcce5ab6d1eb747a0a890
2019-07-25 13:40:18 +10:00
Bartosz Dziewoński
77f4bab8b7 Allow skins/extensions to define custom OOUI themes
This change follows I39cc2a735d9625c87bf4ede6f5fb0ec441d47dcc.

docs/extension.schema.v1.json
docs/extension.schema.v2.json
includes/registration/ExtensionProcessor.php
* The new extension attribute 'OOUIThemePaths' can be used to define
  custom OOUI themes. See I9187a63e509b601b8558ea82850fa828e5c8cc0a
  for an example usage.

includes/resourceloader/ResourceLoaderOOUIModule.php
* Add support for 'OOUIThemePaths'.
* Defining 'images' is now optional. I figure custom themes are
  unlikely to have or need them.
* Use ResourceLoaderFilePath objects to allow skin-/extension-defined
  OOUI module files to use skin/extension's base paths.
  This was previously used to support $wgResourceModuleSkinStyles,
  but only for 'skinStyles' - now ResourceLoaderFileModule needs
  to also handle it for 'skinScripts', and ResourceLoaderImageModule
  for 'images').

includes/resourceloader/ResourceLoaderFilePath.php
* Add getters for local/remote base paths, for when we need to
  construct a new ResourceLoaderFilePath based on existing one.

includes/resourceloader/ResourceLoaderFileModule.php
includes/resourceloader/ResourceLoaderImageModule.php
includes/resourceloader/ResourceLoaderOOUIImageModule.php
* Add or improve handling of ResourceLoaderFilePaths:
  * Replace `(array)` casts with explicit array wrapping, to avoid
    casting objects into associative arrays.
  * Use getLocalPath() instead of string concatenation.

tests/phpunit/includes/resourceloader/ResourceLoaderFileModuleTest.php
tests/phpunit/includes/resourceloader/ResourceLoaderImageModuleTest.php
* Some basic checks for the above.

Bug: T100896
Change-Id: I74362f0fc215b26f1f104ce7bdbbac1e106736ad
2019-07-10 22:08:14 +02:00
Brion Vibber
2e83387666 Relax HTML sniffing checks on image upload
Allows uploaded files to include some HTML tag strings that were
previously forbidden in the first 1k or so of the file:
* <a href
* <img
* <pre
* <table
* <title

They are now allowed as long as the IE MIME type detection heuristic
would not change their types. This should reduce the number of false
positive checks in JPEGs with EXIF data with links.

Also deprecates $wgAllowTitlesInSVG and allows it by default.

This should still protect against malformed PNG attacks on old IE
versions, though false positive checks are conceivable on PNG files
containing comments very close to the beginning of the file.

Adds $wgVerifyMimeTypeIE config var to allow disabling the IE checks
entirely, if desired, but leaves it in place by default. These are
more conservative than the checks that were removed.

Added test cases for the old IE5/6 bug and the particular sort of
JPEG metadata that struck false positives previously.

Bug: T27707
Change-Id: I66642a74fce1a1894cad67d62b0da61020db469a
2019-06-07 14:21:00 -07:00
jenkins-bot
37ef56fadd Merge "registration: Allow @-prefixed keys in manifest_version 2" 2019-02-14 06:38:04 +00:00
Stanislav Malyshev
341bd5d6e9 registration: Allow @-prefixed keys in manifest_version 2
As specified in
<https://www.mediawiki.org/wiki/Manual:Extension_registration#Retaining_documentation>.

Bug: T215650
Change-Id: I2ff523828389fbec39cdf7c55046e7da901a2ecb
2019-02-12 18:50:14 -08:00
Roan Kattouw
fbbd65d2df ResourceLoader: Add support for packageFiles
Package files are files that are part of a module, but are not
immediately executed when the module executes. Instead, they are
lazy-excecuted when require() is called on them. Package files can be
scripts (JS) or data (JSON), and can be real files on the file system,
or virtual files generated by a callback.

Using virtual data files, server-side data and config variables can be
bundled with a module. Support for file-based require() allows us to
import npm modules into ResourceLoader more easily.

The require function passed to each script execution context, which was
previously a reference to the global mw.loader.require() function, is
changed to one that is scoped to the module and the file being executed.
This is needed to support relative paths: require( '../foo.js' ) can
mean a different file depending on the path of the calling file.

The results of require()ing each file (i.e. the value of module.exports
after executing it) are stored, and calling require() on the same file a
second time won't execute it again, but will return the stored value.

Miscellaneous changes:
- Add XmlJsCode::encodeObject(), which combines an associative array of
  XmlJsCode objects into one larger XmlJsCode object. This is needed for
  encoding the packageFiles parameter in mw.loader.implement() calls.

Bug: T133462
Change-Id: I78cc86e626de0720397718cd2bed8ed279579112
2019-02-05 22:05:02 +00:00
Max Semenik
ce071ded51 Display translatable images in wiki language on image pages
We currently show SVGs in default languages unless overridden with
lang=... URL parameter (and we have UI for setting it). This change
makes it display thumbnails in wiki language, if translation is available.

Bug: T210814
Change-Id: Ieb0b5e9e27f45b71ef119bb3c1d3f2cd4d7100e5
2019-02-04 15:57:37 -08:00
jenkins-bot
f7ead5d3ca Merge "Better detection for old MS Office files" 2019-01-23 01:01:43 +00:00
Tim Starling
f12db38048 Better detection for old MS Office files
* Introduce MSCompoundFileReader, which reads the CFB directory and
  detects the file type from well-known names in the root directory
* Do not detect a ZIP file if the EOCDR is not at the end. Other
  containers, especially CFB files, may contain ZIP files embedded
  within them in the last 64KB, but this is not a security concern
  unless the EOCDR is exactly at the end of the file.

Bug: T40432
Change-Id: Id5b1a258ccf3c3c8951e32f6b7a5b1bafe941082
2019-01-23 09:23:17 +11:00
Bartosz Dziewoński
1805fd3be5 resourceloader: Avoid unnecessary xmlns attributes in ResourceLoaderImage
We were creating the `<g>` element without specifying a namespace,
which caused the library to add `xmlns` attributes with the document's
default SVG namespace to elements that we appended underneath it.
(At least, that's what I think was happening.)

Specify the SVG namespace when creating it to avoid the mess and
reduce resulting file size.

Change-Id: Ida27494aeae9dece16f878c16cf9aa582e6deac3
2019-01-16 02:22:45 +00:00
Bartosz Dziewoński
d6043347c3 resourceloader: Avoid toplevel <title> under <g> in ResourceLoaderImage
While it shouldn't be causing any rendering problems,
doing so is semantically incorrect.

Bug: T213507
Change-Id: Ic86cd2bf3028eb24ad60db7ffa9498dd86edd4a5
2019-01-15 22:48:54 +00:00
jenkins-bot
2901b84c49 Merge "TemplateParser: Pass FLAG_MUSTACHELOOKUP to enable parent context access" 2018-09-05 00:33:42 +00:00
Roan Kattouw
3c06d76521 TemplateParser: Pass FLAG_MUSTACHELOOKUP to enable parent context access
This allows variables defined in an outer context to be used in inner
contexts. For example:

<h2>{{foo}}</h2>
<ul>
{{#things}}
    <!-- bar is a property of each thing, foo is an outer variable -->
    <li>{{foo}} is a {{bar}}</li>
{{/things}}
</ul>

Bug: T203209
Change-Id: Ib0ae0fb0b4be6b161f548c79db6fb6f4b831f7c1
2018-09-04 12:39:34 -07:00