Commit graph

203 commits

Author SHA1 Message Date
Alexander Vorwerk
d66fe8b7c7 Bump minimum required version for upgrade to 1.31
The policy allows this and since 1.39 is going to be the next LTS
release, I think it is fine to do this now.

Change-Id: If426e0ee349252ccc0ba9c4222c7d6865ab57fa2
2022-03-15 22:12:59 +01:00
Alexander Vorwerk
71a092e513 Add 1.38 snapshot to sqlite db tests
Change-Id: I72e703473679dbe982c8d851a6c357b17de415eb
2022-03-15 17:05:42 +01:00
mainframe98
de0c4819d1 Add validation for abstract schema
This adds an option to the schema generating maintenance scripts to
validate abstract schemas and schema changes and a structure test to
validate exisiting schemas and schema changes. Schemas are also
validated when generating.

The validation for the schema doesn't impose limits on table, index or
column names as I couldn't find any reliable conventions for them.

The structure tests only cover MediaWiki itself as there is no
convention on where extensions store their abstract schema.
Ideally, auto detection would be possible for sql/, but for now
extensions have to define their own (thankfully trivial) tests.

A couple of invalid definitions were fixed thanks to these tests.

I aimed to be thorough, but not all parts of the abstract schema
are completely clear, and Doctrine's documentation is not complete.
As a result, not everything has a description field.

Bug: T298320
Change-Id: I681d265317d4d1584869142ebb23d4098c06885f
2022-02-22 17:41:08 +00:00
jenkins-bot
99dee6855a Merge "Change return value of ParserOutput::getPageProperty() when property is missing" 2022-02-19 00:49:48 +00:00
C. Scott Ananian
c39ef6c6c9 Change return value of ParserOutput::getPageProperty() when property is missing
The old ParserOutput::getProperty() method returned `false` when a property
was missing.  This requires callers to use the `?:` syntax to supply default
values, which then causes any falsey value to be treated as missing.
So, for example, setting the defaultsort to '0' will cause the default
sort to be ignored.

Modern php convention is to use `null` for missing values, and the `??`
syntax is a better/more restrictive alternative to `?:`.

We renamed `ParserOutput::getProperty()` to `::getPageProperty()` in
1.38 (Ie963eea5aa0f0e984ced7c4dfa0fd65d57313cfa/T287216) but kept the
return value convention.  Before this actually makes it into a 1.38
release, take the opportunity to fix the return value for the new
`ParserOutput::getPageProperty()` method to return `null` when the
property is missing.

We need to do some temporary workarounds to the places we'd
already swapped over to use the new `::getPageProperty()` method
to allow them to handle either `false` or `null` as a return value;
we'll clean that up once this is merged.

Code search:
https://codesearch.wmcloud.org/deployed/?q=-%3EgetPageProperty%5C%28|T301915&i=nope&files=&excludeFiles=&repos=

Bug: T301915
Depends-On: I3f11ce604970e47b41fc1c123792df8c3045626f
Depends-On: Ie7533f49fe4cad01ebfda29760d23c61e9867b10
Depends-On: Ic5c09f5caa4c897bc553c614fbae9cee159566a2
Depends-On: I0278b2eafd90e77e4fee41c45a1165fb79ddf47e
Depends-On: I383abb6b7dc5e96c0061af13957609f6e31a1065
Depends-On: I79f9f4078e415284af29b15047bafd1c823d7f5b
Depends-On: I02276c48c49f5d2d241a69eb0a6cdf439b572d8b
Depends-On: I71628661b4539a4e35ae32846e719f92bcf782e0
Depends-On: I7e215cb43de0ce150a6bcc00f92481dcdcfed383
Change-Id: Iaa25c390118d2db2b6578cdd558f2defd5351d15
2022-02-18 21:15:58 +00:00
C. Scott Ananian
3c211fdb3c Update ParserCache serialization test cases to use valid category keys
Category keys are supposed to be non-null strings.

The test cases use bogus integer values, which causes issues when
refactoring more strictly enforces validity checks on category sort
key values.

Change-Id: If2937a694ba6bd4c522336f33aa58d40949b5a54
2022-02-17 12:12:53 -05:00
C. Scott Ananian
b46dfcc351 Update ParserCache serialization test cases to use a valid index policy
Valid values for ParserCache::$mIndexPolicy are '', 'noindex', and 'index'.

The test cases use the bogus value 'policy1', which causes issues when
refactoring more strictly enforces validity checks on index policy
values.

Change-Id: I2d00ff4e3ded043d18942c8482a39fac14ec60bc
2022-02-09 12:47:27 -05:00
Ed Sanders
9e234e6a37 eslint: Only apply client rules to client files
Change-Id: I757b9616ce69b6020e9f0e556e006d39de59eb01
2022-02-09 02:40:11 +00:00
jenkins-bot
7061fd2ee2 Merge "Reapply "Move limit report rendering to ParserOutput"" 2021-11-11 00:19:44 +00:00
Ed Sanders
9fe2995989 build: Update eslint-config-wikimedia to 0.21.0
* Replace `substr`/`substring` with `slice`. The second argument
  to `substring` is length, not end index as in `substr`/`slice`,
  so convert where necessary.
* Replace `new Date().getTime()` with `Date.now()`
* Documentation fixes
* Replace `throw Error` with `throw new Error`

Change-Id: I532500ea4c99d8ebec01efb21273c8df21626e59
2021-11-09 21:08:55 +00:00
Ppchelko
643fc535c3 Reapply "Move limit report rendering to ParserOutput"
This reverts commit 2bcb3fe567.

Reason for revert: this is a good change,
just needed more work to not break CI

Change-Id: I23768bee242e3cf81b1493a740cf070e7ad1e224
2021-11-09 11:08:08 -08:00
Fomafix
4f71fcdfcc Remove the executable flag from test files
chmod a-x tests/phpunit/data/MSCompoundFileReader/*

Change-Id: Ic2fa5f118b40126bea429d2299563c03fe6b3d1a
2021-10-16 11:23:02 +00:00
Petr Pchelko
a1aa3e0827 Hard-deprecate all public property access on CacheTime and ParserOutput.
- Added a test where ParserOutput objects with CacheTime
properties set are unserialized from previous versions.
- Generate new serialization tests for 1.38

Now all serialization in production is JSON, so changing
property visibility shouldn't affect ParserCache.

Bug: T263851
Depends-On: I283340ff559420ceee8f286ba3ef202c01206a23
Change-Id: I70d6feb1c995a0a0f763b21261141ae8ee6dc570
2021-10-13 13:27:16 -04:00
Amir Sarabadani
c3fff36540 Add 1.36 and 1.37 snapshots to sqlite db tests
To make sure upgrades from those versions work properly

Change-Id: I570cf965286bc5358a43c9601217b8781bb09b0e
2021-09-29 20:13:38 +02:00
Amir Sarabadani
753a4ecdd3 Bump minimum required version for upgrade to 1.29
Since the branch cut has happened, we can bump and get rid of legacy
cruft. According to the policy we can go up to 1.31 but let's keep it
that way to avoid major distruptions.

Change-Id: I9d697445a3bb5047726c8b2a7f808edb8403cdda
2021-09-28 17:57:57 +02:00
Fomafix
ff7f41f928 resourceloader: Append newline to JavaScript only if missing
In debug mode of a packageFiles module there is now only one newline
between the JavaScript code and the closing }.

This change reuses ResourceLoader::ensureNewline and make this to a
interal public static function.

Change-Id: I89519896e3dc56d966c4a63102904686bff6fac9
2021-08-28 12:25:11 +00:00
jenkins-bot
da9f37a177 Merge "resourceloader: Add integration test for ResourceModuleSkinStyles" 2021-08-19 01:32:26 +00:00
Timo Tijhof
76245d70ff resourceloader: Add integration test for ResourceModuleSkinStyles
This was covered in small ways already, but we did not have coverage
of the registration logic where the skin attribute is injected into
the module info, and this then correctly getting read and composed
into a single stylesheet.

This is prep for an optimisation in a subsequent commit.

Change-Id: Ia6fff44a345bb4f5811956638cada8b3e6b6ea71
2021-08-19 01:14:00 +00:00
Fomafix
6157bdaef0 CSSJanus: Update from version 1.3.0 to version 2.0.0
This version fixes a flipping bug.

Bug: T288943
Depends-On: Ida34372a4e5acb3147d83c7a1d008850c8382e32
Change-Id: I0528443e4eae0338f5a37491352db03944c6bd4c
2021-08-16 18:39:41 +00:00
libraryupgrader
5357695270 build: Updating dependencies
composer:
* mediawiki/mediawiki-codesniffer: 36.0.0 → 37.0.0
  The following sniffs now pass and were enabled:
  * Generic.ControlStructures.InlineControlStructure
  * MediaWiki.PHPUnit.AssertCount.NotUsed

npm:
* svgo: 2.3.0 → 2.3.1
  * https://npmjs.com/advisories/1754 (CVE-2021-33587)

Change-Id: I2a9bbee2fecbf7259876d335f565ece4b3622426
2021-07-22 03:36:05 +00:00
Tim Starling
2e507003ca Ignore invalid chunks in PNG files, instead of aborting metadata extraction
* Factor out file read errors and unexpected EOF errors.
* For errors relating to chunk content, instead of throwing an
  exception which is silently discarded, just log an error and continue
  to the next chunk. This allows the dimensions to be extracted even if
  other metadata is mangled.
* As an additional sanity check, verify the CRC of each chunk.

Bug: T286273
Change-Id: I11d0186496324e0bb1bb0a143f438e0368a8e902
2021-07-13 11:11:03 +10:00
Timo Tijhof
3371e4031c tests: Remove unused composer.json sample data
Follows-up c351c4f06e.

Change-Id: I842690849590f2ee57eb951099b9131728ecb7d7
2021-06-08 23:56:01 +00:00
MacFan4000
c351c4f06e Freenode -> Libera per wikimedia moving from freenode to libera
Bug: T283247
Change-Id: Iebbb13ac53830a8d77b42bf2954cf80d76d78467
2021-05-20 22:10:24 +00:00
Timo Tijhof
39e2abd886 registration: Allow multi-line strings in "@" note keys
Change-Id: Ifb1a0f04cc22b122ed99308c096d0ce4296e9097
2021-04-29 02:47:47 +01:00
Ammarpad
e497482ee4 DoctrineSchemaBuilder: Do not add prefix placeholder for Postgres at all
Adding it (only to be removed later) is not useful and interferes with
the internals of the schema builder because it masks the actual table
name during the table sql building and at some stages, this actual name
is required to be known

Bug: T191231
Change-Id: I57cc9b83c81f5c2d0d50c7b9e0dc334e7251c74f
2021-04-06 00:23:14 +00:00
jenkins-bot
e623a91aab Merge "Fix replacement of control chars in DJVU text output" 2021-03-29 15:38:33 +00:00
Inductiveload
32ea3a3fbe Fix replacement of control chars in DJVU text output
The control characters are presented as text, not actual
control characters, so the regexes to replace them are
incorrect.

Added a column and para to the Djvu text on the first page
of the test LoremIpsum.djvu file

Bug: T230415
Change-Id: I4970bc30b3935ce4da062ee7ff687aa667027a00
2021-03-29 06:03:34 +00:00
Derk-Jan Hartman
114d8e1cc5 JPEG2000 MIME fixes
- jp2 is the preferred file extension
- add mime aliases
- add jpg/jpx (jpeg2000 part 2)
- add magic byte detection for both

Bug: T161934
Change-Id: I00f81d9c31bf53a3b950e8a1d2bf2684c3998745
2021-03-15 21:43:24 +01:00
DannyS712
c0d41c57cd test data: remove old cssmin test files
No longer needed, now in the minify repo

Follow-up: I8d71fc64aeecdb31db218b02e361ae9bb0d19b48
Change-Id: Idb8175eb032a3521d08982da5be24041ab8758d3
2021-03-11 20:02:39 +00:00
mainframe98
9b8d6612c6 Support Composer 2.0 in ComposerInstalled
Composer 2.0 provides installed packages under the 'packages' key
in installed.json.

Bug: T266419
Change-Id: I80b690fde1653c7a15f8e211782fa8e60971daa9
2021-03-04 09:29:34 +01:00
James D. Forrester
5a622b6a2e build: Upgrade eslint-config-wikimedia from 0.17.0 to 0.18.1
Change-Id: I5e3687be2b197134578126e1b890ee37dbc1bc1b
2021-02-18 08:39:09 -08:00
Amir Sarabadani
78e6c48156 Rework sqlite schema changes tests
We don't support upgrading from below 1.31 officially and from below 1.27, the
upgrade system doesn't work at all.

The files are copied from the release branches.

Bug: T259771
Change-Id: Ib5251f7ae38148e00ac2dccb7bde76529c9d5076
2021-02-05 16:21:24 +00:00
jenkins-bot
607c4d3910 Merge "CacheTime: remove mVersion field" 2021-01-25 19:12:05 +00:00
Timo Tijhof
f7e8bc6a69 exception: Add test for MWExceptionHandler trace formatting
Change-Id: I00b9607ba7e17c1f75ef065e83b83d2e1a82870c
2021-01-13 23:26:48 +00:00
Timo Tijhof
8ad97d7c32 resourceloader: Fail gracefully if a LESS message is not found
The message cache is originally meant for mw.messages in JS,
which expects non-existent messages to be cleanly omitted.
There is minimal server-side and client-side handling in place for this.

For LESS, however, we were assuming that the blob is complete,
thus not feeding anything to the LESS compiler, thus leading to
a run-time failure where the LESS file can't be parsed at all
due to an undeclared variable.

This could happen sometimes during development or after upgrading
a wiki with a stale LocalisationCache that is still being updated
at that time.

Bug: T267785
Change-Id: I60ff4eb7dce1fee56470acc177afd29ee14b764f
2021-01-07 15:41:02 -08:00
Petr Pchelko
2010a912c2 CacheTime: remove mVersion field
Has to be merged at least one train after the parent patch.

Bug: T269393
Change-Id: Iee98950ad0452a50088e755252da8468f0bec1ae
2021-01-04 10:29:52 -06:00
Daniel Kinzler
48172f794d Revert "Hard-deprecate all public property access on CacheTime and ParserOutput."
This reverts commit b1a30eb0c4.

Reason for revert: T269396

Bug: T269396
Change-Id: I374ca13ccc30418b8fe3bf98f5090f7643aac4d7
2020-12-04 11:47:55 +00:00
Petr Pchelko
b1a30eb0c4 Hard-deprecate all public property access on CacheTime and ParserOutput.
Bug: T263851
Change-Id: I3d3ff7b5b6899150df836e445b56896dfd5b887e
2020-11-19 10:12:41 -07:00
Petr Pchelko
b956c77d27 Merge CacheTime and ParserOutput accessedOptions properties
Change-Id: I5785596d68e8923f8bcbd182ace0b1991bd75c9a
2020-11-19 10:12:39 -07:00
Petr Pchelko
dbdc2a3cd3 Introduce JsonCodec to help with serialization/deserialization
Change-Id: I5433090ae8e2b3f2a4590cc404baf838025546ce
2020-11-19 08:32:21 -07:00
jenkins-bot
d009d75c73 Merge "Start generateSchemaChangeSql.php" 2020-11-16 08:40:36 +00:00
Petr Pchelko
7c68ae9296 Safe ParserOutput extension data and JsonUnserializable helper.
One major difference with what we've had before is that now we
actually write class names into the serialization - given that
this new mechanism is extencible, we can't establish any kind
of mapping of allowed classes. I do not think it's a problem
though.

Bug: T264394
Change-Id: Ia152f3b76b967aabde2d8a182e3aec7d3002e5ea
2020-11-10 11:21:09 -07:00
Petr Pchelko
017cfcf016 Forward-compat for merging CacheTime and ParserOutput mOptions
CacheTime::mUsedOptions and ParserOutput::mAccessedOptions
do exactly the same thing and has to be merged into a single property.
This patch adds forward-compatibility and needs to be deployed
at least one train before the patch which actually merges the properties.

Change-Id: Ic9d71a443994e2545ebf2a826b9155c82961cb88
2020-11-10 07:09:41 -07:00
daniel
cac89b547c ParserOutput: add support for binary properties in JSON.
This introduces a mechanism for encoding binary data in
strings set via setProperty(). This is needed to accommodate compressed
data as used by TemplateData, which uses gzip compression to make the
data fit into the page_props table.

Bug: T266200
Change-Id: I19fa0dea8c25d93fcdec9dc5ddd6f3c9c162b621
2020-11-04 18:52:09 +01:00
Amir Sarabadani
1125167b8d Start generateSchemaChangeSql.php
Creating schema changes from abstract schema and even abstracting a
schema change.

Bug: T230420
Change-Id: If626e866642af820dd70c5f9b0fe7c6a951e0a25
2020-11-01 21:14:53 +01:00
Ammar Abdulhamid
d095b56e25 Expand DoctrineSchemaBuilderTest
* Test the generated table for each platform
* Use the handy dirname() function instead of full raw string path

Bug: T191231
Depends-On: If344395615087c360597a5b3d66ea03e930b7d9b
Change-Id: I936c5d10924a46743c7f76df8526105eec7ff48d
2020-10-17 21:17:46 +00:00
daniel
600f64029f Use JSON for parser cache
This adds JSON serialization and deserialization capabilities
to CacheTime and ParserOutput.

NOTE: JSON serialization is disabled for now. Merging this patch
should not change behavior in production.

Bug: T263579
Change-Id: I18187e8bce573d21f6f1bd29106e07c63a6d2f4d
2020-10-13 16:28:52 -07:00
Petr Pchelko
8a879605d9 Add deserialization acceptance tests for ParserOutput
Bug: T264397
Change-Id: I6476fd9b8eff0e1b61ce5f43280d1cd9b7aaa77c
2020-10-12 08:55:32 +00:00
daniel
6eea7d7ed5 Add test infra for ParserCache serialization/deserialization
Based on Daniel's work at Ia6e70179b7ee5ce4e93888585ccc30d92da165c3
however was changed enough to move into a separate changeset.

More acceptance tests and data will be added in a followup commit.

Bug: T264397
Change-Id: I135187e83cbfa02b97c5656f0752f8bf1ceb58d0
2020-10-09 08:14:57 -06:00
Timo Tijhof
0c01d8cc52 resourceloader: Add skin-based 'mediawiki.skin.variables.less' import
Add the SkinLessImportPaths attribute for skin-specific LESS import
paths, which skins can use to override the mediawiki.skin.variables.less
file.

As a starting point, add the following 5 variables:

* device widths (3x)
  To help phase out 'mediawiki.ui/variables'. These are
  commonly used by MobileFrontend.

* @font-family-sans
  Recommended by Volker. Used by multiple skins.

* @border-radius-base
  Recommended by Volker as example of something that we currently
  hardcode in MediaWiki core for Vector and OOUI/WikimediaUI
  in 'mediawiki.widgets.datetime' but should instead be allowed
  to vary by skin and OOUI theme.

  Remove the hardcoded value for '@border-radius-base' in
  various places in favour of importing from mediawiki.skin.
  The default is a bare default of 0 (as border-radius is off
  by default in the browser).

  The value for Vector is restored there by I47da304667811.
  The value for MonoBook is improved by I000f319ab31b.

Bug: T112747
Change-Id: Icf86c930a3b5524254bb549624737d3b9dccb032
2020-09-15 10:05:53 +00:00