Commit graph

163 commits

Author SHA1 Message Date
vladshapik
1091f7753f Hard-deprecate Parser::mUser public access, Parser::getUser and ParserOptions::getUser
Bug: T285713
Depends-On: Ie75c9cd66d296ce7cf15432e2093817e18004443
Change-Id: I4297aea3489bb66c98c664da2332584c27793bfa
2021-08-17 15:42:05 +00:00
Alexander Vorwerk
60c3deaba1 RepoGroup: inject MimeAnalyzer
Change-Id: Ic3009b368b38d307d00dc3bec5c8d46b2dc97f83
2021-08-13 17:02:42 +02:00
Alexander Vorwerk
135595a818 Remove unnecessary use statement
I10d2b32bd95316e1331b5eb88ef57de55c3ca1a1 removed the last usage of MediaWikiTestCaseTrait, thus the use statement can be removed.

Change-Id: I05b3b56abc8079abd294e453c78128858ef8ea7d
2021-08-04 00:54:08 +00:00
DannyS712
b45ddb2ab3 Use WikiPage::doUserEditContent() instead of ::doEditContent()
Results in passing a user where previously the fallback
to $wgUser was being used, mostly in tests.

Bug: T255507
Change-Id: Iabe24315b23c0ad1272353186425e71974528d23
2021-06-28 00:11:30 -07:00
Arlo Breault
fdd8f864b8 Emit media structure as piloted in Parsoid
Gated behind the flag $wgParserEnableLegacyMediaDOM.  The scattershot
usage of it is a little unfortunate but isn't expected to live very long
so maybe that's acceptable.

Further details can be found at,
https://www.mediawiki.org/wiki/Parsing/Media_structure

Bug: T51097
Bug: T266148
Bug: T271129
Change-Id: I978187f9f6e9e0a105521ab3e26821e36a96b911
2021-06-24 23:32:40 +00:00
Tim Starling
b4849e03b7 Use the unserialized form of image metadata internally
Image metadata is usually a serialized string representing an array.
Passing the string around internally and having everything unserialize
it is an awkward convention.

Also, many image handlers were reading the file twice: once for
getMetadata() and again for getImageSize(). Often getMetadata()
would actually read the width and height and then throw it away.

So, in filerepo:

* Add File::getMetadataItem(), which promises to allow partial
  loading of metadata per my proposal on T275268 in a future commit.
* Add File::getMetadataArray(), which returns the unserialized array.
  Some file handlers were returning non-serializable strings from
  getMetadata(), so I gave them a legacy array form ['_error' => ...]
* Changed MWFileProps to return the array form of metadata.
* Deprecate the weird File::getImageSize(). It was apparently not
  called by anything, but was overridden by UnregisteredLocalFile.
* Wrap serialize/unserialize with File::getMetadataForDb() and
  File::loadMetadataFromDb() in preparation for T275268.

In MediaHandler:

* Merged MediaHandler::getImageSize() and MediaHandler::getMetadata()
  into getSizeAndMetadata(). Deprecated the old methods.
* Instead of isMetadataValid() we now have isFileMetadataValid(), which
  only gets a File object, so it can decide what data it needs to load.
* Simplified getPageDimensions() by having it return false for non-paged
  media. It was not called in that case, but was implemented anyway.

In specific handlers:

* Rename DjVuHandler::getUnserializedMetadata() and
  extractTreesFromMetadata() for clarity. "Metadata" in these function
  names meant an XML string.
* Updated DjVuImage::getImageSize() to provide image sizes in the new
  style.
* In ExifBitmapHandler, getRotationForExif() now takes just the
  Orientation tag, rather than a serialized string. Also renamed for
  clarity.
* In GIFMetadataExtractor, return the width, height and bits per channel
  instead of throwing them away. There was some conflation in
  decodeBPP() which I picked apart. Refer to GIF89a section 18.
* In JpegMetadataExtractor, process the SOF0/SOF2 segment to extract
  bits per channel, width, height and components (channel count). This
  is essentially a port of PHP's getimagesize(), so should be bugwards
  compatible.
* In PNGMetadataExtractor, return the width and height, which were
  previously assigned to unused local variables. I verified the
  implementation by referring to the specification.
* In SvgHandler, retain the version validation from unpackMetadata(),
  but rename the function since it now takes an array as input.

In tests:

* In ExifBitmapTest, refactored some tests by using a provider.
* In GIFHandlerTest and PNGHandlerTest, I removed the tests in which
  getMetadata() returns null, since it doesn't make sense when ported to
  getMetadataArray(). I added tests for empty arrays instead.
* In tests, I retained serialization of input data since I figure it's
  useful to confirm that existing database rows will continue to be read
  correctly. I removed serialization of expected values, replacing them
  with plain data.
* In tests, I replaced access to private class constants like
  BROKEN_FILE with string literals, since stability is essential. If
  the class constant changes, the test should fail.

Elsewhere:

* In maintenance/refreshImageMetadata.php, I removed the check for
  shrinking image metadata, since it's not easy to implement and is
  not future compatible. Image metadata is expected to shrink in
  future.

Bug: T275268
Change-Id: I039785d5b6439d71dcc21dcb972177dba5c3a67d
2021-06-08 17:04:01 +10:00
ZabeMath
947cba5dee ParserTestRunner: use HookContainer instead of Hooks class
Hooks::register and Hooks::clear are deprecated (the second one is
hard deprecated, leading to deprecation warnings). Use
HookContainer::register and HookContainer::clear instead.

Bug: T281613
Change-Id: I10d2b32bd95316e1331b5eb88ef57de55c3ca1a1
2021-05-02 05:12:08 +00:00
James D. Forrester
df5eb22f83 Replace uses of DB_MASTER with DB_PRIMARY
Just an auto-replace from codesniffer for now.

Change-Id: I5240dc9ac5929d291b0ef1c743ea2bfd3f428266
2021-04-29 09:24:31 -07:00
DannyS712
6fb338ae40 MediaHandlerFactory: inject a logger
Instead of using wfDebug
Also normalize the entries

Change-Id: Ie539233c8b95eaae370732f97681989821157299
2021-03-31 17:45:51 +00:00
James D. Forrester
7c74fc35e2 parserTests: Avoid problematic language in comments
Bug: T277986
Change-Id: I1e079d670ecfb5338223a26df507427b45e28121
2021-03-28 21:23:37 -07:00
C. Scott Ananian
ff20e86a4c Make Parser::$mFunctionHooks private
This property was deprecated in 1.35.

Code search:
https://codesearch.wmcloud.org/search/?q=mFunctionHooks&i=nope&files=&excludeFiles=&repos=

No dependencies in WMF-deployed code.  One use in non-WMF-deployed code:
* I5ae19465561b150ee1c74a1fe03fa359964e81c4

Bug: T275160
Change-Id: I96ca88048c5a1cc8032ebcd502015819958680fb
2021-03-16 19:42:25 +00:00
C. Scott Ananian
1c6e24a1a2 Make Parser::$mTagHooks private
This property was deprecated in 1.35.

Code search:
https://codesearch.wmcloud.org/search/?q=mTagHooks&i=nope&files=&excludeFiles=&repos=

Dependencies below are for WMF-deployed code.  Other uses in
non-WMF-deployed code have been patched in:
* I435b0d1ccae9d9bf6fff85dc3e79d3c4b447eb37
* I85ef0e6ce3f0c818df85809d39259d13b56d966c
* Idab6c9475f78ff4040061f2f317560bbe41666d8

Bug: T275160
Depends-On: Ic5445471d770e396421a4fb2bcfbe1490a77e1bf
Depends-On: Ib708e3f84aa871de84aa56561c875f4a85bb000c
Change-Id: I42e23b101e870b66d169cbb731a0359e90f46265
2021-03-16 19:42:12 +00:00
C. Scott Ananian
3f990d5b4c Inline Parser::firstCallInit() into ::__construct()
This has effectively been the case since 1.35; this just cleans up the
remaining code which assumed it still needed to explicitly call
Parser::firstCallInit() on a newly-constructed Parser.

Bug: T250444
Change-Id: I340947c721172f12ff413322b4283627c0b0b3a4
2021-03-16 19:41:56 +00:00
James D. Forrester
bdddfb92ba Drop wgContLang, deprecated in 1.32
Bug: T245940
Depends-On: Ib7fe7318100c0aadc3aa759416bf787913a9b788
Change-Id: I75c3b6715abd5eaf3619337cab8b1844e9a8349a
2021-03-08 13:27:28 -06:00
Umherirrender
a1de8b8700 Tests: Mark more more closures as static
Result of a new sniff I25a17fb22b6b669e817317a0f45051ae9c608208

Bug: T274036
Change-Id: I695873737167a75f0d94901fa40383a33984ca55
2021-02-09 02:55:57 +00:00
Reedy
729f20afc8 Tests: Mark more closures as static
Bug: T274036
Change-Id: I911d3041cebe417d162934223b46ea295c6d20e3
2021-02-07 13:26:56 +01:00
Umherirrender
7691dbeca9 Add missing @param and @return to documentation in tests
Change-Id: Ic663e81cca0bf007804a70772250914a85f1fef4
2021-01-22 19:57:25 +01:00
Arlo Breault
78e85ab9e5 Split out media parser tests
Bug: T111604
Bug: T271129
Change-Id: I9893d11d50b8e5884239da2bb41262e093afc47f
2021-01-13 15:53:33 -05:00
C. Scott Ananian
665eae14ac ParserTestRunner: share more code w/ MediaWikiIntegrationTestCase
Refactor the database setup code to share more code between
ParserTestRunner and MediaWikiIntegrationTestCase.  Made
`::setupAllTestDBs` static so it can be reused from
ParserTestRunner.

Made ParserTestRunner::addArticle more like
MediaWikiIntegrationTestCase::addCoreDBData().  Some additional
refactoring work could be done here in the future to share more code.

After the refactoring the ParserTestTables hook is no longer necessary
and so has been (soft) deprecated.  MediaWikiIntegrationTestCase
clones all database tables, so ParserTestRunner no longer needs to ask
extensions for a list of specific tables it should clone.  Cleaning up
the handful of extensions which define this hook will be left to a
future patch set.

Change-Id: I5124789fac333a664b73b4b4a1e801ecc0a618ca
2021-01-07 23:31:12 -05:00
C. Scott Ananian
71aa8e0e0d Add support for running parsertests with Parsoid in integrated mode
This allows us to move Parsoid-specific extension code from the
Parsoid repo into the extension's own repository and still have
Parsoid parser tests run on it via core's mechanism for running
extension tests.

Factored out some common ParserOptions setup into a common helper
function.  There are a number of features still missing from the
Parsoid test runner, which are marked with @todo comments and
phab task numbers.

Bug: T254181
Change-Id: Ifaf53862b96e9127d8f375ad8dd0cc362cba9f5b
2020-12-23 15:50:50 -05:00
C. Scott Ananian
3bf9a8e577 ParserTestFileSuite: run staticSetup once per suite
We now run staticSetup once per suite, instead of once per test.  This
matches the way that parserTests.php runs the tests, and speeds up
testing.  In addition, it allows simplifying a bit of redundant code
in ParserTestRunner::addArticles() because we know that static setup
has been done.

Change-Id: Id93412cf9eb714af693aa130de1b02c4ef87d6e0
2020-12-23 15:50:50 -05:00
C. Scott Ananian
727a77a19e ParserTestRunner: add interwiki prefixes used by Parsoid tests
Bug: T254181
Change-Id: Ia79992e8e44435746f8512b2f05408c560c80533
2020-12-21 16:53:43 -05:00
C. Scott Ananian
7b4c8f3768 ParserTestRunner: tweak filetype of djvu, allow capital E in !!hooks
Two minor improvements to parser tests:

1) Tweak the filetype of the djvu file to match what commons reports
   nowadays for a more relevant test

2) Allow capital 'E' when registering an extension requirement in the
   `!! hooks` clause of a parser test file.

Bug: T254181
Change-Id: Id4dcd0f5d040f4deb5235d565119a6f76cc7b59d
2020-12-21 16:53:43 -05:00
C. Scott Ananian
3d40c43b1f Deprecate InterwikiLoadPrefix hook
The InterwikiLoadPrefix hook isn't compatible with Parsoid, as it is
unidirectional and doesn't support enumerating all valid prefixes
(T270444).  Set/reset $wgInterwikiCache to mock the interwiki table
for parserTests and other unit tests instead.

This is a soft deprecation, as the used-in-production
Extension:Interwiki still uses InterwikiLoadPrefix, although not in a
way that would break Parsoid (since $wgInterwikiCache is set in
production).

Bug: T270444
Change-Id: If2507017c99c4ee42c104a0890bc45a84d7239d5
2020-12-21 19:28:48 +00:00
Reedy
c75d2e41cc ParserTestRunner: Fix skipping typo
Change-Id: I72f2ed394141acf1fba9a450d10302607545225f
2020-12-16 23:50:51 +00:00
jenkins-bot
fe8779ebf9 Merge "Parser tests: Update TestFileReader to the latest reader from Parsoid" 2020-12-16 17:45:01 +00:00
David Kamholz
a7ad0547bc Implement <langconvert> tag
The <langconvert> tag takes two attributes: from (language variant from) and to (language variant to). It returns the content of the tag converted using LanguageConverter. It returns an error if the attributes are not present, if the variants do not exist, or if the variants belong to different languages. Currently it does not work for IuConverter, because the variants use the code ike rather than iu, and ike isn't in the list of languages with converters available.

This patchset reimplements from a parser function to a tag, and renames from transliterate to langconvert.

Bug: T263082
Change-Id: Idc3a32c66d5a0466c63e7ce8753d2619354c30b0
2020-12-14 19:40:31 -08:00
C. Scott Ananian
02ec9de651 Parser tests: Update TestFileReader to the latest reader from Parsoid
This replaces the 'requirements' from parser tests (hooks and
functionhooks) with a more flexible 'options' clause to allow
additional file-level requirements/options to support running parser
tests in multiple modes.  (For example, with the legacy parser or in
one of two parsoid modes.)

Bug: T254181
Depends-On: I636bd1f2c8aee327acbbd1636e2ac76355f1d80e
Change-Id: I58373d135c3a804f4ce9967112c338435f5cd4b6
2020-12-14 18:49:11 -05:00
Ammar Abdulhamid
71571191d4 Chain MutableRevisionRecord method calls 2
Change-Id: I86578cfbc892f171a4e433283b86d1b78fe4167d
2020-11-27 05:26:54 +01:00
C. Scott Ananian
c64e71615e Replace $wgDisable{Lang,Title}Conversion with LanguageConverterFactory methods
Replace direct access to $wgDisableLangConversion with
LanguageConverterFactory::isConversionDisabled(), and replace direct
access to $wgDisableTitleConversion with
LanguageConverterFactory::isTitleConversionDisabled().  However, most
places that check ::isTitleConversionDisabled() actually want
::isLinkConversionDisabled(), so add that too (and deprecate
isTitleConversionDisabled()).

Code search:
https://codesearch.wmcloud.org/search/?q=Disable%28Lang|Title%29Conversion&i=nope&files=&repos=

This change removes a number of spurious dependencies on the global
configuration and reduces code duplication (for example, if the logic
for disabling language conversion were ever to change).

Depends-On: I6fa8230ae97b0e34c381003548e61f9b7387d363
Change-Id: Icc4687638ff1815003dd903854efdbd904854f1e
2020-11-25 12:47:26 -05:00
Aaron Schulz
90865e0e89 tests: Use FileBackend::quick*() methods in ParserTestRunner
This avoids needs I/O from lock files

Change-Id: I0b8661712154c61f16c37c0c4303909c7b678cad
2020-11-13 06:06:59 +00:00
jenkins-bot
2752717667 Merge "Minor updates to some unrelated PHPDoc tags" 2020-10-30 17:17:57 +00:00
Umherirrender
c85a43561e Improve class property documentation
Reformat existing documentation to match the format

Change-Id: I190b54b5e962f17bab6502dd1b3c02f11dc926d2
2020-10-30 10:38:58 +01:00
Thiemo Kreuz
3306f15ce3 Minor updates to some unrelated PHPDoc tags
The past weeks I collected a few minor updates in my local dev
environment, and would like to submit them now.

Change-Id: Ibe00d72763f1b66c50cf73e00c8fa52d265043fc
2020-10-28 19:00:48 +00:00
C. Scott Ananian
33c506843b phpunit tests: Reset the SpecialPageFactory when Content Language changes
This is causing problems for Parsoid CI, as parser tests fail when
phpunit runs the tests at a different point than they are run in
core's CI due to the side-effects of content-language changes made in
other phpunit tests. (For example, phpunit runs all extension tests
after core tests, so the same parsertest can pass if included in core
and then fail when included in an extension.)

SpecialPageFactory::$aliases has a dependency on the current content
language, with no way to reset it other than to recreate the
SpecialPageFactory.

Change-Id: I278580ed5cf2c85403cbaf601f8af4753e14a9d0
2020-09-23 14:17:31 -04:00
C. Scott Ananian
c443177614 Allow parserTests to declare a dependency on a particular extension
The parsertests file allows certain tests to declare a dependency on
a particular tag hook, but this doesn't work for extensions like
TimedMediaHandler which affect the output but don't register a
unique extension tag name.  Allow using 'extension:Foo' in the
`hooks` clause to register a dependency on the specific extension name,
instead of indirectly on the registered extension tag name.

Change-Id: I2d3f7e1313b4456733f820e6d8c504bb8d7427a7
2020-09-22 12:51:34 -04:00
jenkins-bot
13dc0e893d Merge "Tracking category and parser warning for deprecated uses of {{=}}" 2020-09-15 21:48:22 +00:00
jenkins-bot
448da261a2 Merge "Allow independent parser test files to (re)define articles w/ the same names" 2020-09-15 21:47:43 +00:00
C. Scott Ananian
aeb3f45c20 Tracking category and parser warning for deprecated uses of {{=}}
We plan to add {{=}} as a built-in parser function, expanding to `=`,
in the same way that `{{!}}` is a built-in.  It will be used to
automatically escape uses of `=` in template arguments (again, in the
same way that `{{!}}` can be used to protect uses of `|` in template
arguments).

Some wikis have non-standard definitions of `Template:=`; add a
tracking category to warn these wikis to transition before we turn on
the built-in parser function in a future release.

New parser test file added, so we can re-define Template:= and test
both cases of this new warning.

Bug: T91154
Change-Id: I50ff8a7b6be95901ebb14ffbe64940a0f499cfac
2020-09-15 20:16:37 +00:00
C. Scott Ananian
ce663741bc Allow independent parser test files to (re)define articles w/ the same names
It leads to surprising results when the definitions in one parser test
file leak into all the others.  This can cause spurious test failures
when you happen to have two extensions which define conflicting
article fixtures, and prevents you from using parser tests to test
patches like I50ff8a7b6be95901ebb14ffbe64940a0f499cfac, where you
deliberately want to set up and test two different definitions for the
same template name.

Change-Id: I958c6305a95ca32418d83b7f33f7c180a3b370cd
2020-09-15 16:15:44 -04:00
DannyS712
e834b31b2b Use recordUpload3, hard deprecate recordUpload2
Also reduces references to $wgUser in importImages

Bug: T248813
Change-Id: I17c850000044f65f2fcfdfcfb82f852583a99000
2020-09-10 00:00:42 +00:00
C. Scott Ananian
585cbcd77f Use parser test file parser from Parsoid
One (test file) parser to rule them all.  Reduce a little bit of
redundant code between core and Parsoid by using Parsoid's parser test
file parser to run core's parser tests.

This should have no effect on users of TestFileReader::read() *except*
that Parsoid's test file reader is more strict about bogus lines in
the test file, including duplicate test names, and we've removed support
for the old v1 format (hard deprecated in 1.35).

Next step will be to be able to execute parser tests on extensions
using Parsoid's parser as well.

Bug: T254181
Depends-On: I8ab4a8c59ed1b6837dba428f96a8ba0084b7fb68
Change-Id: I5acaf82819ae964895a831be4f28c31c77a09e84
2020-08-17 14:52:08 -04:00
C. Scott Ananian
fb7ae07c8a Hard-deprecate parser tests targetting Preprocessor_DOM
Support for Preprocessor_DOM was removed in 1.35; it's time to clean up
any old parser tests which required it.

Change-Id: I36c7906b8ce31ef6885aef54175749e67e51d07c
2020-08-04 14:21:32 -04:00
C. Scott Ananian
7bbb14f87d Fix parserTests.php by ensuring change_tag tables are cloned
This fixes a regression in the parser test CLI runner caused by
435d5f4d55, which added a change tag for
"manual reverts".  The parser test framework was triggering this
change tag addition as it set up its mock article database, and then
subsequent attempts to write the change tag to the database failed
because the tables were missing.

Bug: T259186
Change-Id: I232e918dfdc83244a010681b6adffd6c1171cf24
2020-07-29 16:06:20 -04:00
C. Scott Ananian
7101c981b2 Fix parser test class naming
This isn't really user visible, but the algorithm for ensuring there
are no conflicts in automatically-generated parser test class names
had a number of issues which led to inconsistent naming.

Change-Id: I50ff5b72381332c77f0d99af08e689796019a7af
2020-07-24 00:29:12 -04:00
DannyS712
2f4b71fc6c Replace uses of Revision constants
Bug: T257010
Change-Id: Id63123e8b8becd31756d5b68ca11edb238ec8a59
2020-07-03 01:23:44 +00:00
addshore
959bc315f2 MediaWikiTestCase to MediaWikiIntegrationTestCase
The name change happened some time ago, and I think its
about time to start using the name name!
(Done with a find and replace)

My personal motivation for doing this is that I have started
trying out vscode as an IDE for mediawiki development, and
right now it doesn't appear to handle php aliases very well
or at all.

Change-Id: I412235d91ae26e4c1c6a62e0dbb7e7cf3c5ed4a6
2020-06-30 17:02:22 +01:00
DannyS712
a6d16bd03d Remove unneeded creation of revision objects
Clean up some technical debt; use MutableRevisionRecord instead of
manually constructing a Revision from an array, remove last uses of
RevisionStoreDbTestBase::revisionToRow and remove the method.

Each file can be reviewed separately (except that the removal of
revisionToRow depends on replacing its usage)

Bug: T246284
Change-Id: I0bdc069b21a5c41ef8f9e972c5b17ff189d4a741
2020-06-10 09:09:55 +00:00
Thiemo Kreuz
6aa6d10e86 Replace all call_user_func(_array) in all tests
There is native support for all of this now in PHP, thanks to changes
and additions that have been made in later versions. There should be no
need any more to ever use call_user_func() or call_user_func_array().

Reviewing this should be fairly easy: Because this patch touches
exclusivly tests, but no production code, there is no such thing as
"insufficent test coverage". As long as CI goes green, this should be
fine.

Change-Id: Ib9690103687734bb5a85d3dab0e5642a07087bbc
2020-06-06 18:41:20 +02:00
DannyS712
381d873a8b Replace core uses and hard deprecate Parser(Options) Revision methods
Bug: T249384
Change-Id: Iff10e76120eb8b6b4fbb939182dede83c86d3da2
2020-06-03 05:55:35 +00:00