It was possible to run non-compatible tests via the CLI which, among
other things, would have caused a "--parsoid --updateKnownFailures"
test run to add bogus entries to the knownFailures json file.
Change-Id: Ie3c51b97edb9839f775301670018bb4ba3290fbe
This lets us run parser tests on specific directories. This is
particularly useful when syncing tests between Parsoid and core
since it lets us rerun tests for all synced files as well as
update known failures if requires for Parsoid integrated mode
test runs.
Change-Id: I9030ddd63997b0b197991860aeb850ebf54d4dd7
Make parser test discover in core work the same way as it does in
extensions: any file ending with *.txt under tests/parser is run
as a parser test file.
This search is recursive, which is motivation to also move some
unrelated files under tests/parser/preprocess over to
tests/phpunit/data/preprocess where they belong; they are used
by tests/phpunit/includes/parser/PreprocessorTest.php and are
unrelated to the parser test infrastructure.
Change-Id: I8c84b4b853e1309929dceb700aab1e79a598d8ab
This global variable was replaced by auto-discovery of parser test
files in extensions and is no longer needed (or used).
Change-Id: Ib616e60f36db32ffd8bc69c71a096f23c0a27910
This now aligns with Parsoid commit a3d35adf78c3c7e6a8f04e9a55b76f3efefaedb8
The monolithic parserTests.txt have been split up into a number of smaller
test files (more splitting to come) and three existing parser test files have
been renamed:
mediaParserTests.txt => media.txt
legacyMediaParserTests.txt => legacyMedia.txt
extraParserTests.txt => badCharacters.txt
This reduces redundancy and filename length now that we've got a dozen or so
of these.
Change-Id: I2c1461341fcb3f3cef8e49b5575cd30f3a38e500
* Unclear if this is actually useful or helpful, but at least the
legacy and Parsoid test modes do all setup close together now.
* See I776b22a973fbcc945106fbe807376b858dd79e48 for a more detailed
explanation. This patch is a followup from that patch.
Change-Id: I720d31c3573b57b66956bdd98d11e4f89df9cca0
* There are two places in the parser test runner where test options
are processed: perTestSetup, and setupParserOptions. There isn't
enough guidance as to what should be set up where and why.
But, anything that is really a wiki config/global should ideally
be set up in perTestSetup because the ParserOptions object loads
all its defaults from the config values anyway.
* Parsoid gets wiki-specific options from SiteConfig and page-specific
options from PageConfig.
In 61c14054, "externallinktarget" was treated as a wiki-specific opt
by adding it to SiteConfig. So, if Parsoid is to see a test opt,
(a) the corresponding config/global should be initialized
(b) it should be processed before SiteConfig is constructed
* So, all that said, any test option that impacts Parsoid's SiteConfig
should therefore be set in perTestSetup as well whch also matches
my understanding of how it should be set up for legacy tests as well.
* This fix now ensures that external link target tests pass in Parsoid's
integrated mode as well.
* Based on above, couple other things that I'll fix up in followups:
- There are a couple other configs (maxtemplatedepth, maxincludesize)
that are being initialized in setupParserOptions which should
probably be moved to perTestSetup.
- runLegacyTest calls perTestSetup and setupParserOptions back-to-back
but runParsoidTest doesn't do that. Those are separated by code
that sets up SiteConfig and DataAccess. This shouldn't matter if the
above guidelines for initialization are followed appropriately,
but just for consistency's sake, I might try to move those two
closer together in another followup.
Followup-To: f7d6c0190d
Change-Id: I776b22a973fbcc945106fbe807376b858dd79e48
Eliminate a difference between the magic variable (no-arg) and
parser function (1-arg) forms; aka the difference between
{REVISIONTIMESTAMP}} and {{REVISIONTIMESTAMP:{{PAGENAME}}}}.
This is a follow up to I8d25755e4d92bd91988cfb706d85bdb170abb207.
The magic variable contains a MAX_TTS optimisation which reduces
the use of vary-revision-timestamp, since it has severe performance
implications; this patch applies the same optimisation to the
parser function.
The ParserTestRunner has a small issue with test setup:
ParserOptions::setTimestamp() was called with a unix-format timestamp,
where it expected a TS_MW format timestamp. This issue was
fixed, along with tweaking the test timestamps so that a timestamp
coming from ParserOptions would still be distinguishable from one
coming from the revision.
Bug: T204370
Change-Id: I883d42d67013b6fb0da57c61e715b51d3a807879
As a follow-up to Ife14d7319ecbb7b588cbac97fae109b14230c44a, we want to
check that the legacy parser and Parsoid align on what's expected for
nofollow/noreferrer/noopener. Since the wgExternalLinkTarget setting is
modified on the Parsoid side during specific tests, we align the test
runner on the legacy parser to be able to test for this.
Bug: T186241
Change-Id: I2824182eaa5c50cd3e8e0193726d896da285a3a8
* This ensures that titles don't conflict across test runs.
I am surprised this hasn't bitten any one in non parsoid
parser test runs in case titles are repeated across test files.
* I wonder if this was also the source of some of the transient
errors seen in T312709 and T312517.
* Scribunto calls fetchModuleFromParser in two places and passes
it a title. In one case, the title comes from Title::newFromText
(in the call in LuaEngine.php when asked to load a module from
within Lua code). In another case, the title comes from
Title::makeTitleSafe (in Hooks.php to run #invoke on a module).
Title::newFromText uses a title cache whereas Title::makeTitleSafe
constructs a new title.
And, it turned out that in Parsoid runs, the revision lookup
for the module had the wrong revid when it came from Lua code.
This then caused the package lookup to fail only when the module
was loaded in Lua code.
That this test only failed in Parsoid test runs is because we
ran Parsoid tests after legacy parser tests. If we had switched
the order, legacy parser tests would have failed.
I haven't connected all the dots from the cached title in Title.php
to it getting an incorrect revid (but not an incorrect pageid).
But, clearing caches in the parser test runner seems to fix
this problem.
I welcome someone else helping with fleshing out this trail as to
how the title caching leads to only the revid going out of sync.
* Since I could only reproduce this in Jenkins CI (and not locally),
debugging this turned out to be painful.
Bug: T316331
Change-Id: I7c419e1d508c337328db1d5132454fb805bfb1b5
* This should hopefully make the code a bit dryer and easier to read.
* While this should not matter in practice, when testing against Parsoid,
don't create two different mutable revision records for tests where
'lastsavedrevision' option is set.
Change-Id: I5da7ec9ec688df0a1f84f7c1817775fd94a1e092
This reverts commit 0442728ed8, as the issue was fixed upstream in Parsoid. Further, this "fix" inadvertently disabled all but "selser" test modes for Parsoid parser tests run by CI.
Depends-On: Iebd6f6b25aff779efaa55ce9c25eec117403884c
Change-Id: I34d719f43f11e1793b012395d359af2e260f8743
This matches the DB domain logic enforcement in the other class methods.
Cleanup several type hints in related code.
Change-Id: I6668852ccebe16349b38b5a793543bd3ec0a35b8
* processResults continued to report failures even when a test was
newly passing.
* Updated result processing to recognize newly passing tests.
* I removed the CI-focused performance optimization that recognized
an expected failure and bailed early without trying to determine
if there was an unexpected passing test. This could lead to devs
not recognizing changes to test results and not updating the known
failures file to remove newly passing tests.
Change-Id: Iee0b2b3388028b327c30d63b17394149dd8cdc42
In particular, there was a test of __NEWSECTIONLINK__ added by
4e717c0158 (way back when) which seemed
to be intended to check that the appropriate ParserOutputFlag was set,
but that output flag was never included in the `flags=` parser test
output. Address that to improve coverage.
Change-Id: Iccd80e63e9fd724bf7dfb7f243f823a2b9f9915d
* Prevent crashers when selser tests are skipped.
* Don't run processResults when running in 'cache' mode.
* Added a isCachingMode() helper to TestMode.php -- not sure
if this was overkill, but did it anyway.
Change-Id: I2b100bd852306b9ce1de69117c6ca7d7d9c9ce3f
* This matches behavior in standalone mode.
* If we want to run these tests (especially selser),
we should probably add html section. For wt2wt tests,
we may end up tweaking the wikitext to reduce spurious
wt2wt failures because of normalization.
* So for now, skip these tests.
Change-Id: I2b9444ecae1bbf012aebdb81a134ee07ed4582c7
* This let us pass mocked revisions in the parser test runner while
running in Parsoid mode.
* This leads to improvement in wt2html tests results where a revision
id is queried. I've verified this in the Cite extension repo as
also the main parserTests.text file but I cannot enable Parsoid
integrated testing on the main parser tests file without doing a
sweep over all parser tests and adding appropriate test sections
* Currently, PageConfigFactory doesn't have unit tests. Will look
into adding them separately in a followup.
* Moved the setupParsoidTransform function to a more suitable place
in the ParserTestRunner.php file.
Bug: T270310
Change-Id: I94d68c8528bb2f7b367c68d80d14ebc1ab904a7f
* 3 more tests now pass that would only fail when run with other tests
but pass when run with --filter.
Change-Id: Ib0f97cd29b55ff27d891eee1160b66da1e95129c
When a test sets options to change global config, we have to rebuild
the ParsoidSiteConfig instance to pick up the test options since
the service container caches the SiteConfig instance.
For now, we are unconditionally resetting SiteConfig before every
<test,mode> combination. At a later time, we can figure out a way
to do it more conditionally; filed T310283 for that.
Bug: T310283
Change-Id: Ia2dfd37e3da3e1e8b31ab917b1f831eaea5eb5c5
Previously versions of Parsoid represented the !!config section as
a string which then had to be parsed by the ParserTestRunner.
Bug: T307720
Depends-On: Ieaac08e8171fd4c2a233646a84caac70ec30db6f
Change-Id: I3c5666d061c469049bf150ff0fa3010cb907a493
We introduced a new type for "ParserTestMode" and then uniformly pass
this around alongside the ParserTest object itself. (We started by
using a string for the $mode but the "explicit changetree" mode
requires more structured data. We *could* encode this in JSON just to
keep the string type around, but it seems cleaner and more future
proof to wrap this in a proper class type.)
Removed the old TestFileReader wrapper class, which served only as
a thunk to convert the new Test class into the same sort of array
which the legacy parser test runner code expected. There's still
some remaining ::testToArray() usage here for the TestRecorder
framework, but that will be cleaned up in a future patch.
Also updated the TestRecorder with stronger types: use the same
<Test,TestMode> pair when invoking the TestRecorder, instead of the
loosely typed array we'd used previously.
Change-Id: Iec4c0c7972a655b4faccacc67bdc3ca66d4c163d
Run phan over classes in tests/parser
The dependency of classes between parser and phpunit is not clear.
Classes used by both possible needs part of /common/
Change-Id: I2ceca6b7cd447876c127ed3b14e09f479defbd93
This reverts commit 123f785cdf.
Reason for revert: This breaks parserTests.php script since the ParserTestRunner script has references to both TestFileReader classes and this patch ends up using the same reader everywhere.
Change-Id: I22bdd05f9a86674e88335cc9055f1ce6cfa128d1
The ParserTestRunner code was old enough that it used a number of different
mechanisms to supply defaults for options passed via an array; uniformly use
the standard PHP "array add" mechanism instead.
Change-Id: I1791f7f8b1ae2cc6f4dd23351e9810a98b598be1
Long term there isn't going to be a "TestFileReader" and
"ParsoidTestFileReader", the legacy test file reader is just a
temporary hack. And these aren't "Parsoid Test Files" they
are *parser test files* (ie, implementation independent,
more or less). Long story short, adding "Parsoid" to the
class name is a step in the wrong direction.
Change-Id: I9b520c39c6c5b52334c658ec2a97bd38612dbf61
* This adds all of Parsoid's abilities to the parserTests.php script
* Parsoid testing mode requires a "--parsoid" CLI option.
* This supports "--wt2html" and the other testing modes,
"--updateKnownFailures", "--knownFailures", and "--changetree"
options that are part of the Parsoid's parserTests.php script.
* Verified that the output of parserTests.php test run matches
the output of the PHPUnit test run on the Cite extensions'
citeParserTests.txt file.
Change-Id: I87a1f621d0a2579ac47fa80d34cf74d4456bcf62
This patch improves the parser test runner in the following ways:
1. Adds Parsoid selser auto-edit test support
Since it isn't possible to dynamically add tests in PHPUnit
(unlike the home-grown parser test runners), to support selser
auto-edit tests, we create a single composite test that includes
all the individual edits. If a single test fails, the whole test
is considering failing.
The test output from all the individual tests are combined into
composite expected / actual strings and test failure diffs reveals
diffs for individual tests.
Future patches could enhance output further.
2. Ensure that we skip HTML-based Parsoid test modes without appropriate
input HTML
3. Pass additional options to the parser test runner via the
ParserTestTopLevelSuite constructor. These support extended
functionality of the Parsoid parser test runner which are being
made available to core.
At the moment the constructor will attempt to JSON-decode the
PARSERTEST_FLAGS environment variable to supply defaults for
these flags, which provides a hacky sort of way to pass arguments
on the command line. This will be integrated with the existing
$flags passed to ParserTestTopLevelSuite in the future (T308367).
4. Add support for running tests in specific modes as well
as ability to update the known failures json file.
This includes some missing code to run per-test teardown functions
in runParsoidTest.
Unlike the Parsoid parser test runner, you can combine
updateKnownFailures with --filter as well as with subset of
test modes, including possibly specific selser changetrees.
If `{"updateKnownFailures": true}` is set in the PARSERTEST_FLAGS var,
you can also update the knownFailures result for only the tests
that are run.
However, note that this could change the sort order in the
*-knownFailures.json file. This leads to dirty (git) diffs when
a full updateKnownFailures is run on the file. I am not going to
tackle this issue for now since this is an obscure use case that
is unlikely to be used much, if at all.
Overall, if I copy over the -knownFailures.json file from the Parsoid
repo into the Cite repo and update it with a phpunit.php run in core,
it updates the known failures to reflect changes to test results from
an integrated test run. A second run doesn't lead to any additional
changes to the knownFailures.json file or to any test failures. I had
to fix some bugs (referenced in the previous paragraph) to ensure
this idempotence beyond the first run.
This patch also cleans up some function signatures to use Test object
It maps the Test object from the Parsoid repo to an associative array
that a bunch of (legacy) parser test functions expect.
This patch squashes the following previous patches:
I2f46b4b94b44aec76e33403455eb466899376a6b
Ib01991089ec410dd45f77a20b136c8e05d855fa4
Id0ef01e333580892aa6d415b8f66c4393b06870d
Change-Id: I9952ab5ee3edef8e2f00b0e65cbffc42f46c2ea7
