It caused a 20% latency regression by unconditionally parsing extension.json
files on every single load instead of using the existing caching
infrastructure. There are further problems with the use of parsing/loading
extension.json files in a method that is incompatible with the existing
architecture.
This primarily reverts commit 46eabe275c.
Also needed to revert 16381261ae and 7c72347ec1.
Bug: T258664
Change-Id: I34a783c3f0df0447876a26441bb2d12e02368871
Decouple Installer services
Implement injection class Autoloader and i18n messages from extension.json
Implement extension selector by type
Add i18n message key `version-database`
Extensions for testing:
- https://github.com/MWStake/PerconaDB - real Percona extension
- https://github.com/killev/mediawiki-dbext2 - fake extension for test
Bug: T226857, T255151
Change-Id: I9ec8a18ad19283f6be67ac000110ac370afc0815
Jenkins, or more precise, junit, already detects PHPUnit's risky tests.
This is currently mapped to the "UNSTABLE" build status for Jenkins jobs.
Our Zuul confguration does consider UNSTABLE as a failure, so it does
result in the appropiate V-1 score already.
But, "UNSTABLE" is confusing in the Gerrit comments from jenkins-bot,
but it is merely about a PHPUnit test, for example:
https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php72-docker/28970/
This patch lets phpunit fail loudly and explicitly on risky tests,
and thus makes them easier to find and fix.
Change-Id: I95a54d1e4c8bf6b6abcc698bc14c2db5ce6c2a7d
Follows-up 214750d8d2 which created this directory, but forgot
to add it in tests/phpunit/suite.xml.
This was only realized this week, 9 months later, as part of
I29f5e761b977c where an integration test was fixed due to
observing a failure locally when running it directly, and this
not having been caught by CI when the failure was introduced,
because the suite is not being run.
Change-Id: I9edb8fad870ec4345c102e3cc2d280f872d20a22
We don't currently use this flag, and if we were to start doing so, this
would make development impossible on Windows machines as php-invoker is
POSIX-only.
Bug: T243324
Change-Id: I114518ec626020945b948e355eff7fd204fa4ef3
This test file seems to be marked as broken from the beginning, however,
this feature should get some test coverage. This change tries to
re-enable the tests and they should run.
However, there're some parts of the tests that didn't make any sense to
me, like checking the JobQueue after uploading a file, which also breaks
because of a HtmlCacheUpdate job, which might or might not be finished
after uploading a file by URL. However, for the test intention, I don't
see any reason why we should assert on that.
Change-Id: If829c5ac193e4843a443d8beb65e9b79cd691eb3
Follows-up 8373f51cb0 which applied this change to
"/phpunit.xml.dist", which is only used by 'composer phpunit',
whereas the CI coverage jobs use phpunit.php.
Change-Id: I35ad4db9453c6020dcb9fc91cc8ca47f9e1fca26
Going directly to 8 is too much pain.
Depends-On: Iafcea151389663d79e70ad6ab3741ce8b2e25cac
Bug: T192167
Change-Id: I77ea560f4a5344bd1c356ecc3e7857968ad829b8
Following discussion in Ibb8175981092d7f41864e641cc3c118af70a5c76, this patch
proposes to further reduce the scope of what unit tests may access, by removing
the loading of DefaultSettings and GlobalFunctions.php. This also has the
implied effect of disabling the storage backend, as well as the global service
locator.
MediaWikiTestCase is renamed to MediaWikiIntegrationTestCase so it's scope and
purpose is more clear. Whether we still need to keep `@group Database`
annotation around is debatable, as it's unclear to me what the performance costs
are of implying database access for all tests which extend IntegrationTestCase.
As far as I can tell, `@group Database` is primarily used in CI to run faster
tests before slower ones, and with the new UnitTestCase the annotation seems
redundant.
To run all testsuites, use `composer phpunit`. Other composer scripts:
- `composer phpunit:unit` to run unit tests
- `composer phpunit:integration` to run integration tests
- `composer phpunit:coverage` to generate code coverage reports from unit
tests (requires XDebug).
Note that you can pass arguments to composer scripts with `--`, e.g. `composer
phpunit:integration --exclude-group Dump`.
Other changes:
- Rename bootstrap.php to bootstrap.maintenance.php so it's clear it's part of
the legacy PHPUnit-as-maintenance-class setup
- Create new bootstrap.php which loads the minimal configuration necessary for
the tests, and do additional setup in the run() method of the unit/integration
test case classes
- Move the unit-tests.xml file to phpunit.xml.dist in preparation for this being
the default test configuration
For a follow-up patch:
- Find unit/integration tests for extensions/skins
- Migrate other test suites from suite.xml
- Support running all tests via vendor/bin/phpunit
Bug: T84948
Bug: T89432
Bug: T87781
Change-Id: Ie717b0ecf4fcfd089d46248f14853c80b7ef4a76
This changeset lays down the basic groundwork required to implement
T89432 and related tickets and is based on exploration done
at the Prague Hackathon. The goal is to identify tests in MediaWiki core
that can be run without having to install & configure MediaWiki and its dependencies,
and provide a way to execute these tests via the standard phpunit entry point,
allowing for faster development and integration with existing tooling like IDEs.
This changeset creates a new subdirectory under phpunit/ and organizes it
into a separate test suite. The environment for this suite is set up
via a PHPUnit bootstrap file without a custom entry point. For B/C, this
directory is also registered in suite.xml, to ensure that existing CI jobs
still pick up tests in the new suite.
For initial testing, a single test class, PasswordFactoryTest, was moved
to this new suite.
You can run the new suite using the follwoing command:
$ vendor/bin/phpunit -d memory_limit=512M -c tests/phpunit/unit-tests.xml
Bug: T84948
Bug: T89432
Bug: T87781
Change-Id: I69b92db3e70093570e05cc0a64c7780a278b321a
This changeset implements T89432 and related tickets and is based on exploration
done at the Prague Hackathon. The goal is to identify tests in MediaWiki core
that can be run without having to install & configure MediaWiki and its dependencies,
and provide a way to execute these tests via the standard phpunit entry point,
allowing for faster development and integration with existing tooling like IDEs.
The initial set of tests that met these criteria were identified using the work Amir did in
I88822667693d9e00ac3d4639c87bc24e5083e5e8. These tests were then moved into a new subdirectory
under phpunit/ and organized into a separate test suite. The environment for this suite
is set up via a PHPUnit bootstrap file without a custom entry point.
You can execute these tests by running:
$ vendor/bin/phpunit -d memory_limit=512M -c tests/phpunit/unit-tests.xml
Bug: T89432
Bug: T87781
Bug: T84948
Change-Id: Iad01033a0548afd4d2a6f2c1ef6fcc9debf72c0d
Otherwise I get errors every time I try to run PHPUnit on includes/ or
includes/parser, because it tries to run ParserIntegrationTest.php and
fails. Apparently the <exclude> in suite.xml doesn't work if PHPUnit is
invoked on a directory.
Bug: T201278
Change-Id: I7d09576bee2705d8516152e8fa671da8dac40233
* Exclude the data files from PHPUnit coverage.
* Add tests covering the normalize() implementations.
* Fix a small todo about using data providers.
* Set explicit visibility.
Change-Id: Ib104cc3215a36901cff853ad5969d92a6e0cf6a0
Indexing these files takes about 5-10% of the run time when generating
code coverage reports for core. It has no coverage at the moment and
probably won't any time soon given it's just variable declarations.
At least for now, exclude it to significantly speed up generating of
coverage reports.
Change-Id: I0357e004e96d4f4b19036474c8353ddb778e31f3
As of a few releases ago, there are no longer any skins in core
in that directory (base classes and fallback are under includes/).
Removing this makes running phpunit coverage faster locally,
when there are skins installed. This should be excluded, similar
to how extensions/ is not indexed for the core report, either.
Change-Id: Idc01578bf22febc684dbcb7aeb9ec9475e930ae4
Skins, like extensions, can register PHPUnit tests.
The testsuite interface in PHPUnit is designed to execute one run only,
so this should be based on the semantic entry of "skin jobs".
Changing the Jenkins job to run both doesn't work (only one value accepted).
Working around by running phpunit twice also seems wrong.
Bug: T170880
Change-Id: Ic04f4a842b2912574826b01b00d49db93d51c99b
I was bored. What? Don't look at me that way.
I mostly targetted mixed tabs and spaces, but others were not spared.
Note that some of the whitespace changes are inside HTML output,
extended regexps or SQL snippets.
Change-Id: Ie206cc946459f6befcfc2d520e35ad3ea3c0f1e0
This also fixed the testLoggersAreRestoredOnTearDown
test which was broken, and factors it out into
3 seperate tests
Bug: T151081
Change-Id: I00d29b501fa84db22e3bcd3d5642c49b2e99d4a0
Merge the PHPUnit parser test runner with the old parserTests.inc,
taking the good bits of both. Reviewed, pared down and documented the
setup code. parserTests.php is now a frontend to a fully featured
parser test system, with lots of developer options, whereas PHPUnit
provides a simpler interface with increased isolation between test
cases.
Performance of both frontends is much improved, perhaps 2x faster for
parserTests.php and 10x faster for PHPUnit.
General:
* Split out the pre-Setup.php global variable configuration from
phpunit.php into a new class called TestSetup, also called it from
parserTests.php.
* Factored out the setup of TestsAutoLoader into a static method in
Maintenance.
* In Setup.php improved "caches" debug output.
PHPUnit frontend:
* Delete the entire contents of NewParserTest and replace it with a
small wrapper around ParserTestRunner. It doesn't inherit from
MediaWikiTestCase anymore since integrating the setup code was an
unnecessary complication.
* Rename MediaWikiParserTest to ParserTestTopLevelSuite and made it an
instantiable TestSuite class instead of just a static method. Got rid
of the eval(), just construct TestCase objects directly with a
specified name, it works just as well.
* Introduce ParserTestFileSuite for per-file setup.
* Remove parser-related options from phpunit.php, since we don't
support them anymore. Note that --filter now works just as well as
--regex used to.
* Add CoreParserTestSuite, equivalent to ExtensionsParserTestSuite,
for clarity.
* Make it possible to call MediaWikiTestCase::setupTestDB() more than
once, as is implied by the documentation.
parserTests.php frontend:
* Made parserTests.php into a Maintenance subclass, moved CLI-specific
code to it.
* Renamed ParserTest to ParserTestRunner, this is now the generic
backend.
* Add --upload-dir option which sets up an FSFileBackend, similar
to the old default behaviour
Test file reading and interpretation:
* Rename TestFileIterator to TestFileReader, and make it read and buffer
an entire file, instead of iterating.
* The previous code had an associative array representation of test
specifications. Used this form more widely to pass around test data.
* Remove the idea of !!hooks copying hooks from $wgParser, this is
unnecessary now that all extensions use ParserFirstCallInit. Resurrect
an old interpretation of the feature which was accidentally broken: if
a named hook does not exist, skip all tests in the file.
* Got rid of the "subtest" idea for tidy variants, instead use a
human-readable description that appears in the output.
* When all tests in a file are filtered or skipped, don't create the
articles in them. This greatly speeds up execution time when --regex
matches a small number of tests. It may possibly break extensions, but
they would have been randomly broken anyway since there is no
guarantee of test file execution order.
* Remove integrated testing of OutputPage::addCategoryLinks() category
link formatting, life is complicated enough already. It can go in
OutputPageTest if that's a thing we really need.
Result recording and display:
* Make TestRecorder into a generic plugin interface for progress output
etc., which needs to be abstracted for PHPUnit integration.
* Introduce MultiTestRecorder for recorder chaining, instead of using
a long inheritance chain. All test recorders now directly inherit from
TestRecorder.
* Move all console-related code to the new ParserTestPrinter.
* Introduce PhpunitTestRecorder, which is the recorder for the PHPUnit
frontend. Most events are ignored since they are never emitted in the
PHPUnit frontend, which does not call runTests().
* Put more information into ParserTestResult and use it more often.
Setup and teardown:
* Introduce a new API for setup/teardown where setup functions return a
ScopedCallback object which automatically performs the corresponding
teardown when it goes out of scope.
* Rename setUp() to staticSetup(), rewrite. There was a lot of cruft in
here which was simply copied from Setup.php without review, and had
nothing to do with parser tests.
* Rename setupGlobals() to perTestSetup(), mostly rewrite. For
performance, give staticSetup() precedence in cases where they were
both setting up the same thing.
* In support of merged setup code, allow Hooks::clear() to be called
from parserTests.php.
* Remove wgFileExtensions -- it is only used by UploadBase which we
don't call.
* Remove wgUseImageResize -- superseded by MockMediaHandlerFactory which
I imported from NewParserTest.
* Import MockFileBackend from NewParserTest. But instead of
customising the configuration globals, I injected services.
* Remove thumbnail deletion from upload teardown. This makes glob
handling as in the old parserTests.php unnecessary.
* Remove math file from upload teardown, math is actually an extension
now! Also, the relevant parser tests were removed from the Math
extension two years ago in favour of unit tests.
* Make addArticle() private, and introduce addArticles() instead, which
allows setup/teardown to be done once for each batch of articles
instead of every time.
* Remove $wgNamespaceAliases and $wgNamespaceProtection setup. These were
copied in from Setup.php in 2010, and are redundant since we do
actually run Setup.php.
* Use NullLockManager, don't set up a temporary directory just for
this alone.
Fuzz tests:
* Use the new TestSetup class.
* Updated for ParserTestRunner interface change.
* Remove some obsolete references to fuzz tests from the two frontends
where they used to reside.
Bug: T41473
Change-Id: Ia8e17008cb9d9b62ce5645e15a41a3b402f4026a
Since in several cases, with an all-in-one commit, git's file rename
detection failed, I split the renames out into their own commit to
make review easier. Some changes here won't make complete sense without
the following commit.
* Moved TestsAutoLoader to tests/common/. It will be joined by a friend.
* Renamed ParserTest to ParserTestRunner, since the former name was
overly generic.
* Renamed TestFileIterator to TestFileReader. Please see the subsequent
commit for rationale.
* Moved parserTests.php to tests/parser/. It was the only file left in
tests/, and it should have been moved to tests/parser years ago,
analogous to phpunit.php.
* Renamed NewParserTest to ParserIntegrationTest. This was a tricky one,
apparently the name has to end in "Test" or else the structure test
will fail. Analogous to ParserMethodsTest etc. Rationale: because it's
not new anymore.
* Renamed MediaWikiParserTest to ParserTestTopLevelSuite and moved it to
the suites directory. A more descriptive name. Being in suites/
shields it from StructureTests, and is correct anyway.
Change-Id: Iddc6eaf815fdd64b3addb8570b4b6303ab99d634
It's very common these days for test runs on ordinary rigs to skip hundreds of
tests, which makes getting the part of the results you care about (the test
failures) an arduous trip up hundreds of lines in your terminal buffer.
Change-Id: Iaf714b86473379e1ad32e4d2d93e83018a757782
The MediaWiki test suite is painfully slow and delays merging of
changes. More than half of the time is spent in
ParserTest_Parser⁄parserTests::testParserTest which is the PHPUnit
wrapping class for the parser tests.
This patch let us extract the parser tests so we can run them
independently. By running them parallely with the rest of the tests,
that will speed up the gate processing time.
Mark the MediaWikiParserTest and NewParserTest class as belonging to the
test group 'ParserTests'. Will let us filter them out via PHPUnit
option --exclude-group
Introduce a new PHPUnit test suite 'parsertests' which loads the
MediaWiki core parser tests wrapper 'MediaWikiParserTest' and the suite
which loads the extensions parser tests (ExtensionsParserTestSuite.php).
This way we can run solely the parser tests with:
cd tests/phpunit
php phpunit.php --testsuite ParserTests
Wikimedia CI can then be configure to run two jobs:
A) php phpunit.php --exclude-group ParserTests
B) php phpunit.php --testsuite ParserTests
Bug: T114314
Change-Id: Ie819bab43163995048c073691c4c5d258f797c02
To add integration jobs for the MediaWiki skins, we will use the 'skins'
test suite. Add the structure and less validator tests to it. The
skins would then be tested using something like:
php tests/phpunit/phpunit.php --testsuite skins
Bug: T68926
Change-Id: I1be4454b1aa7935ec2e9ac3703eee3907e5101e8
The tests in the structure PHPUnit testsuite are used to verify the
Autoloader, ResourceLoader definitions and some basic other tests.
They are not run by Jenkins for MediaWiki extensions since we invoke:
phpunit.php --testsuite extensions
Add the structure directory to the 'extensions' testsuite. That will
start enforcing them on all extensions, so need to be merged with care.
We will want to backport this patch on all release branches we support
and make sure they pass for all extensions/branches :-/
Bug: T78798
Change-Id: Icb88fb3f10d203cfcb61c36c9c92864b92d6739c
Add the LessFileCompilationTest test case class, which represents the
validation of a LESS file by compilation.
Add the LessTestSuite test suite, which tests all LESS files registered
with the ResourceLoader, and use this to rewrite the checkLess.php
maintenance script.
Bug: 54665
Change-Id: Iedb8dc31e4817d8b4e40b655cf9b8fb092979e90
This has the effect of excluding the tests themselves from the code coverage
and including files that no test loads.
Change-Id: I65674ca3569d1ae6484b6769366213d71c0bdbc4
To better control what we are really covering, this patch uses PHPUnit
forceCoversAnnotation option. That makes it to record covering if and
only if @covers is used.
Also rewrapped the options to uses tabulations.
Change-Id: I467a5077a65b93cce08072a8d088b0b9c652a426
Parser tests are registeredd by appending one or more .txt files to the
$wgParserTestFiles global setting. Since this is shared with MediaWiki
core, I have made MediaWikiParserTest a factory of PHPUnit testsuite
which would filter in/out extensions tests.
The `extensions` test suite now has a second test suite builder which is
simply a wrapper around MediaWikiParserTest factory.
Play cases:
$ php phpunit.php --group Parser --tap
Runs any parser tests including the ones coming from extensions.
With an extension having parser tests such as Cite:
$ php phpunit.php --testsuite extensions --tap
// Extensions tests are run including parser tests.
bug: 42506
Change-Id: Icc3e9d30706b32149aa9dd18552e4241ec4af67e
The phpunit root directory has two test file:
AutoLoaderTest.php
StructureTest.php
The later was registered in phpunit under the `structure` test suite
while the former was not registered and hence never run (bug 47750).
This patch moves both files under the `structure` subdirectory and
change the suite to look in that directory. That will avoid us having
to manually maintain a list of test files.
Updated the __DIR__ in StructureTest.php.
Change-Id: I419c9157f32bdf7e1ff26a42f4bb3f3922b7be37
default .. 2s -> 10s
medium ... 10s -> 30s
I originally thought that using PHPUnit timeout system was a good idea
but that is causing more harm than being helpful. Now default to ten
seconds.
Change-Id: If10a9b6df967130838a253d986b2ed623d8e3b0a
PHPUnit has a timeout protection system which let us put tests in three
categories having different timeout (default: 1s, 10s, 60s). The timeout
only happens when using strict mode and having PHPUnit Invoker
installed.
Recently, the continuous integration server has been upgraded and the
PHPUnit Invoker was installed to let us run a serie of tests related to
dumping the database to XML. They do need some timeout system. Suddenly,
some tests started failing.
By default all tests are in the 1 second timeout group. Since we use a
sqlite backend and the server hard disk can get busy, some test can take
more than one second to execute.
This patch raise the default (small group) timeout to 2seconds and
explicitly define the value for the two other groups (medium and large)
using PHPUnit default values.
Change-Id: I2f5613b0c0215023c413dec8e84804175d8c8d6e
Adds tests/phpunit to the include_path to let PHPUnit find out our files.
Also add to change the suite.xml paths from './path' to 'path' so
stream_resolve_path() correctly find out the suite by using include_path
suite.xml now comes with colors=true. Under Windows, we override the
setting by forging a --no-colors. One can still force color usage
under windows by using --colors.
