When running unit tests, the bootstrap would previously load all
extensions and skins in the filesystem. This was OK for an initial
implementation, but is not acceptable if we want to eventually do that
for all PHPUnit entry points (once we'll have a single config and
bootstrap). Instead, it's desirable to only load the extensions
specified in LocalSettings.php. The problem is that it's pretty much
impossible to load LocalSettings.php without also loading the rest of
MediaWiki, with all the side effects this might have.
This patch introduces a helper script that loads all the config, then
prints what extensions and skins were loaded. The bootstrap file runs
this script via proc_open and then reads the list of extensions to
load. Because the script is run in a separate process, any side effects
only affect the spawned process, not the one where PHPUnit is running.
Currently, there doesn't seem to be a better way to obtain the list of
extensions loaded in LocalSettings.php without all the other side
effects. YAML settings
(https://www.mediawiki.org/wiki/Manual:YAML_settings_file_format) would
probably help, but that's very far from becoming the only supported
config format (if it will ever be).
Also add two TestSuite implementations to replace the '*' wildcard in the
extensions:unit and skins:unit suites. These use the same list of loaded
extensions to determine where to look for tests.
And last but not least: my most sincere apologies to you if the hack
you're seeing here has ruined your day. If you think a better approach
exists, please tell me and I'll be so relieved!
Bug: T227900
Change-Id: Ib578644b8a4c0b64dca607afb9eb8204ca7fc660
This test was not being run because the docs directory wasn't included
in the PHPUnit XML config files. As a direct consequence, the test was
extremely broken: it used the non-namespaced DumpTestCase class which no
longer exists, it referenced XML files with the wrong path, and then the
test had other non-error failures to fix.
Change-Id: I188cbb1de68d39c7a484d4c6f808678af80d11ab
Since the latter is going to be removed, make sure that the former has
everything we need. In particular:
- Add failOnRisky=true from suite.xml
- Reorder config options and add comment about stderr from suite.xml
- Remove non-existent tests/phpunit/skins and
tests/phpunit/documentation from the test suites. These cause an error
when trying to run all the suites with `composer phpunit`.
- Leave the other suites as they are. Compared to suite.xml,
phpunit.xml.dist has separate suites for core, extensions, and skins
unit tests.
- Leave includeUncoveredFiles to false. The rationale in
I3d19627fa36f6cc6666c29fdb638272fdaa30630 seems convincing. CI already
sets it to true: https://w.wiki/72qE
- Leave slowThreshold to 100 as per
Ia6ed404d4d4cc8a7b4b8a48b232f644f400f8103. I believe that change
simply missed suite.xml. The same rationale that led to the threshold
being increased for unit tests is valid a fortiori for integration
tests, which are generally slower.
Rename SuiteDirectoryTest and make it check phpunit.xml.dist too.
Bug: T227900
Change-Id: Ib4b47b337870dffc61dd44817a21d11809075d5b
This resolves an existing TODO in the code. Note that hooks were
deprecated in PHPUnit 9 and replaced by events in PHPUnit 10. However,
PHPUnit 9 does not provide any alternative. Also, PHPUnit 10 has a
requirement of PHP >= 8.1, so we won't update to it any time soon.
Note that this extension isn't needed for unit tests, but it's added to
the unit config file anyway for consistency, given we're going to have a
single config file in the near future.
Bug: T227900
Change-Id: I0a79ee8a9f7f8c8a176acfad9bd26486d038d31f
We currently output 50 slow tests in integration test runs. I'm not sure
if anyone is taking action based on this report, but if so, it seems
like reporting the top 10 (or even fewer, top 5?) would be sufficient.
Change-Id: Id2eed09e404a00090671bc67271d35be5c93fdf3
Move tests from /languages/ to /includes/language/ similiar to the
classes in f5644ba
Clean up suite config too not longer run that folder as test,
remove from coverage, there no php files to cover
Remove converter tests from Language*Test files,
all identical to the *ConverterTest cases (or remove the whole file)
Change-Id: I098ba241bd61d8ed2ee9b65393416077e3544e79
* DeprecatedHooksTest: Don't use assertContains().
* Replace uses of deprecated asserts:
- assertFileNotExists() -> assertFileDoesNotExist()
* Update hierarchy of MediaWikiPHPUnitResultPrinter, since ResultPrinter
is an interface in PHPUnit 9.
* Remove temporary forward-compat methods.
* Remove directories that don't exist from tests/phpunit/suite.xml, since
they now make PHPUnit exit:
- tests/phpunit/skins, it used to have SideBarTest, then moved to
tests/phpunit/includes/skins
- tests/phpunit/documentation, it used to have ReleaseNotesTest, then
moved to tests/phpunit/unit/documentation
* Update configuration with --migrate-configuration and reformat.
* Avoid redefining getMockBuilder() in
ActionModuleBasedHandlerTestTrait, use a @method annotation instead.
* In RCCacheEntryFactoryTest, avoid using internal PHPUnit logic for
HTML validation, and use native PHP methods instead. The code was
copied from Xml::load (moved to \Xml\Loader::load in PHPUnit 9) and
simplified for this use case.
Bug: T243600
Bug: T262076
Change-Id: I851b9158b73d0cfc315eed9d63b15c54b05895e3
* Add stderr=true
* Set php.ini values that are needed when tests are run via composer and/or vendor/bin/phpunit
Bug: T90875
Bug: T227900
Change-Id: I11c34b02e9a18e0a724d14b3bfb2b80c7333fcba
This caused unexpected problems with no obvious fixes. Needs more work.
This reverts commit 7238dff532.
Bug: T310255
Bug: T90875
Change-Id: I3758cbb6d0029b20ec1b0f67dbf2f422031c50ae
* switch to phpunit.xml.dist instead of suites.xml
* switch composer.json to vendor/bin/phpunit
* tests/phpunit/phpunit.php is retained but will be removed after CI
jobs and other references on
codesearch (https://codesearch.wmcloud.org/search/?q=tests%2Fphpunit%2Fphpunit.php&i=nope&files=&excludeFiles=&repos=)
are removed
* add a default bootstrap.integration.php; unit tests in
composer.json use the non-MW bootstrap file (bootstrap.php)
* Migrate the phpunit.php logic into tests/phpunit/BootstrapIntegrationWrapper.php
Depends-On: I19d560bdcdb2ee914ab055e094841f2b5db8be55
Depends-On: Ib23209fc3b095e3c012ed84ce5c11f8b2d27b898
Co-authored-by: Daimona Eaytoy <daimona.wiki@gmail.com>
Bug: T227900
Bug: T90875
Change-Id: I82045c207738d152d5b0006f353637cfaa40bb66
Switch PHPUnit to use the more verbose TestDox format, which allows
us to see exactly which tests have been executed. This is useful when
making changes to test configuration.
Use the MediaWikiPHPUnitResultPrinter to override the CliTestDoxPrinter
and use the actual class names / test method names, instead of the
prettified versions from TestDox, to improve greppability.
Bug: T297287
Change-Id: I9d8561ff868e6ff0a44d32120f2799b34d25e618
Exceptions:
* keep core:unit instead of "unit" from suite.xml
* remove extensions:integration and skins:integration until we decide we
need them. Quibble uses 'phpunit' with no arguments which invokes all
suites, so we don't want to run these long running suites needlessly.
We could copy them back in the future if Quibble always invokes
phpunit with a list of testsuites.
Bug: T227900
Change-Id: Id57e0bb2fb9f3eff7d3360ad1e98d5ac1a4d8816
PHPUnit 8.5.21 no longer converts deprecations to exceptions by default.
However, we rely on such conversions in our tests, otherwise there are
failures:
Failed asserting that exception of type "PHPUnit\Framework\Error\Deprecated" is thrown.
The upstream change is:
fac02620f6
Bug: T291731
Co-Authored-byː Antoine Musso <hashar@free.fr>
Change-Id: Ifacf26905a7fa24ea3aafad79e012ddfb0a797bb
* Turn normalize data into an autoloadable class and use the class name
as key instead of a partial file name with a base directory.
* This protected function is not aimed at re-use in extensions,
has no other usage known to Codesearch. Mark it internal,
and remove the compat introduced by 31283f34bf.
* Update the now ten-year-old documentation comments that described
this as "Temporary". The functionality and comment were introduced
in 2010 by Tim with r60599 (ad19c032b0).
Ref <https://bash.toolforge.org/search?p=0&q=temporary>.
* Replace the broken wikitech-l archive link with a working one,
and describe the thread for easier future reference.
Bug: T225756
Change-Id: I004da1f25c19e1095d0917349f2ba8461f9f0715
PHPUnit parses the directory using the standard PHP glob() function; a
Bash-style “globstar” double asterisk, to match any number of
directories on a path, is not supported.
Change-Id: Iab408a4328e012777245227f5dfff40aaea27510
We don't currently use this flag, and if we were to start doing so, this
would make development impossible on Windows machines as php-invoker is
POSIX-only.
Bug: T243324
Change-Id: I114518ec626020945b948e355eff7fd204fa4ef3
Also move Benchmarker.php to maintenance/includes/ because
that file is not itself a benchmark, and that base class is
actually covered by tests and should remain included.
Change-Id: I4acd88242dde56a884d319dfc141a3511a8221a3
Going directly to 8 is too much pain.
Depends-On: Iafcea151389663d79e70ad6ab3741ce8b2e25cac
Bug: T192167
Change-Id: I77ea560f4a5344bd1c356ecc3e7857968ad829b8
Changing addUncoveredFilesFromWhitelist to false drastically speeds up code
coverage report generation. Including extensions/skins in the config also
removes the need to manually edit your phpunit.xml file when you want to inspect
code coverage for your extension or skin locally.
With this patch you can very quickly generate coverage for an extension/skin or
subsection of core, this is especially useful if you are using an IDE that
supports running a test and importing its coverage into the editor.
The downside to this patch is that the coverage report only contains
information about code that's exercised by the tests, so this patch on its own
is unsuitable for our current processes in integration/config which generate a
full code coverage report per patch for extensions or post-merge for core.
However, since we are not yet using phpunit.xml.dist in CI, this patch should be
OK to merge as is. A follow-up patch to integration/config will use sed or
phpunit-suite-edit.py to set processUncoveredFilesFromWhitelist to true.
Bug: T192078
Change-Id: I3d19627fa36f6cc6666c29fdb638272fdaa30630
The idea is that `vendor/bin/phpunit` could run all of the tests contained in
core + extensions + skins with no overlap, but that we can also provide
granularity for CI which wants to run some testsuites (e.g. core:unit and
core:integration) before extensions/skins, as well as running unit before
integration tests.
Bug: T87781
Change-Id: I2fb57f4a9a58ec7a4042d9cc1821cc484661a6b9
For now only extensions with skin/extension.json which follow PSR-4 are
supported.
DefaultSettings is loaded in bootstrap to work around errors when unit tests are
run followed by integration tests (e.g. `vendor/bin/phpunit`)
Usage:
- composer phpunit:unit will run all unit tests in core and any tests in
`{extensions/skins}/tests/phpunit/unit` which also extend
MediaWikiUnitTestCase, and which have an extension.json file
- Pass a specific directory to only run unit tests for a particular extension,
e.g. `composer phpunit:unit -- extensions/GrowthExperiments`
Bug: T226911
Change-Id: I237a9f82e4d1b05cf2f08b3e4bb7ffcd8d47111c
Following discussion in Ibb8175981092d7f41864e641cc3c118af70a5c76, this patch
proposes to further reduce the scope of what unit tests may access, by removing
the loading of DefaultSettings and GlobalFunctions.php. This also has the
implied effect of disabling the storage backend, as well as the global service
locator.
MediaWikiTestCase is renamed to MediaWikiIntegrationTestCase so it's scope and
purpose is more clear. Whether we still need to keep `@group Database`
annotation around is debatable, as it's unclear to me what the performance costs
are of implying database access for all tests which extend IntegrationTestCase.
As far as I can tell, `@group Database` is primarily used in CI to run faster
tests before slower ones, and with the new UnitTestCase the annotation seems
redundant.
To run all testsuites, use `composer phpunit`. Other composer scripts:
- `composer phpunit:unit` to run unit tests
- `composer phpunit:integration` to run integration tests
- `composer phpunit:coverage` to generate code coverage reports from unit
tests (requires XDebug).
Note that you can pass arguments to composer scripts with `--`, e.g. `composer
phpunit:integration --exclude-group Dump`.
Other changes:
- Rename bootstrap.php to bootstrap.maintenance.php so it's clear it's part of
the legacy PHPUnit-as-maintenance-class setup
- Create new bootstrap.php which loads the minimal configuration necessary for
the tests, and do additional setup in the run() method of the unit/integration
test case classes
- Move the unit-tests.xml file to phpunit.xml.dist in preparation for this being
the default test configuration
For a follow-up patch:
- Find unit/integration tests for extensions/skins
- Migrate other test suites from suite.xml
- Support running all tests via vendor/bin/phpunit
Bug: T84948
Bug: T89432
Bug: T87781
Change-Id: Ie717b0ecf4fcfd089d46248f14853c80b7ef4a76
