These .htaccess files are intended to prohibit all web access. But if
the user sets "Satisfy Any" on a parent directory, in conjunction with
any permissive require directive like "Require all granted", access will
be allowed despite "Require all denied" in .htaccess.
So, override Satisfy so that the "Require all denied" will reliably take
effect.
Note that "Satisfy All" is the default. This only affects non-default
installations.
Change-Id: Ia5862fb69e439b7ea2ed7af011e1ebf8f1b1f6d6
(cherry picked from commit a50d2e69f8ce9e5720b05615d04c35cc9008b6ae)
Why:
- When tests are run in parallel, they are organized by groups--it is
useful to be able to see logs organized by each test group.
What:
- Set an environment variable in ComposerLaunchParallel,
`MW_PHPUNIT_SPLIT_GROUP_ID`
- If the environment variable is set, modify the file names used for
logs in DevelopmentSettings to use `split-group-{group ID}` in the
file name
- CI script for checking if any errors have been logged was updated
in I4283478a89d2a8d3a0502f836459b3fb21d99113
Bug: T375852
Change-Id: If5e08558a6efbb0139c320fda6f53ec73597ba60
Make it clearer in the class documentation that we can’t use most
MediaWiki services, including Shell::command(); that said, as far as I
can tell we should be able to use Shellbox (as it’s a separate library),
so do that.
Change-Id: I73f29214a1527bbbd3fd5c22b7476bdaa4627b03
In T361190 and Quibble 1.9.0, we introduced parallel execution of
PHPUnit tests to speed up the CI jobs. The existing implementation
is purely Python/Quibble, and cannot directly be used by developers
locally. With this patch, we re-implement the parallel test
execution already implemented in CI as a composer task so that the
parallel tests can be run locally.
The `phpunit:parallel:extensions` command expects to be run after
`phpunit:prepare-parallel:extensions`, and expects to find 8 test
suites with the names `split_group_X` (for X in 0 through 7) in the
PHPUnit configuration file. 8 here is currently a hard-coded number
that corresponds to the number of parallel test executions we need
to saturate the CPU of a 4-core developer machine, and experimentally
leads to a good speed-up versus the serial execution.
When this command runs, it forks to launch 8 parallel processes,
each running one of the `split_group_X` suites. The parent process
waits for the children to complete, buffers the output, collects the
exit statuses, then dumps the buffered output and exits with a
non-zero status if any of the child processes failed (i.e. if there
were test failures).
We introduce `phpunit:prepare-parallel:default` as a complement to
`phpunit:prepare-parallel:extensions`, and the two commands
`phpunit:parallel:database` and `phpunit:parallel:databaseless`.
This creates four possible combinations - two different test suites,
and two different test groups. This is a similar setup to that which
we have in CI - the Database and non-Database tests are run in
separate groups, and some jobs use the `extensions` suite while
others just use the default suite.
The `phpunit:parallel:...` commands will fail with a helpful message
if no `split_group_`s are found in the active PHPUnit configuration.
To help test whether the split test runs are really running all the
tests in the suite, we generate and store the PHPUnit results cache
file. Comparing the results cache files from linear versus parallel
runs should tell us if all the tests have been executed.
Bug: T365976
Change-Id: If106802f08edd5d4c841bb7970c69b88ab3bb39b
In T361190 and Quibble 1.9.0, we introduced parallel execution of
PHPUnit tests to speed up the CI jobs. The existing implementation
is purely Python/Quibble, and cannot directly be used by developers
locally. With this patch, we re-implement the test splitting logic
already implemented in CI as a composer task so that the parallel
tests can be run locally.
There are a couple of different approaches to running PHPUnit tests
in parallel. The different approaches have been discussed at length
in T50217. Ideally, we would just install the `paratest` extension
and use that to parallelise the execution. Unfortunately we have
complex test suites (specifically Parser tests and the Scribunto
test suite) that dynamically create tests as they run, which makes
it hard for `paratest` to work out which tests will run.
To overcome this limitation, we use the `phpunit --list-tests`
function to create a list of test classes that would be included in
the execution of the test suite, then scan the filesystem for
classes named in the `tests-list.xml` output. The classes we find
are then collected into smaller groups (`split_group_X`) which we
can run in parallel in separate processes.
We split into 7-8 groups here, as that experimentally leads to an
even spread of the tests and consumes 100% of all cores on a 4-core
processor.
Because `ParserIntegrationTest.php` is a single test class that
generates thousands of integration tests, we put that in its own
bucket rather than allocating it round-robin to one of the split
buckets. This again helps to keep the buckets roughly the same size.
The current implementation only supports splitting the `extensions`
test suite. We need to do some more development and testing to
support splitting other suites.
The new composer command `phpunit:prepare-parallel:extensions` will
generate a `phpunit.xml` file with the same contents as
`phpunit.xml.dist`, but with the split-group suites added. The
result of running all of the split groups should be the same as the
result of running the whole test suite.
Bug: T365976
Change-Id: I2d841ab236c5367961603bb526319053551bec2e
This is internal class is exclusively used by LockFileChecker,
which in turn is used by maintenance/checkComposerLockUpToDate.php,
which in turn is used by PHPUnit and maintenance/update.php.
Both of these cases have to use workarounds to disable localisation,
because they run on the CLI (no request context or user language),
and without access to MediaWikiServices, Database, or LocalSettings.
Follows-up:
* T330228: I2caddfb8f2 (5ecbb62f7f): Add composer check to PHPUnit.
* T333412: Skip composer check for mediawiki-vendor Jenkins jobs.
* If6110eeaec (397e00444a): Fix invisible pre-fatal errors messages.
* T283389: I0d56f23560 (20e77793b8): Refactor and add localization.
* I995a1cb01a (d5d585b9b2): Fix "premature access" during PHPUnit.
* I33bf81087c2: Fix MessageCache crash during update.php.
The localization was added for T283389, but ended up not using it,
since all it does is check isGood and then render a custom interface
message localised as part of the Installer. Hence, remove this to
categorically remove and avoid these problems.
This change effectively reverts I995a1cb01a (d5d585b9b2) and thus
restores the more detailed and useful error details when failing
in the context of a PHPUnit command.
Change-Id: Ia30d39d7e064e9ff0d8397d25918d0bb2ea10914
"Deny from all" is deprecated; the replacement syntax has been
available since Apache 2.4 (originally released in 2012).
See <https://httpd.apache.org/docs/2.4/howto/access.html>.
Bug: T360850
Change-Id: I825053ccefe34f6ca4e04af5ad2601f79e4d51a7
If you try to do `composer update` using a git checkout of a Composer
branch, it fails because "@package_version@" is less than 2.0.0.
So in that case, use Composer::getVersion() which falls back to
Composer::SOURCE_VERSION. The method doesn't exist on very old versions
of Composer, so we can't use it unconditionally.
Change-Id: I53ea01c2f992499a25ba851bd8ed3593c9a13711
Introduce LockFileChecker that used to check whether
composer-installed dependencies (no-dev) are up-to-date.
Bug: T283389
Change-Id: I0d56f235604d5c856bae5d170230f8c7ca0729c6
Dependency of an extension upon particular MediaWiki versions should
be expressed in extension.json. As brought up on wikitech-l, we still
have the code that injects a virtual mediawiki/mediawiki package into
Composer so that extensions can express a dependency upon particular
MediaWiki versions in composer.json. This is duplicate information
potentially introducing inconsistency. This patch removes support for
this capability. Per codesearch, there are no remaining extensions
found using this feature.
The classes in includes/composer are removed immediately as they've
never been intended for extensions to use, while MediaWikiVersionFetcher
is marked for standard deprecation.
Bug: T467
Bug: T249573
Change-Id: I62bacca45e4680812dd42f4e061ca7da17bfcdbb
There is a dedicated directory for this. It should go there.
Since it's not really part of MW, there is no sense in making it
namespaced (doubly so with MediaWiki\). The whole thing should be
removed.
Bug: T321882
Change-Id: I18d7a7542d9445be535e6aa535c68759682a115e
* Xml::loadFile was moved to a new class in PHPUnit 9.5, cf.,
c331d28ac3
* PHPUnit XML coverage element were renamed in
7a494d1fa4
Bug: T320452
Change-Id: Iea9783005522f2851e86a4b56aa0771ed31442bb
In composer starting at version 2.2.0, Package::setProvides must be
called with a map of lowercased package name => Link object rather
than an indexed array.
Bug: T298261
Change-Id: Ic178c00f588a0ff8e4d5fe74c3d6bfb3ebf9d394
Only throw exception if composer 2 is run, and 1.x of
wikimedia/composer-merge-plugin is still installed
Bug: T266419
Change-Id: I55fda5e554ade8df309076eff6b60332dde524f5
Remove the requirement for composer-plugin-api 1.1, so that Composer 2.0
can be used.
Throw an exception if the user tries to update with an old version of
composer-merge-plugin using Composer 2.0.
Change-Id: I0302e250f4a4fd834aee7bd84bd3f2da4fc77700
Many files were in the autoloader despite having potentially harmful
file-scope code.
* Exclude all CommandLineInc maintenance scripts from the autoloader.
* Introduce "NO_AUTOLOAD" tag which excludes the file containing it
from the autoloader. Use it on CommandLineInc.php and a few
suspicious-looking files without classes in case they are refactored
to add classes in the future.
* Add a test which parses all non-PSR4 class files and confirms that
they do not contain dangerous file-scope code. It's slow (15s) but
its results were enlightening.
* Several maintenance scripts define constants in the file scope,
intending to modify the behaviour of MediaWiki. Either move the
define() to a later setup function, or protect with NO_AUTOLOAD.
* Use require_once consistently with Maintenance.php and
doMaintenance.php, per the original convention which is supposed to
allow one maintenance script to use the class of another maintenance
script. Using require breaks autoloading of these maintenance class
files.
* When Maintenance.php is included, check if MediaWiki has already
started, and if so, return early. Revert the fix for T250003 which
is incompatible with this safety measure. Hopefully it was superseded
by splitting out the class file.
* In runScript.php add a redundant PHP_SAPI check since it does some
things in file-scope code before any other check will be run.
* Change the if(false) class_alias(...) to something more hackish and
more compatible with the new test.
* Some site-related scripts found Maintenance.php in a non-standard way.
Use the standard way.
* fileOpPerfTest.php called error_reporting(). Probably debugging code
left in; removed.
* Moved mediawiki.compress.7z registration from the class file to the
caller.
Change-Id: I1b1be90343a5ab678df6f1b1bdd03319dcf6537f
Combine `setLinkAsProvides`, `newMediaWikiLink`,
and `getMediaWikiVersionConstraint` single use
private methods into `setProvidesMediaWiki`
Change-Id: I212ce9f25a85b27a9ff87fe9ae993e23626d0444
Our current recommendation for adjusting PHPUnit configuration for
extension code coverage reports is to manually modify
tests/phpunit/suite.xml. This is cumbersome and annoying, since
suite.xml is version controlled.
This patch proposes a composer script to simplify editing
the (non-version controlled) phpunit.xml config in the root of the
MediaWiki repo. It is useful if you are interested in generating
code coverage reports based on unit tests only, not integration tests.
Usage: composer phpunit:coverage-edit -- extensions/GrowthExperiments
When you're done, you can run `rm phpunit.xml` to switch back to the
default phpunit.xml.dist.
The script will use the default includes/src/maintenance directories,
but a future improvement would be to identify the relevant directories
via the AutoloadClasses property of extension.json.
Another improvement would be to provide an option for passing
arbitrary paths in core, in case you are working on tests for a
specific class and want to generate coverage reports instantly.
We could probably also remove
dockerfiles/quibble-coverage/phpunit-suite-edit.py (which inspired
this patch) and use composer phpunit:coverage-edit instead.
Bug: T100294
Change-Id: Ia0ef41f67ca4a64b0d1ca0ddcee488c29630af0b
I benchmarked this again. The runtime of an unlimited explode() can be
quite high. This is not really a DoS attack vector as it would require to
post megabytes worth of input to the code, which will hit many other
limits before. I still consider it good practice to use unlimited explode()
only when it is actually allowed to return an unlimited amount of elements.
Change-Id: I30f8ca5dba7b317bb4a046b9740fd736b4eea291
The /vendor directory does not need to be web accessible, and to reduce
attack surface, it should not be web accessible. We can use the
post-install-cmd and post-update-cmd hooks to create a .htaccess after
the user has run "composer install" or "composer update". On the first
run of composer, this hook will be invoked twice due to the composer
merge plugin.
If the htaccess file already exists, this hook won't do anything.
Bug: T180237
Change-Id: I2cf6541750c90b5708d7cf5f81b914ae2d9d46d1
The old VersionConstraint class is deprecated in modern versions
of composer which use the version from the composer/semver library.
Bug: T119590
Change-Id: I52ad75e7e25f6583bb4b13489a09d2f2d934e67d
These files have all had treatment before, and these occurrences have either
been missed or have been introduced after.
Change-Id: I06cdab4616b5bff47c85152df28f18c861730a23
- Added spaces after if/foreach/catch
- Added new line before end of file
- Added or removed spaces before/after parenthesis, comma
- Added spaces around string concat
Change-Id: I0590070f1b3542108e242730e8d9a3ba9831e94f
they support via Composer.
This change allows extensions to specify they depend on a specific
version or version range of MediaWiki. This is done by adding the
package mediawiki/mediawiki in their composer.json require section.
As MediaWiki itself is not a Composer package and is quite far away
from becoming one, a workaround was needed, which is provided by
this commit.
It works as follows. When "composer install" or "composer update"
is run, a Composer hook is invoked. This hook programmatically
indicates the root package provides MediaWiki, as it indeed does
when extensions are installed into MediaWiki. The package link
of type "provides" includes the MediaWiki version, which is read
from DefaultSettings.php.
This functionality has been tested and confirmed to work. One needs
a recent Composer version for it to have an effect. The upcoming
Composer alpha8 release will suffice. See
https://github.com/composer/composer/issues/2520
Tests are included. Composer independent tests will run always,
while the Composer specific ones are skipped when Composer is
not installed.
People that already have a composer.json file in their MediaWiki
root directory will need to make the same additions there as this
commit makes to composer-json.example. If this is not done, the
new behaviour will not work for them (though no existing behaviour
will break). The change to the json file has been made in such a
way to minimize the likelihood that any future modifications there
will be needed.
Thanks go to @beausimensen (Sculpin) and @seldaek (Composer) for
their support.
Change-Id: I8df66a92971146ab79cd4fcbd181e559115ca240
