Commit graph

247 commits

Author SHA1 Message Date
Kunal Mehta
6849ebc7d1 Maintenance: Don't try shelling out if it's disabled
Change-Id: Ib30c55aa9e048960f8fa66a41f7e2aa2fa03b253
2018-07-02 22:15:53 +00:00
Marius Hoch
2bb5329595 Maintenance::finalSetup: Make sure we re-create LBFactory
This seems to work anyway (as no connections are needed at
this time), but I guess it's more robust to explicitly
destroy any existing instances.

Bug: T147169
Change-Id: Id56a62d1830fc1464a80dd4420ffddd797bf8b51
2018-06-19 19:46:01 +02:00
Marius Hoch
d74413bbe7 Allow passing the default DB group to use in Maintenance scripts
Also allows setting it via a global, although that is probably
less useful.

Bug: T147169
Change-Id: Ic51204a6f6ce9db4cc96108e823e388512724eff
2018-06-12 23:52:40 +02:00
Tim Starling
72f2e98499 Exit maintenance script with non-zero status if execute() returns false
There's a whole lot of shutdown code in doMaintenance.php that is
skipped when you use exit() directly. In HHVM by default, destructors
are not even called.

There are a few cases where maintenance scripts want to do something after
doMaintenance.php returns, so returning null should continue to mean no exit.

Change-Id: I0891e2ee3af7ef2c64c03b70edcf9e281ce1e7ba
2018-05-31 16:46:14 +10:00
jenkins-bot
9b670fb797 Merge "maintenance: Document secondary purpose of --server" 2018-04-12 18:56:42 +00:00
Kunal Mehta
4ff2040358 Deprecate DO_MAINTENANCE
This has been replaced by RUN_MAINTENANCE_IF_MAIN for quite a while now,
but never officially deprecated.

The main motiviation for this is that tools like CodeSniffer have been
using RUN_MAINTENANCE_IF_MAIN to detect whether a file is a maintenance
script, and missed files that used the old constant.

Change-Id: Iebbe1ecaa7157ba22ebcac9c481592e09410a9f5
2018-04-12 09:44:51 -07:00
jenkins-bot
e2f61aa440 Merge "Fix --memory-limit option to maintenance scripts" 2018-04-06 22:12:03 +00:00
Max Semenik
d28d91e6cd Convert Maintenance to the new shell framework
Change-Id: Ibec8a84071fdf061f9831b2964143d99464cd4cc
2018-04-04 13:59:25 +00:00
Brian Wolff
fba014820b Fix --memory-limit option to maintenance scripts
Follow up 2f293bd0. Previously this option was marked as not
taking an argument, thus if present it would have the value of
true (aka 1). Setting memory limit to 1 results in setting to
256kb, which is way too small.

Change-Id: I2a6e512e384c5ef882f7ba2877091ecb8fe01e35
2018-03-28 01:20:07 +00:00
Timo Tijhof
af46a3351c maintenance: Document secondary purpose of --server
Follows-up to 2de3bf45d1. That commit references "Wikimedia Method"
of https://www.mediawiki.org/wiki/Manual:Wiki_family. This is slightly
confusing given that page explicitly states that hostname-based detection
is only used on the web server, with code for MW_DB and --wiki in place.

Nonetheless, despite this being seemingly unused, unadvertised
and undocumented, if we're going to support this, we might as well
document what its for, to avoid it being removed without notice in
the future.

Change-Id: I9ff333b6e9f15cfe4de21ef01a47931dd73d297e
2018-03-26 18:27:24 -07:00
Kevin Israel
06ba5ca383 Remove internal use of deprecated $wgRequestTime
* Use $_SERVER['REQUEST_TIME_FLOAT'] unconditionally in WebRequest.php
  and libs/Timing.php. WebStart.php was doing this already without issue.
  The key existst since PHP 5.4, for both Web and CLI (we require 5.5).

* In wfDebug() and wfReportTime(), use $_SERVER['REQUEST_TIME_FLOAT'] instead.

* In ApiFormatBase and MWDebug, use WebRequest::getElapsedTime() instead.

* In Maintenance.php, remove setting of $wgRequestTime.

* In rebuildFileCache.php, update mocking to $_SERVER['REQUEST_TIME_FLOAT']
  so that we avoid re-introducing bug T24852.

Change-Id: I1b647da2862f815029caa533b592ec8a05b33806
2018-03-18 04:41:12 +00:00
Umherirrender
74af4f507c Fix phpdoc for Maintenance::getDB
It has the same parameter as wfGetDB including $groups as string

Change-Id: Id531488e1c2bddce71946423010620323e08bc0a
2018-02-24 22:31:10 +01:00
Fomafix
d59af4c341 Use PHP's implode() with the suggested order of arguments
https://secure.php.net/manual/en/function.implode.php defines the order
of arguments as

 string implode ( string $glue , array $pieces )
 string implode ( array $pieces )

Note:
  implode() can, for historical reasons, accept its parameters in
  either order. For consistency with explode(), however, it may be less
  confusing to use the documented order of arguments.

Change-Id: I03bf5712204e283f52d3ede54af9b9ec117d4280
2018-02-22 20:24:00 +01:00
jenkins-bot
220bda9175 Merge "tests: Commit initial version of DevelopmentSettings.php" 2018-02-15 23:52:26 +00:00
Reedy
39f0f919c5 Update suppressWarning()/restoreWarning() calls
Bug: T182273
Change-Id: I9e1b628fe5949ca54258424c2e45b2fb6d491d0f
2018-02-10 08:50:12 +00:00
jenkins-bot
276e1296ac Merge "Don’t write to stderr when testing" 2018-01-26 14:12:46 +00:00
jenkins-bot
07a791ffd1 Merge "Set $_SERVER['SERVER_NAME'] to the value set by --server" 2018-01-24 15:24:57 +00:00
Lucas Werkmeister
dea7243303 Don’t write to stderr when testing
This change makes it possible to test error() messages with
expectOutputString() or expectOutputRegex().

Change-Id: I61970403f9c371d3798f34fd48c70bc72f0c7eda
2018-01-24 15:05:39 +01:00
Umherirrender
ad776c7d5f Use ::class to resolve class names in maintenance scripts
This helps to find renamed or misspelled classes earlier.
Phan will check the class names

Change-Id: I1d4567f47f93eb1436cb98558388e48d35258666
2018-01-23 17:40:16 +00:00
jenkins-bot
bdf062a8e9 Merge "Treat phpdbg as run from the command line when checking PHP_SAPI" 2018-01-08 23:03:41 +00:00
Marius Hoch
803a29127c Fix Maintenance::output when called very early
If a maintenance script is called with --help (or without
required args), this is currently failing, as MediaWikiServices
is only loaded after Maintenance::setup is initially called.
Follows 336454104d.

Change-Id: Ied283d362675b6b98bd3144132a240b432998991
2018-01-05 00:19:55 +01:00
Kunal Mehta
251a0b97e5 Treat phpdbg as run from the command line when checking PHP_SAPI
phpdbg is a gdb-style debugger for PHP that is run from the command
line. However, it has a different PHP_SAPI value, so it was impossible
to run maintenance scripts with it (until now).

To avoid having to check both PHP_SAPI values in a bunch of places,
introduce wfIsCLI() to easily check whether running from the
command-line or not.

We're (CI team) interested in generating code coverage with phpdbg
instead of xdebug, hence this patch.

Bug: T184043
Change-Id: Id1f994ca146d7858cd8bb6ab6cdbb7718ff524fb
2018-01-03 23:00:37 -08:00
Aaron Schulz
336454104d Try to opportunistically flush statsd data in maintenance scripts
This helps to avoid OOMs from buffer build-ups in the statsd
factory object. This piggy-backs on to the same checks used
for deferred update runs. In addition, the output() method
checks if the data size is getting large and emits if needed.

Bug: T181385
Change-Id: I598be98a5770f8358975815e51380c4b8f63a79e
2017-12-30 05:01:21 +00:00
Timo Tijhof
fdb7d36903 tests: Commit initial version of DevelopmentSettings.php
* Store in the includes/ directory.
* Use by default in Travis CI builds.
* Add command-line option to phpunit.php to use it.
* See commit ## for use by Jenkins jobs, which will load it directly
  from LocalSettings.php (instead of via phpunit) so that it applies
  to web entry points and other maintenance scripts as well.

Bug: T177669
Change-Id: I6e5dc5f0dddc1960761980552ed2bb31e6ae9fd9
2017-12-27 22:13:24 +00:00
Tim Starling
659f3779c3 Fix maintenance script failure when run as a child of a FastCGI worker
Use PHP_SAPI==cli instead of checking $_SERVER['REQUEST_METHOD'],
since $_SERVER is populated from the environment when running HHVM in
CLI mode. Environment variables set by a FastCGI worker thus leak
through to child processes run via the shell, and cause this check to
fail.

When I wrote this check in March 2004 (r2803), I didn't know about
PHP_SAPI. Checking PHP_SAPI is quite sufficient to prevent web execution,
we use it in other places.

Bug: T111441
Change-Id: Iad8469ee25df4b0e0c2371e7975a300b1695dd8d
2017-12-08 15:34:23 +11:00
Chad Horohoe
e7c86d8537 Remove some dumb headers with my name in it
Change-Id: Ib34b623b47fb92f1383bc5c55c50a16dd6d49abd
2017-11-30 15:39:46 -08:00
Bryan Davis
9e34eeff23 Maintenance: add fatalError() method
Deprecate the second argument to Maintenance::error() in favor of a new
Maintenance::fatalError() method. This is intended to make it easier to
review flow control in maintenance scripts.

Change-Id: I75699008638f7e99b11210c7bb9e2e131fca7c9e
2017-11-21 21:34:16 -07:00
Max Semenik
dac20d0ffa Introduce Maintenance::getBatchSize()
Just to isolate the internals. Fix most of usages in the core.

Change-Id: I8b3e9ca1f42b7c49ee57f17b88ca2fc7b404f342
2017-11-05 13:26:12 -08:00
physikerwelt
2de3bf45d1 Set $_SERVER['SERVER_NAME'] to the value set by --server
For wiki families that use the "Wikimedia Method" to have domain dependant wikis
the server variable 'SERVER_NAME' might be used to get the database name.
c.f. https://www.mediawiki.org/wiki/Manual:Wiki_family#Wikimedia_Method

This change sets $_SERVER['SERVER_NAME']  to the commandline option passed
as --server to enable the correct loading of the LocalSettings.php

Change-Id: I7e5c009c2403f6c93e81422a8376d7deee4d2b5a
2017-11-02 21:42:16 +00:00
Kunal Mehta
0e1fc6575e Refactor some Installer code into ExecutableFinder
Refactor Installer::locateExecutableInDefaultPaths() into a separate
utility class, ExecutableFinder. This class is already used in plenty of
places outside of the installer, so it's ripe for being extracted.

This class is located in utils/ due to the dependency upon
Shell::command(). Once that no longer has a dependence upon MediaWiki,
this class can be moved to libs/ too.

Change-Id: I175465acc0d64f990445ce05fabcee8b88a0b259
2017-10-26 11:42:05 -07:00
jenkins-bot
128881c1f1 Merge "Remove old workaround for HHVM" 2017-10-13 19:42:22 +00:00
Max Semenik
edda8bbf8f Remove old workaround for HHVM
The bug in question was fixed in HHVM 3.6.0, our minimum requirement is 3.6.5.

Change-Id: Id1d65cf438c7148064d747f09728ef4cb5f51b52
2017-10-12 17:38:34 -07:00
Max Semenik
0392d9bb57 Move wfCountDown() into Maintenance class
Doing this allows to restrict it to maintenance scripts and support
quiet mode.

Change-Id: Iad0858ce1fdd64f746d5f9d4a7d6ed96f21e94df
2017-10-12 17:24:46 -07:00
Umherirrender
f739a8f368 Improve some parameter docs
Add missing @return and @param to function docs and fixed some @param

Change-Id: I810727961057cfdcc274428b239af5975c57468d
2017-09-10 20:32:31 +02:00
Umherirrender
3f1a52805e Use short type bool/int in param documentation
Enable the phpcs sniffs for this and used phpcbf

Change-Id: Iaa36687154ddd2bf663b9dd519f5c99409d37925
2017-08-20 13:20:59 +02:00
Kunal Mehta
d1cf48a397 build: Update mediawiki/mediawiki-codesniffer to 0.10.1
And auto-fix all errors.

The `<exclude-pattern>` stanzas are now included in the default ruleset
and don't need to be repeated.

Change-Id: I928af549dc88ac2c6cb82058f64c7c7f3111598a
2017-07-22 18:24:09 -07:00
Aaron Schulz
dd359741cc Move DB errors to Rdbms namespace
Change-Id: I463bd86123501abc68fdb78b4cda6110f7af2549
2017-04-15 10:47:41 -07:00
Aaron Schulz
21e71e0235 Use IDatabase type hints in /maintenance
Relatedly, move lockTables()/unlockTables() to IMaintainableDatabase

Change-Id: Ib53e9fa948deb2f9a70f0ce16c002613d0060bf9
2017-04-07 23:37:41 +00:00
Aaron Schulz
488a647831 Move IDatabase/IMaintainableDatabase to Rdbms namespace
Change-Id: If7e8a8ff574661fd827de8bcec11d2c39a687300
2017-03-28 15:32:38 -07:00
Aaron Schulz
4a177b34ef Move LBFactory to Rdbms namespace
Change-Id: I5ae10783228d0252284807c9562bc8e328d4becb
2017-02-03 17:24:03 -08:00
addshore
b12086b618 Get ConfigFactory & MainConfig from MediaWikiServices
Change-Id: Iafdd7e00747060572463ffb05aae4543f3a06163
2016-11-23 00:12:38 +00:00
umherirrender
34fe90ac52 Remove empty lines at end of functions
It looks like there is something missing after the last statement
Also remove some other empty lines at begin of functions, ifs or loops
while at these files

Change-Id: Ib00b5cfd31ca4dcd0c32ce33754d3c80bae70641
2016-11-05 11:55:10 +01:00
Tim Starling
d831844364 Parser test file editor
Add editTests.php, which provides an interactive interface to the
parser tests, with semi-automated editing.

Change-Id: I1a20d007ba4627d562a16c03849bbad7aec0e516
2016-10-28 16:15:59 +11:00
Aaron Schulz
d1215eb7a5 Small DB classes IDEA warning fixes and cleanups
Change-Id: I65b5f452a364b329bb671540f4115dbc4f40eebd
2016-10-23 22:03:37 -07:00
Aaron Schulz
30f4b3c103 Replace DatabaseBase => Database in more places
Change-Id: If37a7909056bf2c31a8228cbc84f0fbbf5f1c517
2016-09-28 15:53:02 -07:00
Aaron Schulz
3a839c5927 Inject schema vars into DatabaseMysql
Also fix broken patchSql.php maintenance script

Change-Id: I45bccb0e2e10bd06651a551fa08ed0c66d11eb6a
2016-09-22 21:07:27 +00:00
Aaron Schulz
16b4e3a9f1 Avoid global state in DatabaseBase::factory()/query()
Change-Id: Ibb4f1c0dafea071a1c34e0cd5b5c15b8b4bb7bc6
2016-09-16 00:40:57 +00:00
Aaron Schulz
0e5cd18b74 Rename LBFactory => LBFactoryMW and make LBFactory in /libs
The former extends the later with MW-specific logic.

Also removed a wf* method call from ChronologyProtector.

Change-Id: I325f59b7467ab9c2137731d1ce69816f5a020f03
2016-09-15 17:14:32 +00:00
Tim Starling
6117fb244f Refactor parser tests
Merge the PHPUnit parser test runner with the old parserTests.inc,
taking the good bits of both. Reviewed, pared down and documented the
setup code. parserTests.php is now a frontend to a fully featured
parser test system, with lots of developer options, whereas PHPUnit
provides a simpler interface with increased isolation between test
cases.

Performance of both frontends is much improved, perhaps 2x faster for
parserTests.php and 10x faster for PHPUnit.

General:

* Split out the pre-Setup.php global variable configuration from
  phpunit.php into a new class called TestSetup, also called it from
  parserTests.php.
* Factored out the setup of TestsAutoLoader into a static method in
  Maintenance.
* In Setup.php improved "caches" debug output.

PHPUnit frontend:

* Delete the entire contents of NewParserTest and replace it with a
  small wrapper around ParserTestRunner. It doesn't inherit from
  MediaWikiTestCase anymore since integrating the setup code was an
  unnecessary complication.
* Rename MediaWikiParserTest to ParserTestTopLevelSuite and made it an
  instantiable TestSuite class instead of just a static method. Got rid
  of the eval(), just construct TestCase objects directly with a
  specified name, it works just as well.
* Introduce ParserTestFileSuite for per-file setup.
* Remove parser-related options from phpunit.php, since we don't
  support them anymore. Note that --filter now works just as well as
  --regex used to.
* Add CoreParserTestSuite, equivalent to ExtensionsParserTestSuite,
  for clarity.
* Make it possible to call MediaWikiTestCase::setupTestDB() more than
  once, as is implied by the documentation.

parserTests.php frontend:

* Made parserTests.php into a Maintenance subclass, moved CLI-specific
  code to it.
* Renamed ParserTest to ParserTestRunner, this is now the generic
  backend.
* Add --upload-dir option which sets up an FSFileBackend, similar
  to the old default behaviour

Test file reading and interpretation:

* Rename TestFileIterator to TestFileReader, and make it read and buffer
  an entire file, instead of iterating.
* The previous code had an associative array representation of test
  specifications. Used this form more widely to pass around test data.
* Remove the idea of !!hooks copying hooks from $wgParser, this is
  unnecessary now that all extensions use ParserFirstCallInit. Resurrect
  an old interpretation of the feature which was accidentally broken: if
  a named hook does not exist, skip all tests in the file.
* Got rid of the "subtest" idea for tidy variants, instead use a
  human-readable description that appears in the output.
* When all tests in a file are filtered or skipped, don't create the
  articles in them. This greatly speeds up execution time when --regex
  matches a small number of tests. It may possibly break extensions, but
  they would have been randomly broken anyway since there is no
  guarantee of test file execution order.
* Remove integrated testing of OutputPage::addCategoryLinks() category
  link formatting, life is complicated enough already. It can go in
  OutputPageTest if that's a thing we really need.

Result recording and display:

* Make TestRecorder into a generic plugin interface for progress output
  etc., which needs to be abstracted for PHPUnit integration.
* Introduce MultiTestRecorder for recorder chaining, instead of using
  a long inheritance chain. All test recorders now directly inherit from
  TestRecorder.
* Move all console-related code to the new ParserTestPrinter.
* Introduce PhpunitTestRecorder, which is the recorder for the PHPUnit
  frontend. Most events are ignored since they are never emitted in the
  PHPUnit frontend, which does not call runTests().
* Put more information into ParserTestResult and use it more often.

Setup and teardown:

* Introduce a new API for setup/teardown where setup functions return a
  ScopedCallback object which automatically performs the corresponding
  teardown when it goes out of scope.
* Rename setUp() to staticSetup(), rewrite. There was a lot of cruft in
  here which was simply copied from Setup.php without review, and had
  nothing to do with parser tests.
* Rename setupGlobals() to perTestSetup(), mostly rewrite. For
  performance, give staticSetup() precedence in cases where they were
  both setting up the same thing.
* In support of merged setup code, allow Hooks::clear() to be called
  from parserTests.php.
* Remove wgFileExtensions -- it is only used by UploadBase which we
  don't call.
* Remove wgUseImageResize -- superseded by MockMediaHandlerFactory which
  I imported from NewParserTest.
* Import MockFileBackend from NewParserTest. But instead of
  customising the configuration globals, I injected services.
* Remove thumbnail deletion from upload teardown. This makes glob
  handling as in the old parserTests.php unnecessary.
* Remove math file from upload teardown, math is actually an extension
  now! Also, the relevant parser tests were removed from the Math
  extension two years ago in favour of unit tests.
* Make addArticle() private, and introduce addArticles() instead, which
  allows setup/teardown to be done once for each batch of articles
  instead of every time.
* Remove $wgNamespaceAliases and $wgNamespaceProtection setup. These were
  copied in from Setup.php in 2010, and are redundant since we do
  actually run Setup.php.
* Use NullLockManager, don't set up a temporary directory just for
  this alone.

Fuzz tests:

* Use the new TestSetup class.
* Updated for ParserTestRunner interface change.
* Remove some obsolete references to fuzz tests from the two frontends
  where they used to reside.

Bug: T41473
Change-Id: Ia8e17008cb9d9b62ce5645e15a41a3b402f4026a
2016-09-12 16:11:42 +10:00
jenkins-bot
ba67060734 Merge "Make Maintenance::commitTransaction() avoid using deprecated method" 2016-09-06 06:16:42 +00:00