wiki.techinc.nl/tests/phpunit/MediaWikiCoversValidator.php

58 lines
2 KiB
PHP
Raw Normal View History

<?php
/**
* Copyright (C) 2017 Kunal Mehta <legoktm@debian.org>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
*
*/
use PHPUnit\Framework\CodeCoverageException;
use PHPUnit\Util\Test;
/**
phpunit: Speed up MediaWikiCoversValidator trait Using a data provider for a dynamic or otherwise potentially large amount of cases tends to come with significant overhead due to all the setup and test abstraction involved. Especially when the host class has expensive setUp() or tearDown(), logic such as the case for API integration tests and other test cases that have `group Database`. Each of the test methods invokes the DB reset logic for no good reason. As an example, ApiQueryRecentChangesIntegrationTest takes about 55-60 seconds to run locally for me on master, but takes only 35-40s after this patch. Before: > slow tests (>50ms): > … > 44. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #25 > 45. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #2 > 46. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #7 > 47. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #8 > 48. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #14 > 49. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #17 > 50. 106ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #18 > ...and there are 3 more above your threshold hidden from view > > Time: 59.13 seconds > OK (53 tests) After: > slow tests (>50ms): > … > 26. 122ms to run ApiQueryRecentChangesIntegrationTest:testValidCovers > 27. 108ms to run ApiQueryRecentChangesIntegrationTest:testApiTestGroup > > Time: 40.8 seconds > OK (27 tests) And that's just for running ApiQueryRecentChangesIntegrationTest.php in isolation. I expect the CI jobs overall to come down by much more than that. Bug: T225730 Change-Id: Idae95f7c6f44c02371e2abc6e41d79b9eaced425
2020-03-08 18:32:53 +00:00
* Check that `@covers` tags are valid. PHPUnit only does this when generating
* code coverage reports, which is slow and we generally don't do that when
* running tests during development and pre-merge in CI.
*
* @since 1.31
*/
trait MediaWikiCoversValidator {
/**
phpunit: Speed up MediaWikiCoversValidator trait Using a data provider for a dynamic or otherwise potentially large amount of cases tends to come with significant overhead due to all the setup and test abstraction involved. Especially when the host class has expensive setUp() or tearDown(), logic such as the case for API integration tests and other test cases that have `group Database`. Each of the test methods invokes the DB reset logic for no good reason. As an example, ApiQueryRecentChangesIntegrationTest takes about 55-60 seconds to run locally for me on master, but takes only 35-40s after this patch. Before: > slow tests (>50ms): > … > 44. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #25 > 45. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #2 > 46. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #7 > 47. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #8 > 48. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #14 > 49. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #17 > 50. 106ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #18 > ...and there are 3 more above your threshold hidden from view > > Time: 59.13 seconds > OK (53 tests) After: > slow tests (>50ms): > … > 26. 122ms to run ApiQueryRecentChangesIntegrationTest:testValidCovers > 27. 108ms to run ApiQueryRecentChangesIntegrationTest:testApiTestGroup > > Time: 40.8 seconds > OK (27 tests) And that's just for running ApiQueryRecentChangesIntegrationTest.php in isolation. I expect the CI jobs overall to come down by much more than that. Bug: T225730 Change-Id: Idae95f7c6f44c02371e2abc6e41d79b9eaced425
2020-03-08 18:32:53 +00:00
* Assert that all "test*" methods in the host class have valid `@covers` tags.
*
phpunit: Speed up MediaWikiCoversValidator trait Using a data provider for a dynamic or otherwise potentially large amount of cases tends to come with significant overhead due to all the setup and test abstraction involved. Especially when the host class has expensive setUp() or tearDown(), logic such as the case for API integration tests and other test cases that have `group Database`. Each of the test methods invokes the DB reset logic for no good reason. As an example, ApiQueryRecentChangesIntegrationTest takes about 55-60 seconds to run locally for me on master, but takes only 35-40s after this patch. Before: > slow tests (>50ms): > … > 44. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #25 > 45. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #2 > 46. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #7 > 47. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #8 > 48. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #14 > 49. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #17 > 50. 106ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #18 > ...and there are 3 more above your threshold hidden from view > > Time: 59.13 seconds > OK (53 tests) After: > slow tests (>50ms): > … > 26. 122ms to run ApiQueryRecentChangesIntegrationTest:testValidCovers > 27. 108ms to run ApiQueryRecentChangesIntegrationTest:testApiTestGroup > > Time: 40.8 seconds > OK (27 tests) And that's just for running ApiQueryRecentChangesIntegrationTest.php in isolation. I expect the CI jobs overall to come down by much more than that. Bug: T225730 Change-Id: Idae95f7c6f44c02371e2abc6e41d79b9eaced425
2020-03-08 18:32:53 +00:00
* Don't use a data provider here because this assertion will be run many
* thousands of times in CI, and the implicit overhead from PHPUnit with
* generating and executing a test case around each becomes rather significant
* at that scale. Also, when using a data provider, the setUp() and tearDown()
* of the host class would be re-run for every check, which becomes very
* expensive for integration tests that involve databases.
*/
phpunit: Speed up MediaWikiCoversValidator trait Using a data provider for a dynamic or otherwise potentially large amount of cases tends to come with significant overhead due to all the setup and test abstraction involved. Especially when the host class has expensive setUp() or tearDown(), logic such as the case for API integration tests and other test cases that have `group Database`. Each of the test methods invokes the DB reset logic for no good reason. As an example, ApiQueryRecentChangesIntegrationTest takes about 55-60 seconds to run locally for me on master, but takes only 35-40s after this patch. Before: > slow tests (>50ms): > … > 44. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #25 > 45. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #2 > 46. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #7 > 47. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #8 > 48. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #14 > 49. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #17 > 50. 106ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #18 > ...and there are 3 more above your threshold hidden from view > > Time: 59.13 seconds > OK (53 tests) After: > slow tests (>50ms): > … > 26. 122ms to run ApiQueryRecentChangesIntegrationTest:testValidCovers > 27. 108ms to run ApiQueryRecentChangesIntegrationTest:testApiTestGroup > > Time: 40.8 seconds > OK (27 tests) And that's just for running ApiQueryRecentChangesIntegrationTest.php in isolation. I expect the CI jobs overall to come down by much more than that. Bug: T225730 Change-Id: Idae95f7c6f44c02371e2abc6e41d79b9eaced425
2020-03-08 18:32:53 +00:00
public function testValidCovers() {
$class = static::class;
foreach ( get_class_methods( $this ) as $method ) {
if ( strncmp( $method, 'test', 4 ) === 0 ) {
phpunit: Speed up MediaWikiCoversValidator trait Using a data provider for a dynamic or otherwise potentially large amount of cases tends to come with significant overhead due to all the setup and test abstraction involved. Especially when the host class has expensive setUp() or tearDown(), logic such as the case for API integration tests and other test cases that have `group Database`. Each of the test methods invokes the DB reset logic for no good reason. As an example, ApiQueryRecentChangesIntegrationTest takes about 55-60 seconds to run locally for me on master, but takes only 35-40s after this patch. Before: > slow tests (>50ms): > … > 44. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #25 > 45. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #2 > 46. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #7 > 47. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #8 > 48. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #14 > 49. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #17 > 50. 106ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #18 > ...and there are 3 more above your threshold hidden from view > > Time: 59.13 seconds > OK (53 tests) After: > slow tests (>50ms): > … > 26. 122ms to run ApiQueryRecentChangesIntegrationTest:testValidCovers > 27. 108ms to run ApiQueryRecentChangesIntegrationTest:testApiTestGroup > > Time: 40.8 seconds > OK (27 tests) And that's just for running ApiQueryRecentChangesIntegrationTest.php in isolation. I expect the CI jobs overall to come down by much more than that. Bug: T225730 Change-Id: Idae95f7c6f44c02371e2abc6e41d79b9eaced425
2020-03-08 18:32:53 +00:00
try {
Test::getLinesToBeCovered( $class, $method );
} catch ( CodeCoverageException $ex ) {
$this->fail( "$class::$method: " . $ex->getMessage() );
}
}
}
phpunit: Speed up MediaWikiCoversValidator trait Using a data provider for a dynamic or otherwise potentially large amount of cases tends to come with significant overhead due to all the setup and test abstraction involved. Especially when the host class has expensive setUp() or tearDown(), logic such as the case for API integration tests and other test cases that have `group Database`. Each of the test methods invokes the DB reset logic for no good reason. As an example, ApiQueryRecentChangesIntegrationTest takes about 55-60 seconds to run locally for me on master, but takes only 35-40s after this patch. Before: > slow tests (>50ms): > … > 44. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #25 > 45. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #2 > 46. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #7 > 47. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #8 > 48. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #14 > 49. 107ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #17 > 50. 106ms to run ApiQueryRecentChangesIn…t:testValidCovers with d… #18 > ...and there are 3 more above your threshold hidden from view > > Time: 59.13 seconds > OK (53 tests) After: > slow tests (>50ms): > … > 26. 122ms to run ApiQueryRecentChangesIntegrationTest:testValidCovers > 27. 108ms to run ApiQueryRecentChangesIntegrationTest:testApiTestGroup > > Time: 40.8 seconds > OK (27 tests) And that's just for running ApiQueryRecentChangesIntegrationTest.php in isolation. I expect the CI jobs overall to come down by much more than that. Bug: T225730 Change-Id: Idae95f7c6f44c02371e2abc6e41d79b9eaced425
2020-03-08 18:32:53 +00:00
$this->addToAssertionCount( 1 );
}
}