assertSame() is guaranteed to not do any type conversion. This can be
critical when acciden tially comparing, for example, 0 to 0.0.
Change-Id: Iffcc9bda69573623ba14af655dcd697d0fcce525
* Remove logic for saving slow-to-render parser output. This has
not worked ever since DerivedPageDataUpdater was introduced.
* Make the logic to use cached output actually work. This was
also broken since DerivedPageDataUpdater was added. In order
to pass the output, add a known-revision-output parameter
to both WikiPage::doSecondaryUpdates() and
DerivedPageDataUpdater::prepareUpdate().
* Also factored out some helper methods from runForTitle() in
RefreshLinksJob to make it more readable and avoid the need
for multiple transaction round commit calls. This makes the
case of multiple-title jobs less likely to break again.
* Make use of RefreshLinksJob::runForTitle() return value.
* Add unit tests for multiple-title job case.
Change-Id: I0cd13c424a87653b5a7253c42cd48fe43befd692
This changeset implements T89432 and related tickets and is based on exploration
done at the Prague Hackathon. The goal is to identify tests in MediaWiki core
that can be run without having to install & configure MediaWiki and its dependencies,
and provide a way to execute these tests via the standard phpunit entry point,
allowing for faster development and integration with existing tooling like IDEs.
The initial set of tests that met these criteria were identified using the work Amir did in
I88822667693d9e00ac3d4639c87bc24e5083e5e8. These tests were then moved into a new subdirectory
under phpunit/ and organized into a separate test suite. The environment for this suite
is set up via a PHPUnit bootstrap file without a custom entry point.
You can execute these tests by running:
$ vendor/bin/phpunit -d memory_limit=512M -c tests/phpunit/unit-tests.xml
Bug: T89432
Bug: T87781
Bug: T84948
Change-Id: Iad01033a0548afd4d2a6f2c1ef6fcc9debf72c0d
This reduces confidence in the test. There is no guruantee that
it won't return the same value twice during the duration of a full
PHPUnit run of all test suites, whether twice in a row or 20 minutes
apart.
For a test that needs a string of any kind, use an explicit, consinstent
and cheap literal value.
For a test that specifically needs some kind of uniqueness compared to
something else within the same test case, do so explicitly.
Tests that require something globally unique (for some undefined/vague
definition of "global") were not found, and should not exist anyway.
Also, in libs/objectcache tests, fix order of parameters in some
assertions (expected first, then actual), and use assertFalse/assertSame
instead of assertEqual for cases where false is expected to remove
tolerance of other loosely equal values.
Change-Id: Ifc60e88178da471330b94bfbf12e2731d2efc77d
* Remove duplicate $params check from Job::factory done in Job::__construct.
* In Job::factory(), restore use of a valid title as default for passing as
constructor arg to old job classes. Their constructor may expect it to
be valid.
Keep the invalid dummy in Job::__construct, and document why.
* tests: Update test case for failure mode when using Job::factory
with a class that requires a title. It asserted getting an invalid
title. This now restores the behaviour prior to fc5d51f129,
which is that job classes that require a title, get a valid one.
* tests: Remove test case for testToString that used
an explicitly passed but invalid params value. I've converted
that to expect the exception we now throw instead.
* tests: Update getMockJob(), also used by testToString, which was
relying on undocumented behaviour that 'new Title' is public
and gets namespace=0 and title=''. Before fc5d51f129,
title params weren't in toString() and it asserted outputting
three spaces (delimiter, empty string from formatted title,
delimiter).
In fc5d51f129, this changed to asserting "Special:" which
seems unintentional as we didn't pass it the internally reserved
NS_SPECIAL/'' value, and yet was caught by the dbkey=='' check.
Given this test case doesn't deal with titles, omit it for now.
A job can either have a $title and title/namespace in params,
or neither. This test was asserting an in-memory scenario
where $title can be an object, but title/namespace absent from
params.
Bug: T221368
Depends-On: I89f6ad6967d6f82d87a62c15c0dded901c51b714
Change-Id: I2ec99a12ecc627359a2aae5153d5d7c54156ff46
Simplify the code of jobs that do not care about titles and removes
the direct Title dependency from JobQueue. Remove getTitle() from
IJobSpecification itself. Move all the Job::factory calls into a
single JobQueue::factoryJob() method.
Depends-on: Iee78f4baeca0c0b4d6db073f2fbcc56855114ab0
Change-Id: I9c9d0726d4066bb0aa937665847ad6042ade13ec
Also moved some WikiMap/$wgJobClasses checks to JobQueueGroup::pop
which is the method callers are supposed to use.
Change-Id: I2ab82d8adc4ae1f54697d2935afa2053539cf2db
This already requires a DB domain ID, so there is no reason to have
hacks for trying to handle a wiki ID being passed in instead. If the
provided domain has a schema, it should not simply be ignored in the
comparison.
Change-Id: I9ced7a46fa05f32843a9a7d17391c5d0576b099c
These new classes provide a mechanism for defining the
behavior of slots, like the content models it supports.
This acts as an extension point for extensions that need
to define custom slots, like the MediaInfo extension
for the SDC project.
Bug: T194046
Change-Id: Ia20c98eee819293199e541be75b5521f6413bc2f
Note that we will still be re-parsing either the old or the new
revision. Keeping the rendered version of the old revision cached
for a bit would be nice, but ParserCache currently does not
support this.
Bug: T205369
Change-Id: I86d26e494924eec24e7b1fb32c424ac1284be478
Using domains means thats JobQueueDB has the right value to use for calls
like LoadBalancer::getConnection(). The full domain includes the schema in
the case of Postgres. This makes calls to getConnection() less awkward by
not relying on the fallback logic in reallyOpenConnection() for null schemas.
Make getWikiIdFromDomain/isCurrentWikiDomain account for the schema if it
is both defined and is not simply the generic "mediawiki" schema MediaWiki
uses by default. If all wikis use the default schema, the wiki IDs can get
by with DB/prefix alone, which various config and methods may be built around.
Otherwise, the config callbacks must account for schema and the config must
include it in various wiki domain ID lists to properly disambiguate wikis.
Also, clean up SiteConfiguration::siteFromDB() since it is not meant
to handle schemas unless the callback method was taylored to do so.
Finally, add more comments to DefaultSettings.php about already existing
limitations of wiki domain IDs and their components.
Change-Id: I8d94a650e5c99a19ee50551c5be9544318eb05b1
So far, RefreshLinksJob re-generated the ParserOutput ba calling
Content::getParserOutput. This only works for the main slot. It
needs to instead get the parser output for all slots combiend,
by using RevisionRenderer and RenderedRevision.
Bug: T174035
Change-Id: I253dda26bfa5aefa15f8b1dcc59e69fc7e9d0cb7
The "AUTO" means AUTOCOMMIT, not "automatic transactions"/DBO_TRX,
which is basically the opposite concept. The new name does not
suffer from that ambiguity.
Keep the old constant as an alias for backwards compatibility.
Also remove LoadBalancer comment about non-existing field
Change-Id: I63beeb061fc9be73f320308e4d6393b58628b8c8
When LoadBalancer opens new local domain connections, it currently
assumes that the domain specified by the server info array is the
same. For sanity, make sure that the handle is set to the local
domain.
The main LBFactory/LoadBalancer use $wgDBname/$wgDBprefix as the
local domain, corresponding with wfWikiId(). This relation is set
automatically in MWLBFactory. If $wgLBFactoryConf/$wgDBservers is
manually configured in a way breaking this correspondance, then it
is misconfigured.
Fixes made to avoid test failure:
* Make sure LoadBalancer::setDomainPrefix() updates the local
domain alias member. Also do not bother changing the domain of
foreign connections.
* Use the right domain ID for the connection array key names in
LoadBalancer::openForeignConnection().
* Now that JobQueueTest no longer mistakenly uses the non-test
tables, force it to use the main DB_MASTER handle so that it can
see the unit test tables even if they are TEMPORARY; such tables
are tied to the TCP connection, so separate handles see different
temporary tables.
Change-Id: I56f8b32fe957f984b8c9753e6db3b20abe96b038
Use $wgUpdateRowsPerQuery instead, and use its value at run-time
instead of inside the job parameters. Also helps with deduplication
if batchSize is changed.
Change-Id: Ifef25a3ae5ae2418359a41eba05778613fbb548f
The used phpcs has a bug, so the version 0.9.0 could not be enforced at the moment.
Will be fixed in next version, see T167168
Changed:
- Remove duplicate newline at end of file
- Add space between function and ( for closures
- and -> &&, or -> ||
Change-Id: I4172fb08861729bccd55aecbd07e029e2638d311
$wgJobClasses can now specify a factory function for creating a job,
instead of a class to be instantiated directly. This makes it possible
to inject services in a job constructor, and register a factory function
that calls the constructor with default services.
This follows Ieb85493a7765 and Ia2107dc5af78, which introduced factory
functions for API modules and special pages.
Change-Id: I0461e59da2a8fa6681e3b1fcdfc38bfed7f3ac32
The default will remain PHPUnit 4.x due to PHP 5.5 support.
But, we should allow developers to run tests with newer PHPUnit
versions which are noticably faster (especially for code coverage
reports).
* <https://github.com/sebastianbergmann/phpunit/wiki/Release-Announcement-for-PHPUnit-5.4.0>
PHPUnit 5 deprecates the getMock() shortcut for getMockBuilder()->getMock().
It instead introduces the shortcut createMock() which has better defaults
than getMockBuilder(). For example, it sets 'disableArgumentCloning' and
other things by default.
Going forward, code should either use getMockBuilder directly and configure
it using the setter methods (instead of the confusing variadic arguments
of getMock) or simply use the new minimalistic createMock method. This patch
backports the createMock method to MediaWikiTestCase so that we can start
using it.
Change-Id: I091c0289b21d2b1c876adba89529dc3e72b99af2
Providing the perfectly correct number of pages
that have acctually been added to or removed from a
category is extermly hard.
Rather than providing data that is most likely wrong
and a bit useless ONLY provide a link to Special:WhatLinksHere.
This touches on the following bugs and may mean that
some of them don't need to be thought about any further.
Bug: T126855
Bug: T126407
Bug: T126139
Change-Id: Ida06d822d1955091595c17c9c6c2968a40a93bcd
We currently push a request id into structured logging (monolog/
logstash) to allow seeing all logs that were triggered by the same
request. This extends that to pass the id through jobs so jobs triggered
by a web request also share the same id and can be tracked together.
This web request id will follow jobs both directly created by a request,
and jobs created by those jobs.
This should give us some more visibility when debugging into what
started a particular job, and if a large number of jobs blowing up the
job queue are somehow related.
Change-Id: Iedbd031e6e9bb18fd6f7b923c8c305102255ab4b
This reduces the runtime of database-bound tests by about 40%
(on my system, from 4:55 to 2:47; results from Jenkins are
inconclusive).
The basic idea is to call addCoreDBData() only once, and have
a addDBDataOnce() that is called once per test class, not for
every test method lie addDBData() is. Most tests could be
trivially be changed to implement addDBDataOnce() instead of
addDBData(). The ones for which this did not work immediately
were left out for now. A closer look at the tests that still
implement addDBData() may reveal additional potential for
improvement.
TODO: Once this is merged, try to change addDBData() to
addDBDataOnce() where possible in extensions.
Change-Id: Iec4ed4c8419fb4ad87e6710de808863ede9998b7
I created a basic test yesterday to cover two bugs. Now the test covers
all public methods. I was also able to get rid of the test double.
Change-Id: I53110280e3ef7b7a72d175b11b7fc4ccf1d648b3
* Track queues with non-abandoned jobs per partition server.
The s-queuesWithJobs key can easily be queried to see which
queues need to have periodic tasks run (or for debugging).
* This is requirement for the redis jobchron service to be able to
avoid hitting N=(no. types X no. wikis) queues for periodic tasks
when only a tiny fraction of those actually have any jobs. For WMF,
there are over 30K queues, most of them empty, so doing that can help
lower redis-server CPU (or at least make jobchron more responsive).
* This also allows for jobchron to manage the aggregator by taking the
per-server aggregator sets and merging them. This scales much better
as there are only a modest number of these daemons (18 for WMF) but
vastly more web thread pushing jobs. This cuts down on the connections
to the active aggregator server (the one with the hash table).
* Use Lua unpack() more for stylistic consistency.
Change-Id: I1549f0edc78cc4004dd887b475dec4c0ebd306c6
If we really need this we can do it in MediaWikiTestCase, next
to the setting of wgMainCacheType. But from what I can see the
code being tested here already doesn't use the old $wgMemc.
Change-Id: I9e4b2109b2f3c18d8d5551bbadae5711c1d4c0a6
* Remove some getAcquiredCount() assertions when claimTTL=0
as this is not well defined enough (queues may take a few
minutes to garbage collect the failed jobs).
* Added some tests to make sure push() only de-duplicates
among unclaimed jobs.
Change-Id: Ie0a5e539095c245dfcc8c160417e12824eb7ab83
I noticed JobQueueTest::testRootDeduplication takes ~ 6.5 seconds, which
is due to the test method using sleep(1) and being passed the provider
provider_queueLists which yields six items.
The reason is to have the array returned by Job::newRootJobParams() to
have an incread value for 'rootJobTimestamp'. Instead, just copy the
previous array of parameters and increment the UNIX timestamp and
converting back to TS_MW format.
Change-Id: I75066df73f9f92e56b89eb6d928c41e949a2d6a9
