Commit graph

639 commits

Author SHA1 Message Date
Thiemo Kreuz
e1dd371e11 Make use of PHPUnit's assertCount feature where possible
… and avoid assertEmpty() on arrays, in favor of a much more strict
assertSame( [] ).

Change-Id: I20266b0b1fc38a3a87666ba1b0793cb2b37d94a9
2020-03-02 15:58:41 +00:00
Aaron Schulz
6b12696452 Move UIDGenerator code to a service and put it under /libs
All MediaWiki dependencies have been removed or injected.

Change-Id: I01c9e96edd6b03496c1595670967ffa5a4069c9d
2020-02-18 00:20:40 +00:00
Aaron Schulz
5d6470d37d rdbms: make Database::build(Greatest|Least) support expressions
This makes it possible to use with counter UPDATE queries.

Also add some extra sanity checks for input types.

Change-Id: Ibc2b7173e28022b5ba7bb04d11c594313a47a101
2020-02-15 21:56:57 +00:00
jenkins-bot
991b58bffb Merge "objectcache: fix "coalesceKeys" option name in WANObjectCache" 2020-02-13 17:24:34 +00:00
Aaron Schulz
b9b3f366bb objectcache: fix "coalesceKeys" option name in WANObjectCache
Follow-up to 85bc62c5a8

Fix related unit tests that otherwise break as a result

Change-Id: I28b3a1537d319c68a7c12c578e1acfb916f3ec99
2020-02-13 03:17:34 +00:00
jenkins-bot
c3f20ad3f7 Merge "Add SelectQueryBuilder" 2020-02-13 00:04:11 +00:00
jenkins-bot
3cddc7eb36 Merge "In Database::select() allow an empty array for $table" 2020-02-06 23:36:03 +00:00
Tim Starling
d06a3e049b Add SelectQueryBuilder
Add a query builder class which encapsulates the parameters to
IDatabase::select() and related functions.

Override useIndexClause() and ignoreIndexClause() in DatabaseTestHelper
so that index hints can be tested.

Bug: T243051
Change-Id: I58eec4eeb23bd7fb05b8b77d0a748f1026afee52
2020-02-07 10:10:17 +11:00
Tim Starling
00c8a5cbde In Database::select() allow an empty array for $table
Previously it would give FROM followed by nothing which is always a
syntax error. Easier to fix it here than to convert empty arrays to
empty strings in SelectQueryBuilder.

Bug: T243051
Change-Id: I95a9b6a34cfb5c1ca4cf243c4226b5ed4f968035
2020-02-07 09:57:27 +11:00
Aaron Schulz
85bc62c5a8 objectcache: add "coalesceKeys" option to WANObjectCache for key grouping
This is useful for grouping related keys on the same servers to reduce
the need for cache server connections and availability. A cache key that
uses "lockTSE" can already involve accessing several keys during the
read/write cache-aside paths:
a) The value key itself
b) The check key (named after the main key, a common pattern)
c) The mutex key (used if the value looks stale)
d) The cool-off key (used if regeneration took a while)

Any problems accessing the first two could cause extra value regenerations.
Problems with the mutex key could lead to stampedes due to threads assuming
another thread was regerating a soon-to-expire value when, in fact, none was.
A similar problem could happen with cool-off keys, with threads assuming
that another saved the newly regenerated value when, in fact, none did.

The use of hash stops puts the tiny related keys on the same server as the
main cache key that they serve. This is only for hash-based routing, and not
route prefix routing (e.g. All*Route still sends the key to multiple child
routes, but the PoolRoute/HashRoute function will hash differently).

The option is not enabled by default yet.

Change-Id: I37e92a88f356ef1e2a2b7728263040e2f6f09a13
2020-02-06 20:27:08 +00:00
Brad Jorsch
a04633f678 ParamValidator: Default PresenceBooleanDef to false rather than implicitly null
PresenceBooleanDef is causing ParamValidator::getValue() to return null,
while historical Action API behavior had returned false instead.

While it generally shouldn't make a difference since PHP considers both
falsey, and good arguments could be made either way, we can restore the
historical behavior easily enough by having normalizeSettings()
default PARAM_DEFAULT to false.

Bug: T244440
Change-Id: Iee1d8e5753407674adc3f7384989841bc9b44c54
2020-02-06 09:44:06 -05:00
jenkins-bot
7b54f5dc3d Merge "rdbms: add GREATEST/LEAST wrappers to IDatabase" 2020-02-04 22:03:11 +00:00
Brad Jorsch
d4c2f0d899 Move some validation logic from ApiStructureTest to ParamValidator
ApiStructureTest has a lot of logic for validating Action API settings
arrays during CI. Some of that logic should be part of ParamValidator
instead.

Bug: T242887
Change-Id: I3c3d23e38456de19179ae3e5855397316b6e4c40
Depends-On: I04de72d731b94468d8a12b35df67f359382b3742
2020-02-04 20:29:35 +00:00
jenkins-bot
51280df815 Merge "objectcache: fix cache pollution in WANObectCache Multi* methods" 2020-01-30 18:17:03 +00:00
Aaron Schulz
527fd0109f objectcache: fix cache pollution in WANObectCache Multi* methods
This was triggered by bad reference handling during preemptive refreshes

Bug: T235188
Change-Id: I239a3e1922f478c74c94d8d2debff28f525c7c31
2020-01-29 21:08:39 +00:00
jenkins-bot
0a46bef2db Merge "objectcache: fix storage of null values in WANObjectCache" 2020-01-29 04:14:20 +00:00
Aaron Schulz
4fb5210b62 objectcache: fix storage of null values in WANObjectCache
Bug: T234583
Change-Id: I38a531b9a0acb95d7884519f3381b48cd9d8faa0
2020-01-24 22:49:26 +00:00
Aaron Schulz
314efebb56 rdbms: add GREATEST/LEAST wrappers to IDatabase
Change-Id: I9de931123b03ce10713a3a9bbb34e1332dd5965b
2020-01-17 22:19:08 +00:00
Brad Jorsch
55d3d81803 ParamValidator: Tighten unrecognized value handling
Rename PARAM_IGNORE_INVALID_VALUES to PARAM_IGNORE_UNRECOGNIZED_VALUES,
and make it only ignore the "badvalue" errors thrown by EnumDef for
unrecognized values instead of ignoring every kind of error.

This better matches historical Action API behavior.

Change-Id: Ifdb9063c0a2e82c728e98065e2ac950f4a713552
2020-01-14 17:24:20 -05:00
Brad Jorsch
6e9ec41d0a ParamValidator: Adjust message usage
This fixes a few things in ParamValidator's use of messages:

* Values need to be stringified in various places.
* Don't pass the value to paramvalidator-toomanyvalues, to make it match
  apierror-toomanyvalues. We weren't using it anyway.
* Rename "paramvalidator-help-multi-sep" to match
  "api-help-param-multi-separate".
* Fix attempts to use messages "paramvalidator-param-default" and
  "paramvalidator-param-default-empty" that don't exist.
* EnumDef's logic for selecting between enummulti vs enumnotmulti
  was backwards.
* Sort enum values in getParamInfo() output.
* Adjust some message texts to match equivalent former Action API
  messages.

Change-Id: I1551a59bd110f12e478de7292a00f9a767cd94ca
2020-01-14 17:24:20 -05:00
Brad Jorsch
1ddb16e103 ParamValidator: EnumDef custom deprecation messages
Iea6d4a1d forgot to convert non-true values in PARAM_DEPRECATED_VALUES
to MessageValue, as was originally intended.

Change-Id: If5907f2072c1efdeb9ee8d1b06eaeacaf95cab17
2020-01-14 17:24:19 -05:00
Brad Jorsch
c0150d8bf5 ParamValidator: EnumDef shouldn't warn about deprecated values from default
When the values being validated come from PARAM_DEFAULT, EnumDef
shouldn't warn about them being deprecated.

Making this happen requires a new option passed through from
ParamValidator::getValue() to ::validateValue() to TypeDef::validate().

Change-Id: I8535f8757485748bccb7a756b71ce61fdf28faa1
2020-01-14 17:24:19 -05:00
James D. Forrester
4f2d1efdda Coding style: Auto-fix MediaWiki.Classes.UnsortedUseStatements.UnsortedUse
Change-Id: I94a0ae83c65e8ee419bbd1ae1e86ab21ed4d8210
2020-01-10 09:32:25 -08:00
James D. Forrester
5e9fca47b9 Coding style: Auto-fix MediaWiki.Usage.PHPUnit*
Change-Id: I86fc55a4fc8ceafe368692173211bbcd6d8581d7
2020-01-10 10:17:12 +00:00
daniel
2a763bfc67 DatabaseTest: fix provider for testIsWriteQuery
testIsWriteQuery() never worked. It was using a non-existing provider,
which caused the test to be skipped with a warning. Access to the
protected method under test also needed fixing.

Change-Id: If35789027f2d6c44a7079e68a1c07c86be72d42b
2020-01-07 21:39:39 +01:00
jenkins-bot
9318f46267 Merge "Improve tests in DoctrineSchemaBuilder" 2020-01-06 11:00:54 +00:00
jenkins-bot
56f0bcf6e7 Merge "Remove MWMessagePack" 2020-01-01 23:38:00 +00:00
Daimona Eaytoy
29156b4c84 rdbms: Account for leading spaces in isWriteQuery
Bug: T241034
Change-Id: I313e81dddeb9faef1780c9780fb575a5f8376bf4
2019-12-30 21:24:06 +00:00
TheSandDoctor
01b9bfd5ae Remove MWMessagePack
MWMessagePack has been removed.

Bug: T241220
Change-Id: I1e1823c3b42022d6b045b8d33bebb1be7da95631
2019-12-30 00:00:39 -08:00
jenkins-bot
f73d62d47d Merge "Add more logging to getWithSetCallback()" 2019-12-20 00:36:31 +00:00
Daimona Eaytoy
e501bb3b50 Manual fix of the remaining assertInternalType & Co.
Bug: T192167
Change-Id: I3b8176745f49b1d4bfe6f36978b7879d546f68a9
2019-12-15 00:20:10 +00:00
Daimona Eaytoy
7c9e3db1e6 Fixes for PHPUnit 8 compatibility
Bug: T192167
Change-Id: Ic14f5debc53e55d67146dc96279d26dfd52b4000
2019-12-10 17:02:06 +00:00
jenkins-bot
5e9c40b66d Merge "Remove IE 6 security features from server-side code" 2019-11-28 04:43:35 +00:00
Tim Starling
164a3ac1f0 Remove IE 6 security features from server-side code
* Deprecate WebRequest::checkUrlExtension() and have it always return
  true. This reverts the security fixes made for T30235.
* Remove IEUrlExtension. This is a helper for checkUrlExtension() which
  is not used in any extensions.
* Remove CSS sanitization code which is specific to IE6. This reverts
  the changes made to fix T57332, and related followups. I confirmed
  that the relevant test cases do not result in XSS on IE8.
* Remove related tests.

Bug: T232563
Change-Id: I7318ea4a63210252ebc64968691d4f62d79a63e9
2019-11-28 15:11:56 +11:00
Brad Jorsch
70bc2a5527 libs/Message: Fix value of ParamType::DURATION_SHORT
The values of the constants are supposed to match the keys used for the
corresponding special parameters to Message. This one was different.

Note the values of these constants are considered an internal
implementation detail, as is the format of the XML-ish output of
MessageValue::dump(), so we don't have to worry about compatibility
when changing it here.

Change-Id: If5a7279e661df8c13780552d1dee752215e1d497
2019-11-25 18:23:43 +00:00
Amir Sarabadani
6d4dc90831 Improve tests in DoctrineSchemaBuilder
Also changing its structure to be more readable and more like
https://w.wiki/CXo

My plan is to add more tables to this test.

Bug: T230428
Change-Id: Ia19d5c875cc95238f7fd7cd9adbed1ddb92af078
2019-11-22 19:53:10 +01:00
Tim Starling
29e33aa50a Fix random WANObjectCacheTest failures
We're observing WANObjectCacheTest fail sometimes, possibly due to a
1/825 chance of a cache key being counted as "popular" combined with an
mt_srand() call making this more likely. I tweaked the options to
getWithSetCallback() to avoid the affected mt_rand() call altogether.

I also added a lowTTL option to another getWithSetCallback() call,
eliminating a condition with a 3% chance. It didn't appear to matter
which branch the test took, but I'd rather avoid non-deterministic
behaviour.

There are a few remaining coin tosses being logged, but they all occur
with a chance of less than 10^-5.

Bug: T238197
Change-Id: Iadd3dc00d44b8490210fec97fe1f49236326300f
2019-11-20 05:00:48 +00:00
jenkins-bot
3fb1bd2c72 Merge "Have Database::addQuotes() pass through bare integers without quoting" 2019-11-18 17:42:12 +00:00
jenkins-bot
322abd33da Merge "rdbms: Fix DatabaseMysqlBase::streamStatementEnd() possibly failing" 2019-11-18 15:56:48 +00:00
Tim Starling
24812e4096 Add more logging to getWithSetCallback()
To help isolate the referenced bug, which is a sporadic test failure
in testGetWithSetCallback().

Bug: T238197
Change-Id: If35d60340c804b6bfe1e9ddfcf53c76373c794b1
2019-11-18 16:32:47 +11:00
Tim Starling
673d496f2d Have Database::addQuotes() pass through bare integers without quoting
Quotes started being added to integers in r4984 (August 2004). Before
that, is_numeric() was used to determine whether to add quotes, so
quotes were omitted from numeric strings, which is obviously wrong.

The idea here is to use the type of the variable to hint to the database
as to whether quotes are needed. The results are somewhat inconsistent,
since some callers do not convert numeric strings obtained from user
input to integers. That makes it a more conservative change. Callers can
opt out of unquoted integers by casting them to string.

The reason for doing this is that quoting integers turns out to be not
as harmless as originally assumed. We found a case of it confusing the
MariaDB query planner, causing inappropriate indexes to be used.

I also made addQuotes() consistently return a string, instead of
returning an integer for boolean values. This was already the case for
MySQL, but it seems like a good idea everywhere.

Bug: T238378
Change-Id: I70473280f542ee5ecd79e187f580807410fbd548
2019-11-18 11:40:28 +11:00
jenkins-bot
40b28c73a7 Merge "ParamValidator: Use MessageValue!" 2019-11-13 06:15:51 +00:00
Max Semenik
5bf93709f4 Move a lot of libs tests to the unit test directory
Change-Id: Ie7eb1016cd735f07b00524815598581e0d485f2a
2019-11-11 17:53:56 -08:00
Amir Sarabadani
30c4b22d4e Start of DoctrineSchemaBuilder
This allows us to abstract away schemas into one json file.

Bug: T230419
Change-Id: Ic8a658c97150e4c248bdbc9a5cf5d947b8c71036
2019-11-09 21:32:32 +00:00
Thiemo Kreuz
e6615256c2 rdbms: Fix DatabaseMysqlBase::streamStatementEnd() possibly failing
If a string starts with "DELIMITER" but the word is *not* followed by
a space, the code will set $this->delimiter to null, possibly resulting
in all kinds of hard to track errors.

Even more problematic: the preg_match was *not* case-insensitive, but the
condition was. Which means that all non-uppercase "delimiter …" would
result in the same error.

I'm not sure if it's worth keeping the additional string comparison for
performance reasons. Probably not. That comparison is slow as well, and
preg_match is surprisingly fast (when the pattern is properly written,
which it is).

Change-Id: I33944b7a2410f77e67ce7450af0359a88d39f1aa
2019-11-08 17:25:08 +01:00
Max Semenik
dd7026585c Backwards-compatible PHPUnit 8 preparations
This commit splits changes from Ic14f5debc53e55d6714 to reduce it to
only strictly needed things. It can be merged immediately.

Bug: T192167
Change-Id: I8c541a66ea13421dbe7fa51d197d5455cc4786eb
2019-11-02 12:40:07 -07:00
Brad Jorsch
aa0720d37c ParamValidator: Use MessageValue!
Trying to get away with returning a single code and parameter-list that
was supposed to represent both human-readable and machine-readable data
was a mistake.

This patch converts it to use DataMessageValue, which represents the two
separately and also provides guidance for supplying translations of all
the error codes.

This also eliminates the "describeSettings()" method that was trying to
serve multiple use cases (in terms of the Action API, action=paraminfo
and action=help). It's replaced by two methods that each serve one of
the use cases. Also some of the functionality was moved out of the
TypeDef base class into ParamValidator, to better match where the
constants themselves live.

Also I wound up creating a NumericDef base class so FloatDef can share
the same range-checking logic that IntegerDef has. I probably should
have done that as a separate patch, but untangling it now would be too
much work.

Bug: T235801
Change-Id: Iea6d4a1d05bb4b92d60415b0f03ff9d3dc99a80b
2019-11-01 15:49:31 -04:00
Max Semenik
48a323f702 tests: Add explicit return type void to setUp() and tearDown()
Bug: T192167
Depends-On: I581e54278ac5da3f4e399e33f2c7ad468bae6b43
Change-Id: I3a21fb55db76bac51afdd399cf40ed0760e4f343
2019-10-30 14:31:22 -07:00
Ed Sanders
e20a6e516a Move deflate method to MW util and deprecate EasyDeflate module
This will allow us to easily change the deflate library in future.

Rename the PHP class and i18n to just "Deflate".

Bug: T236210
Change-Id: I88dea5e82a7f672e36556c40b151ac00ff3d044e
2019-10-30 15:03:41 +00:00
jenkins-bot
3627517ed8 Merge "libs/Message: Add DataMessageValue" 2019-10-23 13:35:31 +00:00