Commit graph

447 commits

Author SHA1 Message Date
James D. Forrester
903e8b63de doc: Modernise parameter names and documentation for 'replica' DBs
Non-breaking change. Remaining uses are public interfaces (a constant, two
globals, a config sub-parameter, SQL queries, storage function names), one i18n
message key, and a whole lot of maintenance scripts with calls to the deprecated
function wfWaitForSlaves().

Change-Id: I6ee5ca92ccf6a80c08f53d9efe38ebb4b05064d7
2018-10-31 10:36:48 -07:00
Brad Jorsch
c5a5b02240 Database: Allow selectFieldValues() to accept SQL fragments
The documentation says "This must be a valid SQL fragment", but as
written it breaks if given anything other than a field name. It's easy
enough to fix by adding an alias to the internal select() call.

Bug: T201781
Change-Id: I76428af6d3aadc266254fdb24109a0ac2db3761f
2018-10-17 22:21:40 +00:00
Aaron Schulz
b06f020217 rdbms: re-add DB domain sanity checks to LoadBalancer
Also clean up empty schema handling in DatabaseDomain

This reverts commit f23ac02f4f.

Bug: T193565
Change-Id: I95fde5c069f180ca888a023fade25ec81b846d44
2018-10-16 23:35:05 +00:00
Aaron Schulz
fe0af6cad5 rdbms: Database::selectDB() update the domain and handle failure better
LoadBalancer uses Database::getDomainId() for deciding which keys to use
in the foreign connection handle arrays. This method should reflect any
changes made to the DB selection.

If the query fails, then do not change domain field. This is the sort of
approach that LoadBalancer is expects in openForeignConnection(). Also,
throw an exception when selectDB() fails.

The db/schema/prefix fields of Database no longer exist in favor of just
using the newer currentDomain field.

Also:
* Add IDatabase::selectDomain() method and made selectDB() wrap it.
* Extract the DB name from sqlite files if not explicitly provided.
* Fix inconsistent open() return values from Database subclasses.
* Make a relationSchemaQualifier() method to handle the concern of
  omitting schema names in queries. The means that getDomainId() can
  still return the right value, rather than confusingly omitt the schema.
* Make RevisionStore::checkDatabaseWikiId() account for the domain schema.
  Unlike d2a4d614fc, this does not incorrectly assume the storage is
  always for the current wiki domain. Also, LBFactorySingle sets the local
  domain so it is defined even in install.php.
* Make RevisionStoreDbTestBase actually set the LoadBalancer local domain.
* Make RevisionTest::testLoadFromTitle() account for the domain schema.

Bug: T193565
Change-Id: I6e51cd54c6da78830b38906b8c46789c79498ab5
2018-10-10 12:03:30 -07:00
Brad Jorsch
bf30fcb714 Database: close() should not commit transactions
Transactional databases normally roll back when a connection is closed
with an open transaction rather than committing them, so MediaWiki
committing them is unexpected.

There are two cases being changed here: automatic transactions without
writes and manual transactions. For the former it shouldn't make a
difference if we commit or roll back since no writes were done anyway.
The latter has logged a message since MW 1.31 (I0992f9a8), and that
warning has not been logged in Wikimedia production in the past 60 days
so we should be ok there too.

Bug: T206147
Change-Id: Ieceef4deb49044db8f0622d38ee76c9d9f39704c
2018-10-03 14:55:43 -04:00
Aaron Schulz
46cf714ce7 rdbms: make * consistently act like in select/insertSelect methods
This now matches the documentation of insertSelect()

Bug: T202553
Change-Id: Ie6602fdd3b48b9136de7c65289c85ced5f5f2f1d
2018-10-01 14:31:53 -07:00
Umherirrender
a4caa4d0c6 build: Updating mediawiki/mediawiki-codesniffer to 22.0.0
Added spaces around .
Removed empty return statement which are not required
Removed return after phpunit markTestIncomplete,
which is throwing to exit the test, no need for a return

Change-Id: I2c80b965ee52ba09949e70ea9e7adfc58a1d89ce
2018-09-16 15:51:11 +00:00
Kunal Mehta
8a3bdfc633 Support nested arrays in StaticArrayWriter
This is needed by LCStoreStaticArray.

Bug: T200626
Change-Id: I76abd3849381b3b98e350a4627f3515eeb00fa2d
2018-08-27 22:23:02 +00:00
jenkins-bot
5f7e96fdad Merge "Improve StaticArrayWriterTest and add missing @covers tag" 2018-08-19 17:07:13 +00:00
addshore
82b19bd96a Create and use PrefixingStatsdDataFactoryProxy in PerDbNameStatsdDataFactory
MediaWiki::emitBufferedStatsdData() is never called for the PerDbName factory,
so stats were being dropped. Instead of having two factories, turn the
PerDbName one into a proxy/wrapper around the main factory that just adds a
prefix in front of all of the keys.

Bug: T202144
Change-Id: I31e376446abb58e41353b4ca3814120d2e914104
2018-08-19 00:19:57 -07:00
Kunal Mehta
81f4eb4b71 Improve StaticArrayWriterTest and add missing @covers tag
Checking for proper quoting was suggested by Krinkle in CR.

Change-Id: I7fa599767017a5cfc82899bc4178de6cb34bdd02
2018-08-19 00:07:03 -07:00
Kunal Mehta
3b1e8a5cef Move wfMakeStaticArrayFile() into a class
And include tests :)

This code is independent of MediaWiki, but not really large enough to be
worth librarizing right now.

Bug: T200626
Change-Id: I022c074e8a708fb5219bc4ff4b53e7e31f60dc4b
2018-08-17 22:48:19 -07:00
Timo Tijhof
15110b1ca9 JavaScriptMinifier: Fix bad state after ternary in object literal
The following pattern of input (found in jquery.js) triggered
this bug:

    call( {
        key: 1 ? 0 : function () {
            return this;
        }
    } );

The open brace changes state to PROPERTY_ASSIGNMENT (for object literals).
The colon after 'key' sets state to PROPERTY_EXPRESSION.

Each individual parts of an expression (identifiers and literal values)
is recognised with state *_EXPRESSION_OP, such as PROPERTY_EXPRESSION_OP.

The '1' after 'key:' correctly sets the state to PROPERTY_EXPRESSION_OP.
Upto there it goes well, but after that it goes wrong.

The question mark (TYPE_HOOK) in this context was wrongly switching
back to PROPERTY_EXPRESSION. That is a problem because that does not
handle TYPE_COLON, which meant '0: function' was seen together as a
sequence of continuous PROPERTY_EXPRESSION_OP where TYPE_FUNC may
not be handled.

Fixed by changing handling of TYPE_HOOK in PROPERTY_EXPRESSION to
switch states to EXPRESSION_TERNARY, and also performing a push
so that ternary handling can pop back to the property expression.

This mirrors the handling that already exists for ternaries in
the regular handling of EXPRESSION/EXPRESSION_OP (as opposed to
the variant for object literal properties).

Bug: T201606
Change-Id: I6104c839cfc3416257543b54a91b74cb4aa4193b
2018-08-16 17:02:07 +00:00
Aaron Schulz
5358c41e86 rdbms: use Database::getDomainId for transaction logging calls
Change-Id: I360ca633b145dc8e510bb00b542ac44933283943
2018-08-15 01:29:39 -07:00
Fomafix
0a0d5cb7f7 Fix typos
Bug: T201491
Change-Id: I25a27d11faabe2f5fa02950c7a4fb58b13fb3662
2018-08-14 09:52:19 +00:00
jenkins-bot
dccec12761 Merge "Make MapCacheLRU throw errors for bad $field arguments" 2018-08-14 01:17:17 +00:00
Aaron Schulz
39a63a3c16 Make MapCacheLRU throw errors for bad $field arguments
Change-Id: Ibde33e0ff671d3428b73c66766478f58763726e1
2018-08-14 00:41:33 +00:00
Timo Tijhof
92ce940ec3 JavaScriptMinifier: Add test case for another line-break bug
Discovered by adding a test case that uses file_get_contents()
to pass jquery.js, and then scan the PHPUnit output for an entry
containing `return` by itself on a line where the next line
is isn't `;` - then reducing it to a test case as small as possible.

This was reduced from the definition of jQuery.event.addProp.

Bug: T201606
Change-Id: I1f907436c32630102e60e3ded7092dbeb9669fe8
2018-08-12 00:24:56 +01:00
Timo Tijhof
da3aa28cea JavaScriptMinifier: Move test case for 'x++' to provideLineBreaker()
Easier to read.

Bug: T201606
Change-Id: Ie12910edf5ab6a9d7246000ca78a3c9208aeb152
2018-08-12 00:24:10 +01:00
Timo Tijhof
1f5f6fc204 JavaScriptMinifier: Fix bad state after '{}' in property value
Previously, $push contained:

	self::PROPERTY_EXPRESSION_OP => [
		self::TYPE_PAREN_OPEN => self::PROPERTY_EXPRESSION_OP
	],

But $pop contained:

	self::PROPERTY_EXPRESSION_OP => [ self::TYPE_BRACE_CLOSE => true ]

This meant that when a closing brace was found inside a property
expression, it would wrongly pop the stack, eventhough we are still
inside the property expression.

The impact is that everything after this is one level higher in
the stack than it should be, causing various other types to be
misinterpreted. Including in the following contrived example:

	call( function () {
		try {
		} catch (e) {
			obj = {
				key: 1 ? 0 : {} // A
			}; // B
		} // C
		return name === 'input';
	} );

In the above, the closing brace at A would close the 'obj.key' assignment
(PROPERTY_EXPRESSION_OP), instead of waiting for the closing brace at B to
decide that.

Then the closing brace at B would wrongly close the 'catch' block (instead of
the 'obj' assignment). And lastly, the closing brace at C would close the
function body (STATEMENT).

This resulted in keyword 'return' being interpreted while in state
PAREN_EXPRESSION_OP instead of STATEMENT, where PAREN_EXPRESSION_OP is the
arguments list to `call()`. In an argument list, TYPE_RETURN is not valid,
which means we stay in that state, instead of progressing to EXPRESSION_NO_NL,
which then wrongly allows for a line break to be inserted.

Bug: T201606
Change-Id: I07b809a7ca56e282ecb48b5c89c217b4b8da6856
2018-08-11 00:16:47 +01:00
Timo Tijhof
8cfe4abf8d JavaScriptMinifier: Add better line-breaker tests
Set maxLineLength to '1' instead of messing with filler content.
This makes it easy to see all potential line-breaks, instead of only
at the 999th offset.

Bug: T201606
Change-Id: I220b145c5bc8e7d1a41efacd2a6cea738545f006
2018-08-10 23:22:49 +01:00
Timo Tijhof
53e05f7b9c JavaScriptMinifier: Add test case for T201606
Bug: T201606
Change-Id: I2058765c7f1cfb9e2d644f19e780926d01b9b68c
2018-08-10 21:04:58 +01:00
jenkins-bot
55ffac2024 Merge "objectcache: add "epoch" parameter to WANObjectCache" 2018-08-08 23:16:40 +00:00
Umherirrender
73bf53e94f Remove @uses annotation from XhprofDataTest
The use of a class is visible by the code or @covers

Change-Id: I1acec60da4b524c171686402a01424d66e9fc99f
2018-08-07 12:07:00 +02:00
Aaron Schulz
4af7fbd76a objectcache: add "epoch" parameter to WANObjectCache
This can be used to discard all values before a certain point in
time, such as periods of severe network problems or misconfiguration
that resulted in missed purges.

Change-Id: I612db8f91a5960b912e9f35645a3d3872df47460
2018-08-06 16:43:50 -07:00
jenkins-bot
f506a05aaa Merge "Upstream EasyDeflate library from VisualEditor" 2018-07-26 15:06:41 +00:00
jenkins-bot
cfd0e73f10 Merge "resourceloader: CSSMin::getLocalFileReferences now strips anchors" 2018-07-20 04:59:31 +00:00
Antoine Musso
4030b2cb2b resourceloader: CSSMin::getLocalFileReferences now strips anchors
When processing:

  url( 'foo.svg#anchors' )

getLocalFileReferences() would yield a file 'foo.svg#anchors'. It might
occur with upstream libraries, such as WikiFonts used by the Refreshed
skin.

When a path is found, strip the anchor entirely. Skip the case if that
results in an empty file.

remap() would properly skip the behavior "#default#', also skip url that
are solely an anchor such as '#other'.

Add a few test cases to cover CSSMin::getLocalFileReferences and cover
anchors usage in CSSMin::remap.

Bug: T115436
Change-Id: I1749ddc2b89021807f42d64131931ad7a99a7b43
2018-07-20 04:40:44 +00:00
Aaron Schulz
031a4c747e Add $maxAge parameter to MapCacheLRU get() and getField() methods
Change-Id: I2dee1ad3a5b4f2ef8c182e3466812777a9a9cf7f
2018-07-18 18:08:53 +00:00
Aaron Schulz
b8779134ed objectcache: make BagOStuff::mergeViaLock() timeout more sensible
Interpret "1 attempt" as non-blocking and use 3 seconds otherwise
(instead of 6 seconds). This means that the WAN cache merge call
does not block.

Bug: T198239
Change-Id: Ieb194a949f18785c2cc9dd20fdc4d2e3fed51abf
2018-07-12 16:02:38 +01:00
jenkins-bot
216865344b Merge "rdbms: add IDatabase::lockForUpdate() convenience method" 2018-07-11 19:52:31 +00:00
Aaron Schulz
9eff263e8e rdbms: add IDatabase::lockForUpdate() convenience method
Change-Id: I238fd96407e1122e90058e2c4acf743044a267ec
2018-07-10 20:09:01 +01:00
Aaron Schulz
86dd3b0ac0 Avoid deprecated IDatabase::getWikiId() reference
Change-Id: I26991079b1630335c8a6e907554760fc85c1bad6
2018-07-07 22:55:15 +00:00
jenkins-bot
4b5773a4de Merge "rdbms: fix Sqlite::tableExists() method to avoid STATUS_TRX_ERROR" 2018-07-03 05:40:03 +00:00
jenkins-bot
9fa37f0a8b Merge "objectcache: make MultiWriteBagOStuff handle duplicate add() operations" 2018-06-29 02:16:52 +00:00
Aaron Schulz
6afa7bb86a Make ProcessCacheLRU wrap MapCacheLRU
Change-Id: I190c824af471aee798e2f111b902f38532b8ac99
2018-06-28 20:23:37 +00:00
Aaron Schulz
6612407f9c objectcache: make MultiWriteBagOStuff handle duplicate add() operations
Bug: T198280
Change-Id: Ib1bcde2b3fbfb452f80d8d840c494be2eb70eb87
2018-06-28 16:04:35 +00:00
Aaron Schulz
1f2b2f3f8d Add key expiration and map resizing support to MapCacheLRU
Also implement Serializable in to add stability to anything that
uses naïve serialization (such as for persistent caching).

Change-Id: I03ba654ffd80ba027d47d5d7583abfe48d52818d
2018-06-28 05:53:42 +00:00
Aaron Schulz
d16bfb0a30 rdbms: fix Sqlite::tableExists() method to avoid STATUS_TRX_ERROR
Sqlite used the base implementation of trying a SELECT 1 query and
seeing if it failed. Instead, make it use the sqlite_master table.
Also remove the base version of that method since it would always
cause this problem and all subclasses have proper implementations.

Make LoadBalancerTest::assertWriteAllowed() more explicit and add
more assertions there.

Change-Id: I6c7b0bea8894c45dfe8931748d6687f0e5d1e101
2018-06-22 22:33:45 +00:00
Edward Chernenko
9d4a7e4d75 Xhprof: support tideways-xhprof extension
The "tideways" extension was renamed by their developers, see
https://tideways.com/profiler/blog/releasing-new-tideways-xhprof-extension

While doing so, they also renamed enable/disable functions:
tideways_enable -> tideways_xhprof_enable
tideways_disable -> tideways_xhprof_disable

Change-Id: I63bc97dba461dd46241a094dfc59439c0d28a219
2018-06-19 14:43:40 +00:00
Ed Sanders
230e90b2af Upstream EasyDeflate library from VisualEditor
The library is used to compress large documents before
sending to the server, as HTTP doesn't support upload compression.
(See T68914 for context.)

N.B. Many of the polyfills in the original are omitted as
MediaWiki doesn't support browsers which require them.

Co-Authored-by: Kunal Mehta <legoktm@member.fsf.org>
Change-Id: I12a5879188f46f17dc4c33bdad52317d218d4172
2018-06-10 14:35:27 +00:00
Kunal Mehta
ff18d57c63 Rewrite IEUrlExtensionTest using a data provider
Change-Id: I67568f997d682b88ad99676a57b85395b1b20f85
2018-06-08 23:07:38 -07:00
Tim Starling
4aecdad647 HashRing: use bisection instead of guess and refine
Use bisection to find the position within the ring, as in ketama.
According to my benchmarking this is faster than the guess/refine
algorithm for rings with 100 locations, i.e. 16000 ring nodes requiring
at most 14 bisection iterations.

Change-Id: I6042f382de093081971b5ec210eda8dfa74988f2
2018-06-08 02:36:49 -07:00
Aaron Schulz
f2e6543c46 Rewrite HashRing to use consistent hashing
This scheme avoids disruptive re-hashing when a node is
ejected; only the adjectent clockwise sector is affected,
the later taking responsibility for the former. Addition
of new nodes is likewise less disruptive.

When used with MD5, this maps keys the same as libketama.

Also:
* Make HashRing::getLocations() faster by guessing the
  sector list index and moving clockwise/counter-clockwise.
* Made HashRing Serializable for more robust storage.
* Allow for different hash functions to be specified.

Note:
* This makes PoolCounterRedis use the new hash scheme.
* Likewise for JobQueueFederated (for job de-duplication).
  Switching from one mode to the other will cause some of
  duplicated effort from old jobs only temporarily.

Change-Id: Iff432a4e5dc3fb40b2dda70fb3dbe9dda0b74933
2018-06-07 03:36:22 -07:00
jenkins-bot
33ca739e7f Merge "objectcache: update MemcachedPeclBagOStuff for pecl memcached 3.0.0" 2018-06-05 15:50:29 +00:00
Bartosz Dziewoński
03583f7a91 Use PHP 7 "\u{NNNN}" Unicode codepoint escapes in string literals (part 2)
This is a follow-up to Idc3dee3a7fb5ebfaef395754d8859b18f1f8769a
containing some less trivial changes.

Change-Id: Ia7af2c1d000307d43278cde4a246df413d4ef263
2018-06-04 16:40:48 +00:00
Bartosz Dziewoński
0313128b10 Use PHP 7 "\u{NNNN}" Unicode codepoint escapes in string literals
In cases where we're operating on text data (and not binary data),
use e.g. "\u{00A0}" to refer directly to the Unicode character
'NO-BREAK SPACE' instead of "\xc2\xa0" to specify the bytes C2h A0h
(which correspond to the UTF-8 encoding of that character). This
makes it easier to look up those mysterious sequences, as not all
are as recognizable as the no-break space.

This is not enforced by PHP, but I think we should write those in
uppercase and zero-padded to at least four characters, like the
Unicode standard does.

Note that not all "\xNN" escapes can be automatically replaced:
* We can't use Unicode escapes for binary data that is not UTF-8
  (e.g. in code converting from legacy encodings or testing the
  handling of invalid UTF-8 byte sequences).
* '\xNN' escapes in regular expressions in single-quoted strings
  are actually handled by PCRE and have to be dealt with carefully
  (those regexps should probably be changed to use the /u modifier).
* "\xNN" referring to ASCII characters ("\x7F" and lower) should
  probably be left as-is.

The replacements in this commit were done semi-manually by piping
the existing "\xNN" escapes through the following terrible Ruby
script I devised:

  chars = eval('"' + ARGV[0] + '"').force_encoding('utf-8')
  puts chars.split('').map{|char|
    '\\u{' + char.ord.to_s(16).upcase.rjust(4, '0') + '}'
  }.join('')

Change-Id: Idc3dee3a7fb5ebfaef395754d8859b18f1f8769a
2018-06-04 16:20:13 +00:00
Aaron Schulz
7a730a6f9e objectcache: update MemcachedPeclBagOStuff for pecl memcached 3.0.0
The get() $cas_token parameter was changed into $flags, which can act
as a switch to make the return value an associative array of details.

pecl and PHP-based memcached BagOStuff class both pass all tests now.

Bug: T196125
Change-Id: I4678250331a48db4d50d1fc6c489c991a4dee920
2018-06-02 17:25:22 -07:00
jenkins-bot
285080c311 Merge "objectcache: make RedisBagOStuff pass all tests" 2018-06-02 01:35:26 +00:00
jenkins-bot
966e34619b Merge "objectcache: add setMockTime() method to BagOStuff/WANObjectCache" 2018-06-01 13:19:24 +00:00