This is more consistent with LoadBalancer, modern, and inclusive
of master/master mysql, NDB cluster, and MariaDB galera cluster.
The old constant is an alias now.
Change-Id: I0b37299ecb439cc446ffbe8c341365d1eef45849
Currently if you run updateCollation.php in dry-run mode, it ignores
the other parameters and doesn't give you a row estimate. Now it
will behave the same as an actual run (just without making any
changes to the database).
Change-Id: I25a9751d8ab7554e7975e5f08122dd1ddaaf40a7
2000 writes per wfWaitForSlaves() seems a bit high. There was a
report of this script causing some slave lag when being run.
Note that, the amount of time between wfWaitForSlaves() was
previously increased in r97146.
Bug: T58041
Change-Id: I07a29499775a17255865f25e6b9f1058f898193b
jcrespo reported a lag spike at the very beginning of running this
script. I'm guessing that's due to counting how many rows in
categorylinks to give the progress bar. Since we only need a
rough estimate for the progress meter, make that query run on
a slave. Also add a wfWaitForSlaves() immediately after it for
good measure.
Bug: T58041
Change-Id: I3cba392f0013fcb2ef86803632e2d9b1b88b3b29
We want to update categories in order, to minimize disruption
to users. Previous indexes required a filesort to do this, which
exploded things on large wikis. See bug for details
Bug: T58041
Change-Id: Iee6cd997ff87a313a46fda19d8ab063d0fed8ce8
Using the cl_sortkey index instead (to reduce disruption to a live
site), as currently implemented, seems to have two serious problems:
* MySQL / MariaDB filesorts all rows that "sort above the given row
[the last row of the previous batch]", not just a single category
at a time until the row limit is reached.
* The current approach to pagination is broken in that it does not
work with ENUM columns such as cl_type, causing 'file' rows to be
skipped, or rows of any type to be repeated. See T119173.
This reverts part of commit a43f751cf6.
Bug: T58041
Change-Id: I619564e85b2122f249bdacc45d547b9ce1b3beb5
Add transaction methods to complement getDB().
This makes it easy to grep for direct begin()/commit()
calls to IDatabase by having script use their own
wrapper. Maintenance scripts are one of the few places
that can (and need to) use begin/commit instead of the
start/end atomic methods.
Eventually, there should be almost no direct callers
and those methods can be made stricter about throwing
errors on nested calls.
Change-Id: Ibbfc7a77c0d2a55f7fc2261087f6c3a19061e0aa
In some cases the constructor will work, but trying to access first
letter data will raise an exception, breaking all category pages.
Bug: 46615
Change-Id: I77de040f97080653fe0d1734d38490eaa2d322db
Follows-up I1343872de7, Ia533aedf63 and I2df2f80b81.
Also updated usage in text in documentation and the
installer LocalSettingsGenerator.
Most of them were handled by this regex:
- find: (require|include|require_once|include_once)\s*\(\s*(.+?)\s*\)\s*;$
- replace: $1 $2;
Change-Id: I6b38aad9a5149c9c43ce18bd8edbab14b8ce43fa
Sorry, forgot that method was not in the base class, and I had only tested with uca based collations. This breaks on uppercase type collations.
This reverts commit 6eb84144df
Change-Id: Ib7b9597ff842a76185ba5c153922834ffb741237
Squiz.WhiteSpace.LanguageConstructSpacing:
Language constructs must be followed by a single space;
expected "require_once expression" but found
"require_once(expression)"
It is a keyword (e.g. like `new`, `return` and `print`). As
such the parentheses don't make sense.
Per our code conventions, we use a space after keywords like
these. We appeared to have an unwritten exception for `require`
that doesn't make sense. About 60% of require/include usage
was missing the space and/or had superfluous parentheses.
It is as silly as print("foo") or return("foo"), it works
because keywords have no significance for whitespace between
it and the expression that follows, and since experessions can
be wrapped in parentheses for clarity (e.g. when doing string
concatenation or mathematical operations) the parenthesis
before and after basiclaly just ignored.
Change-Id: I2df2f80b8123714bea7e0771bf94b51ad5bb4b87
Apparently cl_timestamp=cl_timestamp is a workaround for obscure
behaviour of the timestamp type in MySQL
Change-Id: I803f20bcf4e28e8e2833a07bcf00e7edc00ad84b
Have updateCollation.php order by cl_to, so that each category is
updated all at once. This minimises the time during which a category
will appear to be incorrectly sorted, while the maintenance script is in
progress.
Mark the cl_collation index as needing deletion, it was always pretty
pointless. You can't do much better than a full table scan when you're
changing the collation value on a wiki.
Increase the batch size since the lack of a cl_to,cl_from index means
that it will have to filesort each category. A larger batch size means
less sorts. As noted by Liangent on bug 45970, you can't order by
cl_sortkey since that will change during execution.
Also fix an inappropriate use of $wgMiserMode and remove a no-op from
the SET clause of the UPDATE.
Very lightly tested.
Change-Id: I19bc8d6701f5f78040aa9c521427ac98ef488d89
Variables in classes should be declared using public $foo
instead of var $foo for various reasons. As we require PHP 5.3
we don't have to take care about that PHP4 left over, but can
get rid of it in favour of the more clear and better readable
public.
See also: http://php.net/manual/en/language.oop5.visibility.php
(Divided into several commits to keep reviewable)
Change-Id: Ic723d0347ab2e3c78bc0097345c68bbee3dc035a
We can now do this since we finally switched to PHP 5.3 for MW 1.20 and get rid of the silly dirname(__FILE__) stuff :)
Change-Id: Id9b2c9cd2e678197aa81c78adced5d1d31ff57b1
Added a feature allowing updateCollation.php to show a histogram of
sort key sizes, to assess the effect of index size truncation. Added
--dry-run and --target-collation options to allow the index truncation
to be assessed without actually changing the collation.
Change-Id: I497b5d0740384f5d6fdebc6d5ccfea5d853fbd37
* Wait for slaves after every thousand rows rather than after processing every batch. r83992 had 1000 hard-coded, I put it in SYNC_INTERVAL
* Set $lb->waitTimeout(100000). I have no idea why, but it was in the live hack. Maybe Tim or Domas could enlighten me
* Use a STRAIGHT JOIN for the query on categorylinks and page because MySQL appears to want to join the tables the wrong way around
* Use cl_collation='previousValue' rather than cl_collation!='newValue' if possible. This was originally a dirty live hack, but I re-implemented it nicely with a --previous-collation command line option
* Print a status update both before and after the SELECT query. This allows the user to notice when the SELECT queries are getting increasingly slower, which is an indication you may want to set --previous-collation
It selects that many rows, then does PHP processing and an individual
update query for each one. This is not a good idea when each batch is
done in a single transaction: 1000 MySQL updates interspersed with PHP
processing might take a second or more while locks are held.
* Don't run a COUNT(*) query on what's potentially the entire categorylinks table on enwiki (hundreds of millions of rows). Put it in a miser mode check
* Wait for DB replication to catch up before processing the next batch. Implemented LoadBalancer::waitAll() for this purpose, which should behave more nicely than wfWaitForSlaves()
* Added a maintenance script which generates a list of first letters. Unified Han are omitted for performance, and because they shouldn't be used as headings anyway. A future collation specific to Chinese would provide the KangXi radicals as "first letters".
* Provided a precomputed list of first letters. Used Unicode 6.0.0 data and ICU 4.2.
* Moved collation functionality from Language to a Collation class hierarchy with factory function. Removed the recently-added methods from Language and updated all callers.
* Changed Title::getCategorySortkey() to separate its parts with a line break instead of a null character. All collations supported by the intl extension ignore the null character, i.e. "ab" == "a\0b". It would have required a lot of hacking to make it work.
* Fixed the uppercase collation to handle non-ASCII characters, redundantly with r80436. I don't think it's necessary to change the collation name as was done there, so I reverted that in the course of my conflict merge. A --force option to updateCollation.php might be nice though.
I know the uppercase thing is just a standby until a real collation function is written. However in the
mean time, i think it'd be really weird for a wiki with $wgCapitalLinks = false to suddenly have
[[a]] and [[A]] sort under the same letter in a category page, but [[Ä]] and [[ä]] sort no where
near each other, even though on a capitalized wiki they would be the same page.
See discussion on r69816.
Also fix an issue with maintenance/updateCollation.php, where php thinks
that 'uppercase' == 0 (?!). I don't really know what the deal with that
is, but using a ! instead of == 0 seems to fix it. (Follow-up r69961)
Until now, we relied on setting MW_NO_SETUP which was a) hacky, b) irreversable, and c) likely to be forgotten if you didn't use one of the wrappers like runChild().
Instead, move the freaky magic to doMaintenance and have *it* check if it's in a specific call stack that indicates this is being run from the file scope and should be executed. Rename DO_MAINTENANCE to RUN_MAINTENANCE_IF_MAIN so it's nice and clear what magic happens behind the require_once().
