Using the cl_sortkey index instead (to reduce disruption to a live
site), as currently implemented, seems to have two serious problems:
* MySQL / MariaDB filesorts all rows that "sort above the given row
[the last row of the previous batch]", not just a single category
at a time until the row limit is reached.
* The current approach to pagination is broken in that it does not
work with ENUM columns such as cl_type, causing 'file' rows to be
skipped, or rows of any type to be repeated. See T119173.
This reverts part of commit a43f751cf6.
Bug: T58041
Change-Id: I619564e85b2122f249bdacc45d547b9ce1b3beb5
Add transaction methods to complement getDB().
This makes it easy to grep for direct begin()/commit()
calls to IDatabase by having script use their own
wrapper. Maintenance scripts are one of the few places
that can (and need to) use begin/commit instead of the
start/end atomic methods.
Eventually, there should be almost no direct callers
and those methods can be made stricter about throwing
errors on nested calls.
Change-Id: Ibbfc7a77c0d2a55f7fc2261087f6c3a19061e0aa
In some cases the constructor will work, but trying to access first
letter data will raise an exception, breaking all category pages.
Bug: 46615
Change-Id: I77de040f97080653fe0d1734d38490eaa2d322db
Follows-up I1343872de7, Ia533aedf63 and I2df2f80b81.
Also updated usage in text in documentation and the
installer LocalSettingsGenerator.
Most of them were handled by this regex:
- find: (require|include|require_once|include_once)\s*\(\s*(.+?)\s*\)\s*;$
- replace: $1 $2;
Change-Id: I6b38aad9a5149c9c43ce18bd8edbab14b8ce43fa
Sorry, forgot that method was not in the base class, and I had only tested with uca based collations. This breaks on uppercase type collations.
This reverts commit 6eb84144df
Change-Id: Ib7b9597ff842a76185ba5c153922834ffb741237
Squiz.WhiteSpace.LanguageConstructSpacing:
Language constructs must be followed by a single space;
expected "require_once expression" but found
"require_once(expression)"
It is a keyword (e.g. like `new`, `return` and `print`). As
such the parentheses don't make sense.
Per our code conventions, we use a space after keywords like
these. We appeared to have an unwritten exception for `require`
that doesn't make sense. About 60% of require/include usage
was missing the space and/or had superfluous parentheses.
It is as silly as print("foo") or return("foo"), it works
because keywords have no significance for whitespace between
it and the expression that follows, and since experessions can
be wrapped in parentheses for clarity (e.g. when doing string
concatenation or mathematical operations) the parenthesis
before and after basiclaly just ignored.
Change-Id: I2df2f80b8123714bea7e0771bf94b51ad5bb4b87
Apparently cl_timestamp=cl_timestamp is a workaround for obscure
behaviour of the timestamp type in MySQL
Change-Id: I803f20bcf4e28e8e2833a07bcf00e7edc00ad84b
Have updateCollation.php order by cl_to, so that each category is
updated all at once. This minimises the time during which a category
will appear to be incorrectly sorted, while the maintenance script is in
progress.
Mark the cl_collation index as needing deletion, it was always pretty
pointless. You can't do much better than a full table scan when you're
changing the collation value on a wiki.
Increase the batch size since the lack of a cl_to,cl_from index means
that it will have to filesort each category. A larger batch size means
less sorts. As noted by Liangent on bug 45970, you can't order by
cl_sortkey since that will change during execution.
Also fix an inappropriate use of $wgMiserMode and remove a no-op from
the SET clause of the UPDATE.
Very lightly tested.
Change-Id: I19bc8d6701f5f78040aa9c521427ac98ef488d89
Variables in classes should be declared using public $foo
instead of var $foo for various reasons. As we require PHP 5.3
we don't have to take care about that PHP4 left over, but can
get rid of it in favour of the more clear and better readable
public.
See also: http://php.net/manual/en/language.oop5.visibility.php
(Divided into several commits to keep reviewable)
Change-Id: Ic723d0347ab2e3c78bc0097345c68bbee3dc035a
We can now do this since we finally switched to PHP 5.3 for MW 1.20 and get rid of the silly dirname(__FILE__) stuff :)
Change-Id: Id9b2c9cd2e678197aa81c78adced5d1d31ff57b1
Added a feature allowing updateCollation.php to show a histogram of
sort key sizes, to assess the effect of index size truncation. Added
--dry-run and --target-collation options to allow the index truncation
to be assessed without actually changing the collation.
Change-Id: I497b5d0740384f5d6fdebc6d5ccfea5d853fbd37
* Wait for slaves after every thousand rows rather than after processing every batch. r83992 had 1000 hard-coded, I put it in SYNC_INTERVAL
* Set $lb->waitTimeout(100000). I have no idea why, but it was in the live hack. Maybe Tim or Domas could enlighten me
* Use a STRAIGHT JOIN for the query on categorylinks and page because MySQL appears to want to join the tables the wrong way around
* Use cl_collation='previousValue' rather than cl_collation!='newValue' if possible. This was originally a dirty live hack, but I re-implemented it nicely with a --previous-collation command line option
* Print a status update both before and after the SELECT query. This allows the user to notice when the SELECT queries are getting increasingly slower, which is an indication you may want to set --previous-collation
It selects that many rows, then does PHP processing and an individual
update query for each one. This is not a good idea when each batch is
done in a single transaction: 1000 MySQL updates interspersed with PHP
processing might take a second or more while locks are held.
* Don't run a COUNT(*) query on what's potentially the entire categorylinks table on enwiki (hundreds of millions of rows). Put it in a miser mode check
* Wait for DB replication to catch up before processing the next batch. Implemented LoadBalancer::waitAll() for this purpose, which should behave more nicely than wfWaitForSlaves()
* Added a maintenance script which generates a list of first letters. Unified Han are omitted for performance, and because they shouldn't be used as headings anyway. A future collation specific to Chinese would provide the KangXi radicals as "first letters".
* Provided a precomputed list of first letters. Used Unicode 6.0.0 data and ICU 4.2.
* Moved collation functionality from Language to a Collation class hierarchy with factory function. Removed the recently-added methods from Language and updated all callers.
* Changed Title::getCategorySortkey() to separate its parts with a line break instead of a null character. All collations supported by the intl extension ignore the null character, i.e. "ab" == "a\0b". It would have required a lot of hacking to make it work.
* Fixed the uppercase collation to handle non-ASCII characters, redundantly with r80436. I don't think it's necessary to change the collation name as was done there, so I reverted that in the course of my conflict merge. A --force option to updateCollation.php might be nice though.
I know the uppercase thing is just a standby until a real collation function is written. However in the
mean time, i think it'd be really weird for a wiki with $wgCapitalLinks = false to suddenly have
[[a]] and [[A]] sort under the same letter in a category page, but [[Ä]] and [[ä]] sort no where
near each other, even though on a capitalized wiki they would be the same page.
See discussion on r69816.
Also fix an issue with maintenance/updateCollation.php, where php thinks
that 'uppercase' == 0 (?!). I don't really know what the deal with that
is, but using a ! instead of == 0 seems to fix it. (Follow-up r69961)
Until now, we relied on setting MW_NO_SETUP which was a) hacky, b) irreversable, and c) likely to be forgotten if you didn't use one of the wrappers like runChild().
Instead, move the freaky magic to doMaintenance and have *it* check if it's in a specific call stack that indicates this is being run from the file scope and should be executed. Rename DO_MAINTENANCE to RUN_MAINTENANCE_IF_MAIN so it's nice and clear what magic happens behind the require_once().
Per review by Tim, I made two changes:
1) Fix cl_sortkey to be varbinary(255).
2) Expand cl_collation to varbinary(32), and change $wgCollationVersion
to $wgCategoryCollation, to account for the variety of collations we
might have. tinyint is too small. I could have gone with int, but
that's annoyingly inscrutable in practice, as we all know from namespace
fields.
To make the upgrade easier for non-trunk users, I updated the old patch
file to incorporate the new changes, using the updatelog table so that
people upgrading from 1.16 won't have to do two alters on categorylinks.
I didn't test the upgrade-from-1.16 code path yet, so if anyone tests
that and it seems not to break, commenting to that effect would be
appreciated.
Also removed wfDeprecated() from archive(). Do *not* add this to
functions that are still actively used in core. If you think this
function is so terrible that it really mustn't be used, remove callers
yourself, don't pester every single developer with messages in the hope
that someone else will do it for you.
For those crazy Wikinews people, and other DPL users. Why do we use a
crazy auto-updating column type instead of specifying the current time
explicitly when we want to update it, again . . . ?
This removes $wgCategoryPrefixedDefaultSortkey and effectively always
makes it false. The setting was added in the first place to hack around
the default, clearly broken behavior, but this just fixes it instead, so
the setting is no longer needed.
Running maintenance/updateCollation.php for the first time will fix
this, no need to run refreshLinks.php. If you've already run
updateCollation.php, you can do UPDATE categorylinks SET cl_collation =
76; or such and then run the script again.
Patch best viewed with whitespace changes ignored. This will doubtless
introduce a bunch of bugs. Please report any so I can fix them. If
they're big enough and the fix isn't obvious, please revert.
As Bawolff pointed out at [[mw:User talk:Simetrical/Collation]], the
prefixing scheme I was using meant that the page "Z" with sort key of
"F" would sort after a page named "A" with a sort key of "FF", since the
first one's raw sort key would compute to "FZ", and the second's would
compute to "FFA". I've fixed this by separating the prefix from the
unprefixed part by a null byte (cl_sortkey is eventually going to be
totally binary anyway, may as well start now).
In response to feedback by Phillipe Verdy on bug 164. Now if a bunch of
pages have [[Category:Foo| ]], they'll sort amongst themselves according
to page name, instead of in basically random order as it is currently.
This also makes storage more elegant and intuitive: instead of giving
NULL a magic meaning when there's no custom sortkey specified, we just
store an empty string, since there's no prefix.
This means {{defaultsort:}} really now means {{defaultsortprefix:}},
which is slightly confusing, and a lot of code is now slightly
misleading or poorly named. But it should all work fine.
Also, while I was at it, I made updateCollation.php work as a transition
script, so you can apply the SQL patch and then run updateCollation.php
and things will work. However, with the new schema it's not trivial to
reverse this -- you'd have to recover the raw sort keys with some PHP.
Conversion goes at about a thousand rows a second for me, and seems to
be CPU-bound. Could probably be optimized.
I also adjusted the transition script so it will fix rows with collation
versions *greater* than the current one, as well as less. Thus if some
site wants to use their own collation, they can call it 137 or
something, and if they later want to switch back to MediaWiki stock
collation 7, it will work.
Also fixed a silly bug in updateCollation.php where it would say "1000
done" if it did nothing, and changed $res->numRows() >= self::BATCH_SIZE
to == so people don't wonder how it could be bigger (since it can't, I
hope).
It seemed to work correctly, with the newly-created page "bob" sorting
as "BOB", but then I nuked all my cl_sortkey by running the migration
script before refreshLinks.php had finished running, so I'll have to
wait a while to see if it works properly with a non-messed-up database.
It's possible there's something wrong with the display of section
letters in the categories, but otherwise I think this is working right.
