It selects that many rows, then does PHP processing and an individual
update query for each one. This is not a good idea when each batch is
done in a single transaction: 1000 MySQL updates interspersed with PHP
processing might take a second or more while locks are held.
* Don't run a COUNT(*) query on what's potentially the entire categorylinks table on enwiki (hundreds of millions of rows). Put it in a miser mode check
* Wait for DB replication to catch up before processing the next batch. Implemented LoadBalancer::waitAll() for this purpose, which should behave more nicely than wfWaitForSlaves()
* Added a maintenance script which generates a list of first letters. Unified Han are omitted for performance, and because they shouldn't be used as headings anyway. A future collation specific to Chinese would provide the KangXi radicals as "first letters".
* Provided a precomputed list of first letters. Used Unicode 6.0.0 data and ICU 4.2.
* Moved collation functionality from Language to a Collation class hierarchy with factory function. Removed the recently-added methods from Language and updated all callers.
* Changed Title::getCategorySortkey() to separate its parts with a line break instead of a null character. All collations supported by the intl extension ignore the null character, i.e. "ab" == "a\0b". It would have required a lot of hacking to make it work.
* Fixed the uppercase collation to handle non-ASCII characters, redundantly with r80436. I don't think it's necessary to change the collation name as was done there, so I reverted that in the course of my conflict merge. A --force option to updateCollation.php might be nice though.
I know the uppercase thing is just a standby until a real collation function is written. However in the
mean time, i think it'd be really weird for a wiki with $wgCapitalLinks = false to suddenly have
[[a]] and [[A]] sort under the same letter in a category page, but [[Ä]] and [[ä]] sort no where
near each other, even though on a capitalized wiki they would be the same page.
See discussion on r69816.
Also fix an issue with maintenance/updateCollation.php, where php thinks
that 'uppercase' == 0 (?!). I don't really know what the deal with that
is, but using a ! instead of == 0 seems to fix it. (Follow-up r69961)
Until now, we relied on setting MW_NO_SETUP which was a) hacky, b) irreversable, and c) likely to be forgotten if you didn't use one of the wrappers like runChild().
Instead, move the freaky magic to doMaintenance and have *it* check if it's in a specific call stack that indicates this is being run from the file scope and should be executed. Rename DO_MAINTENANCE to RUN_MAINTENANCE_IF_MAIN so it's nice and clear what magic happens behind the require_once().
Per review by Tim, I made two changes:
1) Fix cl_sortkey to be varbinary(255).
2) Expand cl_collation to varbinary(32), and change $wgCollationVersion
to $wgCategoryCollation, to account for the variety of collations we
might have. tinyint is too small. I could have gone with int, but
that's annoyingly inscrutable in practice, as we all know from namespace
fields.
To make the upgrade easier for non-trunk users, I updated the old patch
file to incorporate the new changes, using the updatelog table so that
people upgrading from 1.16 won't have to do two alters on categorylinks.
I didn't test the upgrade-from-1.16 code path yet, so if anyone tests
that and it seems not to break, commenting to that effect would be
appreciated.
Also removed wfDeprecated() from archive(). Do *not* add this to
functions that are still actively used in core. If you think this
function is so terrible that it really mustn't be used, remove callers
yourself, don't pester every single developer with messages in the hope
that someone else will do it for you.
For those crazy Wikinews people, and other DPL users. Why do we use a
crazy auto-updating column type instead of specifying the current time
explicitly when we want to update it, again . . . ?
This removes $wgCategoryPrefixedDefaultSortkey and effectively always
makes it false. The setting was added in the first place to hack around
the default, clearly broken behavior, but this just fixes it instead, so
the setting is no longer needed.
Running maintenance/updateCollation.php for the first time will fix
this, no need to run refreshLinks.php. If you've already run
updateCollation.php, you can do UPDATE categorylinks SET cl_collation =
76; or such and then run the script again.
Patch best viewed with whitespace changes ignored. This will doubtless
introduce a bunch of bugs. Please report any so I can fix them. If
they're big enough and the fix isn't obvious, please revert.
As Bawolff pointed out at [[mw:User talk:Simetrical/Collation]], the
prefixing scheme I was using meant that the page "Z" with sort key of
"F" would sort after a page named "A" with a sort key of "FF", since the
first one's raw sort key would compute to "FZ", and the second's would
compute to "FFA". I've fixed this by separating the prefix from the
unprefixed part by a null byte (cl_sortkey is eventually going to be
totally binary anyway, may as well start now).
In response to feedback by Phillipe Verdy on bug 164. Now if a bunch of
pages have [[Category:Foo| ]], they'll sort amongst themselves according
to page name, instead of in basically random order as it is currently.
This also makes storage more elegant and intuitive: instead of giving
NULL a magic meaning when there's no custom sortkey specified, we just
store an empty string, since there's no prefix.
This means {{defaultsort:}} really now means {{defaultsortprefix:}},
which is slightly confusing, and a lot of code is now slightly
misleading or poorly named. But it should all work fine.
Also, while I was at it, I made updateCollation.php work as a transition
script, so you can apply the SQL patch and then run updateCollation.php
and things will work. However, with the new schema it's not trivial to
reverse this -- you'd have to recover the raw sort keys with some PHP.
Conversion goes at about a thousand rows a second for me, and seems to
be CPU-bound. Could probably be optimized.
I also adjusted the transition script so it will fix rows with collation
versions *greater* than the current one, as well as less. Thus if some
site wants to use their own collation, they can call it 137 or
something, and if they later want to switch back to MediaWiki stock
collation 7, it will work.
Also fixed a silly bug in updateCollation.php where it would say "1000
done" if it did nothing, and changed $res->numRows() >= self::BATCH_SIZE
to == so people don't wonder how it could be bigger (since it can't, I
hope).
It seemed to work correctly, with the newly-created page "bob" sorting
as "BOB", but then I nuked all my cl_sortkey by running the migration
script before refreshLinks.php had finished running, so I'll have to
wait a while to see if it works properly with a non-messed-up database.
It's possible there's something wrong with the display of section
letters in the categories, but otherwise I think this is working right.
