Commit graph

681 commits

Author SHA1 Message Date
jhsoby
660caf9b88 Add custom collation for Northern Sami
This commit adds a custom collation order for
Northern Sami ('se'). Northern Sami exists in ICU,
but the version of ICU that Wikimedia uses is a
few years old, and does *not* include Northern
Sami. It could be years before Wikimedia's production
servers use the one that includes Northern Sami (see
bug), so this is a temporary workaround to amend this
issue.

Bug: T181503
Change-Id: Ib8a48b8db99bef8ec4b05144aace5dbdcacfeded
2017-12-07 21:32:11 +00:00
Brad Jorsch
96bd79b4a3 Avoid DB rows with usable names but ID = 0 by introducing "interwiki" usernames
Importing revisions in MediaWiki has long been weird: if the username on
the imported revision exists locally it's automatically attributed to
the local user, while if the name does not exist locally we wind up with
revision table rows with rev_user = 0 and rev_user_text being a valid
name that someone might later create. "Global" blocks too create rows
with ipb_by = 0 an ipb_by_text being a valid name.

The upcoming actor table change, as things currently stand, would
regularize that a bit by automatically attributing those imported
revisions to the newly-created user. But that's not necessarily what we
actually want to happen. And it would certainly confuse CentralAuth's
attempt to detect its own global blocks.

Thus, this patch introduces "interwiki" usernames that aren't valid for
local use, of the format "iw>Example".[1] Linker will interpret these
names and generate an appropriate interwiki link in history pages and
the like, as if from wikitext like `[[iw:User:Example]]`.

Imports for non-existant local users (and optionally for existing local
users too) will credit the edit to such an interwiki name. There is also
a new hook, 'ImportHandleUnknownUser', to allow extension such as
CentralAuth to create local users as their edits are imported.

Block will no longer accept usable-but-nonexistent names for 'byText' or
->setBlocker(). CentralAuth's global blocks will be submitted with an
interwiki username (see Ieae5d24f9).

Wikis that have imported edits or CentralAuth global blocks should run
the new maintenance/cleanupUsersWithNoId.php maintenance script. This
isn't done by update.php because (1) it needs an interwiki prefix to use
and (2) the updater can't know whether to pass the `--assign` flag.

[1]: '>' was used instead of the more usual ':' because WMF wikis have
many existing usernames containing colons.

Bug: T9240
Bug: T20209
Bug: T111605
Change-Id: I5401941c06102e8faa813910519d55482dff36cb
Depends-On: Ieae5d24f9098c1977447c50a8d4e2cab58a24d9f
2017-11-30 13:12:44 +11:00
jenkins-bot
5dad90f357 Merge "Introduce ClearUserWatchlistJob" 2017-11-28 18:31:52 +00:00
addshore
989ba87562 Introduce ClearUserWatchlistJob
Change-Id: Icea573a10078ea3f09dc2e4e9fdc737bf639935d
2017-11-28 17:11:40 +00:00
Kunal Mehta
bdb5b592f4 shell: Optionally restrict commands' access with firejail
Introduces a FirejailCommand class, which can be used to add additional
restrictions to a command, for increased security. For now, firejail
containment needs to be enabled on a per-command basis.

The following restrictions are implemented:
* NO_ROOT - disallows any root access, including via setuid binaries
* SECCOMP - block dangerous syscalls with seccomp
* PRIVATE_DEV - create a private /dev
* NO_NETWORK - deny all network access
* NO_EXECVE - block the execve syscall

A convenient Shell::RESTRICT_DEFAULT is equivalent to NO_ROOT | SECCOMP
| PRIVATE_DEV, with the expectation that more restrictions may be added
to it in the future.

In addition, specific paths can be whitelisted with
Command::whitelistPaths(). Any file/directory that isn't whitelisted in
that top level directory (e.g. /srv) won't exist inside the firejail.

$wgShellRestrictionMethod can be set to false for no restriction system,
'firejail' to explicitly use it, or 'autodetect' to autodetect whatever
system is available. In the future the default should be changed to
autodetection once firejail is tested more.

Bug: T173370
Change-Id: Id74df0dbba40e1e7c07c4368aacffb6eb06a17c5
2017-11-28 00:06:40 +00:00
jenkins-bot
694120c80e Merge "Crimean Tatar Transliteration" 2017-11-20 22:27:57 +00:00
tjones
a0b511319c Crimean Tatar Transliteration
This is a first pass at Latin/Cyrillic translitertion for Crimean
Tatar (crh).

Includes transliteration tables, prefix/suffix mappings, regex
mappings, and exceptions lists for words and abbreviations.

Regularize CRH language name in messages/* files.

Fix "varient" typos in qqq.json.

Add unit tests for CRH transliteration.

Bug: T23582
Change-Id: I424703f99adf837f6217872b882d1ea26bfdd068
2017-11-20 16:56:38 -05:00
daniel
38edb8ab30 Introduce the UserIdentity interface.
This provides a narrow view on a user identity, providing access to
ID and name.

This has been extracted from I140f43a6fb443b for re-use with Actors,
on Anomie's request.

Change-Id: Ief00db5ce382537c5bf992159eae6baf096ae4be
2017-11-18 19:47:25 +00:00
jenkins-bot
2f15b22b91 Merge "Use Remex in Sanitizer::stripAllTags()" 2017-11-16 20:34:31 +00:00
jenkins-bot
c586a099d7 Merge "Move Sanitizer.php to includes/parser/" 2017-11-16 01:33:21 +00:00
Roan Kattouw
ddb4913f53 Use Remex in Sanitizer::stripAllTags()
Using a real HTML tokenizer fixes bugs when < or > appear in attribute
values. The old implementation used delimiterReplace(), which didn't
handle this case:

    > print Sanitizer::stripAllTags( '<p data-foo="a&lt;b>c">Hello</p>' );
    c">Hello

We also can't use PHP's built-in strip_tags() because it doesn't handle
<?php and <? correctly:

    > print strip_tags('1<span class="<?php">2</span>3');
    1
    > print strip_tags('1<span class="<?">2</span>3');
    1

Bug: T179978
Change-Id: I53b98e6c877c00c03ff110914168b398559c9c3e
2017-11-15 17:31:31 -08:00
Roan Kattouw
7980e38a84 Move Sanitizer.php to includes/parser/
Change-Id: Id08d91c747ec77d715459b89b03eee247ccd4e1b
2017-11-15 15:16:41 -08:00
Reedy
d1442d72f8 Add missing ComposerVendorHtaccessCreator class to autoload.php
Change-Id: Ia70324acf3db2df50c6629d705c2c4728c38aaaa
Follow-up: I2cf6541750c90b5708d7cf5f81b914ae2d9d46d1
2017-11-15 02:55:27 +00:00
addshore
34dad25227 Split interface from WatchedItemStore
Change-Id: Ifb029f5c79ee4865ee225d4f44d3354d95014cce
2017-11-14 19:27:03 +01:00
jenkins-bot
5c39992e39 Merge "Introduce ExternalStoreFactory" 2017-11-14 15:00:47 +00:00
addshore
24b24e493e Introduce ExternalStoreFactory
Change-Id: If0d8f503e3cc9fd83f3b40e2ac8a5f9dc8b7e0ea
2017-11-14 13:22:12 +00:00
jenkins-bot
cc122862fa Merge "Move watcheditem classes to watcheditem directory" 2017-11-12 05:57:28 +00:00
addshore
0000ea39a0 Move watcheditem classes to watcheditem directory
Change-Id: If915c875380b4ecd74fad64df7833de87ea6d6f7
2017-11-09 16:16:48 +00:00
Max Semenik
6df5d031c4 Rewrite userOptions.php
* Convert to use Maintenance
* Clean up
* I want to use the class name UserOptions for something else
  so rename it.

Change-Id: Ic441087702376b1ca0e70554c71cdf7ecad908af
2017-11-07 21:32:56 +00:00
jenkins-bot
f8b3f04ae6 Merge "EditPage: Extract some edit conflict code into EditConflictHelper" 2017-10-31 20:09:30 +00:00
Kunal Mehta
0e1fc6575e Refactor some Installer code into ExecutableFinder
Refactor Installer::locateExecutableInDefaultPaths() into a separate
utility class, ExecutableFinder. This class is already used in plenty of
places outside of the installer, so it's ripe for being extracted.

This class is located in utils/ due to the dependency upon
Shell::command(). Once that no longer has a dependence upon MediaWiki,
this class can be moved to libs/ too.

Change-Id: I175465acc0d64f990445ce05fabcee8b88a0b259
2017-10-26 11:42:05 -07:00
Kunal Mehta
3f88e0c7bf EditPage: Extract some edit conflict code into EditConflictHelper
Extensions with custom content models (e.g. ProofreadPage) will extend
EditPage to provide a customized editing experience. However when
another extension like TwoColConflict wants to change just how edit
conflicts look, this causes problems since only one class can take the
place of EditPage.

So instead, split most of the frontend code for edit conflicts into
TextConflictHelper, and call it from edit page. Extensions can override
with the instance created by calling
EditPage::setEditConflictHelperFactory().

And to make that split possible also move EditPage::addNewLineAtEnd()
and ::buildTextboxAttribs() into a separate TextboxBuilder class that
both EditPage and TextConflictHelper can use.

Bug: T176393
Change-Id: Ie415edd84329c02d5762477f8a171fced85b01a2
2017-10-26 10:32:29 +02:00
Max Semenik
32912b8c8d Introduce Shell\CommandFactory
Bug: T177038
Change-Id: Id875e68ea1fa72b44a463f977ab52270fe1e7088
2017-10-17 18:55:11 -07:00
Ori Livneh
692cf27013 Add lowercase variants to the autoloader for legacy history blob classes
Blobs in the text table created by MediaWiki <1.5 running on PHP 4 had class
names coerced to lowercase. In order to succcessfully resolve these class
names, the autoloader creates a copy of the autoload class array with
lowercased keys, which is pretty inefficient. So instead, make sure the
auto-generated autoload map contains lowercase variants for these specific
classes.

Bug: T166759
Change-Id: I6d751ef516e936e04c056d70a1ecd17960a39ceb
2017-10-16 13:56:35 +00:00
jenkins-bot
e994e89e76 Merge "deferred: Introduce TransactionRoundDefiningUpdate" 2017-10-13 21:54:03 +00:00
Timo Tijhof
6b2f8639b6 deferred: Introduce TransactionRoundDefiningUpdate
This allows scheduling of updates that need to start their own
transaction round. Specifically for cases where the ability to
commit early is not enough (which is already possible via LBFactory
getEmptyTransactionTicket and commitAndWaitForReplication).

Change-Id: I0910587b61c8ddf825f91e92c2f93582cc7ebd80
2017-10-13 21:24:11 +00:00
Max Semenik
0ca822a320 Remove maintenance/checkSyntax.php
Was useful years ago, but now we have much better tools.
Not used anywhere search can find.

Change-Id: Iaa094a686aeefd9a6071abc843ffc33c8daf2a5a
2017-10-12 19:54:05 -07:00
Eddie Greiner-Petter
fdb4557a5a Add maintenance script: Clean up expired userrights
This adds a maintenance script that removes all expired user group
entries from the user_groups table and adds them to the
user_former_groups table.

Bug: T176754
Change-Id: Ie0f4afd3dbecd3af42e89bb99cf03dbe03768d07
2017-10-05 20:38:55 +02:00
Matthew Flaschen
3638695419 Detect query timeouts and throw a specific exception
Throw DBQueryTimeoutError if a database query error is detected
to be a timeout.

Only DatabaseMysqlBase has been updated here.

This is a subclass of DBQueryError, so existing catch'es will work.

Bug: T175775
Change-Id: I4749dc33ad530d9b22504f02106b1ca49e8eb167
2017-09-20 08:47:09 +00:00
Bartosz Dziewoński
06daf807ca Split off remaining helper classes for special pages to separate files
includes/specials/formfields/
* EditWatchlistCheckboxSeriesField (for SpecialEditWatchlist)
* UploadSourceField                (for SpecialUpload)
* Licenses                         (for SpecialUpload)

includes/specials/forms/
* EditWatchlistNormalHTMLForm      (for SpecialEditWatchlist)
* PreferencesForm                  (for SpecialPreferences)
* UploadForm                       (for SpecialUpload)

includes/specials/helpers/
* ImportReporter                   (for SpecialImport)
* License                          (for SpecialUpload)

Change-Id: I58abcbb44dbf9bf1762b4252555f7552bfa7c253
2017-09-12 23:22:58 +02:00
Max Semenik
77ce3b98a0 Replace wfShellExec() with a class
This function has gotten so unwieldy that a helper was
introduced. Instead, here's this class that makes
shelling out easier and more readable.

Example usage:
  $result = Shell::command( 'shell command' )
       ->environment( [ 'ENVIRONMENT_VARIABLE' => 'VALUE' ] )
       ->limits( [ 'time' => 300 ] )
       ->execute();

  $exitCode = $result->getExitCode();
  $output = $result->getStdout();

This is a minimal change, so lots of stuff remains
unrefactored - I'd rather limit the scope of this commit.
A future improvement could be an ability to get stderr
separately from stdout.

Caveat: execution errors (proc_open is disabled/returned error) now
throw errors instead of returning a status code. wfShellExec() still
emulates this behavior though.

Competing commit: I7dccb2b67a4173a8a89b035e444fbda9102e4d0f
<legoktm> MaxSem: so you should continue working on your patch and I'll
          probably refactor on top of it later after its merged :P

Change-Id: I8ac9858b80d7908cf7e7981d7e19d0fc9c2265c0
2017-09-08 21:49:49 -07:00
jenkins-bot
c66360d167 Merge "Move OrderedStreamingForkController class from CirrusSearch to core." 2017-09-07 18:27:49 +00:00
Stanislav Malyshev
c82d30d19c Move OrderedStreamingForkController class from CirrusSearch to core.
This class may be useful for scripts outside CirrusSearch.

Change-Id: I1ed993a3b0b64d828ae5caba6c4b540b77b2f6f5
2017-09-07 10:27:43 -07:00
MusikAnimal
d09554b6ef Add basic IP range support to Special:Contributions
This works by using the new table introduced with T156318.

The only thing that differs from normal Special:Contribs is we are
showing the IP address next to each entry. This is it how it is
displayed if you request to see newbie contributions:
https://en.wikipedia.org/wiki/Special:Contributions?contribs=newbie

For the time being, Special:DeletedContributions does not support
IP ranges. Various other irrelevant links such as Uploads and Logs
are also hidden.

Refer to P4725 for a way to automate creation of edits by random
IPs in your dev environment.

IP::isValidBlock() has been deprecated with this dependent change:
https://gerrit.wikimedia.org/r/#/c/373165/

Bug: T163562
Change-Id: Ice1bdae3d16cf365da14c6df0e8d91d2b914e064
2017-09-05 16:15:33 -04:00
jenkins-bot
b936d263ef Merge "Deprecate IDatabase::nextSequenceValue()" 2017-09-01 09:39:07 +00:00
Tim Starling
95bf0043ef EtcdConfig: allow slashes in config key names
Allowing slashes in config key names allows us to trivially support the
proposed hierarchical structure with a single EtcdConfig object, by
fetching values with the relevant prefixes on startup.

Bug: T156924
Change-Id: Ica0914e61baba9c0462481925be15d79b66dc342
2017-09-01 13:42:28 +10:00
Brad Jorsch
0a9c55bfd3 Deprecate IDatabase::nextSequenceValue()
It's often forgotten because MySQL and Sqlite don't use it, the only
users are PostgreSQL and Oracle. And when used, if inserts to multiple
tables are being done it's easy to get the ordering wrong.

This patch reimplements DatabasePostgres::insertId() in terms of PG's
lastval() function, and adds triggers to the Oracle schema to make it
work the same as the other databases.

Bug: T164900
Change-Id: Ib308190c52673a9266c8495a589ae644f9fbefce
2017-08-31 11:16:26 -04:00
Brad Jorsch
11cf01dd9a Add comment table and code to start using it
A subsequent patch will remove the old columns.

Bug: T166732
Change-Id: Ic3a434c061ed6e443ea072bc62dda09acbeeed7f
2017-08-30 15:05:00 +10:00
jenkins-bot
a8ec960e9d Merge "Produce RDF dump of all categories and subcategories in a wiki." 2017-08-28 11:01:58 +00:00
Stanislav Malyshev
d9bb673b72 Produce RDF dump of all categories and subcategories in a wiki.
Example:
http://en.wiki.local.wmftest.net:8080/wiki/Category:Ducks> a mediawiki:Category ;
        rdfs:label "Ducks" ;
        mediawiki:isInCategory <http://en.wiki.local.wmftest.net:8080/wiki/Category:Birds> .

Bug: T157676
Change-Id: I59b9603581b37af59d17dd6c38247c85aee44911
2017-08-28 00:30:35 -07:00
jenkins-bot
daff410fd5 Merge "Remove MemcachedClient compat class names" 2017-08-22 15:16:13 +00:00
jenkins-bot
33bb6888c8 Merge "Remove RunningStat compat class" 2017-08-22 15:16:09 +00:00
jenkins-bot
5918cef6fe Merge "Remove IPSet compat classes" 2017-08-22 15:07:22 +00:00
jenkins-bot
05fa5fa13a Merge "Remove Cdb compat class names" 2017-08-22 15:07:06 +00:00
Kunal Mehta
434ea4f421 Remove MemcachedClient compat class names
Deprecated since 1.27, and unused in Wikimedia Git.

Change-Id: I1d73efac2fddb771124bcd31b3d40769e751410c
2017-08-21 21:50:31 -07:00
Kunal Mehta
4c6fd2bc69 Remove RunningStat compat class
Deprecated since 1.27, unused in Wikimedia Git.

Change-Id: I0086c1cd945865b10a1e7fcc34c08642db2474af
2017-08-21 21:38:16 -07:00
Kunal Mehta
015b74691b Remove IPSet compat classes
Deprecated in 1.26, unused in Wikimedia Git.

Change-Id: I7719a204160ddd0ba71a1a5d8f7088f7d552acbd
2017-08-21 21:34:50 -07:00
Kunal Mehta
7f0486b18e Remove Cdb compat class names
Deprecated since 1.25, and unused in Wikimedia Git.

Change-Id: I05bdefe2e2bf5ffcadb9846651af7367e8e7a814
2017-08-21 21:29:00 -07:00
Brad Jorsch
918e4c394c Delete maintenance/deleteRevision.php
It hasn't been updated properly since 2006 so many fields aren't being
copied to the archive table. Tim suggests it'd be best to just delete it
and, if someone needs the ability to delete or revdel revisions from the
command line, properly abstract out the deletion code instead of
duplicating it.

Change-Id: I400b8ac30b31802e7dd9f6e4d0ec10918eba0183
2017-08-21 13:30:01 -04:00
Florian Schmidt
499f643bef Move RawMessage out of Message.php to its own file
Change-Id: Idae9617dafa3c314085eb097f78f1c8d38672f31
2017-08-14 17:33:52 +00:00