Commit graph

3407 commits

Author SHA1 Message Date
jenkins-bot
fd3020b8cd Merge "Allow setting max execution time to several special pages" 2021-12-16 01:54:31 +00:00
Amir Sarabadani
627c1c2c30 Allow setting max execution time to several special pages
These special pages and their counter-part APIs are:
 - Special:RecentChanges
 - Special:Watchlist
 - Special:Log
 - Special:Contributions

This also changes the way MAX_EXECUTION_TIME works from taking the value
as milliseconds, it takes seconds which is more intuitive for users.

Bug: T297708
Depends-On: I126e7181422d8da1a63afc3717faa4f72a687dd9
Change-Id: I3ff78751c3df3b6342f1865d35c2075f4415185d
2021-12-15 22:59:06 +01:00
jenkins-bot
c0e00b43bd Merge "externalstore: Improve overall documentation" 2021-12-15 05:27:48 +00:00
Timo Tijhof
f21caa41d3 externalstore: Improve overall documentation
* Consolidate various duplicate docs to the main entry point.
  Especially the duplication that followed after ExternalStoreFactory
  and ExternalStoreAccess were introduced with ExternalStore
  left as deprecated wrapper.

* Officially mark ExternalStoreFactory as internal, pointing
  to ExternalStoreAccess instead. And document why the latter exists,
  since it seems all its methods could trivially move to the factory
  class. Follows-up I40c3b553, where it was explained that the factory
  is meant to be an internal implementation detail, with the access
  class meant to resemble ExternalStoreMedium.

* Use consistent terms for "protocol", "location" and "object"
  (reduce use of words like medium, backend, type, blob, revision,
  etc. within this code).

* Document that ExternalStore is fundamentally designed as an
  append-only store (per T247383).

Bug: T247383
Change-Id: I1094af7d35a14f9e47b8791f3e4cc888e8cbfc7f
2021-12-15 02:03:47 +00:00
jenkins-bot
d81f7ee110 Merge "rdbms: add query timeout support to Database::select()" 2021-12-13 20:33:22 +00:00
jenkins-bot
f499129780 Merge "rdbms: Remove five deprecated methods from IDatabase" 2021-12-13 20:24:16 +00:00
jenkins-bot
7032a10719 Merge "Remove some more comments that literally repeat the code" 2021-12-13 20:13:26 +00:00
Aaron Schulz
46425659fa rdbms: add query timeout support to Database::select()
This will modify the SQL to add a timeout specifier, depending
on whether the server is running MySQL or MariaDB, and whether
such features are supported for the version number.

Bug: T129093
Bug: T195792
Change-Id: I12e095fe82622064e555a42f1812eb99d201f10c
2021-12-13 20:08:16 +00:00
jenkins-bot
74202df144 Merge "rdbms: Explicitly set Sqlite PDO ATTR_ERRMODE to ERRMODE_SILENT" 2021-12-13 17:34:00 +00:00
Aaron Schulz
a82b186df1 rdbms: Explicitly set Sqlite PDO ATTR_ERRMODE to ERRMODE_SILENT
This to maintain forward-compat and consistency for the newer PDO
version in PHP 8, which changes the default from ERRMODE_SILENT to
ERRMODE_EXCEPTION.

https://www.php.net/manual/en/pdo.error-handling.php

Change-Id: Ieb6bbcbe884fcb8861e38c10b745d8a7502db7db
2021-12-13 17:13:30 +00:00
Amir Sarabadani
8dca355526 rdbms: Remove five deprecated methods from IDatabase
They are not used in anything deployed to production
These methods are:
  - ::getTopologyRootMaster() since 1.37
  - ::masterPosWait() since 1.37
  - ::dataSeek() soft-deprecated since 1.37
  - ::onTransactionIdle() soft-deprecated since 1.32
  - ::getMasterPos() since 1.37

Bug: T296960
Change-Id: I1d196172b3f50e1200980c6325beccecac7db903
2021-12-13 09:07:54 +01:00
Translation updater bot
4e43885ccb Localisation updates from https://translatewiki.net.
Change-Id: I6aca0d11cf0ed6b86de67d22b9fdef27c39b074c
2021-12-13 08:08:38 +01:00
jenkins-bot
b44af6c975 Merge "rdbms: Drop IDatabase::aggregateValue()" 2021-12-12 02:06:50 +00:00
Aaron Schulz
99bd8bddec rdbms: remove pointless DatabasePostgres::tableName() method
Change-Id: I89d92755c6e662a58dcebe70493a29ecbebf6fbd
2021-12-09 17:52:40 -08:00
Amir Sarabadani
41ca4f821b rdbms: Fix query prefix logging
Right now, the check is on whether topologyRole is truthy and it's
always trruthy making logger to prefix every query with "query-m".

Follows-up I437ce43432660 (fb621c26a3).

Change-Id: I691ac772010b914b3f13b6b9897764b829a2c632
2021-12-09 21:11:05 +00:00
Thiemo Kreuz
b4c63c64ae Remove some more comments that literally repeat the code
Nothing to learn from these.

You can find a longer explanation in the comments in I93751e6.

Change-Id: I195aae70fc282b58be5b18160783f27d38605d15
2021-12-09 19:01:36 +01:00
Amir Sarabadani
76c761a845 rdbms: Drop IDatabase::aggregateValue()
Unused

Bug: T296960
Change-Id: I098d8e8d47709d2c754c1bb107a9a57e811ec6f4
2021-12-09 15:54:11 +01:00
jenkins-bot
7a554d38d6 Merge "Add IDatabase::factorConds()" 2021-12-08 23:10:57 +00:00
Amir Sarabadani
72ada3c108 rdbms: Remove IDatabase::getServerUptime()
Unused, why it's even there?

Bug: T296960
Change-Id: I4e2acc34665c2f3a94db4a8e2f269137600c9903
2021-12-08 11:34:24 +01:00
Tim Starling
4ba64a1967 Add IDatabase::factorConds()
Generalize makeWhere2d() to provide factorConds(), which simplifies any
boolean expression consisting of equality conditions in a
sum-of-products form.

Change-Id: Ic2d6b054b69fd0be0577807bcc4afb919af7d693
2021-12-08 11:02:46 +11:00
Amir Sarabadani
3522939464 rdbms: Remove IDatabase::pendingWriteRowsAffected()
Completely unused

Bug: T296960
Change-Id: I0f3128e56ec6d8a7ba6f8deda6397c8f486c0c5a
2021-12-02 20:15:51 +01:00
Amir Sarabadani
6c4ad39c43 Remove IDatabase::preCommitCallbacksPending()
Completely unused

Bug: T296960
Change-Id: I7626a442a4295386e52f481a20e658630bc2fadd
2021-12-02 20:12:31 +01:00
Translation updater bot
2682461394 Localisation updates from https://translatewiki.net.
Change-Id: I61bcc53970621c7d58bbbcf729002958c5e26e8f
2021-12-02 08:22:01 +01:00
Aaron Schulz
12946b629a objectcache: improve error status codes in MemcachedPeclBagOStuff
Use ERR_UNREACHABLE and ERR_NO_RESPONSE in various error cases,
rather than just using ERR_UNEXPECTED.

Bug: T252951
Change-Id: I5c15362132b8c80aad973520e0addc474bb38c22
2021-12-01 04:33:17 +00:00
Kunal Mehta
12dbaa8638 Log headers in MultiHttpClient
These are useful when debugging requests that vary behavior based on
the headers specified, like Swift's x-copy-from. Additionally,
knowing the content-length is helpful when looking at how long
the request took.

We do know that some headers are likely to contain sensitive
passwords/tokens, so try to redact those ahead of time using a regex of:

 (authorization|auth|password|cookie)

This will catch the standard Authorization header, plus Swift's
X-Auth-Token and X-Auth-Key. The regex is a best effort, but not
intended to be fully comprehensive as there's already plenty of
private data being included in logs.

Change-Id: I6e695f357f92ce8d40b54ea39002fb7858f5596e
2021-11-29 22:57:37 +00:00
jenkins-bot
1d3b49bd8b Merge "rdbms: Add DB host to TransactionProfiler logging and fix time fields" 2021-11-29 16:33:07 +00:00
Amir Sarabadani
7f54729e20 rdbms: Add DB host to TransactionProfiler logging and fix time fields
DB hostnames are extremely useful in getting the info when they are on
different MariaDB versions for example or their table stats need
refreshing.

Time fields are not being counted as numbers in Logstash because these
values are conflicting with other values that are string. Let's fix it.

Bug: T295706
Change-Id: Iff1c1087a7b64a51b5cbd7ecf13afc87d452c352
2021-11-29 16:14:04 +00:00
Martin Urbanec
a1683f52c6 ILBFactory: Add @throws for newExternalLB/getExternalLB
Change-Id: I98d0a44a42aa8a98844927e187937213fa5b7041
2021-11-28 15:09:29 +01:00
Gergő Tisza
0ff1138537
StatusValue: improve toString formatting
Don't just cut off the message key at 25 characters, it makes
debugging very annoying. But do cut off the parameter at some
reasonable length, it could be arbitrarily large.

Change-Id: I8a2665434d6370477eb36a6827eb634e230d280a
2021-11-25 21:53:21 -08:00
Kunal Mehta
5cadb281d6 MultiHttpClient: Check if $wgLocalHTTPProxy is false, not null
The default not configured value for $wgLocalHTTPProxy is false, not
null. So in MultiHttpClient make sure it defaults to false and check for
false when deciding whether to use the proxy.

MWHttpRequest already has the correct behavior.

Bug: T296312
Change-Id: I3962c24f9cb159fb7d1e194e59c8bef5261371c3
2021-11-23 10:17:50 -08:00
jenkins-bot
3690fb0f5d Merge "Use DeletePage in FileDeleteForm and fix output of ApiDelete" 2021-11-23 09:55:55 +00:00
jenkins-bot
479cde3fca Merge "Support $wgLocalHTTPProxy in MultiHttpClient" 2021-11-22 21:36:42 +00:00
Reedy
c054b10423 Remove or replace usages of "sane"
Bug: T254646
Change-Id: Ia660ab95353cd8f05c50e60f30d29fd22b018a43
2021-11-22 15:48:03 +00:00
Reedy
2a2bb1e9bd Remove or replace usages of "sane"
Bug: T254646
Change-Id: I096b2cf738a1395a14f1d47bcbed0c2c686c2581
2021-11-22 13:35:17 +00:00
Reedy
7bf779524a Remove or replace usages of "sanity"
Bug: T254646
Change-Id: I2b120f0b9c9e1dc1a6c216bfefa3f2463efe1001
2021-11-19 23:19:42 +00:00
jenkins-bot
cba4a2e8d8 Merge "rdbms: Document not adding varying ids to SQL comments" 2021-11-19 00:30:01 +00:00
jenkins-bot
e42a8536c3 Merge "rdbms: Add full query to transaction profiler" 2021-11-19 00:26:41 +00:00
jenkins-bot
1dafb67bc4 Merge "MemoizedCallable: Remove outdated php5-apc support" 2021-11-18 20:19:25 +00:00
Amir Sarabadani
c9afd8c4e0 rdbms: Add full query to transaction profiler
Full query is a blocker for migrating off and killing tendril.

Bug: T295706
Change-Id: Id848d3bd90e05d9a5c3a8fb8d848702ac4404ca5
2021-11-18 07:42:39 +01:00
Amir Sarabadani
0b34aba500 rdbms: Make TransactionProfiler logs more useful
- Remove TrxId from GeneralizedSql. It's actually making the whole
  "generalized" part of sql useless and takes away the ability to
  aggregate based on normalized value.
- Add TrxId as part of violation message.

Bug: T295706
Change-Id: I3d0d99b5eef312d3ba5f8aecae2f4bfd027fa34a
2021-11-18 04:44:05 +00:00
Amir Sarabadani
8d5be723fe rdbms: Document not adding varying ids to SQL comments
As the comment explains it would break aggregation of similar queries
in analysis tools.

Bug: T193050
Change-Id: I6e9939e34287d274302f984327685de81a19cdec
2021-11-18 03:51:13 +01:00
Daimona Eaytoy
b77ae63e48 Use DeletePage in FileDeleteForm and fix output of ApiDelete
- Use DeletePage in FileDeleteForm instead of
  WikiPage::doDeleteArticleReal
- Properly handle scheduled deletions in FileDeleteForm: previously, a
  null status value could indicate a missing page OR a scheduled
  deletion, but the code always assumed the first, and it would generate
  a duplicated log entry. The API response would also not contain the
  "delete-scheduled" message. This has been broken since the introduction
  of scheduled deletion.
- In ApiDelete, for file deletions, check whether the status is OK not
  good. The two might be equivalent, but this way it's more consistent.
- Add some documentation for the Status objects returned by file-related
  methods. This is still incomplete, as there are many methods using
  Status and none of them says what the status could be. In particular,
  this means that for now we keep checking whether the status is OK
  instead of good, even though it's unclear what could produce a
  non-fatal error.
- In LocalFileDeleteBatch, avoid using a class property for the returned
  status, as that's hard to follow. Instead, use a local variable and
  pass it around when needed.

Bug: T288758
Change-Id: I22d60c05bdd4a3ea531e63dbb9e49efc36935137
2021-11-18 01:41:51 +01:00
Kunal Mehta
f539b6f58c Support $wgLocalHTTPProxy in MultiHttpClient
This follows the same approach as MWHttpRequest, we inject
$wgLocalVirtualHosts, and for each request check whether it matches that
domain list, rewriting the request as necessary.

Unfortunately this requires a decent amount of code duplication because
MultiHttpClient is in includes/libs/ and can't depend on the same code
in MWHttpRequest.

Bug: T288848
Change-Id: Ia16d8f86b1cb20dde9fe487729d67d92af650cfe
2021-11-16 11:13:10 -08:00
jenkins-bot
990ec777dc Merge "Add Message/MessageValue user group member parameter type" 2021-11-16 19:08:33 +00:00
Timo Tijhof
3445c87ecf MemoizedCallable: Remove outdated php5-apc support
Change-Id: I4a3932a33a200cd0f51c5abf14c94476a755156c
2021-11-16 18:23:22 +00:00
TChin
349819dc5a Add Message/MessageValue user group member parameter type
* Added ParamType::OBJECT, which allows Stringable objects to be passed into MessageValue

Bug: T278482
Change-Id: Ib4990f87d4ad70b7525d7aa05c8b97e90c121674
2021-11-16 11:24:35 -05:00
jenkins-bot
81d2446493 Merge "Stop using LocalFile::lock()" 2021-11-16 16:07:31 +00:00
jenkins-bot
c5b2c30085 Merge "rdbms: make Database transaction IDs more collision resistant" 2021-11-15 22:54:22 +00:00
Tim Starling
216f813239 Stop using LocalFile::lock()
LocalFile::lock() held an explicit DB transaction open while a
potentially slow file backend operation was in progress, causing
production errors.

Instead, protect write operations with non-blocking locks. If a file is
locked, report the failure to the user. Like an edit conflict, informing
the user is appropriate in most cases. For example, if two admins try to
delete the same file at the same time, only one should succeed, and the
second does not need to wait for the first.

In upgradeRow(), locks were used to protect operations which read from a
file and write the result back to the database. The lock was supposed
to prevent overwriting of fresh data with stale data. Use a timestamp
check instead.

Bug: T283045
Change-Id: Iaf9c9eef617044656b5b0d91a9632eacd84a5a78
2021-11-15 13:56:53 +11:00
Aaron Schulz
66ca4ecef0 rdbms: rename Database::wasConnectionError() and make it protected
This method is for internal use only

Change-Id: I4c0bd1530dc76b521ce9cbfc75baf0321ee1e9a0
2021-11-10 15:01:08 -08:00