Commit graph

165 commits

Author SHA1 Message Date
James D. Forrester
9fe4138f3d Follow-up I68198bc3: Hard-deprecate getMasterDB() methods
Also switch one use that we missed in the rebase.

Bug: T254646
Depends-On: I632610dec1aa6bb59af5423e6f86164fc791aadd
Change-Id: I9873c889f943fcbd7d867d534a7857f20ef0c5dc
2021-07-15 09:12:17 +01:00
James D. Forrester
597376e12c Replace getMasterDB methods with getPrimaryDB
Bug: T254646
Change-Id: I68198bc39b174ea1920b4acc2617cb6c6ce406e9
2021-07-14 16:21:04 -07:00
Tim Starling
9251f3c9e3 Manual and automatic image metadata reserialization
* Add automatic splitting of large metadata on upload or refresh. If
  the reserializeMetadata option is enabled, metadata stored with PHP
  serialization will be automatically reserialized to JSON.
* Inject configuration variable $wgUpdateCompatibleMetadata via
  LocalRepo instead of accessing it directly.
* In refreshImageMetadata.php and rebuildImages.php, construct a new
  LocalRepo with config overrides, instead of overwriting config
  globals. Add a helper to RepoGroup to help with this.
* In refreshImageMetadata.php, add new options --convert-to-json and
  --split which reserialize metadata and optionally split out large
  items to blob storage.

Also, refreshImageMetadata.php was totally broken in the non-force mode
since metadata refresh on page view was disabled in b814245d9f. The
maintenance script was relying on newFileFromRow() magically upgrading
the row, which doesn't happen anymore. So, call maybeUpgradeRow()
directly.

Bug: T275268
Change-Id: I7bf7d9cef71641e287ca4346b568b381f4ada50e
2021-06-26 22:50:17 +00:00
Tim Starling
68ebdfc77f Optionally split out parts of file metadata to BlobStore
* Optionally store metadata in the database in JSON format instead of
  PHP serialization. The new JSON format has a top-level "envelope"
  array which gives us a place to store things that are not part of the
  handler metadata.
* Optionally split metadata items, putting items above a threshold into
  the text table. The FileRepo and MediaHandler must both opt in.
* For staged deployment, the read side of these changes is always
  active. Only the write side is configurable.

Bug: T275268
Change-Id: I876ea5c9d3a1881e278f689d2f8a3ae20240c703
2021-06-11 08:01:26 +10:00
Petr Pchelko
fb6529e653 FileRepo::findFile - support Authority
Change-Id: Ib42b7f7d5aa88447b4fb363f52062b08a1af30c3
2021-05-26 19:01:12 -07:00
James D. Forrester
f2f9345e39 Replace uses of DB_MASTER with DB_PRIMARY in documentation and local variables
This is just a start.

Bug: T254646
Change-Id: I9213aad4660e27afe7ff9e5d2e730cbf03911068
2021-05-14 12:40:34 -07:00
James D. Forrester
df5eb22f83 Replace uses of DB_MASTER with DB_PRIMARY
Just an auto-replace from codesniffer for now.

Change-Id: I5240dc9ac5929d291b0ef1c743ea2bfd3f428266
2021-04-29 09:24:31 -07:00
daniel
4eebc3491a FileRepo: replace Title in method signatures
NOTE: FileRepo is extended (via LocalRepo) by a couple of extensions,
even though it is not stable to extend. Hwever, none of these extensions
overrides any of the methods changed here.
https://codesearch.wmcloud.org/extensions/?q=extends%20%5Cw%2BRepo%5Cb&i=nope&files=&excludeFiles=&repos=

Bug: T278459
Change-Id: I0c236bfa354b01b03f864dd343371e4ae323b56f
2021-04-16 11:01:48 +00:00
Umherirrender
8de3b7d324 Use static closures where safe to use
This is micro-optimization of closure code to avoid binding the closure
to $this where it is not needed.

Created by I25a17fb22b6b669e817317a0f45051ae9c608208

Change-Id: I0ffc6200f6c6693d78a3151cb8cea7dce7c21653
2021-02-11 00:13:52 +00:00
jenkins-bot
3e91f1be2a Merge "filerepo: remove repo name from getSharedCacheKey()" 2020-11-17 03:09:21 +00:00
Aaron Schulz
57b2e2d6ac filerepo: remove repo name from getSharedCacheKey()
The name is only used for local config and internal virtual URLs.
A key use case is having a LocalRepo named "local" on central repo
wiki and a ForeignDBViaLBRepo named "shared" on the client wikis,
the later being a reference to the former. The keys should be be
shared regardless of this naming difference.

Bug: T267668
Change-Id: Ic239bb980740ec5237edc878af4274885034a96f
2020-11-16 17:28:31 -08:00
Thiemo Kreuz
b0130ca649 Update a lot of unspecific "array" types in PHPDocs
This includes fixing some mistakes, as well as removing
redundant text that doesn't add new information, either because
it literally repeats what the code already says, or is actually
duplicated.

Change-Id: I3a8dd8ce57192deda8916cc444c87d7ab1a36515
2020-10-28 11:01:33 +01:00
Aaron Schulz
70c4255978 filerepo: clean up shared cache keys to avoid key metrics clutter
This avoids statsd metrics with names that begin with DB domains

Bug: T261534
Change-Id: I69fa4482c639bd49bdc2dec48a8dcdd74d455432
2020-10-21 13:29:15 -07:00
DannyS712
8dc5e1413d Remove fallback to $wgUser in LocalRepo::findFiles
Use RequestContext::getMain()->getUser(), per task

Bug: T263033
Change-Id: I78a2af031c5157d58f369c58c9f98376ba44a921
2020-09-25 17:44:51 +00:00
C. Scott Ananian
5e703cdf66 Hard deprecate File::userCan() with $user=null
The ArchivedFile::userCan and OldLocalFile::userCan() methods, along
with a number of other methods where the user parameter was optional,
were deprecated in 1.35, but this case was overlooked. This patch is
intended for backport to 1.35, so that the $user parameter can be
removed in 1.36 in accordance with the deprecation policy.

This path is known to be used by LocalRepo::findFile(),
FileRepo::findFile(), and FileRepo::findFileFromKey(), so hacky
workarounds have been added in this patch to avoid triggering
deprecation warnings in 1.35. T263033 has been filed to fix these
'correctly' in 1.36.

Bug: T263014
Change-Id: I17cab8ce043a5965aeab089392068b91c686025e
2020-09-16 12:27:51 -04:00
Ed Sanders
7683f7d839 Use strict (in)equality with namespaces constants when LHS is definitely an integer
Change-Id: I8fede00dfe1270d93c5d78d3c36e788cddfc8a99
2020-07-31 18:03:28 +01:00
Tim Starling
47a1619027 Remove terminating line breaks from debug messages
A terminating line break has not been required in wfDebug() since 2014,
however no migration was done. Some of these line breaks found their way
into LoggerInterface::debug() calls, where they mess up the formatting
of the debug log.

So, remove terminating line breaks from wfDebug() and
LoggerInterface::debug() calls.

Also:
* Fix the stripping of leading line breaks from the log header emitted
  by Setup.php. This feature, accidentally broken in 2014, allows
  requests to be distinguished in the log file.
* Avoid using the global variable $self.
* Move the logging of the client IP back to Setup.php. It was moved to
  WebRequest in the hopes that it would not always be needed, however
  $wgRequest->getIP() is now called unconditionally a few lines up in
  Setup.php. This means that it is put in its proper place after the
  "start request" message.
* Wrap the log header code in a closure so that variables like $name do
  not leak into global scope.
* In Linker.php, remove a few instances of an unnecessary second
  parameter to wfDebug().

Change-Id: I96651d3044a95b9d210b51cb8368edc76bebbb9e
2020-06-03 12:01:16 +10:00
Aaron Schulz
88e17d3f7c filerepo: make LocalRepo::getSharedCacheKey() use makeGlobalKey()
Also rename image_redirect key to file_redirect while at it.
This assures that stale keys are not still in use.

Bug: T253405
Change-Id: I31a9bb6672b33fbfa1b974955d78fdfd8d58f5da
2020-05-23 17:32:55 +00:00
Reedy
af063dd794 Fix more Squiz.Scope.MethodScope.Missing
Change-Id: I44cd7ba39a898a27f0f66cf34238ab95370d2279
2020-05-18 21:02:14 +00:00
James D. Forrester
4f2d1efdda Coding style: Auto-fix MediaWiki.Classes.UnsortedUseStatements.UnsortedUse
Change-Id: I94a0ae83c65e8ee419bbd1ae1e86ab21ed4d8210
2020-01-10 09:32:25 -08:00
zoranzoki21
dcbbe6e68d Remove deprecated function getSlaveDB() from includes/filerepo/LocalRepo.php
Change-Id: I65562cbe8ab5b58fcb683994d7addab615e2ac0a
2019-11-30 16:38:25 +00:00
Daimona Eaytoy
69cadf44a9 Unsuppress PhanParamsTooMany
This is the last repo-wide suppressed issue. Hurrah.

Bug: T231636
Change-Id: I3dc939f115bea14848c9c40bc52a7601f86ca0a7
2019-10-10 12:44:08 -07:00
Aaron Schulz
2c87b59f35 filerepo: Use @method LocalRepo::newFile() to document the return type
This avoids PhpStorm IDE warnings

Change-Id: I521aa1b1243ceabd40e7da18bf73ab95af7039cc
2019-09-04 20:12:23 +02:00
Marko Obrovac
f65beabcfd LocalRepo: Remove leading 0 from 32-byte SHA1 keys
Bug: T230667
Change-Id: I3d9de7a92495e894b4b44b0a5b0646b6d720f7c2
2019-09-03 16:04:14 +02:00
Aryeh Gregor
9b90a1fde7 Add some LocalRepo integration tests
Coverage is still far from 100%, but better than nothing.

Change-Id: Ifb22491414653a5730a83434c4ea31a3540f7d26
2019-08-30 12:36:46 +03:00
Thiemo Kreuz
734a969d55 Safe replacement of a lot of !count() with === []
This was originally a global search and replace. I manually checked all
replacements and reverted them if (due to the lack of type hints) either
null (that would be 0 when counted) or a Countable object can end in the
variable or property in question.

Now this patch only touches places where I'm sure nothing can break.

For the sanity of the honorable reviewers this patch is exclusively touching
negated counts. You should not find a single `!== []` in this patch, that
would be a mistake.

Change-Id: I5eafd4d8fccdb53a668be8e6f25a566f9c3a0a95
2019-01-15 17:28:49 +01:00
Aaron Schulz
8baffeae4f filerepo: Inject the WAN cache info FileRepo instances
Also fixed some intellij IDEA warnings.

Change-Id: I667ed7ef9b49c1da1c8c7c9914daadb7c74b3c79
2018-10-25 20:57:40 +00:00
Umherirrender
40d3ce8d17 Reduce calls to MediaWikiServices::getInstance()
In some functions MediaWikiServices::getInstance() was called twices or
in loops. Extract the variable to reduce calls.

Change-Id: I2705db11d7a9ea73efb9b5a5c40747ab0b3ea36f
2018-08-18 06:02:39 +02:00
Aryeh Gregor
90d4f56fe4 Mass conversion of $wgContLang to service
Brought to you by vim macros.

Bug: T200246
Change-Id: I79e919f4553e3bd3eb714073fed7a43051b4fb2a
2018-08-11 22:44:29 -06:00
Max Semenik
8085c58a4b Replace call_user_func_array(), part 1
Uses new PHP 5.6 syntax like ...parameter unpacking and
calling anything looking like a callback to make the code more readable.
There are much more occurrences but this commit is intentionally limited
to an easily reviewable size.

Change-Id: Idcec077ef3fdf029b632cceafd0150851ad723e3
2018-06-04 23:39:04 -07:00
WMDE-Fisch
2330c9afad Replace deprecated ObjectCache and CommentStore calls in filerepo
Change-Id: I0e863dc8e4bc6dac938a2ed437c2691aace19690
2018-05-07 11:00:23 +02:00
Thiemo Kreuz
645d96b0e1 Fix and improve PHPDoc type hints in FileBackend and FileRepo
Change-Id: I311b4a6777946b451d70c6fdb2133dbbd73f159f
2018-04-19 10:32:09 +02:00
Umherirrender
3124a990a2 Use ::class to resolve class names in includes files
This helps to find renamed or misspelled classes earlier.
Phan will check the class names

Change-Id: I07a925c2a9404b0865e8a8703864ded9d14aa769
2018-01-27 20:34:29 +01:00
Brad Jorsch
3488f49532 Replace selectFields() methods with getQueryInfo()
Several classes have a "selectFields()" static method to tell callers
which fields to select from the database. With the recent comment table
change and the upcoming actor table change, this pattern has become too
simplistic as a SELECT will need to join several tables to be able to
retrieve all the needed fields.

Thus, we deprecate the selectFields() methods in favor of getQueryInfo()
methods that return tables and join conditions in addition to the
fields.

Change-Id: Idcfd15568489d9f03a7ba4460e96610d33bc4089
2017-10-30 22:57:33 +00:00
Matěj Suchánek
c2720d5686 Drop compatibility with PHP 5.3
Since 5.4, $this works inside closures.

Change-Id: I6987f76a1e33d6d7ab020a57186e5477186bbf2e
2017-09-02 20:39:13 +02:00
Umherirrender
be42e09aa8 build: Prepare for mediawiki/mediawiki-codesniffer to 0.9.0
The used phpcs has a bug, so the version 0.9.0 could not be enforced at the moment.
Will be fixed in next version, see T167168

Changed:
- Remove duplicate newline at end of file
- Add space between function and ( for closures
- and -> &&, or -> ||

Change-Id: I4172fb08861729bccd55aecbd07e029e2638d311
2017-06-26 17:14:31 +00:00
Aaron Schulz
d735dc562d Move Database and subclasses to Rdbms namespace
Change-Id: I52bef87512f9ddd155d1f4cc0052f6b7a0db5b42
2017-04-12 10:43:57 -07:00
Aaron Schulz
488a647831 Move IDatabase/IMaintainableDatabase to Rdbms namespace
Change-Id: If7e8a8ff574661fd827de8bcec11d2c39a687300
2017-03-28 15:32:38 -07:00
Aaron Schulz
e01fd44388 Move ResultWrapper subclasses to Rdbms
Change-Id: I6f3f0e85e268b24c57c537aa6ad8016e0b4cdddb
2017-03-03 00:44:41 +00:00
Aaron Schulz
f525c72590 Rename getSlaveDB() FileRepo method to getReplicaDB()
The old name is left as an alias.

Change-Id: I60ab2cd5ce05df4247d5e25b017d2debee56554e
2016-11-18 07:42:39 -08:00
Kevin Israel
6ec4c12f13 LocalRepo::checkRedirect(): Fix use of __METHOD__ in closure
Change-Id: Ic8b4d2d54f1986fe4b52bc595221c9384389379b
2016-11-05 21:07:16 +00:00
Aaron Schulz
8585cc9ffb Fix numerous FileRepo IDEA warnings
Change-Id: I3522f37b675efffb68dec9d125faacd8c8776e64
2016-10-05 16:38:15 -07:00
Aaron Schulz
30f4b3c103 Replace DatabaseBase => Database in more places
Change-Id: If37a7909056bf2c31a8228cbc84f0fbbf5f1c517
2016-09-28 15:53:02 -07:00
Aaron Schulz
a2790b1b80 Remove wfGetCaller() dependency from DatabaseBase
Change-Id: I3e240b2eb5c1f6a21f1bc974c3d28f5755c7451a
2016-09-15 14:40:00 -07:00
Aaron Schulz
950cf6016c Rename DB_SLAVE constant to DB_REPLICA
This is more consistent with LoadBalancer, modern, and inclusive
of master/master mysql, NDB cluster, and MariaDB galera cluster.

The old constant is an alias now.

Change-Id: I0b37299ecb439cc446ffbe8c341365d1eef45849
2016-09-05 22:55:53 -07:00
Aaron Schulz
16266edff3 Change "slave" => "replica DB" in /includes
Change-Id: Icb716219c9335ff8fa447b1733d04b71d9712bf9
2016-09-05 21:01:01 +00:00
Aaron Schulz
ea1f4e8a53 Add TTL_PROC_* constants for clarity
This makes the reason for the cache times more obvious

Change-Id: Ie75df6be77c513feeb6cff3417ccbc124a6a1141
2016-05-06 23:45:41 +00:00
Aaron Schulz
e2427d3487 Lower pcTTL in checkRedirect() to 30
The process cache does not see other purges, so lower this
for sanity in case of long-running scripts.

Change-Id: I545ce6b160bb10ffe1877284a397a247c960d885
2016-03-26 10:43:21 +00:00
Ori Livneh
d1c4eafef5 Request-local caching of image_redirect
LocalRepo::checkRedirect() cache image redirect checks in memory to avoid
unnecessary trips to memcached. Same idea as Idbd11637.

Change-Id: Idaa12376382aaa0bd8cb988240ec1da0b688359d
2016-03-25 12:09:10 -07:00
Aaron Schulz
dcb5ec5cbf Make UploadBase use TempFSFile to wrap the temporary file
* The cleanupTempFile() method now only marks it as ready
  for unlinking when unreferenced.
* Pass TempFSFile down the FileRepo call chain so that it
  can build off a5d903860a and allow more fast async writes
  for secondary backends. Previously, the 'store' operation
  used on upload forced sync file change replication writes.
* Also fix bogus method call in LocalFile::publishTo().

Bug: T91869
Change-Id: I06caa6e5d8bdec9a7cae2b68cb02dd2e64b9ac74
2016-03-07 20:54:06 +00:00