Commit graph

45 commits

Author SHA1 Message Date
Daimona Eaytoy
598c4d7fcb build: Upgrade phan to 0.9.0
Scalar casts are still allowed (for now), because there's a huge amount
of false positives. Ditto for invalid array offsets.

Thoughts about the rest: luckily, many false positives with array offsets
have gone. Moreover, since *Internal issues are suppressed in the base
config, we can remove inline suppressions.

Unfortunately, there are a couple of new issues about array additions
with only false positives, because apparently they don't take
branches into account.

Change-Id: I5a3913c6e762f77bfdae55051a395fae95d1f841
2019-12-07 20:16:19 +00:00
Aaron Schulz
5b3ce5dc8c filebackend: reduce hashing I/O in SwiftFileBackend::doStoreInternal()
Compute both hashes with one read of the local source file

Change-Id: I9679b50d99a46747a6606ca059ef0005ee1aaaa6
2019-09-15 21:30:50 -07:00
jenkins-bot
e80ff96c37 Merge "filebackend: rename and simplify header sanitizing methods in SwiftFileBackend" 2019-09-14 00:12:05 +00:00
Derick Alangi
bf25b795ea filebackend: Use HTTPFileStreamer::send404Message() in caller instead
Replaced usage of StreamFile::STEAM_HEADLESS (now making it unused) by
HTTPFileStreamer::STREAM_HEADLESS. Also, deprecated these unused constants.
Searched around and found nothing and CS also says so too;

https://codesearch.wmflabs.org/search/?q=StreamFile%3A%3A(STREAM_ALLOW_OB%7CSTREAM_HEADLESS)&i=nope&files=&repos=

Change-Id: Ia3fc2cf072fa2fddb910c64a547d12cb243290d5
2019-09-12 14:09:39 +01:00
Aaron Schulz
8d1e7c00c6 filebackend: rename and simplify header sanitizing methods in SwiftFileBackend
In addition:
* Remove restriction that Content-Type was ignored in 'copy' and 'move'.
* Fix failing tests due to copy/move not respecting 'ignoreMissingSource'
  when using doQuickOperations().

Change-Id: Id1befeefeaaf75f0a6bf0e68b1e62271d432f953
2019-09-08 12:42:25 -07:00
Daimona Eaytoy
b3e4a6f61f Add docblocks to a bunch of methods
Copied from I014ea7f048862cc8cc387a68bf0f1cc5acbd9c6e.

Change-Id: Ic3216386af173cfb9cbffda0826879829e52ffbb
2019-09-08 18:33:31 +00:00
Aaron Schulz
e390198c4e filebackend: use self:: instead of FileBackend:: for some constant uses
Change-Id: Iae82a074ef9da3c36b34713d31ec65407d2a90b7
2019-09-08 14:15:19 +00:00
Aaron Schulz
903f1810c8 filebackend: improve internal use of FileBackend constants
Add more result constants and split up FileBackend::UNKNOWN for
clarity. This follows up 5719815f3b, which added that constant.

Make internal FileBackendStore::doGet* methods distinguish I/O errors
from missing files; the return types of public FileBackend methods are
unchanged. Avoid process caching any mtime/size/sha1 values in the
case of I/O errors. Use error constants consistently for stat methods
when given invalid paths.

Also:
* Factor out FileBackendStore::processCacheAndPersistStatEntries() method
  to reduce significant code duplication.
* Consolidate duplicated isPathUsable() checks in FileOp subclasses to
  FileOp::precheck().
* Remove null process cache value check from FileBackend::getFileStat()
  as null values are never stored in the process cache to begin with.
* Reformat some oddly wrapped lines to look cleaner.

Change-Id: Id0e4b0da0bb2ed3184847b35142d587c7f3d953d
2019-09-03 19:31:17 -07:00
Daimona Eaytoy
e70b5b3309 Unsuppress other phan issues (part 4)
Bug: T231636
Depends-On: I58e67c2b38389df874438deada4239510d21654f
Change-Id: I6e5fba7bd273219b1206559420b5bdb78734aa84
2019-08-31 17:13:39 +00:00
Daimona Eaytoy
7f7efbe026 Globally unsuppress phan issues with low count
All of these suppression prevent the detection of many common mistakes,
and could easily prevent things like T231488. Especially if there are
few issues of a given type, it's way better to suppress them inline,
instead of disabling them for the whole core.
This patch only touches the one with a lower count (although those
counts may be out of date).

Bug: T231636
Change-Id: Ica50297ec7c71a81ba2204f9763499da925067bd
2019-08-30 09:40:47 +00:00
Aaron Schulz
5719815f3b filebackend: add idiom constant to FileBackend for null results
Change-Id: I65f043e87d82192c13d3627fb45ef222f0290bf8
2019-08-29 01:19:33 -07:00
Aaron Schulz
d01d98012b filebackend: use AtEase in FileBackend related classes
Change-Id: I819e0e923dfd3ffa7299acd1d6103e3cdf2baa93
2019-08-23 13:46:30 +00:00
Aryeh Gregor
a83b33582a TempFSFileFactory service
This replaces TempFSFile::factory(), which is now deprecated.

Change-Id: I9e65c3867e26c16687560dccc7d9f3e195a8bdd6
2019-08-21 16:26:05 +03:00
Aaron Schulz
bcfba52cd1 filebackend: avoid bogus SwiftFileBackend log warnings due to HTTP 202 responses
Change-Id: I12897bd2e29f18bbdce9a54a18415e83acc31365
2019-06-28 13:28:49 -07:00
Umherirrender
4b489fa2a7 Fix type hint for properties holding MapCacheLRU
Change-Id: Ia22b1dbcde3b96b3f437a213da95b3556fa42d70
2019-06-05 20:32:16 +02:00
James D. Forrester
460bcf81e7 build: Upgrade mediawiki/mediawiki-phan-config from 0.5.0 to 0.6.0 and make pass
Fix five instances of PhanPluginDuplicateConditionalNullCoalescing;
escape the rest for now.

Bug: T219114
Change-Id: Ic4bb30c43c5315ce6b878b37b432c6e219414f8b
2019-05-13 14:57:07 +00:00
Reedy
7f04a47d68 Split filebackend files to class per file
Change-Id: Idf9f4177fb9a523ce41227bda2af923bf418396b
2019-04-14 23:02:13 +01:00
Aaron Schulz
e1497e3593 filebackend: avoiding computing file SHA-1 hashes unless needed
FileBackendStore already supports stat info not returning SHA-1.
Build on that logic with a "requireSHA1" parameter to getFileStat()
to move some logic from SwiftFileBackend to the parent class and
avoid computing missing SHA-1's for Swift when nothing actually
requested the SHA-1. Only getFileSha1Base36() needs to trigger this
lazy-population logic.

Note that thumbnails only use doQuickOperations(), which does not
need to examine SHA-1s, it only does regular getFileStat() calls.

Also renamed addMissingMetadata() to addMissingHashMetadata().

Bug: T204174
Change-Id: I2a378cb2a34608a6da2f8abe604861ff391ffaa7
2018-12-10 22:51:26 +00:00
Aaron Schulz
ec7dfb4a34 filebackend: switch callers from ProcessCacheLRU to MapCacheLRU
Change-Id: I6efe7122ff30053a85461d31be7b5ddad0a5d2af
2018-07-14 11:27:09 +00:00
Umherirrender
130ec2523d Fix PhanTypeMismatchDeclaredParam
Auto fix MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam sniff

Change-Id: I865323fd0295aabd06f3e3c75e0e5043fb31069e
2018-07-07 00:34:30 +00:00
Max Semenik
1e680456b4 Get rid of call_user_func(_array)(), part 3
Also cleaned up nearby code in a couple places.

Change-Id: Ibf44ee7c0ceb739d7e79406e4ff39303c316e285
2018-06-10 02:21:24 +00:00
Bartosz Dziewoński
485f66f174 Use PHP 7 '??' operator instead of '?:' with 'isset()' where convenient
Find: /isset\(\s*([^()]+?)\s*\)\s*\?\s*\1\s*:\s*/
Replace with: '\1 ?? '

(Everywhere except includes/PHPVersionCheck.php)
(Then, manually fix some line length and indentation issues)

Then manually reviewed the replacements for cases where confusing
operator precedence would result in incorrect results
(fixing those in I478db046a1cc162c6767003ce45c9b56270f3372).

Change-Id: I33b421c8cb11cdd4ce896488c9ff5313f03a38cf
2018-05-30 18:06:13 -07:00
Timo Tijhof
a89095bd52 filebackend: Fix undefined 'req_params' context in FileOperation log
Follows-up 9d49075b38. Noticed various log messages on Beta Cluster
from this code path that used "{params}" even in the real messages
(not normalised). Turned out, it isn't being defined due to a typo.

Change-Id: Ib9cf644fdf413035504d9d7486696cffa7737d4a
2018-05-20 01:39:36 +02:00
Gilles Dubuc
cd233f1165 Separate additional swift users for public and private
For better security seggregation it can be desirable
to set different additional swift users when dealing
with private containers.

Bug: T187822
Change-Id: I66d26ec81b3a3577d274c0d28cf86db7b505e082
2018-02-26 10:25:59 +01:00
Reedy
39f0f919c5 Update suppressWarning()/restoreWarning() calls
Bug: T182273
Change-Id: I9e1b628fe5949ca54258424c2e45b2fb6d491d0f
2018-02-10 08:50:12 +00:00
Chad Horohoe
5c93fabafd Drop XCache support
It was never super popular anyway, APC was always the best option.
The project has no plans to move to PHP7, so it's quickly reaching
its end of life. Oh, and Fedora dropped it from their repos 2 years
ago.

Change-Id: Ia3257e86a6323d8943f04a5c05c72c0bd4c4b0a9
2018-02-07 13:45:40 -08:00
Aaron Schulz
795412ccaf Clean up formatting of SwiftFileBackend::sanitizeHdrsStrict()
Change-Id: I5e411f4bf50836d22db01220853776494a9e69d5
2017-11-28 09:07:37 -08:00
Cormac Parle
d36d987c4e Do not strip Content-Type header for POST requests to swift
libcurl adds 'Content-Type: application/x-www-form-urlencoded'
to a POST request if the 'Content-Type' header is not set
manually. Because data in swift is updated via POST, the
Content-Type header must be set explicitly to stop a run of
refreshFileHeaders.php from changing the Content-Type of all
files in swift to application/x-www-form-urlencoded

Bug: T178849
Change-Id: I43c21bc1b73e37104cf07cd5f1c1557f472b9898
2017-11-28 14:24:27 +00:00
Aaron Schulz
205f4606af SwiftFileBackend::resolveContainerPath() check the proper length
The length sanity check should use this instead of urlencode()
see it is rawurlencode() that is actually used.

Change-Id: I5632e30c14c8ab27c8324c3e31311ca8bff7c162
2017-11-17 23:37:04 +00:00
Chad Horohoe
9d49075b38 Fix grouping for remaining logger calls in SwiftFileBackend
Change-Id: I80ad8424de3e2a6a0c4b5e69537012a8f775cd68
2017-11-03 22:22:33 +00:00
Chad Horohoe
4f2d55babb Group Swift metadata errors by making $path a parameter
Change-Id: I3beb3743e1d13b06a9110c44f84fdc8d1b193137
2017-10-26 10:33:42 -07:00
Gilles Dubuc
75a3043b71 Give thumbor swift user r/w access to containers
readUsers and writeUsers are new optional values
of the swift backend configuration. They allow
giving read and/or write rights to additional
users than the default swift user.

Bug: T144479
Change-Id: I0f81a013ec994eee3f156a89f29f4fcfc37c42b7
2017-09-25 21:45:39 +00:00
Kunal Mehta
0cb2a09b6d Always put type information before variable name for @var tags
Like other tags, @var should have the type information before the
variable name.

<https://docs.phpdoc.org/references/phpdoc/tags/var.html>

Change-Id: I9eca6957b1990fa8cc687103dc02ee38af5f9086
2017-09-04 11:06:51 -07:00
Umherirrender
718e63694d Add missing @param and @return documentation
Change-Id: I1d1098eec3933df6561cceef646576013ddc08c8
2017-08-11 22:17:01 +02:00
Umherirrender
a9007e8baf Add missing & to @param documentation to match functon call
Change-Id: I81e68310abcbc59964b22e0e74842d509f6b1fb9
2017-08-11 18:47:46 +02:00
Aaron Schulz
82e2c924e4 Remove "@author Aaron Schulz" annotations
Bug: T139301
Change-Id: Ib5248e8e27d60611c7373bce4b29dd5e85aa3489
2017-06-27 15:24:14 -07:00
Aaron Schulz
8eb82c0f64 Handle proxy-based TLS when placed in front of Swift
Force the URL for storage operations to use https if the authentication
URL is already forced to use https. This avoids following an http URL
based on the response of a Swift proxy unaware that a terminator placed
in front of it was used for the authentication.

Bug: T160616
Change-Id: Ia6c8c99ebb38d5828773b2f7aa8cf14ced6bdfde
2017-05-10 17:01:13 +00:00
Aaron Schulz
8d4b97f364 Use executeOpHandlesInternal() in SwiftFileBackend
Instead calling doExecuteOpHandlesInternal() directly, use the
wrapper parent class method instead so that closeResources() is
called on all of the handles.

Change-Id: Iab0d4f7c08d155639a44f48147b78283059a6d24
2017-03-20 18:04:40 +00:00
Aaron Schulz
17c686a802 Make sure Swift store operations close the source file handle
Bug: T159607
Change-Id: I9a8907a20ba83f33cc5fc96bc15a1728770f656a
2017-03-14 20:08:46 +00:00
James D. Forrester
b715f432f4 includes/libs: Replace implicit Bugzilla bug numbers with Phab ones
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.

Skipping jsminplus.php as those bug numbers aren't Wikimedia's, nor obviously
someone else's.

Change-Id: I9a2210e17852ee56f11282b980ac66d8c7a95671
2017-02-21 18:13:08 +00:00
Erik Bernhardson
d67197fa11 Cleanup some incorrect return annotations
Most of these are simply changing annotations to reflect
reality. If a function can return false to indicate failure
the @return should indicate it.

Some are fixing preg_match calls, preg match returns 1, 0 or false,
but the functions all claim to return booleans.

This is far from all the incorrect return types in mediawiki, there
are around 250 detected by phan, but have to start somewhere.

Change-Id: I1bbdfee6190747bde460f8a7084212ccafe169ef
2016-12-12 10:15:05 -08:00
jenkins-bot
69ae945e8d Merge "Update weblinks in comments from HTTP to HTTPS" 2016-11-08 21:32:00 +00:00
Fomafix
202f695f67 Update weblinks in comments from HTTP to HTTPS
Use HTTPS instead of HTTP where the HTTP link is a redirect to the HTTPS link.

Also update some defect links.

Change-Id: Ic3a5eac910d098ed5c2a21e9f47c9b6ee06b2643
2016-11-07 15:24:46 +01:00
umherirrender
34fe90ac52 Remove empty lines at end of functions
It looks like there is something missing after the last statement
Also remove some other empty lines at begin of functions, ifs or loops
while at these files

Change-Id: Ib00b5cfd31ca4dcd0c32ce33754d3c80bae70641
2016-11-05 11:55:10 +01:00
Aaron Schulz
51f00e3db8 Move SwiftFileBackend to /libs
Change-Id: Ied0be9891d70930caf415d75cb3b89043347f47e
2016-09-24 15:03:56 -07:00
Renamed from includes/filebackend/SwiftFileBackend.php (Browse further)