Commit graph

102 commits

Author SHA1 Message Date
James D. Forrester
960e4d09ca Documentation: Replace misuse of 'later' when we meant 'latter'
Change-Id: I01bfdb8e0bfaa132533732873f7c840cea7935eb
2016-09-07 13:13:42 -07:00
Aaron Schulz
1ba6d52141 Use WAN cache adaptiveTTL() method in FileBackendStore
Change-Id: I2e8b4e85b12c8892e8640abc63ee13b4b170645d
2016-09-02 15:59:52 -07:00
Aaron Schulz
d42a05475a Added Range support to FileBackend::streamFile()
* Added HTTP options headers parameter to streamFile().
* Refactored doStreamFile() to either call StreamFile::stream()
  or delagate that to the subclass. SwiftFileBackend now relays
  the full Swift response rather than manually making the headers.
  This also makes Range headers easy to support.
* Made use of this in img_auth.php for performance on private wikis.
* Elimate stat call in streamFile() for Swift if "headers" is empty.
* Refactored StreamFile a bit to inject request headers instead
  of using the $_SERVER global. A header options parameter is used
  instead, which also supports Range.
* Removed now unused prepareForStream().
* Cleaned up streamFile() unit tests.

Change-Id: I2ccbcbca6caabb8cf65bd6b3084cede2e6ea628a
2016-06-04 05:43:43 -07:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
Reedy
00c426e3c2 Replace wfBaseConvert with Wikimedia\base_convert
Change-Id: Iadab3d018c3559daf79be90edb23d131729bdb68
2015-11-24 22:51:42 +00:00
jenkins-bot
b85aeec5e2 Merge "Improve MIME detection in FileBackend" 2015-11-03 11:10:41 +00:00
Aaron Schulz
de290cd02d Improve MIME detection in FileBackend
The content type detector will now inspect the file contents
to better handle extensionless files.

Also dependency inject the callback and make the default one
use FileInfo.

Change-Id: Iad59bf6c6a416b706f976a4c425763fd30e2debb
2015-11-02 00:36:54 +00:00
Aaron Schulz
642ca3862b Normalize header case for FileBackend operations
Normalize all headers to lower case at the start of the
FileBackend operation methods. This makes it easy for
subclasses to check for certain headers, e.g. content-type.

Change-Id: Ia69976326d17a51bcaa61f2781aa669ae7bd9c28
2015-10-27 12:34:35 -07:00
Aaron Schulz
5262d41a6c Minor fix to primeFileCache() comment
Change-Id: I7d040c5a914574df72fa31612df0eb28e15fd46c
2015-10-23 17:18:56 -07:00
Aaron Schulz
fb0994d5c5 Prevent IDEA error in doOperationsInternal()
Change-Id: I2f0a1885803fff9dc0eb8ca4a595a83736ce421b
2015-09-20 23:04:29 -07:00
Aaron Schulz
5cea7769d3 Cleaned up getScopedFileLocks() return value
* Just return the ScopedLock itself without wrappig it in an array.
  This also makes getScopedLocksForOps() actually return the correct
  type according to the docs

Change-Id: Ic2d01d56c3fcb22af3adb5e5d4d1586d62316552
2015-09-18 17:58:11 +00:00
Aaron Schulz
8d1eecab49 Allow "." in filebackend container prefixes
* This character in $wgDBname seems to be common enough.
  The wiki ID is the default container prefix, and if it
  is not accepted, then confusing errors occur that block
  all file uploads. The case of dots is handled now.
* Improved variable naming in resolveStoragePath().

Bug: T46066
Change-Id: I68458a876855894d01e8cf880bd6cfcfdae17bd0
2015-06-26 06:49:20 +00:00
Kunal Mehta
f6e5079a69 Use mediawiki/at-ease library for suppressing warnings
wfSuppressWarnings() and wfRestoreWarnings() were split out into a
separate library. All usages in core were replaced with the new
functions, and the wf* global functions are marked as deprecated.

Additionally, some uses of @ were replaced due to composer's autoloader
being loaded even earlier.

Ie1234f8c12693408de9b94bf6f84480a90bd4f8e adds the library to
mediawiki/vendor.

Bug: T100923
Change-Id: I5c35079a0a656180852be0ae6b1262d40f6534c4
2015-06-11 18:49:29 +00:00
Aaron Schulz
48cffdff82 Converted FileBackendStore to using the WAN cache
Bug: T93141
Change-Id: I76efb0dc45a697c5fdfc50932e1198a6d663a445
2015-04-30 01:30:38 +00:00
jenkins-bot
b7ec2c9b33 Merge "Fix and make PHPDoc tags in FileBackend more specific" 2015-03-18 16:45:20 +00:00
Thiemo Mättig
d71f05290e Fix and make PHPDoc tags in FileBackend more specific
A pure documentation patch. Should almost be a no-brainer. ;-)

Change-Id: I9bf4d6056f7cde193fe5882a04a7973af9b5a7f4
2015-03-18 17:22:52 +01:00
Aaron Schulz
8d2f7b9178 Never treat persistent stat cache as "latest" in FileBackend
bug: T89184
Change-Id: I035c0f85a641efa0e4e3209b2bcf9847c08ab0d2
2015-02-10 23:30:55 +00:00
Aaron Schulz
4ff8136807 Removed remaining profile calls
Change-Id: I31c81c78715048004fc8fca0f27d09c1fa71c118
2015-01-08 02:49:33 -08:00
Aaron Schulz
3226337f17 Converted some section profiling to scopedProfileIn
Change-Id: If69de34a046f4f33fc57b93beaf264f522ce7386
2015-01-07 14:19:06 -08:00
Ricordisamoa
12dec5d85d Fix some stuttering in comments and documentation
Change-Id: I9c0088b9aab37335203cad45a1d6fa8ac3f43321
2014-12-17 19:44:10 +00:00
umherirrender
b883e8c7c0 Cleanup some docs (includes/[e-l])
- Swap "$variable type" to "type $variable"
- Added missing types
- Fixed spacing inside docs
- Makes beginning of @param/@return/@var/@throws in capital
- Changed some types to match the more common spelling

Change-Id: I41a84e8e1dec39170aa655250325ffc485eaeaef
2014-07-24 19:43:03 +02:00
umherirrender
abfa0d7854 Standardize @todo
Always use lowercase @todo
Changed some "@FIXME" to "@todo FIXME: "

Change-Id: I016d1f9511e6f3bbd22427f0fffeccd77ab5e0db
2014-07-23 22:27:20 +02:00
umherirrender
53c420e278 Fixed spacing
- use tab as indent instead of spaces
- Added space after closures "function"
- Added spaces around string_concat
- Added newline inside empty blocks
- Removed four spaces after comma

Change-Id: I4425b0c6a69b36f40acfea6511b8950cf09ce2b2
2014-07-20 21:41:41 +02:00
Aaron Schulz
0fa9a391e3 filebackend: Avoid using popen() when "parallelize" is disabled
Bug: 66467
Change-Id: Iaf8eb2ecfad166e26e265a98933399dc4bc255a3
2014-06-18 20:12:03 +00:00
Aaron Schulz
1d59d6152e Improved FileBackend failure logging a bit
* This partly follows up on 2cfce9f

Change-Id: I9d15f44d8000f52fd7b307d358c895452ee96a8d
2014-05-22 17:37:33 -07:00
jenkins-bot
a21d59cc6e Merge "Make room for preloadFileStat() call in FileBackend::doOperationsInternal" 2014-04-21 17:13:35 +00:00
jenkins-bot
e40b19ed4e Merge "Bail out on FileBackend operations if the initial stat calls failed" 2014-04-21 17:13:33 +00:00
umherirrender
957adbef22 Fixed some @params documentation (includes/[file...|upload])
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Also added some missing @param.

Change-Id: I8804ebe0922d3a414863b162a2110e0b9e49b80f
2014-04-19 17:19:17 +02:00
umherirrender
b9cd789fce docs: closure -> Closure; callback -> callable
Changed closure to capital word Closure in doc and type hint,
also changed callback in docs to callable

Change-Id: I52c8e8f13d38a837052101c38b9986be780ca057
2014-04-19 08:43:31 +02:00
Aaron Schulz
d52a559fba Make room for preloadFileStat() call in FileBackend::doOperationsInternal
Change-Id: I60f71155abcf7f69423639b10ec301ac192df728
2014-04-18 17:02:19 -07:00
Aaron Schulz
2cfce9fd8d Bail out on FileBackend operations if the initial stat calls failed
* The preloadFileStat() call acts as a good canary for whether all
  affected servers are up. If anything failed in it, then it's not
  worth bothering to send the actual write requests.
* Also made FileBackend::preloadFileStat abstract while at it since
  the two subclasses implemented it and so should any others.
* Fixed a silly comment typo.

Change-Id: I5bd427f654aa4a9d6bfe4ed7566276e8ac520b30
2014-04-18 12:41:23 -07:00
Aaron Schulz
02e0d115a1 Tweaked "latest" handling of filebackend stat entries
* Let "latest" stat entries override non "latest" so
  that future getFileStat() calls with the "latest" flag
  can actually have a cache hit.

Change-Id: I1e9391039537d608b89773b4d51575e3b364a751
2014-03-24 06:55:01 +00:00
Aaron Schulz
6112acac29 Make use of strong consistency with Ceph RGW
Change-Id: Ic3b764363e91c3279d3dc017ddce630dae26bec5
2014-03-19 17:27:48 +00:00
Aaron Schulz
f5f2233daa Avoid key fragmentation in FileBackend
* Make sure that global backends do not prefix the cache keys
  with local wiki IDs for container and file stat entries.
  This causes fragmentation and breaks invalidation.

Change-Id: Ic02d6e41f828dc82cc07c7d24c8af29c46392a9c
2014-02-24 13:22:15 -08:00
Aaron Schulz
6b1a60a1ca Added concurrent HEAD request support for Swift
* This adds a new preloadFileStat() method to FileBackend.
* FileBackend::doOperations() makes use of this instead of primeFileCache().
  The later was mostly useless due to the 'latest' flag, which stat entries
  rarely had (and since they use add(), 'latest' ones do not override the
  non-'latest' ones).
* Simplified primeContainerCache/primeFileCache to no longer accept
  FileOp objects and made sure nothing passes those in anymore.

Change-Id: Iee3878cadd69c3a594c701c0aff81aab5f473de5
2014-02-20 08:50:08 +00:00
Siebrand Mazeland
f077c4b1d6 Update formatting
Change-Id: I18aff576262479c9bb1c56eb8e1d1aaae200e4b1
2014-02-06 09:27:05 +01:00
Aaron Schulz
c745ff9a58 Fixed bogus var usage in SwiftFileBackend
Change-Id: Ia0dec914208cb3100b7f5bbd2291985eac05dc02
2014-01-09 12:39:13 -08:00
Aaron Schulz
ac14c1c8a6 filebackend: Added supported for retrieving file metadata/headers
* This can be useful for carrying over metadata when copying files around
* Also fixed a bug in sanitizeHdrs() for Swift (broken content-disposition)

Change-Id: I4534e9acac2b306086797b3677f85c05b98e39fc
2014-01-08 13:22:11 -08:00
Aaron Schulz
a0878fe2b4 filebackend: exception handling cleanups
* Split out FileBackendException class and reduced direct use of MWException

Change-Id: I325c1798b6d90972c12a5dccc37989af34d857f3
2013-12-07 23:22:07 -07:00
Siebrand Mazeland
8c0cbe6f83 Update ternary notation
Change-Id: I553dd1d42477326848e6e40b824058b6790e2eb5
2013-12-04 09:52:51 +01:00
Siebrand Mazeland
c232652303 Remove unused local variable
Just count if there are array members

Change-Id: I4f9cc8ba554404796822317fdb285e2f560d01dd
2013-11-25 06:48:23 +01:00
Siebrand Mazeland
34a4bcbd48 Update documentation of file backend classes
Change-Id: I2113aecc53f1b361d9c276113bc0a9f7076cd21c
2013-11-23 19:24:43 +01:00
Siebrand Mazeland
a5c35215e2 Update formatting of file backend classes
Change-Id: Ie69c324e2f1ea0b8a654adee06ebda494cebd10c
2013-11-22 22:17:15 +01:00
Aaron Schulz
beb7359243 Removed excess profile calls
Change-Id: Idd57e8d7fff56c62a66c7fcaf4627b9096f0c363
2013-11-16 23:44:41 -08:00
Aaron Schulz
2afdc066f5 filebackend: improved "adviseStat" performance
* Use the normal page size instead of limiting it way down to the cache
  size. Track the stat information in the pages and load into into the
  stat cache as entries are accessed. This should also be less prone to
  evictions causing HEAD requests (or memcached hits).
* Also bumped CACHE_CHEAP_SIZE up to 500.
* Fix a few doc bits

Change-Id: I8d44a072e7bcc56c83d8d9c8c9ac9864530bccf8
2013-11-08 04:35:57 +00:00
Aaron Schulz
584de11eb1 Fixed broken quickDescribe() method
Change-Id: Ic68f483c4008ff6ae8c4accef2f9d1f180a5d377
2013-10-24 16:34:07 -07:00
Aaron Schulz
ef2f2610ab filebackend: try to combine SH and EX lock acquisition
* getPathsToLockForOpsInternal() now returns an array in the format LockManager can use
* Also fixed getScopedLocksForOps() for multiwrite backend (it should accept proxy backend paths)
* Updated getScopedFileLocks() docs

Change-Id: Id0dc50c159c5fbc8fca10a9e46c5be23afcb3e9a
2013-10-18 22:57:38 +00:00
Aaron Schulz
91cab51cb5 Factored out a small filebackend getContentType() function
Change-Id: I221483ab4a0ed36ea34c19db1740dfaecf223677
2013-09-25 09:57:07 -07:00
Aaron Schulz
ad372af149 Switched FileBackendStore to use ProfileSection when possible.
* Also removed outer profiling layer.

Change-Id: Ice3b921fe0c9981ac16f005d66a78bf4f4a5c601
2013-06-19 00:43:54 +00:00
Aaron Schulz
7242d938f3 Cleaned up variable/type order in docs.
* A few other minor doc cleanups and obvious type hints.

Change-Id: I454fad0c1942068b4f3632ebd1b605bfbb9dfe7c
2013-06-13 11:18:52 -07:00