Commit graph

32 commits

Author SHA1 Message Date
Aaron
159cc27fde [FileBackend] A few code cleanups and some error message improvements.
Change-Id: I75f066104b98638ca956042c4e877c0f6327509c
2012-05-17 11:34:15 -07:00
Tim Starling
f03819f8eb Merge "[FileBackend] Added doQuickOperations() function for things like purging thumbnails." 2012-05-17 07:42:07 +00:00
Aaron
957f09b8a4 [FileBackend] Added doQuickOperations() function for things like purging thumbnails.
* doQuickOperations() lets us do things like purge thumbnails as fast as possible.
  Stat calls, SHA1 checks, and RTTs in general are avoided. It also avoids the
  slowness of lazy population of thumbnail SHA1s in Swift (for those made by rewrite.py).
* Removed supportedOperations() to avoid the extra complexity.
* Made a few variable type and exception documentation cleanups.
* Cleaned up unit test file removal a bit and made some functions private.

Change-Id: I6922368c6af7752a6927d96402519132203108a1
2012-05-16 10:30:34 -07:00
Aaron Schulz
6a460ed446 Renamed new BagOStuff getBatch() function to getMulti() for consistency.
Change-Id: Icfe673919f65192565aa114c165c90d97d33785a
2012-05-15 01:48:22 -07:00
Aaron
059823dae6 [FileBackend] Removed likely surperflous memcached retry code.
Change-Id: I934cd658d14c051f99f72d7e37e12f8646e59458
2012-05-14 18:26:56 -07:00
Aaron
23a0490b78 [FileBackend] Added support for concurrent file write operations.
* FS backends work via popen() and basic shell commands.
* Swift backends use the custom SwiftCloudFiles async features.
* Refactored storagePathsRead()/storagePathsChanged() to normalize the paths
  for correct dependency detection in FileOpBatch.
* Cleaned up SwiftFileBackend exception handling to make debugging easier.
* Added a quick and dirty performance testing script.
* Updated unit tests to include a run with parallelize=implicit.
* Improved file test failure output a bit.

Change-Id: I6a5ed743c30c598e0dd7301dbdb3631c460332fd
2012-05-09 05:37:41 -07:00
Alexandre Emsenhuber
d5381d0ea0 Added missing GPLv2 headers in some places.
Also made file/class documentation more consistent.

Change-Id: Iad15427368d13a6cc11c8512b7265e8c7b8878a5
2012-05-07 09:11:33 +02:00
Aaron
769a1cd261 Adding file stat caching to avoid RTTs to high latency backends.
Also tweaked a few of the container cache functions a bit.

Change-Id: Ic91b8e04aa509410578b3a4dc232871c0c5561d0
2012-04-27 10:49:50 -07:00
Brion VIBBER
c9480d2eee Merge "[FileBackend] Some documentation and minor cleanups." 2012-04-26 23:40:12 +00:00
Brion VIBBER
0efd8c5acc Merge "[FileBackend] Tweaked various cache parameters and bumped tiny Swift auth TTL a bit." 2012-04-26 23:32:32 +00:00
Brion VIBBER
e8eddd20de Merge "[FileBackend] Added 'recursive' flag to directory clean() function." 2012-04-26 23:30:05 +00:00
Brion VIBBER
4ef894d9e1 Merge "[FileBackend] Added container stat caching to reduce RTTs to high latency backends." 2012-04-26 22:51:54 +00:00
Aaron
104e8bb0d7 [FileBackend] Added 'recursive' flag to directory clean() function.
Change-Id: I8fadd25c05840e2324bf8433dc0a4846daf8e2c3
2012-04-26 12:41:33 -07:00
Aaron
d6d4fb1219 [FileBackend] Tweaked various cache parameters and bumped tiny Swift auth TTL a bit.
Change-Id: Ibb1a43624a94ff737e9f5c73a0bedec75aba37ff
2012-04-26 10:38:32 -07:00
Aaron
23e2fbcd2f [FileBackend] Added container stat caching to reduce RTTs to high latency backends.
Change-Id: I73575fd65ca06a238803b30f8de6873801a224b3
2012-04-25 18:07:11 -07:00
Aaron
a2b9285634 [FileBackend] Some documentation and minor cleanups.
Change-Id: Ia5f01679c928a9053cfd05d26fe1a03b67dac433
2012-04-25 15:53:22 -07:00
Brion VIBBER
f2e06b1f04 Merge "[FileBackend] Added some basic directory functions." 2012-04-25 19:38:12 +00:00
Demon
509403683f Merge "[FileBackend] Reduced code duplication with new getPathsToLockForOpsInternal() function." 2012-04-25 14:18:15 +00:00
Aaron
22a7665de2 [FileBackend] Reduced code duplication with new getPathsToLockForOpsInternal() function.
Change-Id: I0ca8a81f2e65acaf849e02af3d59fcc3d40026d8
2012-04-24 11:06:34 -07:00
Aaron Schulz
6419ce51ca [FileBackend] Renamed getOperations() to reflect that it is "internal".
Change-Id: I11e1b1b3ad494280f753d517792dcd1dff400aa6
2012-04-24 03:55:11 -07:00
Aaron Schulz
dd5326ccf1 [FileBackend] Added some basic directory functions.
* Added directoryExists() function to check if a dir exists.
* Added getDirectoryList() and getTopDirectoryList() functions to list out subdirs.
* Added getTopFileList() function to list out immediate files in a dir. This wraps getFileList(), which was modified.

Change-Id: I087835f40d76c41f5d6af55b9a34a51e0f44a675
2012-04-21 21:33:41 -07:00
Aaron
26ab914aa9 Fixed missing argument to pingExpensiveCache() call.
Change-Id: If3cf56e9cc804388cf3396e8ab15cae2f80de828
2012-04-11 12:02:20 -07:00
Aaron Schulz
01ce4e032c [FileBackend] Use LRU for stat/sha1 cache.
Change-Id: I0a61720357aacb42a362c6666ba7f806d6cc14cf
2012-04-06 13:16:23 -07:00
Aaron Schulz
215be530dd [FileBackend]
* Added FileJournal class to log file changes for file backends. This can be used for migrations (like moving to Swift), syncing mirror repos, consistency checks, finishing/reverting operation batches, and such. The default journal is the "null" journal, which simply does nothing.
* Added the optional schema change required for using the DBFileJournal (MySQL, SQLite).

Change-Id: I33c9f9a598ba1f164c862b9dc3c718f9172db02b
2012-03-27 15:27:14 +11:00
Aaron Schulz
a1b46f5dd4 Added per-backend profiling calls.
Change-Id: I380d06838649ab77f905cb6fed46b8068dbdcb9f
2012-03-21 16:54:22 -07:00
Roan Kattouw
0fca9a7991 Revert r107309, r113601, r113704, r113742, r113792, r113838, r113859, r113893, r113894, r113952, r114047, r114252, r114256, r114257. This reverts the remaining 'new' revisions in core.
All of these revisions are tagged with 'gerritmigration' and will be resubmitted into Gerrit after the Gerrit switchover. See also http://lists.wikimedia.org/pipermail/wikitech-l/2012-March/059124.html
2012-03-21 00:16:50 +00:00
Aaron Schulz
d19f54602f [FileBackend]
* Added FileJournal class to log file changes for file backends. This can be used for migrations (like moving to Swift), syncing mirror repos, consistency checks, finishing/reverting operation batches, and such. The default journal is the "null" journal, which simply does nothing.
* Added the optional schema change required for using the DBFileJournal (MySQL, SQLite).
2012-03-13 01:46:33 +00:00
Aaron Schulz
4fc819e92d r113416: added missing parameter to message 2012-03-09 16:01:11 +00:00
Aaron Schulz
bcdecf8fff Added backend-fail-maxsize failure message and use it instead of the vague "could not store" type message. 2012-03-08 22:51:43 +00:00
Aaron Schulz
3e3da50e8a r109469: ported r113208 by bumping max file size from 1GB to 4GiB. 2012-03-07 02:00:55 +00:00
Aaron Schulz
3ce1792d97 [FileBackend]
* Various documentation improvements.
* Moved a few protected FileBackendStoreShardListIterator functions down. Same with normalizeContainerPath().
2012-03-03 19:14:50 +00:00
Aaron Schulz
ea99d8290a Moved FileBackendStore and helper classes to their own file (no code changes). Updated AutoLoader. 2012-03-03 18:29:38 +00:00