Commit graph

66 commits

Author SHA1 Message Date
Sam Reed
462ab60ab6 Bad Aaron 2012-02-16 02:05:49 +00:00
Sam Reed
c052fc3b7b Documentation followups from lastnight 2012-02-10 15:37:33 +00:00
Aaron Schulz
571488a663 Cleaned up some docs after r111101 2012-02-10 00:49:10 +00:00
Sam Reed
76246b9bf5 More return documentation 2012-02-09 21:33:27 +00:00
Sam Reed
e1d83d5721 Fixing some of the "@return true" or "@return false", need to be "@return bool" and then the metadata can say true if foo, false if bar
Other documentation improvements
2012-02-09 18:01:54 +00:00
Aaron Schulz
8dd63c9580 r111036: some doc tweaks 2012-02-09 17:54:14 +00:00
Sam Reed
138ddc452f Fixing some of the "@return true" or "@return false", need to be "@return bool" and then the metadata can say true if foo, false if bar
Other documentation improvements
2012-02-09 17:42:35 +00:00
Antoine Musso
20dba09fac reword doc from r110938 2012-02-09 14:41:15 +00:00
Aaron Schulz
c2861f924c * Corrected and improved FileBackend::getFileList() comments.
* Added unit tests for using getFileList() iterators twice.
2012-02-08 22:13:03 +00:00
Antoine Musso
394c8b5234 enhance filerepo doc structure 2012-02-08 15:51:16 +00:00
Aaron Schulz
e0c78101e5 In FileBackend:
* Added simple getReadOnly()/getReadOnlyReason() functions.
* Allow directly passing a LockManager object into __construct(), useful for testing.
* Fixed bug in FSFileBackend were creating empty files would result in a failing status.
* Added more file stat unit tests.
2012-02-06 05:25:26 +00:00
Aaron Schulz
d9c7111092 Per r108369 CR comment, use StringUtils::escapeRegexReplacement() since backend name can have $1 or such. Also added comments to discourage such characters in backend names. 2012-02-03 18:05:33 +00:00
Aaron Schulz
f229505f96 r110253: reworded "expensive" cache fields and function per CR 2012-02-01 23:47:08 +00:00
Aaron Schulz
d010928d5a Quick fix to getContainerHashLevels() comparison from r110435. 2012-02-01 04:58:19 +00:00
Aaron Schulz
d84684ae30 Expanded 'shardViaHashLevels' config var in FileBackendStore to be able to recognize FileRepo-style deleted zone hash paths (which are different than for the other zones). 2012-01-31 21:52:58 +00:00
Aaron Schulz
7ccbeab5f0 * r108353: Made FileBackendMultiWrite consistency checks configurable.
* Removed redundant code from r110259 and added a comment to isValidContainerName().
2012-01-30 08:00:19 +00:00
Aaron Schulz
92408acbfb Renamed FileBackend to FileBackendStore and ugly FileBackendBase to FileBackend. Callers doing things like FileBackend::extensionFromPath() are now more proper. 2012-01-29 22:22:28 +00:00
Aaron Schulz
9a1452ae92 In FileBackendBase/FileBackend:
* Moved some public static functions from FileBackend to FileBackendBase as the later defines the public API.
* Made splitStoragePath() return null if the backend or container name is empty.
* Made normalizeContainerPath() kill leading directory separators.
* Added more unit tests and made some documentation tweaks.
In FSFileBackend:
* Added resolveContainerName() to disallow '.' a container name, since this would cause a traversal.
2012-01-29 21:28:31 +00:00
Aaron Schulz
d70c7e6b28 * Split up process cache in FileBackend into separate arrays for expensive and inexpensive entries. Put the local reference FSFile object cache in the former since it takes up disk space on /tmp.
* Removed FileBackendBase::resolveWikiId(); doesn't really work well with FileBackendMultiWrite and the functionality is best handled in resolveContainerName(). Follows-up r108303.
* Gave FileOp::doAttempt() a default implementation (a no-op) to be more libertarian.
* Some documentation tweaks.
2012-01-29 19:23:26 +00:00
Aaron Schulz
9d8c4393a5 r107980: added missing dir param to FileBackendShardListIterator 2012-01-27 18:23:40 +00:00
Aaron Schulz
2d0f069800 r107980: added missing dir param to FileBackendShardListIterator 2012-01-27 18:21:58 +00:00
Aaron Schulz
bc8dbb6d23 In FileBackendBase/FileBackend:
* Added normalizeStoragePath() function. Use it for normalizing paths in the stat cache. This way, if someone clears a non-normal form of path from the cache, it will actually work as expected.
* Removed 'abstract' flag from FileBackendBase::clearCache().
* Documentation tweaks.
2012-01-26 21:05:03 +00:00
Aaron Schulz
d1b159ef63 In FileBackend:
* Use 'b' param in some fopen() calls as needed for Windows and newline handling.
* Removed some useless padding code in FileBackend::getContainerShard(). Initialized $m to make IDE happy.
* Updated some code comments.
In SwiftFileBackend:
* Manually set the ETag when using php-cloudfiles for creating files to avoid https://github.com/rackspace/php-cloudfiles/issues/59.
* Manually set the content type based on how StreamFile::getType(). This makes it safe to read files directly out of the proxy to end-users. The streamFile() backend functions already uses a similar content-type check.
2012-01-25 01:57:28 +00:00
Antoine Musso
3f32bbb551 reverts r109814 r109815
* Ternary operator avoided a call to wfWikiID()
* We really want to explicitly choose the manager
  use LockManagerGroup->get( 'nullLockManager' )
* defaultConfig no more needed if it handles just one parameter that
  we want to set explicitly
2012-01-23 21:56:00 +00:00
Antoine Musso
71abcf2f4c cleanly handle wikiId on FileBackendBase construction
Use the default array added by r109814
2012-01-23 11:03:45 +00:00
Antoine Musso
528897ee86 get a null lock manager by default 2012-01-23 10:59:02 +00:00
Aaron Schulz
a0bce8f10e * Added some wfProfileIn() calls to file backend code.
* Made FileBackend::getFileProps() final.
* Added exception needed in SwiftFileBackend::getConnection().
* Various FileBackendTests fixes and additions. Optimized it a bit by keeping the backend instance in memory.
2012-01-23 08:33:31 +00:00
Aaron Schulz
2d1d7f644b * Fixed bugs in SwiftFileBackend file listing code
* Clear container cache in SwiftFileBackend when clearCache() is called
* Updated a comment and fixed a comment typo
2012-01-22 00:06:18 +00:00
Aaron Schulz
abed714fee * r109659: actually return the exact type we say we do
* Made use of FileRepo::cleanupBatch() in ForeignAPIFile::purgeThumbnails()
2012-01-21 00:04:09 +00:00
Aaron Schulz
eb92ac5557 * Added FileBackend::parentStoragePath() convenience function for getting parent directories.
* In StoreBatchTest: sed proper clean() function to remove temp dirs (follows up r109641). Also removed some commented out code.
* Fixed temp dir leakage in FileBackendTest.
2012-01-20 22:46:35 +00:00
Aaron Schulz
efa202ab2d Make sure FSFileBackend clears the php stat cache in clearCache(). Otherwise it will just clear the FileBackend cache and refetch the same cached data from PHP's stat cache. 2012-01-20 21:55:15 +00:00
Aaron Schulz
e2e0c0d9b7 * Follow-up r109009: Check that paths are usable in FileOp::doPrecheck(). Also lock parent directories to avoid prepare()/clean() race conditions for FS backends.
* Fixed bogus $params var in logException() call in SwiftFileBackend.
* Added 'latest' param to FileBackendMultliWrite::consistencyCheck().
* Dummy-proof FileBackend::getFileStat() w.r.t the 'latest' param and removed related FileOp::allowStaleReads() comment.
* Tweaked backend-fail-batchsize message from r109469.
2012-01-19 23:18:03 +00:00
Aaron Schulz
d0fc7d5d89 Renamed 'overwriteDest' FileBackend operation parameter to just 'overwrite', which is shorter and more consistent with 'overwriteSame' 2012-01-19 02:24:49 +00:00
Aaron Schulz
95f31706ee In FileBackend/FileOp:
* Added a sane default max file size to FileBackend. Operation batches need to check this before trying anything.
* Temporarily adjust the PHP execution time limit in attemptBatch() to reduce the chance of dying in the middle of it. Also added a maximum batch size limit.
* Added some code comments.
2012-01-19 02:07:48 +00:00
Aaron Schulz
f1a06be61c * Fixed type check in StreamFile::prepareForStream() for 404s
* Code comment tweaks and fixes
2012-01-17 05:34:01 +00:00
Aaron Schulz
c4a01a7afd In FileOp/FileBackend:
* Added SHA-1 to FileOp::attemptBatch $predicates (since concatenate was removed from FileOp). This lets us get rid of temp file backups, as the remaining failure case is just the backend medium going down (the ops would break in that case anyway). Doing so reduces RTTs and backup file I/O overhead. This also simplifies expiry object support by avoiding having to stash the expiry values of temp backup objects somewhere.
* Improved precheck() and attempt() status logic in FileOp::attemptBatch() a bit. Use separate $subStatus var to check if each op is OK.
* A few minor code cleanups and comment tweaks.
* Fixed MoveFileOp bug found in unit tests when the source/dest are the same and an overwriteDest/overwriteSame param is given (the file would just get deleted). Improved CopyFileOp in this case too.
Other:
* Added more unit tests.
2012-01-15 22:45:14 +00:00
Aaron Schulz
9d49164df1 * Renamed ContainerShardListIterator -> FileBackendShardListIterator
* Renamed FSFileIterator -> FSFileBackendFileList
* Renamed SwiftFileIterator -> SwiftFileBackendFileList and added it to autoloader
* A few minor doc fixes
2012-01-14 01:52:19 +00:00
Aaron Schulz
36c143110a In FileBackend:
* Made secure() call doPrepare() for caller dummy proofing (especially those that don't check the status).
In FSFileBackend:
* Removed redundant wfMkdirParents() calls in FSFileBackend, we already have prepare() for this purpose. This also keeps it's behavior more consistent with the other backends.
* Made use of 'backend-fail-store' message in doStoreInternal().
Other:
* Updated unit tests and renamed $src => $source in some functions for consistency.
* Added some documentation comments and @since tags.
2012-01-13 23:30:46 +00:00
Aaron Schulz
164ce2bc63 * Distinguish "does not exist" from failure in FSFileBackend::doGetFileStat().
* Added clearstatcache() to doConcatenate() to make it safe for callers to stat the file.
* A few minor comment tweaks.
2012-01-13 04:32:28 +00:00
Aaron Schulz
9956408b55 * Collapse multiple directory separators in FileBackend::normalizeStoragePath(). Also renamed the function to normalizeContainerPath() to be less confusing.
* Made SwiftFileBackend respect the 'latest' parameter of various functions via "X-Newest: true" header. Added to TODO in the doGetFileStat() function, as cloudfiles needs upstream changes for that.
* Fixed SwiftFileIterator to return paths relative to the given directory.
* Clean up for FSFileBackend trailing slash handling.
2012-01-12 22:01:02 +00:00
Aaron Schulz
5a6559d8b1 Reverted r108743 per CR comment. This should at least be discussed first. 2012-01-12 19:41:18 +00:00
Sam Reed
a5c200a25b Kill "* @return void" 2012-01-12 19:12:24 +00:00
Aaron Schulz
aad6cb07d6 r108353: Distinguish null/false in FileBackend::fileExists(). This is intended for things that might really care. 2012-01-10 02:18:27 +00:00
Aaron Schulz
3cd02f7a38 * r107986: Added readOnly checks to prepare(), secure(), and clean() in FileBackendBase.
* Added some prepare()/clean() tests.
2012-01-09 00:20:28 +00:00
Aaron Schulz
50a7097436 * Fixed 'success' value of doOperations() Status to match documentation.
* Made 'success', 'successCount', and 'failCount' fields reflect the overall operation in FileBackendMultiWrite::doOperationsInternal(). This makes it match up with single-write backends.
* Made FileBackend::clearCache() part of the public API.
2012-01-08 22:10:53 +00:00
Aaron Schulz
94c3039106 * Follow-up r107170: Moved FileBackend::concatenate() outside of doOperations() as it's own separate operation. It does not mutate storage files like the others and having it in doOperations() broke FileBackendMultiWrite. This change also makes overriding doOperationsInternal() easier (suching as using a custom batch operation storage API).
* Added sanity check to FileBackendMultiWrite constructor.
* Moved FileBackend::create() function up a bit.
2012-01-08 09:25:15 +00:00
Aaron Schulz
360e3376bf In FileBackend:
* Added getFileSize()/getFileStat() functions. Refactored some functions to use the stat function for better reuse and caching/consistency.
* Refactored streamFile() to allow for subclasses to avoid local file copying with less duplication. Also make last-modified check actually work since we always get the timestamp of the original file.
* Renamed 'ignoreErrors' parameter to 'force'.
In FileBackendMultiWrite:
* Simplified how read ops are done (use 'master' backend for consistency).
* Added consistency check to doOperationsInternal() to check if the files are synced.
* Various fixes after testing.
In StreamFile:
* Split out prepareForStream() function from stream() in StreamFile for code reuse.
In FileBackendTest:
* Properly cover FileBackendMultiWrite in tests.
* Various test improvements.
2012-01-08 08:40:00 +00:00
Aaron Schulz
73e83a48b9 * Fixed bogus dollar signs left in $tmpGlobals array keys in r108300.
* Various documentation cleanups.
2012-01-07 03:46:54 +00:00
Aaron Schulz
049b7002fc Partially reverted bits from r108300. Not allowing underscores is too restrictive and breaks on default DB name value suggested by installer. Azure can use resolveContainerName() to enforce its extra restrictions. A new resolveWikiId() function was added to allow for it to mangle the wiki ID by translating underscores. Also restored previous size limit in isValidContainerName() since it makes sense to either go all the way lowest-common-denominator or not here (in this case not). Preserved "first char is a alphanumeric" check since it's pretty weird to have anything else. 2012-01-07 03:17:17 +00:00
Aaron Schulz
7d923a9360 In FileBackend/FileOp:
* Replaced 'media' portion of container names with the repo name. This makes it easy for multiple repos to use the same backend without 'wikiId' hacks. Full container names are now like <wiki>-<repo>-<zone> (or <repo>-<zone> if 'wikiId' is set to an empty string).
* Restricted isValidContainerName() more in light of Azure portability and shorted shard suffix.
* Bumped $maxCacheSize to 75 storage paths.
* Code comment cleanups and additions.
Unit tests:
* Updated related tests and marked testBug29408() as broken (I can't find the problem).
* Reduced leakage in UploadFromUrlTestSuite a bit.
2012-01-07 01:33:23 +00:00