Commit graph

105 commits

Author SHA1 Message Date
Aaron Schulz
d2d935483f Make makeKeyInternal() limit more conservative
This should avoid error log entries for long WAN cache keys

Change-Id: I401482d25dd5bf47052a3c6729c5f8bc9fd68770
2015-11-01 19:37:21 -08:00
Aaron Schulz
7cddc22fb8 objectcache: Allow bounded HashBagOStuff sizes and limit it in WANObjectCache
Change-Id: Icca2474b1ea6feb7134f8958aecf79aa51b7f71e
2015-10-31 02:04:01 +00:00
Aaron Schulz
6d0108d533 Fix slow callbacks in getWithSetCallback() using lockTSE
* Keys that take a long time to generate would run into
  the MAX_SNAPSHOT_LAG check and have set() no-op. This
  would be fine except that leaves no key there to figure
  out the time since expiry and therefore whether to use
  the mutex, so it didn't. This now saves the keys but with
  a FLG_STALE bit set, making the next caller that sees it
  perform a regeneration (unless it can't get the mutex).
* Bumped LOCK_TTL so that keys can stay locked much longer.
* This is easy to test via sleep(5) in tagUsageStatistics()
  and two Special:Tags browser tabs.

Bug: T91535
Change-Id: I549e70ace3d2e7da40d3c4346ebacc36024cd522
2015-10-28 23:31:37 +00:00
jenkins-bot
33859dfcf5 Merge "Make WAN cache HOLDOFF_TTL smaller by combining db/snapshot lag" 2015-10-28 15:10:40 +00:00
Aaron Schulz
31b144a7cf Improve WAN cache delete() docs a bit
Also use more TTL constants in getWithSetCallback() examples

Change-Id: I866f0464c07fa0c8977c6e908e92186b2ef9de43
2015-10-27 22:06:46 -07:00
Timo Tijhof
e8275758fe objectcache: Introduce IExpiringStore for convenient TTL constants
Also consistently use self:: instead of BagOStuff:: for constants
referenced within the BagOStuff class.

Change-Id: I20fde9fa5cddcc9e92fa6a02b05dc7effa846742
2015-10-28 04:07:25 +00:00
Aaron Schulz
05a3ee2468 Make WAN cache HOLDOFF_TTL smaller by combining db/snapshot lag
* In the common ~0 second lag case, transactions up to ~7 seconds
  long will have normal set() behavior (instead of just ~5 seconds).
* Like wise with ~0 second transactions tolerating ~7 seconds of
  lag (instead of just ~5).
* The lower hold-off time lets caching resume 3 seconds sooner.

Change-Id: I21e2a0a6915492cec422b6a6af5541937745c15b
2015-10-27 11:38:54 -07:00
jenkins-bot
30135f1dec Merge "Protect WAN cache sets() against uncommitted data" 2015-10-27 10:18:52 +00:00
Aaron Schulz
dc646b464c Protect WAN cache sets() against uncommitted data
This generally only effects wikis with no slave DBs,
but also matters if the master has non-zero LB load.
If the master ends up being used for DB_SLAVE, care
should be shown for cache-aside writes

Interesting WAN cache events are now logged.

Change-Id: I2cd8e84138263c13ea23beb9ab3d7562340e1fd3
2015-10-24 17:21:22 -07:00
jenkins-bot
b9778e1a10 Merge "Fix bogus calls to encodeKey/decodeKey in the PHP memcached class" 2015-10-24 23:39:50 +00:00
Timo Tijhof
22b9599cda objectcache: Remove outdated comment about back-compat defaults
Follows-up 0dd27b1da9.

Change-Id: Ic21b02d93d6047eb4713dfec3cbb2fff4d51fdff
2015-10-24 23:41:44 +01:00
Aaron Schulz
51fe42fc1a Fix bogus calls to encodeKey/decodeKey in the PHP memcached class
* Follow up to cdb5432728
* These are no longer needed

Change-Id: Id41c3aae50a13f3c53f2bbd9662027b36a1f4142
2015-10-24 15:38:23 -07:00
Aaron Schulz
f36b73e96c Move some memcached classes to /libs
Also removed a few MW dependencies from MemcachedPeclBagOStuff.
It still uses an IP class method, so it has to stay for now.

Change-Id: I8c5c83046c58fb58091d6ce11b2385208262460f
2015-10-24 15:27:42 -07:00
Aaron Schulz
cce813a922 Move MultiWriteBagOStuff to /libs
Also moved related tests files to /libs.

Change-Id: I806eeaa30205733d497adde933baf0c4157f7aae
2015-10-24 12:15:26 -07:00
Ori Livneh
0c9fb12265 Escape colons in BagOStuff key segments
For the sake of safety and correctness, the following BagOStuff::makeKey()
invocations should return distinct keys:

   $cache->makeKey( 'ab:', 'cd' );
   $cache->makeKey( 'ab', ':cd' );

That is not currently the case, because while we use ':' as a key path
separator, we don't escape ':' in the input supplied to makeKey(). So, make
BagOStuff::makeKeyInternal() URL-encode colons.

To prevent this from messing up the max. key length calculations, reproduce
this logic in MemcachedBagOStuff::makeKeyInternal(), in lieu of having the
method call its parent.

Change-Id: I83ea7e7336a1c9e64aa42284c2517089a736efe5
2015-10-23 20:26:49 -07:00
Aaron Schulz
fb7022c54d Bound BagOStuff::lock() RTT estimate for sanity
The timestamp difference might be 0 (or in rare cases, less)

Change-Id: I0298b413df637d9da5ff33c078ad49dadc8a46ce
2015-10-21 20:14:54 -07:00
jenkins-bot
409e7d6a70 Merge "objectcache: Fix grammar error in BagOStuff::getWithToken doc" 2015-10-22 02:33:16 +00:00
Aaron Schulz
5c8ef13306 Add WRITE_SYNC flag to BagOStuff::set()/merge()
* This blocks on writing to all replicas
  and returns false if any failed.
* This is useful if ChronologyProtector is to work across
  domains by having the writes go everywhere so that later
  reads will see them (and be local at the same time).
* Redundant doc comments were also removed.

Change-Id: I9ed098d563c64dba605e7809bc96731da3b3e79d
2015-10-22 01:44:09 +00:00
Timo Tijhof
92b197dc59 objectcache: Fix grammar error in BagOStuff::getWithToken doc
Follows-up 1fa1235d32.

Change-Id: I19a4430609423f5ff32a1014ed21131175469adf
2015-10-22 02:35:35 +01:00
Aaron Schulz
922f15e8e0 Remove ObjectCache dependency from MultiWriteBagOStuff
This brings it closer to being able to move to /libs.

Change-Id: Ia733f9023e56d4a25ffcb99ca0cc8b29cbb2ad45
2015-10-20 00:16:51 +00:00
jenkins-bot
f17437d8c2 Merge "Add make(Global)Key() method to WANObjectCache" 2015-10-20 00:14:54 +00:00
jenkins-bot
105517c68c Merge "objectcache: Remove getWithSetCallback() signature backwards-compatability" 2015-10-19 23:56:11 +00:00
Aaron Schulz
f84b32af37 objectcache: Remove getWithSetCallback() signature backwards-compatability
All callers have been migrated

Change-Id: I7b6b87594dd724434ba24d8206fe07d66c1d5d25
2015-10-19 23:47:04 +00:00
Aaron Schulz
d4895d5c70 Add make(Global)Key() method to WANObjectCache
Change-Id: I8e739fbe8614c9bd1c3595bbf16c8cd423aff7cc
2015-10-19 12:55:30 -07:00
jenkins-bot
ba405dd13b Merge "Mention WANObjectCache::TTL_INDEFINITE in set() method" 2015-10-17 22:55:25 +00:00
jenkins-bot
7436cd31a2 Merge "Add makeKey and makeGlobalKey to BagOStuff" 2015-10-17 01:06:12 +00:00
Aaron Schulz
d7021c077d Mention WANObjectCache::TTL_INDEFINITE in set() method
Change-Id: I6e5cb2e7ba34ceaf4ca128e4b32da99d08917156
2015-10-15 08:09:48 -07:00
Timo Tijhof
68a54d6d7a objectcache: Rename WANObjectCache TTL_NONE to TTL_INDEFINITE
"None" has a somewhat unobvious meaning, also ambiguous with "uncachable".

Change-Id: I799de385427aeb8f581d51800606552bdc998252
2015-10-14 17:08:59 -04:00
Ori Livneh
6916548490 Add makeKey and makeGlobalKey to BagOStuff
* Add a string `keyspace` member to BagOStuff instances. The default
  implementation, meant for simple key/value stores, treats the key space
  as a string prefix to prepend to keys. By default, its value is `local`,
  but any instance created via ObjectCache::newFromParams() (or or one of
  its callers) will have that default to $wgCachePrefix / wfWikiID().
* Add `makeKey` and `makeGlobalKey` methods to the base BagOStuff class.
  These methods are not static to allow for BagOStuff types which require
  a configured instance to know the underlying storage engine's key semantics.
* Make wfMemcKey() and wfGlobalCacheKey() delegate to these methods on the main
  ObjectCache instance.

Change-Id: Ib7fc2f939be3decfa97f66af8c2431c51039905f
2015-10-13 09:32:48 -04:00
Aaron Schulz
1a6151e3a6 Caching documentation tweaks and doxygen cleanups
Change-Id: I4b37a6c7d267d4eb0b39b313d92e6c0bf87d460e
2015-10-10 01:06:37 -07:00
Aaron Schulz
968e28f894 Add process cache support to WANObjectCache
* Make getWithSetCallback() accept a TTL field for this
* Make ChangeTag callers use this flag to avoid hundreds of
  duplicate queries at Special:Tags

Change-Id: Ic1ed28294f5d557e02f39a7f20d36766244b9ded
2015-10-10 06:14:50 +00:00
jenkins-bot
9e56569f9f Merge "Minor getWithSetCallback() example formatting tweaks" 2015-10-10 04:55:52 +00:00
Aaron Schulz
17e8bd3f32 Better explain WANObjectCache::TTL_NONE constant
Change-Id: I2ce363c957f35e75a17f05b2160979a6fa4aa642
2015-10-10 00:12:37 +00:00
Aaron Schulz
3c021116e4 Minor getWithSetCallback() example formatting tweaks
Change-Id: I6fc7164770fbbd32374b01d1aa27dbaa9263c940
2015-10-08 15:49:57 -07:00
jenkins-bot
dad851600f Merge "Add get() flags support to BagOStuff::getWithSetCallback()" 2015-10-08 15:53:17 +00:00
Aaron Schulz
e5878c522b Update getWithSetCallback() examples to use the new signature
Change-Id: I81916b4d1884bad5d30f689698aa35cdb9f139eb
2015-10-07 21:52:43 -07:00
Aaron Schulz
b39fcc855c Add get() flags support to BagOStuff::getWithSetCallback()
Change-Id: Ib6046f5bc5d8c0481f90b56c381ba761fee558d6
2015-10-07 20:51:00 -07:00
Aaron Schulz
5aee45ee9e Fix default $oldOpts from 09cbebb224
This fixes the default $oldOpts value for the case when
$opts is given check keys and no $oldOpts value is passed
in. In that case, the getWithSetCallback() method
variable $opts would be null instead of array().

Change-Id: I825062533478b37766511aae1d06dd4b81f8dba1
2015-10-07 20:26:38 -07:00
Timo Tijhof
09cbebb224 WANObjectCache: Change getWithSetCallback() signature to key/ttl/callback/opts
* Put 'checkKeys' param in opts array instead of as a separate parameter.
  It's neat to not have to skip unnamed/positional arguments that are optional.
* Move TTL to be before callback instead of after. This avoids dangling integers
  toward the bottom of a large code block that have no obvious meaning.
  Matches BagOStuff::getWithSetCallback.

Add unit test for lockTSE to confirm

Change-Id: I60d6b64fc5dff14108741bafe801fd1fde16d1df
2015-10-07 18:03:15 -07:00
Aaron Schulz
667e9ef44b Bump WANObjectCache::LOW_TTL to 30
This better handles values that take a long time to
generate. For example if it took 10 sec, then the old
value would not regenerate in time. 30 is a good, high,
value that is the default PHP timeout.

Change-Id: I2e4372bf4c6d28d09347012926d8c0a9deda4183
2015-10-06 23:50:28 -07:00
Aaron Schulz
1fa1235d32 Clean up BagOStuff::get() interface
* Callers of get() no longer have to contend with
  the annoying $casToken parameter, which is there
  but totally unusable to non-BagOStuff code.
* The default get() now delegates to doGet(),
  which callers must implement instead. They can
  ignore the overhead of generating $casToken if
  they do not implement cas(), which applies to
  callers that use the stock merge(). If cas() is
  used for merge(), then getWithToken() must be
  implemented.
* Also add BagOStuff::READ_LATEST to mergeViaCas()
  for sanity, as that missing before.
  Likewise with mergeViaLock().

Change-Id: I4efce6a9ab4b1eadd2f161dff641004a7239c516
2015-10-07 02:54:57 +00:00
Aaron Schulz
54758dd160 Change getCacheSetOptions() callers to use "Database"
* This is less verbose that DatabaseBase
* Also add a few WAN cache doc comments

Change-Id: I5b6de6d0ffa06753ea96c50b63db7dae796475dc
2015-10-05 22:06:46 -07:00
Aaron Schulz
355ba8530e objectcache: Add BagOStuff::getWithSetCallback() convenience method
Change-Id: I9cc162ff1cc48c1c500f2999327bd18ba235bfd0
2015-10-05 17:26:32 -07:00
Aaron Schulz
db0b9ef264 Make WANObjectCache sets account for slave lag
* This gets lag information that is useful when
  the calling code is about to run queries that
  will have their results cached.
* This is now used in place of trxTimestamp() for
  WANObjectCache set() and getWithSetCallback().
* The WAN cache will use a low TTL if the lag is
  too high to avoid caching stale data for weeks.
* Bumped MAX_COMMIT_DELAY as nothing enforces it.

Bug: T113204
Change-Id: I2a95b4088cff42d6e980351555f81a4b13519e24
2015-10-05 16:45:13 -07:00
Aaron Schulz
54991403ff objectcache: Add BagOStuff::READ_VERIFIED flag to get()
* This lets multiwrite backends upgrade cached items
  to higher tiers using UPGRADE_TTL.
* This is useful for memcached/sql tiers or apc/memcached.

Change-Id: I34b30ce8b54f8de36429d48c80f6768aed310272
2015-10-05 16:03:00 -07:00
Aaron Schulz
7aea96befa objectcache: Add some newlines to WANObjectCache docs
* Doxygen needs this or the paragraph ends up in the
  last bullet point
* Also removed some redundant comments

Change-Id: Ie4bf2b67fff27ecad5f24f55688161fe484c300d
2015-10-01 20:42:16 +00:00
Aaron Schulz
3f3d5a10b9 Cleaned up WANObjectCache toy example code blocks
The list order logic was backwards

Change-Id: Id26e13b69d9e1e6ef132792bdcc693836757cbd8
2015-09-29 23:05:33 -07:00
Aaron Schulz
4ff4296333 Improved getWithSetCallback() docs to have more use cases
Change-Id: I428617c159009f22fa78129c16a8964ac2e830b9
2015-09-29 17:25:38 -07:00
Aaron Schulz
a9665bf4d6 Clarified WAN cache docs by using "datacenter" consistently
Change-Id: I448c57cebd0481809fc17a7c2ac698c73b17dcba
2015-09-28 18:22:52 -07:00
Aaron Schulz
75f06d5294 Added $opts to WANObjectCache::set() to detect snapshot lag
* This can avoid some stale write race conditions
* Made use of this option in a few key places
* HOLDOFF_TTL was also bumped

Change-Id: I83505a59cce0119e456191c3100f7d97bc86bdbf
2015-09-28 23:03:36 +00:00