Commit graph

1624 commits

Author SHA1 Message Date
Fomafix
6866cfec37 Simplify PHP by using ?? and ?:
Also remove not necessary surrounding parentheses.

Change-Id: I0eb5c9c1bdfb09a800258379cdcefb5fd4d3d21c
2018-07-10 20:03:17 +00:00
Aaron Schulz
c9ae846db7 Remove deprecated CONN_TRX_AUTO alias
Change-Id: I0bc9b89e998cd3b324f9c72507d40a473bfb6118
2018-07-03 13:03:55 +01:00
jenkins-bot
ab5edd6ea2 Merge "RESTBagOStuff: improve timeouts and logging" 2018-07-03 11:18:43 +00:00
jenkins-bot
4b5773a4de Merge "rdbms: fix Sqlite::tableExists() method to avoid STATUS_TRX_ERROR" 2018-07-03 05:40:03 +00:00
jenkins-bot
9fa37f0a8b Merge "objectcache: make MultiWriteBagOStuff handle duplicate add() operations" 2018-06-29 02:16:52 +00:00
Aaron Schulz
6afa7bb86a Make ProcessCacheLRU wrap MapCacheLRU
Change-Id: I190c824af471aee798e2f111b902f38532b8ac99
2018-06-28 20:23:37 +00:00
jenkins-bot
fa219d9652 Merge "Add key expiration and map resizing support to MapCacheLRU" 2018-06-28 19:22:03 +00:00
Aaron Schulz
6612407f9c objectcache: make MultiWriteBagOStuff handle duplicate add() operations
Bug: T198280
Change-Id: Ib1bcde2b3fbfb452f80d8d840c494be2eb70eb87
2018-06-28 16:04:35 +00:00
Aaron Schulz
1f2b2f3f8d Add key expiration and map resizing support to MapCacheLRU
Also implement Serializable in to add stability to anything that
uses naïve serialization (such as for persistent caching).

Change-Id: I03ba654ffd80ba027d47d5d7583abfe48d52818d
2018-06-28 05:53:42 +00:00
Aaron Schulz
61a7e1acd0 objectcache: define makeKey()/makeGlobalKey() for ReplicatedBagOStuff
Proxy to the "master"/"write" cache object method. This is similar to the
approach taken in MultiWriteBagOStuff

Bug: T198279
Change-Id: If0933246b7ef4fc07ebeec4c3c9625b1137dbe05
2018-06-27 14:16:01 +00:00
jenkins-bot
5856fd2f6f Merge "rdbms: make some LBFactory fields private" 2018-06-24 03:06:28 +00:00
Aaron Schulz
d7707fa814 rdbms: rearrange LoadBalancer fields and make some private
Put the object/callback fields first, followed by config fields
and then state fields, similar to LBFactory.

Change-Id: I40ceaa48842b505f15efe5e41ca7a6de437b6ff1
2018-06-24 01:29:59 +00:00
Aaron Schulz
d16bfb0a30 rdbms: fix Sqlite::tableExists() method to avoid STATUS_TRX_ERROR
Sqlite used the base implementation of trying a SELECT 1 query and
seeing if it failed. Instead, make it use the sqlite_master table.
Also remove the base version of that method since it would always
cause this problem and all subclasses have proper implementations.

Make LoadBalancerTest::assertWriteAllowed() more explicit and add
more assertions there.

Change-Id: I6c7b0bea8894c45dfe8931748d6687f0e5d1e101
2018-06-22 22:33:45 +00:00
Tim Starling
c21daeebe0 RESTBagOStuff: improve timeouts and logging
* Provide a sensible default for the HTTP timeout, and make it configurable.
* Also make HTTP proxy and caBundlePath configurable.
* Pass through the logger object to the HTTP client, as in EtcdConfig.
* Do not treat a 404 response code for DELETE or 204 for PUT as errors. The
  doc comment already said that delete() should return true if the item
  was not found.

Change-Id: I6a90a25339c3a433381a5300c8c7c1a644e2a10f
2018-06-21 09:34:02 +00:00
Aaron Schulz
ce1773b70d rdbms: make some LBFactory fields private
Change-Id: I3fbd35048556593e298731cb497cc83335c5ad8e
2018-06-21 01:56:39 +01:00
jenkins-bot
5572895c9f Merge "Xhprof: support tideways-xhprof extension" 2018-06-20 22:55:26 +00:00
Aaron Schulz
626c5f6972 objectcache: change "miss" to "renew" in metric name for preemptive refreshes
This will usually let one distinguish asynchronous preemptive cache refreshes
from synchronous value recomputations due to actual cache misses.

Change-Id: I3424a89cc96a12fe283460eeaf722ee0cf299534
2018-06-20 15:23:35 +00:00
Edward Chernenko
9d4a7e4d75 Xhprof: support tideways-xhprof extension
The "tideways" extension was renamed by their developers, see
https://tideways.com/profiler/blog/releasing-new-tideways-xhprof-extension

While doing so, they also renamed enable/disable functions:
tideways_enable -> tideways_xhprof_enable
tideways_disable -> tideways_xhprof_disable

Change-Id: I63bc97dba461dd46241a094dfc59439c0d28a219
2018-06-19 14:43:40 +00:00
jenkins-bot
e43395cf79 Merge "Allow passing the default DB group to use in Maintenance scripts" 2018-06-19 05:22:44 +00:00
jenkins-bot
f2d0c3492b Merge "Improve timeouts in MultiHttpClient" 2018-06-15 23:28:12 +00:00
Timo Tijhof
fad1d90760 stats: Hard-deprecate BufferingStatsdDataFactory::getBuffer()
Follows-up 7fdc3d09a3. Also document the getData method.

This method seems unused from a quick glance at Code Search, but
there's a lot of false positives from Monolog instances.

To play safe, rather than removing, add hard-deprecation for
a few days just in case we find another hit.

Change-Id: I8aab93744e66c1a583c67f0bbac27a45250a8fbb
2018-06-15 23:15:26 +01:00
Tim Starling
792dc3c74a Improve timeouts in MultiHttpClient
* PHP 7 only checks for request/connection timeout after the timeout
  passed to curl_multi_select() expires. So it's necessary to use a
  select timeout which is much shorter than 10s. I filed
  https://bugs.php.net/bug.php?id=76480 for this.
* Use millisecond resolution timeout parameters. These have been
  available since libcurl 7.16.2, released Jan 2008, available in
  Debian 7, Ubuntu 14.04 and CentOS 6.

Change-Id: Ia07b824dde179b33e14b81a76d580ce547bca315
2018-06-15 21:04:05 +10:00
jenkins-bot
aa85156042 Merge "rdbms: avoid redundant SPI logging fields in ChronologyProtector log entries" 2018-06-14 14:54:53 +00:00
Niklas Laxström
0e1566505f Fix typo in method documentation
Change-Id: I891f9725acfac993966238abff255e59e9aee07a
2018-06-14 10:47:08 +02:00
Aaron Schulz
6a6771e553 rdbms: disable ChronologyProtector if EmptyBagOStuff is used
Bug: T197206
Change-Id: Ic07634224fac1ae72d1ba20176c028fdda5ba415
2018-06-14 06:57:31 +00:00
jenkins-bot
c4568ad41b Merge "rdbms: make getCPInfoFromCookieValue() stricter about allowed values" 2018-06-12 23:51:49 +00:00
Marius Hoch
d74413bbe7 Allow passing the default DB group to use in Maintenance scripts
Also allows setting it via a global, although that is probably
less useful.

Bug: T147169
Change-Id: Ic51204a6f6ce9db4cc96108e823e388512724eff
2018-06-12 23:52:40 +02:00
Aaron Schulz
44b47b43ee rdbms: make getCPInfoFromCookieValue() stricter about allowed values
All components, not just the write index, must now be present.

Bug: T194403
Change-Id: I279ba3e16d470aca09fdb74cec91d28efb5e2f95
2018-06-12 18:18:54 +00:00
Aaron Schulz
f696f47c97 rdbms: avoid redundant SPI logging fields in ChronologyProtector log entries
Also renamed the field to a better name.

Change-Id: I8bd13a01415a7518b5d9f7dc393b32848efebbf6
2018-06-11 12:44:18 -07:00
Fomafix
e1630b6a53 PHP: Use short ternary operator (?:) where possible
Change-Id: Idcc7e4fcdd4d8302ceda44bf6d294fa8c2219381
2018-06-11 11:26:35 +02:00
jenkins-bot
9bc186e021 Merge "Get rid of call_user_func(_array)(), part 3" 2018-06-10 02:46:30 +00:00
Max Semenik
1e680456b4 Get rid of call_user_func(_array)(), part 3
Also cleaned up nearby code in a couple places.

Change-Id: Ibf44ee7c0ceb739d7e79406e4ff39303c316e285
2018-06-10 02:21:24 +00:00
Max Semenik
9bab7de5f8 Clean up CSPRNG support for PHP7
Replace it all with random_bytes(), leave
only MWCryptRand::generateHex() as a convenience helper.

Change-Id: Ic30376a90e66d8f00dab86e7e6466fb3a750b87d
2018-06-10 00:52:04 +00:00
jenkins-bot
2f8e83821b Merge "Replace call_user_func_array(), part 2" 2018-06-09 14:25:26 +00:00
Aaron Schulz
9130b0395c Add more logging to ChronologyProtector::initPositions()
Bug: T194403
Change-Id: I8f1ccb3c6e257ae48ae6bbecd8f2a8f51cd2ed41
2018-06-08 15:14:05 -07:00
Tim Starling
4aecdad647 HashRing: use bisection instead of guess and refine
Use bisection to find the position within the ring, as in ketama.
According to my benchmarking this is faster than the guess/refine
algorithm for rings with 100 locations, i.e. 16000 ring nodes requiring
at most 14 bisection iterations.

Change-Id: I6042f382de093081971b5ec210eda8dfa74988f2
2018-06-08 02:36:49 -07:00
Max Semenik
6e956d55aa Replace call_user_func_array(), part 2
Uses new PHP 5.6 syntax like ...parameter unpacking and
calling anything looking like a callback to make the code more readable.
There are much more occurrences but this commit is intentionally limited
to an easily reviewable size.

In one occurrence, a simple conditional instead of trickery was much more readable.

This patch finishes all the easy stuf in the core, the remainder is either unobvious
or would result in smaller readability gains. It will be carefully dealt with in
further commits.

Change-Id: I79a16c48bfb98b75e5b99f2f6f4fa07b3ae02c5b
2018-06-07 20:19:26 -07:00
Aaron Schulz
f2e6543c46 Rewrite HashRing to use consistent hashing
This scheme avoids disruptive re-hashing when a node is
ejected; only the adjectent clockwise sector is affected,
the later taking responsibility for the former. Addition
of new nodes is likewise less disruptive.

When used with MD5, this maps keys the same as libketama.

Also:
* Make HashRing::getLocations() faster by guessing the
  sector list index and moving clockwise/counter-clockwise.
* Made HashRing Serializable for more robust storage.
* Allow for different hash functions to be specified.

Note:
* This makes PoolCounterRedis use the new hash scheme.
* Likewise for JobQueueFederated (for job de-duplication).
  Switching from one mode to the other will cause some of
  duplicated effort from old jobs only temporarily.

Change-Id: Iff432a4e5dc3fb40b2dda70fb3dbe9dda0b74933
2018-06-07 03:36:22 -07:00
jenkins-bot
dc6d8d2c3e Merge "objectcache: clean up language about expiration/invalidation" 2018-06-06 00:12:01 +00:00
Aaron Schulz
bde2c5aa9e objectcache: clean up language about expiration/invalidation
Change-Id: I593e7bf9bd4f7fbb615d3d164e77273bd4033dde
2018-06-05 23:45:19 +00:00
jenkins-bot
33ca739e7f Merge "objectcache: update MemcachedPeclBagOStuff for pecl memcached 3.0.0" 2018-06-05 15:50:29 +00:00
jenkins-bot
a93cf8f4ee Merge "clientpool: refactor Redis authentication error handling" 2018-06-05 02:05:04 +00:00
jenkins-bot
390d35a1af Merge "rdbms: include client ID hash in ChronologyProtector cookies" 2018-06-04 22:08:22 +00:00
Bartosz Dziewoński
0313128b10 Use PHP 7 "\u{NNNN}" Unicode codepoint escapes in string literals
In cases where we're operating on text data (and not binary data),
use e.g. "\u{00A0}" to refer directly to the Unicode character
'NO-BREAK SPACE' instead of "\xc2\xa0" to specify the bytes C2h A0h
(which correspond to the UTF-8 encoding of that character). This
makes it easier to look up those mysterious sequences, as not all
are as recognizable as the no-break space.

This is not enforced by PHP, but I think we should write those in
uppercase and zero-padded to at least four characters, like the
Unicode standard does.

Note that not all "\xNN" escapes can be automatically replaced:
* We can't use Unicode escapes for binary data that is not UTF-8
  (e.g. in code converting from legacy encodings or testing the
  handling of invalid UTF-8 byte sequences).
* '\xNN' escapes in regular expressions in single-quoted strings
  are actually handled by PCRE and have to be dealt with carefully
  (those regexps should probably be changed to use the /u modifier).
* "\xNN" referring to ASCII characters ("\x7F" and lower) should
  probably be left as-is.

The replacements in this commit were done semi-manually by piping
the existing "\xNN" escapes through the following terrible Ruby
script I devised:

  chars = eval('"' + ARGV[0] + '"').force_encoding('utf-8')
  puts chars.split('').map{|char|
    '\\u{' + char.ord.to_s(16).upcase.rjust(4, '0') + '}'
  }.join('')

Change-Id: Idc3dee3a7fb5ebfaef395754d8859b18f1f8769a
2018-06-04 16:20:13 +00:00
Aaron Schulz
7a730a6f9e objectcache: update MemcachedPeclBagOStuff for pecl memcached 3.0.0
The get() $cas_token parameter was changed into $flags, which can act
as a switch to make the return value an associative array of details.

pecl and PHP-based memcached BagOStuff class both pass all tests now.

Bug: T196125
Change-Id: I4678250331a48db4d50d1fc6c489c991a4dee920
2018-06-02 17:25:22 -07:00
Aaron Schulz
fb51330084 rdbms: include client ID hash in ChronologyProtector cookies
Previously, if an internal service forwarded the cookies for a
user (e.g. for permissions) but not the User-Agent header or not
the IP address (e.g. XFF), ChronologyProtector could timeout
waiting for a matching writeIndex to appear for the wrong key.

The cookie now tethers the client to the key that holds the
DB positions from their last state-changing request.

Bug: T194403
Bug: T190082
Change-Id: I84f2cbea82532d911cdfed14644008894498813a
2018-06-02 03:57:30 +00:00
jenkins-bot
285080c311 Merge "objectcache: make RedisBagOStuff pass all tests" 2018-06-02 01:35:26 +00:00
jenkins-bot
966e34619b Merge "objectcache: add setMockTime() method to BagOStuff/WANObjectCache" 2018-06-01 13:19:24 +00:00
Aaron Schulz
acace9a049 objectcache: add setMockTime() method to BagOStuff/WANObjectCache
Change-Id: I3e5760814fb7dbe628eb0d979d690c3275fc3c15
2018-06-01 03:46:58 +00:00
Aaron Schulz
13f7232bf4 objectcache: make RedisBagOStuff pass all tests
* Provide a default lock-based BagOStuff::cas implementation
* Make RedisBagOStuff::merge() use mergeViaCas()
* Use the raw unserialized string as the redis CAS token to
  avoid any bad interaction with __wakeup() methods changing
  field values every time
* Make RedisBagOStuff::incr() return false when there is no
  such key, not null
* Rewrite merge() test to make the order of write/cas phase
  of the parent and child merge() calls well defined instead
  of arbitrary usleep() calls
* Avoid cache key reuse in test runs

Change-Id: I388ec173cf3858bb2fc7a8c8a00cda68703074ce
2018-05-31 20:43:10 -07:00