Commit graph

541 commits

Author SHA1 Message Date
jenkins-bot
afecdf769f Merge "objectcache: suppress TransactionProfiler in occasionallyGarbageCollect()" 2022-11-17 16:12:28 +00:00
Aaron Schulz
9c7ddf1f0a objectcache: use multi-row upsert support in SqlBagOStuff
This cuts down on needless roundtrips and lock contention

Change-Id: Ic9681512a4b17f000da21b45cacad173e644a44e
2022-11-17 07:49:01 +00:00
Aaron Schulz
0f837a1460 objectcache: suppress TransactionProfiler in occasionallyGarbageCollect()
Bug: T258125
Change-Id: I3b0f5b5efb85ccd868b8d797c43565d1786a86b6
2022-11-10 08:02:18 +00:00
jenkins-bot
43830cde86 Merge "Use array_key_first()/array_key_last() in some places" 2022-10-21 21:57:34 +00:00
jenkins-bot
a85966542d Merge "Use short array destructuring instead of list()" 2022-10-21 11:11:06 +00:00
Tim Starling
0077c5da15 Use short array destructuring instead of list()
Introduced in PHP 7.1. Because it's shorter and looks nice.

I used regex replacement.

Change-Id: I0555e199d126cd44501f859cb4589f8bd49694da
2022-10-21 15:33:37 +11:00
Tim Starling
1aeccebaf2 Use array_key_first()/array_key_last() in some places
Introduced in PHP 7.3. I used it to replace reset()/end() followed by
key() where the return value of reset() is not captured and internal
pointer iteration is not locally occuring.

I also used it in a couple of places when reset() is absent but
array_key_first() is semantically desired.

Change-Id: I750d3fa71420cbdca5fb00d82ac5ca40821769d4
2022-10-21 15:01:59 +11:00
Tim Starling
43a93d9782 Use the null coalescing assignment operator
Available since PHP 7.4.

Automated search, manual replacement.

Change-Id: Ibb163141526e799bff08cfeb4037b52144bb39fa
2022-10-21 13:26:49 +11:00
jenkins-bot
06740c77b8 Merge "SqlBagOStuff: Fix modtoken comparison" 2022-08-19 22:44:58 +00:00
Tim Starling
3cabe8bf8a SqlBagOStuff: Fix modtoken comparison
String offsets in MariaDB are 1-based, except in "Oracle compatible"
mode. SUBSTR(modtoken,0,13) was always the empty string and so the
modtoken comparison was always true. I was able to reproduce a failure
to reach consistency using ring replication.

Add regression test.

Bug: T315271
Change-Id: I74e54e8aba44505dd04426c12d91a9ea0de17f22
2022-08-19 11:11:58 +10:00
Derick Alangi
354535977e SqlBagOStuff: Migrate from IDatabase::select to SelectQueryBuilder
Bug: T311866
Change-Id: I0101a0d5f4008bfd79e823248bb625e15f7135bd
2022-08-18 19:36:54 +01:00
jenkins-bot
4225028026 Merge "SqlBagOStuff: use cancelAtomic()" 2022-08-17 16:49:02 +00:00
Tim Starling
8034769392 SqlBagOStuff: use cancelAtomic()
* When an exception is thrown from
  modifyTableSpecificBlobsForIncrInit(), call cancelAtomic() on the
  connection. Otherwise a subsequent call to
  LBFactory::commitPrimaryChanges() will throw a user-visible error,
  violating the goal of graceful failure.
* After getting a DBConnRef, call ensureConnection() on it. Otherwise
  the try/catch blocks around getConnection() have no purpose since
  connections are always lazy-initialised.

Bug: T315274
Change-Id: I8a5bc90bb340f34bd4a88872a1a2e72612ac91f4
2022-08-17 12:54:25 +10:00
Timo Tijhof
38cf6eed98 objectcache: Add trace to SqlBagOStuff DBError logging
Bug: T315274
Change-Id: I7551f2d41238f2dd064e1049723b22805b495516
2022-08-17 00:41:48 +01:00
Timo Tijhof
ed94214c4f objectcache: Remove unused WRITE_SYNC flag
Bug: T270225
Depends-On: I7e72c1180b7ba9e479ade62ab3dd3139d7bd5bb0
Change-Id: I9f59ff35bbb806d7b1375739001d1cf458f366a8
2022-08-02 17:05:26 -07:00
daniel
bf092744c9 PHPUnit: introduce setMainCache
The main object cache is disabled during testing. Some integration tests
need it though. This provides a clean way to enable it, to replace the hacks
that were used so far.

Note that we may want to enable the main cache during testing soon. When
that happens, this method is still useful to disable the cache in certain
tests, and to set a specific cache instance.

Change-Id: I04ae1bf1b6b2c8f6310acd2edf89459d01a9c870
2022-07-07 16:25:59 +10:00
Tim Starling
79afc20506 objectcache: Optimise SqlBagOStuff::incrWithInit with WRITE_BACKGROUND
Bug: T310662
Bug: T261744
Change-Id: I9549722ff6f0c4d62d1bcbe8de55f51758157ec4
2022-07-04 22:43:45 +00:00
Tim Starling
dbd52291a0 objectcache: Deprecate BagOStuff::addBusyCallback() and reduce to a stub
It's not called by anything since it was removed from
ChronologyProtector, and it is not needed for production.

Pass the timeout directly to LoadBalancer::waitForAll() instead of
using a WaitConditionLoop, as it was before e99bb1b7dc

Change-Id: I5a1903e6e9ac145c06f77c690a0f6a487b4aa43f
2022-05-27 15:05:10 +00:00
Tim Starling
bcfb90cdd7 objectcache: Simplify SqlBagOStuff class configuration
Substantive changes:

* Remove the parameters globalKeyLB and localKeyLB, not needed in
  production, probably not used anywhere.
* Add a "cluster" parameter, for production.
* Don't call markServerDown() in LoadBalancer mode.

Style changes:

* Document config parameters separately from constructor parameters, to
  clarify the role of the wiring normalisation.
* Inject a load balancer from ObjectCache to SqlBagOStuff, using a
  closure to defer construction.
* Clarify the split between LoadBalancer mode and server array mode by
  introducing a useLB property. Use it instead of special shard index
  values and array counts.
* multiPrimaryModeType wasn't needed anywhere and wasn't available in
  the constructor, so it just became a boolean multiPrimaryMode.
* Add a test covering most of the server array (!useLB) branches.

Bug: T212129
Change-Id: Ib097082efb40b514a29a42286dc362f2e3743ee4
2022-05-27 15:04:43 +00:00
jenkins-bot
31caa2921c Merge "objectcache: reduce function_exists() calls in SqlBagOStuff" 2022-05-13 22:05:11 +00:00
Matěj Suchánek
e47c441078 Fix many typos in comments
Found using IntelliJ's "Typo" code inspection.

Change-Id: I746220ebe6e1e39f6cb503390ec9053e6518cf16
2022-05-10 12:46:11 +00:00
Aaron Schulz
d283032252 objectcache: reduce function_exists() calls in SqlBagOStuff
Change-Id: I221c5bf35277f0aaf458a3343ac583d332e204d2
2022-05-05 14:48:11 -07:00
Klein Muçi
803b873beb docs: Fix 'the the' typo
Bug: T201491
Change-Id: Idd776953dcf7dadc136e57d586c09b24363005a6
2022-05-05 15:40:26 +00:00
Aryeh Gregor
4a52bf553f Use MainConfigNames instead of string literals, #3
This edition brought to you by:

grep -ERIn $(grep -o "'[A-Za-z0-9_]*'" includes/MainConfigNames.php | tr
"\n" '|' | sed 's/|$/\n/') includes/

I only corrected a fraction of the results provided by that command. I'm
submitting the partial patch now so it doesn't bitrot.

Bug: T305805
Change-Id: If1918c0b3d88cdf90403921e4310740e206d6962
2022-04-26 14:31:26 +03:00
Aaron Schulz
cc1bb73490 objectcache: remove "multiPrimaryMode" DB type assertion
Since no mysql-specific queries are required, allow sqlite and
Postgres setups to use "multiPrimaryMode" for easier testing.

Avoid "CASE types text and integer cannot be matched" Postgres
error by making dbEncodeSerialValue() cast the result to a string
so that addQuotes() applies in buildUpsertSetForOverwrite().

Bug: T212129
Change-Id: Ic84a804c272a7070779c8d41d3a33003852d0839
2022-04-21 16:20:39 -07:00
Aaron Schulz
3a73b852b8 objectcache: fix comments for SqlBagOStuff::deleteServerObjectsExpiringBefore()
Change-Id: I7eef82820187317aece1f53601598ddf9293281f
2022-04-14 15:54:30 +00:00
Timo Tijhof
c2bb1a1b2f objectcache: Simplify docs of SqlBagOStuff 'purgePeriod' option
Parsing "the reciprocal of the probability of purging on any write"
is fine in isolation, but requires too much effort when trying to
undertand the larger picture at the same time.

Bug: T212129
Change-Id: I6f21727c05b4fc0449de5d6acec14e49ed1991ef
2022-04-13 23:43:47 +01:00
jenkins-bot
bf48104bf6 Merge "installer: make 'db-replicated' an alias for CACHE_DB for sqlite" 2022-04-05 21:23:09 +00:00
Aaron Schulz
b81b5fee46 installer: make 'db-replicated' an alias for CACHE_DB for sqlite
By default, CACHE_DB and db-replicated are similar, with the main
difference being that db-replicated will prefer to read from a local
replica server rather than the primary server. Setups using sqlite
do not have use for replication logic, since everything uses local
database files. To reduce contention, SqliteInstaller makes CACHE_DB
use a different database file (rather than a table in the main wiki
database file).

This change makes 'db-replicated' reuse the same cache instance.

Tweak ObjectCache::newFromParams() to make ObjectFactory usable.

Also set $wgResourceLoaderUseObjectCacheForDeps to true to avoid
similar contention in SqlModuleDependencyStore.

Bug: T303225
Change-Id: Ia644f35698b69fe340fc1bcae8899c7e332f332d
2022-04-05 12:40:18 -07:00
Aaron Schulz
c96dec041e objectcache: make "multiPrimaryMode" work with LB-based SqlBagOStuff
This avoids the previous database type exception in the constructor.

Bug: T212129
Change-Id: I43c6a86883f4122f7395b022bdc6ed7a0cf88121
2022-04-05 01:43:53 +00:00
Timo Tijhof
51d7ea8505 docs: Document how SqlBagOStuff::createTables() works
Bug: T272512
Change-Id: I8f1fc754f096543b677f50a357fb430de8ec6c7c
2022-03-25 00:36:13 +00:00
Aaron Schulz
cfbd854d05 http,objectcache: Use local-unaware floats in sprintf() calls
Bug: T303628
Change-Id: I7d92bc9dd2bc30711572abff36a0e1545c88479b
2022-03-24 14:59:03 +00:00
Umherirrender
1f71eccf63 phan: Disable null_casts_as_any_type setting
Make phan stricter about null types by setting null_casts_as_any_type to
false (the default in mediawiki-phan-config)
Remaining false positive issues are suppressed.
The suppression and the setting change can only be done together

Bug: T242536
Bug: T301991
Change-Id: I0f295382b96fb3be8037a01c10487d9d591e7e01
2022-03-21 18:25:07 +00:00
Aaron Schulz
4718e2c529 objectcache: add makeTimestampedModificationToken() type hint
All callers pass in floats and this is the expected type, used
to fill $microseconds.

Bug: T303628
Change-Id: I6c6176b4cc308646d86a5b9d2afdc91aadda4a0d
2022-03-19 21:26:35 +00:00
Umherirrender
fe48e1223a cache: Fix various scalar and null types to match documentation
Adding float as type when INF is used as argument

Found by phan strict checks

Change-Id: I7e19cd0d9b09bf685e4e8387c900f303b669ad54
2022-03-14 17:36:04 +00:00
jenkins-bot
6fd4c95f5e Merge "Use updated ObjectFactory namespace" 2022-03-12 00:19:53 +00:00
Alexander Vorwerk
9b09bf3112 Use updated ObjectFactory namespace
Depends-On: I99c5e5664d2401c36a9890f148eba7c25e6e8324
Depends-On: I48ab818b2965da14af15ef370aa83ad9455badd9
Depends-On: I018371e4b77911e56152ca7b2df734afc73f58a5
Change-Id: I04ebdb52102f6191d49a9cc70b1f98308299e72f
2022-03-09 23:04:51 +00:00
jenkins-bot
0200221a93 Merge "Fix various documentation related to null types (part II)" 2022-03-09 22:42:42 +00:00
Umherirrender
e4829c6ea3 SQLBagOStuff: Fix dbdomain argument in getConnectionViaLoadBalancer
getMaintenanceConnectionRef second parameter is the group,
not the domain
Break another line to make the argument order more visible

Follow-Up: 618b61a4c6
Change-Id: Id6dd0c2f8e5aded759fe403f8435ee7251a46b8c
2022-03-09 20:36:34 +00:00
Umherirrender
d7248d63fb Fix various documentation related to null types (part II)
The functions returning null or the class property is set explict null.
Some function should not accept null or return null.

Found by phan strict checks

Change-Id: Ie50f23249282cdb18caa332f562a3945a58d86ff
2022-03-08 23:45:31 +00:00
Umherirrender
9ed1ed185f Fix various documentation related to false
Some function already document "False on failure", but does not document
it on the used type

Found by phan strict checks

Change-Id: I12eb8bbc99179833ee3e42c1a7d1dc1443682ca6
2022-03-03 21:33:31 +01:00
Umherirrender
9efd9ca45e Add explicit casts between scalar types
* Some functions accept only string, cast ints and floats to string
* After preg_matches or explode() casts numbers to int to do maths
* Cast unix timestamps to int to do maths
* Cast return values from timestamp format function to int
* Cast bitwise operator to bool when needed as bool

* php internal functions like floor/round/ceil documented to return
  float, most cases the result is used as int, added casts

Found by phan strict checks

Change-Id: Icb2de32107f43817acc45fe296fb77acf65c1786
2022-03-01 18:19:33 +01:00
daniel
a5277ce87f Avoid references to DefaultSettings.
This replaces references to DefaultSettings with
references to config-schema.yaml where appropriate.

NOTE: this does not yet change Setup.php. DefaultSettings.php
remains intact and is still being used.

NOTE: this does not remove usages in the installer, see I5d8843a1062fbf
for that.

Bug: T300129
Change-Id: Ie6152cf510c3be61bc22167ca6d90dfc28910a45
2022-02-22 19:43:50 +01:00
Aaron Schulz
bc79ce7fb7 objectcache: deprecate BagOStuff incr() and decr() methods
All callers in Wikimedia gerrit repos have since been migrated to
incrWithInit(), which is usually what is desired and is easier to
implement in various backends. Newer memcached versions of the
memcached protocol even support auto-initializing increments
(including an initial TTL) via the 'ma' command. The incr()/decr()
methods currently bloat the interface and subclasses (often with
slow and buggy implementations). These methods are also hard to
implement in a multi-DC store (either extremely slow or racey)
even though callers might assume they handle high concurrency.

Also, force each MediumSpecificBagOStuff subclass to implement
incrWithInit() instead of having some rely on the base method

In mcc.php, add incrWithInit() and simplify parameters, making
it possible to test non-memcached backends.

Change-Id: I53c9c2c839a1e71d5c104913fea0680c30d11108
2022-02-18 11:15:27 +11:00
Aaron Schulz
618b61a4c6 objectcache: add "globalKeyLbDomain" option to use with "globalKeyLB"
Previously, "globalKeyLB" was generally useless for sharing keys since
the LoadBalancer would still use the default domain.

This does not effect configuration using "server"/"servers".

Follow-up to 6a8943d8c5.

Bug: T212129
Change-Id: I5b20c9d3f153650d6e03581fd523cba4961f664f
2022-01-31 19:14:04 +00:00
Siddharth VP
804dcac8a3 Fix typos in comments (S)
Change-Id: Id86769ca81ff65707edde520585aa9ea117123e1
2022-01-09 23:28:53 +05:30
Umherirrender
fe7bd754ee objectcache: Change variable name from master to primary
Change-Id: I3574ea348f32ccd289a0b4158fe93b5cb48ddffa
2021-10-14 19:40:01 +00:00
Umherirrender
2e4ee47c3d Cleanup mixed space/tab line indent
Change-Id: I833052a656b1ce419c0929f6f0514f2a33c2c4cc
2021-09-04 00:52:31 +02:00
Aaron Schulz
7e12163708 objectcache: simplify BagOStuff::ATTR_* flags and set them for backends
* Remove unused ATTR_SYNCWRITES/QOS_SYNCWRITES_* constants
* Remove unused ATTR_LOCALITY/QOS_LOCALITY_* constants

Bug: T279977
Change-Id: I5ada228fbd504f8e94833df8a8414ed8da0bce4b
2021-08-31 11:23:28 -07:00
jenkins-bot
5f83cbffa4 Merge "objectcache: call silenceTransactionProfiler() in SqlBagOStuff::fetchBlobs()" 2021-08-28 23:16:06 +00:00