Commit graph

214 commits

Author SHA1 Message Date
jenkins-bot
78a2b8f485 Merge "Fix new phan errors, part 5" 2019-10-20 01:39:44 +00:00
Daimona Eaytoy
d3252fc870 Fix new phan errors, part 5
Some of the errors are suppressed because they're phan false positives.
The idea behind this is that they'll be fixed in a future version of
phan, and we'll just have to remove the suppressions.

Note: I'm disabling UnusedPluginSuppression so that we can start suppressing
issues even if they're still disabled. The sniff should be re-enabled
as soon as we upgrade phan.

Bug: T231636
Change-Id: I0f7fa06a9e03fbb86c7a5eb6e50a850bb258a7f7
2019-10-18 20:45:53 +00:00
daniel
e1f8a8148d SqlBlobStore HOT FIX: remove caching from getBlobBatch
This is a TEMPORARY HACK to fix a problem with cached values
getting mixed up when using getBlobBatch(). As of the creation of this
patch, the actual cause is unknown. Caching should be put back once the
cause of the problem has been found.

Note that getBlobBatch is only used in maintenance scripts.
Removal of caching in that method has no effect on web requests.

getBlobBatch() was introduced in I56306c50a661 and is used by
RevisionStore::getContentBlobsForBatch(), which in turn is needed
by the Translate extension to remove its dependency on the deprecated
rev_text_id field.

Bug: T235188
Change-Id: I94c6f9ba7b9caeebaa9b055916f15f7bbdcd8fb6
2019-10-16 23:02:52 +00:00
Max Semenik
8a98dd9d59 Convert some private static arrays to constants
Remove @since for some private ones as we don't guarantee anything
about private class members.

Change-Id: Ifb898353c02082e9ef69d67f69339345c6cd154d
2019-10-16 01:30:54 +00:00
jenkins-bot
b02b2704b9 Merge "NameTableStore: ensure consistency upon rollback." 2019-10-10 15:40:17 +00:00
daniel
135673b98e NameTableStore: ensure consistency upon rollback.
This ensures consistent behavior when an ID for a name is first acquired
within a transaction that is rolled back. The documentation for acquireId
now reads:

@note If called within a transaction, there is a chance for the acquired ID to be lost
if the transaction is rolled back. A best effort is made to re-insert the mapping
after a rollback, and consistency of the cache with the database table is ensured
by re-loading the map after a failed transaction. However, there is no guarantee
that an ID returned by this method is valid outside the transaction in which it
was produced. This means that calling code should not retain the return value beyond
the scope of a transaction, but rather call acquireId() again after the transaction
is complete. In some rare cases, this may produce an ID different from the first call.

Bug: T224949
Change-Id: I6d05e4112a649675bfb9083cab2d1bbe394e65b3
2019-10-10 12:03:29 +02:00
James D. Forrester
4afe9030c2 Services: Convert BlobStoreFactory's static to a const now HHVM is gone
Change-Id: Ia319c881f0457fde71e355eca4873ac157e605d3
2019-10-08 11:27:22 -07:00
Timo Tijhof
01095f8075 Storage: SqlBlobStore no longer needs Lanugage object
Constructing a Language object in order to initialize the
BlobStoreFactory service causes a circular dependency
(see T231866).

SqlBlobStore was using the Language object to all iconv.
But nothing language specific is done in Language::iconv,
so we can just inline the call.

Bug: T231866
Change-Id: I90c25decbcff10ea762a2c7474a12fd2041b3abc
2019-10-02 09:26:34 -07:00
Daimona Eaytoy
e5444ea55a docs: Avoid some scalar juggling
Phan can treat scalar types as non-interchangeable with
`scalar_implicit_cast` set to false. This patch fixes some of those
issues (which are in total >1000), namely the ones with alphabetic order
< includes/actions.

Change-Id: Ib1c6573ab899088bc319b9da9ceaffc850da3dbe
2019-09-19 17:21:24 +00:00
Timo Tijhof
fe3c418925 Storage: Use a more descriptive WAN cache keyclass in 'SqlBlobStore'
In order to make statistics from WANObjectCache more useful,
each kind of key should generally have a unique first make*Key
component. This usually corresponds 1:1 to a call to make*Key,
unless the same key format is duplicated in muliple places.

In this case, change ('BlobStore', 'address', …, …)
to ('SqlBlobStore-blob', …, …).

Bug: T197849
Change-Id: Iba3ae1bf947d6f56c72df2cf3f2160d458d38c66
2019-09-14 02:22:01 +00:00
Petr Pchelko
7b4eafda0d Replace usages of deprecated User::isAllowed. Step 2.
Bug: T220191
Depends-On: I509cd88355cfd0cc074f4135ec86399b4075f89f
Change-Id: I2341e6f1d108c92924620fba3c64ec090cb09918
2019-09-11 16:00:03 -07:00
Daimona Eaytoy
c659bc6308 Unsuppress another phan issue (part 7)
Bug: T231636
Depends-On: I2cd24e73726394e3200a570c45d5e86b6849bfa9
Depends-On: I4fa3e6aad872434ca397325ed7a83f94973661d0
Change-Id: Ie6233561de78457cae5e4e44e220feec2d1272d8
2019-09-03 17:19:21 +00:00
Petr Pchelko
fa36773e58 Introduce BlobStore::getBlobBatch method
Bug: T230834
Change-Id: I56306c50a6617dc91e4eb362ef010703ac25d951
2019-08-30 09:40:49 +00:00
jenkins-bot
358348d9bf Merge "rdbms: add setTempTablesOnlyMode() to suppress CONN_TRX_AUTOCOMMIT during tests" 2019-08-25 15:02:48 +00:00
Aaron Schulz
b1b01d8e3c Make NameTableStore use LoadBalancer::getConnectionRef()
This method handles reuseConnection() calls, disallows writes to
DB_REPLICA role handles, is more consistent with wfGetDB(), and
is more efficient with I540b08920997c5.

Change-Id: I8edaef0ea4718414e2004fad04afca8ec5797669
2019-08-25 01:24:16 +00:00
Aaron Schulz
0b8a4daaa4 rdbms: add setTempTablesOnlyMode() to suppress CONN_TRX_AUTOCOMMIT during tests
Bug: T202116
Change-Id: Ib3e80eb04580b2d5a75ede8fb5546409c09a751a
2019-08-19 18:47:21 -04:00
jenkins-bot
e09547f59b Merge "Remove title protection correctly for undeletions and imports" 2019-08-17 09:27:34 +00:00
Petr Pchelko
e3ecfe4544 Check all slots in DerivedPadeDataUpdater::isCountable.
For example when the page is moved, the slots are not actually
modified, so countability of the page after the update should
not be based only on modified slots, but instead on all the page
slots.

Bug: T212706
Change-Id: Ie352ec9307b89b04af09ed85b542ca2907856bbf
2019-08-16 09:55:01 -07:00
GeoffreyT2000
00104e2c0e Remove title protection correctly for undeletions and imports
Undeletions and imports can both produce "new" pages with more than
1 revision. The fact that $options['created'] for the associated
DerivedPageDataUpdater instance is unexpectedly changed to false is due
to the fact that the latest revision often has a nonzero rev_parent_id.
This causes WikiPage::onArticleCreate not to be applied correctly, and
so title protection is not correctly removed. The part about "created"
being forced is therefore being fixed.

Bug: T200088
Change-Id: I8c899893486976285eb6fac87d641e93a616676d
2019-08-16 14:13:48 +00:00
David Causse
bc0de57e75 Simplify SearchUpdate constructor and hard deprecate some param types
Change-Id: I5677041169402014f1afc1a9012460c760ca24b6
2019-08-01 17:24:42 +02:00
jenkins-bot
22f137ced8 Merge "Convert LoadBalancer::getConnection() callers to LoadBalancer::getConnectionRef()" 2019-07-26 14:02:29 +00:00
Aaron Schulz
5099ee9f72 parser: add speculative page IDs to use with {{PAGEID}}
This works similarly to speculative rev IDs with {{REVISIONID}}.
Re-parses can be avoided if the page ID is correctly guessed.

Also make the {{PAGEID:X}} parser function set vary-page-id.

Bug: T226785
Change-Id: I0b19be45e6ddd6cde330bfcd09d243e4e5beda01
2019-07-26 16:41:00 +10:00
Aaron Schulz
83e294a892 Convert LoadBalancer::getConnection() callers to LoadBalancer::getConnectionRef()
Change-Id: Ie1ae322202105ce093779e6d0b00f335c71be31f
2019-07-21 01:02:49 -07:00
jenkins-bot
3e74b6eeaa Merge "parser: add vary-revision-sha1 and related ParserOutput methods" 2019-07-17 23:57:17 +00:00
Aaron Schulz
dd6ed7840f parser: add vary-revision-sha1 and related ParserOutput methods
This can be used to avoid double parsed on save if the prior output
can be reused in-spite of involving a self content reference.

Change-Id: Idcd30a3fa3f7012dac76ce8bbf46625453ae331f
2019-07-17 05:12:18 +00:00
Aaron Schulz
f72ae0f6e6 Switch various LoadBalancer::getConnection() callers to getConnectionRef()
This is the preferred method as it enforces read-only mode for DB_REPLICA
and handles LoadBalancer::reuseConnection() calls automatically.

Change-Id: Iab9439ba8e0810fa14c302661ed7a3534f6bfc0d
2019-07-12 10:56:30 -07:00
jenkins-bot
1cc50ab8f5 Merge "Migrate various callers away from wfWikiId() to WikiMap" 2019-07-12 17:33:51 +00:00
jenkins-bot
0d076eae58 Merge "Some fixes to page updater docs" 2019-07-12 08:40:28 +00:00
Aaron Schulz
7710afe569 Migrate various callers away from wfWikiId() to WikiMap
Change-Id: Idc2980c05873c428fb5ffa6ab89f723d33217a8e
2019-07-12 01:02:00 +00:00
jenkins-bot
316509d908 Merge "Various fixes and simplifications to RefreshLinksJob::runTitle()" 2019-07-11 06:19:24 +00:00
Aaron Schulz
f588586e16 Various fixes and simplifications to RefreshLinksJob::runTitle()
* Remove logic for saving slow-to-render parser output. This has
  not worked ever since DerivedPageDataUpdater was introduced.
* Make the logic to use cached output actually work. This was
  also broken since DerivedPageDataUpdater was added. In order
  to pass the output, add a known-revision-output parameter
  to both WikiPage::doSecondaryUpdates() and
  DerivedPageDataUpdater::prepareUpdate().
* Also factored out some helper methods from runForTitle() in
  RefreshLinksJob to make it more readable and avoid the need
  for multiple transaction round commit calls. This makes the
  case of multiple-title jobs less likely to break again.
* Make use of RefreshLinksJob::runForTitle() return value.
* Add unit tests for multiple-title job case.

Change-Id: I0cd13c424a87653b5a7253c42cd48fe43befd692
2019-07-11 06:06:02 +00:00
Tim Starling
dda7a2c3ed Some fixes to page updater docs
A couple of typos, and also remove a thing that says isDeleted() should
be skipped for the current revision -- isDeleted() does not do a DB
query so there's no performance rationale for spreading assumptions and
increasing complexity in that way.

Change-Id: I661a6889618f188a9aa72f2f9a2c2ebf69ba09c4
2019-07-11 15:59:56 +10:00
Aaron Schulz
baafb5adb4 Make ExternalStore wrap ExternalStoreFactory and create access class
* Inject settings and global instances as dependencies to the
  ExternalStoreMedium instances. This includes the local wiki
  domain, so that wfWikiId() calls are not scattered around.
* Create ExternalStoreAccess service for read/write logic.
* Deprecate the ExternalStore wrapper methods.
* Add some exception cases for bogus store URLs are used instead
  of just giving PHP warnings and failing later.
* Make moveToExternal.php require the type/protocol to decide
  which ExternalStoreMedium to use instead of assuming "DB".
* Convert logging calls to use LoggerInterface.

Change-Id: I40c3b5534fc8a31116c4c5eb64ee6e4903a6197a
2019-06-28 14:31:44 -07:00
Aaron Schulz
a9a13a1266 storage: rename various $wikiId fields/parameters to $dbDomain
These fields are passed to methods like LoadBalancer::getConnection() and are
already expected to be DB domains. Update various comments as well.

Fix a few minor IDEA warnings.

Change-Id: If480cc4d7d4a3c2d176ab346e6307e80cd2540c5
2019-06-28 00:57:04 +00:00
jenkins-bot
20e65a1915 Merge "parser: inject the time for {{REVISIONTIMESTAMP}} on pre-save parse" 2019-06-23 22:13:48 +00:00
jenkins-bot
7b3631abc8 Merge "Storage: Type against ILBFactory and ILoadBalancer in storage classes" 2019-06-14 00:05:03 +00:00
Umherirrender
29cc0aa0f9 Storage: Type against ILBFactory and ILoadBalancer in storage classes
Instead of the LBFactory/LoadBalancer implementations.

Change-Id: Ia58d9056bd60f3f094fbb16f7b9afa64e11143c8
2019-06-13 22:10:58 +00:00
Daimona Eaytoy
f5eaf6470b externalstore: Fix insert*() return docs and remove redundant checks
ExternalStorage::insertWithFallback is reported to return false on
failure, but it doesn't. It has a single exit point, and return value is
checked with strlen(), so actually it can only return the URL or throw.
Thus, update any related doc and remove a redundant check from code
calling insertToDefault.

Change-Id: Ic95c3aed19118b987aef105f8077d55558f39127
2019-06-13 18:23:23 +00:00
jenkins-bot
699920cc66 Merge "objectcache: add object segmentation support to BagOStuff" 2019-06-13 18:19:17 +00:00
Aaron Schulz
b09b3980f9 objectcache: add object segmentation support to BagOStuff
Use it for ApiStashEdit so that large PaserOutput can be stored.

Add flag to allow for value segmentation on set() in BagOStuff.
Also add a flag for immediate deletion of segments on delete().

BagOStuff now has base serialize()/unserialize() methods.

Bug: T204742
Change-Id: I0667a02612526d8ddfd91d5de48b6faa78bd1ab5
2019-06-11 16:14:17 +01:00
Aaron Schulz
e85fe191c9 parser: inject the time for {{REVISIONTIMESTAMP}} on pre-save parse
DerivedPageDataUpdater::prepareContent already locks in the revision
timestamp before insertion, so inject that into the parser options
used for any pre-save parse (e.g for edit filters).

This means that a reparse is no longer needed within in the same save
request to get the post-save canonical output. A parse will still be
required if the edit filter output used an edit stash output, since
the revision timestamp is not set at stash time.

Instead of using vary-revision, add a vary-revision-timestamp flag
for the revision timestamp words. The month/day/hour variants retain
their prior optimizations for allowing edit stash output reuse for
the post-save canonical output.

Change-Id: Ic2c13db4d21197c79a89de0de56745ca32918eb6
2019-06-09 13:12:57 +01:00
Aaron Schulz
3496f0fca3 Make sure that each DataUpdate still has outer transaction scope
Bug: T221577
Change-Id: I620e461d791416ca37fa9ca4fca501e28d778cf5
2019-05-30 20:53:18 +00:00
Aaron Schulz
bde1c2e28b Cleanup DerivedPageDataUpdater::doSecondaryDataUpdates and remove redundant caller option
Change-Id: I8bbc6cd76d035635d47d1158df34c625879a8cc3
2019-05-24 14:01:23 -07:00
Aryeh Gregor
18ec468633 Don't pass Config to service constructors
We don't want to depend on the entire site configuration when we only
need a few specific settings.

This change additionally means that these services no longer see a live
version of the settings, but rather a copy. This means in tests you
really do have to call overrideMwServices() if you want services to pick
up your config changes.

ResourceLoader and SearchEngineConfig will need more work to port,
because they expose their member Config in a getter, and the getter is
actually used.

Parser and NamespaceInfo are also relatively complicated, so I split
them into separate patches.

Tested with 100% code coverage. \o/

Depends-On: If6534b18f6657ec1aba7327463f2661037f995b3
Change-Id: I1a3f358e8659b49de4502dc8216ecb6f35f4e02a
2019-05-02 11:33:56 +03:00
jenkins-bot
7a658df6ad Merge "Alter two uses of hasOrMadeRecentMasterChanges() for consistency" 2019-04-23 21:31:38 +00:00
jenkins-bot
0ffc58d935 Merge "Avoid pre-send parse in DerivedPageDataUpdater due to isCountable()" 2019-04-23 20:54:03 +00:00
Aaron Schulz
7a69b598f2 Alter two uses of hasOrMadeRecentMasterChanges() for consistency
Rather than have the behavior vary and possibly break code or tests
when small changes happen, make User/NameTableStore more explicit
about when cache key purges happens.

This should reduce problems with certain fragile tests, such as those
that could be affected by 0390811263 when --use-normal tables is not
used. Ideally, any fragility should be ironed out of effected code.

Change-Id: Ibe5d1bb4bece2526bc0da99648f7ba73bdc0ffa5
2019-04-23 11:34:09 -07:00
Aaron Schulz
9de4bc9df1 editstash: segregate stats by content type for more useful graphing
Change-Id: Ice590a163ab7790993e44995220ca093f90c7c4f
2019-04-20 00:28:04 +00:00
Aaron Schulz
6cd6b43c31 Avoid pre-send parse in DerivedPageDataUpdater due to isCountable()
Bug: T216306
Change-Id: I08087f1019049fb8f31fed0d1e757f4df8733a05
2019-04-19 17:08:49 -07:00
Aaron Schulz
2859306684 Refactor edit stashing into a PageEditStash service
Additional code cleanup:
* Call setCacheTime() in parseAndStash instead of relying
  on the one in DerivedPageDataUpdater.
* Improve the SPI logging by adding more extra fields.
* Treat requests in CLI/job mode (aside from tests) like
  those from bots with regard to checking the stash.
  This should avoid stats/logging pollution.

Change-Id: I8c6be919e399378e401a60502add0ecec7764d2d
2019-04-17 12:53:03 -07:00