Commit graph

2250 commits

Author SHA1 Message Date
Petr Pchelko
b964bd2e8d Make ParserOptions::isSafeToCache more robust
Bug: T273120
Change-Id: I516133d07d9ff4d7930de68b88ed4b40ca1c6150
2021-01-27 18:34:26 -06:00
jenkins-bot
d491f23b90 Merge "Respect used options for ParserOptions::isSafeToCache" 2021-01-25 19:13:53 +00:00
jenkins-bot
607c4d3910 Merge "CacheTime: remove mVersion field" 2021-01-25 19:12:05 +00:00
jenkins-bot
e845067ab6 Merge "Adopt pipe trick with Arabic comma" 2021-01-16 03:29:36 +00:00
Niklas Laxström
8d42b76017 Fix @deprecated for Parser::enableOOUI
Change-Id: I0d6937a82413f50a054ba0e3869efa3863a6632b
2021-01-13 15:12:50 +01:00
jenkins-bot
7200e36c9f Merge "parser: convert Preprocessor to WANCache and inject dependencies" 2021-01-12 05:04:47 +00:00
Aaron Schulz
de6eeead21 parser: convert Preprocessor to WANCache and inject dependencies
Make the caching logic use getWithSetCallback() and simplify
the code given that there is only one Preprocessor subclass.
Also, keep the cached values JSON serializable but rely on
the serialization in BagOStuff instead for simplicity.

Add related class constants for injecting preprocessor flags.

Bug: T254608
Change-Id: I72f9f0c0bc352ed5120469090c71294ff0c24999
2021-01-11 20:21:10 -08:00
Ebrahim Byagowi
9fe1d1f734 Adopt pipe trick with Arabic comma
Currently MediaWiki turns `[[test, abc]]` to `[[test, abc|test]]`
while saving the page but that comma isn't in use in Persian
so this patch makes MediaWiki to treat Arabic comma the same way
as regular comma.

Change-Id: Ib8051023abc25b7c4f97a3f50246f35650057ec9
2021-01-11 21:43:33 +00:00
jenkins-bot
a3a3e9f3af Merge "CoreTagHooks: First argument passed to parser tags can be null" 2021-01-07 17:29:28 +00:00
Petr Pchelko
7e8d1a11c8 Return back accidentally removed ParserCache 'hit' metric
Change-Id: Ibd69e532a2f373f9d0129ac2a2c6ac70039c9bec
2021-01-05 14:44:19 -06:00
C. Scott Ananian
a41f284324 CoreTagHooks: First argument passed to parser tags can be null
Document and enforce the correct type for the first argument to
a Parser tag hook, which will be `null` if the tag is self-closed.

Mark the methods in CoreTagHooks @internal.  They are apparently
unused outside MediaWiki core:
  https://codesearch.wmcloud.org/search/?q=CoreTagHooks&i=nope&files=&repos=

Add coverage test cases to ensure that all tag hooks properly handle
the `null` value of the first argument; prior to this patch the
`<html>` tag emitted a broken strip tag in this case.  The other hooks
passed the null to other callees in violation of their type
signatures, but eventually every other hook managed to safely cast the
null to the empty string without throwing an exception or emitting a
warning.  For those, this patch does not change existing behavior---it
just makes the cast to the empty string much more obvious to the
reader.

Change-Id: I69fde6c06eabb2db27bb1cc23d2cb19b99273391
2021-01-05 14:19:44 -05:00
Petr Pchelko
46b66f093a Respect used options for ParserOptions::isSafeToCache
Bug: T269293
Change-Id: Ic3cf908265ad470815f0ac81442d33bde04a5665
2021-01-04 10:32:34 -06:00
Petr Pchelko
2010a912c2 CacheTime: remove mVersion field
Has to be merged at least one train after the parent patch.

Bug: T269393
Change-Id: Iee98950ad0452a50088e755252da8468f0bec1ae
2021-01-04 10:29:52 -06:00
Umherirrender
a30fe542ae build: Enable SecurityCheck-DoubleEscaped and suppress issues
This issue type was globally suppressed in
I849ac4f120fd15b483e8939d4db45c98dc351259 to make reviewer easier.

This adds inline suppressions or @suppress directives on function
docs for false positives, mostly restoring those removed in
I849ac4f120fd15b483e8939d4db45c98dc351259

Bug: T231311
Change-Id: I1b1d814bd907e9d49fcc39f777982936574fc7c6
2020-12-30 23:34:20 +00:00
Umherirrender
e9e784a09e build: Enable phan-taint-check-plugin and suppress issues
Taint check checks for possible security issues by tracking html
escaping and more by using phan.
This slows done the phan-job a bit and requires more ram

Keep the DoubleEscaped issues out to make reviewer easier

Adds suppression for false positives
Adds taint-annotation to help taint-check
Removes suppression for code phan now understand better by the tracking
of keys in taint-check
Fix some small issues by adding int cast or htmlspecialchars calls

Bug: T216348
Bug: T268920
Change-Id: I849ac4f120fd15b483e8939d4db45c98dc351259
2020-12-30 19:02:22 +01:00
jenkins-bot
4257367a01 Merge "Record all used options in metadata." 2020-12-16 17:59:54 +00:00
jenkins-bot
add80561d2 Merge "ParserOptions: Deprecate fallback to $wgUser" 2020-12-16 15:17:05 +00:00
Petr Pchelko
71bb51ed55 ParserCache: general code cleanup, abstracted expiration checks.
Change-Id: I7374f30d582064236b8f782e6a2528eb692e3010
2020-12-16 12:09:55 +00:00
DannyS712
25db391d53 ParserOptions: Deprecate fallback to $wgUser
Emit deprecation notices in the constructor
if falling back to $wgUser, and fix core
calls

Bug: T246861
Depends-On: I51117931d527a3bdda468b48de577a7faafbcd69
Change-Id: Ibd0a8ffd0494c17a378cc43e6b6164166130adf4
2020-12-16 09:03:29 +00:00
jenkins-bot
f0d9e5e182 Merge "Implement caching for old revision HTML endpoint" 2020-12-16 00:41:27 +00:00
daniel
637f630fe9 Implement caching for old revision HTML endpoint
Bug: T269663
Change-Id: I2d17ec37d25f3a6e1c4836c05576bf0fabb7d429
2020-12-15 23:40:08 +01:00
jenkins-bot
b3a8ea7076 Merge "Remove 'stubthreshold' from ParserCache key." 2020-12-15 20:21:14 +00:00
jenkins-bot
ea12ffec23 Merge "Remove ParserCacheFactory::getInstance" 2020-12-15 19:25:04 +00:00
jenkins-bot
c8edeeeaed Merge "CacheTime: Remove the version check." 2020-12-15 19:00:26 +00:00
Petr Pchelko
5413678f06 Record all used options in metadata.
Bug: T269293
Change-Id: I69ac24fcf8bf81de1d25c5a331b2155d9fe21977
2020-12-15 12:51:14 -06:00
jenkins-bot
a10ba8d59e Merge "Implement <langconvert> tag" 2020-12-15 06:18:11 +00:00
David Kamholz
a7ad0547bc Implement <langconvert> tag
The <langconvert> tag takes two attributes: from (language variant from) and to (language variant to). It returns the content of the tag converted using LanguageConverter. It returns an error if the attributes are not present, if the variants do not exist, or if the variants belong to different languages. Currently it does not work for IuConverter, because the variants use the code ike rather than iu, and ike isn't in the list of languages with converters available.

This patchset reimplements from a parser function to a tag, and renames from transliterate to langconvert.

Bug: T263082
Change-Id: Idc3a32c66d5a0466c63e7ce8753d2619354c30b0
2020-12-14 19:40:31 -08:00
Petr Pchelko
b56a3c0d23 CacheTime: Remove the version check.
Fot the sake for forward compatibility, first we need
to remove the check for version in case of a train rollback.

Bug: T269393
Change-Id: Ic7011e20a23e8ad99eb872b050e97baf1ef5c6a2
2020-12-14 19:29:24 -06:00
Petr Pchelko
5a995333fb Remove ParserCacheFactory::getInstance
Added in 1.36, no deprecation process needed.

Bug: T269869
Depends-On: I5d81622d5857bc1d06a3600d8e9554882532714a
Change-Id: I182cb0cb7e4c569a64e2455ca80eda2d1f570608
2020-12-14 19:12:04 -06:00
jenkins-bot
f3e1a5aa22 Merge "api: Use GetLinkColours hook to export CSS classes in page info" 2020-12-15 00:27:48 +00:00
C. Scott Ananian
f9433de05e api: Use GetLinkColours hook to export CSS classes in page info
Several important extensions (Disambiguator, ProofreadPage, and
SoftRedirector) use the GetLinkColours hook to add additional CSS
classes to links on article pages. Parsoid previously relied on
backdoor knowledge of the way Disambiguator used the page property
table to support these, but they should be exported properly from the
API.

Bug: T237538
Change-Id: I945940aa872541d7e01f1e543ca854231c857fe2
2020-12-14 17:51:50 -05:00
daniel
00a3439dce Introduce RevisionOutputCache
Bug: T267981
Change-Id: Ib1dc641ed10d786918362b25bd655780d5844ba1
2020-12-14 16:50:28 +00:00
Umherirrender
f46ca9a63c build: Updating mediawiki/mediawiki-phan-config to 0.10.5
Change-Id: I343d2bae626a3903eb1e67c05bf5caef4314b7dd
2020-12-12 14:42:25 +01:00
Petr Pchelko
235c56d649 Remove 'stubthreshold' from ParserCache key.
Stubthreshold option used to be a cache-varying option,
but in all places where we interact with the ParserCache
we are checking that it's 0 before using the cache.

Instead, we can just remove all the special cases for
stubthreshold option, remove it from cache key and rely
on ParserOptions::isSafeToCache to avoid caching non-default
stubthreshold outputs.

Bug: T264351
Change-Id: Ifaf69a3e651eef21c88da3aa3044b490059958ca
2020-12-07 14:47:05 -06:00
Petr Pchelko
66cc685b45 Make ParserCache use CachedBagOStuff
Bug: T269593
Change-Id: I21e6e39eccad22b781252b142c1e5b079c1ee0b4
2020-12-07 10:28:30 -06:00
Daniel Kinzler
48172f794d Revert "Hard-deprecate all public property access on CacheTime and ParserOutput."
This reverts commit b1a30eb0c4.

Reason for revert: T269396

Bug: T269396
Change-Id: I374ca13ccc30418b8fe3bf98f5090f7643aac4d7
2020-12-04 11:47:55 +00:00
jenkins-bot
e2c4b32c58 Merge "Make ParserCache respect ParserOptions::isSafeToCache" 2020-12-02 20:31:58 +00:00
Petr Pchelko
4417b13d58 Make ParserCache respect ParserOptions::isSafeToCache
Bug: T269154
Change-Id: I8e9ecd2787aa8d172e708ba64ea936e63fbc6b36
2020-12-02 14:02:36 -06:00
jenkins-bot
6278daab01 Merge "ParserOutput: temporary undeprecate getting dynamic properties." 2020-12-02 16:11:47 +00:00
Petr Pchelko
869962e7bb ParserOutput: temporary undeprecate getting dynamic properties.
Apparently I've missed a few instances where dynamic properties
were still written. To fix those we need some grace period for
ParserCache to expire when we would need to have get fallback,
so undeprecate reading dynamic properties.

Bug: T263851
Change-Id: I123605f7b5b907cc1b0ae6f183f3015b41835e8b
2020-12-02 15:35:40 +00:00
Petr Pchelko
7978bfab37 Parser: use setter instead of accessing ParserOutput property
Bug: T269236
Change-Id: I195160d24a5d5bf4eb4cb0890c39c6720e8ec3ac
2020-12-02 08:57:39 -06:00
Ammar Abdulhamid
71571191d4 Chain MutableRevisionRecord method calls 2
Change-Id: I86578cfbc892f171a4e433283b86d1b78fe4167d
2020-11-27 05:26:54 +01:00
C. Scott Ananian
c64e71615e Replace $wgDisable{Lang,Title}Conversion with LanguageConverterFactory methods
Replace direct access to $wgDisableLangConversion with
LanguageConverterFactory::isConversionDisabled(), and replace direct
access to $wgDisableTitleConversion with
LanguageConverterFactory::isTitleConversionDisabled().  However, most
places that check ::isTitleConversionDisabled() actually want
::isLinkConversionDisabled(), so add that too (and deprecate
isTitleConversionDisabled()).

Code search:
https://codesearch.wmcloud.org/search/?q=Disable%28Lang|Title%29Conversion&i=nope&files=&repos=

This change removes a number of spurious dependencies on the global
configuration and reduces code duplication (for example, if the logic
for disabling language conversion were ever to change).

Depends-On: I6fa8230ae97b0e34c381003548e61f9b7387d363
Change-Id: Icc4687638ff1815003dd903854efdbd904854f1e
2020-11-25 12:47:26 -05:00
jenkins-bot
4c9899ea99 Merge "languages: Language::formatNum() should accept any valid number" 2020-11-23 20:54:40 +00:00
C. Scott Ananian
e099c38ef4 languages: Language::formatNum() should accept any valid number
The PHP function is_numeric() returns true for numbers like '123.456'
and even '1.23e45'. However, it returns false for (string)NAN,
(string)INF, and (string)-INF (which are "NAN", "INF" and "-INF"
respectively).  We can return the appropriate unicode characters for
the infinities to localize these/make them universal, and allow a
localization of the "Not a Number" message.

Make the corresponding change to Language::parseFormattedNumber() so
that its remains the inverse operation to ::formatNum().

Accept "NAN"/"INF"/"-INF" only when they stand alone in the string;
in the legacy case where text and numbers are intermingled, split
only on "traditional" numbers; I think we're more likely to find
INF/NAN "innocently" in the middle of text than we are to find it
as a "real" number.

Change-Id: I3ff227a4aac66fc938182dc9fb8a7b743e94faca
2020-11-23 15:20:43 -05:00
Umherirrender
201980999a build: Updating mediawiki/mediawiki-phan-config to 0.10.4
Change-Id: I56538eaa498ab6d312240f9a534c2d2da11c34cb
2020-11-20 17:33:22 +01:00
Petr Pchelko
b1a30eb0c4 Hard-deprecate all public property access on CacheTime and ParserOutput.
Bug: T263851
Change-Id: I3d3ff7b5b6899150df836e445b56896dfd5b887e
2020-11-19 10:12:41 -07:00
Petr Pchelko
b956c77d27 Merge CacheTime and ParserOutput accessedOptions properties
Change-Id: I5785596d68e8923f8bcbd182ace0b1991bd75c9a
2020-11-19 10:12:39 -07:00
Petr Pchelko
dbdc2a3cd3 Introduce JsonCodec to help with serialization/deserialization
Change-Id: I5433090ae8e2b3f2a4590cc404baf838025546ce
2020-11-19 08:32:21 -07:00
jenkins-bot
e6a6592ecf Merge "Fix some unit tests accessing MediaWikiServices" 2020-11-17 18:36:37 +00:00