Commit graph

535 commits

Author SHA1 Message Date
Thiemo Kreuz
b4c63c64ae Remove some more comments that literally repeat the code
Nothing to learn from these.

You can find a longer explanation in the comments in I93751e6.

Change-Id: I195aae70fc282b58be5b18160783f27d38605d15
2021-12-09 19:01:36 +01:00
Ppchelko
643fc535c3 Reapply "Move limit report rendering to ParserOutput"
This reverts commit 2bcb3fe567.

Reason for revert: this is a good change,
just needed more work to not break CI

Change-Id: I23768bee242e3cf81b1493a740cf070e7ad1e224
2021-11-09 11:08:08 -08:00
Ppchelko
2bcb3fe567 Revert "Move limit report rendering to ParserOutput"
This reverts commit 89028e0b8e.

Reason for revert: Temporary until we deal with T295357

Change-Id: I556de18dbf900a9bc58d5ae22d1bf194682d0840
2021-11-09 15:57:18 +00:00
Petr Pchelko
89028e0b8e Move limit report rendering to ParserOutput
This does not move the actual limit report data into
ParserOptions yet, that should be done separately
given that it will require serialization changes.
Let's get this change settled first before messing
with serialization.

This unifies canonical and non-canonical ParserOptions,
so ParserCache can now be used with both. It is hard
to say how this will affect the ParserCache capacity,
so we should monitor it after releasing this.

Change-Id: I154c0a77a5b0287b5572614d56339fb57ac56c33
2021-11-08 12:45:41 -08:00
jdlrobson
24949480eb Give skins more flexibility over table of contents render
* Do not store table of contents in parser output
* Instead inject table of contents via strpos where needed
  inside Article based on Skin "toc" option
* Use <mw:tocplace> as a TOC placeholder; for Parsoid compatibility
  this will be replaced with a <meta> tag in a followup patch.

Bug: T287767
Change-Id: I44045b3b9e78e7ab793da3f37e3c0dbc91cd7d39
2021-10-25 22:26:41 +00:00
Umherirrender
b581e6d40e Hide deprecation on tests for ParserOutput::addWarning
Change-Id: Ifc5f3ef93720a944f0e0fffc1666047b46f1683b
2021-10-20 21:32:49 +02:00
C. Scott Ananian
4834340ec0 Deprecate ParserOutput::addWarning() in favor of ::addWarningMsg()
Encourage localization and factor out common code by taking a message
key as the first argument to ::addWarningMsg() instead of a wikitext
string.  This also plays nicer with Parsoid by separating out the
localization code from the parse.

Bug: T293515
Change-Id: I6a7c04c67ac586ab00d4edcbb3d09485a7794e23
2021-10-15 16:06:13 -04:00
C. Scott Ananian
6bf78153bf Hard deprecate ParserOutput::addTrackingCategory()
Non-WMF-deployed patches:
I403b646852064fed22e8e9bbfadd221990363881 (CollaborationKit)

Code search:
https://codesearch.wmcloud.org/search/?q=%5BOo%5Dutput%28%5C%28%5C%29%29%3F-%3EaddTrackingCategory%5C%28&i=nope&files=&excludeFiles=&repos=

Depends-On: I339a14907c8059c52ffce5d8a554e97f1f0b58d6
Depends-On: Ie6a09796b908dd105ec17d5a1f0f9586eed9bd8c
Depends-On: I983d370a03413086acafe7e67a2b812400e1a38d
Depends-On: Ie41ad70f1fb5de202355314a96a1243234fb267c
Depends-On: I715f788c5026a6c07d4995980a4fcc6c8b84e2b4
Depends-On: I1330a806febf898b3a0a2998fac857f0f737a0a3
Change-Id: I340c138f26e25ce971a46470a4da02d011b5a638
2021-10-15 14:18:26 -04:00
C. Scott Ananian
9632dfb041 Move ::addTrackingCategory() implementation to TrackingCategories
This moves the implementation of ParserOutput::addTrackingCategory()
to the TrackingCategories class as a non-static method. This makes
invocation from ParserOutput awkward, but when invoking as
Parser::addTrackingCategory() all the necessary services are
available.  As a result, we've also soft-deprecated
ParserOutput::addTrackingCategory(); new users should use the
TrackingCategories::addTrackingCategory() method, or else
Parser::addTrackingCategory() if the parser object is available.

The Parser class is already kind of bloated as it is (alas), but there
aren't too many callsites which invoke
ParserOutput::addTrackingCategory() and don't have the corresponding
Parser object handy; see:

https://codesearch.wmcloud.org/search/?q=%5BOo%5Dutput%28%5C%28%5C%29%29%3F-%3EaddTrackingCategory%5C%28&i=nope&files=&excludeFiles=&repos=

Change-Id: I697ce188a912e445a6a748121575548e79aabac6
2021-10-15 14:17:58 -04:00
C. Scott Ananian
5ae946d3a6 Rename ParserOutput::setCategoryLinks() and ::getCategoryLinks()
Make ::setCategory() consistent with the corresponding singular method,
which is ::addCategory(), not ::addCategoryLink().  Also, don't return
a value.

This renaming is in preparation for factoring out a write-only base
class from ParserOutput suitable to be used by Parsoid.

Note that OutputPage does distinguish a 'category link' from a
'category list', and there are separate OutputPage::getCategories()
and OutputPage::getCategoryLinks() methods.  However, the category
map in ParserOutput isn't exactly the same as either of these:
it's actually a map (or list of pairs) of category name to sort key.

Rename ParserOutput::getCategoryLinks() to ::getCategoryNames()
in order to clarify that the concept involved is not the same as
the OutputPage "category links" methods.

Code search:
https://codesearch.wmcloud.org/deployed/?q=-%3E(get%7Cset)CategoryLinks%5C(&i=nope&files=&excludeFiles=&repos=

(Note that many of the code search matches are for the methods in
OutputPage, which we are trying to disambiguate here.)

Bug: T287216
Change-Id: Idb383d3d9ef7b76f8a0208a057a3cb8c639465c9
2021-10-15 09:45:36 -07:00
jenkins-bot
8ab400e920 Merge "Sanitizer: Replace RFC 3454 by RFC 8264 for clearUrl" 2021-10-14 16:15:44 +00:00
jenkins-bot
388e5f200c Merge "Sanitizer: Use \u{xxxx} syntax in cleanUrl" 2021-10-14 16:07:43 +00:00
jenkins-bot
2504a3bb68 Merge "ParserOutput: remove Title from public interface" 2021-10-14 16:02:28 +00:00
daniel
845b36c7bb ParserOutput: remove Title from public interface
Usages of Title in the public interface of ParsrOutput are being
replaced with LinkTarget or PageReference, as appropriate.

Change-Id: I571cf18fd7e666a59ef3947b09c2e75357bcac04
2021-10-14 10:46:53 +00:00
C. Scott Ananian
3056737420 Make Parser::$mStripState private
This property was deprecated in 1.35.  The replacement function
Parser::getStripState() was introduced in MediaWiki 1.34.

Code search:
https://codesearch.wmcloud.org/search/?q=-%3EmStripState&i=nope&files=&excludeFiles=&repos=

Depends-On clauses below are for WMF-deployed code.  Other uses in
non-WMF-deployed code have been patched in:
* https://github.com/SemanticMediaWiki/SemanticMediaWiki/pull/4936
* Idc2fadf5105d6eb30777a16dff0035bceff17174 (BlueSpiceSocial)
* I130fd61a8fe2d28e6b116a3fcc767b8abd466cea (ContributionScores)
* I3676fe9882ce9de5732cb7230528134df544ff98 (HierarchyBuilder)
* Ic392afd1e93ae0003fd0ab65114ec1ff38bb2927 (Mpdf)
* I4b01017da752def982777c4fea5fad5e21e4c7ea (MsLinks)
* I09726078ee62eb99e032b8faa5f938e20107f48c (Negref)
* Ibfd6b7064a8e650c3492e0d2764d4f7afc4937bf (PageForms)
* Ia865435688d36178508f21cffae79538c919035c (PageInCat)
* Ib94db0e6d365e4cb3f51121340a04d31b88add62 (ParserFun)
* I8660c0691b7e9842106d7dcb224ff5ecf374e4bc (PhpTags)
* I1ad5f78e5a937767123400ceca4967941e256e5e (RandomImageByCategory)
* I4539e7cea597f71b2a2d9a6cae137bc25085ed6b (ReplaceSet)
* If8ff2e21952b3f08d3a8950d42e2afb56973fb89 (SemanticDrilldown)
* I4a5bd64760cdde5b614a7d4e2b09e8d0634b2056 (SemanticPageSeries)
* Ia04f1aac1d8ae4ea16c98cfbbe72195fffe653b6 (SemanticRating)
* Id2a2e2d024922e3babf756ebae1a4f59b4358146 (Spark)
* I4a979024b18ec4834dc06b51ee0f018d749c6dab (Tooltip)
* Iaf179914863998b32bfecc16c874c3cffd6c26e9 (VIKI)
* I2de0e7a6c133c2e1f3cb7502a81d809c4489db4c (XSL)
* https://gitlab.com/hydrawiki/extensions/characterescapes/-/merge_requests/1 (characterescapes)
* https://github.com/JeroenDeDauw/Validator/pull/38 (Validator)
* https://github.com/lingua-libre/CustomSubtitle/pull/3 (CustomSubtitle)
* https://github.com/mkroetzsch/AutoCreatePage/pull/12 (AutoCreatePage)
* https://gitlab.com/nonsensopedia/extensions/advancedbacklinks/-/merge_requests/95 (advancedbacklinks)
* https://github.com/vonloxley/Shariff-Mediawiki/pull/16 (Shariff-Mediawiki)

Bug: T275160
Depends-On: I062ac8b69756a7ad35d8cc744b4735fd2e70f13e
Depends-On: Ic4be2bad176f2c59a1104219be10045cd5929261
Depends-On: I3cb117a91c8c57331b6b513f64ddb68d6ae2758c
Depends-On: I67b5926f2f851b3dc709d044eec5dd3df5065482
Depends-On: I7806068e1cd6e4da66adfe7bb75095d4bfb5d6bc
Depends-On: I429da35ca4e276c852b8d6ee102ff19f742c22c0
Change-Id: I4af85a46cfcafba15aa5ee50fda9f7b04681d6e6
2021-10-13 19:58:50 -04:00
Petr Pchelko
a1aa3e0827 Hard-deprecate all public property access on CacheTime and ParserOutput.
- Added a test where ParserOutput objects with CacheTime
properties set are unserialized from previous versions.
- Generate new serialization tests for 1.38

Now all serialization in production is JSON, so changing
property visibility shouldn't affect ParserCache.

Bug: T263851
Depends-On: I283340ff559420ceee8f286ba3ef202c01206a23
Change-Id: I70d6feb1c995a0a0f763b21261141ae8ee6dc570
2021-10-13 13:27:16 -04:00
C. Scott Ananian
af5d13c5de Rename ParserOutput::{get,set,unset}Property to {get,set,unset}PageProperty
The ::getProperty() naming is too generic and doesn't clearly indicate
that these are "page properties" (which have their own table in the DB).
As part of refactoring a clean API out of ParserOutput which can be used
by Parsoid, clean up the naming here.

Soft-deprecation in this patch, there are a handful of external users
which need to be cleaned up before we hard-deprecate.

Bug: T287216
Change-Id: Ie963eea5aa0f0e984ced7c4dfa0fd65d57313cfa
2021-10-08 10:07:17 -04:00
jenkins-bot
e5b8997536 Merge "Remove "auto-number headings" preference" 2021-10-05 19:18:40 +00:00
Amir Sarabadani
649bbdd6c5 Remove "auto-number headings" preference
Bug: T284921
Change-Id: Ic9ed88f419419cf4cc5cc32010539eea8b76314b
2021-10-03 00:47:08 +02:00
C. Scott Ananian
df3cc40fac Rename ParserOutput::{allow,prevent}Clickjacking() -> ::{get,set}PreventClickjacking()
This name is consist with the rest of the setter and getter methods
in ParserOutput.  Renamed the methods in OutputPage, ImageHistoryList,
ImageHistoryPseudoPager, and ContribsPager as well for consistency;
it also makes chasing down lingering references in codesearch easier.

Soft-deprecated the old name for 1.38.  Hard-deprecation will follow,
but there are a number of users in production that should be chased
down first.

Code search:

https://codesearch.https://codesearch.wmcloud.org/deployed/?q=(allow%7Cprevent)Clickjacking&i=nope&files=&excludeFiles=&repos=

Bug: T287216
Change-Id: I9822c60c180d204bd30cb4447a1120155d456da4
2021-10-01 14:13:47 -04:00
C. Scott Ananian
db81b56adf Rename ParserOutput::hideNewSection() -> ::setHideNewSection()
This name is consist with the rest of the setter and getter methods
in ParserOutput (note that ParserOutput::getHideNewSection() already
exists and is consistently named).

Hard deprecated the old name for 1.38.  Rarely used outside core, and
a pull request already created for the one outside user:
https://github.com/SkizNet/mediawiki-WikiMirror/pull/15

Code search:
https://codesearch.wmcloud.org/search/?q=hideNewSection&i=nope&files=&excludeFiles=&repos=

Bug: T287216
Change-Id: Ia553373eef78f875a83ad0eebfe2e465ce33272f
2021-09-29 17:47:54 -04:00
jenkins-bot
eaf10f600b Merge "Expand local URLs to absolute URLs in ParserOutput" 2021-09-27 14:48:02 +00:00
Alexander Vorwerk
04dfdc3653 Hard deprecate User::setOption()
deprecated since 1.35

Bug: T277818
Change-Id: Ic251d624e5d6fa857aa92f9c5dd3df44714ac610
2021-09-26 17:18:54 +02:00
Petr Pchelko
d334de960a Expand local URLs to absolute URLs in ParserOutput
New option 'absoluteURLs' was added to getText method
of the ParserOutput object that replaces all links
in the page HTML with absolute URLs.

Removing the action=render special case from Title
seems safe cause we will end up replacing the result
with absolute URL if we're in a render action no matter
where Title::getLocalUrl was called from.

This change is safely revertable from the perspective
of ParserCache.

Bug: T263581
Change-Id: Id660e1026192f40181587199d3418568f0fdb6d3
2021-09-23 11:48:51 -07:00
Umherirrender
2e4ee47c3d Cleanup mixed space/tab line indent
Change-Id: I833052a656b1ce419c0929f6f0514f2a33c2c4cc
2021-09-04 00:52:31 +02:00
Fomafix
4706b9bfb5 Sanitizer: Replace RFC 3454 by RFC 8264 for clearUrl
RFC 3454 is obsoleted by RFC 7564 obsoleted by RFC 8264.

The mapping section 3.1 "Commonly mapped to nothing"
https://datatracker.ietf.org/doc/html/rfc3454#section-3.1
was replaced by section 9.13. "PrecisIgnorableProperties (M)"
https://datatracker.ietf.org/doc/html/rfc8264#section-9.13
which uses the Default_Ignorable_Code_Point from
https://www.unicode.org/Public/UCD/latest/ucd/DerivedCoreProperties.txt

U+1806 (MONGOLIAN TODO SOFT HYPHEN) is not included anymore.

Change-Id: Ib86e619ba9181cdc7f731a96c684a119527e3b76
2021-08-26 21:02:21 +00:00
Fomafix
c2a598d5eb Sanitizer: Use \u{xxxx} syntax in cleanUrl
Also add a simple testcase to verify the handling of these Unicode
characters.

Change-Id: I611a7cf816bb6f6e61b834e48be28943f73e0908
2021-08-26 21:02:21 +00:00
vladshapik
1091f7753f Hard-deprecate Parser::mUser public access, Parser::getUser and ParserOptions::getUser
Bug: T285713
Depends-On: Ie75c9cd66d296ce7cf15432e2093817e18004443
Change-Id: I4297aea3489bb66c98c664da2332584c27793bfa
2021-08-17 15:42:05 +00:00
Petr Pchelko
79475042a5 ParserOptions: support setting a default for lazy options
If the default is not set for a lazy loaded option,
canonical parser output key ends up being 'lazy_option=default',
because when computing the keys we exclude defaults.

If on the other hand we register a default for a lazy option,
it does not get loaded, since in lazyLoadOption we believe
the default is already the loaded value.

Change-Id: I92b3e18fabef4eecac2ec2a4844f1be2716e5d89
Needed-By: I3bce04684070ad306685dabbc51267def25773cd
2021-08-10 10:22:05 -07:00
Umherirrender
d1950d924a parser: Replace deprecated MWHttpRequest::factory
Change-Id: Id5fe298209cfbc09037799a2cdc117c9b7119172
2021-08-04 13:22:50 +00:00
Petr Pchelko
1aa68d183d Remove depecated ParserCache::getKey and ::getEtag
Change-Id: Idea037eaab851110d0c58f537dafcb2153cd2613
2021-07-27 14:47:49 -07:00
DannyS712
cd58ec2a36 Tests: remove unneeded tearDown functions
No need to manually reset MWTimestamp fake time,
MediaWikiTestCaseTrait::fakeTimestampTearDown() does
that after all tests anyway.

Change-Id: Ie45e837c693d7f21b6c7e25471cdaad6f60a3de9
2021-07-27 13:15:40 +00:00
libraryupgrader
5357695270 build: Updating dependencies
composer:
* mediawiki/mediawiki-codesniffer: 36.0.0 → 37.0.0
  The following sniffs now pass and were enabled:
  * Generic.ControlStructures.InlineControlStructure
  * MediaWiki.PHPUnit.AssertCount.NotUsed

npm:
* svgo: 2.3.0 → 2.3.1
  * https://npmjs.com/advisories/1754 (CVE-2021-33587)

Change-Id: I2a9bbee2fecbf7259876d335f565ece4b3622426
2021-07-22 03:36:05 +00:00
DannyS712
29ec3ec7e3 Remove $wgUser fallback in ParserOptions
ParserOptions::__construct() and ::newCanonical()
no longer accept null and fallback to the global
$wgUser - instead, ::__construct() has a typehint
for a UserIdentity, and ::newCanonical() will throw
an exception.

Bug: T284977
Change-Id: I35865e160190582ab10abaa696c6fc6686cc8989
2021-06-24 02:55:20 +00:00
DannyS712
e11c0244dc ParserTest: remove unused $mockConfig
Follow-up: I58f7b3ba1b1d62851b2db71197a8d9129e8d473d
Change-Id: I461dfd088710459ebc07ffec866699a68fa43e08
2021-05-29 01:24:52 +00:00
DannyS712
bea9165e61 Tests: use UserIdentityValue instead of mock User objects
- ParserMethodsTest for creating ParserOptions
- SimpleAntiSpamConstraintTest

Change-Id: I5f8ffbee5ee1ff8e08cae7e6ace30ff7b186539c
2021-05-03 15:41:48 +00:00
James D. Forrester
64898405cb build: Upgrade mediawiki-codesniffer from v35.0.0 to v36.0.0
Change-Id: I8905d0d69738a1cd6997c104080fdf128d315e8b
2021-04-29 13:00:15 -07:00
daniel
4880a82555 Parser: remove Title from method signatures
Bug: T281068
Change-Id: I3280e38dd82d71845c343eeb911e71dd33bb380b
2021-04-29 18:11:46 +02:00
Daimona Eaytoy
535d7abf59 phpunit: Mass-replace setMethods with onlyMethods and adjust
Ended up using
  grep -Prl '\->setMethods\(' . | xargs sed -r -i 's/setMethods\(/onlyMethods\(/g'

special-casing setMethods( null ) -> onlyMethods( [] )

and then manual fix of failing test (from PS2 onwards).

Bug: T278010
Change-Id: I012dca7ae774bb430c1c44d50991ba0b633353f1
2021-04-16 20:15:00 +02:00
jenkins-bot
37ab31298c Merge "Remove $actor field from UsererIdentityValue" 2021-04-14 14:40:12 +00:00
daniel
fed7f0b179 Remove $actor field from UsererIdentityValue
Code that needs to store an actor ID in the database to
represent a UserIdentity, or needs to construct a UserIdentity based on
an actor ID loaded from the database, should use the ActorNormalization
service.

Note: The getActorId() method is removed from the UserIdentity interface,
but all concrete classes continue to support it for now.
UsererIdentityValue::getActorId() is hard deprecated and should
be removed in 1.37. It always returns 0.
User::getActorId() is not deprecated at this point.

Bug: T274179
Depends-On: Id2b3ddf6a2a7cdf90f8936a69148d2cce6fde237
Change-Id: I9925906d11e47efaec3c1f48d5cb3f9896a982c1
2021-04-13 18:18:06 +00:00
Petr Pchelko
d1f481f242 ParserCache: only use in-process caching for metadata
CachedBagOStuff caches negatives, so it breaks PoolCounter.
We only need to cache metadata in-process, since it's commonly
used twice within the request.

Bug: T277829
Change-Id: I11a147c24b6cdb275b521b48802d6f3d0e1a4387
2021-04-06 17:53:38 -06:00
Petr Pchelko
f642215aed Convert ParserCache to PageRecord
ParserOptions not updated cause they depend on Title::getLanguage
implementation.

Tests converted to not require a DB anymore. Can't be proper unit
tests yet due to globals in ParserOptions and fake time hacks,
but exec time does go down from 70 seconds to 9 seconds.

Page content model is still emitted in the metrics since
it was considered useful. Should be removed when we get
something like a page type concept.

Change-Id: Ib16fd0b5b87ffc3cb4d21f4aa43d1203cb7206d2
2021-04-02 21:14:54 -06:00
James D. Forrester
5c83c8952c tests: Avoid problematic language in comments and internal code
We can't avoid this entirely where we're testing upstream features
that are yet to be renamed, but this is a start.

Bug:T277986
Change-Id: I5c1ae717cc008ca147ea31fe35c44b19d1fe5d98
2021-03-28 21:21:28 -07:00
Umherirrender
d01d47683c Fix spacing after yield and use statements
Change-Id: Iacb93e96168ec0cd895130c5c8f66b6b44317e34
2021-03-26 23:55:58 +01:00
Petr Pchelko
7bf51ccef3 Convert ParserOptions to UserIdentity
We still need a lot of refactoring in ParserOptions
constructions, but for now converting the public interface
should be enough.

Change-Id: I04663c39ca037129b827b33555c3f59def5f9b59
2021-03-24 09:40:42 -06:00
DannyS712
24d26873e3 CoreParserFunctionsTest: fix duplication
testSubjectpagename was probably copied from
testTalkpagename and so uses the wrong data provider
and tests the wrong code

Change-Id: Ied4023a8fdd57a1c2b538eca6034c37e94967bc7
2021-03-22 01:48:03 +00:00
Petr Pchelko
0f0a11f6dc Make Parser use UserIdentity instead of User
Change-Id: Idf8578e88af1fd4824f49417a200b16befdbca51
2021-03-17 13:51:52 -06:00
jenkins-bot
76a2acf48a Merge "Parser: initialize preprocessor in constructor" 2021-03-17 16:17:30 +00:00
daniel
afa236ef53 RevisionStore: getRevisionbyId should take a PageIdentity
Providing a PageIdentity to getRevisionById removes the need
to look up page related data, and to construct a Title.

Bug: T275531
Bug: T273284
Depends-On: I8d0dc30ecd0c0c65c337170d271c2e2531914686
Change-Id: I683362ccd669acf2aadf56853afeb1ca6dee6f8c
2021-03-17 13:46:41 +00:00