Provide a way for backend code to determine the primary language of a
ParserOutput, eg for setting the Content-Language header of an API
response.
This is read-only and backed by extension data at the moment for
transition purposes; if this API sticks we'll graduate it to a
"real" property in the future, with appropriate serialization
to/from JSON (T303329).
Similarly, this patch only includes the most basic code to handle
the various ParserOutput merge cases in
ParserOutput::merge{Internal,Html,Tracking}MetaDataFrom(),
ParserOutput::collectMetadata(), and
OutputPage::addParserOutput{Content,Metadata,Text,}(); mostly
inherited from the fact that the storage is backed by extension
data at the moment.
Generally only the "top-level" parser output gets to set the
primary language; we'll presumably need to ensure that the
language is consistent during merge.
Change-Id: I767daba22805a877d9b806fd77334e508902844b
Before 1.39 we used <mw:toc> and in 1.39 we switched to <mw:tocplace/>
(commit 24949480eb). This was changed
to a <meta> tag in 1.40 (commit
0b10563895 and
fa8646ca7b) and the old content has long
since expired from the ParserCache. Clean up the old ParserCache
transition code.
Change-Id: I3254d0acba31e107b50767797a2b0ad28aba59ee
* Rather than computing TOC HTML in Parser and setting it in
ParserOutput, compute it on demand based on section metadata.
This will let Parsoid set section metadata in ParserOutput
and have the TOC generated automatically.
* This required fixing some "bugs" in Linker's generateTOC
which didn't properly close tags and relied on Tidy to fix
up unclosed li and ul tags.
* This patch relies on converting section metadata objects to
array objects, but Linker::generateTOC could be converted to
use TOC data instead.
* Since TOC generation is now moved to getText(), this is done
post-PC load and this eliminates the parser cache split on
user language for TOC heading localization.
Bug: T293513
Change-Id: Ief1bba326d3612b40930440c872a61abadffab10
* ParserOutput::setSections()/::getSections() are expected
to be deprecated. Uses in extensions and skins will need to be
migrated in follow up patches once the new interface has stabilized.
* In the skins code, the metadata is converted back to an array.
Downstream skin TOC consumers will need to be migrated as well
before we can remove the toLegacy() conversion.
* Fixed SerializationTestTrait's validation method
- Not sure if this is overkill but should handle all future
complex objects we might stuff into the ParserCache.
* This patch emits a backward-compatible Sections property in order to
avoid changing the parser cache serialization format. T327439 has
been filed to eventually use the JsonCodec support for object
serialization, but for this initial patch it makes sense to avoid
the need for a concurrent ParserCache format migration by using a
backward-compatible serialization.
* TOCData is nullable because the intent is that
ParserOutput::setTOCData() is MW_MERGE_STRATEGY_WRITE_ONCE; that is,
only the top-level fragment composing a page will set the TOCData.
This will be enforced in the future via wfDeprecated() (T327429),
but again our first patch is as backward-compatible as possible.
Bug: T296025
Depends-On: I1b267d23cf49d147c5379b914531303744481b68
Co-Authored-By: C. Scott Ananian <cananian@wikimedia.org>
Co-Authored-By: Subramanya Sastry <ssastry@wikimedia.org>
Change-Id: I8329864535f0b1dd5f9163868a08d6cb1ffcb78f
Split out from the I44045b3b9e78e change.
This is consistent with what Parsoid will use for the TOC marker.
Bug: T287767
Bug: T270199
Bug: T311502
Depends-On: I1f607cf1ef1b61fb4d2e1880de756fb94d5a6b22
Change-Id: Ie63eed07b9bca1bfa07d4c256aba3728cedd8f93
Split out from the I44045b3b9e78e and Ie63eed07b9bca changes. We
first add code to handle the new tag as well as the old tag in
ParserCache contents. This will allow us to safely rollback if needed
when deploying the follow-on patch which actually changes the tag
used.
Bug: T287767
Bug: T270199
Bug: T311502
Change-Id: Ib3e5e010b9f5ca2c4ea7c4fe28080170b6a88812
Previously:
* It was unclear that generate-html is an optional optimization
* Most of MediaWiki core was doing $parserOutput->setText('') if
html wasn't generated. However this is wrong and will cause
$parserOutput->hasText() to return true and also potentially cause
cache pollution if a content handler both does that and supports
parser cache (Like MassMessage; see T299896)
* The default value of mText in the constructor was '', and most
of the time MW used that default. This doesn't seem right. If
setText() is never called, the ParserOutput should not be considered
to have text
* It was impossible to set mText to null, as $parserOutput->setText(null)
was a no-op. Docs implied you were supposed to do this, so it was very
confusing.
This patch clarifies docs, changes the default value for ParserOutput::$mText
from '' to null, and makes $parserOutput->setText(null) do what you
expect it to. The last two are arguably breaking changes, although
the previous behaviours were unexpected, mostly undocumented and
based on a code search do not appear to be relied on.
It seems like the main reason this only broke MassMessage is most
content handlers either don't support generateHtml, or they don't
support parser cache.
Bug: T306591
Change-Id: I49cdf21411c6b02ac9a221a13393bebe17c7871e
Depends-On: I68ad491735b2df13951399312a4f9c37b63a08fa
The old ParserOutput::getProperty() method returned `false` when a property
was missing. This requires callers to use the `?:` syntax to supply default
values, which then causes any falsey value to be treated as missing.
So, for example, setting the defaultsort to '0' will cause the default
sort to be ignored.
Modern php convention is to use `null` for missing values, and the `??`
syntax is a better/more restrictive alternative to `?:`.
We renamed `ParserOutput::getProperty()` to `::getPageProperty()` in
1.38 (Ie963eea5aa0f0e984ced7c4dfa0fd65d57313cfa/T287216) but kept the
return value convention. Before this actually makes it into a 1.38
release, take the opportunity to fix the return value for the new
`ParserOutput::getPageProperty()` method to return `null` when the
property is missing.
We need to do some temporary workarounds to the places we'd
already swapped over to use the new `::getPageProperty()` method
to allow them to handle either `false` or `null` as a return value;
we'll clean that up once this is merged.
Code search:
https://codesearch.wmcloud.org/deployed/?q=-%3EgetPageProperty%5C%28|T301915&i=nope&files=&excludeFiles=&repos=
Bug: T301915
Depends-On: I3f11ce604970e47b41fc1c123792df8c3045626f
Depends-On: Ie7533f49fe4cad01ebfda29760d23c61e9867b10
Depends-On: Ic5c09f5caa4c897bc553c614fbae9cee159566a2
Depends-On: I0278b2eafd90e77e4fee41c45a1165fb79ddf47e
Depends-On: I383abb6b7dc5e96c0061af13957609f6e31a1065
Depends-On: I79f9f4078e415284af29b15047bafd1c823d7f5b
Depends-On: I02276c48c49f5d2d241a69eb0a6cdf439b572d8b
Depends-On: I71628661b4539a4e35ae32846e719f92bcf782e0
Depends-On: I7e215cb43de0ce150a6bcc00f92481dcdcfed383
Change-Id: Iaa25c390118d2db2b6578cdd558f2defd5351d15
There is a common and reasonable need for longer lines in tests.
The nudge for shorter lines doesn't seem valuable here. The natural
breaks will likely still fall in 80-100 given the enforced practice
for non-test code, e.g. whether through habit, or 80-100 column markers
in text editors, or the finite width of diff and code review
interfaces.
Change-Id: I879479e13551789a67624ce66f0946d2f185e6ee
Soft-deprecate the use of ::setExtensionData() to destructively update
the value stored under a single key. Add the new
::appendExtensionData() method to use where multiple values are
desired. This accomodates the asynchronous and incremental parsing
goals on the Parsoid roadmap.
Bug: T300981
Change-Id: I2dea4ba71ea506428854a9983c1abd906b2efd5f
Deprecate ParserOutput::addJsConfigVars() and add setter methods which
better ensure that the ParserOutput contents are independent of parse
order. This accomodates the asynchronous and incremental parsing goals
on the Parsoid roadmap.
Bug: T300307
Change-Id: I4f08d1098da211f7bf5c43c08c620de224cbf37f
* Do not store table of contents in parser output
* Instead inject table of contents via strpos where needed
inside Article based on Skin "toc" option
* Use <mw:tocplace> as a TOC placeholder; for Parsoid compatibility
this will be replaced with a <meta> tag in a followup patch.
Bug: T287767
Change-Id: I44045b3b9e78e7ab793da3f37e3c0dbc91cd7d39
Encourage localization and factor out common code by taking a message
key as the first argument to ::addWarningMsg() instead of a wikitext
string. This also plays nicer with Parsoid by separating out the
localization code from the parse.
Bug: T293515
Change-Id: I6a7c04c67ac586ab00d4edcbb3d09485a7794e23
Usages of Title in the public interface of ParsrOutput are being
replaced with LinkTarget or PageReference, as appropriate.
Change-Id: I571cf18fd7e666a59ef3947b09c2e75357bcac04
The ::getProperty() naming is too generic and doesn't clearly indicate
that these are "page properties" (which have their own table in the DB).
As part of refactoring a clean API out of ParserOutput which can be used
by Parsoid, clean up the naming here.
Soft-deprecation in this patch, there are a handful of external users
which need to be cleaned up before we hard-deprecate.
Bug: T287216
Change-Id: Ie963eea5aa0f0e984ced7c4dfa0fd65d57313cfa
This name is consist with the rest of the setter and getter methods
in ParserOutput. Renamed the methods in OutputPage, ImageHistoryList,
ImageHistoryPseudoPager, and ContribsPager as well for consistency;
it also makes chasing down lingering references in codesearch easier.
Soft-deprecated the old name for 1.38. Hard-deprecation will follow,
but there are a number of users in production that should be chased
down first.
Code search:
https://codesearch.https://codesearch.wmcloud.org/deployed/?q=(allow%7Cprevent)Clickjacking&i=nope&files=&excludeFiles=&repos=
Bug: T287216
Change-Id: I9822c60c180d204bd30cb4447a1120155d456da4
New option 'absoluteURLs' was added to getText method
of the ParserOutput object that replaces all links
in the page HTML with absolute URLs.
Removing the action=render special case from Title
seems safe cause we will end up replacing the result
with absolute URL if we're in a render action no matter
where Title::getLocalUrl was called from.
This change is safely revertable from the perspective
of ParserCache.
Bug: T263581
Change-Id: Id660e1026192f40181587199d3418568f0fdb6d3
No need to manually reset MWTimestamp fake time,
MediaWikiTestCaseTrait::fakeTimestampTearDown() does
that after all tests anyway.
Change-Id: Ie45e837c693d7f21b6c7e25471cdaad6f60a3de9
CacheTime::mUsedOptions and ParserOutput::mAccessedOptions
do exactly the same thing and has to be merged into a single property.
This patch adds forward-compatibility and needs to be deployed
at least one train before the patch which actually merges the properties.
Change-Id: Ic9d71a443994e2545ebf2a826b9155c82961cb88
Based on Daniel's work at Ia6e70179b7ee5ce4e93888585ccc30d92da165c3
however was changed enough to move into a separate changeset.
More acceptance tests and data will be added in a followup commit.
Bug: T264397
Change-Id: I135187e83cbfa02b97c5656f0752f8bf1ceb58d0
This reverts commit a4dc6d82af.
I've reverted the merged patch since I didn't do enough testing
on serialized/reserialized ParserOutput and CacheTime. Now I'm
confident serialization/deserialization works.
Changes since original reverted version:
- Use __get/__set instead of DeprecationHelper in order to
avoid $deprecateProperties array to be serialized.
- Add test for old format serialization new format deserialization.
Change-Id: Ic911c2724ad709931d3316e609781fb89b5b7b28
I've been trying to find how -1 can actually get to
the cacheTime and could find any usages in core or
extensions. The proper way of making ParserOutput
uncacheable is to use updateExpiry( 0 ), that is
documented and has some warning about performance costs.
Setting the cacheTime to -1 seems to be an undocumented
feature (it is mentioned in comments as undocumented)
that doesn't seem to be used anywhere anymore.
I intend to remove it entirely after the deprecation is
deployed and we so not see any warnings for a few weeks.
Change-Id: Ie90b7e4a21faae726940fa9082f2e6a6ea8df613
When ParserOutput encounters a bad page title in an editsection
placeholder, this should not cause a fatal error. We can just not
produce an edit link and continue.
It's still worth logging though, since the parser shouldn't be putting
invalid links into editsection placeholders.
Bug: T261347
Change-Id: I154e85aec4b408e659e6281b02473c51f370865d
This adds methods to ParserOutput ::addExtraCSPStyleSrc,
::addExtraCSPDefaultSrc, and ::addExtraCSPScriptSrc, to easily
allow parser tags/functions to add additional CSP sources if their
tag needs it. Previously such an extension would need to use
and OutputPage hook. This is modeled on how addModules() works.
The immediate use case is for Kartographer (T240960), although
its expected that lots of extensions might do something like this,
especially extensions used outside of Wikimedia.
Change-Id: I24e5f0b4edff58025a0c2a3e1a9aa3f62eb7db7b
This patch tries to make assertions in tests more readable by using more
self-documenting assertions as provided by modern PHPUnit versions. Among
a few others, these two main changes are done:
* I found a lot of assertions with the expected value being the *second*
parameter. I did not changed all of them. Only some that can be replaced
with assertNull() and such.
* I try to replace all `assertTrue( is_…() )` with dedicated assertions.
Change-Id: I1fc72188fbd0edacf13886e7f9a9eacbd85f13c2
This was done automatically by replacing every assertContains with
string *needle*. Then verifying the results.
Bug: T192167
Change-Id: Id8cbbf3b01e948f80046714183cc299f86be21fd
And add a test which is confirmed to fail on HHVM prior to this change
with the error message "serialize(): "" returned as member variable from
__sleep() but does not exist".
Bug: T229366
Change-Id: I236bb4d64bc2e9f7756885e8c418399804eac5e1
These fields are passed to methods like LoadBalancer::getConnection() and are
already expected to be DB domains. Update various comments as well.
Fix a few minor IDEA warnings.
Change-Id: I7cf76700690aec449872a80d30b5ba540d2bf315
The addModuleScripts() methods were deprecated in 1.31 and 1.32,
these are now removed.
The getModuleScripts() are now deprecated as well, always returning
an empty array. To be removed in 1.34.
Depends on commits for bundled/wmf-deployed extensions that
remove the last few remaining callers to the deprecated functions
in: 3D, Collection, Flow, GlobalUserPage, and Wikibase.
Bug: T188689
Depends-On: If9f0bc6aef85117587fa1929f34f8861c8d80314
Depends-On: Ia8d41b97fbf6822f5f8f7ac889408acce1ac9a3a
Depends-On: I503b919739ea474ff33726815b0da55e2f7e2724
Depends-On: I236ef637fd03b810a46eb361e25067a037e9d183
Depends-On: I62e17779753b977a452cc0c9694947941e999cc3
Change-Id: I5a19b8f164ccf666485d2971202194b747f882df
This reverts commit 1bb5b58eb1.
A month has passed, the workaround for old parser cache entries
should not be needed anymore.
Bug: T203716
Change-Id: I446b47cc6b4c43aaae33675d62086d842b04ddcb
Previously, getCacheTime would default to the current time, potentially
causing the return value to change over subsequent calls. With this change,
the value is determined on the first call, and then remembered for subsequent
calls.
Bug: T205464
Change-Id: If240161c71d523ad5b0d33b9378950e0bebceb6e
Partially and temporarily reverts I1641b7995 to deal with cached
HTML the same way the old code did.
Bug: T203716
Change-Id: I29846a6513f6b580b429c0bfe6c310ada50b28bb
This allows optimization for situations in which a caller
needs the meta-data of a ParserOutput, and the respective
ContentHandler can provide that meta-data without generating
HTML output.
Bug: T194048
Change-Id: I786d294d18a6a2e3cea61577313e21b578c44f1e
RevisionRenderer is the MCR replacement for Content::getParserOutput,
as outlined in <https://www.mediawiki.org/wiki/User:Daniel_Kinzler_(WMDE)/MCR-PageUpdater>.
Note: This change also introduces quite a bit of code for
merging ParserOutput objects.
Bug: T194048
Change-Id: I871978bf79f67c9e7954fb3fc8528d6e365f2cc1
Instead of applying wrapping the the parser and unwrapping in
ParserOutput::getText(), turn this around and apply wrapping in getText(),
and only if desired.
This avoids search&replace logic for unwrapping, and it also makes it a lot
easier to merge the output of multiple slots for MCR output.
This changes behavior in two hopefully irrelevant ways:
1) the limit report comments will be inside the wrapper div, instead of
following it.
2) if HTML with a wrapper div is explicitly injected into a ParserOutput
object, it will not be possible to unwrap the text.
Bug: T174035
Change-Id: I1641b7995af9bd297f1acd610d583fbf874f34e0
This also removes all the in-core calls that had been kept for the
benefit of extensions, and causes them to not have any effect since
anything that had been calling them was already either a no-op or will
probably be broken now that nothing in core is setting or checking the
flags.
Change-Id: Id22c1a5a6d6a249debb14063ae3f8838d105b634
This transformation will find <style> tag with a "data-mw-deduplicate"
attribute. For each value of the attribute, the first instance will be
kept as-is, while any subsequent tags with the same value will be
replaced by a <link rel="mw-deduplicated-inline-style"> with its href
referring to the "data-mw-deduplicate" value using a custom scheme.
This also adds an $attribs parameter to Html::inlineStyle() so the
data-mw-deduplicate attribute can be added.
Note this doesn't actually depend on Ib931e25c, but action=mobileview
will break if it starts being used without that patch.
Bug: T160563
Change-Id: I055abdf4d73ec65771eaa4fe0999ec907c831568
Depends-On: Ib931e25ce85072000e62c486bbe5907f03372494
And deprecate passing false for ParserOptions::setWrapOutputClass().
There are three cases for the Parser wrapper: the default
mw-parser-output, a custom wrapper, or no wrapper. As things currently
stand, we have to fragment the parser cache on each of these options,
which uses a nontrival amount of storage space (T167784).
Ideally we'd do all the wrapping as a post-cache transform, but
TemplateStyles needs to know the wrapper in use in order to properly
prefix its CSS rules (that's why we added the wrapper in the first
place). So, second best option is to make *un*wrapping be a post-cache
transform and make "custom wrapper" be uncacheable.
This patch does the first bit (unwrapping as a post-cache transform),
and a followup will do the second part once the deprecation process is
satisfied.
Bug: T181846
Change-Id: Iba16e78c41be992467101e7d83e9c3134765b101
Clean up use of @codingStandardsIgnore
- @codingStandardsIgnoreFile -> phpcs:ignoreFile
- @codingStandardsIgnoreLine -> phpcs:ignore
- @codingStandardsIgnoreStart -> phpcs:disable
- @codingStandardsIgnoreEnd -> phpcs:enable
For phpcs:disable always the necessary sniffs are provided.
Some start/end pairs are changed to line ignore
Change-Id: I92ef235849bcc349c69e53504e664a155dd162c8
This is a hard deprecation, with getSecondaryDataUpdates returning an
empty array and addSecondaryDataUpdate throwing an exception. This seems
prudent since there are no known users of these methods, and they
interfere with the parser cache:
DataUpdates are basically jobs, they need access to services to
function. That makes them inherently non-serializable. This interferes
with the function of the parser cache, which serializes ParserOutput
objects in order to persist them.
This could be solved by splitting DataUpdates into DataUpdateDefinitions
and DataUpdateHandlers, similar to how JobSpecification works with
wgJobClasses. That however seems pointless and overkill, since
ParserOutput already has a mechanism for storing arbitrary data,
including any info needed by an UpdateJob: the setExtensionData method.
After this change, the preferred method to introduce custom data updates
is to store any relevant data using setExtensionData and
implement Content::getSecondaryDataUpdates() if possible. If not,
use the 'SecondaryDataUpdates' hook to construct the necessary update
objects from the info stored using setExtensionData.
Change-Id: I0f6f49e61fa3d8904e55f42c99f342a3dc357495
My previous patch broke this: ApiStashEdit would stash ParserOutput
with no custom DataUpdates, but calling getSecondaryDataUpdates still
failed after unserialization. This patch should fix that.
Bug: T86305
Change-Id: Ic114e521c5dfd0d3c028ea7d16e93eace758deef
Extensions sometimes need to stash information in the ParserOutput
for later use. This change provides a clean way to do that.
Change-Id: I8bc571d13c9a70bb71430862c2ab679ff1947126
Setting $wgRegisterInternalExternals = false for proto server should not
store the http/https links in externallinks table
Also fix detection of own links for links with query or anchor or
nothing
new also detected:
//localhost
//localhost?query
//localhost#anchor
already detected:
//localhost/path
Change-Id: Idd03d309cc3b71728a8cbea460efa12b10348d64
