Commit graph

1554 commits

Author SHA1 Message Date
Ebrahim Byagowi
e1385d3bdf Add {{#dir}} parser function
Template:Dir is one of the most used templates in Wikimedia Commons,
this tries to provide parts of its functionality in hope we can
perhaps simplify or get rid of the template eventually for clarity and
performance reasons.

As a convenience, `{{#dir}}` and `{{#dir:}}` are synonyms for
`{{#dir:{{PAGELANGUAGE}}}}`: they return the direction of the target
language.  For articles, the target language is the content language;
for messages, the target language is the user language.

In addition, to avoid confusion between BCP-47 language codes and
MediaWiki-internal language codes, an optional second parameter can be
supplied.  If the second parameter is the (localizable) string
'bcp47', the language code given in the first parameter will be
treated as a BCP-47 code.  For example: `{{#dir:sr-Cyrl|bcp47}}`.

(See LanguageCode::bcp47ToInternal() for a description of the
differences and overlaps between MediaWiki internal and BCP-47
codes.  These overlaps *so far* don't result in any case where
encouraging editors to be precise about which set of enumerated
string values they are using for consistency with other
language-related functions, and because MediaWiki internally
differentiates between BCP-47 codes and internal codes.)

Bug: T359761
Change-Id: I19c3e91a924e080f37dc95a0d4e61493583b533e
2024-07-19 16:57:48 -04:00
Arlo Breault
f3e9477465 Sync up core repo with Parsoid
This now aligns with Parsoid commit 3e10bb7a56619fe3881ea7c759ada21b96dc592e

Change-Id: Iff282e927bdcdb1feef00ba630e33c54c63bec5f
2024-07-10 14:41:17 -04:00
Lucas Werkmeister
d19f2543c1 Update expected test output after i18n change
The Arabic translation of red-link-title was modified again in change
Id00b720194 (commit 30c622c091), requiring another fix mirroring change
I8f2930802a (commit 6cbd9e5263).

Test with:

    composer phpunit -- --testsuite parsertests --filter=T236183

Bug: T369694
Change-Id: I19090fe523e0a5c17bc1c30ee31edce24f541e6b
Follows-Up: Id00b720194d3a715050cbce55e40ca11b34212ce
2024-07-10 11:38:39 +02:00
Arlo Breault
b8258c5c15 Sync up core repo with Parsoid
This now aligns with Parsoid commit fefcac4e949707536530828366d74f06aac88861

Change-Id: I6bf0be053982454d7ccba7a03268440644813cf2
2024-06-27 11:09:10 -04:00
jenkins-bot
1dba954b3f Merge "Remove image and gallery image caption trimming" 2024-06-27 13:54:58 +00:00
jenkins-bot
f412e1b3b8 Merge "Fix mw-selflink-fragment on variant fragment links" 2024-06-27 10:42:44 +00:00
Arlo Breault
6b05fa3a21 Remove image and gallery image caption trimming
Post I5039c7ef9e07199c256fd568b4f94714e5831d17, gallery image captions
are no longer placed on new lines, so the presence of leading whitespace
shouldn't be significant.

This fixes an inconsistency in gallery image caption trimming, where
only the first and last option had start and end trimming, respectively.

It also matches Parsoid output, where no trimming takes places, as seen
in the updated tests.

Change-Id: I2a80198c43598dc8c7fa61cb4b0340a97d2ee895
2024-06-26 21:51:40 -04:00
Lucas Werkmeister
6cbd9e5263 Update expected test output after i18n change
The Arabic translation of red-link-title was modified in change
I47769df5dc (commit dfd748033e).

Bug: T368383
Change-Id: I8f2930802a6b161ba44205d6b6d114d223be3cde
Follows-Up: I47769df5dc91ad12d817e117fe931b1452e8b2ef
2024-06-25 14:59:14 +00:00
Arlo Breault
c356dfed72 Fix mw-selflink-fragment on variant fragment links
Should have been part of 1fca3b5b.

The fix to doVariants can be seen in old output linking [[Dуна#Foo]] to
Дуна despite [[Dуна]] being a self-link in the test above.

Bug: T198652
Change-Id: Id38cfc47041492c5cc68b4f8f9566f421c9168bd
2024-06-19 08:50:46 +00:00
Arlo Breault
867b158c51 Fix test about applying mw-selflink-fragment to variant links
A byte sequence (d0 b0) was presumably erroneously introduced in the
link [[Duna#Foo]] which prevented it from being recognized as the same
target as the title associated with the current test.  An article (Дуна)
that's a variant of Duna exists, and variant conversion determines that
the link points at that page instead.  Removing those bytes updates the
output to add the mw-selflink-fragment class.

The test itself is updated to reflect that, as of 1fca3b5b, fragment
links are selflinks and a FIXME is added for a follow-up to correct the
variant fragment links to add the class.

Change-Id: Iee0326f85c919a672397d0378f3549f583c17e28
2024-06-18 15:31:35 -04:00
Arlo Breault
adf0beeaab Sync up core repo with Parsoid
This now aligns with Parsoid commit af12018f9905bceeb7fbf201f8685bcc39d8cdf4

A lot of langParserTests were split out of parserTests in
Idcce741402233fb4768ba06868f09bff0397172a but langParserTests.txt wasn't
being sync'd to the core repo.

Change-Id: Icff4a70fba4bf154f7438d354d17972173c4469c
2024-06-17 16:03:28 -04:00
jenkins-bot
45c105ec46 Merge "Parser: Avoid extra escaping in replaceTableOfContentsMarker" 2024-06-12 19:51:18 +00:00
vahurzpu
fbba3bb2cf Parser: Avoid extra escaping in replaceTableOfContentsMarker
I60fdfc2c52 changed replaceTableOfContentsMarker from using
preg_replace, which supports backreferences in the replacement, and
thus expects literal backslashes and dollar signs to to be escaped,
to using preg_replace_callback, which does not expect any escaping.
This caused unwanted backslashes in headings. This patch removes the
escaping.

Bug: T365413
Change-Id: Idbdc3074c7ad007627c4c259a1aaf090a5d0c7f9
2024-06-12 06:10:58 -04:00
C. Scott Ananian
c8e77a3707 Sync up core repo with Parsoid
This now aligns with Parsoid commit 2508e24a2aeb54b55eb54f7f65bedc4d477fc9cf

Change-Id: Ibb9f1c6287c6ec3e982f0fa3ddf908b01484973a
2024-06-10 23:29:02 -04:00
jenkins-bot
bbadf63fa8 Merge "Move Linker::makeExternalLink() to the LinkRenderer service" 2024-06-10 19:58:45 +00:00
C. Scott Ananian
b855c62f66 Move Linker::makeExternalLink() to the LinkRenderer service
Move Linker::makeExternalLink to the LinkRenderer service, as has been
done with the other static methods of Linker.

In order to allow phan's SecurityCheckPlugin to perform a more accurate
analysis of taintedness, tweak the API of Linker::makeExternalLink to
clearly indicate via the type system whether the link text has already
been escaped or not: a `string` argument will always be escaped, and
if the argument is already escaped it should be passed as an HtmlArmor
object.  In refactoring, `Message` arguments were also common, and accept
them as-is to avoid the caller having to think about whether to call
Message::text() or Message::escaped().

This allows us to provide a more precise taint type to the $text argument,
avoids an opaque boolean argument, and avoids spurious errors from
SecurityCheck.

We also require the caller to explicitly pass a Title context, instead
of implicitly relying on the global $wgTitle.  This works cleanly
everywhere except for CommentParser, which has a $selfLinkTarget which
generally works as the title context for the external link, but which
is nullable.  The original Linker::makeExternalLink() used $wgTitle as
a fallback, but $wgTitle can also be null in some circumstances.  The
title context only determines how $wgNoFollowNsExceptions is handled,
so existing code basically just ignored $wgNoFollowNsExceptions when
$wgTitle was null, which isn't terrible.  A future refactor could/should
clean up CommentParser to ensure that there is always a non-null title
context that can be used.

Change-Id: I9bcf4780f388ba639a9cc882dd9dd42eda5736ae
2024-06-10 18:47:32 +00:00
Arlo Breault
2d407a10f2 Fix dropped space from headings parsertests
Follows-Up: I1180169a8e83af834c2984ba16089e6277f2a8dd
Change-Id: I33d4e13332e42c85705f6b179fc64bfcdd341df0
2024-06-07 20:10:36 -04:00
Isabelle Hurbain-Palatin
36132fc91b Sync up core repo with Parsoid
This now aligns with Parsoid commit 2b45686645a87155cb6148e295f5a67f29f261c8

Change-Id: I75ea7f96548033edd2f3925b5ecf2db9cf6bcbb0
2024-06-07 09:06:26 +02:00
Ebrahim Byagowi
d21cc67450 Add namespace and deprecation alias to FileBackend
This patch introduces a namespace declaration for the
Wikimedia\FileBackend to FileBackend and establishes a class
alias marked as deprecated since version 1.43.

Bug: T353458
Change-Id: Id897687b1d679fd7d179e3a32e617aae10ebff33
2024-05-19 22:35:58 +03:30
Ebrahim Byagowi
a717db8e60 Add namespace and deprecation alias to FormatJson
This patch introduces a namespace declaration for the
MediaWiki\Json to FormatJson and establishes a class
alias marked as deprecated since version 1.43.

Bug: T353458
Change-Id: I5e1311e4eb7a878a7db319b725ae262f40671c32
2024-05-16 16:28:01 +03:30
Bartosz Dziewoński
f0c7fa9234 Move section edit links outside headings (new heading HTML)
Legacy parser can now output headings using a more accessible markup,
which is also identical to the markup used by the Parsoid parser.

Changes to client-side JS and CSS necessary to support the new markup
have already been merged in earlier commits.

includes/skins/Skin.php
includes/ServiceWiring.php
* Define a new skin option, 'supportsMwHeading', which can be used
  to toggle the new markup per-skin.
* Update the built-in fallback skin to enable it. This affects the
  output in parser tests.

docs/config-schema.yaml
includes/config-schema.php
includes/config-vars.php
includes/MainConfigNames.php
includes/MainConfigSchema.php
* Add a new configuration setting, 'ParserEnableLegacyHeadingDOM',
  which can be used to toggle the new markup per-site.

includes/OutputTransform/Stages/HandleSectionLinks.php
* Output new heading HTML for skins that enabled the option.

tests/*
* Duplicate parser tests that cover heading generation to cover both
  new and old markup. Update other parser tests to use new markup.
* Add some unit and integration tests for the behavior of the skin
  option and some parser tests for edge cases of the new markup.

Bug: T13555
Change-Id: I1180169a8e83af834c2984ba16089e6277f2a8dd
2024-05-06 12:25:33 -04:00
Timo Tijhof
c02513c97e phpunit: Fix tests relying on implicit wgScript/wgArticlePath
A number of tests have hardcoded expections that pass only in WMF CI
where Quibble has LocalSettings.php with $wgScript and $wgArticlePath
set a certain way.

We could fix these by adding setMwGlobals() in their tests, as we
often do, but these are so often forgotten that I'd rather we just
add them to TestSetup.php so that it is simply impossible to write a
test that that passes locally for you (if you have the same config)
but not for someone else.

There is a larger project in there somewhere about expanding this
slowly such that we basically only pluck DB-settings and extension
enablement from LocalSettings and otherwise run the tests with the
default settings in PHPUnit. Pretty much by definition, any (other)
setting you have in LocalSettings is irrelevant because it either:
1. has no effect on the test (majority, harmless either way),
2. has a custom default via TestSetup.php (which has precedence over
   LocalSettings.php),
3. is relevant to the code being tested and the test case correctly
   calls setMwGlobals() to ensure a consistent value during test.
4. is relevant to the tested code but has no override, thus only
   passes if you happen to have the "right" value set for it
   (undesirable).

Case 4 is already categorically impossible for the most common config
settings that influence random code because we give them a value
in TestSetup.php. This patch expands that to include $wgScript
and $wgArticlePath. Perhaps in the future we can think about a way
to do this automatically by either re-applying MainConfigSchema
(sans db settings) or by only selectively applying LocalSettings.php
in the first place.

This patch follows-up I072ddf89562fe, which added a test case in
WikitextContentHandlerIntegrationTest.php that assumed "/index.php"
as the value of $wgScript. This passes in WMF CI since Quibble uses
that value, but the tests failed in most local development installs
since those tend to use "/w" instead.

Rather than one-off fixing that one test with overrideConfigValues(),
switch to a more general fixture, since the precise values don't
matter for this test.

Bug: T349087
Bug: T277470
Change-Id: If4304b7ca4a838bd892d4516a0b5c6dfbc30986e
2024-05-05 00:00:01 +00:00
Subramanya Sastry
33f2164096 Sync up core repo with Parsoid
This now aligns with Parsoid commit 902eb345ed701b635b98f03557276aa48b564cc2

Change-Id: I91c663a4f2ca00157fbd9337d1d0c72a98452591
2024-04-26 14:57:58 +05:30
Arlo Breault
3c9b410e17 Add a file to the ParserTestRunner
Bug: T363520
Change-Id: I0d5b577d19f98d4fc6f22930c9199a24fc459b9a
2024-04-25 23:47:54 -04:00
Umherirrender
a89a00ffb8 tests: Migrate to IDatabase::newInsertQueryBuilder
Changed some inserts to use multi-row insert for small performance
benefit where possible and not already used.
InsertQueryBuilder does not return a value, deprecated since 1.33

Bug: T353219
Change-Id: I2380ebc8ec8db178dd790247aefbdd798b6d62ff
2024-04-14 21:56:07 +02:00
Arlo Breault
de01ef7d20 Sync up core repo with Parsoid
This now aligns with Parsoid commit c296dca4af9a1d47200a3699e12d9884acc43150

Change-Id: I5a0e246171e9b58d77b2be945b802f381c1f40b2
2024-04-11 12:59:32 -04:00
jenkins-bot
2472cd9247 Merge "Substitute category default sort key when filling links table, not at parse time" 2024-04-11 14:59:33 +00:00
jenkins-bot
71b809f9c2 Merge "Don't strip non-newline whitespace from left side of language links" 2024-04-04 16:56:28 +00:00
jenkins-bot
31a686f9da Merge "Sync up core repo with Parsoid" 2024-04-01 04:00:01 +00:00
Subramanya Sastry
0cd8ecf2a5 Sync up core repo with Parsoid
This now aligns with Parsoid commit 16e27722c6c50618c78230952c1ad27948fc3a0b

Change-Id: I21067c1b22a494422184abf7c4bb50424b4fad56
2024-04-01 08:16:27 +05:30
C. Scott Ananian
63293370e5 Don't strip non-newline whitespace from left side of language links
This follows up on I5e87b33a956e296cdaf671fa99c9555944b73479 and makes
(invisible) language links consistent with how we handle (invisible)
category links.

Bug: T359886
Followup-To: I5e87b33a956e296cdaf671fa99c9555944b73479
Change-Id: I3e5567a91b47e0b04da928450644f3f475aaf51b
2024-03-29 18:46:16 -04:00
C. Scott Ananian
bf7120f80e Don't strip non-newline whitespace from left side of [[Category]] links
This follows up on a long series of tweaks to whitespace handling around
[[Category]] links (T2087, T87753, T174639) which aimed to simplify and
make intelligible the whitespace handling around category links without
allowing categories to break lists or paragraphs in which they are found.

Removing newlines but not other whitespace on the left-hand side of
category links should preserve the valuable features of T2087 et al
while still ensuring that the following all render equivalently:

  ABC [[Category:Foo]]DEF
  ABC[[Category:Foo]] DEF
  ABC [[Category:Foo]] DEF

Added parser test to document the new behavior; it's worth noting
that although there were plenty of tests documenting the expected
interaction of category links and newlines, there were previously
no tests covering the interaction of non-newline whitespace and
category links; the one test which needed to be altered added
non-semantic whitespace (ie, extra whitespace to the test output
which did not affect the way the HTML would display).

This patch brings the legacy parser into parity which Parsoid parsing
of category links.

Bug: T359886
Change-Id: I5e87b33a956e296cdaf671fa99c9555944b73479
2024-03-29 22:30:59 +00:00
C. Scott Ananian
c2df535b9c Substitute category default sort key when filling links table, not at parse time
This ensures uniform treatment of all places that call `addCategory`
without duplicating the `defaultsort` code; it also ensures that the
effect of the {{DEFAULTSORT}} parser function is independent of page
position.

Bug: T40435
Bug: T353530
Change-Id: I4480a6d59e766fa4eddc9ec9117c58b66771bb47
2024-03-29 18:30:02 -04:00
Daimona Eaytoy
3a2f16d3f1 parsertests: Misc cleanup
Change-Id: Id1b073d80e896a26493206141bddfcf65c171d0b
2024-03-15 00:34:23 +00:00
Derick Alangi
b06f079e2b
ParserTestRunner: Remove parser test caching setup code
As of today, MediaWikiIntegrationTestCase together with TestSetup
already do the correct thing by setting up tests with appropriate
caching backends with HashBagOStuff so this hack is no longer needed.

Change-Id: I7a02ffb1bd7807901fe5d515b6a158865e8e83ea
2024-03-14 21:06:44 +03:00
jenkins-bot
f0c9885b0c Merge "[tests] Remove use of dynamic property Parser::$static_tag_buf" 2024-03-05 16:01:25 +00:00
jenkins-bot
210772516f Merge "parser: Fix formatdate parser function for ISO year 0 = 1 BC" 2024-03-01 21:45:31 +00:00
jenkins-bot
376e81c2a7 Merge "Support "showmedia" parsertests option" 2024-02-29 02:49:23 +00:00
thiemowmde
a15b6d516f parser: Fix formatdate parser function for ISO year 0 = 1 BC
I'm not sure how this ever happened, but I'm sure it's a mistake.
The following test scenario should make it very obvious:

* {{#formatdate:-0002-12-31|mdy}}
* {{#formatdate:-0001-12-31|mdy}}
* {{#formatdate:0000-12-31|mdy}}
* {{#formatdate:0001-12-31|mdy}}
* {{#formatdate:0002-12-31|mdy}}

Expected output: 3 BC, 2 BC, 1 BC, 1, 2, …
Current output: 3 BC, 2 BC, 0 (?), 1, 2, …

Note how "1 BC" is skipped and shown as "0" instead. Everything else
is correct, e.g. the ISO year -1 is already displayed as "2 BC".
It's really only this single outlier.

In case you don't know: There is no year 0 when the BC specifier is
used. There is either year 1 after or year 1 before Christ. This is
different in ISO, mostly to make calculations easier. That's why the
DateFormater already does an extra `- 1` and `+ 1` in the two
makeIsoYear and makeNormalYear methods.

The problematic line of code was originally written in 2003, see
https://phabricator.wikimedia.org/rMW98fc03e6
The core parser function exists since 2009, see
https://phabricator.wikimedia.org/rMWb9ffb5a7

Change-Id: Iaeb7a954579a409fefd87dab4e2a15778ab39fb4
2024-02-27 17:17:36 +01:00
jenkins-bot
58e97df366 Merge "Drop comments in cleanUpTocLine" 2024-02-24 02:55:48 +00:00
Arlo Breault
5ed94aba15 Drop comments in cleanUpTocLine
Needed-By: Ie6760dd25f937d4f6acbab1c0e1475b54878d4ed
Change-Id: I10f96435f892b188cffe64b92cdf2701a3e2058b
2024-02-22 19:06:15 -05:00
C. Scott Ananian
93f0648a35 [tests] Remove use of dynamic property Parser::$static_tag_buf
There are only two test cases using the <statictag> tag extension and
it's no longer entirely clear what regression they were intended to
prevent.

In any case, we can use ParserOutput to maintain our parser state
instead of the dynamic Parser::$static_tag_buf property.  That's not a
perfect solution -- T300979 says at some point in the future we won't
guarantee parse order -- but it will work for now.

Bug: T357838
Change-Id: Ia23049fc729c90a94fc16231ad89c199db5b7bc9
2024-02-22 16:02:19 -05:00
Arlo Breault
5fc737eb91 Support "showmedia" parsertests option
Bug: T358158
Change-Id: I2db79f9a4658e6cb638ccf2716bd6c3d9917a3cc
2024-02-22 15:38:29 -05:00
C. Scott Ananian
3cebc721bb Sync up core repo with Parsoid
This now aligns with Parsoid commit 51baccc8741108a9e3f763f2c19c6ce6eda55ac4

Three tests needed to be disabled because they had dependencies on features
not included in core's CI:

* {{#if}} used in tests added by I71c38b42ac9bfb7137f2e34df70bdfa139abced7
  but only provided by the ParserFunctions extension
* <poem> used in tests added by I5a6356a82251881a5f841b36a7f26879fc611138
  but only provided by the Poem extension

In addition, the "multiline" part of the "Expansion of multi-line..."
parser tests seems to have been lost at some point.  My best guess is
that the definition of `Template:1x` initially included an extra
newline which was lost, maybe during an unrelated stripping of
leading/trailing whitespace in `!! article` clauses.  In any case,
these tests are no longer testing the thing they say they are.

These will be fixed in a follow up.

Change-Id: Ia9144634625f176fbea11f3d2ef4b21a5492e99b
2024-02-21 15:04:08 -05:00
Reedy
2295da3004 Fix more incorrect casing of MediaWiki
Change-Id: I331e5636823a0beae8d804148f648cfaffd6a1f8
2024-02-19 14:35:34 +00:00
C. Scott Ananian
cf5aabc42f ParserTestRunner: add [[pmid:]] interwiki prefix
This supports writing tests for the behavior requested in
T145590#8608455 whereby magic links are preferentially serialized
as interwiki links.

Bug: T145590
Change-Id: I2db78bdba6a32fbc2ef625e8c4a34091455bc234
2024-02-17 01:58:04 +00:00
Subramanya Sastry
e55cc517da Move Parser to Mediawiki\Parser namespace
Bug: T166010
Co-Authored-By: Daimona Eaytoy <daimona.wiki@gmail.com>
Co-Authored-By: James Forrester <jforrester@wikimedia.org>
Co-Authored-By: Subramanya Sastry <ssastry@wikimedia.org>
Change-Id: I79b4e732c45095eedbaa80afa5eb7479b387ed8a
2024-02-16 09:18:38 -05:00
James D. Forrester
060a1b1668 Replace last remaining wfGetDB() calls in core, except ResourceLoader
Bug: T330641
Change-Id: I6d30af6ff9f667e367d39befb80c2bb0bf5fb29e
2024-02-14 11:02:01 -05:00
jenkins-bot
3ab381ec90 Merge "ParserTestRunner: clean up ::addArticles()" 2024-02-13 00:09:04 +00:00
jenkins-bot
1c058326af Merge "ParserTestRunner: Fix behavior of bare parsoid-compatible file option" 2024-02-12 23:46:12 +00:00