- Use a non-capturing group and positive look-ahead assertion, use the
overall capture as result
- Use substr_compare to compare strings
- Remove unneeded variable assigment
- Use array_pop to get the last element
- Remove unneeded isset/empty checks
- Change behaviour of $wsEnd to avoid +1 everywhere
Change-Id: Ifc1728b09743ec91908b59c09ac34598d9cd8b49
The regex searching for </$name> is case-insensitive (/i),
the cache to avoid the regex should be case-insensitive as well to avoid
that the regex is triggered for same tag name varying in case.
Change-Id: I5cdbc623be1ebd3cb8439c0ed4781da86f246c94
Since "segmentable" requires the use of strings, serialize the
tree as JSON, as it was before de6eeead21.
Bug: T309126
Change-Id: Ib5a902c6a8e03a129dec47426c2aa5f85e17c99f
Here is an example wikitext to play around with:
a
<!----> <!---> <!--->
b
a
<!----> <!---> <!--->b
https://www.mediawiki.org/w/api.php?action=parse&text=a%0A%3C!----%3E%20%3C!---%3E%20%3C!---%3E%0Ab%0A%0Aa%0A%3C!----%3E%20%3C!---%3E%20%3C!---%3Eb&prop=parsetree
Notice how the comments are parsed different in the two situations.
<comment><!----> </comment><comment><!---> </comment><comment><!--->\n</comment>
<comment><!----></comment> <comment><!---> <!---></comment>
The first one is wrong because <!---> is not an HTML comment. There
is one dash missing. It should be parsed as <!-- followed by the
content "-> <!-" followed by the closing -->, just as in the second
example.
Change-Id: Iaf8e52c5d41ca3f28890e8b66cfdf1d86126c357
All the other ways of doing it were ridiculous and much harder to read,
and usually required repeating the needle expression (to get its
length). I found these occurrences by grepping for various expressions,
but I undoubtedly missed some.
I didn't try replacing the many instances of strpos(...) === 0 with
str_starts_with(...), because I think they're readable enough as-is
(although less efficient). Likewise I didn't try porting strpos(...) !==
false to str_contains(...). For case-insensitive comparisons, Tim
Starling requested that we stick with substr_compare() because it's more
efficient than calling strtolower().
On PHP < 8 these functions will be included with a polyfill via
vendor/autoload.php. This is included at the beginning of
includes/AutoLoader.php, so if our autoloader has been included the
polyfill will be available. This means it should be safe to call these
functions from any code that would not be usable without our autoloader.
Three uses that Tim Starling identified as being performance-sensitive
have been split out to a separate commit for porting after the switch to
PHP 8.
Change-Id: I113a8d052b6845852c15969a2f0e6fbbe3e9f8d9
Make the caching logic use getWithSetCallback() and simplify
the code given that there is only one Preprocessor subclass.
Also, keep the cached values JSON serializable but rely on
the serialization in BagOStuff instead for simplicity.
Add related class constants for injecting preprocessor flags.
Bug: T254608
Change-Id: I72f9f0c0bc352ed5120469090c71294ff0c24999
Replace direct access to $wgDisableLangConversion with
LanguageConverterFactory::isConversionDisabled(), and replace direct
access to $wgDisableTitleConversion with
LanguageConverterFactory::isTitleConversionDisabled(). However, most
places that check ::isTitleConversionDisabled() actually want
::isLinkConversionDisabled(), so add that too (and deprecate
isTitleConversionDisabled()).
Code search:
https://codesearch.wmcloud.org/search/?q=Disable%28Lang|Title%29Conversion&i=nope&files=&repos=
This change removes a number of spurious dependencies on the global
configuration and reduces code duplication (for example, if the logic
for disabling language conversion were ever to change).
Depends-On: I6fa8230ae97b0e34c381003548e61f9b7387d363
Change-Id: Icc4687638ff1815003dd903854efdbd904854f1e
HHVM does not support variadic arguments with type hints. This is
mostly not a big problem, because we can just drop the type hint, but
for some reason PHPUnit adds a type hint of "array" when it creates
mocks, so a class with a variadic method can't be mocked (at least in
some cases). As such, I left alone all the classes that seem like
someone might like to mock them, like Title and User. If anyone wants
to mock them in the future, they'll have to switch back to
func_get_args(). Some of the changes are definitely safe, like
functions and test classes.
In most cases, func_get_args() (and/or func_get_arg(), func_num_args() )
were only present because the code was written before we required PHP
5.6, and writing them as variadic functions is strictly superior. In
some cases I left them alone, aside from HHVM compatibility:
* Forwarding all arguments to another function. It's useful to keep
func_get_args() here where we want to keep the list of expected
arguments and their meanings in the function signature line for
documentation purposes, but don't want to copy-paste a long line of
argument names.
* Handling deprecated calling conventions.
* One or two miscellaneous cases where we're basically using the
arguments individually but want to use them as an array as well for
some reason.
Change-Id: I066ec95a7beb7c0665146195a08e7cce1222c788
Added spaces around .
Removed empty return statement which are not required
Removed return after phpunit markTestIncomplete,
which is throwing to exit the test, no need for a return
Change-Id: I2c80b965ee52ba09949e70ea9e7adfc58a1d89ce
Find: /isset\(\s*([^()]+?)\s*\)\s*\?\s*\1\s*:\s*/
Replace with: '\1 ?? '
(Everywhere except includes/PHPVersionCheck.php)
(Then, manually fix some line length and indentation issues)
Then manually reviewed the replacements for cases where confusing
operator precedence would result in incorrect results
(fixing those in I478db046a1cc162c6767003ce45c9b56270f3372).
Change-Id: I33b421c8cb11cdd4ce896488c9ff5313f03a38cf
Whitelist the remaining usages of assert(), and reinstate the PHPCS sniff
that forbids usage of it. Add FIXME comments as well, so any casual readers
of the code will not think that the disabling and usage is intentional.
Change-Id: I7cabe715c0e6aa6a9ef3ffe5657f3de7fd8e662b
Clean up use of @codingStandardsIgnore
- @codingStandardsIgnoreFile -> phpcs:ignoreFile
- @codingStandardsIgnoreLine -> phpcs:ignore
- @codingStandardsIgnoreStart -> phpcs:disable
- @codingStandardsIgnoreEnd -> phpcs:enable
For phpcs:disable always the necessary sniffs are provided.
Some start/end pairs are changed to line ignore
Change-Id: I92ef235849bcc349c69e53504e664a155dd162c8
- mostly auto fixes
- some too long lines fixed
- ignore amp space in one case passing by reference
Change-Id: I6472f83bc3cbf4bd629d83050cc3319b19ec465c
This revises 2877402276, which was
reverted in master due to unexpected issues with `-{{...}} ` markup
on translatewiki and enwiki. Test cases are added to ensure that this
is parsed as a template, not as language converter markup.
https://www.mediawiki.org/wiki/Preprocessor_ABNF is the canonical
documentation for the preprocessor; this will be updated after this
patch is merged. The basic principles described in that page are
maintained in this patch:
* Rightmost opening structure has precedence: `-{{` is parsed as a
dash followed by template opening.
* `{{{` has precedence over `{{` and `-{`: `-{{{{` is parsed as
`-{` `{{{` since we first grab the rightmost `{{{`.
A bunch of test cases were added to verify the "ideal precedence"
order described on that wiki page.
This patch introduced some minor incompatibilities in existing
markup, in particular with chemical formulae in templates.
Fixes for these are being tracked at
https://www.mediawiki.org/wiki/Parsoid/Language_conversion/Preprocessor_fixups
Bug: T146304
Bug: T153761
Change-Id: I2f0c186c75e392c95e1a3d89266cae2586349150
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.
Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
This effectively reverts commit 2877402276 in
order to unblock the deploy train. The underlying behavior might not be
incorrect, but it was unexpected.
Bug: T153761
Change-Id: Ifc9c7cf3482dd5d222ff4da24a6d4cc401e9d965
This ensures that `{{echo|-{R|foo}-}}` is parsed correctly as
a template invocation with a single argument, not as two separate
arguments split by the `|`.
Bug: T146304
Change-Id: I709d007c70a3fd19264790055042c615999b2f67
The singly-linked list data structure of Preprocessor_Hash was causing
stack exhaustion due to the need for a recursion depth proportional to
the number of children of a given PPNode, in serialize() and on
object destruction. So, switch to array-based storage. PPNode_* becomes
a temporary proxy around the underlying storage, which avoids circular
references and keeps the storage very compact. Preprocessor_DOM uses
similar temporary PPNode objects, so the fact that
$node->getFirstChild() !== $node->getFirstChild()
should not cause any new problems.
* Increment cache version
* Use JSON serialization of the store array instead of serialize(),
since JSON is more compact, even after gzipping.
* For efficiency, make $accum a plain array, and use it as an array
where possible, instead of using helper functions.
Performance and memory usage for typical input are slightly improved:
something like 4% faster for the whole parse, and 20% less memory for
the tree.
Bug: T73486
Change-Id: I0d6c162b790d6dc1ddb0352aba6e4753854f4c56
(Previously done in f51d0d9a81 and
reverted in 543f46e9c08e0ff8c5e8b4e917fcc045730ef1bc.)
I think it's saner to treat this as invalid syntax, and output the
mismatched tag code verbatim. The current behavior is particularly
annoying for <ref> tags, which often swallow everything afterwards.
This does not affect HTML tags, though. Assuming Tidy is enabled, they
are still auto-closed at the end of the page content. (For tags that
"shadow" a HTML tag name, this results in the tag being treated as a
HTML tag. This currently only affects <pre> tags: if unclosed, they
are still displayed as preformatted text, but without suppressing
wikitext formatting.)
It also does not affect <includeonly>, <noinclude> and <onlyinclude>
tags. Changing this behavior now would be too disruptive to existing
content, and is the reason why previous attempt was reverted. (They
are already special-cased enough that this isn't too weird, for example
mismatched closing tags are hidden.)
Related to T17712 and T58306. I think this brings the PHP parser closer
to Parsoid's interpretation.
It reduces performance somewhat in the worst case, though. Testing with
https://phabricator.wikimedia.org/F3245989 (a 1 MB page starting with
3000 opening tags of 15 different types), parsing time rises from
~0.2 seconds to ~1.1 seconds on my setup. We go from O(N) to O(kN),
where N is bytes of input and k is the number of types of tags present
on the page. Maximum k shouldn't exceed 30 or so in reasonable setups
(depends on installed extensions, it's 20 on English Wikipedia).
Change-Id: Ide8b034e464eefb1b7c9e2a48ed06e21a7f8d434
This is about template parameters. They can be indexed by position (int) or
name (string). The returned value is always a string, or false (bool) on
failure.
Change-Id: I565210ad485505281246ef2bb3086a675b905976
This reverts commit f51d0d9a81.
Breaks templates with non-closed </noinclude> tags, which
were previously acceptable.
Bug: T125754
Change-Id: I8bafb15eefac4e1d3e727c1c84782636d8b82c2b
I think it's saner to treat this as invalid syntax, and output the
mismatched tag code verbatim. The current behavior is particularly
annoying for <ref> tags, which often swallow everything afterwards.
This does not affect HTML tags, though. Assuming Tidy is enabled, they
are still auto-closed at the end of the page content.
Related to T17712 and T58306. I think this brings the PHP parser closer
to Parsoid's interpretation.
It reduces performance somewhat in the worst case, though. Testing with
https://phabricator.wikimedia.org/F3245989 (a 1 MB page starting with
3000 opening tags of 15 different types), parsing time rises from
~0.2 seconds to ~1.1 seconds on my setup. We go from O(N) to O(kN),
where N is bytes of input and k is the number of types of tags present
on the page. Maximum k shouldn't exceed 30 or so in reasonable setups
(depends on installed extensions, it's 20 on English Wikipedia).
To consider:
* Should we keep previous behavior for unclosed <includeonly> /
<noinclude>? This would be particularly disruptive for these if
someone relied on the old behavior, and they're already
special-cased in places.
* Unclosed <pre> tags are now treated as HTML tags, and are still
displayed as preformatted text, but without suppressing wikitext
formatting.
Change-Id: Ia2f24dbfb3567c4b0778761585e6c0303d11ddd0
