RFC 3629 defines the legal range of characters as U+0000..U+10FFFF
and forbids overlong forms (encodings of a character that use more
bytes than necessary). Let's make StringUtils::isUtf8() match the
specification.
* Changed the maximum value in the pure PHP code path and added a
check for overlong forms.
* Added another check, specific to PHP 5.3's mbstring extension,
for values above U+10FFFF.
* Fixed the mbstring test errors in PHP 5.4 using changes to
StringUtilsTest by Platonides <platonides@gmail.com>.
* Uncommented some other tests that could fail because of the
missing check for overlong forms.
* Added additional tests for extra continuation bytes, overlong
sequences/forms, and values in the UTF-16 surrogate range.
The changes to the function were so extensive that I might as
well say I rewrote it.
Bug: 43679
Change-Id: I56ae496d17ffc3747550e06a72dacab3ac55da61
Also update some previous inconsistencies pointed out by Krinkle in change IDs:
* Ide20743a2e84ff68549286120e6cff9d9f396f54
* I811ca957b6588085d67606ebc0cd4033a1e53839
Change-Id: Ife33b931870d0d7e04fcb40974997436d27f528f
tests/phpunit/includes/StringUtilsTest.php:42:3
- avoid function calls in a FOR loop test part
tests/phpunit/includes/PathRouterTest.php:155 to 165
- fix space/tabs indenting
- convert double quotes to single for consistency
tests/phpunit/data/xmp/7.result.php
- reindent to use tabs instead of spaces
- removed trailing whitespaces
tests/phpunit/includes/filebackend/FileBackendTest.php
- spaces in indentation
Change-Id: I9c15803f961fc88b798ada3dd6c2b292c1de2143
Language class had a code snippet to verify whether a text is valid
UTF-8 though that could not be used from another place. The snippet use
mb_check_encoding() and fallback to some regex whenever mbstring is not
available.
* introduce StringUtils::isUtf8() which is mostly code moved out of the
language class.
* Enhance regex readability by using an expanded regex (//x)
* Made the regex to recognize longer sequences
* Add some unit tests to the mbstring and the PHP native implementation
* An optional second parameter can be passed to isUtf8() to force the
use of our PHP implementation. This is used for unit testing.
Change-Id: I4cf4dfe2eb02f046db1726f4654ba649e01419f2
