Followup r91591, r93020: patch to jsminplus to support Unicode chars and char escapes in identifiers
Fast-path check keeps runtime about the same on most scripts (eg jquery.js parsing was abround 4100ms both before and after on my test machine)
Slow-path code kicks in if plain ASCII word chars don't extend all the way to the next whitespace or punctuation char.
Using PCRE's Unicode properties magic to ensure that we're catching everything, following ECMA-262 edition 5.1 spec.
Note that identifiers using escapes don't get normalized to their UTF-8 form; this might be a nice thing to do as it saves a couple bytes, but currently there's no change made to output.
Added QUnit tests to verify that unicode letter & escapes work in identifiers in all supported browsers (ok back to IE 6, yay)
This fixes the Notices that would be produced if several JSTokenizer were instantiated.
jsminplus was added in r91591.
Defines code was generated with:
$js = new JSTokenizer();
foreach ($js->opTypeNames as $operand => $name) echo "define('OP_$name', '$operand');\n";
foreach ($js->keywords as $keyword) echo "define('KEYWORD_" . strtoupper($keyword) . "', '$keyword');\n";
