The C tokenizer was silently truncating input at the first NUL byte (\x00)
because it used '\0' both as a valid input character and as the EOF sentinel.
Root cause:
- Tokenizer_read() returned '\0' for both:
1. Real NUL bytes in the input
2. End-of-input (when index >= text.length)
- This made them indistinguishable, causing real NULs to be treated as EOF
Fix:
1. Define TOKENIZER_EOF as 0x110000 (first invalid Unicode code point)
2. Update Tokenizer_read() and Tokenizer_read_backwards() to return
TOKENIZER_EOF instead of '\0' for out-of-bounds reads
3. Replace all `!this` and `'\0'` checks with explicit `TOKENIZER_EOF` checks
4. Remove '\0' from the MARKERS array (no longer needed as EOF marker)
5. Move EOF check before is_marker() in main parse loop to ensure
TOKENIZER_EOF doesn't try to emit as a character
6. Fix Tokenizer_has_leading_whitespace() to recognize TOKENIZER_EOF
The Python tokenizer already preserved NUL bytes correctly; this brings
the C tokenizer into parity.
Regression test added: test_nul_byte_preservation() verifies that both
tokenizers now preserve NUL bytes in plain text, templates, and
multiple-NUL scenarios.
This regression seems more severe than the bug the commit was
attempting to fix (incorrect parsing of nested wikilinks in normal
links), so that bug is reintroduced until localization-aware parsing
that allows us to detect file links is added.
This commit partially reverts fac60dee48.
* Proposed fix for https://github.com/earwig/mwparserfromhell/issues/197
* Port the fix for #197 to the C tokenizer
* Fix parsing of external links where the URL is terminated by some special character
- One existing test case has been found wrong -- current MediaWiki
version always terminates the URL when an opening bracket is
encountered.
- Other test cases added: double quote, two single quotes and angles
always terminate the URL (regardless if it is a free link or external
link inside brackets). One single quote does not terminate the URL.
* Fix case-insensitive parsing of URI schemes
pytest is the preferred way to write and run unit tests these days and
it has a cleaner interface - so lets switch to it. The tokenizer tests
especially are much easier to read/understand.
This was mostly done with find/replace regexes and then cleaned up
manually.
Step one of refactoring - making SmartList into its own
package, with each class having its own file. No code
changes were made.
Note that SmartList and ListProxy import each other,
so had to import SmartList as a full package name
rather than use from ... import ... construct.
Also removed the max cycles stop-gap, allowing much more complex pages
to be parsed quickly without losing nodes at the end
Also fixes#65, fixes#102, fixes#165, fixes#183
Also fixes#81 (Rafael Nadal parsing bug)
Also fixes#53, fixes#58, fixes#88, fixes#152 (duplicate issues)
June Kim
Cateon
E. Seiver
David Ebbo
Jakub Klinkovský
Kunal Mehta
Yuri Astrakhan
Hugo