June Kim
158b8967e1
fix: check for EOF instead of truthiness in tag close handling
Tokenizer_read returns TOKENIZER_EOF for end-of-input, not a falsy
value. The old truthiness check let NUL bytes truncate parsing.
5ヶ月前
June Kim
0f5f898bb7
fix: replace remaining !last checks with TOKENIZER_EOF comparison
Four start-of-input checks in the main parse loop still used !last
(falsy NUL) to detect beginning-of-input. After the TOKENIZER_EOF
sentinel change, Tokenizer_read_backwards returns 0x110000 instead of
'\0', so !last is always false and headings, lists, and horizontal
rules at position 0 would silently fail to parse.
5ヶ月前
June Kim
986b71a854
Fix C tokenizer NUL byte truncation (issue #355 )
The C tokenizer was silently truncating input at the first NUL byte (\x00)
because it used '\0' both as a valid input character and as the EOF sentinel.
Root cause:
- Tokenizer_read() returned '\0' for both:
1. Real NUL bytes in the input
2. End-of-input (when index >= text.length)
- This made them indistinguishable, causing real NULs to be treated as EOF
Fix:
1. Define TOKENIZER_EOF as 0x110000 (first invalid Unicode code point)
2. Update Tokenizer_read() and Tokenizer_read_backwards() to return
TOKENIZER_EOF instead of '\0' for out-of-bounds reads
3. Replace all `!this` and `'\0'` checks with explicit `TOKENIZER_EOF` checks
4. Remove '\0' from the MARKERS array (no longer needed as EOF marker)
5. Move EOF check before is_marker() in main parse loop to ensure
TOKENIZER_EOF doesn't try to emit as a character
6. Fix Tokenizer_has_leading_whitespace() to recognize TOKENIZER_EOF
The Python tokenizer already preserved NUL bytes correctly; this brings
the C tokenizer into parity.
Regression test added: test_nul_byte_preservation() verifies that both
tokenizers now preserve NUL bytes in plain text, templates, and
multiple-NUL scenarios.
5ヶ月前
dependabot[bot]
1daaf888f4
Bump pytest from 9.0.2 to 9.0.3 ( #354 )
Bumps [pytest](https://github.com/pytest-dev/pytest ) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases )
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3 )
---
updated-dependencies:
- dependency-name: pytest
dependency-version: 9.0.3
dependency-type: direct:development
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6ヶ月前
dependabot[bot]
f9ae021a41
Bump pygments from 2.19.2 to 2.20.0 ( #351 )
Bumps [pygments](https://github.com/pygments/pygments ) from 2.19.2 to 2.20.0.
- [Release notes](https://github.com/pygments/pygments/releases )
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES )
- [Commits](https://github.com/pygments/pygments/compare/2.19.2...2.20.0 )
---
updated-dependencies:
- dependency-name: pygments
dependency-version: 2.20.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6ヶ月前
dependabot[bot]
bc84526c80
Bump requests from 2.32.5 to 2.33.0 ( #350 )
Bumps [requests](https://github.com/psf/requests ) from 2.32.5 to 2.33.0.
- [Release notes](https://github.com/psf/requests/releases )
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md )
- [Commits](https://github.com/psf/requests/compare/v2.32.5...v2.33.0 )
---
updated-dependencies:
- dependency-name: requests
dependency-version: 2.33.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6ヶ月前
Ben Kurtovic
892bffb0ae
One more action bump
7ヶ月前
Ben Kurtovic
8a0738bf93
Apparently we disagree on clang-format behavior
7ヶ月前
Ben Kurtovic
00824ee9bb
Run uv lock --upgrade
7ヶ月前
Ben Kurtovic
85ef04c795
Update GitHub Actions config
7ヶ月前
Ben Kurtovic
fbd09aa654
Add Python 3.14 to build and remove EOL 3.9
7ヶ月前
Ben Kurtovic
d9a889d417
release/0.7.2
1年前
Ben Kurtovic
36e6157197
Improve Wikicode/Node typing
1年前
Ben Kurtovic
4bcd87e8a8
Clean up unnecessary API docs
1年前
Ben Kurtovic
4860387cf3
Disable C tokenizer building on PyPy ( fixes #341 )
1年前
Ben Kurtovic
dd614455c2
Update release script
1年前
Ben Kurtovic
c0d0e84c9b
release/0.7.1
1年前
Ben Kurtovic
35cb3b4c4b
Fix building in C23 ( fixes #340 )
1年前
Ben Kurtovic
067baef238
Support $WITH_EXTENSION to control building the C tokenizer
1年前
Ben Kurtovic
68aebb08d3
Clean up docs
1年前
Ben Kurtovic
8ac8a66c01
Fix coverage action and add pre-commit hook action
1年前
Ben Kurtovic
2a04a68930
Try to get coveralls working again
1年前
Ben Kurtovic
9bffd6278e
Try to get coveralls working again
1年前
Ben Kurtovic
4540b2d58a
Add some debugging
1年前
Ben Kurtovic
afff3e5529
Fix typo
1年前
Ben Kurtovic
5402ba8485
Try to get coveralls working again
1年前
Ben Kurtovic
9109c867e8
Update release script
1年前
Ben Kurtovic
fd1b08c4ba
Try again for Windows
1年前
Ben Kurtovic
e946c5f175
Try to get coveralls working again
1年前
Ben Kurtovic
d0f697e5ba
Try to get coveralls working again
1年前
Ben Kurtovic
c1e62d4f48
Try to get coveralls working again
1年前
Ben Kurtovic
5d2aff1090
Try to get coveralls working again
1年前
Ben Kurtovic
5d50de52e9
Try to get coveralls working again
1年前
Ben Kurtovic
0335c50d09
Fix workflow again
1年前
Ben Kurtovic
463a40d2f2
Try to fix workflow
1年前
Ben Kurtovic
a9abc78698
Fix workflow
1年前
Ben Kurtovic
432069770c
Fix workflow
1年前
Ben Kurtovic
ccc1bc69a4
Clean up github actions workflows
1年前
Ben Kurtovic
90f5bf65b3
Experiment with cibuildwheel
1年前
Ben Kurtovic
8cf2514836
More packaging cleanup
1年前
Ben Kurtovic
ed9d988db4
Update docs
1年前
Ben Kurtovic
96f51aad61
Add pyright pre-commit hook
1年前
Ben Kurtovic
17d993594d
Switch to pyproject.toml + uv
1年前
Ben Kurtovic
16819ac0ce
Fix docs config
1年前
Ben Kurtovic
4d8ac13e54
Version bump
1年前
Ben Kurtovic
374affe037
release/0.7.0
1年前
Ben Kurtovic
16a7761827
Update changelog
1年前
Ben Kurtovic
ec7f98ddb6
Encoding required on Windows
1年前
Ben Kurtovic
d46f047dcb
Fix build-wheels workflow
1年前
Ben Kurtovic
ab4caf3805
Fix a failing test
1年前