Ben Kurtovic
22e869b142
Fix a failing HTML entity test in the C tokenizer.
Remove some extraneous whitespace in string_mixin.py.
il y a 11 ans
Ben Kurtovic
496475c977
Whoops, that should be one larger ( #25 ).
il y a 11 ans
Ben Kurtovic
9ede1121ba
Fix tokenizer.c on Windows; add another template test ( #25 )
Mostly by @gdooms , with tweaks.
il y a 11 ans
Ben Kurtovic
debcb6577e
Fix recursion issues by giving up at a certain point ( closes #16 ).
- Stop parsing new templates if the template depth gets above
MAX_DEPTH (40) or if we've already tried to parse over MAX_CYCLES
(100,000) templates.
- Add two tests to ensure recursion works somewhat correctly.
- Fix parsing the string "{{" with the Python tokenizer; add a test.
il y a 11 ans
Ben Kurtovic
f803269514
Add a USES_C field to the tokenizers; add TestParser.test_use_c()
il y a 11 ans
Ben Kurtovic
054a84afe0
A bit of misc cleanup.
il y a 11 ans
Ben Kurtovic
718fcb24c8
Fix eight failing tests; all template parsing tests now passing ( #25 ).
il y a 11 ans
Ben Kurtovic
5a0a00ba98
Change the way verify_safe() handles template params ( #25 ).
- Newlines are now allowed in template param names.
- Changes also affect handling of arguments like {{{foo}}}.
- Update unit tests: remove some unnecessary ones, and add some to cover the changes.
- Update StringMixIn tests to actually work for some of the methods.
- Update copyright notices for the C extensions.
il y a 11 ans
Ben Kurtovic
0803417901
Port CTokenizer's verify_safe method to Python to solve a failing test.
il y a 11 ans
Ben Kurtovic
d6f2723a06
Fix safety checks on template params in some odd cases ( closes #24 ).
Also, fix parsing of wikilinks in both tokenizers such that newlines
in any location within the title are an automatic failure.
il y a 11 ans
Ben Kurtovic
11cf5def75
Fix handling of sections headers with equal signs ( closes #20 )
il y a 11 ans
Ben Kurtovic
dab6cc39ef
Fix handling of errors when parsing template param keys ( closes #13 )
il y a 11 ans
Ben Kurtovic
1f47e10f04
Start some Py3k compatibility stuff.
il y a 11 ans
Ben Kurtovic
ca11d77efd
Shorten some longer lines.
il y a 11 ans
Ben Kurtovic
4a725b7ac5
Fix another couple bugs regarding template contexts and verify_safe().
il y a 11 ans
Ben Kurtovic
1abdb478c9
Fix a couple bugs.
il y a 11 ans
Ben Kurtovic
88a2b95030
Remove a bunch of white space.
il y a 11 ans
Ben Kurtovic
3a6335d969
Clean up init_tokenizer().
il y a 11 ans
Ben Kurtovic
06f02b9753
Clean up whitespace / newlines.
il y a 11 ans
Ben Kurtovic
0e78571672
Forgot to check numeric/hexadecimal entities for string chars.
il y a 11 ans
Ben Kurtovic
a1e6dfcef6
Ensure that there is null terminator at the end of entitydefs.
il y a 11 ans
Ben Kurtovic
19e7c3b6ac
Fix type.
il y a 11 ans
Ben Kurtovic
06b20dd8c0
More updates.
il y a 11 ans
Ben Kurtovic
16024494cf
Mostly implement Tokenizer_really_parse_entity().
il y a 11 ans
Ben Kurtovic
087d606097
Recover failing contexts when due to a template or link opening.
il y a 11 ans
Ben Kurtovic
7de34ed5a6
Rewrite verify_safe completely.
il y a 11 ans
Ben Kurtovic
b2f933dddc
Fix memory leak.
il y a 11 ans
Ben Kurtovic
c5fea8ea34
Render the entire textbuffer correctly.
il y a 11 ans
Ben Kurtovic
cdef073a9b
Replace textbuffer with our own datastructure.
il y a 11 ans
Ben Kurtovic
f0a36f3262
Store tokens in a global variable instead of retrieving anew each time.
il y a 11 ans
Ben Kurtovic
6d73eeeab1
Replace Python list of stacks with a singly linked list.
il y a 11 ans
Ben Kurtovic
d2868731e8
Update some abstract protocols.
il y a 11 ans
Ben Kurtovic
1bfd364a4a
Process contexts in Tokenizer_handle_template_param() correctly.
il y a 11 ans
Ben Kurtovic
230c89711b
Fix verify_save and propogate errors correctly in the parse func.
il y a 11 ans
Ben Kurtovic
467aef3651
FromStringAndSize instead of FromString.
il y a 11 ans
Ben Kurtovic
29ef2a6682
Propogate fix from b447e7bfc1
il y a 11 ans
Ben Kurtovic
ff0b4439f8
Replace Python exception machinery with a global and some macros.
il y a 11 ans
Ben Kurtovic
8e4819ea37
Fix exception implementation; some other cleanup and fixes.
il y a 11 ans
Ben Kurtovic
9993f1ba76
Fix object calls.
il y a 11 ans
Ben Kurtovic
0be18bc5b8
Fix Py_UNICODE comparisons.
il y a 11 ans
Ben Kurtovic
17a09e395a
Fix some bugs.
il y a 11 ans
Ben Kurtovic
c50de647e2
Implement Tokenizer_handle_heading_end() and malloc for HeadingData.
il y a 11 ans
Ben Kurtovic
48188bfa99
Implement Tokenizer_parse_heading(), some other stuff.
il y a 11 ans
Ben Kurtovic
150f331129
Implement Tokenizer_parse_entity(), Tokenizer_parse_comment().
il y a 11 ans
Ben Kurtovic
7c29a2a65e
Implement Tokenizer_handle_wikilink_separator()/_end().
il y a 11 ans
Ben Kurtovic
707ecc3837
Implement Tokenizer_parse_wikilink() and more.
il y a 11 ans
Ben Kurtovic
f401ede179
Implementing more stuff.
il y a 11 ans
Ben Kurtovic
41535992a1
Implement Tokenizer_handle_template_param_value().
il y a 11 ans
Ben Kurtovic
17af353fb6
Implement Tokenizer_handle_template_param().
il y a 11 ans
Ben Kurtovic
849016f734
Implement Tokenizer_verify_safe() and some others.
il y a 11 ans