Ben Kurtovic
08cafc0576
Raise ParserError for internal problems. Improve coverage. Cleanup.
pirms 10 gadiem
Ben Kurtovic
5c5fd6b3cb
Fix a bug involving nested links ( closes #61 and #62 ).
pirms 10 gadiem
Ben Kurtovic
e5f17eea00
Update copyright notices for 2014.
pirms 10 gadiem
Ben Kurtovic
1946cf621d
Add a temporary skip_style_tags until we resolve some issues.
pirms 10 gadiem
Ben Kurtovic
066049b46a
Update email address.
pirms 10 gadiem
Ben Kurtovic
38050f6878
C code cleanup and speed improvements.
pirms 10 gadiem
Ben Kurtovic
1bf9868753
Proper sentinel handling with free links in the C tokenizer.
pirms 10 gadiem
Ben Kurtovic
fcdc0abd22
Fix autofail contexts.
pirms 10 gadiem
Ben Kurtovic
2561cf5b5e
Fix all bugs in C implementation of external links.
pirms 10 gadiem
Ben Kurtovic
7dcfa3fe92
Implement Tokenizer_really_parse_external_link(), some other fixes
pirms 10 gadiem
Ben Kurtovic
6ecf15cad4
Tokenizer_parse_external_link()
pirms 10 gadiem
Ben Kurtovic
a1948b06aa
Tokenizer_parse_bracketed/free_uri_scheme(), other adjustments
pirms 10 gadiem
Ben Kurtovic
7b84b3f0df
Refactor out C's is_marker(); hooks for ext links.
pirms 10 gadiem
Ben Kurtovic
d42e05a554
Implement improved wikilink handling.
pirms 10 gadiem
Ben Kurtovic
5e6e5b6301
tag_defs.py -> definitions.py; more outline stuff
pirms 10 gadiem
Ben Kurtovic
cbf67c7842
Add hooks for some ext link stuff; add a INVALID_LINK aggregate context.
pirms 10 gadiem
Ben Kurtovic
8923d96a57
More unification.
pirms 10 gadiem
Ben Kurtovic
b5ec7f3beb
Fix py3k module importing; stick a bunch of macros in one place.
pirms 10 gadiem
Ben Kurtovic
25d53cacf8
Begin porting C tokenizer to Python 3.
pirms 10 gadiem
Ben Kurtovic
ebf99d722c
Combine emit()/emit_first() internally.
pirms 10 gadiem
Ben Kurtovic
51ac97de04
Make macros out of the failing/unsafe contexts.
pirms 10 gadiem
Ben Kurtovic
df9f7388b6
emit_FAST(), emit_first_FAST(); update comment parsing
pirms 10 gadiem
Ben Kurtovic
c20d3f2a6a
handle_list_marker() and handle_list()
pirms 10 gadiem
Ben Kurtovic
9b98907751
Add C hooks and prototypes for wiki-markup tags.
pirms 10 gadiem
Ben Kurtovic
4663563ce4
Remove unnecessary markers.
pirms 10 gadiem
Ben Kurtovic
e83f321340
Rearrange functions; remove useless prototypes.
pirms 10 gadiem
Ben Kurtovic
e3fc27c9e3
Refactor TagData code into dedicated functions.
pirms 10 gadiem
Ben Kurtovic
d02a6da81e
Implement Tokenizer_handle_tag_space(); refactor textbuffer writing.
- Add a test for very long strings of text.
pirms 10 gadiem
Ben Kurtovic
9365fcf6e4
Implement Tokenizer_handle_tag_data(); add a read-backwards macro.
pirms 10 gadiem
Ben Kurtovic
e636bf77cf
Implement Tokenizer_push_tag_buffer()
pirms 11 gadiem
Ben Kurtovic
653071379b
Finish porting misc changes; add prototypes for remaining functions.
pirms 11 gadiem
Ben Kurtovic
aca0f78cd7
Port more Python tokenizer updates to C.
pirms 11 gadiem
Ben Kurtovic
f67cf46900
Start C port of tag tokenization; refactor the init func.
pirms 11 gadiem
Ben
a689467577
Replace broken log2 function; add a missing comment.
pirms 11 gadiem
Ben Kurtovic
9ede1121ba
Fix tokenizer.c on Windows; add another template test ( #25 )
Mostly by @gdooms , with tweaks.
pirms 11 gadiem
Ben Kurtovic
debcb6577e
Fix recursion issues by giving up at a certain point ( closes #16 ).
- Stop parsing new templates if the template depth gets above
MAX_DEPTH (40) or if we've already tried to parse over MAX_CYCLES
(100,000) templates.
- Add two tests to ensure recursion works somewhat correctly.
- Fix parsing the string "{{" with the Python tokenizer; add a test.
pirms 11 gadiem
Ben Kurtovic
054a84afe0
A bit of misc cleanup.
pirms 11 gadiem
Ben Kurtovic
718fcb24c8
Fix eight failing tests; all template parsing tests now passing ( #25 ).
pirms 11 gadiem
Ben Kurtovic
5a0a00ba98
Change the way verify_safe() handles template params ( #25 ).
- Newlines are now allowed in template param names.
- Changes also affect handling of arguments like {{{foo}}}.
- Update unit tests: remove some unnecessary ones, and add some to cover the changes.
- Update StringMixIn tests to actually work for some of the methods.
- Update copyright notices for the C extensions.
pirms 11 gadiem
Ben Kurtovic
0803417901
Port CTokenizer's verify_safe method to Python to solve a failing test.
pirms 11 gadiem
Ben Kurtovic
d6f2723a06
Fix safety checks on template params in some odd cases ( closes #24 ).
Also, fix parsing of wikilinks in both tokenizers such that newlines
in any location within the title are an automatic failure.
pirms 11 gadiem
Ben Kurtovic
1f47e10f04
Start some Py3k compatibility stuff.
pirms 11 gadiem
Ben Kurtovic
4a725b7ac5
Fix another couple bugs regarding template contexts and verify_safe().
pirms 11 gadiem
Ben Kurtovic
06b20dd8c0
More updates.
pirms 11 gadiem
Ben Kurtovic
16024494cf
Mostly implement Tokenizer_really_parse_entity().
pirms 11 gadiem
Ben Kurtovic
7de34ed5a6
Rewrite verify_safe completely.
pirms 11 gadiem
Ben Kurtovic
cdef073a9b
Replace textbuffer with our own datastructure.
pirms 11 gadiem
Ben Kurtovic
f0a36f3262
Store tokens in a global variable instead of retrieving anew each time.
pirms 11 gadiem
Ben Kurtovic
6d73eeeab1
Replace Python list of stacks with a singly linked list.
pirms 11 gadiem
Ben Kurtovic
ff0b4439f8
Replace Python exception machinery with a global and some macros.
pirms 11 gadiem