David Winegar
0128b1f78a
Implement CTokenizer for tables
CTokenizer is completely implemented in this commit - it didn't
make much sense to me to split it up. All tests passing, memory test
shows no leaks on Linux.
10 years ago
David Winegar
2d945b30e5
Use uint64_t for context
For the C tokenizer, include `<stdint.h>` and use `uint64_t` instead
of `int` for context. Changes to tables mean that context can be
larger than 32 bits, and it is possible for `int` to only have 16
bits anyways (though this is very unlikely).
10 years ago
David Winegar
406dd3a157
All tokenizer end methods return a stack
For C compatability, switch table cell end to return the stack.
Now context is kept by using `keep_context` when calling `self._pop()`.
10 years ago
David Winegar
e6ec5dc4de
Refactor methods to avoid returning tuples
Various changes to avoid returning tuples - working on the C tokenizer
made me realize this was a bad idea for compatability/similarity between
the two.
10 years ago
David Winegar
151a73e437
Fix issue with incorrect table attributes
Fix problem in which invalid table attributes were being parsed
incorrectly. Added tests.
10 years ago
David Winegar
8b5d6f9a3b
Changes to table close handling
Fix problem in which fake table closes were causing a problem inside
cells. Changed inline table handling to fix this.
10 years ago
David Winegar
457b224045
Add padding to table cell tags
Padding now included on all wiki table cells. With wiki table cells
that include attributes, `wiki_markup` is also included (unchanged).
10 years ago
David Winegar
ddaa3ec92a
Reorder table tokenizer methods for forward declaration
Make sure py tokenizer methods only call methods that have been declared
earlier. Not necessary but makes it much easier to maintain/write
the C tokenizer if methods are in the same order.
10 years ago
David Winegar
842af20c38
fixed hacky table cell style exception, added tests
Removed the `StopIteration()` exception for handling table style
and instead call `_handle_table_cell_end()` with a new parameter.
Also added some random tests for table openings.
10 years ago
David Winegar
f1664a8d67
Updated row and table handling
Changed row recursion handling to make sure the tag is emitted even
when hitting recursion limits. Need to test table recursion to make
sure that works. Also fixed a bug in which tables were eating the
trailing token. Added several tests for rows and trailing tokens with
tables.
10 years ago
David Winegar
ec08001871
Tables and rows now use newline as padding
Tables and rows use newlines as padding, partly because these characters
are pretty important to the integrity of the table. They might need
to be in the preceding whitespace of inner tags instead as padding after,
not sure.
10 years ago
David Winegar
9e4bb0c7e5
Clean up and style changes
Added comments, tried to keep to 80 character lines.
10 years ago
David Winegar
d356a570b3
Added closing_wiki_markup support to Tag node
Added support for allowing different wiki syntax for replacing the opening
and closing tags. Added for table support.
10 years ago
David Winegar
9f159ecfa2
Add table start/row start style attribute support
Started styling attributes for table row and table start. Still not entirely
sure about this, definitely need to make changes regarding padding.
10 years ago
David Winegar
0bba69d5dc
Added tests/support for header cells
Support for header cells was mostly in already, just needed minor
changes. Added two tests as well.
10 years ago
David Winegar
a13bc948fa
Started table cell attribute support
Started support for parsing table style attributes. I suspect some
of this is incorrect, need to add more tests to see.
10 years ago
David Winegar
b7e40d7b5a
Table cells now recurse
Added another stack layer for tokenizing table cells because of
styling/correctness of implementation. Added many tests cases.
10 years ago
David Winegar
a8d2983161
Started table parsing in PyTokenizer
Started parsing table support and added the start of table support.
This is a big commit (ugh) and it should probably be split up into
multiple smaller ones if possible, but that seems unworkable as of
right now because of all the dependencies. Also breaks tests of
CTokenizer (double ugh) because I haven't started table support there.
May want to pick line by line on this commit later but I need to save
my work for now.
10 years ago
Ben Kurtovic
87e0079512
Take proper advantage of Sphinx's default domains.
10 years ago
Ben Kurtovic
6954480263
Fix template parsing when comments are inside the name ( fixes #59 ).
10 years ago
Ben Kurtovic
871d48c688
Solve a couple more coverage issues; tighten.
10 years ago
Ben Kurtovic
ded89fb14e
Add a few unit tests for untested code; remove a useless conditional.
10 years ago
Ben Kurtovic
963cb2f780
Store builder handlers in a dictionary. ~5% speedup.
10 years ago
Ben Kurtovic
b997e4cd71
Support attributes quoted with '; add required quotes in value setter.
10 years ago
Ben Kurtovic
a4c2fd023a
Remove some useless code in the tokenizers.
10 years ago
Ben Kurtovic
08cafc0576
Raise ParserError for internal problems. Improve coverage. Cleanup.
10 years ago
Ben Kurtovic
02eff0fc49
Fully fix #74 . Add another tokenizer test.
10 years ago
Ben Kurtovic
34a6c7cc4c
Typo fix.
10 years ago
Ben Kurtovic
b4b62026f8
Update changelog, docs.
10 years ago
Ben Kurtovic
0101c038fb
Python 3, grr.
10 years ago
Ben Kurtovic
0497b54f03
Fix _handle_single_tag_end()'s token search order ( fixes #74 )
10 years ago
Ben Kurtovic
d342831af8
Allow passing skip_style_tags to parse() ( fixes #73 )
10 years ago
Marcio Faustino
f616e6b3b7
Avoid custom constructor and sub-class `dict` directly instead.
Before:
$ python -m timeit -s 'from mwparserfromhell.parser.tokens import Text' 'Text(text = "abc")'
1000000 loops, best of 3: 0.91 usec per loop
After:
$ python -m timeit -s 'from mwparserfromhell.parser.tokens import Text' 'Text(text = "abc")'
1000000 loops, best of 3: 0.223 usec per loop
10 years ago
Ben Kurtovic
5c5fd6b3cb
Fix a bug involving nested links ( closes #61 and #62 ).
10 years ago
Ben Kurtovic
1312a1fb8a
Some clean up for Python 2.6 support.
* Removed unittest2 dependency on Python >2.6.
* Moved discover_tests.py into tests/.
* tokenizer.c: Fixed errors noted by -Wshorten-64-to-32.
10 years ago
Marcio Faustino
ee194fb07a
Use a generator expression instead to support Python 2.6.
10 years ago
Ben Kurtovic
e5f17eea00
Update copyright notices for 2014.
10 years ago
Ben Kurtovic
45aec31629
Move range/xrange compatibility code to compat.py.
10 years ago
Ben Kurtovic
1946cf621d
Add a temporary skip_style_tags until we resolve some issues.
10 years ago
Ben Kurtovic
066049b46a
Update email address.
10 years ago
Ben Kurtovic
38050f6878
C code cleanup and speed improvements.
10 years ago
Ben Kurtovic
951a8737a5
Don't pass underlying context if this is a bracketed link.
10 years ago
Ben Kurtovic
287bf71158
Condense code.
10 years ago
Ben Kurtovic
1bf9868753
Proper sentinel handling with free links in the C tokenizer.
10 years ago
Ben Kurtovic
565a04256f
Proper sentinel handling with free links in the Python tokenizer.
10 years ago
Ben Kurtovic
77092e066c
Fix C tokenizer behavior re: some single_only tag edge cases.
10 years ago
Ben Kurtovic
6784ff73bf
Fix an edge case when we recurse too deeply.
10 years ago
Ben Kurtovic
bdfd0632b3
A couple very minor nitpicks.
10 years ago
Ben Kurtovic
4d04cae780
Fix a segfault with GCC.
10 years ago
Ben Kurtovic
fcdc0abd22
Fix autofail contexts.
10 years ago