A Python parser for MediaWiki wikicode https://mwparserfromhell.readthedocs.io/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Ben Kurtovic b354c56755 Version bump; cleanup [ci skip] 6 years ago
docs Version bump; cleanup [ci skip] 6 years ago
mwparserfromhell Version bump; cleanup [ci skip] 6 years ago
scripts Version bump; cleanup [ci skip] 6 years ago
tests Fix test on old Python versions 6 years ago
.coveragerc Solve a couple more coverage issues; tighten. 10 years ago
.gitignore Raise ParserError for internal problems. Improve coverage. Cleanup. 10 years ago
.travis.yml Try Python 3.6 in Travis (#173) 7 years ago
CHANGELOG Version bump; cleanup [ci skip] 6 years ago
LICENSE Don't get stuck in tags with unclosed quoted attributes (fixes #190). 6 years ago
MANIFEST.in Fix some issues during 0.4.1 release. 9 years ago
README.rst Version bump; cleanup [ci skip] 6 years ago
appveyor.yml Version bump; cleanup [ci skip] 6 years ago
setup.py Add Python 3.7 to trove classifiers. 6 years ago



.. image:: https://img.shields.io/travis/earwig/mwparserfromhell/develop.svg
:alt: Build Status
:target: http://travis-ci.org/earwig/mwparserfromhell

.. image:: https://img.shields.io/coveralls/earwig/mwparserfromhell/develop.svg
:alt: Coverage Status
:target: https://coveralls.io/r/earwig/mwparserfromhell

**mwparserfromhell** (the *MediaWiki Parser from Hell*) is a Python package
that provides an easy-to-use and outrageously powerful parser for MediaWiki_
wikicode. It supports Python 2 and Python 3.

Developed by Earwig_ with contributions from `Σ`_, Legoktm_, and others.
Full documentation is available on ReadTheDocs_. Development occurs on GitHub_.


The easiest way to install the parser is through the `Python Package Index`_;
you can install the latest release with ``pip install mwparserfromhell``
(`get pip`_). Make sure your pip is up-to-date first, especially on Windows.

Alternatively, get the latest development version::

git clone https://github.com/earwig/mwparserfromhell.git
cd mwparserfromhell
python setup.py install

You can run the comprehensive unit testing suite with
``python setup.py test -q``.


Normal usage is rather straightforward (where ``text`` is page text)::

>>> import mwparserfromhell
>>> wikicode = mwparserfromhell.parse(text)

``wikicode`` is a ``mwparserfromhell.Wikicode`` object, which acts like an
ordinary ``str`` object (or ``unicode`` in Python 2) with some extra methods.
For example::

>>> text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?"
>>> wikicode = mwparserfromhell.parse(text)
>>> print(wikicode)
I has a template! {{foo|bar|baz|eggs=spam}} See it?
>>> templates = wikicode.filter_templates()
>>> print(templates)
>>> template = templates[0]
>>> print(template.name)
>>> print(template.params)
['bar', 'baz', 'eggs=spam']
>>> print(template.get(1).value)
>>> print(template.get("eggs").value)

Since nodes can contain other nodes, getting nested templates is trivial::

>>> text = "{{foo|{{bar}}={{baz|{{spam}}}}}}"
>>> mwparserfromhell.parse(text).filter_templates()
['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}']

You can also pass ``recursive=False`` to ``filter_templates()`` and explore
templates manually. This is possible because nodes can contain additional
``Wikicode`` objects::

>>> code = mwparserfromhell.parse("{{foo|this {{includes a|template}}}}")
>>> print(code.filter_templates(recursive=False))
['{{foo|this {{includes a|template}}}}']
>>> foo = code.filter_templates(recursive=False)[0]
>>> print(foo.get(1).value)
this {{includes a|template}}
>>> print(foo.get(1).value.filter_templates()[0])
{{includes a|template}}
>>> print(foo.get(1).value.filter_templates()[0].get(1).value)

Templates can be easily modified to add, remove, or alter params. ``Wikicode``
objects can be treated like lists, with ``append()``, ``insert()``,
``remove()``, ``replace()``, and more. They also have a ``matches()`` method
for comparing page or template names, which takes care of capitalization and

>>> text = "{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}"
>>> code = mwparserfromhell.parse(text)
>>> for template in code.filter_templates():
... if template.name.matches("Cleanup") and not template.has("date"):
... template.add("date", "July 2012")
>>> print(code)
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{uncategorized}}
>>> code.replace("{{uncategorized}}", "{{bar-stub}}")
>>> print(code)
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
>>> print(code.filter_templates())
['{{cleanup|date=July 2012}}', '{{bar-stub}}']

You can then convert ``code`` back into a regular ``str`` object (for
saving the page!) by calling ``str()`` on it::

>>> text = str(code)
>>> print(text)
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
>>> text == code

Likewise, use ``unicode(code)`` in Python 2.


While the MediaWiki parser generates HTML and has access to the contents of
templates, among other things, mwparserfromhell acts as a direct interface to
the source code only. This has several implications:

* Syntax elements produced by a template transclusion cannot be detected. For
example, imagine a hypothetical page ``"Template:End-bold"`` that contained
the text ``</b>``. While MediaWiki would correctly understand that
``<b>foobar{{end-bold}}`` translates to ``<b>foobar</b>``, mwparserfromhell
has no way of examining the contents of ``{{end-bold}}``. Instead, it would
treat the bold tag as unfinished, possibly extending further down the page.

* Templates adjacent to external links, as in ``http://example.com{{foo}}``,
are considered part of the link. In reality, this would depend on the
contents of the template.

* When different syntax elements cross over each other, as in
``{{echo|''Hello}}, world!''``, the parser gets confused because this cannot
be represented by an ordinary syntax tree. Instead, the parser will treat the
first syntax construct as plain text. In this case, only the italic tag would
be properly parsed.

**Workaround:** Since this commonly occurs with text formatting and text
formatting is often not of interest to users, you may pass
*skip_style_tags=True* to ``mwparserfromhell.parse()``. This treats ``''``
and ``'''`` as plain text.

A future version of mwparserfromhell may include multiple parsing modes to
get around this restriction more sensibly.

Additionally, the parser lacks awareness of certain wiki-specific settings:

* `Word-ending links`_ are not supported, since the linktrail rules are

* Localized namespace names aren't recognized, so file links (such as
``[[File:...]]``) are treated as regular wikilinks.

* Anything that looks like an XML tag is treated as a tag, even if it is not a
recognized tag name, since the list of valid tags depends on loaded MediaWiki


``mwparserfromhell`` is used by and originally developed for EarwigBot_;
``Page`` objects have a ``parse`` method that essentially calls
``mwparserfromhell.parse()`` on ``page.get()``.

If you're using Pywikibot_, your code might look like this::

import mwparserfromhell
import pywikibot

def parse(title):
site = pywikibot.Site()
page = pywikibot.Page(site, title)
text = page.get()
return mwparserfromhell.parse(text)

If you're not using a library, you can parse any page using the following
Python 3 code (via the API_)::

import json
from urllib.parse import urlencode
from urllib.request import urlopen
import mwparserfromhell
API_URL = "https://en.wikipedia.org/w/api.php"

def parse(title):
data = {"action": "query", "prop": "revisions", "rvlimit": 1,
"rvprop": "content", "format": "json", "titles": title}
raw = urlopen(API_URL, urlencode(data).encode()).read()
res = json.loads(raw)
text = res["query"]["pages"].values()[0]["revisions"][0]["*"]
return mwparserfromhell.parse(text)

.. _MediaWiki: http://mediawiki.org
.. _ReadTheDocs: http://mwparserfromhell.readthedocs.io
.. _Earwig: http://en.wikipedia.org/wiki/User:The_Earwig
.. _Σ: http://en.wikipedia.org/wiki/User:%CE%A3
.. _Legoktm: http://en.wikipedia.org/wiki/User:Legoktm
.. _GitHub: https://github.com/earwig/mwparserfromhell
.. _Python Package Index: http://pypi.python.org
.. _get pip: http://pypi.python.org/pypi/pip
.. _Word-ending links: https://www.mediawiki.org/wiki/Help:Links#linktrail
.. _EarwigBot: https://github.com/earwig/earwigbot
.. _Pywikibot: https://www.mediawiki.org/wiki/Manual:Pywikibot
.. _API: http://mediawiki.org/wiki/API