A Python parser for MediaWiki wikicode https://mwparserfromhell.readthedocs.io/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

README.rst 8.2 KiB

12 years ago
12 years ago
12 years ago
12 years ago
11 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228
  1. mwparserfromhell
  2. ================
  3. .. image:: https://img.shields.io/coveralls/earwig/mwparserfromhell/main.svg
  4. :alt: Coverage Status
  5. :target: https://coveralls.io/r/earwig/mwparserfromhell
  6. **mwparserfromhell** (the *MediaWiki Parser from Hell*) is a Python package
  7. that provides an easy-to-use and outrageously powerful parser for MediaWiki_
  8. wikicode. It supports Python 3.8+.
  9. Developed by Earwig_ with contributions from `Σ`_, Legoktm_, and others.
  10. Full documentation is available on ReadTheDocs_. Development occurs on GitHub_.
  11. Installation
  12. ------------
  13. The easiest way to install the parser is through the `Python Package Index`_;
  14. you can install the latest release with ``pip install mwparserfromhell``
  15. (`get pip`_). Make sure your pip is up-to-date first, especially on Windows.
  16. Alternatively, get the latest development version:
  17. .. code-block:: sh
  18. git clone https://github.com/earwig/mwparserfromhell.git
  19. cd mwparserfromhell
  20. python setup.py install
  21. The comprehensive unit testing suite requires `pytest`_ (``pip install pytest``)
  22. and can be run with ``python -m pytest``.
  23. Usage
  24. -----
  25. Normal usage is rather straightforward (where ``text`` is page text):
  26. .. code-block:: python
  27. >>> import mwparserfromhell
  28. >>> wikicode = mwparserfromhell.parse(text)
  29. ``wikicode`` is a ``mwparserfromhell.Wikicode`` object, which acts like an
  30. ordinary ``str`` object with some extra methods. For example:
  31. .. code-block:: python
  32. >>> text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?"
  33. >>> wikicode = mwparserfromhell.parse(text)
  34. >>> print(wikicode)
  35. I has a template! {{foo|bar|baz|eggs=spam}} See it?
  36. >>> templates = wikicode.filter_templates()
  37. >>> print(templates)
  38. ['{{foo|bar|baz|eggs=spam}}']
  39. >>> template = templates[0]
  40. >>> print(template.name)
  41. foo
  42. >>> print(template.params)
  43. ['bar', 'baz', 'eggs=spam']
  44. >>> print(template.get(1).value)
  45. bar
  46. >>> print(template.get("eggs").value)
  47. spam
  48. Since nodes can contain other nodes, getting nested templates is trivial:
  49. .. code-block:: python
  50. >>> text = "{{foo|{{bar}}={{baz|{{spam}}}}}}"
  51. >>> mwparserfromhell.parse(text).filter_templates()
  52. ['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}']
  53. You can also pass ``recursive=False`` to ``filter_templates()`` and explore
  54. templates manually. This is possible because nodes can contain additional
  55. ``Wikicode`` objects:
  56. .. code-block:: python
  57. >>> code = mwparserfromhell.parse("{{foo|this {{includes a|template}}}}")
  58. >>> print(code.filter_templates(recursive=False))
  59. ['{{foo|this {{includes a|template}}}}']
  60. >>> foo = code.filter_templates(recursive=False)[0]
  61. >>> print(foo.get(1).value)
  62. this {{includes a|template}}
  63. >>> print(foo.get(1).value.filter_templates()[0])
  64. {{includes a|template}}
  65. >>> print(foo.get(1).value.filter_templates()[0].get(1).value)
  66. template
  67. Templates can be easily modified to add, remove, or alter params. ``Wikicode``
  68. objects can be treated like lists, with ``append()``, ``insert()``,
  69. ``remove()``, ``replace()``, and more. They also have a ``matches()`` method
  70. for comparing page or template names, which takes care of capitalization and
  71. whitespace:
  72. .. code-block:: python
  73. >>> text = "{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}"
  74. >>> code = mwparserfromhell.parse(text)
  75. >>> for template in code.filter_templates():
  76. ... if template.name.matches("Cleanup") and not template.has("date"):
  77. ... template.add("date", "July 2012")
  78. ...
  79. >>> print(code)
  80. {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{uncategorized}}
  81. >>> code.replace("{{uncategorized}}", "{{bar-stub}}")
  82. >>> print(code)
  83. {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
  84. >>> print(code.filter_templates())
  85. ['{{cleanup|date=July 2012}}', '{{bar-stub}}']
  86. You can then convert ``code`` back into a regular ``str`` object (for
  87. saving the page!) by calling ``str()`` on it:
  88. .. code-block:: python
  89. >>> text = str(code)
  90. >>> print(text)
  91. {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
  92. >>> text == code
  93. True
  94. Limitations
  95. -----------
  96. While the MediaWiki parser generates HTML and has access to the contents of
  97. templates, among other things, mwparserfromhell acts as a direct interface to
  98. the source code only. This has several implications:
  99. * Syntax elements produced by a template transclusion cannot be detected. For
  100. example, imagine a hypothetical page ``"Template:End-bold"`` that contained
  101. the text ``</b>``. While MediaWiki would correctly understand that
  102. ``<b>foobar{{end-bold}}`` translates to ``<b>foobar</b>``, mwparserfromhell
  103. has no way of examining the contents of ``{{end-bold}}``. Instead, it would
  104. treat the bold tag as unfinished, possibly extending further down the page.
  105. * Templates adjacent to external links, as in ``http://example.com{{foo}}``,
  106. are considered part of the link. In reality, this would depend on the
  107. contents of the template.
  108. * When different syntax elements cross over each other, as in
  109. ``{{echo|''Hello}}, world!''``, the parser gets confused because this cannot
  110. be represented by an ordinary syntax tree. Instead, the parser will treat the
  111. first syntax construct as plain text. In this case, only the italic tag would
  112. be properly parsed.
  113. **Workaround:** Since this commonly occurs with text formatting and text
  114. formatting is often not of interest to users, you may pass
  115. *skip_style_tags=True* to ``mwparserfromhell.parse()``. This treats ``''``
  116. and ``'''`` as plain text.
  117. A future version of mwparserfromhell may include multiple parsing modes to
  118. get around this restriction more sensibly.
  119. Additionally, the parser lacks awareness of certain wiki-specific settings:
  120. * `Word-ending links`_ are not supported, since the linktrail rules are
  121. language-specific.
  122. * Localized namespace names aren't recognized, so file links (such as
  123. ``[[File:...]]``) are treated as regular wikilinks.
  124. * Anything that looks like an XML tag is treated as a tag, even if it is not a
  125. recognized tag name, since the list of valid tags depends on loaded MediaWiki
  126. extensions.
  127. Integration
  128. -----------
  129. ``mwparserfromhell`` is used by and originally developed for EarwigBot_;
  130. ``Page`` objects have a ``parse`` method that essentially calls
  131. ``mwparserfromhell.parse()`` on ``page.get()``.
  132. If you're using Pywikibot_, your code might look like this:
  133. .. code-block:: python
  134. import mwparserfromhell
  135. import pywikibot
  136. def parse(title):
  137. site = pywikibot.Site()
  138. page = pywikibot.Page(site, title)
  139. text = page.get()
  140. return mwparserfromhell.parse(text)
  141. If you're not using a library, you can parse any page with the following
  142. Python 3 code (using the API_ and the requests_ library):
  143. .. code-block:: python
  144. import mwparserfromhell
  145. import requests
  146. API_URL = "https://en.wikipedia.org/w/api.php"
  147. def parse(title):
  148. params = {
  149. "action": "query",
  150. "prop": "revisions",
  151. "rvprop": "content",
  152. "rvslots": "main",
  153. "rvlimit": 1,
  154. "titles": title,
  155. "format": "json",
  156. "formatversion": "2",
  157. }
  158. headers = {"User-Agent": "My-Bot-Name/1.0"}
  159. req = requests.get(API_URL, headers=headers, params=params)
  160. res = req.json()
  161. revision = res["query"]["pages"][0]["revisions"][0]
  162. text = revision["slots"]["main"]["content"]
  163. return mwparserfromhell.parse(text)
  164. .. _MediaWiki: https://www.mediawiki.org
  165. .. _ReadTheDocs: https://mwparserfromhell.readthedocs.io
  166. .. _Earwig: https://en.wikipedia.org/wiki/User:The_Earwig
  167. .. _Σ: https://en.wikipedia.org/wiki/User:%CE%A3
  168. .. _Legoktm: https://en.wikipedia.org/wiki/User:Legoktm
  169. .. _GitHub: https://github.com/earwig/mwparserfromhell
  170. .. _Python Package Index: https://pypi.org/
  171. .. _get pip: https://pypi.org/project/pip/
  172. .. _pytest: https://docs.pytest.org/
  173. .. _Word-ending links: https://www.mediawiki.org/wiki/Help:Links#linktrail
  174. .. _EarwigBot: https://github.com/earwig/earwigbot
  175. .. _Pywikibot: https://www.mediawiki.org/wiki/Manual:Pywikibot
  176. .. _API: https://www.mediawiki.org/wiki/API:Main_page
  177. .. _requests: https://2.python-requests.org