A Python parser for MediaWiki wikicode https://mwparserfromhell.readthedocs.io/
Você não pode selecionar mais de 25 tópicos Os tópicos devem começar com uma letra ou um número, podem incluir traços ('-') e podem ter até 35 caracteres.

README.rst 8.0 KiB

12 anos atrás
10 anos atrás
11 anos atrás
10 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
11 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
7 anos atrás
12 anos atrás
12 anos atrás
12 anos atrás
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212
  1. mwparserfromhell
  2. ================
  3. .. image:: https://img.shields.io/travis/earwig/mwparserfromhell/develop.svg
  4. :alt: Build Status
  5. :target: http://travis-ci.org/earwig/mwparserfromhell
  6. .. image:: https://img.shields.io/coveralls/earwig/mwparserfromhell/develop.svg
  7. :alt: Coverage Status
  8. :target: https://coveralls.io/r/earwig/mwparserfromhell
  9. **mwparserfromhell** (the *MediaWiki Parser from Hell*) is a Python package
  10. that provides an easy-to-use and outrageously powerful parser for MediaWiki_
  11. wikicode. It supports Python 2 and Python 3.
  12. Developed by Earwig_ with contributions from `Σ`_, Legoktm_, and others.
  13. Full documentation is available on ReadTheDocs_. Development occurs on GitHub_.
  14. Installation
  15. ------------
  16. The easiest way to install the parser is through the `Python Package Index`_;
  17. you can install the latest release with ``pip install mwparserfromhell``
  18. (`get pip`_). Make sure your pip is up-to-date first, especially on Windows.
  19. Alternatively, get the latest development version::
  20. git clone https://github.com/earwig/mwparserfromhell.git
  21. cd mwparserfromhell
  22. python setup.py install
  23. You can run the comprehensive unit testing suite with
  24. ``python setup.py test -q``.
  25. Usage
  26. -----
  27. Normal usage is rather straightforward (where ``text`` is page text):
  28. >>> import mwparserfromhell
  29. >>> wikicode = mwparserfromhell.parse(text)
  30. ``wikicode`` is a ``mwparserfromhell.Wikicode`` object, which acts like an
  31. ordinary ``str`` object (or ``unicode`` in Python 2) with some extra methods.
  32. For example:
  33. >>> text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?"
  34. >>> wikicode = mwparserfromhell.parse(text)
  35. >>> print(wikicode)
  36. I has a template! {{foo|bar|baz|eggs=spam}} See it?
  37. >>> templates = wikicode.filter_templates()
  38. >>> print(templates)
  39. ['{{foo|bar|baz|eggs=spam}}']
  40. >>> template = templates[0]
  41. >>> print(template.name)
  42. foo
  43. >>> print(template.params)
  44. ['bar', 'baz', 'eggs=spam']
  45. >>> print(template.get(1).value)
  46. bar
  47. >>> print(template.get("eggs").value)
  48. spam
  49. Since nodes can contain other nodes, getting nested templates is trivial:
  50. >>> text = "{{foo|{{bar}}={{baz|{{spam}}}}}}"
  51. >>> mwparserfromhell.parse(text).filter_templates()
  52. ['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}']
  53. You can also pass ``recursive=False`` to ``filter_templates()`` and explore
  54. templates manually. This is possible because nodes can contain additional
  55. ``Wikicode`` objects:
  56. >>> code = mwparserfromhell.parse("{{foo|this {{includes a|template}}}}")
  57. >>> print(code.filter_templates(recursive=False))
  58. ['{{foo|this {{includes a|template}}}}']
  59. >>> foo = code.filter_templates(recursive=False)[0]
  60. >>> print(foo.get(1).value)
  61. this {{includes a|template}}
  62. >>> print(foo.get(1).value.filter_templates()[0])
  63. {{includes a|template}}
  64. >>> print(foo.get(1).value.filter_templates()[0].get(1).value)
  65. template
  66. Templates can be easily modified to add, remove, or alter params. ``Wikicode``
  67. objects can be treated like lists, with ``append()``, ``insert()``,
  68. ``remove()``, ``replace()``, and more. They also have a ``matches()`` method
  69. for comparing page or template names, which takes care of capitalization and
  70. whitespace:
  71. >>> text = "{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}"
  72. >>> code = mwparserfromhell.parse(text)
  73. >>> for template in code.filter_templates():
  74. ... if template.name.matches("Cleanup") and not template.has("date"):
  75. ... template.add("date", "July 2012")
  76. ...
  77. >>> print(code)
  78. {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{uncategorized}}
  79. >>> code.replace("{{uncategorized}}", "{{bar-stub}}")
  80. >>> print(code)
  81. {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
  82. >>> print(code.filter_templates())
  83. ['{{cleanup|date=July 2012}}', '{{bar-stub}}']
  84. You can then convert ``code`` back into a regular ``str`` object (for
  85. saving the page!) by calling ``str()`` on it:
  86. >>> text = str(code)
  87. >>> print(text)
  88. {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
  89. >>> text == code
  90. True
  91. Likewise, use ``unicode(code)`` in Python 2.
  92. Limitations
  93. -----------
  94. While the MediaWiki parser generates HTML and has access to the contents of
  95. templates, among other things, mwparserfromhell acts as a direct interface to
  96. the source code only. This has several implications:
  97. * Syntax elements produced by a template transclusion cannot be detected. For
  98. example, imagine a hypothetical page ``"Template:End-bold"`` that contained
  99. the text ``</b>``. While MediaWiki would correctly understand that
  100. ``<b>foobar{{end-bold}}`` translates to ``<b>foobar</b>``, mwparserfromhell
  101. has no way of examining the contents of ``{{end-bold}}``. Instead, it would
  102. treat the bold tag as unfinished, possibly extending further down the page.
  103. * Templates adjacent to external links, as in ``http://example.com{{foo}}``,
  104. are considered part of the link. In reality, this would depend on the
  105. contents of the template.
  106. * When different syntax elements cross over each other, as in
  107. ``{{echo|''Hello}}, world!''``, the parser gets confused because this cannot
  108. be represented by an ordinary syntax tree. Instead, the parser will treat the
  109. first syntax construct as plain text. In this case, only the italic tag would
  110. be properly parsed.
  111. **Workaround:** Since this commonly occurs with text formatting and text
  112. formatting is often not of interest to users, you may pass
  113. *skip_style_tags=True* to ``mwparserfromhell.parse()``. This treats ``''``
  114. and ``'''`` as plain text.
  115. A future version of mwparserfromhell may include multiple parsing modes to
  116. get around this restriction more sensibly.
  117. Additionally, the parser lacks awareness of certain wiki-specific settings:
  118. * `Word-ending links`_ are not supported, since the linktrail rules are
  119. language-specific.
  120. * Localized namespace names aren't recognized, so file links (such as
  121. ``[[File:...]]``) are treated as regular wikilinks.
  122. * Anything that looks like an XML tag is treated as a tag, even if it is not a
  123. recognized tag name, since the list of valid tags depends on loaded MediaWiki
  124. extensions.
  125. Integration
  126. -----------
  127. ``mwparserfromhell`` is used by and originally developed for EarwigBot_;
  128. ``Page`` objects have a ``parse`` method that essentially calls
  129. ``mwparserfromhell.parse()`` on ``page.get()``.
  130. If you're using Pywikibot_, your code might look like this:
  131. .. code-block:: python
  132. import mwparserfromhell
  133. import pywikibot
  134. def parse(title):
  135. site = pywikibot.Site()
  136. page = pywikibot.Page(site, title)
  137. text = page.get()
  138. return mwparserfromhell.parse(text)
  139. If you're not using a library, you can parse any page using the following
  140. Python 3 code (via the API_):
  141. .. code-block:: python
  142. import json
  143. from urllib.parse import urlencode
  144. from urllib.request import urlopen
  145. import mwparserfromhell
  146. API_URL = "https://en.wikipedia.org/w/api.php"
  147. def parse(title):
  148. data = {"action": "query", "prop": "revisions", "rvprop": "content",
  149. "rvslots": "main", "rvlimit": 1, "titles": title,
  150. "format": "json", "formatversion": "2"}
  151. raw = urlopen(API_URL, urlencode(data).encode()).read()
  152. res = json.loads(raw)
  153. revision = res["query"]["pages"][0]["revisions"][0]
  154. text = revision["slots"]["main"]["content"]
  155. return mwparserfromhell.parse(text)
  156. .. _MediaWiki: http://mediawiki.org
  157. .. _ReadTheDocs: http://mwparserfromhell.readthedocs.io
  158. .. _Earwig: http://en.wikipedia.org/wiki/User:The_Earwig
  159. .. _Σ: http://en.wikipedia.org/wiki/User:%CE%A3
  160. .. _Legoktm: http://en.wikipedia.org/wiki/User:Legoktm
  161. .. _GitHub: https://github.com/earwig/mwparserfromhell
  162. .. _Python Package Index: http://pypi.python.org
  163. .. _get pip: http://pypi.python.org/pypi/pip
  164. .. _Word-ending links: https://www.mediawiki.org/wiki/Help:Links#linktrail
  165. .. _EarwigBot: https://github.com/earwig/earwigbot
  166. .. _Pywikibot: https://www.mediawiki.org/wiki/Manual:Pywikibot
  167. .. _API: http://mediawiki.org/wiki/API