Merge branch 'develop' into feature/copyvios

Conflicts:
	earwigbot/wiki/constants.py
	earwigbot/wiki/copyvios/__init__.py
	earwigbot/wiki/page.py

.gitignore

@@ -1,19 +1,6 @@
-# Ignore python bytecode:
 *.pyc
-
-# Ignore bot-specific config file:
-config.json
-
-# Ignore logs directory:
-logs/
-
-# Ignore cookies file:
-.cookies
-
-# Ignore OS X's crud:
+*.egg
+*.egg-info
 .DS_Store
-
-# Ignore pydev's nonsense:
-.project
-.pydevproject
-.settings/
+build
+docs/_build

README.md

@@ -1,39 +0,0 @@
-[EarwigBot](http://en.wikipedia.org/wiki/User:EarwigBot) is a
-[Python](http://python.org/) robot that edits
-[Wikipedia](http://en.wikipedia.org/) and interacts with people over
-[IRC](http://en.wikipedia.org/wiki/Internet_Relay_Chat).
-
-# History
-
-Development began, based on the
-[Pywikipedia framework](http://pywikipediabot.sourceforge.net/), in early 2009.
-Approval for its fist task, a
-[copyright violation detector](http://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/EarwigBot_1),
-was carried out in May, and the bot has been running consistently ever since
-(with the exception of Jan/Feb 2011). It currently handles
-[several ongoing tasks](http://en.wikipedia.org/wiki/User:EarwigBot#Tasks),
-ranging from statistics generation to category cleanup, and on-demand tasks
-such as WikiProject template tagging. Since it started running, the bot has
-made over 45,000 edits.
-
-A project to rewrite it from scratch began in early April 2011, thus moving
-away from the Pywikipedia framework and allowing for less overall code, better
-integration between bot parts, and easier maintenance.
-
-# Installation
-
-## Dependencies
-
-EarwigBot uses the MySQL library
-[oursql](http://packages.python.org/oursql/) (>= 0.9.2) for communicating with
-MediaWiki databases, and some tasks use their own tables for storage.
-Additionally, the afc_history task uses
-[matplotlib](http://matplotlib.sourceforge.net/) and
-[numpy](http://numpy.scipy.org/) for graphing AfC statistics. Neither of these
-modules are required for the main bot itself.
-
-`earwigbot.wiki.copyright` requires access to a search engine for detecting
-copyright violations. Currently,
-[Yahoo! BOSS](http://developer.yahoo.com/search/boss/) is the only engine
-supported, and this requires
-[oauth2](https://github.com/simplegeo/python-oauth2).

README.rst

@@ -0,0 +1,205 @@
+EarwigBot
+=========
+
+EarwigBot_ is a Python_ robot that edits Wikipedia_ and interacts with people
+over IRC_. This file provides a basic overview of how to install and setup the
+bot; more detailed information is located in the ``docs/`` directory (available
+online at PyPI_).
+
+History
+-------
+
+Development began, based on the `Pywikipedia framework`_, in early 2009.
+Approval for its fist task, a `copyright violation detector`_, was carried out
+in May, and the bot has been running consistently ever since (with the
+exception of Jan/Feb 2011). It currently handles `several ongoing tasks`_
+ranging from statistics generation to category cleanup, and on-demand tasks
+such as WikiProject template tagging. Since it started running, the bot has
+made over 50,000 edits.
+
+A project to rewrite it from scratch began in early April 2011, thus moving
+away from the Pywikipedia framework and allowing for less overall code, better
+integration between bot parts, and easier maintenance.
+
+Installation
+------------
+
+This package contains the core ``earwigbot``, abstracted enough that it should
+be usable and customizable by anyone running a bot on a MediaWiki site. Since
+it is component-based, the IRC components can be disabled if desired. IRC
+commands and bot tasks specific to `my instance of EarwigBot`_ that I don't
+feel the average user will need are available from the repository
+`earwigbot-plugins`_.
+
+It's recommended to run the bot's unit tests before installing. Run ``python
+setup.py test`` from the project's root directory. Note that some
+tests require an internet connection, and others may take a while to run.
+Coverage is currently rather incomplete.
+
+Latest release (v0.1)
+~~~~~~~~~~~~~~~~~~~~~
+
+EarwigBot is available from the `Python Package Index`_, so you can install the
+latest release with ``pip install earwigbot`` (`get pip`_).
+
+You can also install it from source [1]_ directly::
+
+    curl -Lo earwigbot.tgz https://github.com/earwig/earwigbot/tarball/v0.1
+    tar -xf earwigbot.tgz
+    cd earwig-earwigbot-*
+    python setup.py install
+    cd ..
+    rm -r earwigbot.tgz earwig-earwigbot-*
+
+Development version
+~~~~~~~~~~~~~~~~~~~
+
+You can install the development version of the bot from ``git`` by using
+setuptools/distribute's ``develop`` command [1]_, probably on the ``develop``
+branch which contains (usually) working code. ``master`` contains the latest
+release. EarwigBot uses `git flow`_, so you're free to
+browse by tags or by new features (``feature/*`` branches)::
+
+    git clone git://github.com/earwig/earwigbot.git earwigbot
+    cd earwigbot
+    python setup.py develop
+
+Setup
+-----
+
+The bot stores its data in a "working directory", including its config file and
+databases. This is also the location where you will place custom IRC commands
+and bot tasks, which will be explained later. It doesn't matter where this
+directory is, as long as the bot can write to it.
+
+Start the bot with ``earwigbot path/to/working/dir``, or just ``earwigbot`` if
+the working directory is the current directory. It will notice that no
+``config.yml`` file exists and take you through the setup process.
+
+There is currently no way to edit the ``config.yml`` file from within the bot
+after it has been created, but YAML is a very straightforward format, so you
+should be able to make any necessary changes yourself. Check out the
+`explanation of YAML`_ on Wikipedia for help.
+
+After setup, the bot will start. This means it will connect to the IRC servers
+it has been configured for, schedule bot tasks to run at specific times, and
+then wait for instructions (as commands on IRC). For a list of commands, say
+"``!help``" (commands are messages prefixed with an exclamation mark).
+
+You can stop the bot at any time with Control+C, same as you stop a normal
+Python program, and it will try to exit safely. You can also use the
+"``!quit``" command on IRC.
+
+Customizing
+-----------
+
+The bot's working directory contains a ``commands`` subdirectory and a
+``tasks`` subdirectory. Custom IRC commands can be placed in the former,
+whereas custom wiki bot tasks go into the latter. Developing custom modules is
+explained below, and in more detail through the bot's documentation on PyPI_
+(or in the ``docs/`` dir).
+
+Note that custom commands will override built-in commands and tasks with the
+same name.
+
+``Bot`` and ``BotConfig``
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`earwigbot.bot.Bot`_ is EarwigBot's main class. You don't have to instantiate
+this yourself, but it's good to be familiar with its attributes and methods,
+because it is the main way to communicate with other parts of the bot. A
+``Bot`` object is accessible as an attribute of commands and tasks (i.e.,
+``self.bot``).
+
+`earwigbot.config.BotConfig`_ stores configuration information for the bot. Its
+docstring explains what each attribute is used for, but essentially each "node"
+(one of ``config.components``, ``wiki``, ``irc``, ``commands``, ``tasks``, and
+``metadata``) maps to a section of the bot's ``config.yml`` file. For example,
+if ``config.yml`` includes something like::
+
+    irc:
+        frontend:
+            nick: MyAwesomeBot
+            channels:
+                - "##earwigbot"
+                - "#channel"
+                - "#other-channel"
+
+...then ``config.irc["frontend"]["nick"]`` will be ``"MyAwesomeBot"`` and
+``config.irc["frontend"]["channels"]`` will be ``["##earwigbot", "#channel",
+"#other-channel"]``.
+
+Custom IRC commands
+~~~~~~~~~~~~~~~~~~~
+
+Custom commands are subclasses of `earwigbot.commands.Command`_ that override
+``Command``'s ``process()`` (and optionally ``check()`` or ``setup()``)
+methods.
+
+The bot has a wide selection of built-in commands and plugins to act as sample
+code and/or to give ideas. Start with test_, and then check out chanops_ and
+afc_status_ for some more complicated scripts.
+
+Custom bot tasks
+~~~~~~~~~~~~~~~~
+
+Custom tasks are subclasses of `earwigbot.tasks.Task`_ that override ``Task``'s
+``run()`` (and optionally ``setup()``) methods.
+
+See the built-in wikiproject_tagger_ task for a relatively straightforward
+task, or the afc_statistics_ plugin for a more complicated one.
+
+The Wiki Toolset
+----------------
+
+EarwigBot's answer to the `Pywikipedia framework`_ is the Wiki Toolset
+(``earwigbot.wiki``), which you will mainly access through ``bot.wiki``.
+
+``bot.wiki`` provides three methods for the management of Sites -
+``get_site()``, ``add_site()``, and ``remove_site()``. Sites are objects that
+simply represent a MediaWiki site. A single instance of EarwigBot (i.e. a
+single *working directory*) is expected to relate to a single site or group of
+sites using the same login info (like all WMF wikis with CentralAuth).
+
+Load your default site (the one that you picked during setup) with
+``site = bot.wiki.get_site()``.
+
+Not all aspects of the toolset are covered in the docs. Explore `its code and
+docstrings`_ to learn how to use it in a more hands-on fashion. For reference,
+``bot.wiki`` is an instance of ``earwigbot.wiki.SitesDB`` tied to the
+``sites.db`` file in the bot's working directory.
+
+Footnotes
+---------
+
+- Questions, comments, or suggestions about the documentation? `Let me know`_
+  so I can improve it for other people.
+
+.. [1] ``python setup.py install``/``develop`` may require root, or use the
+       ``--user`` switch to install for the current user only.
+
+.. _EarwigBot:                      http://en.wikipedia.org/wiki/User:EarwigBot
+.. _Python:                         http://python.org/
+.. _Wikipedia:                      http://en.wikipedia.org/
+.. _IRC:                            http://en.wikipedia.org/wiki/Internet_Relay_Chat
+.. _PyPI:                           http://packages.python.org/earwigbot
+.. _Pywikipedia framework:          http://pywikipediabot.sourceforge.net/
+.. _copyright violation detector:   http://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/EarwigBot_1
+.. _several ongoing tasks:          http://en.wikipedia.org/wiki/User:EarwigBot#Tasks
+.. _my instance of EarwigBot:       http://en.wikipedia.org/wiki/User:EarwigBot
+.. _earwigbot-plugins:              https://github.com/earwig/earwigbot-plugins
+.. _Python Package Index:           http://pypi.python.org
+.. _get pip:                        http://pypi.python.org/pypi/pip
+.. _git flow:                       http://nvie.com/posts/a-successful-git-branching-model/
+.. _explanation of YAML:            http://en.wikipedia.org/wiki/YAML
+.. _earwigbot.bot.Bot:              https://github.com/earwig/earwigbot/blob/develop/earwigbot/bot.py
+.. _earwigbot.config.BotConfig:     https://github.com/earwig/earwigbot/blob/develop/earwigbot/config.py
+.. _earwigbot.commands.Command:     https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/__init__.py
+.. _test:                           https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/test.py
+.. _chanops:                        https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/chanops.py
+.. _afc_status:                     https://github.com/earwig/earwigbot-plugins/blob/develop/commands/afc_status.py
+.. _earwigbot.tasks.Task:           https://github.com/earwig/earwigbot/blob/develop/earwigbot/tasks/__init__.py
+.. _wikiproject_tagger:             https://github.com/earwig/earwigbot/blob/develop/earwigbot/tasks/wikiproject_tagger.py
+.. _afc_statistics:                 https://github.com/earwig/earwigbot-plugins/blob/develop/tasks/afc_statistics.py
+.. _its code and docstrings:        https://github.com/earwig/earwigbot/tree/develop/earwigbot/wiki
+.. _Let me know:                    ben.kurtovic@verizon.net

bot.py

@@ -1,70 +0,0 @@
-#! /usr/bin/env python
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot
-
-This is a thin wrapper for EarwigBot's main bot code, specified by bot_script.
-The wrapper will automatically restart the bot when it shuts down (from
-!restart, for example). It requests the bot's password at startup and reuses it
-every time the bot restarts internally, so you do not need to re-enter the
-password after using !restart.
-
-For information about the bot as a whole, see the attached README.md file (in
-markdown format!), the docs/ directory, and the LICENSE file for licensing
-information. EarwigBot is released under the MIT license.
-"""
-from getpass import getpass
-from subprocess import Popen, PIPE
-from os import path
-from sys import executable
-from time import sleep
-
-import earwigbot
-
-bot_script = path.join(earwigbot.__path__[0], "runner.py")
-
-def main():
-    print "EarwigBot v{0}\n".format(earwigbot.__version__)
-
-    is_encrypted = earwigbot.config.config.load()
-    if is_encrypted:  # Passwords in the config file are encrypted
-        key = getpass("Enter key to unencrypt bot passwords: ")
-    else:
-        key = None
-
-    while 1:
-        bot = Popen([executable, bot_script], stdin=PIPE)
-        print >> bot.stdin, path.dirname(path.abspath(__file__))
-        if is_encrypted:
-            print >> bot.stdin, key
-        return_code = bot.wait()
-        if return_code == 1:
-            exit()  # Let critical exceptions in the subprocess cause us to
-                    # exit as well
-        else:
-            sleep(5)  # Sleep between bot runs following a non-critical
-                      # subprocess exit
-
-if __name__ == "__main__":
-    main()

docs/Makefile

@@ -0,0 +1,153 @@
+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+PAPER         =
+BUILDDIR      = _build
+
+# Internal variables.
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+
+.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
+
+help:
+	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  html       to make standalone HTML files"
+	@echo "  dirhtml    to make HTML files named index.html in directories"
+	@echo "  singlehtml to make a single large HTML file"
+	@echo "  pickle     to make pickle files"
+	@echo "  json       to make JSON files"
+	@echo "  htmlhelp   to make HTML files and a HTML help project"
+	@echo "  qthelp     to make HTML files and a qthelp project"
+	@echo "  devhelp    to make HTML files and a Devhelp project"
+	@echo "  epub       to make an epub"
+	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
+	@echo "  text       to make text files"
+	@echo "  man        to make manual pages"
+	@echo "  texinfo    to make Texinfo files"
+	@echo "  info       to make Texinfo files and run them through makeinfo"
+	@echo "  gettext    to make PO message catalogs"
+	@echo "  changes    to make an overview of all changed/added/deprecated items"
+	@echo "  linkcheck  to check all external links for integrity"
+	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
+
+clean:
+	-rm -rf $(BUILDDIR)/*
+
+html:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+dirhtml:
+	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+singlehtml:
+	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+	@echo
+	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+pickle:
+	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+	@echo
+	@echo "Build finished; now you can process the pickle files."
+
+json:
+	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+	@echo
+	@echo "Build finished; now you can process the JSON files."
+
+htmlhelp:
+	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+	@echo
+	@echo "Build finished; now you can run HTML Help Workshop with the" \
+	      ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+qthelp:
+	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+	@echo
+	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
+	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/EarwigBot.qhcp"
+	@echo "To view the help file:"
+	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/EarwigBot.qhc"
+
+devhelp:
+	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+	@echo
+	@echo "Build finished."
+	@echo "To view the help file:"
+	@echo "# mkdir -p $$HOME/.local/share/devhelp/EarwigBot"
+	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/EarwigBot"
+	@echo "# devhelp"
+
+epub:
+	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+	@echo
+	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+latex:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo
+	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+	@echo "Run \`make' in that directory to run these through (pdf)latex" \
+	      "(use \`make latexpdf' here to do that automatically)."
+
+latexpdf:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through pdflatex..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+text:
+	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+	@echo
+	@echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+man:
+	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+	@echo
+	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+texinfo:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo
+	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+	@echo "Run \`make' in that directory to run these through makeinfo" \
+	      "(use \`make info' here to do that automatically)."
+
+info:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo "Running Texinfo files through makeinfo..."
+	make -C $(BUILDDIR)/texinfo info
+	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+gettext:
+	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+	@echo
+	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+changes:
+	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+	@echo
+	@echo "The overview file is in $(BUILDDIR)/changes."
+
+linkcheck:
+	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	@echo
+	@echo "Link check complete; look for any errors in the above output " \
+	      "or in $(BUILDDIR)/linkcheck/output.txt."
+
+doctest:
+	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+	@echo "Testing of doctests in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/doctest/output.txt."

docs/api/earwigbot.commands.rst

@@ -0,0 +1,9 @@
+commands Package
+================
+
+:mod:`commands` Package
+-----------------------
+
+.. automodule:: earwigbot.commands
+    :members:
+    :undoc-members:

docs/api/earwigbot.irc.rst

@@ -0,0 +1,46 @@
+irc Package
+===========
+
+:mod:`irc` Package
+------------------
+
+.. automodule:: earwigbot.irc
+    :members:
+    :undoc-members:
+
+:mod:`connection` Module
+------------------------
+
+.. automodule:: earwigbot.irc.connection
+    :members:
+    :undoc-members:
+
+:mod:`data` Module
+------------------
+
+.. automodule:: earwigbot.irc.data
+    :members:
+    :undoc-members:
+
+:mod:`frontend` Module
+----------------------
+
+.. automodule:: earwigbot.irc.frontend
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`rc` Module
+----------------
+
+.. automodule:: earwigbot.irc.rc
+    :members:
+    :undoc-members:
+
+:mod:`watcher` Module
+---------------------
+
+.. automodule:: earwigbot.irc.watcher
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/api/earwigbot.rst

@@ -0,0 +1,56 @@
+earwigbot Package
+=================
+
+:mod:`earwigbot` Package
+------------------------
+
+.. automodule:: earwigbot.__init__
+    :members:
+    :undoc-members:
+
+:mod:`bot` Module
+-----------------
+
+.. automodule:: earwigbot.bot
+    :members:
+    :undoc-members:
+
+:mod:`config` Module
+--------------------
+
+.. automodule:: earwigbot.config
+    :members:
+    :undoc-members:
+
+:mod:`exceptions` Module
+------------------------
+
+.. automodule:: earwigbot.exceptions
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`managers` Module
+----------------------
+
+.. automodule:: earwigbot.managers
+    :members: _ResourceManager, CommandManager, TaskManager
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`util` Module
+------------------
+
+.. automodule:: earwigbot.util
+    :members:
+    :undoc-members:
+
+Subpackages
+-----------
+
+.. toctree::
+
+    earwigbot.commands
+    earwigbot.irc
+    earwigbot.tasks
+    earwigbot.wiki

docs/api/earwigbot.tasks.rst

@@ -0,0 +1,9 @@
+tasks Package
+=============
+
+:mod:`tasks` Package
+--------------------
+
+.. automodule:: earwigbot.tasks
+    :members:
+    :undoc-members:

docs/api/earwigbot.wiki.rst

@@ -0,0 +1,59 @@
+wiki Package
+============
+
+:mod:`wiki` Package
+-------------------
+
+.. automodule:: earwigbot.wiki
+    :members:
+    :undoc-members:
+
+:mod:`category` Module
+----------------------
+
+.. automodule:: earwigbot.wiki.category
+    :members:
+    :undoc-members:
+
+:mod:`constants` Module
+-----------------------
+
+.. automodule:: earwigbot.wiki.constants
+    :members:
+    :undoc-members:
+
+:mod:`copyright` Module
+-----------------------
+
+.. automodule:: earwigbot.wiki.copyright
+    :members:
+    :undoc-members:
+
+:mod:`page` Module
+------------------
+
+.. automodule:: earwigbot.wiki.page
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+:mod:`site` Module
+------------------
+
+.. automodule:: earwigbot.wiki.site
+    :members:
+    :undoc-members:
+
+:mod:`sitesdb` Module
+---------------------
+
+.. automodule:: earwigbot.wiki.sitesdb
+    :members:
+    :undoc-members:
+
+:mod:`user` Module
+------------------
+
+.. automodule:: earwigbot.wiki.user
+    :members:
+    :undoc-members:

docs/api/modules.rst

@@ -0,0 +1,7 @@
+earwigbot
+=========
+
+.. toctree::
+   :maxdepth: 4
+
+   earwigbot

docs/conf.py

@@ -0,0 +1,242 @@
+# -*- coding: utf-8 -*-
+#
+# EarwigBot documentation build configuration file, created by
+# sphinx-quickstart on Sun Apr 29 01:42:25 2012.
+#
+# This file is execfile()d with the current directory set to its containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys, os
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath('..'))
+
+# -- General configuration -----------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be extensions
+# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.coverage', 'sphinx.ext.viewcode']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix of source filenames.
+source_suffix = '.rst'
+
+# The encoding of source files.
+#source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'EarwigBot'
+copyright = u'2009, 2010, 2011, 2012 by Ben Kurtovic'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = '0.1'
+# The full version, including alpha/beta/rc tags.
+release = '0.1.dev'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#today = ''
+# Else, today_fmt is used as the format for a strftime call.
+#today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ['_build']
+
+# The reST default role (used for this markup: `text`) to use for all documents.
+#default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+#modindex_common_prefix = []
+
+
+# -- Options for HTML output ---------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+html_theme = 'nature'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+#html_theme_path = []
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+#html_title = None
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+#html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+#html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+#html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+#html_additional_pages = {}
+
+# If false, no module index is generated.
+#html_domain_indices = True
+
+# If false, no index is generated.
+#html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+#html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+#html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+#html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+#html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a <link> tag referring to it.  The value of this option must be the
+# base URL from which the finished HTML is served.
+#html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+#html_file_suffix = None
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'EarwigBotdoc'
+
+
+# -- Options for LaTeX output --------------------------------------------------
+
+latex_elements = {
+# The paper size ('letterpaper' or 'a4paper').
+#'papersize': 'letterpaper',
+
+# The font size ('10pt', '11pt' or '12pt').
+#'pointsize': '10pt',
+
+# Additional stuff for the LaTeX preamble.
+#'preamble': '',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title, author, documentclass [howto/manual]).
+latex_documents = [
+  ('index', 'EarwigBot.tex', u'EarwigBot Documentation',
+   u'Ben Kurtovic', 'manual'),
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+#latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+#latex_use_parts = False
+
+# If true, show page references after internal links.
+#latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+#latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+#latex_appendices = []
+
+# If false, no module index is generated.
+#latex_domain_indices = True
+
+
+# -- Options for manual page output --------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    ('index', 'earwigbot', u'EarwigBot Documentation',
+     [u'Ben Kurtovic'], 1)
+]
+
+# If true, show URL addresses after external links.
+#man_show_urls = False
+
+
+# -- Options for Texinfo output ------------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+  ('index', 'EarwigBot', u'EarwigBot Documentation',
+   u'Ben Kurtovic', 'EarwigBot', 'One line description of project.',
+   'Miscellaneous'),
+]
+
+# Documents to append as an appendix to all manuals.
+#texinfo_appendices = []
+
+# If false, no module index is generated.
+#texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+#texinfo_show_urls = 'footnote'

docs/customizing.rst

@@ -0,0 +1,240 @@
+Customizing
+===========
+
+The bot's working directory contains a :file:`commands` subdirectory and a
+:file:`tasks` subdirectory. Custom IRC commands can be placed in the former,
+whereas custom wiki bot tasks go into the latter. Developing custom modules is
+explained in detail in this documentation.
+
+Note that custom commands will override built-in commands and tasks with the
+same name.
+
+:py:class:`~earwigbot.bot.Bot` and :py:class:`~earwigbot.bot.BotConfig`
+-----------------------------------------------------------------------
+
+:py:class:`earwigbot.bot.Bot` is EarwigBot's main class. You don't have to
+instantiate this yourself, but it's good to be familiar with its attributes and
+methods, because it is the main way to communicate with other parts of the bot.
+A :py:class:`~earwigbot.bot.Bot` object is accessible as an attribute of
+commands and tasks (i.e., :py:attr:`self.bot`).
+
+The most useful attributes are:
+
+- :py:attr:`~earwigbot.bot.Bot.config`: an instance of
+  :py:class:`~earwigbot.config.BotConfig`, for accessing the bot's
+  configuration data (see below).
+
+- :py:attr:`~earwigbot.bot.Bot.commands`: the bot's
+  :py:class:`~earwigbot.managers.CommandManager`, which is used internally to
+  run IRC commands (through
+  :py:meth:`commands.call() <earwigbot.managers.CommandManager.call>`, which
+  you shouldn't have to use); you can safely reload all commands with
+  :py:meth:`commands.load() <earwigbot.managers._ResourceManager.load>`.
+
+- :py:attr:`~earwigbot.bot.Bot.tasks`: the bot's
+  :py:class:`~earwigbot.managers.TaskManager`, which can be used to start tasks
+  with :py:meth:`tasks.start(task_name, **kwargs)
+  <earwigbot.managers.TaskManager.start>`. :py:meth:`tasks.load()
+  <earwigbot.managers._ResourceManager.load>` can be used to safely reload all
+  tasks.
+
+- :py:attr:`~earwigbot.bot.Bot.frontend` /
+  :py:attr:`~earwigbot.bot.Bot.watcher`: instances of
+  :py:class:`earwigbot.irc.Frontend <earwigbot.irc.frontend.Frontend>` and
+  :py:class:`earwigbot.irc.Watcher <earwigbot.irc.watcher.Watcher>`,
+  respectively, which represent the bot's connections to these two servers; you
+  can, for example, send a message to the frontend with
+  :py:meth:`frontend.say(chan, msg)
+  <earwigbot.irc.connection.IRCConnection.say>` (more on communicating with IRC
+  below).
+
+- :py:attr:`~earwigbot.bot.Bot.wiki`: interface with the
+  :doc:`Wiki Toolset <toolset>`.
+
+- Finally, :py:meth:`~earwigbot.bot.Bot.restart` (restarts IRC components and
+  reloads config, commands, and tasks) and :py:meth:`~earwigbot.bot.Bot.stop`
+  can be used almost anywhere. Both take an optional "reason" that will be
+  logged and used as the quit message when disconnecting from IRC.
+
+:py:class:`earwigbot.config.BotConfig` stores configuration information for the
+bot. Its docstrings explains what each attribute is used for, but essentially
+each "node" (one of :py:attr:`config.components
+<earwigbot.config.BotConfig.components>`,
+:py:attr:`~earwigbot.config.BotConfig.wiki`,
+:py:attr:`~earwigbot.config.BotConfig.irc`,
+:py:attr:`~earwigbot.config.BotConfig.commands`,
+:py:attr:`~earwigbot.config.BotConfig.tasks`, or
+:py:attr:`~earwigbot.config.BotConfig.metadata`) maps to a section
+of the bot's :file:`config.yml` file. For example, if :file:`config.yml`
+includes something like::
+
+    irc:
+        frontend:
+            nick: MyAwesomeBot
+            channels:
+                - "##earwigbot"
+                - "#channel"
+                - "#other-channel"
+
+...then :py:attr:`config.irc["frontend"]["nick"]` will be ``"MyAwesomeBot"``
+and :py:attr:`config.irc["frontend"]["channels"]` will be
+``["##earwigbot", "#channel", "#other-channel"]``.
+
+Custom IRC commands
+-------------------
+
+Custom commands are subclasses of :py:class:`earwigbot.commands.Command` that
+override :py:class:`~earwigbot.commands.Command`'s
+:py:meth:`~earwigbot.commands.Command.process` (and optionally
+:py:meth:`~earwigbot.commands.Command.check` or
+:py:meth:`~earwigbot.commands.Command.setup`) methods.
+
+:py:class:`~earwigbot.commands.Command`'s docstrings should explain what each
+attribute and method is for and what they should be overridden with, but these
+are the basics:
+
+- Class attribute :py:attr:`~earwigbot.commands.Command.name` is the name of
+  the command. This must be specified.
+
+- Class attribute :py:attr:`~earwigbot.commands.Command.commands` is a list of
+  names that will trigger this command. It defaults to the command's
+  :py:attr:`~earwigbot.commands.Command.name`, but you can override it with
+  multiple names to serve as aliases. This is handled by the default
+  :py:meth:`~earwigbot.commands.Command.check` implementation (see below), so
+  if :py:meth:`~earwigbot.commands.Command.check` is overridden, this is
+  ignored by everything except the help_ command (so ``!help alias`` will
+  trigger help for the actual command).
+
+- Class attribute :py:attr:`~earwigbot.commands.Command.hooks` is a list of the
+  "IRC events" that this command might respond to. It defaults to ``["msg"]``,
+  but options include ``"msg_private"`` (for private messages only),
+  ``"msg_public"`` (for channel messages only), and ``"join"`` (for when a user
+  joins a channel). See the afc_status_ plugin for a command that responds to
+  other hook types.
+
+- Method :py:meth:`~earwigbot.commands.Command.setup` is called *once* with no
+  arguments immediately after the command is first loaded. Does nothing by
+  default; treat it like an :py:meth:`__init__` if you want
+  (:py:meth:`~earwigbot.tasks.Command.__init__` does things by default and a
+  dedicated setup method is often easier than overriding
+  :py:meth:`~earwigbot.tasks.Command.__init__` and using :py:obj:`super`).
+
+- Method :py:meth:`~earwigbot.commands.Command.check` is passed a
+  :py:class:`~earwigbot.irc.data.Data` object, and should return ``True`` if
+  you want to respond to this message, or ``False`` otherwise. The default
+  behavior is to return ``True`` only if :py:attr:`data.is_command` is ``True``
+  and :py:attr:`data.command` ``==``
+  :py:attr:`~earwigbot.commands.Command.name` (or :py:attr:`data.command
+  <earwigbot.irc.data.Data.command>` is in
+  :py:attr:`~earwigbot.commands.Command.commands` if that list is overriden;
+  see above), which is suitable for most cases. A possible reason for
+  overriding is if you want to do something in response to events from a
+  specific channel only. Note that by returning ``True``, you prevent any other
+  commands from responding to this message.
+
+- Method :py:meth:`~earwigbot.commands.Command.process` is passed the same
+  :py:class:`~earwigbot.irc.data.Data` object as
+  :py:meth:`~earwigbot.commands.Command.check`, but only if
+  :py:meth:`~earwigbot.commands.Command.check` returned ``True``. This is where
+  the bulk of your command goes. To respond to IRC messages, there are a number
+  of methods of :py:class:`~earwigbot.commands.Command` at your disposal. See
+  the test_ command for a simple example, or look in
+  :py:class:`~earwigbot.commands.Command`'s
+  :py:meth:`~earwigbot.commands.Command.__init__` method for the full list.
+
+  The most common ones are :py:meth:`say(chan_or_user, msg)
+  <earwigbot.irc.connection.IRCConnection.say>`, :py:meth:`reply(data, msg)
+  <earwigbot.irc.connection.IRCConnection.reply>` (convenience function; sends
+  a reply to the issuer of the command in the channel it was received),
+  :py:meth:`action(chan_or_user, msg)
+  <earwigbot.irc.connection.IRCConnection.action>`,
+  :py:meth:`notice(chan_or_user, msg)
+  <earwigbot.irc.connection.IRCConnection.notice>`, :py:meth:`join(chan)
+  <earwigbot.irc.connection.IRCConnection.join>`, and
+  :py:meth:`part(chan) <earwigbot.irc.connection.IRCConnection.part>`.
+
+Commands have access to :py:attr:`config.commands[command_name]` for config
+information, which is a node in :file:`config.yml` like every other attribute
+of :py:attr:`bot.config`. This can be used to store, for example, API keys or
+SQL connection info, so that these can be easily changed without modifying the
+command itself.
+
+The command *class* doesn't need a specific name, but it should logically
+follow the command's name. The filename doesn't matter, but it is recommended
+to match the command name for readability. Multiple command classes are allowed
+in one file.
+
+The bot has a wide selection of built-in commands and plugins to act as sample
+code and/or to give ideas. Start with test_, and then check out chanops_ and
+afc_status_ for some more complicated scripts.
+
+Custom bot tasks
+----------------
+
+Custom tasks are subclasses of :py:class:`earwigbot.tasks.Task` that
+override :py:class:`~earwigbot.tasks.Task`'s
+:py:meth:`~earwigbot.tasks.Task.run` (and optionally
+:py:meth:`~earwigbot.tasks.Task.setup`) methods.
+
+:py:class:`~earwigbot.tasks.Task`'s docstrings should explain what each
+attribute and method is for and what they should be overridden with, but these
+are the basics:
+
+- Class attribute :py:attr:`~earwigbot.tasks.Task.name` is the name of the
+  task. This must be specified.
+
+- Class attribute :py:attr:`~earwigbot.tasks.Task.number` can be used to store
+  an optional "task number", possibly for use in edit summaries (to be
+  generated with :py:meth:`~earwigbot.tasks.Task.make_summary`). For
+  example, EarwigBot's :py:attr:`config.wiki["summary"]` is
+  ``"([[WP:BOT|Bot]]; [[User:EarwigBot#Task $1|Task $1]]): $2"``, which the
+  task class's :py:meth:`make_summary(comment)
+  <earwigbot.tasks.Task.make_summary>` method will take and replace
+  ``$1`` with the task number and ``$2`` with the details of the edit.
+
+  Additionally, :py:meth:`~earwigbot.tasks.Task.shutoff_enabled` (which checks
+  whether the bot has been told to stop on-wiki by checking the content of a
+  particular page) can check a different page for each task using similar
+  variables. EarwigBot's :py:attr:`config.wiki["shutoff"]["page"]` is
+  ``"User:$1/Shutoff/Task $2"``; ``$1`` is substituted with the bot's username,
+  and ``$2`` is substituted with the task number, so, e.g., task #14 checks the
+  page ``[[User:EarwigBot/Shutoff/Task 14]].`` If the page's content does *not*
+  match :py:attr:`config.wiki["shutoff"]["disabled"]` (``"run"`` by default),
+  then shutoff is considered to be *enabled* and
+  :py:meth:`~earwigbot.tasks.Task.shutoff_enabled` will return ``True``,
+  indicating the task should not run. If you don't intend to use either of
+  these methods, feel free to leave this attribute blank.
+
+- Method :py:meth:`~earwigbot.tasks.Task.setup` is called *once* with no
+  arguments immediately after the task is first loaded. Does nothing by
+  default; treat it like an :py:meth:`__init__` if you want
+  (:py:meth:`~earwigbot.tasks.Task.__init__` does things by default and a
+  dedicated setup method is often easier than overriding
+  :py:meth:`~earwigbot.tasks.Task.__init__` and using :py:obj:`super`).
+
+- Method :py:meth:`~earwigbot.tasks.Task.run` is called with any number of
+  keyword arguments every time the task is executed (by
+  :py:meth:`tasks.start(task_name, **kwargs)
+  <earwigbot.managers.TaskManager.start>`, usually). This is where the bulk of
+  the task's code goes. For interfacing with MediaWiki sites, read up on the
+  :doc:`Wiki Toolset <toolset>`.
+
+Tasks have access to :py:attr:`config.tasks[task_name]` for config information,
+which is a node in :file:`config.yml` like every other attribute of
+:py:attr:`bot.config`. This can be used to store, for example, edit summaries
+or templates to append to user talk pages, so that these can be easily changed
+without modifying the task itself.
+
+The task *class* doesn't need a specific name, but it should logically follow
+the task's name. The filename doesn't matter, but it is recommended to match
+the task name for readability. Multiple tasks classes are allowed in one file.
+
+See the built-in wikiproject_tagger_ task for a relatively straightforward
+task, or the afc_statistics_ plugin for a more complicated one.
+
+.. _help:               https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/help.py
+.. _afc_status:         https://github.com/earwig/earwigbot-plugins/blob/develop/commands/afc_status.py
+.. _test:               https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/test.py
+.. _chanops:            https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/chanops.py
+.. _wikiproject_tagger: https://github.com/earwig/earwigbot/blob/develop/earwigbot/tasks/wikiproject_tagger.py
+.. _afc_statistics:     https://github.com/earwig/earwigbot-plugins/blob/develop/tasks/afc_statistics.py

docs/index.rst

@@ -0,0 +1,48 @@
+EarwigBot v0.1 Documentation
+============================
+
+EarwigBot_ is a Python_ robot that edits Wikipedia_ and interacts with people
+over IRC_.
+
+History
+-------
+
+Development began, based on the `Pywikipedia framework`_, in early 2009.
+Approval for its fist task, a `copyright violation detector`_, was carried out
+in May, and the bot has been running consistently ever since (with the
+exception of Jan/Feb 2011). It currently handles `several ongoing tasks`_
+ranging from statistics generation to category cleanup, and on-demand tasks
+such as WikiProject template tagging. Since it started running, the bot has
+made over 50,000 edits.
+
+A project to rewrite it from scratch began in early April 2011, thus moving
+away from the Pywikipedia framework and allowing for less overall code, better
+integration between bot parts, and easier maintenance.
+
+.. _EarwigBot:                      http://en.wikipedia.org/wiki/User:EarwigBot
+.. _Python:                         http://python.org/
+.. _Wikipedia:                      http://en.wikipedia.org/
+.. _IRC:                            http://en.wikipedia.org/wiki/Internet_Relay_Chat
+.. _Pywikipedia framework:          http://pywikipediabot.sourceforge.net/
+.. _copyright violation detector:   http://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/EarwigBot_1
+.. _several ongoing tasks:          http://en.wikipedia.org/wiki/User:EarwigBot#Tasks
+
+Contents
+--------
+
+.. toctree::
+   :maxdepth: 2
+
+   installation
+   setup
+   customizing
+   toolset
+   tips
+   API Reference <api/modules>
+
+Indices and tables
+------------------
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/installation.rst

@@ -0,0 +1,55 @@
+Installation
+============
+
+This package contains the core :py:mod:`earwigbot`, abstracted enough that it
+should be usable and customizable by anyone running a bot on a MediaWiki site.
+Since it is component-based, the IRC components can be disabled if desired. IRC
+commands and bot tasks specific to `my instance of EarwigBot`_ that I don't
+feel the average user will need are available from the repository
+`earwigbot-plugins`_.
+
+It's recommended to run the bot's unit tests before installing. Run
+:command:`python setup.py test` from the project's root directory. Note that
+some tests require an internet connection, and others may take a while to run.
+Coverage is currently rather incomplete.
+
+Latest release (v0.1)
+---------------------
+
+EarwigBot is available from the `Python Package Index`_, so you can install the
+latest release with :command:`pip install earwigbot` (`get pip`_).
+
+You can also install it from source [1]_ directly::
+
+    curl -Lo earwigbot.tgz https://github.com/earwig/earwigbot/tarball/v0.1
+    tar -xf earwigbot.tgz
+    cd earwig-earwigbot-*
+    python setup.py install
+    cd ..
+    rm -r earwigbot.tgz earwig-earwigbot-*
+
+Development version
+-------------------
+
+You can install the development version of the bot from :command:`git` by using
+setuptools/`distribute`_'s :command:`develop` command [1]_, probably on the
+``develop`` branch which contains (usually) working code. ``master`` contains
+the latest release. EarwigBot uses `git flow`_, so you're free to browse by
+tags or by new features (``feature/*`` branches)::
+
+    git clone git://github.com/earwig/earwigbot.git earwigbot
+    cd earwigbot
+    python setup.py develop
+
+.. rubric:: Footnotes
+
+.. [1] :command:`python setup.py install`/:command:`develop` may require root,
+       or use the :command:`--user` switch to install for the current user
+       only.
+
+.. _my instance of EarwigBot: http://en.wikipedia.org/wiki/User:EarwigBot
+.. _earwigbot-plugins:        https://github.com/earwig/earwigbot-plugins
+.. _Python Package Index:     http://pypi.python.org
+.. _get pip:                  http://pypi.python.org/pypi/pip
+.. _distribute:               http://pypi.python.org/pypi/distribute
+.. _git flow:                 http://nvie.com/posts/a-successful-git-branching-model/

docs/setup.rst

@@ -0,0 +1,28 @@
+Setup
+=====
+
+The bot stores its data in a "working directory", including its config file and
+databases. This is also the location where you will place custom IRC commands
+and bot tasks, which will be explained later. It doesn't matter where this
+directory is, as long as the bot can write to it.
+
+Start the bot with :command:`earwigbot path/to/working/dir`, or just
+:command:`earwigbot` if the working directory is the current directory. It will
+notice that no :file:`config.yml` file exists and take you through the setup
+process.
+
+There is currently no way to edit the :file:`config.yml` file from within the
+bot after it has been created, but YAML is a very straightforward format, so
+you should be able to make any necessary changes yourself. Check out the
+`explanation of YAML`_ on Wikipedia for help.
+
+After setup, the bot will start. This means it will connect to the IRC servers
+it has been configured for, schedule bot tasks to run at specific times, and
+then wait for instructions (as commands on IRC). For a list of commands, say
+"``!help``" (commands are messages prefixed with an exclamation mark).
+
+You can stop the bot at any time with :kbd:`Control-c`, same as you stop a
+normal Python program, and it will try to exit safely. You can also use the
+"``!quit``" command on IRC.
+
+.. _explanation of YAML: http://en.wikipedia.org/wiki/YAML

docs/tips.rst

@@ -0,0 +1,46 @@
+Tips
+====
+
+- Logging_ is a fantastic way to monitor the bot's progress as it runs. It has
+  a slew of built-in loggers, and enabling log retention (so logs are saved to
+  :file:`logs/` in the working directory) is highly recommended. In the normal
+  setup, there are three log files, each of which "rotate" at a  specific time
+  (:file:`filename.log` becomes :file:`filename.log.2012-04-10`, for example).
+  The :file:`debug.log` file rotates every hour, and maintains six hours of
+  logs of every level (``DEBUG`` and up). :file:`bot.log` rotates every day at
+  midnight, and maintains seven days of non-debug logs (``INFO`` and up).
+  Finally, :file:`error.log` rotates every Sunday night, and maintains four
+  weeks of logs indicating unexpected events (``WARNING`` and up).
+
+  To use logging in your commands or tasks (recommended),
+  :py:class:~earwigbot.commands.BaseCommand` and
+  :py:class:~earwigbot.tasks.BaseTask` provide :py:attr:`logger` attributes
+  configured for the specific command or task. If you're working with other
+  classes, :py:attr:`bot.logger` is the root logger
+  (:py:obj:`logging.getLogger("earwigbot")` by default), so you can use
+  :py:func:`~logging.Logger.getChild` to make your logger. For example, task
+  loggers are essentially
+  :py:attr:`bot.logger.getChild("tasks").getChild(task.name) <bot.logger>`.
+
+- A very useful IRC command is "``!reload``", which reloads all commands and
+  tasks without restarting the bot. [1]_ Combined with using the `!git plugin`_
+  for pulling repositories from IRC, this can provide a seamless command/task
+  development workflow if the bot runs on an external server and you set up
+  its working directory as a git repo.
+
+- You can run a task by itself instead of the entire bot with
+  :command:`earwigbot path/to/working/dir --task task_name`.
+
+- Questions, comments, or suggestions about the documentation? `Let me know`_,
+  or `create an issue`_ so I can improve it for other people.
+
+.. rubric:: Footnotes
+
+.. [1] In reality, all this does is call :py:meth:`bot.commands.load()
+       <earwigbot.managers._ResourceManager.load>` and
+       :py:meth:`bot.tasks.load() <earwigbot.managers._ResourceManager.load>`!
+
+.. _logging:         http://docs.python.org/library/logging.html
+.. _!git plugin:     https://github.com/earwig/earwigbot-plugins/blob/develop/commands/git.py
+.. _Let me know:     ben.kurtovic@verizon.net
+.. _create an issue: https://github.com/earwig/earwigbot/issues

docs/toolset.rst

@@ -0,0 +1,244 @@
+The Wiki Toolset
+================
+
+EarwigBot's answer to the `Pywikipedia framework`_ is the Wiki Toolset
+(:py:mod:`earwigbot.wiki`), which you will mainly access through
+:py:attr:`bot.wiki <earwigbot.bot.Bot.wiki>`.
+
+:py:attr:`bot.wiki <earwigbot.bot.Bot.wiki>` provides three methods for the
+management of Sites - :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.get_site`,
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.add_site`, and
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.remove_site`. Sites are objects that
+simply represent a MediaWiki site. A single instance of EarwigBot (i.e. a
+single *working directory*) is expected to relate to a single site or group of
+sites using the same login info (like all WMF wikis with `CentralAuth`_).
+
+Load your default site (the one that you picked during setup) with
+``site = bot.wiki.get_site()``.
+
+Dealing with other sites
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Skip this section if you're only working with one site.*
+
+If a site is *already known to the bot* (meaning that it is stored in the
+:file:`sites.db` file, which includes just your default wiki at first), you can
+load a site with ``site = bot.wiki.get_site(name)``, where ``name`` might be
+``"enwiki"`` or ``"frwiktionary"`` (you can also do
+``site = bot.wiki.get_site(project="wikipedia", lang="en")``). Recall that not
+giving any arguments to ``get_site()`` will return the default site.
+
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.add_site` is used to add new sites to
+the sites database. It may be called with similar arguments as
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.get_site`, but the difference is
+important. :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.get_site` only needs
+enough information to identify the site in its database, which is usually just
+its name; the database stores all other necessary connection info. With
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.add_site`, you need to provide enough
+connection info so the toolset can successfully access the site's API/SQL
+databases and store that information for later. That might not be much; for WMF
+wikis, you can usually use code like this::
+
+    project, lang = "wikipedia", "es"
+    try:
+        site = bot.wiki.get_site(project=project, lang=lang)
+    except earwigbot.SiteNotFoundError:
+        # Load site info from http://es.wikipedia.org/w/api.php:
+        site = bot.wiki.add_site(project=project, lang=lang)
+
+This works because EarwigBot assumes that the URL for the site is
+``"//{lang}.{project}.org"`` and the API is at ``/w/api.php``; this might
+change if you're dealing with non-WMF wikis, where the code might look
+something more like::
+
+    project, lang = "mywiki", "it"
+    try:
+        site = bot.wiki.get_site(project=project, lang=lang)
+    except earwigbot.SiteNotFoundError:
+        # Load site info from http://mysite.net/mywiki/it/s/api.php:
+        base_url = "http://mysite.net/" + project + "/" + lang
+        db_name = lang + project + "_p"
+        sql = {host: "sql.mysite.net", db: db_name}
+        site = bot.wiki.add_site(base_url=base_url, script_path="/s", sql=sql)
+
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.remove_site` does the opposite of
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.add_site`: give it a site's name or a
+project/lang pair like :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.get_site`
+takes, and it'll remove that site from the sites database.
+
+Sites
+~~~~~
+
+:py:class:`earwigbot.wiki.Site <earwigbot.wiki.site.Site>` objects provide the
+following attributes:
+
+- :py:attr:`~earwigbot.wiki.site.Site.name`: the site's name (or "wikiid"),
+  like ``"enwiki"``
+- :py:attr:`~earwigbot.wiki.site.Site.project`: the site's project name, like
+  ``"wikipedia"``
+- :py:attr:`~earwigbot.wiki.site.Site.lang`: the site's language code, like
+  ``"en"``
+- :py:attr:`~earwigbot.wiki.site.Site.domain`: the site's web domain, like
+  ``"en.wikipedia.org"``
+- :py:attr:`~earwigbot.wiki.site.Site.url`: the site's full base URL, like
+  ``"https://en.wikipedia.org"``
+
+and the following methods:
+
+- :py:meth:`api_query(**kwargs) <earwigbot.wiki.site.Site.api_query>`: does an
+  API query with the given keyword arguments as params
+- :py:meth:`sql_query(query, params=(), ...)
+  <earwigbot.wiki.site.Site.sql_query>`: does an SQL query and yields its
+  results (as a generator)
+- :py:meth:`~earwigbot.wiki.site.Site.get_replag`: returns the estimated
+  database replication lag (if we have the site's SQL connection info)
+- :py:meth:`namespace_id_to_name(id, all=False)
+  <earwigbot.wiki.site.Site.namespace_id_to_name>`: given a namespace ID,
+  returns the primary associated namespace name (or a list of all names when
+  ``all`` is ``True``)
+- :py:meth:`namespace_name_to_id(name)
+  <earwigbot.wiki.site.Site.namespace_name_to_id>`: given a namespace name,
+  returns the associated namespace ID
+- :py:meth:`get_page(title, follow_redirects=False, ...)
+  <earwigbot.wiki.site.Site.get_page>`: returns a ``Page`` object for the given
+  title (or a :py:class:`~earwigbot.wiki.category.Category` object if the
+  page's namespace is "``Category:``")
+- :py:meth:`get_category(catname, follow_redirects=False, ...)
+  <earwigbot.wiki.site.Site.get_category>`: returns a ``Category`` object for
+  the given title (sans namespace)
+- :py:meth:`get_user(username) <earwigbot.wiki.site.Site.get_user>`: returns a
+  :py:class:`~earwigbot.wiki.user.User` object for the given username
+- :py:meth:`delegate(services, ...) <earwigbot.wiki.site.Site.delegate>`:
+  delegates a task to either the API or SQL depending on various conditions,
+  such as server lag
+
+Pages and categories
+~~~~~~~~~~~~~~~~~~~~
+
+Create :py:class:`earwigbot.wiki.Page <earwigbot.wiki.page.Page>` objects with
+:py:meth:`site.get_page(title) <earwigbot.wiki.site.Site.get_page>`,
+:py:meth:`page.toggle_talk() <earwigbot.wiki.page.Page.toggle_talk>`,
+:py:meth:`user.get_userpage() <earwigbot.wiki.user.User.get_userpage>`, or
+:py:meth:`user.get_talkpage() <earwigbot.wiki.user.User.get_talkpage>`. They
+provide the following attributes:
+
+- :py:attr:`~earwigbot.wiki.page.Page.site`: the page's corresponding
+  :py:class:`~earwigbot.wiki.site.Site` object
+- :py:attr:`~earwigbot.wiki.page.Page.title`: the page's title, or pagename
+- :py:attr:`~earwigbot.wiki.page.Page.exists`: whether or not the page exists
+- :py:attr:`~earwigbot.wiki.page.Page.pageid`: an integer ID representing the
+  page
+- :py:attr:`~earwigbot.wiki.page.Page.url`: the page's URL
+- :py:attr:`~earwigbot.wiki.page.Page.namespace`: the page's namespace as an
+  integer
+- :py:attr:`~earwigbot.wiki.page.Page.protection`: the page's current
+  protection status
+- :py:attr:`~earwigbot.wiki.page.Page.is_talkpage`: ``True`` if the page is a
+  talkpage, else ``False``
+- :py:attr:`~earwigbot.wiki.page.Page.is_redirect`: ``True`` if the page is a
+  redirect, else ``False``
+
+and the following methods:
+
+- :py:meth:`~earwigbot.wiki.page.Page.reload`: forcibly reloads the page's
+  attributes (emphasis on *reload* - this is only necessary if there is reason
+  to believe they have changed)
+- :py:meth:`toggle_talk(...) <earwigbot.wiki.page.Page.toggle_talk>`: returns a
+  content page's talk page, or vice versa
+- :py:meth:`~earwigbot.wiki.page.Page.get`: returns page content
+- :py:meth:`~earwigbot.wiki.page.Page.get_redirect_target`: if the page is a
+  redirect, returns its destination
+- :py:meth:`~earwigbot.wiki.page.Page.get_creator`: returns a
+  :py:class:`~earwigbot.wiki.user.User` object representing the first user to
+  edit the page
+- :py:meth:`edit(text, summary, minor=False, bot=True, force=False)
+  <earwigbot.wiki.page.Page.edit>`: replaces the page's content with ``text``
+  or creates a new page
+- :py:meth:`add_section(text, title, minor=False, bot=True, force=False)
+  <earwigbot.wiki.page.Page.add_section>`: adds a new section named ``title``
+  at the bottom of the page
+- :py:meth:`copyvio_check(...)
+  <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_check>`: checks the page for
+  copyright violations
+- :py:meth:`copyvio_compare(url, ...)
+  <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_compare>`: checks the page like
+  :py:meth:`~earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_check`, but
+  against a specific URL
+- :py:meth:`check_exclusion(username=None, optouts=None)
+  <earwigbot.wiki.page.Page.check_exclusion>`: checks whether or not we are
+  allowed to edit the page per ``{{bots}}``/``{{nobots}}``
+
+Additionally, :py:class:`~earwigbot.wiki.category.Category` objects (created
+with :py:meth:`site.get_category(name) <earwigbot.wiki.site.Site.get_category>`
+or :py:meth:`site.get_page(title) <earwigbot.wiki.site.Site.get_page>` where
+``title`` is in the ``Category:`` namespace) provide the following additional
+attributes:
+
+- :py:attr:`~earwigbot.wiki.category.Category.size`: the total number of
+  members in the category
+- :py:attr:`~earwigbot.wiki.category.Category.pages`: the number of pages in
+  the category
+- :py:attr:`~earwigbot.wiki.category.Category.files`: the number of files in
+  the category
+- :py:attr:`~earwigbot.wiki.category.Category.subcats`: the number of
+  subcategories in the category
+
+And the following additional method:
+
+- :py:meth:`get_members(limit=None, ...)
+  <earwigbot.wiki.category.Category.get_members>`: iterates over
+  :py:class:`~earwigbot.wiki.page.Page`\ s in the category, until either the
+  category is exhausted or (if given) ``limit`` is reached
+
+Users
+~~~~~
+
+Create :py:class:`earwigbot.wiki.User <earwigbot.wiki.user.User>` objects with
+:py:meth:`site.get_user(name) <earwigbot.wiki.site.Site.get_user>` or
+:py:meth:`page.get_creator() <earwigbot.wiki.page.Page.get_creator>`. They
+provide the following attributes:
+
+- :py:attr:`~earwigbot.wiki.user.User.site`: the user's corresponding
+  :py:class:`~earwigbot.wiki.site.Site` object
+- :py:attr:`~earwigbot.wiki.user.User.name`: the user's username
+- :py:attr:`~earwigbot.wiki.user.User.exists`: ``True`` if the user exists, or
+  ``False`` if they do not
+- :py:attr:`~earwigbot.wiki.user.User.userid`: an integer ID representing the
+  user
+- :py:attr:`~earwigbot.wiki.user.User.blockinfo`: information about any current
+  blocks on the user (``False`` if no block, or a dict of
+  ``{"by": blocking_user, "reason": block_reason,
+  "expiry": block_expire_time}``)
+- :py:attr:`~earwigbot.wiki.user.User.groups`: a list of the user's groups
+- :py:attr:`~earwigbot.wiki.user.User.rights`: a list of the user's rights
+- :py:attr:`~earwigbot.wiki.user.User.editcount`: the number of edits made by
+  the user
+- :py:attr:`~earwigbot.wiki.user.User.registration`: the time the user
+  registered as a :py:obj:`time.struct_time`
+- :py:attr:`~earwigbot.wiki.user.User.emailable`: ``True`` if you can email the
+  user, ``False`` if you cannot
+- :py:attr:`~earwigbot.wiki.user.User.gender`: the user's gender (``"male"``,
+  ``"female"``, or ``"unknown"``)
+
+and the following methods:
+
+- :py:meth:`~earwigbot.wiki.user.User.reload`: forcibly reloads the user's
+  attributes (emphasis on *reload* - this is only necessary if there is reason
+  to believe they have changed)
+- :py:meth:`~earwigbot.wiki.user.User.get_userpage`: returns a
+  :py:class:`~earwigbot.wiki.page.Page` object representing the user's userpage
+- :py:meth:`~earwigbot.wiki.user.User.get_talkpage`: returns a
+  :py:class:`~earwigbot.wiki.page.Page` object representing the user's talkpage
+
+Additional features
+~~~~~~~~~~~~~~~~~~~
+
+Not all aspects of the toolset are covered here. Explore `its code and
+docstrings`_ to learn how to use it in a more hands-on fashion. For reference,
+:py:attr:`bot.wiki <earwigbot.bot.Bot.wiki>` is an instance of
+:py:class:`earwigbot.wiki.SitesDB <earwigbot.wiki.sitesdb.SitesDB>` tied to the
+:file:`sites.db` file in the bot's working directory.
+
+.. _Pywikipedia framework:   http://pywikipediabot.sourceforge.net/
+.. _CentralAuth:             http://www.mediawiki.org/wiki/Extension:CentralAuth
+.. _its code and docstrings: https://github.com/earwig/earwigbot/tree/develop/earwigbot/wiki

earwigbot/__init__.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
-# 
+#
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -21,17 +21,42 @@
 # SOFTWARE.
 
 """
-EarwigBot - http://earwig.github.com/earwig/earwigbot
-See README.md for a basic overview, or the docs/ directory for details.
+`EarwigBot <https://github.com/earwig/earwigbot>`_ is a Python robot that edits
+Wikipedia and interacts with people over IRC.
+
+See :file:`README.rst` for an overview, or the :file:`docs/` directory for
+details. This documentation is also available `online
+<http://packages.python.org/earwigbot>`_.
 """
 
 __author__ = "Ben Kurtovic"
-__copyright__ = "Copyright (C) 2009, 2010, 2011 by Ben Kurtovic"
+__copyright__ = "Copyright (C) 2009, 2010, 2011, 2012 by Ben Kurtovic"
 __license__ = "MIT License"
 __version__ = "0.1.dev"
 __email__ = "ben.kurtovic@verizon.net"
+__release__ = False
+
+if not __release__:
+    def _get_git_commit_id():
+        """Return the ID of the git HEAD commit."""
+        from git import Repo
+        from os.path import split, dirname
+        path = split(dirname(__file__))[0]
+        commit_id = Repo(path).head.object.hexsha
+        return commit_id[:8]
+    try:
+        __version__ += ".git+" + _get_git_commit_id()
+    except Exception:
+        pass
+    finally:
+        del _get_git_commit_id
 
-from earwigbot import (
-    blowfish, config, classes, commands, config, frontend, main, rules, tasks,
-    tests, watcher, wiki
-)
+from earwigbot import bot
+from earwigbot import commands
+from earwigbot import config
+from earwigbot import exceptions
+from earwigbot import irc
+from earwigbot import managers
+from earwigbot import tasks
+from earwigbot import util
+from earwigbot import wiki

earwigbot/blowfish.py

@@ -1,556 +0,0 @@
-#! /usr/bin/env python
-# -*- coding: utf-8  -*-
-#
-# blowfish.py
-# Copyright (C) 2002 Michael Gilfix <mgilfix@eecs.tufts.edu>
-# Copyright (C) 2011, 2012 Ben Kurtovic <ben.kurtovic@verizon.net>
-#
-# This module is open source; you can redistribute it and/or
-# modify it under the terms of the GPL or Artistic License.
-# These licenses are available at http://www.opensource.org
-#
-# This software must be used and distributed in accordance
-# with the law. The author claims no liability for its
-# misuse.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-#
-
-"""
-Blowfish Encryption
-
-This module is a pure python implementation of Bruce Schneier's encryption
-scheme 'Blowfish'. Blowish is a 16-round Feistel Network cipher and offers
-substantial speed gains over DES.
-
-The key is a string of length anywhere between 64 and 448 bits, or equivalently
-8 and 56 bytes. The encryption and decryption functions operate on 64-bit
-blocks, or 8 byte strings.
-
-The entire Blowfish() class (excluding verify_key()) is by Michael Gilfix
-<mgilfix@eecs.tufts.edu>.
-
-Blowfish.verify_key(), exception classes, encrypt() and decrypt() wrappers, and
-interactive mode are by Ben Kurtovic <ben.kurtovic@verizon.net>.
-"""
-
-class BlowfishError(Exception):
-    """Base exception class for errors involving blowfish
-    encryption/decryption."""
-
-class BlockSizeError(BlowfishError):
-    """Attempted to handle a block not 8 bytes in length."""
-
-class KeyLengthError(BlowfishError):
-    """Attempted to use a key that is either less than 8 bytes or more than 56
-    bytes in length."""
-
-class DecryptionError(BlowfishError):
-    """Attempted to decrypt malformed cyphertext (e.g., not evenly divisible
-    into 8-byte blocks) or attempted to decrypt using a bad key."""
-
-class Blowfish(object):
-    """Blowfish encryption Scheme
-
-    This class implements the encryption and decryption
-    functionality of the Blowfish cipher.
-
-    Public functions:
-
-        def __init__ (self, key)
-            Creates an instance of blowfish using 'key'
-            as the encryption key. Key is a string of
-            length ranging from 8 to 56 bytes (64 to 448
-            bits). Once the instance of the object is
-            created, the key is no longer necessary.
-
-        def encrypt (self, data):
-            Encrypt an 8 byte (64-bit) block of text
-            where 'data' is an 8 byte string. Returns an
-            8-byte encrypted string.
-
-        def decrypt (self, data):
-            Decrypt an 8 byte (64-bit) encrypted block
-            of text, where 'data' is the 8 byte encrypted
-            string. Returns an 8-byte string of plaintext.
-
-        def cipher (self, xl, xr, direction):
-            Encrypts a 64-bit block of data where xl is
-            the upper 32-bits and xr is the lower 32-bits.
-            'direction' is the direction to apply the
-            cipher, either ENCRYPT or DECRYPT constants.
-            returns a tuple of either encrypted or decrypted
-            data of the left half and right half of the
-            64-bit block.
-
-    Private members:
-
-        def __round_func (self, xl)
-            Performs an obscuring function on the 32-bit
-            block of data 'xl', which is the left half of
-            the 64-bit block of data. Returns the 32-bit
-            result as a long integer.
-    """
-
-    # Cipher directions
-    ENCRYPT = 0
-    DECRYPT = 1
-
-    # For the __round_func
-    modulus = long (2) ** 32
-
-    def __init__ (self, key):
-        self.verify_key(key)
-
-        self.p_boxes = [
-            0x243F6A88, 0x85A308D3, 0x13198A2E, 0x03707344,
-            0xA4093822, 0x299F31D0, 0x082EFA98, 0xEC4E6C89,
-            0x452821E6, 0x38D01377, 0xBE5466CF, 0x34E90C6C,
-            0xC0AC29B7, 0xC97C50DD, 0x3F84D5B5, 0xB5470917,
-            0x9216D5D9, 0x8979FB1B
-        ]
-
-        self.s_boxes = [
-            [
-                0xD1310BA6, 0x98DFB5AC, 0x2FFD72DB, 0xD01ADFB7,
-                0xB8E1AFED, 0x6A267E96, 0xBA7C9045, 0xF12C7F99,
-                0x24A19947, 0xB3916CF7, 0x0801F2E2, 0x858EFC16,
-                0x636920D8, 0x71574E69, 0xA458FEA3, 0xF4933D7E,
-                0x0D95748F, 0x728EB658, 0x718BCD58, 0x82154AEE,
-                0x7B54A41D, 0xC25A59B5, 0x9C30D539, 0x2AF26013,
-                0xC5D1B023, 0x286085F0, 0xCA417918, 0xB8DB38EF,
-                0x8E79DCB0, 0x603A180E, 0x6C9E0E8B, 0xB01E8A3E,
-                0xD71577C1, 0xBD314B27, 0x78AF2FDA, 0x55605C60,
-                0xE65525F3, 0xAA55AB94, 0x57489862, 0x63E81440,
-                0x55CA396A, 0x2AAB10B6, 0xB4CC5C34, 0x1141E8CE,
-                0xA15486AF, 0x7C72E993, 0xB3EE1411, 0x636FBC2A,
-                0x2BA9C55D, 0x741831F6, 0xCE5C3E16, 0x9B87931E,
-                0xAFD6BA33, 0x6C24CF5C, 0x7A325381, 0x28958677,
-                0x3B8F4898, 0x6B4BB9AF, 0xC4BFE81B, 0x66282193,
-                0x61D809CC, 0xFB21A991, 0x487CAC60, 0x5DEC8032,
-                0xEF845D5D, 0xE98575B1, 0xDC262302, 0xEB651B88,
-                0x23893E81, 0xD396ACC5, 0x0F6D6FF3, 0x83F44239,
-                0x2E0B4482, 0xA4842004, 0x69C8F04A, 0x9E1F9B5E,
-                0x21C66842, 0xF6E96C9A, 0x670C9C61, 0xABD388F0,
-                0x6A51A0D2, 0xD8542F68, 0x960FA728, 0xAB5133A3,
-                0x6EEF0B6C, 0x137A3BE4, 0xBA3BF050, 0x7EFB2A98,
-                0xA1F1651D, 0x39AF0176, 0x66CA593E, 0x82430E88,
-                0x8CEE8619, 0x456F9FB4, 0x7D84A5C3, 0x3B8B5EBE,
-                0xE06F75D8, 0x85C12073, 0x401A449F, 0x56C16AA6,
-                0x4ED3AA62, 0x363F7706, 0x1BFEDF72, 0x429B023D,
-                0x37D0D724, 0xD00A1248, 0xDB0FEAD3, 0x49F1C09B,
-                0x075372C9, 0x80991B7B, 0x25D479D8, 0xF6E8DEF7,
-                0xE3FE501A, 0xB6794C3B, 0x976CE0BD, 0x04C006BA,
-                0xC1A94FB6, 0x409F60C4, 0x5E5C9EC2, 0x196A2463,
-                0x68FB6FAF, 0x3E6C53B5, 0x1339B2EB, 0x3B52EC6F,
-                0x6DFC511F, 0x9B30952C, 0xCC814544, 0xAF5EBD09,
-                0xBEE3D004, 0xDE334AFD, 0x660F2807, 0x192E4BB3,
-                0xC0CBA857, 0x45C8740F, 0xD20B5F39, 0xB9D3FBDB,
-                0x5579C0BD, 0x1A60320A, 0xD6A100C6, 0x402C7279,
-                0x679F25FE, 0xFB1FA3CC, 0x8EA5E9F8, 0xDB3222F8,
-                0x3C7516DF, 0xFD616B15, 0x2F501EC8, 0xAD0552AB,
-                0x323DB5FA, 0xFD238760, 0x53317B48, 0x3E00DF82,
-                0x9E5C57BB, 0xCA6F8CA0, 0x1A87562E, 0xDF1769DB,
-                0xD542A8F6, 0x287EFFC3, 0xAC6732C6, 0x8C4F5573,
-                0x695B27B0, 0xBBCA58C8, 0xE1FFA35D, 0xB8F011A0,
-                0x10FA3D98, 0xFD2183B8, 0x4AFCB56C, 0x2DD1D35B,
-                0x9A53E479, 0xB6F84565, 0xD28E49BC, 0x4BFB9790,
-                0xE1DDF2DA, 0xA4CB7E33, 0x62FB1341, 0xCEE4C6E8,
-                0xEF20CADA, 0x36774C01, 0xD07E9EFE, 0x2BF11FB4,
-                0x95DBDA4D, 0xAE909198, 0xEAAD8E71, 0x6B93D5A0,
-                0xD08ED1D0, 0xAFC725E0, 0x8E3C5B2F, 0x8E7594B7,
-                0x8FF6E2FB, 0xF2122B64, 0x8888B812, 0x900DF01C,
-                0x4FAD5EA0, 0x688FC31C, 0xD1CFF191, 0xB3A8C1AD,
-                0x2F2F2218, 0xBE0E1777, 0xEA752DFE, 0x8B021FA1,
-                0xE5A0CC0F, 0xB56F74E8, 0x18ACF3D6, 0xCE89E299,
-                0xB4A84FE0, 0xFD13E0B7, 0x7CC43B81, 0xD2ADA8D9,
-                0x165FA266, 0x80957705, 0x93CC7314, 0x211A1477,
-                0xE6AD2065, 0x77B5FA86, 0xC75442F5, 0xFB9D35CF,
-                0xEBCDAF0C, 0x7B3E89A0, 0xD6411BD3, 0xAE1E7E49,
-                0x00250E2D, 0x2071B35E, 0x226800BB, 0x57B8E0AF,
-                0x2464369B, 0xF009B91E, 0x5563911D, 0x59DFA6AA,
-                0x78C14389, 0xD95A537F, 0x207D5BA2, 0x02E5B9C5,
-                0x83260376, 0x6295CFA9, 0x11C81968, 0x4E734A41,
-                0xB3472DCA, 0x7B14A94A, 0x1B510052, 0x9A532915,
-                0xD60F573F, 0xBC9BC6E4, 0x2B60A476, 0x81E67400,
-                0x08BA6FB5, 0x571BE91F, 0xF296EC6B, 0x2A0DD915,
-                0xB6636521, 0xE7B9F9B6, 0xFF34052E, 0xC5855664,
-                0x53B02D5D, 0xA99F8FA1, 0x08BA4799, 0x6E85076A
-            ],
-            [
-                0x4B7A70E9, 0xB5B32944, 0xDB75092E, 0xC4192623,
-                0xAD6EA6B0, 0x49A7DF7D, 0x9CEE60B8, 0x8FEDB266,
-                0xECAA8C71, 0x699A17FF, 0x5664526C, 0xC2B19EE1,
-                0x193602A5, 0x75094C29, 0xA0591340, 0xE4183A3E,
-                0x3F54989A, 0x5B429D65, 0x6B8FE4D6, 0x99F73FD6,
-                0xA1D29C07, 0xEFE830F5, 0x4D2D38E6, 0xF0255DC1,
-                0x4CDD2086, 0x8470EB26, 0x6382E9C6, 0x021ECC5E,
-                0x09686B3F, 0x3EBAEFC9, 0x3C971814, 0x6B6A70A1,
-                0x687F3584, 0x52A0E286, 0xB79C5305, 0xAA500737,
-                0x3E07841C, 0x7FDEAE5C, 0x8E7D44EC, 0x5716F2B8,
-                0xB03ADA37, 0xF0500C0D, 0xF01C1F04, 0x0200B3FF,
-                0xAE0CF51A, 0x3CB574B2, 0x25837A58, 0xDC0921BD,
-                0xD19113F9, 0x7CA92FF6, 0x94324773, 0x22F54701,
-                0x3AE5E581, 0x37C2DADC, 0xC8B57634, 0x9AF3DDA7,
-                0xA9446146, 0x0FD0030E, 0xECC8C73E, 0xA4751E41,
-                0xE238CD99, 0x3BEA0E2F, 0x3280BBA1, 0x183EB331,
-                0x4E548B38, 0x4F6DB908, 0x6F420D03, 0xF60A04BF,
-                0x2CB81290, 0x24977C79, 0x5679B072, 0xBCAF89AF,
-                0xDE9A771F, 0xD9930810, 0xB38BAE12, 0xDCCF3F2E,
-                0x5512721F, 0x2E6B7124, 0x501ADDE6, 0x9F84CD87,
-                0x7A584718, 0x7408DA17, 0xBC9F9ABC, 0xE94B7D8C,
-                0xEC7AEC3A, 0xDB851DFA, 0x63094366, 0xC464C3D2,
-                0xEF1C1847, 0x3215D908, 0xDD433B37, 0x24C2BA16,
-                0x12A14D43, 0x2A65C451, 0x50940002, 0x133AE4DD,
-                0x71DFF89E, 0x10314E55, 0x81AC77D6, 0x5F11199B,
-                0x043556F1, 0xD7A3C76B, 0x3C11183B, 0x5924A509,
-                0xF28FE6ED, 0x97F1FBFA, 0x9EBABF2C, 0x1E153C6E,
-                0x86E34570, 0xEAE96FB1, 0x860E5E0A, 0x5A3E2AB3,
-                0x771FE71C, 0x4E3D06FA, 0x2965DCB9, 0x99E71D0F,
-                0x803E89D6, 0x5266C825, 0x2E4CC978, 0x9C10B36A,
-                0xC6150EBA, 0x94E2EA78, 0xA5FC3C53, 0x1E0A2DF4,
-                0xF2F74EA7, 0x361D2B3D, 0x1939260F, 0x19C27960,
-                0x5223A708, 0xF71312B6, 0xEBADFE6E, 0xEAC31F66,
-                0xE3BC4595, 0xA67BC883, 0xB17F37D1, 0x018CFF28,
-                0xC332DDEF, 0xBE6C5AA5, 0x65582185, 0x68AB9802,
-                0xEECEA50F, 0xDB2F953B, 0x2AEF7DAD, 0x5B6E2F84,
-                0x1521B628, 0x29076170, 0xECDD4775, 0x619F1510,
-                0x13CCA830, 0xEB61BD96, 0x0334FE1E, 0xAA0363CF,
-                0xB5735C90, 0x4C70A239, 0xD59E9E0B, 0xCBAADE14,
-                0xEECC86BC, 0x60622CA7, 0x9CAB5CAB, 0xB2F3846E,
-                0x648B1EAF, 0x19BDF0CA, 0xA02369B9, 0x655ABB50,
-                0x40685A32, 0x3C2AB4B3, 0x319EE9D5, 0xC021B8F7,
-                0x9B540B19, 0x875FA099, 0x95F7997E, 0x623D7DA8,
-                0xF837889A, 0x97E32D77, 0x11ED935F, 0x16681281,
-                0x0E358829, 0xC7E61FD6, 0x96DEDFA1, 0x7858BA99,
-                0x57F584A5, 0x1B227263, 0x9B83C3FF, 0x1AC24696,
-                0xCDB30AEB, 0x532E3054, 0x8FD948E4, 0x6DBC3128,
-                0x58EBF2EF, 0x34C6FFEA, 0xFE28ED61, 0xEE7C3C73,
-                0x5D4A14D9, 0xE864B7E3, 0x42105D14, 0x203E13E0,
-                0x45EEE2B6, 0xA3AAABEA, 0xDB6C4F15, 0xFACB4FD0,
-                0xC742F442, 0xEF6ABBB5, 0x654F3B1D, 0x41CD2105,
-                0xD81E799E, 0x86854DC7, 0xE44B476A, 0x3D816250,
-                0xCF62A1F2, 0x5B8D2646, 0xFC8883A0, 0xC1C7B6A3,
-                0x7F1524C3, 0x69CB7492, 0x47848A0B, 0x5692B285,
-                0x095BBF00, 0xAD19489D, 0x1462B174, 0x23820E00,
-                0x58428D2A, 0x0C55F5EA, 0x1DADF43E, 0x233F7061,
-                0x3372F092, 0x8D937E41, 0xD65FECF1, 0x6C223BDB,
-                0x7CDE3759, 0xCBEE7460, 0x4085F2A7, 0xCE77326E,
-                0xA6078084, 0x19F8509E, 0xE8EFD855, 0x61D99735,
-                0xA969A7AA, 0xC50C06C2, 0x5A04ABFC, 0x800BCADC,
-                0x9E447A2E, 0xC3453484, 0xFDD56705, 0x0E1E9EC9,
-                0xDB73DBD3, 0x105588CD, 0x675FDA79, 0xE3674340,
-                0xC5C43465, 0x713E38D8, 0x3D28F89E, 0xF16DFF20,
-                0x153E21E7, 0x8FB03D4A, 0xE6E39F2B, 0xDB83ADF7
-            ],
-            [
-                0xE93D5A68, 0x948140F7, 0xF64C261C, 0x94692934,
-                0x411520F7, 0x7602D4F7, 0xBCF46B2E, 0xD4A20068,
-                0xD4082471, 0x3320F46A, 0x43B7D4B7, 0x500061AF,
-                0x1E39F62E, 0x97244546, 0x14214F74, 0xBF8B8840,
-                0x4D95FC1D, 0x96B591AF, 0x70F4DDD3, 0x66A02F45,
-                0xBFBC09EC, 0x03BD9785, 0x7FAC6DD0, 0x31CB8504,
-                0x96EB27B3, 0x55FD3941, 0xDA2547E6, 0xABCA0A9A,
-                0x28507825, 0x530429F4, 0x0A2C86DA, 0xE9B66DFB,
-                0x68DC1462, 0xD7486900, 0x680EC0A4, 0x27A18DEE,
-                0x4F3FFEA2, 0xE887AD8C, 0xB58CE006, 0x7AF4D6B6,
-                0xAACE1E7C, 0xD3375FEC, 0xCE78A399, 0x406B2A42,
-                0x20FE9E35, 0xD9F385B9, 0xEE39D7AB, 0x3B124E8B,
-                0x1DC9FAF7, 0x4B6D1856, 0x26A36631, 0xEAE397B2,
-                0x3A6EFA74, 0xDD5B4332, 0x6841E7F7, 0xCA7820FB,
-                0xFB0AF54E, 0xD8FEB397, 0x454056AC, 0xBA489527,
-                0x55533A3A, 0x20838D87, 0xFE6BA9B7, 0xD096954B,
-                0x55A867BC, 0xA1159A58, 0xCCA92963, 0x99E1DB33,
-                0xA62A4A56, 0x3F3125F9, 0x5EF47E1C, 0x9029317C,
-                0xFDF8E802, 0x04272F70, 0x80BB155C, 0x05282CE3,
-                0x95C11548, 0xE4C66D22, 0x48C1133F, 0xC70F86DC,
-                0x07F9C9EE, 0x41041F0F, 0x404779A4, 0x5D886E17,
-                0x325F51EB, 0xD59BC0D1, 0xF2BCC18F, 0x41113564,
-                0x257B7834, 0x602A9C60, 0xDFF8E8A3, 0x1F636C1B,
-                0x0E12B4C2, 0x02E1329E, 0xAF664FD1, 0xCAD18115,
-                0x6B2395E0, 0x333E92E1, 0x3B240B62, 0xEEBEB922,
-                0x85B2A20E, 0xE6BA0D99, 0xDE720C8C, 0x2DA2F728,
-                0xD0127845, 0x95B794FD, 0x647D0862, 0xE7CCF5F0,
-                0x5449A36F, 0x877D48FA, 0xC39DFD27, 0xF33E8D1E,
-                0x0A476341, 0x992EFF74, 0x3A6F6EAB, 0xF4F8FD37,
-                0xA812DC60, 0xA1EBDDF8, 0x991BE14C, 0xDB6E6B0D,
-                0xC67B5510, 0x6D672C37, 0x2765D43B, 0xDCD0E804,
-                0xF1290DC7, 0xCC00FFA3, 0xB5390F92, 0x690FED0B,
-                0x667B9FFB, 0xCEDB7D9C, 0xA091CF0B, 0xD9155EA3,
-                0xBB132F88, 0x515BAD24, 0x7B9479BF, 0x763BD6EB,
-                0x37392EB3, 0xCC115979, 0x8026E297, 0xF42E312D,
-                0x6842ADA7, 0xC66A2B3B, 0x12754CCC, 0x782EF11C,
-                0x6A124237, 0xB79251E7, 0x06A1BBE6, 0x4BFB6350,
-                0x1A6B1018, 0x11CAEDFA, 0x3D25BDD8, 0xE2E1C3C9,
-                0x44421659, 0x0A121386, 0xD90CEC6E, 0xD5ABEA2A,
-                0x64AF674E, 0xDA86A85F, 0xBEBFE988, 0x64E4C3FE,
-                0x9DBC8057, 0xF0F7C086, 0x60787BF8, 0x6003604D,
-                0xD1FD8346, 0xF6381FB0, 0x7745AE04, 0xD736FCCC,
-                0x83426B33, 0xF01EAB71, 0xB0804187, 0x3C005E5F,
-                0x77A057BE, 0xBDE8AE24, 0x55464299, 0xBF582E61,
-                0x4E58F48F, 0xF2DDFDA2, 0xF474EF38, 0x8789BDC2,
-                0x5366F9C3, 0xC8B38E74, 0xB475F255, 0x46FCD9B9,
-                0x7AEB2661, 0x8B1DDF84, 0x846A0E79, 0x915F95E2,
-                0x466E598E, 0x20B45770, 0x8CD55591, 0xC902DE4C,
-                0xB90BACE1, 0xBB8205D0, 0x11A86248, 0x7574A99E,
-                0xB77F19B6, 0xE0A9DC09, 0x662D09A1, 0xC4324633,
-                0xE85A1F02, 0x09F0BE8C, 0x4A99A025, 0x1D6EFE10,
-                0x1AB93D1D, 0x0BA5A4DF, 0xA186F20F, 0x2868F169,
-                0xDCB7DA83, 0x573906FE, 0xA1E2CE9B, 0x4FCD7F52,
-                0x50115E01, 0xA70683FA, 0xA002B5C4, 0x0DE6D027,
-                0x9AF88C27, 0x773F8641, 0xC3604C06, 0x61A806B5,
-                0xF0177A28, 0xC0F586E0, 0x006058AA, 0x30DC7D62,
-                0x11E69ED7, 0x2338EA63, 0x53C2DD94, 0xC2C21634,
-                0xBBCBEE56, 0x90BCB6DE, 0xEBFC7DA1, 0xCE591D76,
-                0x6F05E409, 0x4B7C0188, 0x39720A3D, 0x7C927C24,
-                0x86E3725F, 0x724D9DB9, 0x1AC15BB4, 0xD39EB8FC,
-                0xED545578, 0x08FCA5B5, 0xD83D7CD3, 0x4DAD0FC4,
-                0x1E50EF5E, 0xB161E6F8, 0xA28514D9, 0x6C51133C,
-                0x6FD5C7E7, 0x56E14EC4, 0x362ABFCE, 0xDDC6C837,
-                0xD79A3234, 0x92638212, 0x670EFA8E, 0x406000E0
-            ],
-            [
-                0x3A39CE37, 0xD3FAF5CF, 0xABC27737, 0x5AC52D1B,
-                0x5CB0679E, 0x4FA33742, 0xD3822740, 0x99BC9BBE,
-                0xD5118E9D, 0xBF0F7315, 0xD62D1C7E, 0xC700C47B,
-                0xB78C1B6B, 0x21A19045, 0xB26EB1BE, 0x6A366EB4,
-                0x5748AB2F, 0xBC946E79, 0xC6A376D2, 0x6549C2C8,
-                0x530FF8EE, 0x468DDE7D, 0xD5730A1D, 0x4CD04DC6,
-                0x2939BBDB, 0xA9BA4650, 0xAC9526E8, 0xBE5EE304,
-                0xA1FAD5F0, 0x6A2D519A, 0x63EF8CE2, 0x9A86EE22,
-                0xC089C2B8, 0x43242EF6, 0xA51E03AA, 0x9CF2D0A4,
-                0x83C061BA, 0x9BE96A4D, 0x8FE51550, 0xBA645BD6,
-                0x2826A2F9, 0xA73A3AE1, 0x4BA99586, 0xEF5562E9,
-                0xC72FEFD3, 0xF752F7DA, 0x3F046F69, 0x77FA0A59,
-                0x80E4A915, 0x87B08601, 0x9B09E6AD, 0x3B3EE593,
-                0xE990FD5A, 0x9E34D797, 0x2CF0B7D9, 0x022B8B51,
-                0x96D5AC3A, 0x017DA67D, 0xD1CF3ED6, 0x7C7D2D28,
-                0x1F9F25CF, 0xADF2B89B, 0x5AD6B472, 0x5A88F54C,
-                0xE029AC71, 0xE019A5E6, 0x47B0ACFD, 0xED93FA9B,
-                0xE8D3C48D, 0x283B57CC, 0xF8D56629, 0x79132E28,
-                0x785F0191, 0xED756055, 0xF7960E44, 0xE3D35E8C,
-                0x15056DD4, 0x88F46DBA, 0x03A16125, 0x0564F0BD,
-                0xC3EB9E15, 0x3C9057A2, 0x97271AEC, 0xA93A072A,
-                0x1B3F6D9B, 0x1E6321F5, 0xF59C66FB, 0x26DCF319,
-                0x7533D928, 0xB155FDF5, 0x03563482, 0x8ABA3CBB,
-                0x28517711, 0xC20AD9F8, 0xABCC5167, 0xCCAD925F,
-                0x4DE81751, 0x3830DC8E, 0x379D5862, 0x9320F991,
-                0xEA7A90C2, 0xFB3E7BCE, 0x5121CE64, 0x774FBE32,
-                0xA8B6E37E, 0xC3293D46, 0x48DE5369, 0x6413E680,
-                0xA2AE0810, 0xDD6DB224, 0x69852DFD, 0x09072166,
-                0xB39A460A, 0x6445C0DD, 0x586CDECF, 0x1C20C8AE,
-                0x5BBEF7DD, 0x1B588D40, 0xCCD2017F, 0x6BB4E3BB,
-                0xDDA26A7E, 0x3A59FF45, 0x3E350A44, 0xBCB4CDD5,
-                0x72EACEA8, 0xFA6484BB, 0x8D6612AE, 0xBF3C6F47,
-                0xD29BE463, 0x542F5D9E, 0xAEC2771B, 0xF64E6370,
-                0x740E0D8D, 0xE75B1357, 0xF8721671, 0xAF537D5D,
-                0x4040CB08, 0x4EB4E2CC, 0x34D2466A, 0x0115AF84,
-                0xE1B00428, 0x95983A1D, 0x06B89FB4, 0xCE6EA048,
-                0x6F3F3B82, 0x3520AB82, 0x011A1D4B, 0x277227F8,
-                0x611560B1, 0xE7933FDC, 0xBB3A792B, 0x344525BD,
-                0xA08839E1, 0x51CE794B, 0x2F32C9B7, 0xA01FBAC9,
-                0xE01CC87E, 0xBCC7D1F6, 0xCF0111C3, 0xA1E8AAC7,
-                0x1A908749, 0xD44FBD9A, 0xD0DADECB, 0xD50ADA38,
-                0x0339C32A, 0xC6913667, 0x8DF9317C, 0xE0B12B4F,
-                0xF79E59B7, 0x43F5BB3A, 0xF2D519FF, 0x27D9459C,
-                0xBF97222C, 0x15E6FC2A, 0x0F91FC71, 0x9B941525,
-                0xFAE59361, 0xCEB69CEB, 0xC2A86459, 0x12BAA8D1,
-                0xB6C1075E, 0xE3056A0C, 0x10D25065, 0xCB03A442,
-                0xE0EC6E0E, 0x1698DB3B, 0x4C98A0BE, 0x3278E964,
-                0x9F1F9532, 0xE0D392DF, 0xD3A0342B, 0x8971F21E,
-                0x1B0A7441, 0x4BA3348C, 0xC5BE7120, 0xC37632D8,
-                0xDF359F8D, 0x9B992F2E, 0xE60B6F47, 0x0FE3F11D,
-                0xE54CDA54, 0x1EDAD891, 0xCE6279CF, 0xCD3E7E6F,
-                0x1618B166, 0xFD2C1D05, 0x848FD2C5, 0xF6FB2299,
-                0xF523F357, 0xA6327623, 0x93A83531, 0x56CCCD02,
-                0xACF08162, 0x5A75EBB5, 0x6E163697, 0x88D273CC,
-                0xDE966292, 0x81B949D0, 0x4C50901B, 0x71C65614,
-                0xE6C6C7BD, 0x327A140A, 0x45E1D006, 0xC3F27B9A,
-                0xC9AA53FD, 0x62A80F00, 0xBB25BFE2, 0x35BDD2F6,
-                0x71126905, 0xB2040222, 0xB6CBCF7C, 0xCD769C2B,
-                0x53113EC0, 0x1640E3D3, 0x38ABBD60, 0x2547ADF0,
-                0xBA38209C, 0xF746CE76, 0x77AFA1C5, 0x20756060,
-                0x85CBFE4E, 0x8AE88DD8, 0x7AAAF9B0, 0x4CF9AA7E,
-                0x1948C25C, 0x02FB8A8C, 0x01C36AE4, 0xD6EBE1F9,
-                0x90D4F869, 0xA65CDEA0, 0x3F09252D, 0xC208E69F,
-                0xB74E6132, 0xCE77E25B, 0x578FDFE3, 0x3AC372E6
-            ]
-        ]
-
-        # Cycle through the p-boxes and round-robin XOR the
-        # key with the p-boxes
-        key_len = len (key)
-        index = 0
-        for i in range (len (self.p_boxes)):
-            val = (ord (key[index % key_len]) << 24) + \
-                  (ord (key[(index + 1) % key_len]) << 16) + \
-                  (ord (key[(index + 2) % key_len]) << 8) + \
-                   ord (key[(index + 3) % key_len])
-            self.p_boxes[i] = self.p_boxes[i] ^ val
-            index = index + 4
-
-        # For the chaining process
-        l, r = 0, 0
-
-        # Begin chain replacing the p-boxes
-        for i in range (0, len (self.p_boxes), 2):
-            l, r = self.cipher (l, r, self.ENCRYPT)
-            self.p_boxes[i] = l
-            self.p_boxes[i + 1] = r
-
-        # Chain replace the s-boxes
-        for i in range (len (self.s_boxes)):
-            for j in range (0, len (self.s_boxes[i]), 2):
-                l, r = self.cipher (l, r, self.ENCRYPT)
-                self.s_boxes[i][j] = l
-                self.s_boxes[i][j + 1] = r
-
-    def cipher(self, xl, xr, direction):
-        if direction == self.ENCRYPT:
-            for i in range (16):
-                xl = xl ^ self.p_boxes[i]
-                xr = self.__round_func (xl) ^ xr
-                xl, xr = xr, xl
-            xl, xr = xr, xl
-            xr = xr ^ self.p_boxes[16]
-            xl = xl ^ self.p_boxes[17]
-        else:
-            for i in range (17, 1, -1):
-                xl = xl ^ self.p_boxes[i]
-                xr = self.__round_func (xl) ^ xr
-                xl, xr = xr, xl
-            xl, xr = xr, xl
-            xr = xr ^ self.p_boxes[1]
-            xl = xl ^ self.p_boxes[0]
-        return xl, xr
-
-    def __round_func(self, xl):
-        a = (xl & 0xFF000000) >> 24
-        b = (xl & 0x00FF0000) >> 16
-        c = (xl & 0x0000FF00) >> 8
-        d = xl & 0x000000FF
-
-        # Perform all ops as longs then and out the last 32-bits to
-        # obtain the integer
-        f = (long (self.s_boxes[0][a]) + long (self.s_boxes[1][b])) % self.modulus
-        f = f ^ long (self.s_boxes[2][c])
-        f = f + long (self.s_boxes[3][d])
-        f = (f % self.modulus) & 0xFFFFFFFF
-
-        return f
-
-    def encrypt(self, data):
-        if not len(data) == 8:
-            e = "blocks must be 8 bytes long, but tried to encrypt one {0} bytes long"
-            raise BlockSizeError(e.format(len(data)))
-
-        # Use big endianess since that's what everyone else uses
-        xl = ord (data[3]) | (ord (data[2]) << 8) | (ord (data[1]) << 16) | (ord (data[0]) << 24)
-        xr = ord (data[7]) | (ord (data[6]) << 8) | (ord (data[5]) << 16) | (ord (data[4]) << 24)
-
-        cl, cr = self.cipher (xl, xr, self.ENCRYPT)
-        chars = ''.join ([
-            chr ((cl >> 24) & 0xFF), chr ((cl >> 16) & 0xFF), chr ((cl >> 8) & 0xFF), chr (cl & 0xFF),
-            chr ((cr >> 24) & 0xFF), chr ((cr >> 16) & 0xFF), chr ((cr >> 8) & 0xFF), chr (cr & 0xFF)
-        ])
-        return chars
-
-    def decrypt(self, data):
-        if not len(data) == 8:
-            e = "blocks must be 8 bytes long, but tried to decrypt one {0} bytes long"
-            raise BlockSizeError(e.format(len(data)))
-
-        # Use big endianess since that's what everyone else uses
-        cl = ord (data[3]) | (ord (data[2]) << 8) | (ord (data[1]) << 16) | (ord (data[0]) << 24)
-        cr = ord (data[7]) | (ord (data[6]) << 8) | (ord (data[5]) << 16) | (ord (data[4]) << 24)
-
-        xl, xr = self.cipher (cl, cr, self.DECRYPT)
-        chars = ''.join ([
-            chr ((xl >> 24) & 0xFF), chr ((xl >> 16) & 0xFF), chr ((xl >> 8) & 0xFF), chr (xl & 0xFF),
-            chr ((xr >> 24) & 0xFF), chr ((xr >> 16) & 0xFF), chr ((xr >> 8) & 0xFF), chr (xr & 0xFF)
-        ])
-        return chars
-
-    def blocksize(self):
-        return 8
-
-    def key_length(self):
-        return 56
-
-    def key_bits(self):
-        return 56 * 8
-
-    def verify_key(self, key):
-        """Make sure our key is not too short or too long.
-        
-        If there's a problem, raise KeyTooShortError() or KeyTooLongError().
-        """
-        if not key:
-            raise KeyLengthError("no key given")
-        if len(key) < 8:
-            e = "key is {0} bytes long, but it must be at least 8"
-            raise KeyLengthError(e.format(len(key)))
-        if len(key) > 56:
-            e = "key is {0} bytes long, but it must be less than 56"
-            raise KeyLengthError(e.format(len(key)))
-
-def encrypt(key, plaintext):
-    """Encrypt any length of plaintext using a given key that must be between
-    8 and 56 bytes in length. This is a convienence function that can handle
-    plaintext that is not a single block in length. It will auto-pad blocks
-    that are less than 8 bytes with spaces that are automatically removed by
-    decrypt(). Actual spaces in plaintext are preserved."""
-    cypher = Blowfish(key)
-
-    msg = "TRUE{0}|{1}".format(len(plaintext), plaintext)
-    while len(msg) % 8 > 0:
-        msg += " "  # pad message to form complete 8-byte blocks
-
-    blocks = [msg[f:f+8] for f in range(0, len(msg), 8)]
-    cyphertext = ''.join(map(cypher.encrypt, blocks))
-
-    return cyphertext.encode('hex')
-
-def decrypt(key, cyphertext):
-    """Decrypt the result of encrypt() using the original key, or raise
-    DecryptionError()."""
-    cypher = Blowfish(key)
-
-    try:
-        cyphertext = cyphertext.decode("hex")
-    except (TypeError, AttributeError) as error:
-        e = error.message
-        raise DecryptionError("cyphertext could not be decoded: " + e.lower())
-
-    if len(cyphertext) % 8 > 0:
-        e = "cyphertext cannot be broken into 8-byte blocks evenly"
-        raise DecryptionError(e)
-
-    blocks = [cyphertext[f:f+8] for f in range(0, len(cyphertext), 8)]
-    msg = ''.join(map(cypher.decrypt, blocks))
-
-    # Sanity check to ensure valid decryption:
-    if not msg.startswith("TRUE"):
-        e = "the given key is incorrect, or part of the cyphertext is malformed"
-        raise DecryptionError(e)
-
-    size, msg = msg[4:].split("|", 1)
-    while len(msg) > int(size):
-        msg = msg[:-1]  # Remove the padding that we applied earlier
-
-    return msg
-
-if __name__ == '__main__':
-    action = raw_input("Would you like to [e]ncrypt or [d]ecrypt? ")
-    if action.lower().startswith("e"):
-        key = raw_input("Enter a key: ")
-        plaintext = raw_input("Enter a message to encrypt: ")
-        print "\n" + encrypt(key, plaintext)
-    elif action.lower().startswith("d"):
-        key = raw_input("Enter a key: ")
-        cyphertext = raw_input("Enter a message to decrypt: ")
-        print "\n" + decrypt(key, cyphertext)
-    else:
-        print "Unknown action: '{0}'".format(action)

earwigbot/bot.py

@@ -0,0 +1,212 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import logging
+from threading import Lock, Thread, enumerate as enumerate_threads
+from time import sleep, time
+
+from earwigbot import __version__
+from earwigbot.config import BotConfig
+from earwigbot.irc import Frontend, Watcher
+from earwigbot.managers import CommandManager, TaskManager
+from earwigbot.wiki import SitesDB
+
+__all__ = ["Bot"]
+
+class Bot(object):
+    """
+    **EarwigBot: Main Bot Class**
+
+    The :py:class:`Bot` class is the core of EarwigBot, essentially responsible
+    for starting the various bot components and making sure they are all happy.
+
+    EarwigBot has three components that can run independently of each other: an
+    IRC front-end, an IRC watcher, and a wiki scheduler.
+
+    - The IRC front-end runs on a normal IRC server and expects users to
+      interact with it/give it commands.
+    - The IRC watcher runs on a wiki recent-changes server and listens for
+      edits. Users cannot interact with this part of the bot.
+    - The wiki scheduler runs wiki-editing bot tasks in separate threads at
+      user-defined times through a cron-like interface.
+
+    The :py:class:`Bot` object is accessible from within commands and tasks as
+    :py:attr:`self.bot`. This is the primary way to access data from other
+    components of the bot. For example, our
+    :py:class:`~earwigbot.config.BotConfig` object is accessable from
+    :py:attr:`bot.config`, tasks can be started with
+    :py:meth:`bot.tasks.start() <earwigbot.managers.TaskManager.start>`, and
+    sites can be loaded from the wiki toolset with
+    :py:meth:`bot.wiki.get_site() <earwigbot.wiki.sitesdb.SitesDB.get_site>`.
+    """
+
+    def __init__(self, root_dir, level=logging.INFO):
+        self.config = BotConfig(root_dir, level)
+        self.logger = logging.getLogger("earwigbot")
+        self.commands = CommandManager(self)
+        self.tasks = TaskManager(self)
+        self.wiki = SitesDB(self)
+        self.frontend = None
+        self.watcher = None
+
+        self.component_lock = Lock()
+        self._keep_looping = True
+
+        self.config.load()
+        self.commands.load()
+        self.tasks.load()
+
+    def __repr__(self):
+        """Return the canonical string representation of the Bot."""
+        return "Bot(config={0!r})".format(self.config)
+
+    def __str__(self):
+        """Return a nice string representation of the Bot."""
+        return "<Bot at {0}>".format(self.config.root_dir)
+
+    def _dispatch_irc_component(self, name, klass):
+        """Create a new IRC component, record it internally, and start it."""
+        component = klass(self)
+        setattr(self, name, component)
+        Thread(name="irc_" + name, target=component.loop).start()
+
+    def _start_irc_components(self):
+        """Start the IRC frontend/watcher in separate threads if enabled."""
+        if self.config.components.get("irc_frontend"):
+            self.logger.info("Starting IRC frontend")
+            self._dispatch_irc_component("frontend", Frontend)
+        if self.config.components.get("irc_watcher"):
+            self.logger.info("Starting IRC watcher")
+            self._dispatch_irc_component("watcher", Watcher)
+
+    def _start_wiki_scheduler(self):
+        """Start the wiki scheduler in a separate thread if enabled."""
+        def wiki_scheduler():
+            while self._keep_looping:
+                time_start = time()
+                self.tasks.schedule()
+                time_end = time()
+                time_diff = time_start - time_end
+                if time_diff < 60:  # Sleep until the next minute
+                    sleep(60 - time_diff)
+
+        if self.config.components.get("wiki_scheduler"):
+            self.logger.info("Starting wiki scheduler")
+            thread = Thread(name="wiki_scheduler", target=wiki_scheduler)
+            thread.daemon = True  # Stop if other threads stop
+            thread.start()
+
+    def _keep_irc_component_alive(self, name, klass):
+        """Ensure that IRC components stay connected, else restart them."""
+        component = getattr(self, name)
+        if component:
+            component.keep_alive()
+            if component.is_stopped():
+                log = "IRC {0} has stopped; restarting".format(name)
+                self.logger.warn(log)
+                self._dispatch_irc_component(name, klass)
+
+    def _stop_irc_components(self, msg):
+        """Request the IRC frontend and watcher to stop if enabled."""
+        if self.frontend:
+            self.frontend.stop(msg)
+        if self.watcher:
+            self.watcher.stop(msg)
+
+    def _stop_daemon_threads(self):
+        """Notify the user of which threads are going to be killed.
+
+        Unfortunately, there is no method right now of stopping command and
+        task threads safely. This is because there is no way to tell them to
+        stop like the IRC components can be told; furthermore, they are run as
+        daemons, and daemon threads automatically stop without calling any
+        __exit__ or try/finally code when all non-daemon threads stop. They
+        were originally implemented as regular non-daemon threads, but this
+        meant there was no way to completely stop the bot if tasks were
+        running, because all other threads would exit and threading would
+        absorb KeyboardInterrupts.
+
+        The advantage of this is that stopping the bot is truly guarenteed to
+        *stop* the bot, while the disadvantage is that the threads are given no
+        advance warning of their forced shutdown.
+        """
+        tasks = []
+        non_tasks = self.config.components.keys() + ["MainThread", "reminder"]
+        for thread in enumerate_threads():
+            if thread.name not in non_tasks and thread.is_alive():
+                tasks.append(thread.name)
+        if tasks:
+            log = "The following commands or tasks will be killed: {0}"
+            self.logger.warn(log.format(" ".join(tasks)))
+
+    def run(self):
+        """Main entry point into running the bot.
+
+        Starts all config-enabled components and then enters an idle loop,
+        ensuring that all components remain online and restarting components
+        that get disconnected from their servers.
+        """
+        self.logger.info("Starting bot (EarwigBot {0})".format(__version__))
+        self._start_irc_components()
+        self._start_wiki_scheduler()
+        while self._keep_looping:
+            with self.component_lock:
+                self._keep_irc_component_alive("frontend", Frontend)
+                self._keep_irc_component_alive("watcher", Watcher)
+            sleep(2)
+
+    def restart(self, msg=None):
+        """Reload config, commands, tasks, and safely restart IRC components.
+
+        This is thread-safe, and it will gracefully stop IRC components before
+        reloading anything. Note that you can safely reload commands or tasks
+        without restarting the bot with :py:meth:`bot.commands.load()
+        <earwigbot.managers._ResourceManager.load>` or
+        :py:meth:`bot.tasks.load() <earwigbot.managers._ResourceManager.load>`.
+        These should not interfere with running components or tasks.
+
+        If given, *msg* will be used as our quit message.
+        """
+        if msg:
+            self.logger.info('Restarting bot ("{0}")'.format(msg))
+        else:
+            self.logger.info("Restarting bot")
+        with self.component_lock:
+            self._stop_irc_components(msg)
+            self.config.load()
+            self.commands.load()
+            self.tasks.load()
+            self._start_irc_components()
+
+    def stop(self, msg=None):
+        """Gracefully stop all bot components.
+
+        If given, *msg* will be used as our quit message.
+        """
+        if msg:
+            self.logger.info('Stopping bot ("{0}")'.format(msg))
+        else:
+            self.logger.info("Stopping bot")
+        with self.component_lock:
+            self._stop_irc_components(msg)
+        self._keep_looping = False
+        self._stop_daemon_threads()

earwigbot/classes/base_command.py

@@ -1,75 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-import logging
-
-__all__ = ["BaseCommand"]
-
-class BaseCommand(object):
-    """A base class for commands on IRC.
-
-    This docstring is reported to the user when they use !help <command>.
-    """
-    # This is the command's name, as reported to the user when they use !help:
-    name = None
-
-    # Hooks are "msg", "msg_private", "msg_public", and "join". "msg" is the
-    # default behavior; if you wish to override that, change the value in your
-    # command subclass:
-    hooks = ["msg"]
-
-    def __init__(self, connection):
-        """Constructor for new commands.
-
-        This is called once when the command is loaded (from
-        commands._load_command()). `connection` is a Connection object,
-        allowing us to do self.connection.say(), self.connection.send(), etc,
-        from within a method.
-        """
-        self.connection = connection
-        logger_name = ".".join(("earwigbot", "commands", self.name))
-        self.logger = logging.getLogger(logger_name)
-        self.logger.setLevel(logging.DEBUG)
-
-    def check(self, data):
-        """Returns whether this command should be called in response to 'data'.
-
-        Given a Data() instance, return True if we should respond to this
-        activity, or False if we should ignore it or it doesn't apply to us.
-
-        Most commands return True if data.command == self.name, otherwise they
-        return False. This is the default behavior of check(); you need only
-        override it if you wish to change that.
-        """
-        if data.is_command and data.command == self.name:
-            return True
-        return False
-
-    def process(self, data):
-        """Main entry point for doing a command.
-
-        Handle an activity (usually a message) on IRC. At this point, thanks
-        to self.check() which is called automatically by the command handler,
-        we know this is something we should respond to, so (usually) something
-        like 'if data.command != "command_name": return' is unnecessary.
-        """
-        pass

earwigbot/classes/base_task.py

@@ -1,117 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-import logging
-
-from earwigbot.config import config
-from earwigbot import wiki
-
-__all__ = ["BaseTask"]
-
-class BaseTask(object):
-    """A base class for bot tasks that edit Wikipedia."""
-    name = None
-    number = 0
-
-    def __init__(self):
-        """Constructor for new tasks.
-
-        This is called once immediately after the task class is loaded by
-        the task manager (in tasks._load_task()).
-        """
-        pass
-
-    def _setup_logger(self):
-        """Set up a basic module-level logger."""
-        logger_name = ".".join(("earwigbot", "tasks", self.name))
-        self.logger = logging.getLogger(logger_name)
-        self.logger.setLevel(logging.DEBUG)
-
-    def run(self, **kwargs):
-        """Main entry point to run a given task.
-
-        This is called directly by tasks.start() and is the main way to make a
-        task do stuff. kwargs will be any keyword arguments passed to start()
-        which are entirely optional.
-
-        The same task instance is preserved between runs, so you can
-        theoretically store data in self (e.g.
-        start('mytask', action='store', data='foo')) and then use it later
-        (e.g. start('mytask', action='save')).
-        """
-        pass
-
-    def make_summary(self, comment):
-        """Makes an edit summary by filling in variables in a config value.
-
-        config.wiki["summary"] is used, where $2 is replaced by the main
-        summary body, given as a method arg, and $1 is replaced by the task
-        number.
-
-        If the config value is not found, we just return the arg as-is.
-        """
-        try:
-            summary = config.wiki["summary"]
-        except KeyError:
-            return comment
-        return summary.replace("$1", str(self.number)).replace("$2", comment)
-
-    def shutoff_enabled(self, site=None):
-        """Returns whether on-wiki shutoff is enabled for this task.
-
-        We check a certain page for certain content. This is determined by
-        our config file: config.wiki["shutoff"]["page"] is used as the title,
-        with $1 replaced by our username and $2 replaced by the task number,
-        and config.wiki["shutoff"]["disabled"] is used as the content.
-
-        If the page has that content or the page does not exist, then shutoff
-        is "disabled", meaning the bot is supposed to run normally, and we
-        return False. If the page's content is something other than what we
-        expect, shutoff is enabled, and we return True.
-
-        If a site is not provided, we'll try to use self.site if it's set.
-        Otherwise, we'll use our default site.
-        """
-        if not site:
-            try:
-                site = self.site
-            except AttributeError:
-                site = wiki.get_site()
-
-        try:
-            cfg = config.wiki["shutoff"]
-        except KeyError:
-            return False
-        title = cfg.get("page", "User:$1/Shutoff/Task $2")
-        username = site.get_user().name()
-        title = title.replace("$1", username).replace("$2", str(self.number))
-        page = site.get_page(title)
-
-        try:
-            content = page.get()
-        except wiki.PageNotFoundError:
-            return False
-        if content == cfg.get("disabled", "run"):
-            return False
-
-        self.logger.warn("Emergency task shutoff has been enabled!")
-        return True

earwigbot/classes/connection.py

@@ -1,115 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-import socket
-import threading
-
-__all__ = ["BrokenSocketException", "Connection"]
-
-class BrokenSocketException(Exception):
-    """A socket has broken, because it is not sending data. Raised by
-    Connection.get()."""
-    pass
-
-class Connection(object):
-    """A class to interface with IRC."""
-
-    def __init__(self, host=None, port=None, nick=None, ident=None,
-                 realname=None, logger=None):
-        self.host = host
-        self.port = port
-        self.nick = nick
-        self.ident = ident
-        self.realname = realname
-        self.logger = logger
-
-        # A lock to prevent us from sending two messages at once:
-        self.lock = threading.Lock()
-
-    def connect(self):
-        """Connect to our IRC server."""
-        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        try:
-            self.sock.connect((self.host, self.port))
-        except socket.error:
-            self.logger.critical("Couldn't connect to IRC server", exc_info=1)
-            exit(1)
-        self.send("NICK %s" % self.nick)
-        self.send("USER %s %s * :%s" % (self.ident, self.host, self.realname))
-
-    def close(self):
-        """Close our connection with the IRC server."""
-        try:
-            self.sock.shutdown(socket.SHUT_RDWR) # shut down connection first
-        except socket.error:
-            pass # ignore if the socket is already down
-        self.sock.close()
-
-    def get(self, size=4096):
-        """Receive (i.e. get) data from the server."""
-        data = self.sock.recv(4096)
-        if not data:
-            # Socket isn't giving us any data, so it is dead or broken:
-            raise BrokenSocketException()
-        return data
-
-    def send(self, msg):
-        """Send data to the server."""
-        # Ensure that we only send one message at a time with a blocking lock:
-        with self.lock:
-            self.sock.sendall(msg + "\r\n")
-            self.logger.debug(msg)
-
-    def say(self, target, msg):
-        """Send a private message to a target on the server."""
-        message = "".join(("PRIVMSG ", target, " :", msg))
-        self.send(message)
-
-    def reply(self, data, msg):
-        """Send a private message as a reply to a user on the server."""
-        message = "".join((chr(2), data.nick, chr(0x0f), ": ", msg))
-        self.say(data.chan, message)
-
-    def action(self, target, msg):
-        """Send a private message to a target on the server as an action."""
-        message = "".join((chr(1), "ACTION ", msg, chr(1)))
-        self.say(target, message)
-
-    def notice(self, target, msg):
-        """Send a notice to a target on the server."""
-        message = "".join(("NOTICE ", target, " :", msg))
-        self.send(message)
-
-    def join(self, chan):
-        """Join a channel on the server."""
-        message = " ".join(("JOIN", chan))
-        self.send(message)
-
-    def part(self, chan):
-        """Part from a channel on the server."""
-        message = " ".join(("PART", chan))
-        self.send(message)
-
-    def mode(self, chan, level, msg):
-        """Send a mode message to the server."""
-        message = " ".join(("MODE", chan, level, msg))
-        self.send(message)

earwigbot/classes/data.py

@@ -1,79 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-import re
-
-__all__ = ["KwargParseException", "Data"]
-
-class KwargParseException(Exception):
-    """Couldn't parse a certain keyword argument in self.args, probably because
-    it was given incorrectly: e.g., no value (abc), just a value (=xyz), just
-    an equal sign (=), instead of the correct (abc=xyz)."""
-    pass
-
-class Data(object):
-    """Store data from an individual line received on IRC."""
-    
-    def __init__(self, line):
-        self.line = line
-        self.chan = self.nick = self.ident = self.host = self.msg = ""
-
-    def parse_args(self):
-        """Parse command args from self.msg into self.command and self.args."""
-        args = self.msg.strip().split()
-
-        while "" in args:
-            args.remove("")
-
-        # Isolate command arguments:
-        self.args = args[1:]
-        self.is_command = False  # is this message a command?
-
-        try:
-            self.command = args[0]
-        except IndexError:
-            self.command = None
-
-        try:
-            if self.command.startswith('!') or self.command.startswith('.'):
-                self.is_command = True
-                self.command = self.command[1:]  # Strip the '!' or '.'
-                self.command = self.command.lower()
-        except AttributeError:
-            pass
-
-    def parse_kwargs(self):
-        """Parse keyword arguments embedded in self.args.
-        
-        Parse a command given as "!command key1=value1 key2=value2..." into a
-        dict, self.kwargs, like {'key1': 'value2', 'key2': 'value2'...}.
-        """
-        self.kwargs = {}
-        for arg in self.args[2:]:
-            try:
-                key, value = re.findall("^(.*?)\=(.*?)$", arg)[0]
-            except IndexError:
-                raise KwargParseException(arg)
-            if key and value:
-                self.kwargs[key] = value
-            else:
-                raise KwargParseException(arg)

earwigbot/commands/__init__.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,92 +20,103 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-"""
-EarwigBot's IRC Command Manager
-
-This package provides the IRC "commands" used by the bot's front-end component.
-In __init__, you can find some functions used to load and run these commands.
-"""
-
-import logging
-import os
-import sys
-
-from earwigbot.classes import BaseCommand
-from earwigbot.config import config
-
-__all__ = ["load", "get_all", "check"]
+__all__ = ["Command"]
 
-# Base directory when searching for commands:
-base_dir = os.path.dirname(os.path.abspath(__file__))
-
-# Store commands in a dict, where the key is the command's name and the value
-# is an instance of the command's class:
-_commands = {}
-
-# Logger for this module:
-logger = logging.getLogger("earwigbot.tasks")
-
-def _load_command(connection, filename):
-    """Try to load a specific command from a module, identified by file name.
-
-    Given a Connection object and a filename, we'll first try to import it,
-    and if that works, make an instance of the 'Command' class inside (assuming
-    it is an instance of BaseCommand), add it to _commands, and report the
-    addition to the user. Any problems along the way will either be ignored or
-    reported.
+class Command(object):
     """
-    global _commands
-
-    # Strip .py from the end of the filename and join with our package name:
-    name = ".".join(("commands", filename[:-3]))
-    try:
-         __import__(name)
-    except:
-        logger.exception("Couldn't load file {0}".format(filename))
-        return
-
-    command = sys.modules[name].Command(connection)
-    if not isinstance(command, BaseCommand):
-        return
+    **EarwigBot: Base IRC Command**
 
-    _commands[command.name] = command
-    logger.debug("Added command {0}".format(command.name))
+    This package provides built-in IRC "commands" used by the bot's front-end
+    component. Additional commands can be installed as plugins in the bot's
+    working directory.
 
-def load(connection):
-    """Load all valid commands into the _commands global variable.
+    This class (import with ``from earwigbot.commands import Command``), can be
+    subclassed to create custom IRC commands.
 
-    `connection` is a Connection object that is given to each command's
-    constructor.
+    This docstring is reported to the user when they type ``"!help
+    <command>"``.
     """
-    files = os.listdir(base_dir)
-    files.sort()
-
-    for filename in files:
-        if filename.startswith("_") or not filename.endswith(".py"):
-            continue
-        try:
-            _load_command(connection, filename)
-        except AttributeError:
-            pass  # The file is doesn't contain a command, so just move on
-
-    msg = "Found {0} commands: {1}"
-    logger.info(msg.format(len(_commands), ", ".join(_commands.keys())))
-
-def get_all():
-    """Return our dict of all loaded commands."""
-    return _commands
-
-def check(hook, data):
-    """Given an event on IRC, check if there's anything we can respond to."""
-    # Parse command arguments into data.command and data.args:
-    data.parse_args()
-
-    for command in _commands.values():
-        if hook in command.hooks:
-            if command.check(data):
-                try:
-                    command.process(data)
-                except:
-                    logger.exception("Error executing command '{0}'".format(data.command))
-                break
+    # The command's name, as reported to the user when they use !help:
+    name = None
+
+    # A list of names that will trigger this command. If left empty, it will
+    # be triggered by the command's name and its name only:
+    commands = []
+
+    # Hooks are "msg", "msg_private", "msg_public", and "join". "msg" is the
+    # default behavior; if you wish to override that, change the value in your
+    # command subclass:
+    hooks = ["msg"]
+
+    def __init__(self, bot):
+        """Constructor for new commands.
+
+        This is called once when the command is loaded (from
+        :py:meth:`commands.load() <earwigbot.managers._ResourceManager.load>`).
+        *bot* is out base :py:class:`~earwigbot.bot.Bot` object. Don't override
+        this directly; if you do, remember to place
+        ``super(Command, self).__init()`` first. Use :py:meth:`setup` for
+        typical command-init/setup needs.
+        """
+        self.bot = bot
+        self.config = bot.config
+        self.logger = bot.commands.logger.getChild(self.name)
+
+        # Convenience functions:
+        self.say = lambda target, msg, hidelog=False: self.bot.frontend.say(target, msg, hidelog)
+        self.reply = lambda data, msg, hidelog=False: self.bot.frontend.reply(data, msg, hidelog)
+        self.action = lambda target, msg, hidelog=False: self.bot.frontend.action(target, msg, hidelog)
+        self.notice = lambda target, msg, hidelog=False: self.bot.frontend.notice(target, msg, hidelog)
+        self.join = lambda chan, hidelog=False: self.bot.frontend.join(chan, hidelog)
+        self.part = lambda chan, msg=None, hidelog=False: self.bot.frontend.part(chan, msg, hidelog)
+        self.mode = lambda t, level, msg, hidelog=False: self.bot.frontend.mode(t, level, msg, hidelog)
+        self.ping = lambda target, hidelog=False: self.bot.frontend.ping(target, hidelog)
+        self.pong = lambda target, hidelog=False: self.bot.frontend.pong(target, hidelog)
+
+        self.setup()
+
+    def __repr__(self):
+        """Return the canonical string representation of the Command."""
+        res = "Command(name={0!r}, commands={1!r}, hooks={2!r}, bot={3!r})"
+        return res.format(self.name, self.commands, self.hooks, self.bot)
+
+    def __str__(self):
+        """Return a nice string representation of the Command."""
+        return "<Command {0} of {1}>".format(self.name, self.bot)
+
+    def setup(self):
+        """Hook called immediately after the command is loaded.
+
+        Does nothing by default; feel free to override.
+        """
+        pass
+
+    def check(self, data):
+        """Return whether this command should be called in response to *data*.
+
+        Given a :py:class:`~earwigbot.irc.data.Data` instance, return ``True``
+        if we should respond to this activity, or ``False`` if we should ignore
+        it and move on. Be aware that since this is called for each message
+        sent on IRC, it should be cheap to execute and unlikely to throw
+        exceptions.
+
+        Most commands return ``True`` only if :py:attr:`data.command
+        <earwigbot.irc.data.Data.command>` ``==`` :py:attr:`self.name <name>`,
+        or :py:attr:`data.command <earwigbot.irc.data.Data.command>` is in
+        :py:attr:`self.commands <commands>` if that list is overriden. This is
+        the default behavior; you should only override it if you wish to change
+        that.
+        """
+        if self.commands:
+            return data.is_command and data.command in self.commands
+        return data.is_command and data.command == self.name
+
+    def process(self, data):
+        """Main entry point for doing a command.
+
+        Handle an activity (usually a message) on IRC. At this point, thanks
+        to :py:meth:`check` which is called automatically by the command
+        handler, we know this is something we should respond to. Place your
+        command's body here.
+        """
+        pass

earwigbot/commands/_old.py

@@ -1,979 +0,0 @@
-# -*- coding: utf-8  -*-
-###### 
-###### NOTE:
-###### This is an old commands file from the previous version of EarwigBot.
-###### It is not used by the new EarwigBot and is simply here for reference
-###### when developing new commands.
-###### 
-### EarwigBot
-
-## Import basics.
-import sys, socket, string, time, codecs, os, traceback, thread, re, urllib, web, math, unicodedata
-
-## Import our functions.
-import config
-
-## Set up constants.
-HOST, PORT, NICK, IDENT, REALNAME, CHANS, REPORT_CHAN, WELCOME_CHAN, HOST2, CHAN2, OWNER, ADMINS, ADMINS_R, PASS = config.host, config.port, config.nick, config.ident, config.realname, config.chans, config.report_chan, config.welcome_chan, config.host2, config.chan2, config.owner, config.admins, config.admin_readable, config.password
-
-def get_commandList():
-	return {'quiet': 'quiet',
-	'welcome': 'welcome',
-	'greet': 'welcome',
-	'linker': 'linker',
-	'auth': 'auth',
-	'access': 'access',
-	'join': 'join',
-	'part': 'part',
-	'restart': 'restart',
-	'quit': 'quit',
-	'die': 'quit',
-	'msg': 'msg',
-	'me': 'me',
-	'calc': 'calc',
-	'dice': 'dice',
-	'tock': 'tock',
-	'beats': 'beats',
-	'copyvio': 'copyvio', 
-	'copy': 'copyvio',
-	'copyright': 'copyvio',
-	'dict': 'dictionary',
-	'dictionary': 'dictionary',
-	'ety': 'etymology',
-	'etymology': 'etymology',
-	'lang': 'langcode',
-	'langcode': 'langcode',
-	'num': 'number',
-	'number': 'number',
-	'count': 'number',
-	'c': 'number',
-	'nick': 'nick',
-	'op': 'op',
-	'deop': 'deop',
-	'voice': 'voice',
-	'devoice': 'devoice',
-	'pend': 'pending',
-	'pending': 'pending',
-	'sub': 'submissions',
-	'submissions': 'submissions',
-	'praise': 'praise',
-	'leonard': 'leonard',
-	'groovedog': 'groovedog',
-	'earwig': 'earwig',
-	'macmed': 'macmed',
-	'cubs197': 'cubs197',
-	'sparksboy': 'sparksboy',
-	'tim_song': 'tim_song',
-	'tim': 'tim_song',
-	'blurpeace': 'blurpeace',
-	'sausage': 'sausage',
-	'mindstormskid': 'mindstormskid',
-	'mcjohn': 'mcjohn',
-	'fetchcomms': 'fetchcomms',
-	'trout': 'trout',
-	'kill': 'kill',
-	'destroy': 'kill',
-	'murder': 'kill',
-	'fish': 'fish',
-	'report': 'report',
-	'commands': 'commands',
-	'help': 'help',
-	'doc': 'help',
-	'documentation': 'help',
-	'mysql': 'mysql',
-	'remind': 'reminder',
-	'reminder': 'reminder',
-	'notes': 'notes',
-	'note': 'notes',
-	'about': 'notes',
-	'data': 'notes',
-	'database': 'notes',
-	'hash': 'hash',
-	'lookup': 'lookup',
-	'ip': 'lookup'
-	}
-
-def main(command, line, line2, nick, chan, host, auth, notice, say, reply, s):
-	try:
-		parse(command, line, line2, nick, chan, host, auth, notice, say, reply, s)
-	except Exception:
-		trace = traceback.format_exc() # Traceback.
-		print trace # Print.
-		lines = list(reversed(trace.splitlines())) # Convert lines to process traceback....
-		report2 = [lines[0].strip()]
-		for line in lines: 
-			line = line.strip()
-			if line.startswith('File "/'): 
-				report2.append(line[0].lower() + line[1:])
-				break
-		else: report2.append('source unknown')
-		say(report2[0] + ' (' + report2[1] + ')', chan)
-
-def parse(command, line, line2, nick, chan, host, auth, notice, say, reply, s):
-	authy = auth(host)
-	if command == "access":
-		a = 'The bot\'s owner is "%s".' % OWNER
-		b = 'The bot\'s admins are "%s".' % ', '.join(ADMINS_R)
-		reply(a, chan, nick)
-		reply(b, chan, nick)
-		return
-	if command == "join":
-		if authy == "owner" or authy == "admin":
-			try:
-				channel = line2[4]
-			except Exception:
-				channel = chan
-			s.send("JOIN %s\r\n" % channel)
-		else:
-			reply("You aren't authorized to use that command.", chan, nick)
-		return
-	if command == "part":
-		if authy == "owner" or authy == "admin":
-			try:
-				channel = line2[4]
-			except Exception:
-				channel = chan
-			s.send("PART %s\r\n" % channel)
-		else:
-			reply("You aren't authorized to use that command.", chan, nick)
-		return
-	if command == "restart":
-		import thread
-		if authy == "owner":
-			s.send("QUIT\r\n")
-			time.sleep(5)
-			os.system("nice -15 python main.py")
-			exit()
-		else:
-			reply("Only the owner, %s, can stop the bot. This incident will be reported." % OWNER, chan, nick)
-		return
-	if command == "quit" or command == "die":
-		if authy != "owner":
-			if command != "suicide":
-				reply("Only the owner, %s, can stop the bot. This incident will be reported." % OWNER, chan, nick)
-			else:
-				say("\x01ACTION hands %s a gun... have fun :D\x01" % nick, nick)
-		else:
-			if command == "suicide":
-				say("\x01ACTION stabs himself with a knife.\x01", chan)
-				time.sleep(0.2)
-			try:
-				s.send("QUIT :%s\r\n" % ' '.join(line2[4:]))
-			except Exception:
-				s.send("QUIT\r\n")
-			__import__('os')._exit(0)
-		return
-	if command == "msg":
-		if authy == "owner" or authy == "admin":
-			say(' '.join(line2[5:]), line2[4])
-		else:
-			reply("You aren't authorized to use that command.", chan, nick)
-		return
-	if command == "me":
-		if authy == "owner" or authy == "admin":
-			say("\x01ACTION %s\x01" % ' '.join(line2[5:]), line2[4])
-		else:
-			reply("You aren't authorized to use that command.", chan, nick)
-		return
-	if command == "calc":
-		r_result = re.compile(r'(?i)<A NAME=results>(.*?)</A>')
-		r_tag = re.compile(r'<\S+.*?>')
-		subs = [
-				(' in ', ' -> '), 
-				(' over ', ' / '), 
-				(u'£', 'GBP '), 
-				(u'€', 'EUR '), 
-				('\$', 'USD '), 
-				(r'\bKB\b', 'kilobytes'), 
-				(r'\bMB\b', 'megabytes'), 
-				(r'\bGB\b', 'kilobytes'), 
-				('kbps', '(kilobits / second)'), 
-				('mbps', '(megabits / second)')
-		]
-		try:
-			q = ' '.join(line2[4:])
-		except Exception:
-			say("0?", chan)
-			return
-		query = q[:]
-		for a, b in subs: 
-			query = re.sub(a, b, query)
-		query = query.rstrip(' \t')
-
-		precision = 5
-		if query[-3:] in ('GBP', 'USD', 'EUR', 'NOK'): 
-			precision = 2
-		query = web.urllib.quote(query.encode('utf-8'))
-
-		uri = 'http://futureboy.us/fsp/frink.fsp?fromVal='
-		bytes = web.get(uri + query)
-		m = r_result.search(bytes)
-		if m: 
-			result = m.group(1)
-			result = r_tag.sub('', result) # strip span.warning tags
-			result = result.replace('&gt;', '>')
-			result = result.replace('(undefined symbol)', '(?) ')
-
-			if '.' in result: 
-				try: result = str(round(float(result), precision))
-				except ValueError: pass
-
-			if not result.strip(): 
-				result = '?'
-			elif ' in ' in q: 
-				result += ' ' + q.split(' in ', 1)[1]
-
-			say(q + ' = ' + result[:350], chan)
-		else: reply("Sorry, can't calculate that.", chan, nick)
-		return
-	if command == "dice":
-		import random
-		try:
-			set = range(int(line2[4]), int(line2[5]) + 1)
-		except Exception:
-			set = range(1, 7)
-		num = random.choice(set)
-		reply("You rolled a %s." % num, chan, nick)
-		if len(set) < 30:
-			say("Set consisted of %s." % set, nick)
-		else:
-			say("Set consisted of %s... and %s others." % (set[:30], len(set) - 30), nick)
-		return
-	if command == "tock":
-		u = urllib.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl')
-		info = u.info()
-		u.close()
-		say('"' + info['Date'] + '" - tycho.usno.navy.mil', chan)
-		return
-	if command == "beats":
-		beats = ((time.time() + 3600) % 86400) / 86.4
-		beats = int(math.floor(beats))
-		say('@%03i' % beats, chan)
-		return
-	if command == "copyvio" or command == "copy" or command == "copyright":
-		url = "http://en.wikipedia.org/wiki/User:EarwigBot/AfC copyvios"
-		query = urllib.urlopen(url)
-		data = query.read()
-		url = "http://toolserver.org/~earwig/earwigbot/pywikipedia/error.txt"
-		query = urllib.urlopen(url)
-		data2 = query.read()
-		if "critical" in data2:
-			text = "AfC copyvio situation is CRITICAL: Major disaster."
-		elif "exceed" in data2:
-			text = "AfC copyvio situation is CRITICAL: Queries exceeded error."
-		elif "spam" in data2:
-			text = "AfC copyvio situation is CRITICAL: Spamfilter error."
-		elif "<h3>" in data:
-			text = "AfC copyvio situation is BAD: Unsolved copyvios at [[User:EarwigBot/AfC copyvios]]"
-		else:
-			text = "AfC copyvio situation is OK: OK."
-		reply(text, chan, nick)
-		return
-	if command == "dict" or command == "dictionary":
-		def trim(thing): 
-			if thing.endswith('&nbsp;'): 
-				thing = thing[:-6]
-			return thing.strip(' :.')
-		r_li = re.compile(r'(?ims)<li>.*?</li>')
-		r_tag = re.compile(r'<[^>]+>')
-		r_parens = re.compile(r'(?<=\()(?:[^()]+|\([^)]+\))*(?=\))')
-		r_word = re.compile(r'^[A-Za-z0-9\' -]+$')
-		uri = 'http://encarta.msn.com/dictionary_/%s.html'
-		r_info = re.compile(r'(?:ResultBody"><br /><br />(.*?)&nbsp;)|(?:<b>(.*?)</b>)')
-		try:
-			word = line2[4]
-		except Exception:
-			reply("Please enter a word.", chan, nick)
-			return
-		word = urllib.quote(word.encode('utf-8'))
-		bytes = web.get(uri % word)
-		results = {}
-		wordkind = None
-		for kind, sense in r_info.findall(bytes): 
-			kind, sense = trim(kind), trim(sense)
-			if kind: wordkind = kind
-			elif sense: 
-				results.setdefault(wordkind, []).append(sense)
-		result = word.encode('utf-8') + ' - '
-		for key in sorted(results.keys()): 
-			if results[key]: 
-				result += (key or '') + ' 1. ' + results[key][0]
-				if len(results[key]) > 1: 
-					result += ', 2. ' + results[key][1]
-				result += '; '
-		result = result.rstrip('; ')
-		if result.endswith('-') and (len(result) < 30): 
-			reply('Sorry, no definition found.', chan, nick)
-		else: say(result, chan)
-		return
-	if command == "ety" or command == "etymology":
-		etyuri = 'http://etymonline.com/?term=%s'
-		etysearch = 'http://etymonline.com/?search=%s'
-		r_definition = re.compile(r'(?ims)<dd[^>]*>.*?</dd>')
-		r_tag = re.compile(r'<(?!!)[^>]+>')
-		r_whitespace = re.compile(r'[\t\r\n ]+')
-		abbrs = [
-			'cf', 'lit', 'etc', 'Ger', 'Du', 'Skt', 'Rus', 'Eng', 'Amer.Eng', 'Sp', 
-			'Fr', 'N', 'E', 'S', 'W', 'L', 'Gen', 'J.C', 'dial', 'Gk', 
-			'19c', '18c', '17c', '16c', 'St', 'Capt', 'obs', 'Jan', 'Feb', 'Mar', 
-			'Apr', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec', 'c', 'tr', 'e', 'g'
-		]
-		t_sentence = r'^.*?(?<!%s)(?:\.(?= [A-Z0-9]|\Z)|\Z)'
-		r_sentence = re.compile(t_sentence % ')(?<!'.join(abbrs))
-		def unescape(s): 
-			s = s.replace('&gt;', '>')
-			s = s.replace('&lt;', '<')
-			s = s.replace('&amp;', '&')
-			return s
-		def text(html): 
-			html = r_tag.sub('', html)
-			html = r_whitespace.sub(' ', html)
-			return unescape(html).strip()
-		try:
-			word = line2[4]
-		except Exception:
-			reply("Please enter a word.", chan, nick)
-			return
-		def ety(word):
-			if len(word) > 25: 
-				raise ValueError("Word too long: %s[...]" % word[:10])
-			word = {'axe': 'ax/axe'}.get(word, word)
-			bytes = web.get(etyuri % word)
-			definitions = r_definition.findall(bytes)
-			if not definitions: 
-				return None
-			defn = text(definitions[0])
-			m = r_sentence.match(defn)
-			if not m: 
-				return None
-			sentence = m.group(0)
-			try: 
-				sentence = unicode(sentence, 'iso-8859-1')
-				sentence = sentence.encode('utf-8')
-			except: pass
-			maxlength = 275
-			if len(sentence) > maxlength: 
-				sentence = sentence[:maxlength]
-				words = sentence[:-5].split(' ')
-				words.pop()
-				sentence = ' '.join(words) + ' [...]'
-			sentence = '"' + sentence.replace('"', "'") + '"'
-			return sentence + ' - ' + (etyuri % word)
-		try:
-			result = ety(word.encode('utf-8'))
-		except IOError: 
-			msg = "Can't connect to etymonline.com (%s)" % (etyuri % word)
-			reply(msg, chan, nick)
-			return
-		except AttributeError: 
-			result = None
-		if result is not None: 
-			reply(result, chan, nick)
-		else: 
-			uri = etysearch % word
-			msg = 'Can\'t find the etymology for "%s". Try %s' % (word, uri)
-			reply(msg, chan, nick)
-		return
-	if command == "num" or command == "number" or command == "count" or command == "c":
-		try:
-			params = string.lower(line2[4])
-		except Exception:
-			params = False
-		if params == "old" or params == "afc" or params == "a":
-			number = unicode(int(len(re.findall("title=", urllib.urlopen("http://en.wikipedia.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Pending_AfC_submissions&cmlimit=500").read()))) - 2)
-			reply("There are currently %s pending AfC submissions." % number, chan, nick)
-		elif params == "redirect" or params == "redir" or params == "redirs" or params == "redirects" or params == "r":
-			redir_data = urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Wikipedia:Articles_for_creation/Redirects").read()
-			redirs = (string.count(redir_data, "<h2>") - 1) - (string.count(redir_data, '<table class="navbox collapsible collapsed" style="text-align: left; border: 0px; margin-top: 0.2em;">'))
-			reply("There are currently %s open redirect requests." % redirs, chan, nick)
-		elif params == "files" or params == "ffu" or params == "file" or params == "image" or params == "images" or params == "ifu" or params == "f":
-			file_data = re.sub("<h2>Contents</h2>", "", urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Wikipedia:Files_for_upload").read())
-			files = (string.count(file_data, "<h2>") - 1) - (string.count(file_data, '<table class="navbox collapsible collapsed" style="text-align: left; border: 0px; margin-top: 0.2em;">'))
-			reply("There are currently %s open file upload requests." % files, chan, nick)
-		elif params == "aggregate" or params == "agg":
-			subs = unicode(int(len(re.findall("title=", urllib.urlopen("http://en.wikipedia.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Pending_AfC_submissions&cmlimit=500").read()))) - 2)
-			redir_data = urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Wikipedia:Articles_for_creation/Redirects").read()
-			file_data = re.sub("<h2>Contents</h2>", "", urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Wikipedia:Files_for_upload").read())
-			redirs = (string.count(redir_data, "<h2><span class=\"editsection\">")) - (string.count(redir_data, '<table class="navbox collapsible collapsed" style="text-align: left; border: 0px; margin-top: 0.2em;">'))
-			files = (string.count(file_data, "<h2>") - 1) - (string.count(file_data, '<table class="navbox collapsible collapsed" style="text-align: left; border: 0px; margin-top: 0.2em;">'))
-			aggregate = (int(subs) * 5) + (int(redirs) * 2) + (int(files) * 2)
-			if aggregate == 0:
-				stat = "clear"
-			elif aggregate < 60:
-				stat = "almost clear"
-			elif aggregate < 125:
-				stat = "small backlog"
-			elif aggregate < 175:
-				stat = "average backlog"
-			elif aggregate < 250:
-				stat = "backlogged"
-			elif aggregate < 300:
-				stat = "heavily backlogged"
-			else:
-				stat = "severely backlogged"
-			reply("Aggregate is currently %s (%s)." % (aggregate, stat), chan, nick)
-		else:
-			subs = unicode(int(len(re.findall("title=", urllib.urlopen("http://en.wikipedia.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Pending_AfC_submissions&cmlimit=500").read()))) - 2)
-			redir_data = urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Wikipedia:Articles_for_creation/Redirects").read()
-			file_data = re.sub("<h2>Contents</h2>", "", urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Wikipedia:Files_for_upload").read())
-			redirs = (string.count(redir_data, "<h2><span class=\"editsection\">")) - (string.count(redir_data, '<table class="navbox collapsible collapsed" style="text-align: left; border: 0px; margin-top: 0.2em;">'))
-			files = (string.count(file_data, "<h2>") - 1) - (string.count(file_data, '<table class="navbox collapsible collapsed" style="text-align: left; border: 0px; margin-top: 0.2em;">'))
-			reply("There are currently %s pending submissions, %s open redirect requests, and %s open file upload requests." % (subs, redirs, files), chan, nick)
-		return
-	if command == "nick":
-		if authy == "owner":
-			try:
-				new_nick = line2[4]
-			except Exception:
-				reply("Please specify a nick to change to.", chan, nick)
-				return
-			s.send("NICK %s\r\n" % new_nick)
-		else:
-			reply("You aren't authorized to use that command.", chan, nick)
-		return
-	if command == "op" or command == "deop" or command == "voice" or command == "devoice":
-		if authy == "owner" or authy == "admin":
-			try:
-				user = line2[4]
-			except Exception:
-				user = nick
-			say("%s %s %s" % (command, chan, user), "ChanServ")
-		else:
-			reply("You aren't authorized to use that command.", chan, nick)
-		return
-	if command == "pend" or command == "pending":
-		say("Pending submissions status page: <http://en.wikipedia.org/wiki/WP:AFC/S>.", chan)
-		say("Pending submissions category: <http://en.wikipedia.org/wiki/Category:Pending_AfC_submissions>.", chan)
-		return
-	if command == "sub" or command == "submissions":
-		try:
-			number = int(line2[4])
-		except Exception:
-			reply("Please enter a number.", chan, nick)
-			return
-		do_url = False
-		try:
-			if "url" in line2[5:]: do_url = True
-		except Exception:
-			pass
-		url = "http://en.wikipedia.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Pending_AfC_submissions&cmlimit=500&cmsort=timestamp"
-		query = urllib.urlopen(url)
-		data = query.read()
-		pages = re.findall("title=&quot;(.*?)&quot;", data)
-		try:
-			pages.remove("Wikipedia:Articles for creation/Redirects")
-		except Exception:
-			pass
-		try:
-			pages.remove("Wikipedia:Files for upload")
-		except Exception:
-			pass
-		pages.reverse()
-		pages = pages[:number]
-		if not do_url:
-			s = string.join(pages, "]], [[")
-			s = "[[%s]]" % s
-		else:
-			s = string.join(pages, ">, <http://en.wikipedia.org/wiki/")
-			s = "<http://en.wikipedia.org/wiki/%s>" % s
-			s = re.sub(" ", "_", s)
-			s = re.sub(">,_<", ">, <", s)
-		report = "\x02First %s pending AfC submissions:\x0F %s" % (number, s)
-		say(report, chan)
-		return
-	if command == "praise" or command == "leonard" or command == "groovedog" or command == "earwig" or command == "macmed" or command == "cubs197" or command == "sparksboy" or command == "tim_song" or command == "tim" or command == "sausage" or command == "mindstormskid" or command == "mcjohn" or command == "fetchcomms" or command == "blurpeace":
-		bad = False
-		if command == "leonard":
-			special = "AfC redirect reviewer"
-			user = "Leonard^Bloom"
-		elif command == "groovedog":
-			special = "heh"
-			user = "GrooveDog"
-		elif command == "earwig":
-			special = "Python programmer"
-			user = "Earwig"
-		elif command == "macmed":
-			special = "CSD tagger"
-			user = "MacMed"
-		elif command == "mindstormskid":
-			special = "Lego fanatic"
-			user = "MindstormsKid"
-		elif command == "cubs197":
-			special = "IRC dude"
-			user = "Cubs197"
-		elif command == "sparksboy":
-			special = "pet owner"
-			user = "SparksBoy"
-		elif command == "tim_song" or command == "tim":
-			special = "JavaScript programmer"
-			user = "Tim_Song"
-		elif command == "sausage":
-			special = "helper"
-			user = "chzz"
-		elif command == "mcjohn":
-			special = "edit summary writer"
-			user = "McJohn"
-		elif command == "fetchcomms":
-			special = "n00b"
-			user = "Fetchcomms"
-		elif command == "blurpeace":
-			special = "Commons admin"
-			user = "Blurpeace"
-		else:
-			say("Only a true fool would use that command, %s." % nick, chan)
-			# say("The users who you can praise are: Leonard^Bloom, GrooveDog, Earwig, MacMed, Cubs197, SparksBoy, MindstormsKid, Chzz, McJohn, Tim_Song, Fetchcomms, and Blurpeace.", chan)
-			return
-		if not bad:
-			say("\x02%s\x0F is the bestest %s evah!" % (user, special), chan)
-		if bad:
-			say("\x02%s\x0F is worstest %s evah!" % (user, special), chan)
-		return
-	if command == "trout":
-		try:
-			user = line2[4]
-			user = ' '.join(line2[4:])
-		except Exception:
-			reply("Hahahahahahahaha...", chan, nick)
-			return
-		normal = unicodedata.normalize('NFKD', unicode(string.lower(user)))
-		if "itself" in normal:
-			reply("I'm not that stupid ;)", chan, nick)
-			return
-		elif "earwigbot" in normal:
-			reply("I'm not that stupid ;)", chan, nick)
-		elif "earwig" not in normal and "ear wig" not in normal:
-			text = 'slaps %s around a bit with a large trout.' % user
-			msg = '\x01ACTION %s\x01' % text
-			say(msg, chan)
-		else:
-			reply("I refuse to hurt anything with \"Earwig\" in its name :P", chan, nick)
-		return
-	if command == "kill" or command == "destroy" or command == "murder":
-		reply("Who do you think I am? The Mafia?", chan, nick)
-		return
-	if command == "fish":
-		try:
-			user = line2[4]
-			fish = ' '.join(line2[5:])
-		except Exception:
-			reply("Hahahahahahahaha...", chan, nick)
-			return
-		normal = unicodedata.normalize('NFKD', unicode(string.lower(user)))
-		if "itself" in normal:
-			reply("I'm not that stupid ;)", chan, nick)
-			return
-		elif "earwigbot" in normal:
-			reply("I'm not that stupid ;)", chan, nick)
-		elif "earwig" not in normal and "ear wig" not in normal:
-			text = 'slaps %s around a bit with a %s.' % (user, fish)
-			msg = '\x01ACTION %s\x01' % text
-			say(msg, chan)
-		else:
-			reply("I refuse to hurt anything with \"Earwig\" in its name :P", chan, nick)
-		return
-	if command == "report":
-		def find_status(name="", talk=False):
-			enname = re.sub(" ", "_", name)
-			if talk == True:
-				enname = "Wikipedia_talk:Articles_for_creation/%s" % enname
-				if talk == False:
-					enname = "Wikipedia:Articles_for_creation/%s" % enname
-				url = "http://en.wikipedia.org/w/api.php?action=query&titles=%s&prop=revisions&rvprop=content" % enname
-				query = urllib.urlopen(url)
-				data = query.read()
-				status = ""
-				if "{{AFC submission|D" in data or "{{AFC submission|d" in data:
-					reason = re.findall("(D|d)\|(.*?)\|", data)
-					if reason[0][1] != "reason":
-						status = "Declined, reason is '%s'" % reason[0][1]
-					if reason[0][1] == "reason":
-						status = "Declined, reason is a custom reason"
-				if "{{AFC submission|H" in data or "{{AFC submission|h" in data:
-					reason = re.findall("(H|h)\|(.*?)\|", data)
-					if reason[0][1] != "reason":
-						status = "Held, reason is '%s'" % reason[0][1]
-					if reason[0][1] == "reason":
-						status = "Held, reason is a custom reason"
-				if "{{AFC submission||" in data:
-					status = "Pending"
-				if "{{AFC submission|R" in data or "{{AFC submission|r" in data:
-					status = "Reviewing"
-				if not status:
-					exist = exists(name=enname)
-					if exist == True:
-						status = "Accepted"
-					if exist == False:
-						status = "Not found"
-				return status
-		def exists(name=""):
-			url = "http://en.wikipedia.org/wiki/%s" % name
-			query = urllib.urlopen(url)
-			data = query.read()
-			if "Wikipedia does not have a" in data:
-				return False
-			return True
-		def get_submitter(name="", talk=False):
-			enname = re.sub(" ", "_", name)
-			if talk == True:
-				enname = "Wikipedia_talk:Articles_for_creation/%s" % enname
-			if talk == False:
-				enname = "Wikipedia:Articles_for_creation/%s" % enname
-			url = "http://en.wikipedia.org/w/api.php?action=query&titles=%s&prop=revisions&rvprop=user&rvdir=newer&rvlimit=1" % enname
-			query = urllib.urlopen(url)
-			data = query.read()
-			extract = re.findall("user=&quot;(.*?)&quot;", data)
-			if "anon=" in data:
-				anon = True
-			else:
-				anon = False
-			try:
-				return extract[0], anon
-			except BaseException:
-				print extract
-				return "", anon
-		try:
-			rawSub = line2[4]
-			rawSub = ' '.join(line2[4:])
-		except Exception:
-			reply("You need to specify a submission name in order to use %s!" % command, chan, nick)
-			return
-		talk = False
-		if "[[" in rawSub and "]]" in rawSub:
-			name = re.sub("\[\[(.*)\]\]", "\\1", rawSub)
-			name = re.sub(" ", "_", name)
-			name = urllib.quote(name, ":/")
-			name = "http://en.wikipedia.org/wiki/%s" % name
-			if "talk:" in name:
-				talk = True
-		elif "http://" in rawSub:
-			name = rawSub
-			if "talk:" in name:
-				talk = True
-		elif "en.wikipedia.org" in rawSub:
-			name = "http://%s" % rawSub
-			if "talk:" in name:
-				talk = True
-		elif "Wikipedia:" in rawSub or "Wikipedia_talk:" in rawSub or "Wikipedia talk:" in rawSub:
-			name = re.sub(" ", "_", rawSub)
-			name = urllib.quote(name, ":/")
-			name = "http://en.wikipedia.org/wiki/%s" % name
-			if "talk:" in name:
-				talk = True
-		else:
-			url = "http://en.wikipedia.org/wiki/"
-			pagename = re.sub(" ", "_", rawSub)
-			pagename = urllib.quote(pagename, ":/")
-			pagename = "Wikipedia:Articles_for_creation/%s" % pagename
-			page = urllib.urlopen("%s%s" % (url, pagename))
-			text = page.read()
-			name = "http://en.wikipedia.org/wiki/%s" % pagename
-			if "Wikipedia does not have a" in text:
-				pagename = re.sub(" ", "_", rawSub)
-				pagename = urllib.quote(pagename, ":/")
-				pagename = "Wikipedia_talk:Articles_for_creation/%s" % pagename
-				page = urllib.urlopen("%s%s" % (url, pagename))
-				name = "http://en.wikipedia.org/wiki/%s" % pagename
-				talk = True
-		unname = re.sub("http://en.wikipedia.org/wiki/Wikipedia:Articles_for_creation/", "", name)
-		unname = re.sub("http://en.wikipedia.org/wiki/Wikipedia_talk:Articles_for_creation/", "", unname)
-		unname = re.sub("_", " ", unname)
-		if "talk" in unname:
-			talk = True
-		submitter, anon = get_submitter(name=unname, talk=talk)
-		status = find_status(name=unname, talk=talk)
-		if submitter != "":
-			if anon == True:
-				submitter_page = "Special:Contributions/%s" % submitter
-			if anon == False:
-				unsubmit = re.sub(" ", "_", submitter)
-				unsubmit = urllib.quote(unsubmit, ":/")
-				submitter_page = "User:%s" % unsubmit
-			if status == "Accepted":
-				submitterm = "Reviewer"
-			else:
-				submitterm = "Submitter"
-		line1 = "\x02AfC submission report for %s:" % unname
-		line2 = "\x02URL: \x0301\x0F%s" % name
-		if submitter != "":
-			line3 = "\x02%s: \x0F\x0302%s (\x0301\x0Fhttp://en.wikipedia.org/wiki/%s)." % (submitterm, submitter, submitter_page)
-		line4 = "\x02Status: \x0F\x0302%s." % status
-		say(line1, chan)
-		time.sleep(0.1)
-		say(line2, chan)
-		time.sleep(0.1)
-		if submitter != "":
-			say(line3, chan)
-			time.sleep(0.1)
-		say(line4, chan)
-		return
-	if command == "commands":
-		if chan.startswith("#"):
-			reply("Please use that command in a private message.", chan, nick)
-			return
-		others2 = get_commandList().values()
-		others = []
-		for com in others2:
-			if com == "copyvio" or com == "number" or com == "pending" or com == "report" or com == "submissions" or com == "access" or com == "help" or com == "join" or com == "linker" or com == "nick" or com == "op" or com == "part" or com == "quiet" or com == "quit" or com == "restart" or com == "voice" or com == "welcome" or com == "fish" or com == "praise" or com == "trout" or com == "notes":
-				continue
-			if com in others: continue
-			others.append(com)
-		others.sort()
-		say("\x02AFC commands:\x0F copyvio, number, pending, report, submissions.", chan)
-		time.sleep(0.1)
-		say("\x02Bot operation and channel maintaince commands:\x0F access, help, join, linker, nick, op, part, quiet, quit, restart, voice, welcome.", chan)
-		time.sleep(0.1)
-		say("\x02Fun commands:\x0F fish, praise, trout, and numerous easter eggs", chan)
-		time.sleep(0.1)
-		say("\x02Other commands:\x0F %s" % ', '.join(others), chan)
-		time.sleep(0.1)
-		say("The bot maintains a mini-wiki. Type \"!notes help\" for more information.", chan)
-		time.sleep(0.1)
-		say("See http://enwp.org/User:The_Earwig/Bots/IRC for details. For help on a specific command, type '!help command'.", chan)
-		return
-	if command == "help" or command == "doc" or command == "documentation":
-		try:
-			com = line2[4]
-		except Exception:
-			reply("Hi, I'm a bot that does work for Articles for Creation. You can find information about me at http://enwp.org/User:The_Earwig/Bots/IRC. Say \"!commands\" to me in a private message for some of my abilities. Earwig is my owner and creator, and you can contact him at http://enwp.org/User_talk:The_Earwig.", chan, nick)
-			return
-		say("Sorry, command documentation has not been implemented yet.", chan)
-		return
-	if command == "mysql":
-		if authy != "owner":
-			reply("You aren't authorized to use this command.", chan, nick)
-			return
-		import MySQLdb
-		try:
-			strings = line2[4]
-			strings = ' '.join(line2[4:])
-			if "db:" in strings:
-				database = re.findall("db\:(.*?)\s", strings)[0]
-			else:
-				database = "enwiki_p"
-			if "time:" in strings:
-				times = int(re.findall("time\:(.*?)\s", strings)[0])
-			else:
-				times = 60
-			file = re.findall("file\:(.*?)\s", strings)[0]
-			sqlquery = re.findall("query\:(.*?)\Z", strings)[0]
-		except Exception:
-			reply("You did not specify enough data for the bot to continue.", chan, nick)
-			return
-		database2 = database[:-2] + "-p"
-		db = MySQLdb.connect(db=database, host="%s.rrdb.toolserver.org" % database2, read_default_file="/home/earwig/.my.cnf")
-		db.query(sqlquery)
-		r = db.use_result()
-		data = r.fetch_row(0)
-		try:
-			f = codecs.open("/home/earwig/public_html/reports/%s/%s" % (database[:-2], file), 'r')
-			reply("A file already exists with that name.", chan, nick)
-			return
-		except Exception:
-			pass
-		f = codecs.open("/home/earwig/public_html/reports/%s/%s" % (database[:-2], file), 'a', 'utf-8')
-		for line in data:
-			new_line = []
-			for l in line:
-				new_line.append(str(l))
-			f.write('	'.join(new_line) + "\n")
-		f.close()
-		reply("Query completed successfully. See http://toolserver.org/~earwig/reports/%s/%s. I will delete the report in %s seconds." % (database[:-2], file, times), chan, nick)
-		time.sleep(times)
-		os.remove("/home/earwig/public_html/reports/%s/%s" % (database[:-2], file))
-		return
-	if command == "remind" or command == "reminder":
-		try:
-			times = int(line2[4])
-			content = ' '.join(line2[5:])
-		except Exception:
-			reply("Please specify a time and a note in the following format: !remind <time> <note>.", chan, nick)
-			return
-		reply("Set reminder for \"%s\" in %s seconds." % (content, times), chan, nick)
-		time.sleep(times)
-		reply(content, chan, nick)
-		return
-	if command == "notes" or command == "note" or command == "about" or command == "data" or command == "database":
-		try:
-			action = line2[4]
-		except BaseException:
-			reply("What do you want me to do? Type \"!notes help\" for more information.", chan, nick)
-			return
-		import MySQLdb
-		db = MySQLdb.connect(db="u_earwig_ircbot", host="sql", read_default_file="/home/earwig/.my.cnf")
-		specify = ' '.join(line2[5:])
-		if action == "help" or action == "manual":
-			shortCommandList = "read, write, change, undo, delete, move, author, category, list, report, developer"
-			if specify == "read":
-				say("To read an entry, type \"!notes read <entry>\".", chan)
-			elif specify == "write":
-				say("To write a new entry, type \"!notes write <entry> <content>\". This will create a new entry only if one does not exist, see the below command...", chan)
-			elif specify == "change":
-				say("To change an entry, type \"!notes change <entry> <new content>\". The old entry will be stored in the database, so it can be undone later.", chan)
-			elif specify == "undo":
-				say("To undo a change, type \"!notes undo <entry>\".", chan)
-			elif specify == "delete":
-				say("To delete an entry, type \"!notes delete <entry>\". For security reasons, only bot admins can do this.", chan)
-			elif specify == "move":
-				say("To move an entry, type \"!notes move <old_title> <new_title>\".", chan)
-			elif specify == "author":
-				say("To return the author of an entry, type \"!notes author <entry>\".", chan)
-			elif specify == "category" or specify == "cat":
-				say("To change an entry's category, type \"!notes category <entry> <category>\".", chan)
-			elif specify == "list":
-				say("To list all categories in the database, type \"!notes list\". Type \"!notes list <category>\" to get all entries in a certain category.", chan)
-			elif specify == "report":
-				say("To give some statistics about the mini-wiki, including some debugging information, type \"!notes report\" in a PM.", chan)
-			elif specify == "developer":
-				say("To do developer work, such as writing to the database directly, type \"!notes developer <command>\". This can only be done by the bot owner.", chan)
-			else:
-				db.query("SELECT * FROM version;")
-				r = db.use_result()
-				data = r.fetch_row(0)
-				version = data[0]
-				reply("The Earwig Mini-Wiki: running v%s." % version, chan, nick)
-				reply("The full list of commands, for reference, are: %s." % shortCommandList, chan, nick)
-				reply("For an explaination of a certain command, type \"!notes help <command>\".", chan, nick)
-				reply("You can also access the database from the Toolserver: http://toolserver.org/~earwig/cgi-bin/irc_database.py", chan, nick)
-				time.sleep(0.4)
-			return
-		elif action == "read":
-			specify = string.lower(specify)
-			if " " in specify: specify = string.split(specify, " ")[0]
-			if not specify or "\"" in specify:
-				reply("Please include the name of the entry you would like to read after the command, e.g. !notes read earwig", chan, nick)
-				return
-			try:
-				db.query("SELECT entry_content FROM entries WHERE entry_title = \"%s\";" % specify)
-				r = db.use_result()
-				data = r.fetch_row(0)
-				entry = data[0][0]
-				say("Entry \"\x02%s\x0F\": %s" % (specify, entry), chan)
-			except Exception:
-				reply("There is no entry titled \"\x02%s\x0F\"." % specify, chan, nick)
-			return
-		elif action == "delete" or action == "remove":
-			specify = string.lower(specify)
-			if " " in specify: specify = string.split(specify, " ")[0]
-			if not specify or "\"" in specify:
-				reply("Please include the name of the entry you would like to delete after the command, e.g. !notes delete earwig", chan, nick)
-				return
-			if authy == "owner" or authy == "admin":
-				try:
-					db.query("DELETE from entries where entry_title = \"%s\";" % specify)
-					r = db.use_result()
-					db.commit()
-					reply("The entry on \"\x02%s\x0F\" has been removed." % specify, chan, nick)
-				except Exception:
-					phenny.reply("Unable to remove the entry on \"\x02%s\x0F\", because it doesn't exist." % specify, chan, nick)
-			else:
-				reply("Only bot admins can remove entries.", chan, nick)
-			return
-		elif action == "developer":
-			if authy == "owner":
-				db.query(specify)
-				r = db.use_result()
-				try:
-					print r.fetch_row(0)
-				except Exception:
-					pass
-				db.commit()
-				reply("Done.", chan, nick)
-			else:
-				reply("Only the bot owner can modify the raw database.", chan, nick)
-			return
-		elif action == "write":
-			try:
-				write = line2[5]
-				content = ' '.join(line2[6:])
-			except Exception:
-				reply("Please include some content in your entry.", chan, nick)
-				return
-			db.query("SELECT * from entries WHERE entry_title = \"%s\";" % write)
-			r = db.use_result()
-			data = r.fetch_row(0)
-			if data:
-				reply("An entry on %s already exists; please use \"!notes change %s %s\"." % (write, write, content), chan, nick)
-				return
-			content2 = content.replace('"', '\\' + '"')
-			db.query("INSERT INTO entries (entry_title, entry_author, entry_category, entry_content, entry_content_old) VALUES (\"%s\", \"%s\", \"uncategorized\", \"%s\", NULL);" % (write, nick, content2))
-			db.commit()
-			reply("You have written an entry titled \"\x02%s\x0F\", with the following content: \"%s\"" % (write, content), chan, nick)
-			return
-		elif action == "change":
-			reply("NotImplementedError", chan, nick)
-		elif action == "undo":
-			reply("NotImplementedError", chan, nick)
-		elif action == "move":
-			reply("NotImplementedError", chan, nick)
-		elif action == "author":
-			try:
-				entry = line2[5]
-			except Exception:
-				reply("Please include the name of the entry you would like to get information for after the command, e.g. !notes author earwig", chan, nick)
-				return
-			db.query("SELECT entry_author from entries WHERE entry_title = \"%s\";" % entry)
-			r = db.use_result()
-			data = r.fetch_row(0)
-			if data:
-				say("The author of \"\x02%s\x0F\" is \x02%s\x0F." % (entry, data[0][0]), chan)
-				return
-			reply("There is no entry titled \"\x02%s\x0F\"." % entry, chan, nick)
-			return
-		elif action == "cat" or action == "category":
-			reply("NotImplementedError", chan, nick)
-		elif action == "list":
-			reply("NotImplementedError", chan, nick)
-		elif action == "report":
-			reply("NotImplementedError", chan, nick)
-	if command == "hash":
-		import hashlib
-		try:
-			hashVia = line2[4]
-			hashText = line2[5]
-			hashText = ' '.join(line2[5:])
-		except Exception:
-			reply("Please provide a string and method to hash by.", chan, nick)
-			return
-		try:
-			hashed = eval("hashlib.%s(\"%s\").hexdigest()" % (hashVia, hashText))
-			reply(hashed, chan, nick)
-		except Exception:
-			try:
-				hashing = hashlib.new(hashVia)
-				hashing.update(hashText)
-				hashed = hashing.hexdigest()
-				reply(hashed, chan, nick)
-			except Exception:
-				reply("Error.", chan, nick)
-	if command == "langcode" or command == "lang" or command == "language":
-		try:
-			lang = line2[4]
-		except Exception:
-			reply("Please specify an ISO code.", chan, nick)
-			return
-		data = urllib.urlopen("http://toolserver.org/~earwig/cgi-bin/swmt.py?action=iso").read()
-		data = string.split(data, "\n")
-		result = False
-		for datum in data:
-			if datum.startswith(lang):
-				result = re.findall(".*? (.*)", datum)[0]
-				break
-		if result:
-			reply(result, chan, nick)
-			return
-		reply("Not found.", chan, nick)
-		return
-	if command == "lookup" or command == "ip":
-		try:
-			hexIP = line2[4]
-		except Exception:
-			reply("Please specify a hex IP address.", chan, nick)
-			return
-		hexes = [hexIP[:2], hexIP[2:4], hexIP[4:6], hexIP[6:8]]
-		hashes = []
-		for hexHash in hexes:
-			newHex = int(hexHash, 16)
-			hashes.append(newHex)
-		normalizedIP = "%s.%s.%s.%s" % (hashes[0], hashes[1], hashes[2], hashes[3])
-		reply(normalizedIP, chan, nick)
-		return

earwigbot/commands/afc_pending.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from earwigbot.commands import Command
+
+class AFCPending(Command):
+    """Link the user to the pending AFC submissions page and category."""
+    name = "pending"
+    commands = ["pending", "pend"]
+
+    def process(self, data):
+        msg1 = "pending submissions status page: http://enwp.org/WP:AFC/ST"
+        msg2 = "pending submissions category: http://enwp.org/CAT:PEND"
+        self.reply(data, msg1)
+        self.reply(data, msg2)

earwigbot/commands/afc_report.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,31 +20,29 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-import re
-
-from earwigbot.classes import BaseCommand
-from earwigbot import tasks
 from earwigbot import wiki
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class AFCReport(Command):
     """Get information about an AFC submission by name."""
     name = "report"
 
     def process(self, data):
-        self.site = wiki.get_site()
-        self.site._maxlag = None
+        self.site = self.bot.wiki.get_site()
         self.data = data
 
         try:
-            self.statistics = tasks.get("afc_statistics")
+            self.statistics = self.bot.tasks.get("afc_statistics")
         except KeyError:
-            e = "Cannot run command: requires afc_statistics task."
+            e = "Cannot run command: requires afc_statistics task (from earwigbot_plugins)"
             self.logger.error(e)
+            msg = "command requires afc_statistics task (from earwigbot_plugins)"
+            self.reply(data, msg)
             return
 
         if not data.args:
             msg = "what submission do you want me to give information about?"
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
 
         title = " ".join(data.args)
@@ -68,21 +66,20 @@ class Command(BaseCommand):
         if page:
             return self.report(page)
 
-        msg = "submission \x0302{0}\x0301 not found.".format(title)
-        self.connection.reply(data, msg)
+        self.reply(data, "submission \x0302{0}\x0301 not found.".format(title))
 
     def get_page(self, title):
         page = self.site.get_page(title, follow_redirects=False)
-        if page.exists()[0]:
+        if page.exists == page.PAGE_EXISTS:
             return page
 
     def report(self, page):
-        url = page.url().replace("en.wikipedia.org/wiki", "enwp.org")
-        short = self.statistics.get_short_title(page.title())
+        url = page.url.replace("en.wikipedia.org/wiki", "enwp.org")
+        short = self.statistics.get_short_title(page.title)
         status = self.get_status(page)
-        user = self.site.get_user(page.creator())
-        user_name = user.name()
-        user_url = user.get_talkpage().url()
+        user = page.get_creator()
+        user_name = user.name
+        user_url = user.get_talkpage().url
 
         msg1 = "AfC submission report for \x0302{0}\x0301 ({1}):"
         msg2 = "Status: \x0303{0}\x0301"
@@ -90,14 +87,14 @@ class Command(BaseCommand):
         if status == "accepted":
             msg3 = "Reviewed by \x0302{0}\x0301 ({1})"
 
-        self.connection.reply(self.data, msg1.format(short, url))
-        self.connection.say(self.data.chan, msg2.format(status))
-        self.connection.say(self.data.chan, msg3.format(user_name, user_url))
+        self.reply(self.data, msg1.format(short, url))
+        self.say(self.data.chan, msg2.format(status))
+        self.say(self.data.chan, msg3.format(user_name, user_url))
 
     def get_status(self, page):
-        if page.is_redirect():
+        if page.is_redirect:
             target = page.get_redirect_target()
-            if self.site.get_page(target).namespace() == wiki.NS_MAIN:
+            if self.site.get_page(target).namespace == wiki.NS_MAIN:
                 return "accepted"
             return "redirect"
 

earwigbot/commands/afc_status.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,36 +22,32 @@
 
 import re
 
-from earwigbot import wiki
-from earwigbot.classes import BaseCommand
-from earwigbot.config import config
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class AFCStatus(Command):
     """Get the number of pending AfC submissions, open redirect requests, and
     open file upload requests."""
     name = "status"
+    commands = ["status", "count", "num", "number"]
     hooks = ["join", "msg"]
 
     def check(self, data):
-        commands = ["status", "count", "num", "number"]
-        if data.is_command and data.command in commands:
+        if data.is_command and data.command in self.commands:
             return True
-
         try:
             if data.line[1] == "JOIN" and data.chan == "#wikipedia-en-afc":
-                if data.nick != config.irc["frontend"]["nick"]:
+                if data.nick != self.config.irc["frontend"]["nick"]:
                     return True
         except IndexError:
             pass
         return False
 
     def process(self, data):
-        self.site = wiki.get_site()
-        self.site._maxlag = None
+        self.site = self.bot.wiki.get_site()
 
         if data.line[1] == "JOIN":
             status = " ".join(("\x02Current status:\x0F", self.get_status()))
-            self.connection.notice(data.nick, status)
+            self.notice(data.nick, status)
             return
 
         if data.args:
@@ -59,17 +55,17 @@ class Command(BaseCommand):
             if action.startswith("sub") or action == "s":
                 subs = self.count_submissions()
                 msg = "there are \x0305{0}\x0301 pending AfC submissions (\x0302WP:AFC\x0301)."
-                self.connection.reply(data, msg.format(subs))
+                self.reply(data, msg.format(subs))
 
             elif action.startswith("redir") or action == "r":
                 redirs = self.count_redirects()
                 msg = "there are \x0305{0}\x0301 open redirect requests (\x0302WP:AFC/R\x0301)."
-                self.connection.reply(data, msg.format(redirs))
+                self.reply(data, msg.format(redirs))
 
             elif action.startswith("file") or action == "f":
                 files = self.count_redirects()
                 msg = "there are \x0305{0}\x0301 open file upload requests (\x0302WP:FFU\x0301)."
-                self.connection.reply(data, msg.format(files))
+                self.reply(data, msg.format(files))
 
             elif action.startswith("agg") or action == "a":
                 try:
@@ -80,21 +76,21 @@ class Command(BaseCommand):
                     agg_num = self.get_aggregate_number(agg_data)
                 except ValueError:
                     msg = "\x0303{0}\x0301 isn't a number!"
-                    self.connection.reply(data, msg.format(data.args[1]))
+                    self.reply(data, msg.format(data.args[1]))
                     return
                 aggregate = self.get_aggregate(agg_num)
                 msg = "aggregate is \x0305{0}\x0301 (AfC {1})."
-                self.connection.reply(data, msg.format(agg_num, aggregate))
+                self.reply(data, msg.format(agg_num, aggregate))
 
             elif action.startswith("nocolor") or action == "n":
-                self.connection.reply(data, self.get_status(color=False))
+                self.reply(data, self.get_status(color=False))
 
             else:
                 msg = "unknown argument: \x0303{0}\x0301. Valid args are 'subs', 'redirs', 'files', 'agg', 'nocolor'."
-                self.connection.reply(data, msg.format(data.args[0]))
+                self.reply(data, msg.format(data.args[0]))
 
         else:
-            self.connection.reply(data, self.get_status())
+            self.reply(data, self.get_status())
 
     def get_status(self, color=True):
         subs = self.count_submissions()
@@ -111,12 +107,9 @@ class Command(BaseCommand):
 
     def count_submissions(self):
         """Returns the number of open AFC submissions (count of CAT:PEND)."""
-        cat = self.site.get_category("Pending AfC submissions")
-        subs = len(cat.members(limit=2500, use_sql=True))
-
-        # Remove [[Wikipedia:Articles for creation/Redirects]] and
+        # Subtract two for [[Wikipedia:Articles for creation/Redirects]] and
         # [[Wikipedia:Files for upload]], which aren't real submissions:
-        return subs - 2
+        return self.site.get_category("Pending AfC submissions").pages - 2
 
     def count_redirects(self):
         """Returns the number of open redirect submissions. Calculated as the
@@ -140,30 +133,30 @@ class Command(BaseCommand):
     def get_aggregate(self, num):
         """Returns a human-readable AFC status based on the number of pending
         AFC submissions, open redirect requests, and open FFU requests. This
-        does not match {{AFC status}} directly because my algorithm factors in
+        does not match {{AFC status}} directly because the algorithm factors in
         WP:AFC/R and WP:FFU while the template only looks at the main
-        submissions. My reasoning is that AFC/R and FFU are still part of
+        submissions. The reasoning is that AFC/R and FFU are still part of
         the project, so even if there are no pending submissions, a backlog at
         FFU (for example) indicates that our work is *not* done and the
         project-wide backlog is most certainly *not* clear."""
         if num == 0:
             return "is \x02\x0303clear\x0301\x0F"
-        elif num < 125:  # < 25 subs
+        elif num <= 200:
             return "is \x0303almost clear\x0301"
-        elif num < 200:  # < 40 subs
+        elif num <= 400:
             return "is \x0312normal\x0301"
-        elif num < 275:  # < 55 subs
+        elif num <= 600:
             return "is \x0307lightly backlogged\x0301"
-        elif num < 350:  # < 70 subs
+        elif num <= 900:
             return "is \x0304backlogged\x0301"
-        elif num < 500:  # < 100 subs
+        elif num <= 1200:
             return "is \x02\x0304heavily backlogged\x0301\x0F"
-        else:  # >= 100 subs
+        else:
             return "is \x02\x1F\x0304severely backlogged\x0301\x0F"
 
     def get_aggregate_number(self, (subs, redirs, files)):
         """Returns an 'aggregate number' based on the real number of pending
         submissions in CAT:PEND (subs), open redirect submissions in WP:AFC/R
         (redirs), and open files-for-upload requests in WP:FFU (files)."""
-        num = (subs * 5) + (redirs * 2) + (files * 2)
+        num = subs + (redirs / 2) + (files / 2)
         return num

earwigbot/commands/afc_submissions.py

@@ -0,0 +1,59 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from earwigbot.commands import Command
+
+class AFCSubmissions(Command):
+    """Link the user directly to some pending AFC submissions."""
+    name = "submissions"
+    commands = ["submissions", "subs"]
+
+    def setup(self):
+        try:
+            self.ignore_list = self.config.commands[self.name]["ignoreList"]
+        except KeyError:
+            try:
+                ignores = self.config.tasks["afc_statistics"]["ignoreList"]
+                self.ignore_list = ignores
+            except KeyError:
+                self.ignore_list = []
+
+    def process(self, data):
+        if data.args:
+            try:
+                number = int(data.args[0])
+            except ValueError:
+                self.reply(data, "argument must be a number.")
+                return
+            if number > 5:
+                msg = "cannot get more than five submissions at a time."
+                self.reply(data, msg)
+                return
+        else:
+            number = 3
+
+        site = self.bot.wiki.get_site()
+        category = site.get_category("Pending AfC submissions")
+        members = category.get_members(limit=number + len(self.ignore_list))
+        urls = [member.url for member in members if member.title not in self.ignore_list]
+        pages = ", ".join(urls[:number])
+        self.reply(data, "{0} pending AfC subs: {1}".format(number, pages))

earwigbot/commands/calc.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,16 +23,16 @@
 import re
 import urllib
 
-from earwigbot.classes import BaseCommand
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Calc(Command):
     """A somewhat advanced calculator: see http://futureboy.us/fsp/frink.fsp
     for details."""
     name = "calc"
 
     def process(self, data):
         if not data.args:
-            self.connection.reply(data, "what do you want me to calculate?")
+            self.reply(data, "what do you want me to calculate?")
             return
 
         query = ' '.join(data.args)
@@ -47,7 +47,7 @@ class Command(BaseCommand):
 
         match = r_result.search(result)
         if not match:
-            self.connection.reply(data, "Calculation error.")
+            self.reply(data, "Calculation error.")
             return
 
         result = match.group(1)
@@ -58,26 +58,26 @@ class Command(BaseCommand):
 
         if not result:
             result = '?'
-        elif " in " in query: 
+        elif " in " in query:
             result += " " + query.split(" in ", 1)[1]
 
         res = "%s = %s" % (query, result)
-        self.connection.reply(data, res)
+        self.reply(data, res)
 
     def cleanup(self, query):
         fixes = [
-            (' in ', ' -> '), 
-            (' over ', ' / '), 
-            (u'£', 'GBP '), 
-            (u'€', 'EUR '), 
-            ('\$', 'USD '), 
-            (r'\bKB\b', 'kilobytes'), 
-            (r'\bMB\b', 'megabytes'), 
-            (r'\bGB\b', 'kilobytes'), 
-            ('kbps', '(kilobits / second)'), 
+            (' in ', ' -> '),
+            (' over ', ' / '),
+            (u'£', 'GBP '),
+            (u'€', 'EUR '),
+            ('\$', 'USD '),
+            (r'\bKB\b', 'kilobytes'),
+            (r'\bMB\b', 'megabytes'),
+            (r'\bGB\b', 'kilobytes'),
+            ('kbps', '(kilobits / second)'),
             ('mbps', '(megabits / second)')
         ]
 
-        for original, fix in fixes: 
+        for original, fix in fixes:
             query = re.sub(original, fix, query)
         return query.strip()

earwigbot/commands/chanops.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,36 +20,69 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseCommand
-from earwigbot.config import config
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
-    """Voice, devoice, op, or deop users in the channel."""
+class ChanOps(Command):
+    """Voice, devoice, op, or deop users in the channel, or join or part from
+    other channels."""
     name = "chanops"
-
-    def check(self, data):
-        commands = ["chanops", "voice", "devoice", "op", "deop"]
-        if data.is_command and data.command in commands:
-            return True
-        return False
+    commands = ["chanops", "voice", "devoice", "op", "deop", "join", "part"]
 
     def process(self, data):
         if data.command == "chanops":
-            msg = "available commands are !voice, !devoice, !op, and !deop."
-            self.connection.reply(data, msg)
+            msg = "available commands are !voice, !devoice, !op, !deop, !join, and !part."
+            self.reply(data, msg)
             return
-
-        if data.host not in config.irc["permissions"]["admins"]:
-            msg = "you must be a bot admin to use this command."
-            self.connection.reply(data, msg)
+        if data.host not in self.config.irc["permissions"]["admins"]:
+            self.reply(data, "you must be a bot admin to use this command.")
             return
 
-        # If it is just !op/!devoice/whatever without arguments, assume they
-        # want to do this to themselves:
-        if not data.args:
-            target = data.nick
+        if data.command == "join":
+            self.do_join(data)
+        elif data.command == "part":
+            self.do_part(data)
         else:
-            target = data.args[0]
+            # If it is just !op/!devoice/whatever without arguments, assume
+            # they want to do this to themselves:
+            if not data.args:
+                target = data.nick
+            else:
+                target = data.args[0]
+            command = data.command.upper()
+            self.say("ChanServ", " ".join((command, data.chan, target)))
+            log = "{0} requested {1} on {2} in {3}"
+            self.logger.info(log.format(data.nick, command, target, data.chan))
+
+    def do_join(self, data):
+        if data.args:
+            channel = data.args[0]
+            if not channel.startswith("#"):
+                channel = "#" + channel
+        else:
+            msg = "you must specify a channel to join or part from."
+            self.reply(data, msg)
+            return
+
+        self.join(channel)
+        log = "{0} requested JOIN to {1}".format(data.nick, channel)
+        self.logger.info(log)
+
+    def do_part(self, data):
+        channel = data.chan
+        reason = None
+        if data.args:
+            if data.args[0].startswith("#"):
+                # "!part #channel reason for parting"
+                channel = data.args[0]
+                if data.args[1:]:
+                    reason = " ".join(data.args[1:])
+            else:  # "!part reason for parting"; assume current channel
+                reason = " ".join(data.args)
 
-        msg = " ".join((data.command, data.chan, target))
-        self.connection.say("ChanServ", msg)
+        msg = "Requested by {0}".format(data.nick)
+        log = "{0} requested PART from {1}".format(data.nick, channel)
+        if reason:
+            msg += ": {0}".format(reason)
+            log += ' ("{0}")'.format(reason)
+        self.part(channel, msg)
+        self.logger.info(log)

earwigbot/commands/crypt.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,29 +22,25 @@
 
 import hashlib
 
-from earwigbot.classes import BaseCommand
-from earwigbot import blowfish
+from Crypto.Cipher import Blowfish
 
-class Command(BaseCommand):
+from earwigbot.commands import Command
+
+class Crypt(Command):
     """Provides hash functions with !hash (!hash list for supported algorithms)
-    and blowfish encryption with !encrypt and !decrypt."""
+    and Blowfish encryption with !encrypt and !decrypt."""
     name = "crypt"
-
-    def check(self, data):
-        commands = ["crypt", "hash", "encrypt", "decrypt"]
-        if data.is_command and data.command in commands:
-            return True
-        return False
+    commands = ["crypt", "hash", "encrypt", "decrypt"]
 
     def process(self, data):
         if data.command == "crypt":
             msg = "available commands are !hash, !encrypt, and !decrypt."
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
 
         if not data.args:
             msg = "what do you want me to {0}?".format(data.command)
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
 
         if data.command == "hash":
@@ -52,29 +48,29 @@ class Command(BaseCommand):
             if algo == "list":
                 algos = ', '.join(hashlib.algorithms)
                 msg = algos.join(("supported algorithms: ", "."))
-                self.connection.reply(data, msg)
+                self.reply(data, msg)
             elif algo in hashlib.algorithms:
                 string = ' '.join(data.args[1:])
                 result = getattr(hashlib, algo)(string).hexdigest()
-                self.connection.reply(data, result)
+                self.reply(data, result)
             else:
                 msg = "unknown algorithm: '{0}'.".format(algo)
-                self.connection.reply(data, msg)
+                self.reply(data, msg)
 
         else:
             key = data.args[0]
-            text = ' '.join(data.args[1:])
+            text = " ".join(data.args[1:])
 
             if not text:
                 msg = "a key was provided, but text to {0} was not."
-                self.connection.reply(data, msg.format(data.command))
+                self.reply(data, msg.format(data.command))
                 return
 
+            cipher = Blowfish.new(hashlib.sha256(key).digest())
             try:
                 if data.command == "encrypt":
-                    self.connection.reply(data, blowfish.encrypt(key, text))
+                    self.reply(data, cipher.encrypt(text).encode("hex"))
                 else:
-                    self.connection.reply(data, blowfish.decrypt(key, text))
-            except blowfish.BlowfishError as error:
-                msg = "{0}: {1}.".format(error.__class__.__name__, error)
-                self.connection.reply(data, msg)
+                    self.reply(data, cipher.decrypt(text.decode("hex")))
+            except ValueError as error:
+                self.reply(data, error.message)

earwigbot/commands/ctcp.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,13 +23,12 @@
 import platform
 import time
 
-import earwigbot
-from earwigbot.classes import BaseCommand
-from earwigbot.config import config
+from earwigbot import __version__
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
-    """Not an actual command, this module is used to respond to the CTCP
-    commands PING, TIME, and VERSION."""
+class CTCP(Command):
+    """Not an actual command; this module implements responses to the CTCP
+    requests PING, TIME, and VERSION."""
     name = "ctcp"
     hooks = ["msg_private"]
 
@@ -53,17 +52,17 @@ class Command(BaseCommand):
         if command == "PING":
             msg = " ".join(data.line[4:])
             if msg:
-                self.connection.notice(target, "\x01PING {0}\x01".format(msg))
+                self.notice(target, "\x01PING {0}\x01".format(msg))
             else:
-                self.connection.notice(target, "\x01PING\x01")
+                self.notice(target, "\x01PING\x01")
 
         elif command == "TIME":
             ts = time.strftime("%a, %d %b %Y %H:%M:%S %Z", time.localtime())
-            self.connection.notice(target, "\x01TIME {0}\x01".format(ts))
+            self.notice(target, "\x01TIME {0}\x01".format(ts))
 
         elif command == "VERSION":
             default = "EarwigBot - $1 - Python/$2 https://github.com/earwig/earwigbot"
-            vers = config.irc.get("version", default)
-            vers = vers.replace("$1", earwigbot.__version__)
+            vers = self.config.irc.get("version", default)
+            vers = vers.replace("$1", __version__)
             vers = vers.replace("$2", platform.python_version())
-            self.connection.notice(target, "\x01VERSION {0}\x01".format(vers))
+            self.notice(target, "\x01VERSION {0}\x01".format(vers))

earwigbot/commands/editcount.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,18 +22,13 @@
 
 from urllib import quote_plus
 
-from earwigbot.classes import BaseCommand
-from earwigbot import wiki
+from earwigbot import exceptions
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Editcount(Command):
     """Return a user's edit count."""
     name = "editcount"
-
-    def check(self, data):
-        commands = ["ec", "editcount"]
-        if data.is_command and data.command in commands:
-            return True
-        return False
+    commands = ["ec", "editcount"]
 
     def process(self, data):
         if not data.args:
@@ -41,18 +36,18 @@ class Command(BaseCommand):
         else:
             name = ' '.join(data.args)
 
-        site = wiki.get_site()
-        site._maxlag = None
+        site = self.bot.wiki.get_site()
         user = site.get_user(name)
 
         try:
-            count = user.editcount()
-        except wiki.UserNotFoundError:
+            count = user.editcount
+        except exceptions.UserNotFoundError:
             msg = "the user \x0302{0}\x0301 does not exist."
-            self.connection.reply(data, msg.format(name))
+            self.reply(data, msg.format(name))
             return
 
-        safe = quote_plus(user.name())
-        url = "http://toolserver.org/~tparis/pcount/index.php?name={0}&lang=en&wiki=wikipedia"
+        safe = quote_plus(user.name.encode("utf8"))
+        url = "http://toolserver.org/~tparis/pcount/index.php?name={0}&lang={1}&wiki={2}"
+        fullurl = url.format(safe, site.lang, site.project)
         msg = "\x0302{0}\x0301 has {1} edits ({2})."
-        self.connection.reply(data, msg.format(name, count, url.format(safe)))
+        self.reply(data, msg.format(name, count, fullurl))

earwigbot/commands/geolocate.py

@@ -0,0 +1,72 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import json
+import urllib2
+
+from earwigbot.commands import Command
+
+class Geolocate(Command):
+    """Geolocate an IP address (via http://ipinfodb.com/)."""
+    name = "geolocate"
+    commands = ["geolocate", "locate", "geo", "ip"]
+
+    def setup(self):
+        self.config.decrypt(self.config.commands, self.name, "apiKey")
+        try:
+            self.key = self.config.commands[self.name]["apiKey"]
+        except KeyError:
+            self.key = None
+            log = 'Cannot use without an API key for http://ipinfodb.com/ stored as config.commands["{0}"]["apiKey"]'
+            self.logger.warn(log.format(self.name))
+
+    def process(self, data):
+        if not data.args:
+            self.reply(data, "please specify an IP to lookup.")
+            return
+
+        if not self.key:
+            msg = 'I need an API key for http://ipinfodb.com/ stored as \x0303config.commands["{0}"]["apiKey"]\x0301.'
+            log = 'Need an API key for http://ipinfodb.com/ stored as config.commands["{0}"]["apiKey"]'
+            self.reply(data, msg.format(self.name) + ".")
+            self.logger.error(log.format(self.name))
+            return
+
+        address = data.args[0]
+        url = "http://api.ipinfodb.com/v3/ip-city/?key={0}&ip={1}&format=json"
+        query = urllib2.urlopen(url.format(self.key, address)).read()
+        res = json.loads(query)
+
+        try:
+            country = res["countryName"].title()
+            region = res["regionName"].title()
+            city = res["cityName"].title()
+            latitude = res["latitude"]
+            longitude = res["longitude"]
+            utcoffset = res["timeZone"]
+        except KeyError:
+            self.reply(data, "IP \x0302{0}\x0301 not found.".format(address))
+            return
+
+        msg = "{0}, {1}, {2} ({3}, {4}), UTC {5}"
+        geo = msg.format(country, region, city, latitude, longitude, utcoffset)
+        self.reply(data, geo)

earwigbot/commands/git.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,61 +20,92 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-import shlex
-import subprocess
-import re
+import time
+
+import git
 
-from earwigbot.classes import BaseCommand
-from earwigbot.config import config
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Git(Command):
     """Commands to interface with the bot's git repository; use '!git' for a
     sub-command list."""
     name = "git"
 
+    def setup(self):
+        try:
+            self.repos = self.config.commands[self.name]["repos"]
+        except KeyError:
+            self.repos = None
+
     def process(self, data):
         self.data = data
-        if data.host not in config.irc["permissions"]["owners"]:
+        if data.host not in self.config.irc["permissions"]["owners"]:
             msg = "you must be a bot owner to use this command."
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
-
-        if not data.args:
+        if not data.args or data.args[0] == "help":
             self.do_help()
             return
+        if not self.repos:
+            self.reply(data, "no repos are specified in the config file.")
+            return
 
-        if data.args[0] == "help":
-            self.do_help()
+        command = data.args[0]
+        try:
+            repo_name = data.args[1]
+        except IndexError:
+            repos = self.get_repos()
+            msg = "which repo do you want to work with (options are {0})?"
+            self.reply(data, msg.format(repos))
+            return
+        if repo_name not in self.repos:
+            repos = self.get_repos()
+            msg = "repository must be one of the following: {0}."
+            self.reply(data, msg.format(repos))
+            return
+        self.repo = git.Repo(self.repos[repo_name])
 
-        elif data.args[0] == "branch":
+        if command == "branch":
             self.do_branch()
-
-        elif data.args[0] == "branches":
+        elif command == "branches":
             self.do_branches()
-
-        elif data.args[0] == "checkout":
+        elif command == "checkout":
             self.do_checkout()
-
-        elif data.args[0] == "delete":
+        elif command == "delete":
             self.do_delete()
-
-        elif data.args[0] == "pull":
+        elif command == "pull":
             self.do_pull()
-
-        elif data.args[0] == "status":
+        elif command == "status":
             self.do_status()
-
         else:  # They asked us to do something we don't know
             msg = "unknown argument: \x0303{0}\x0301.".format(data.args[0])
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
 
-    def exec_shell(self, command):
-        """Execute a shell command and get the output."""
-        command = shlex.split(command)
-        result = subprocess.check_output(command, stderr=subprocess.STDOUT)
-        if result:
-            result = result[:-1]  # Strip newline
-        return result
+    def get_repos(self):
+        data = self.repos.iteritems()
+        repos = ["\x0302{0}\x0301 ({1})".format(k, v) for k, v in data]
+        return ", ".join(repos)
+
+    def get_remote(self):
+        try:
+            remote_name = self.data.args[2]
+        except IndexError:
+            remote_name = "origin"
+        try:
+            return getattr(self.repo.remotes, remote_name)
+        except AttributeError:
+            msg = "unknown remote: \x0302{0}\x0301.".format(remote_name)
+            self.reply(self.data, msg)
+
+    def get_time_since(self, date):
+        diff = time.mktime(time.gmtime()) - date
+        if diff < 60:
+            return "{0} seconds".format(int(diff))
+        if diff < 60 * 60:
+            return "{0} minutes".format(int(diff / 60))
+        if diff < 60 * 60 * 24:
+            return "{0} hours".format(int(diff / 60 / 60))
+        return "{0} days".format(int(diff / 60 / 60 / 24))
 
     def do_help(self):
         """Display all commands."""
@@ -86,110 +117,124 @@ class Command(BaseCommand):
             "pull": "update everything from the remote server",
             "status": "check if we are up-to-date",
         }
-        msg = ""
+        subcommands = ""
         for key in sorted(help.keys()):
-            msg += "\x0303{0}\x0301 ({1}), ".format(key, help[key])
-        msg = msg[:-2]  # Trim last comma and space
-        self.connection.reply(self.data, "sub-commands are: {0}.".format(msg))
+            subcommands += "\x0303{0}\x0301 ({1}), ".format(key, help[key])
+        subcommands = subcommands[:-2]  # Trim last comma and space
+        msg = "sub-commands are: {0}; repos are: {1}. Syntax: !git \x0303subcommand\x0301 \x0302repo\x0301."
+        self.reply(self.data, msg.format(subcommands, self.get_repos()))
 
     def do_branch(self):
         """Get our current branch."""
-        branch = self.exec_shell("git name-rev --name-only HEAD")
+        branch = self.repo.active_branch.name
         msg = "currently on branch \x0302{0}\x0301.".format(branch)
-        self.connection.reply(self.data, msg)
+        self.reply(self.data, msg)
 
     def do_branches(self):
         """Get a list of branches."""
-        branches = self.exec_shell("git branch")
-        # Remove extraneous characters:
-        branches = branches.replace('\n* ', ', ')
-        branches = branches.replace('* ', ' ')
-        branches = branches.replace('\n  ', ', ')
-        branches = branches.strip()
-        msg = "branches: \x0302{0}\x0301.".format(branches)
-        self.connection.reply(self.data, msg)
+        branches = [branch.name for branch in self.repo.branches]
+        msg = "branches: \x0302{0}\x0301.".format(", ".join(branches))
+        self.reply(self.data, msg)
 
     def do_checkout(self):
         """Switch branches."""
         try:
-            branch = self.data.args[1]
-        except IndexError: # no branch name provided
-            self.connection.reply(self.data, "switch to which branch?")
+            target = self.data.args[2]
+        except IndexError:  # No branch name provided
+            self.reply(self.data, "switch to which branch?")
             return
 
-        current_branch = self.exec_shell("git name-rev --name-only HEAD")
+        current_branch = self.repo.active_branch.name
+        if target == current_branch:
+            msg = "already on \x0302{0}\x0301!".format(target)
+            self.reply(self.data, msg)
+            return
 
         try:
-            result = self.exec_shell("git checkout %s" % branch)
-            if "Already on" in result:
-                msg = "already on \x0302{0}\x0301!".format(branch)
-                self.connection.reply(self.data, msg)
-            else:
-                ms = "switched from branch \x0302{1}\x0301 to \x0302{1}\x0301."
-                msg = ms.format(current_branch, branch)
-                self.connection.reply(self.data, msg)
-
-        except subprocess.CalledProcessError:
-            # Git couldn't switch branches; assume the branch doesn't exist:
-            msg = "branch \x0302{0}\x0301 doesn't exist!".format(branch)
-            self.connection.reply(self.data, msg)
+            ref = getattr(self.repo.branches, target)
+        except AttributeError:
+            msg = "branch \x0302{0}\x0301 doesn't exist!".format(target)
+            self.reply(self.data, msg)
+        else:
+            ref.checkout()
+            ms = "switched from branch \x0302{0}\x0301 to \x0302{1}\x0301."
+            msg = ms.format(current_branch, target)
+            self.reply(self.data, msg)
+            log = "{0} checked out branch {1} of {2}"
+            logmsg = log.format(self.data.nick, target, self.repo.working_dir)
+            self.logger.info(logmsg)
 
     def do_delete(self):
         """Delete a branch, while making sure that we are not already on it."""
         try:
-            delete_branch = self.data.args[1]
-        except IndexError: # no branch name provided
-            self.connection.reply(self.data, "delete which branch?")
+            target = self.data.args[2]
+        except IndexError:  # No branch name provided
+            self.reply(self.data, "delete which branch?")
             return
 
-        current_branch = self.exec_shell("git name-rev --name-only HEAD")
-
-        if current_branch == delete_branch:
+        current_branch = self.repo.active_branch.name
+        if current_branch == target:
             msg = "you're currently on this branch; please checkout to a different branch before deleting."
-            self.connection.reply(self.data, msg)
+            self.reply(self.data, msg)
             return
 
         try:
-            self.exec_shell("git branch -d %s" % delete_branch)
+            ref = getattr(self.repo.branches, target)
+        except AttributeError:
+            msg = "branch \x0302{0}\x0301 doesn't exist!".format(target)
+            self.reply(self.data, msg)
+        else:
+            self.repo.git.branch("-d", ref)
             msg = "branch \x0302{0}\x0301 has been deleted locally."
-            self.connection.reply(self.data, msg.format(delete_branch))
-        except subprocess.CalledProcessError:
-            # Git couldn't switch branches; assume the branch doesn't exist:
-            msg = "branch \x0302{0}\x0301 doesn't exist!".format(delete_branch)
-            self.connection.reply(self.data, msg)
+            self.reply(self.data, msg.format(target))
+            log = "{0} deleted branch {1} of {2}"
+            logmsg = log.format(self.data.nick, target, self.repo.working_dir)
+            self.logger.info(logmsg)
 
     def do_pull(self):
         """Pull from our remote repository."""
-        branch = self.exec_shell("git name-rev --name-only HEAD")
+        branch = self.repo.active_branch.name
         msg = "pulling from remote (currently on \x0302{0}\x0301)..."
-        self.connection.reply(self.data, msg.format(branch))
+        self.reply(self.data, msg.format(branch))
 
-        result = self.exec_shell("git pull")
-
-        if "Already up-to-date." in result:
-            self.connection.reply(self.data, "done; no new changes.")
+        remote = self.get_remote()
+        if not remote:
+            return
+        result = remote.pull()
+        updated = [info for info in result if info.flags != info.HEAD_UPTODATE]
+
+        if updated:
+            branches = ", ".join([info.ref.remote_head for info in updated])
+            msg = "done; updates to \x0302{0}\x0301 (from {1})."
+            self.reply(self.data, msg.format(branches, remote.url))
+            log = "{0} pulled {1} of {2} (updates to {3})"
+            self.logger.info(log.format(self.data.nick, remote.name,
+                                        self.repo.working_dir, branches))
         else:
-            regex = "\s*((.*?)\sfile(.*?)tions?\(-\))"
-            changes = re.findall(regex, result)[0][0]
-            try:
-                cmnd_remt = "git config --get branch.{0}.remote".format(branch)
-                remote = self.exec_shell(cmnd_remt)
-                cmnd_url = "git config --get remote.{0}.url".format(remote)
-                url = self.exec_shell(cmnd_url)
-                msg = "done; {0} [from {1}].".format(changes, url)
-                self.connection.reply(self.data, msg)
-            except subprocess.CalledProcessError:
-                # Something in .git/config is not specified correctly, so we
-                # cannot get the remote's URL. However, pull was a success:
-                self.connection.reply(self.data, "done; %s." % changes)
+            self.reply(self.data, "done; no new changes.")
+            log = "{0} pulled {1} of {2} (no updates)"
+            self.logger.info(log.format(self.data.nick, remote.name,
+                                        self.repo.working_dir))
 
     def do_status(self):
-        """Check whether we have anything to pull."""
-        last = self.exec_shell('git log -n 1 --pretty="%ar"')
-        result = self.exec_shell("git fetch --dry-run")
-        if not result:  # Nothing was fetched, so remote and local are equal
-            msg = "last commit was {0}. Local copy is \x02up-to-date\x0F with remote."
-            self.connection.reply(self.data, msg.format(last))
+        """Check if we have anything to pull."""
+        remote = self.get_remote()
+        if not remote:
+            return
+        since = self.get_time_since(self.repo.head.object.committed_date)
+        result = remote.fetch(dry_run=True)
+        updated = [info for info in result if info.flags != info.HEAD_UPTODATE]
+
+        if updated:
+            branches = ", ".join([info.ref.remote_head for info in updated])
+            msg = "last local commit was \x02{0}\x0F ago; updates to \x0302{1}\x0301."
+            self.reply(self.data, msg.format(since, branches))
+            log = "{0} got status of {1} of {2} (updates to {3})"
+            self.logger.info(log.format(self.data.nick, remote.name,
+                                        self.repo.working_dir, branches))
         else:
-            msg = "last local commit was {0}. Remote is \x02ahead\x0F of local copy."
-            self.connection.reply(self.data, msg.format(last))
+            msg = "last commit was \x02{0}\x0F ago. Local copy is up-to-date with remote."
+            self.reply(self.data, msg.format(since))
+            log = "{0} pulled {1} of {2} (no updates)"
+            self.logger.info(log.format(self.data.nick, remote.name,
+                                        self.repo.working_dir))

earwigbot/commands/help.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,47 +22,50 @@
 
 import re
 
-from earwigbot.classes import BaseCommand, Data
-from earwigbot import commands
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Help(Command):
     """Displays help information."""
     name = "help"
 
+    def check(self, data):
+        if data.is_command:
+            if data.command == "help":
+                return True
+            if not data.command and data.trigger == data.my_nick:
+                return True
+        return False
+
     def process(self, data):
-        self.cmnds = commands.get_all()
-        if not data.args:
-            self.do_main_help(data)
-        else:
+        if not data.command:
+            self.do_hello(data)
+        elif data.args:
             self.do_command_help(data)
+        else:
+            self.do_main_help(data)
 
     def do_main_help(self, data):
         """Give the user a general help message with a list of all commands."""
         msg = "Hi, I'm a bot! I have {0} commands loaded: {1}. You can get help for any command with '!help <command>'."
-        cmnds = sorted(self.cmnds.keys())
+        cmnds = sorted([cmnd.name for cmnd in self.bot.commands])
         msg = msg.format(len(cmnds), ', '.join(cmnds))
-        self.connection.reply(data, msg)
+        self.reply(data, msg)
 
     def do_command_help(self, data):
         """Give the user help for a specific command."""
-        command = data.args[0]
+        target = data.args[0]
 
-        # Create a dummy message to test which commands pick up the user's
-        # input:
-        dummy = Data(":foo!bar@example.com PRIVMSG #channel :msg!".split())
-        dummy.command = command.lower()
-        dummy.is_command = True
+        for command in self.bot.commands:
+            if command.name == target or target in command.commands:
+                if command.__doc__:
+                    doc = command.__doc__.replace("\n", "")
+                    doc = re.sub("\s\s+", " ", doc)
+                    msg = 'help for command \x0303{0}\x0301: "{1}"'
+                    self.reply(data, msg.format(target, doc))
+                    return
 
-        for cmnd in self.cmnds.values():
-            if not cmnd.check(dummy):
-                continue
-            if cmnd.__doc__:
-                doc = cmnd.__doc__.replace("\n", "")
-                doc = re.sub("\s\s+", " ", doc)
-                msg = "info for command \x0303{0}\x0301: \"{1}\""
-                self.connection.reply(data, msg.format(command, doc))
-                return
-            break
+        msg = "sorry, no help for \x0303{0}\x0301.".format(target)
+        self.reply(data, msg)
 
-        msg = "sorry, no help for \x0303{0}\x0301.".format(command)
-        self.connection.reply(data, msg)
+    def do_hello(self, data):
+        self.say(data.chan, "Yes, {0}?".format(data.nick))

earwigbot/commands/langcode.py

@@ -0,0 +1,53 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from earwigbot.commands import Command
+
+class Langcode(Command):
+    """Convert a language code into its name and a list of WMF sites in that
+    language."""
+    name = "langcode"
+    commands = ["langcode", "lang", "language"]
+
+    def process(self, data):
+        if not data.args:
+            self.reply(data, "please specify a language code.")
+            return
+
+        code = data.args[0]
+        site = self.bot.wiki.get_site()
+        matrix = site.api_query(action="sitematrix")["sitematrix"]
+        del matrix["count"]
+        del matrix["specials"]
+
+        for site in matrix.itervalues():
+            if site["code"] == code:
+                name = site["name"].encode("utf8")
+                localname = site["localname"].encode("utf8")
+                if name != localname:
+                    name += " ({0})".format(localname)
+                sites = ", ".join([s["url"] for s in site["site"]])
+                msg = "\x0302{0}\x0301 is {1} ({2})".format(code, name, sites)
+                self.reply(data, msg)
+                return
+
+        self.reply(data, "language \x0302{0}\x0301 not found.".format(code))

earwigbot/commands/link.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,35 +23,27 @@
 import re
 from urllib import quote
 
-from earwigbot.classes import BaseCommand
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Link(Command):
     """Convert a Wikipedia page name into a URL."""
     name = "link"
 
-    def check(self, data):
-        # if ((data.is_command and data.command == "link") or
-        # (("[[" in data.msg and "]]" in data.msg) or
-        # ("{{" in data.msg and "}}" in data.msg))):
-        if data.is_command and data.command == "link":
-            return True
-        return False
-
     def process(self, data):
         msg = data.msg
 
         if re.search("(\[\[(.*?)\]\])|(\{\{(.*?)\}\})", msg):
             links = self.parse_line(msg)
             links = " , ".join(links)
-            self.connection.reply(data, links)
+            self.reply(data, links)
 
         elif data.command == "link":
             if not data.args:
-                self.connection.reply(data, "what do you want me to link to?")
+                self.reply(data, "what do you want me to link to?")
                 return
             pagename = ' '.join(data.args)
             link = self.parse_link(pagename)
-            self.connection.reply(data, link)
+            self.reply(data, link)
 
     def parse_line(self, line):
         results = []

earwigbot/commands/notes.py

@@ -0,0 +1,167 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from earwigbot.commands import Command
+
+class Notes(Command):
+    """A mini IRC-based wiki for storing notes, tips, and reminders."""
+    name = "notes"
+
+    def process(self, data):
+        pass
+
+
+class OldCommand(object):
+    def parse(self):
+        if command == "notes" or command == "note" or command == "about" or command == "data" or command == "database":
+            try:
+                action = line2[4]
+            except BaseException:
+                reply("What do you want me to do? Type \"!notes help\" for more information.", chan, nick)
+                return
+            import MySQLdb
+            db = MySQLdb.connect(db="u_earwig_ircbot", host="sql", read_default_file="/home/earwig/.my.cnf")
+            specify = ' '.join(line2[5:])
+            if action == "help" or action == "manual":
+                shortCommandList = "read, write, change, undo, delete, move, author, category, list, report, developer"
+                if specify == "read":
+                    say("To read an entry, type \"!notes read <entry>\".", chan)
+                elif specify == "write":
+                    say("To write a new entry, type \"!notes write <entry> <content>\". This will create a new entry only if one does not exist, see the below command...", chan)
+                elif specify == "change":
+                    say("To change an entry, type \"!notes change <entry> <new content>\". The old entry will be stored in the database, so it can be undone later.", chan)
+                elif specify == "undo":
+                    say("To undo a change, type \"!notes undo <entry>\".", chan)
+                elif specify == "delete":
+                    say("To delete an entry, type \"!notes delete <entry>\". For security reasons, only bot admins can do this.", chan)
+                elif specify == "move":
+                    say("To move an entry, type \"!notes move <old_title> <new_title>\".", chan)
+                elif specify == "author":
+                    say("To return the author of an entry, type \"!notes author <entry>\".", chan)
+                elif specify == "category" or specify == "cat":
+                    say("To change an entry's category, type \"!notes category <entry> <category>\".", chan)
+                elif specify == "list":
+                    say("To list all categories in the database, type \"!notes list\". Type \"!notes list <category>\" to get all entries in a certain category.", chan)
+                elif specify == "report":
+                    say("To give some statistics about the mini-wiki, including some debugging information, type \"!notes report\" in a PM.", chan)
+                elif specify == "developer":
+                    say("To do developer work, such as writing to the database directly, type \"!notes developer <command>\". This can only be done by the bot owner.", chan)
+                else:
+                    db.query("SELECT * FROM version;")
+                    r = db.use_result()
+                    data = r.fetch_row(0)
+                    version = data[0]
+                    reply("The Earwig Mini-Wiki: running v%s." % version, chan, nick)
+                    reply("The full list of commands, for reference, are: %s." % shortCommandList, chan, nick)
+                    reply("For an explaination of a certain command, type \"!notes help <command>\".", chan, nick)
+                    reply("You can also access the database from the Toolserver: http://toolserver.org/~earwig/cgi-bin/irc_database.py", chan, nick)
+                    time.sleep(0.4)
+                return
+            elif action == "read":
+                specify = string.lower(specify)
+                if " " in specify: specify = string.split(specify, " ")[0]
+                if not specify or "\"" in specify:
+                    reply("Please include the name of the entry you would like to read after the command, e.g. !notes read earwig", chan, nick)
+                    return
+                try:
+                    db.query("SELECT entry_content FROM entries WHERE entry_title = \"%s\";" % specify)
+                    r = db.use_result()
+                    data = r.fetch_row(0)
+                    entry = data[0][0]
+                    say("Entry \"\x02%s\x0F\": %s" % (specify, entry), chan)
+                except Exception:
+                    reply("There is no entry titled \"\x02%s\x0F\"." % specify, chan, nick)
+                return
+            elif action == "delete" or action == "remove":
+                specify = string.lower(specify)
+                if " " in specify: specify = string.split(specify, " ")[0]
+                if not specify or "\"" in specify:
+                    reply("Please include the name of the entry you would like to delete after the command, e.g. !notes delete earwig", chan, nick)
+                    return
+                if authy == "owner" or authy == "admin":
+                    try:
+                        db.query("DELETE from entries where entry_title = \"%s\";" % specify)
+                        r = db.use_result()
+                        db.commit()
+                        reply("The entry on \"\x02%s\x0F\" has been removed." % specify, chan, nick)
+                    except Exception:
+                        phenny.reply("Unable to remove the entry on \"\x02%s\x0F\", because it doesn't exist." % specify, chan, nick)
+                else:
+                    reply("Only bot admins can remove entries.", chan, nick)
+                return
+            elif action == "developer":
+                if authy == "owner":
+                    db.query(specify)
+                    r = db.use_result()
+                    try:
+                        print r.fetch_row(0)
+                    except Exception:
+                        pass
+                    db.commit()
+                    reply("Done.", chan, nick)
+                else:
+                    reply("Only the bot owner can modify the raw database.", chan, nick)
+                return
+            elif action == "write":
+                try:
+                    write = line2[5]
+                    content = ' '.join(line2[6:])
+                except Exception:
+                    reply("Please include some content in your entry.", chan, nick)
+                    return
+                db.query("SELECT * from entries WHERE entry_title = \"%s\";" % write)
+                r = db.use_result()
+                data = r.fetch_row(0)
+                if data:
+                    reply("An entry on %s already exists; please use \"!notes change %s %s\"." % (write, write, content), chan, nick)
+                    return
+                content2 = content.replace('"', '\\' + '"')
+                db.query("INSERT INTO entries (entry_title, entry_author, entry_category, entry_content, entry_content_old) VALUES (\"%s\", \"%s\", \"uncategorized\", \"%s\", NULL);" % (write, nick, content2))
+                db.commit()
+                reply("You have written an entry titled \"\x02%s\x0F\", with the following content: \"%s\"" % (write, content), chan, nick)
+                return
+            elif action == "change":
+                reply("NotImplementedError", chan, nick)
+            elif action == "undo":
+                reply("NotImplementedError", chan, nick)
+            elif action == "move":
+                reply("NotImplementedError", chan, nick)
+            elif action == "author":
+                try:
+                    entry = line2[5]
+                except Exception:
+                    reply("Please include the name of the entry you would like to get information for after the command, e.g. !notes author earwig", chan, nick)
+                    return
+                db.query("SELECT entry_author from entries WHERE entry_title = \"%s\";" % entry)
+                r = db.use_result()
+                data = r.fetch_row(0)
+                if data:
+                    say("The author of \"\x02%s\x0F\" is \x02%s\x0F." % (entry, data[0][0]), chan)
+                    return
+                reply("There is no entry titled \"\x02%s\x0F\"." % entry, chan, nick)
+                return
+            elif action == "cat" or action == "category":
+                reply("NotImplementedError", chan, nick)
+            elif action == "list":
+                reply("NotImplementedError", chan, nick)
+            elif action == "report":
+                reply("NotImplementedError", chan, nick)

earwigbot/commands/praise.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,32 +20,29 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-import random
-
-from earwigbot.classes import BaseCommand
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Praise(Command):
     """Praise people!"""
     name = "praise"
 
+    def setup(self):
+        try:
+            self.praises = self.config.commands[self.name]["praises"]
+        except KeyError:
+            self.praises = []
+
     def check(self, data):
-        commands = ["praise", "earwig", "leonard", "leonard^bloom", "groove",
-                  "groovedog"]
-        return data.is_command and data.command in commands
+        check = data.command == "praise" or data.command in self.praises
+        return data.is_command and check
 
     def process(self, data):
-        if data.command == "earwig":
-            msg = "\x02Earwig\x0F is the bestest Python programmer ever!"
-        elif data.command in ["leonard", "leonard^bloom"]:
-            msg = "\x02Leonard^Bloom\x0F is the biggest slacker ever!"
-        elif data.command in ["groove", "groovedog"]:
-            msg = "\x02GrooveDog\x0F is the bestest heh evar!"
-        else:
-            if not data.args:
-                msg = "You use this command to praise certain people. Who they are is a secret."
-            else:
-                msg = "You're doing it wrong."
-            self.connection.reply(data, msg)
+        if data.command in self.praises:
+            msg = self.praises[data.command]
+            self.say(data.chan, msg)
             return
-
-        self.connection.say(data.chan, msg)
+        if not data.args:
+            msg = "You use this command to praise certain people. Who they are is a secret."
+        else:
+            msg = "you're doing it wrong."
+        self.reply(data, msg)

earwigbot/commands/quit.py

@@ -0,0 +1,68 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from earwigbot.commands import Command
+
+class Quit(Command):
+    """Quit, restart, or reload components from the bot. Only the owners can
+    run this command."""
+    name = "quit"
+    commands = ["quit", "restart", "reload"]
+
+    def process(self, data):
+        if data.host not in self.config.irc["permissions"]["owners"]:
+            self.reply(data, "you must be a bot owner to use this command.")
+            return
+        if data.command == "quit":
+            self.do_quit(data)
+        elif data.command == "restart":
+            self.do_restart(data)
+        else:
+            self.do_reload(data)
+
+    def do_quit(self, data):
+        args = data.args
+        if data.trigger == data.my_nick:
+            reason = " ".join(args)
+        else:
+            if not args or args[0].lower() != data.my_nick:
+                self.reply(data, "to confirm this action, the first argument must be my name.")
+                return
+            reason = " ".join(args[1:])
+
+        if reason:
+            self.bot.stop("Stopped by {0}: {1}".format(data.nick, reason))
+        else:
+            self.bot.stop("Stopped by {0}".format(data.nick))
+
+    def do_restart(self, data):
+        if data.args:
+            msg = " ".join(data.args)
+            self.bot.restart("Restarted by {0}: {1}".format(data.nick, msg))
+        else:
+            self.bot.restart("Restarted by {0}".format(data.nick))
+
+    def do_reload(self, data):
+        self.logger.info("{0} requested command/task reload".format(data.nick))
+        self.bot.commands.load()
+        self.bot.tasks.load()
+        self.reply(data, "IRC commands and bot tasks reloaded.")

earwigbot/commands/registration.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,18 +22,13 @@
 
 import time
 
-from earwigbot.classes import BaseCommand
-from earwigbot import wiki
+from earwigbot import exceptions
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Registration(Command):
     """Return when a user registered."""
     name = "registration"
-
-    def check(self, data):
-        commands = ["registration", "age"]
-        if data.is_command and data.command in commands:
-            return True
-        return False
+    commands = ["registration", "reg", "age"]
 
     def process(self, data):
         if not data.args:
@@ -41,30 +36,28 @@ class Command(BaseCommand):
         else:
             name = ' '.join(data.args)
 
-        site = wiki.get_site()
-        site._maxlag = None
+        site = self.bot.wiki.get_site()
         user = site.get_user(name)
 
         try:
-            reg = user.registration()
-        except wiki.UserNotFoundError:
+            reg = user.registration
+        except exceptions.UserNotFoundError:
             msg = "the user \x0302{0}\x0301 does not exist."
-            self.connection.reply(data, msg.format(name))
+            self.reply(data, msg.format(name))
             return
 
         date = time.strftime("%b %d, %Y at %H:%M:%S UTC", reg)
         age = self.get_diff(time.mktime(reg), time.mktime(time.gmtime()))
 
-        g = user.gender()
-        if g == "male":
+        if user.gender == "male":
             gender = "He's"
-        elif g == "female":
+        elif user.gender == "female":
             gender = "She's"
         else:
-            gender = "They're"
-        
+            gender = "They're"  # Singluar they?
+
         msg = "\x0302{0}\x0301 registered on {1}. {2} {3} old."
-        self.connection.reply(data, msg.format(name, date, gender, age))
+        self.reply(data, msg.format(name, date, gender, age))
 
     def get_diff(self, t1, t2):
         parts = {"years": 31536000, "days": 86400, "hours": 3600,

earwigbot/commands/remind.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,36 +20,32 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-import threading
+from threading import Timer
 import time
 
-from earwigbot.classes import BaseCommand
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Remind(Command):
     """Set a message to be repeated to you in a certain amount of time."""
     name = "remind"
-
-    def check(self, data):
-        if data.is_command and data.command in ["remind", "reminder"]:
-            return True
-        return False
+    commands = ["remind", "reminder"]
 
     def process(self, data):
         if not data.args:
             msg = "please specify a time (in seconds) and a message in the following format: !remind <time> <msg>."
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
 
         try:
             wait = int(data.args[0])
         except ValueError:
             msg = "the time must be given as an integer, in seconds."
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
         message = ' '.join(data.args[1:])
         if not message:
             msg = "what message do you want me to give you when time is up?"
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
 
         end = time.localtime(time.time() + wait)
@@ -58,14 +54,9 @@ class Command(BaseCommand):
 
         msg = 'Set reminder for "{0}" in {1} seconds (ends {2}).'
         msg = msg.format(message, wait, end_time_with_timezone)
-        self.connection.reply(data, msg)
+        self.reply(data, msg)
 
-        t_reminder = threading.Thread(target=self.reminder,
-                                      args=(data, message, wait))
+        t_reminder = Timer(wait, self.reply, args=(data, message))
         t_reminder.name = "reminder " + end_time
         t_reminder.daemon = True
         t_reminder.start()
-
-    def reminder(self, data, message, wait):
-        time.sleep(wait)
-        self.connection.reply(data, message)

earwigbot/commands/replag.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -24,16 +24,22 @@ from os.path import expanduser
 
 import oursql
 
-from earwigbot.classes import BaseCommand
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Replag(Command):
     """Return the replag for a specific database on the Toolserver."""
     name = "replag"
 
+    def setup(self):
+        try:
+            self.default = self.config.commands[self.name]["default"]
+        except KeyError:
+            self.default = None
+
     def process(self, data):
         args = {}
         if not data.args:
-            args["db"] = "enwiki_p"
+            args["db"] = self.default or self.bot.wiki.get_site().name + "_p"
         else:
             args["db"] = data.args[0]
         args["host"] = args["db"].replace("_", "-") + ".rrdb.toolserver.org"
@@ -41,10 +47,11 @@ class Command(BaseCommand):
 
         conn = oursql.connect(**args)
         with conn.cursor() as cursor:
-            query = "SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(rc_timestamp) FROM recentchanges ORDER BY rc_timestamp DESC LIMIT 1"
+            query = """SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(rc_timestamp)
+                       FROM recentchanges ORDER BY rc_timestamp DESC LIMIT 1"""
             cursor.execute(query)
             replag = int(cursor.fetchall()[0][0])
         conn.close()
 
-        msg = "Replag on \x0302{0}\x0301 is \x02{1}\x0F seconds."
-        self.connection.reply(data, msg.format(args["db"], replag))
+        msg = "replag on \x0302{0}\x0301 is \x02{1}\x0F seconds."
+        self.reply(data, msg.format(args["db"], replag))

earwigbot/commands/rights.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,18 +20,13 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseCommand
-from earwigbot import wiki
+from earwigbot import exceptions
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Rights(Command):
     """Retrieve a list of rights for a given username."""
     name = "rights"
-
-    def check(self, data):
-        commands = ["rights", "groups", "permissions", "privileges"]
-        if data.is_command and data.command in commands:
-            return True
-        return False
+    commands = ["rights", "groups", "permissions", "privileges"]
 
     def process(self, data):
         if not data.args:
@@ -39,15 +34,14 @@ class Command(BaseCommand):
         else:
             name = ' '.join(data.args)
 
-        site = wiki.get_site()
-        site._maxlag = None
+        site = self.bot.wiki.get_site()
         user = site.get_user(name)
 
         try:
-            rights = user.groups()
-        except wiki.UserNotFoundError:
+            rights = user.groups
+        except exceptions.UserNotFoundError:
             msg = "the user \x0302{0}\x0301 does not exist."
-            self.connection.reply(data, msg.format(name))
+            self.reply(data, msg.format(name))
             return
 
         try:
@@ -55,4 +49,4 @@ class Command(BaseCommand):
         except ValueError:
             pass
         msg = "the rights for \x0302{0}\x0301 are {1}."
-        self.connection.reply(data, msg.format(name, ', '.join(rights)))
+        self.reply(data, msg.format(name, ', '.join(rights)))

earwigbot/commands/test.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,15 +22,16 @@
 
 import random
 
-from earwigbot.classes import BaseCommand
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Test(Command):
     """Test the bot!"""
     name = "test"
 
     def process(self, data):
+        user = "\x02" + data.nick + "\x0F"  # Wrap nick in bold
         hey = random.randint(0, 1)
         if hey:
-            self.connection.say(data.chan, "Hey \x02%s\x0F!" % data.nick)
+            self.say(data.chan, "Hey {0}!".format(user))
         else:
-            self.connection.say(data.chan, "'sup \x02%s\x0F?" % data.nick)
+            self.say(data.chan, "'sup {0}?".format(user))

earwigbot/commands/threads.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,25 +23,18 @@
 import threading
 import re
 
-from earwigbot import tasks
-from earwigbot.classes import BaseCommand, Data, KwargParseException
-from earwigbot.config import config
+from earwigbot.commands import Command
 
-class Command(BaseCommand):
+class Threads(Command):
     """Manage wiki tasks from IRC, and check on thread status."""
     name = "threads"
-
-    def check(self, data):
-        commands = ["tasks", "task", "threads", "tasklist"]
-        if data.is_command and data.command in commands:
-            return True
-        return False
+    commands = ["tasks", "task", "threads", "tasklist"]
 
     def process(self, data):
         self.data = data
-        if data.host not in config.irc["permissions"]["owners"]:
+        if data.host not in self.config.irc["permissions"]["owners"]:
             msg = "you must be a bot owner to use this command."
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
             return
 
         if not data.args:
@@ -49,7 +42,7 @@ class Command(BaseCommand):
                 self.do_list()
             else:
                 msg = "no arguments provided. Maybe you wanted '!{0} list', '!{0} start', or '!{0} listall'?"
-                self.connection.reply(data, msg.format(data.command))
+                self.reply(data, msg.format(data.command))
             return
 
         if data.args[0] == "list":
@@ -63,7 +56,7 @@ class Command(BaseCommand):
 
         else:  # They asked us to do something we don't know
             msg = "unknown argument: \x0303{0}\x0301.".format(data.args[0])
-            self.connection.reply(data, msg)
+            self.reply(data, msg)
 
     def do_list(self):
         """With !tasks list (or abbreviation !tasklist), list all running
@@ -72,15 +65,14 @@ class Command(BaseCommand):
         threads = threading.enumerate()
 
         normal_threads = []
-        task_threads = []
+        daemon_threads = []
 
         for thread in threads:
             tname = thread.name
             if tname == "MainThread":
-                tname = self.get_main_thread_name()
-                t = "\x0302{0}\x0301 (as main thread, id {1})"
-                normal_threads.append(t.format(tname, thread.ident))
-            elif tname in ["irc-frontend", "irc-watcher", "wiki-scheduler"]:
+                t = "\x0302MainThread\x0301 (id {0})"
+                normal_threads.append(t.format(thread.ident))
+            elif tname in self.config.components:
                 t = "\x0302{0}\x0301 (id {1})"
                 normal_threads.append(t.format(tname, thread.ident))
             elif tname.startswith("reminder"):
@@ -90,28 +82,28 @@ class Command(BaseCommand):
             else:
                 tname, start_time = re.findall("^(.*?) \((.*?)\)$", tname)[0]
                 t = "\x0302{0}\x0301 (id {1}, since {2})"
-                task_threads.append(t.format(tname, thread.ident, start_time))
+                daemon_threads.append(t.format(tname, thread.ident,
+                                               start_time))
 
-        if task_threads:
-            msg = "\x02{0}\x0F threads active: {1}, and \x02{2}\x0F task threads: {3}."
+        if daemon_threads:
+            if len(daemon_threads) > 1:
+                msg = "\x02{0}\x0F threads active: {1}, and \x02{2}\x0F command/task threads: {3}."
+            else:
+                msg = "\x02{0}\x0F threads active: {1}, and \x02{2}\x0F command/task thread: {3}."
             msg = msg.format(len(threads), ', '.join(normal_threads),
-                             len(task_threads), ', '.join(task_threads))
+                             len(daemon_threads), ', '.join(daemon_threads))
         else:
-            msg = "\x02{0}\x0F threads active: {1}, and \x020\x0F task threads."
+            msg = "\x02{0}\x0F threads active: {1}, and \x020\x0F command/task threads."
             msg = msg.format(len(threads), ', '.join(normal_threads))
 
-        self.connection.reply(self.data, msg)
+        self.reply(self.data, msg)
 
     def do_listall(self):
         """With !tasks listall or !tasks all, list all loaded tasks, and report
         whether they are currently running or idle."""
-        all_tasks = tasks.get_all().keys()
         threads = threading.enumerate()
         tasklist = []
-
-        all_tasks.sort()
-
-        for task in all_tasks:
+        for task in sorted([task.name for task in self.bot.tasks]):
             threadlist = [t for t in threads if t.name.startswith(task)]
             ids = [str(t.ident) for t in threadlist]
             if not ids:
@@ -123,10 +115,10 @@ class Command(BaseCommand):
                 t = "\x0302{0}\x0301 (\x02active\x0F as ids {1})"
                 tasklist.append(t.format(task, ', '.join(ids)))
 
-        tasklist = ", ".join(tasklist)
+        tasks = ", ".join(tasklist)
 
-        msg = "{0} tasks loaded: {1}.".format(len(all_tasks), tasklist)
-        self.connection.reply(self.data, msg)
+        msg = "\x02{0}\x0F tasks loaded: {1}.".format(len(tasklist), tasks)
+        self.reply(self.data, msg)
 
     def do_start(self):
         """With !tasks start, start any loaded task by name with or without
@@ -136,32 +128,16 @@ class Command(BaseCommand):
         try:
             task_name = data.args[1]
         except IndexError:  # No task name given
-            self.connection.reply(data, "what task do you want me to start?")
-            return
-
-        try:
-            data.parse_kwargs()
-        except KwargParseException, arg:
-            msg = "error parsing argument: \x0303{0}\x0301.".format(arg)
-            self.connection.reply(data, msg)
+            self.reply(data, "what task do you want me to start?")
             return
 
-        if task_name not in tasks.get_all().keys():
+        if task_name not in [task.name for task in self.bot.tasks]:
             # This task does not exist or hasn't been loaded:
-            msg = "task could not be found; either bot/tasks/{0}.py doesn't exist, or it wasn't loaded correctly."
-            self.connection.reply(data, msg.format(task_name))
+            msg = "task could not be found; either it doesn't exist, or it wasn't loaded correctly."
+            self.reply(data, msg.format(task_name))
             return
 
         data.kwargs["fromIRC"] = True
-        tasks.start(task_name, **data.kwargs)
+        self.bot.tasks.start(task_name, **data.kwargs)
         msg = "task \x0302{0}\x0301 started.".format(task_name)
-        self.connection.reply(data, msg)
-
-    def get_main_thread_name(self):
-        """Return the "proper" name of the MainThread."""
-        if "irc_frontend" in config.components:
-            return "irc-frontend"
-        elif "wiki_schedule" in config.components:
-            return "wiki-scheduler"
-        else:
-            return "irc-watcher"
+        self.reply(data, msg)

earwigbot/commands/time.py

@@ -0,0 +1,68 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from datetime import datetime
+from math import floor
+from time import time
+
+try:
+    import pytz
+except ImportError:
+    pytz = None
+
+from earwigbot.commands import Command
+
+class Time(Command):
+    """Report the current time in any timezone (UTC default), or in beats."""
+    name = "time"
+    commands = ["time", "beats", "swatch"]
+
+    def process(self, data):
+        if data.command in ["beats", "swatch"]:
+            self.do_beats(data)
+            return
+        if data.args:
+            timezone = data.args[0]
+        else:
+            timezone = "UTC"
+        if timezone in ["beats", "swatch"]:
+            self.do_beats(data)
+        else:
+            self.do_time(data, timezone)
+
+    def do_beats(self, data):
+        beats = ((time() + 3600) % 86400) / 86.4
+        beats = int(floor(beats))
+        self.reply(data, "@{0:0>3}".format(beats))
+
+    def do_time(self, data, timezone):
+        if not pytz:
+            msg = "this command requires the 'pytz' module: http://pytz.sourceforge.net/"
+            self.reply(data, msg)
+            return
+        try:
+            tzinfo = pytz.timezone(timezone)
+        except pytz.exceptions.UnknownTimeZoneError:
+            self.reply(data, "unknown timezone: {0}.".format(timezone))
+            return
+        now = pytz.utc.localize(datetime.utcnow()).astimezone(tzinfo)
+        self.reply(data, now.strftime("%Y-%m-%d %H:%M:%S %Z"))

earwigbot/commands/trout.py

@@ -0,0 +1,46 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from unicodedata import normalize
+
+from earwigbot.commands import Command
+
+class Trout(Command):
+    """Slap someone with a trout, or related fish."""
+    name = "trout"
+    commands = ["trout", "whale"]
+
+    def setup(self):
+        try:
+            self.exceptions = self.config.commands[self.name]["exceptions"]
+        except KeyError:
+            self.exceptions = {}
+
+    def process(self, data):
+        animal = data.command
+        target = " ".join(data.args) or data.nick
+        normal = normalize("NFKD", target.decode("utf8")).lower()
+        if normal in self.exceptions:
+            self.reply(data, self.exceptions[normal])
+        else:
+            msg = "slaps \x02{0}\x0F around a bit with a large {1}."
+            self.action(data.chan, msg.format(target, animal))

earwigbot/config.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,105 +20,97 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-"""
-EarwigBot's JSON Config File Parser
-
-This handles all tasks involving reading and writing to our config file,
-including encrypting and decrypting passwords and making a new config file from
-scratch at the inital bot run.
-
-Usually you'll just want to do "from earwigbot.config import config", which
-returns a singleton _BotConfig object, with data accessible from various
-attributes and functions:
-
-* config.components  - enabled components
-* config.wiki        - information about wiki-editing
-* config.tasks       - information for bot tasks
-* config.irc         - information about IRC
-* config.metadata    - miscellaneous information
-* config.schedule()  - tasks scheduled to run at a given time
-
-Additionally, _BotConfig has some functions used in config loading:
-* config.load()     - loads and parses our config file, returning True if
-                      passwords are stored encrypted or False otherwise
-* config.decrypt()  - given a key, decrypts passwords inside our config
-                      variables; won't work if passwords aren't encrypted
-"""
-
-import json
+from getpass import getpass
+from hashlib import sha256
 import logging
 import logging.handlers
 from os import mkdir, path
 
-from earwigbot import blowfish
+from Crypto.Cipher import Blowfish
+import bcrypt
+import yaml
 
-__all__ = ["config"]
+from earwigbot.exceptions import NoConfigError
 
-class _ConfigNode(object):
-    def __iter__(self):
-        for key in self.__dict__.iterkeys():
-            yield key
+__all__ = ["BotConfig"]
 
-    def __getitem__(self, item):
-        return self.__dict__.__getitem__(item)
+class BotConfig(object):
+    """
+    **EarwigBot: YAML Config File Manager**
 
-    def _dump(self):
-        data = self.__dict__.copy()
-        for key, val in data.iteritems():
-            if isinstance(val, _ConfigNode):
-                data[key] = val.dump()
-        return data
+    This handles all tasks involving reading and writing to our config file,
+    including encrypting and decrypting passwords and making a new config file
+    from scratch at the inital bot run.
 
-    def _load(self, data):
-        self.__dict__ = data.copy()
+    BotConfig has a few attributes and methods, including the following:
 
-    def _decrypt(self, key, intermediates, item):
-        base = self.__dict__
-        try:
-            for inter in intermediates:
-                base = base[inter]
-        except KeyError:
-            return
-        if item in base:
-            base[item] = blowfish.decrypt(key, base[item])
+    - :py:attr:`root_dir`:   bot's working directory; contains
+      :file:`config.yml`, :file:`logs/`
+    - :py:attr:`path`:       path to the bot's config file
+    - :py:attr:`components`: enabled components
+    - :py:attr:`wiki`:       information about wiki-editing
+    - :py:attr:`irc`:        information about IRC
+    - :py:attr:`commands`:   information about IRC commands
+    - :py:attr:`tasks`:      information for bot tasks
+    - :py:attr:`metadata`:   miscellaneous information
+    - :py:meth:`schedule`:   tasks scheduled to run at a given time
 
-    def get(self, *args, **kwargs):
-        return self.__dict__.get(*args, **kwargs)
+    BotConfig also has some methods used in config loading:
 
+    - :py:meth:`load`:    loads (or reloads) and parses our config file
+    - :py:meth:`decrypt`: decrypts an object in the config tree
+    """
 
-class _BotConfig(object):
-    def __init__(self):
-        self._script_dir = path.dirname(path.abspath(__file__))
-        self._root_dir = path.split(self._script_dir)[0]
-        self._config_path = path.join(self._root_dir, "config.json")
+    def __init__(self, root_dir, level):
+        self._root_dir = root_dir
+        self._logging_level = level
+        self._config_path = path.join(self._root_dir, "config.yml")
         self._log_dir = path.join(self._root_dir, "logs")
-        self._decryption_key = None
+        self._decryption_cipher = None
         self._data = None
 
         self._components = _ConfigNode()
         self._wiki = _ConfigNode()
-        self._tasks = _ConfigNode()
         self._irc = _ConfigNode()
+        self._commands = _ConfigNode()
+        self._tasks = _ConfigNode()
         self._metadata = _ConfigNode()
 
-        self._nodes = [self._components, self._wiki, self._tasks, self._irc,
-                       self._metadata]
+        self._nodes = [self._components, self._wiki, self._irc, self._commands,
+                       self._tasks, self._metadata]
+
+        self._decryptable_nodes = [  # Default nodes to decrypt
+            (self._wiki, ("password",)),
+            (self._wiki, ("search", "credentials", "key")),
+            (self._wiki, ("search", "credentials", "secret")),
+            (self._irc, ("frontend", "nickservPassword")),
+            (self._irc, ("watcher", "nickservPassword")),
+        ]
+
+    def __repr__(self):
+        """Return the canonical string representation of the BotConfig."""
+        res = "BotConfig(root_dir={0!r}, level={1!r})"
+        return res.format(self.root_dir, self.logging_level)
+
+    def __str__(self):
+        """Return a nice string representation of the BotConfig."""
+        return "<BotConfig at {0}>".format(self.root_dir)
 
     def _load(self):
-        """Load data from our JSON config file (config.json) into _config."""
+        """Load data from our JSON config file (config.yml) into self._data."""
         filename = self._config_path
         with open(filename, 'r') as fp:
             try:
-                self._data = json.load(fp)
-            except ValueError as error:
+                self._data = yaml.load(fp)
+            except yaml.YAMLError:
                 print "Error parsing config file {0}:".format(filename)
-                print error
-                exit(1)
+                raise
 
     def _setup_logging(self):
         """Configures the logging module so it works the way we want it to."""
         log_dir = self._log_dir
         logger = logging.getLogger("earwigbot")
+        logger.handlers = []  # Remove any handlers already attached to us
         logger.setLevel(logging.DEBUG)
 
         if self.metadata.get("enableLogging"):
@@ -134,7 +126,7 @@ class _BotConfig(object):
                 else:
                     msg = "log_dir ({0}) exists but is not a directory!"
                     print msg.format(log_dir)
-                    exit(1)
+                    return
 
             main_handler = hand(logfile("bot.log"), "midnight", 1, 7)
             error_handler = hand(logfile("error.log"), "W6", 1, 4)
@@ -148,41 +140,64 @@ class _BotConfig(object):
                 h.setFormatter(formatter)
                 logger.addHandler(h)
 
-            stream_handler = logging.StreamHandler()
-            stream_handler.setLevel(logging.DEBUG)
-            stream_handler.setFormatter(color_formatter)
-            logger.addHandler(stream_handler)
+        self._stream_handler = stream = logging.StreamHandler()
+        stream.setLevel(self._logging_level)
+        stream.setFormatter(color_formatter)
+        logger.addHandler(stream)
 
-        else:
-            logger.addHandler(logging.NullHandler())
+    def _decrypt(self, node, nodes):
+        """Try to decrypt the contents of a config node. Use self.decrypt()."""
+        try:
+            node._decrypt(self._decryption_cipher, nodes[:-1], nodes[-1])
+        except ValueError:
+            print "Error decrypting passwords:"
+            raise
 
     def _make_new(self):
         """Make a new config file based on the user's input."""
-        encrypt = raw_input("Would you like to encrypt passwords stored in config.json? [y/n] ")
-        if encrypt.lower().startswith("y"):
-            is_encrypted = True
-        else:
-            is_encrypted = False
-
-        return is_encrypted
-
-    @property
-    def script_dir(self):
-        return self._script_dir
+        #m = "Would you like to encrypt passwords stored in config.yml? [y/n] "
+        #encrypt = raw_input(m)
+        #if encrypt.lower().startswith("y"):
+        #    is_encrypted = True
+        #else:
+        #    is_encrypted = False
+        raise NotImplementedError()
+        # yaml.dumps() config.yml file (self._config_path)
+        # Create root_dir/, root_dir/commands/, root_dir/tasks/
+        # Give a reasonable message after config has been created regarding
+        # what to do next...
 
     @property
     def root_dir(self):
+        """The bot's root directory containing its config file and more."""
         return self._root_dir
 
     @property
-    def config_path(self):
+    def logging_level(self):
+        """The minimum logging level for messages logged via stdout."""
+        return self._logging_level
+
+    @logging_level.setter
+    def logging_level(self, level):
+        self._logging_level = level
+        self._stream_handler.setLevel(level)
+
+    @property
+    def path(self):
+        """The path to the bot's config file."""
         return self._config_path
 
     @property
     def log_dir(self):
+        """The directory containing the bot's logs."""
         return self._log_dir
 
     @property
+    def data(self):
+        """The entire config file as a decoded JSON object."""
+        return self._data
+
+    @property
     def components(self):
         """A dict of enabled components."""
         return self._components
@@ -193,90 +208,103 @@ class _BotConfig(object):
         return self._wiki
 
     @property
-    def tasks(self):
-        """A dict of information for bot tasks."""
-        return self._tasks
-
-    @property
     def irc(self):
         """A dict of information about IRC."""
         return self._irc
 
     @property
+    def commands(self):
+        """A dict of information for IRC commands."""
+        return self._commands
+
+    @property
+    def tasks(self):
+        """A dict of information for bot tasks."""
+        return self._tasks
+
+    @property
     def metadata(self):
         """A dict of miscellaneous information."""
         return self._metadata
 
     def is_loaded(self):
-        """Return True if our config file has been loaded, otherwise False."""
+        """Return ``True`` if our config file has been loaded, or ``False``."""
         return self._data is not None
 
     def is_encrypted(self):
-        """Return True if passwords are encrypted, otherwise False."""
+        """Return ``True`` if passwords are encrypted, otherwise ``False``."""
         return self.metadata.get("encryptPasswords", False)
 
-    def load(self, config_path=None, log_dir=None):
+    def load(self):
         """Load, or reload, our config file.
 
         First, check if we have a valid config file, and if not, notify the
         user. If there is no config file at all, offer to make one, otherwise
         exit.
 
-        Store data from our config file in five _ConfigNodes (components,
-        wiki, tasks, irc, metadata) for easy access (as well as the internal
-        _data variable).
-
-        If everything goes well, return True if stored passwords are
-        encrypted in the file, or False if they are not.
+        Data from the config file is stored in six
+        :py:class:`~earwigbot.config._ConfigNode`\ s (:py:attr:`components`,
+        :py:attr:`wiki`, :py:attr:`irc`, :py:attr:`commands`, :py:attr:`tasks`,
+        :py:attr:`metadata`) for easy access (as well as the lower-level
+        :py:attr:`data` attribute). If passwords are encrypted, we'll use
+        :py:func:`~getpass.getpass` for the key and then decrypt them. If the
+        config is being reloaded, encrypted items will be automatically
+        decrypted if they were decrypted earlier.
         """
-        if config_path:
-            self._config_path = config_path
-        if log_dir:
-            self._log_dir = log_dir
-
         if not path.exists(self._config_path):
-            print "You haven't configured the bot yet!"
-            choice = raw_input("Would you like to do this now? [y/n] ")
+            print "Config file not found:", self._config_path
+            choice = raw_input("Would you like to create a config file now? [y/n] ")
             if choice.lower().startswith("y"):
-                return self._make_new()
+                self._make_new()
             else:
-                exit(1)
+                raise NoConfigError()
 
         self._load()
         data = self._data
         self.components._load(data.get("components", {}))
         self.wiki._load(data.get("wiki", {}))
-        self.tasks._load(data.get("tasks", {}))
         self.irc._load(data.get("irc", {}))
+        self.commands._load(data.get("commands", {}))
+        self.tasks._load(data.get("tasks", {}))
         self.metadata._load(data.get("metadata", {}))
 
         self._setup_logging()
-        return self.is_encrypted()
+        if self.is_encrypted():
+            if not self._decryption_cipher:
+                key = getpass("Enter key to decrypt bot passwords: ")
+                self._decryption_cipher = Blowfish.new(sha256(key).digest())
+            signature = self.metadata["signature"]
+            assert bcrypt.hashpw(key, signature) == signature
+            for node, nodes in self._decryptable_nodes:
+                self._decrypt(node, nodes)
 
     def decrypt(self, node, *nodes):
-        """Use self._decryption_key to decrypt an object in our config tree.
+        """Decrypt an object in our config tree.
+
+        :py:attr:`_decryption_cipher` is used as our key, retrieved using
+        :py:func:`~getpass.getpass` in :py:meth:`load` if it wasn't already
+        specified. If this is called when passwords are not encrypted (check
+        with :py:meth:`is_encrypted`), nothing will happen. We'll also keep
+        track of this node if :py:meth:`load` is called again (i.e. to reload)
+        and automatically decrypt it.
 
-        If this is called when passwords are not encrypted (check with
-        config.is_encrypted()), nothing will happen.
+        Example usage::
 
-        An example usage would be:
-        config.decrypt(config.irc, "frontend", "nickservPassword")
+            >>> config.decrypt(config.irc, "frontend", "nickservPassword")
+            # decrypts config.irc["frontend"]["nickservPassword"]
         """
-        if not self.is_encrypted():
-            return
-        try:
-            node._decrypt(self._decryption_key, nodes[:-1], nodes[-1])
-        except blowfish.BlowfishError as error:
-            print "\nError decrypting passwords:"
-            print "{0}: {1}.".format(error.__class__.__name__, error)
-            exit(1)
+        signature = (node, nodes)
+        if signature in self._decryptable_nodes:
+            return  # Already decrypted
+        self._decryptable_nodes.append(signature)
+        if self.is_encrypted():
+            self._decrypt(node, nodes)
 
     def schedule(self, minute, hour, month_day, month, week_day):
         """Return a list of tasks scheduled to run at the specified time.
 
-        The schedule data comes from our config file's 'schedule' field, which
-        is stored as self._data["schedule"]. Call this function as
-        config.schedule(args).
+        The schedule data comes from our config file's ``schedule`` field,
+        which is stored as :py:attr:`self.data["schedule"] <data>`.
         """
         # Tasks to run this turn, each as a list of either [task_name, kwargs],
         # or just the task_name:
@@ -305,6 +333,57 @@ class _BotConfig(object):
         return tasks
 
 
+class _ConfigNode(object):
+    def __iter__(self):
+        for key in self.__dict__:
+            yield key
+
+    def __getitem__(self, item):
+        return self.__dict__.__getitem__(item)
+
+    def _dump(self):
+        data = self.__dict__.copy()
+        for key, val in data.iteritems():
+            if isinstance(val, _ConfigNode):
+                data[key] = val._dump()
+        return data
+
+    def _load(self, data):
+        self.__dict__ = data.copy()
+
+    def _decrypt(self, cipher, intermediates, item):
+        base = self.__dict__
+        for inter in intermediates:
+            try:
+                base = base[inter]
+            except KeyError:
+                return
+        if item in base:
+            ciphertext = base[item].decode("hex")
+            base[item] = cipher.decrypt(ciphertext).rstrip("\x00")
+
+    def get(self, *args, **kwargs):
+        return self.__dict__.get(*args, **kwargs)
+
+    def keys(self):
+        return self.__dict__.keys()
+
+    def values(self):
+        return self.__dict__.values()
+
+    def items(self):
+        return self.__dict__.items()
+
+    def iterkeys(self):
+        return self.__dict__.iterkeys()
+
+    def itervalues(self):
+        return self.__dict__.itervalues()
+
+    def iteritems(self):
+        return self.__dict__.iteritems()
+
+
 class _BotFormatter(logging.Formatter):
     def __init__(self, color=False):
         self._format = super(_BotFormatter, self).format
@@ -330,6 +409,3 @@ class _BotFormatter(logging.Formatter):
         if record.levelno == logging.CRITICAL:
             record.lvl = l.join(("\x1b[1m\x1b[31m", "\x1b[0m"))  # Bold red
         return record
-
-
-config = _BotConfig()

earwigbot/exceptions.py

@@ -0,0 +1,256 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+"""
+**EarwigBot: Exceptions**
+
+This module contains all exceptions used by EarwigBot::
+
+    EarwigBotError
+     +-- NoConfigError
+     +-- IRCError
+     |    +-- BrokenSocketError
+     +-- WikiToolsetError
+          +-- SiteNotFoundError
+          +-- ServiceError
+          |    +-- APIError
+          |    +-- SQLError
+          +-- NoServiceError
+          +-- LoginError
+          +-- NamespaceNotFoundError
+          +-- PageNotFoundError
+          +-- InvalidPageError
+          +-- RedirectError
+          +-- UserNotFoundError
+          +-- EditError
+          |    +-- PermissionsError
+          |    +-- EditConflictError
+          |    +-- NoContentError
+          |    +-- ContentTooBigError
+          |    +-- SpamDetectedError
+          |    +-- FilteredError
+          +-- CopyvioCheckError
+               +-- UnknownSearchEngineError
+               +-- UnsupportedSearchEngineError
+               +-- SearchQueryError
+"""
+
+class EarwigBotError(Exception):
+    """Base exception class for errors in EarwigBot."""
+
+class NoConfigError(EarwigBotError):
+    """The bot cannot be run without a config file.
+
+    This occurs if no config file exists, and the user said they did not want
+    one to be created.
+    """
+
+class IRCError(EarwigBotError):
+    """Base exception class for errors in IRC-relation sections of the bot."""
+
+class BrokenSocketError(IRCError):
+    """A socket has broken, because it is not sending data.
+
+    Raised by :py:meth:`IRCConnection._get
+    <earwigbot.irc.connection.IRCConnection._get>`.
+    """
+
+class WikiToolsetError(EarwigBotError):
+    """Base exception class for errors in the Wiki Toolset."""
+
+class SiteNotFoundError(WikiToolsetError):
+    """A particular site could not be found in the sites database.
+
+    Raised by :py:class:`~earwigbot.wiki.sitesdb.SitesDB`.
+    """
+
+class ServiceError(WikiToolsetError):
+    """Base exception class for an error within a service (the API or SQL).
+
+    This is caught by :py:meth:`Site.delegate
+    <earwigbot.wiki.site.Site.delegate>` to indicate a service is
+    non-functional so another, less-preferred one can be tried.
+    """
+
+class APIError(ServiceError):
+    """Couldn't connect to a site's API.
+
+    Perhaps the server doesn't exist, our URL is wrong or incomplete, or
+    there are temporary problems on their end.
+
+    Raised by :py:meth:`Site.api_query <earwigbot.wiki.site.Site.api_query>`.
+    """
+
+class SQLError(ServiceError):
+    """Some error involving SQL querying occurred.
+
+    Raised by :py:meth:`Site.sql_query <earwigbot.wiki.site.Site.sql_query>`.
+    """
+
+class NoServiceError(WikiToolsetError):
+    """No service is functioning to handle a specific task.
+
+    Raised by :py:meth:`Site.delegate <earwigbot.wiki.site.Site.delegate>`.
+    """
+
+class LoginError(WikiToolsetError):
+    """An error occured while trying to login.
+
+    Perhaps the username/password is incorrect.
+
+    Raised by :py:meth:`Site._login <earwigbot.wiki.site.Site._login>`.
+    """
+
+class NamespaceNotFoundError(WikiToolsetError):
+    """A requested namespace name or namespace ID does not exist.
+
+    Raised by :py:meth:`Site.namespace_id_to_name
+    <earwigbot.wiki.site.Site.namespace_id_to_name>` and
+    :py:meth:`Site.namespace_name_to_id
+    <earwigbot.wiki.site.Site.namespace_name_to_id>`.
+    """
+
+class PageNotFoundError(WikiToolsetError):
+    """Attempted to get information about a page that does not exist.
+
+    Raised by :py:class:`~earwigbot.wiki.page.Page`.
+    """
+
+class InvalidPageError(WikiToolsetError):
+    """Attempted to get information about a page whose title is invalid.
+
+    Raised by :py:class:`~earwigbot.wiki.page.Page`.
+    """
+
+class RedirectError(WikiToolsetError):
+    """A redirect-only method was called on a malformed or non-redirect page.
+
+    Raised by :py:meth:`Page.get_redirect_target
+    <earwigbot.wiki.page.Page.get_redirect_target>`.
+    """
+
+class UserNotFoundError(WikiToolsetError):
+    """Attempted to get certain information about a user that does not exist.
+
+    Raised by :py:class:`~earwigbot.wiki.user.User`.
+    """
+
+class EditError(WikiToolsetError):
+    """An error occured while editing.
+
+    This is used as a base class for all editing errors; this one specifically
+    is used only when a generic error occurs that we don't know about.
+
+    Raised by :py:meth:`Page.edit <earwigbot.wiki.page.Page.edit>` and
+    :py:meth:`Page.add_section <earwigbot.wiki.page.Page.add_section>`.
+    """
+
+class PermissionsError(EditError):
+    """A permissions error ocurred while editing.
+
+    We tried to do something we don't have permission to, like trying to delete
+    a page as a non-admin, or trying to edit a page without login information
+    and AssertEdit enabled.
+
+    Raised by :py:meth:`Page.edit <earwigbot.wiki.page.Page.edit>` and
+    :py:meth:`Page.add_section <earwigbot.wiki.page.Page.add_section>`.
+    """
+
+class EditConflictError(EditError):
+    """We gotten an edit conflict or a (rarer) delete/recreate conflict.
+
+    Raised by :py:meth:`Page.edit <earwigbot.wiki.page.Page.edit>` and
+    :py:meth:`Page.add_section <earwigbot.wiki.page.Page.add_section>`.
+    """
+
+class NoContentError(EditError):
+    """We tried to create a page or new section with no content.
+
+    Raised by :py:meth:`Page.edit <earwigbot.wiki.page.Page.edit>` and
+    :py:meth:`Page.add_section <earwigbot.wiki.page.Page.add_section>`.
+    """
+
+class ContentTooBigError(EditError):
+    """The edit we tried to push exceeded the article size limit.
+
+    Raised by :py:meth:`Page.edit <earwigbot.wiki.page.Page.edit>` and
+    :py:meth:`Page.add_section <earwigbot.wiki.page.Page.add_section>`.
+    """
+
+class SpamDetectedError(EditError):
+    """The spam filter refused our edit.
+
+    Raised by :py:meth:`Page.edit <earwigbot.wiki.page.Page.edit>` and
+    :py:meth:`Page.add_section <earwigbot.wiki.page.Page.add_section>`.
+    """
+
+class FilteredError(EditError):
+    """The edit filter refused our edit.
+
+    Raised by :py:meth:`Page.edit <earwigbot.wiki.page.Page.edit>` and
+    :py:meth:`Page.add_section <earwigbot.wiki.page.Page.add_section>`.
+    """
+
+class CopyvioCheckError(WikiToolsetError):
+    """An error occured when checking a page for copyright violations.
+
+    This is a base class for multiple exceptions; usually one of those will be
+    raised instead of this.
+
+    Raised by :py:meth:`Page.copyvio_check
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_check>` and
+    :py:meth:`Page.copyvio_compare
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_compare>`.
+    """
+
+class UnknownSearchEngineError(CopyvioCheckError):
+    """Attempted to do a copyvio check with an unknown search engine.
+
+    Search engines are specified in :file:`config.yml` as
+    :py:attr:`config.wiki["search"]["engine"]`.
+
+    Raised by :py:meth:`Page.copyvio_check
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_check>` and
+    :py:meth:`Page.copyvio_compare
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_compare>`.
+    """
+
+class UnsupportedSearchEngineError(CopyvioCheckError):
+    """Attmpted to do a copyvio check using an unavailable engine.
+
+    This might occur if, for example, an engine requires oauth2 but the package
+    couldn't be imported.
+
+    Raised by :py:meth:`Page.copyvio_check
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_check>` and
+    :py:meth:`Page.copyvio_compare
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_compare>`.
+    """
+
+class SearchQueryError(CopyvioCheckError):
+    """Some error ocurred while doing a search query.
+
+    Raised by :py:meth:`Page.copyvio_check
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_check>` and
+    :py:meth:`Page.copyvio_compare
+    <earwigbot.wiki.copyvios.CopyvioMixIn.copyvio_compare>`.
+    """

earwigbot/frontend.py

@@ -1,137 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot's IRC Frontend Component
-
-The IRC frontend runs on a normal IRC server and expects users to interact with
-it and give it commands. Commands are stored as "command classes", subclasses
-of BaseCommand in irc/base_command.py. All command classes are automatically
-imported by irc/command_handler.py if they are in irc/commands.
-"""
-
-import logging
-import re
-
-from earwigbot import  commands
-from earwigbot.classes import Connection, Data, BrokenSocketException
-from earwigbot.config import config
-
-__all__ = ["get_connection", "startup", "main"]
-
-connection = None
-logger = logging.getLogger("earwigbot.frontend")
-sender_regex = re.compile(":(.*?)!(.*?)@(.*?)\Z")
-
-def get_connection():
-    """Return a new Connection() instance with information about our server
-    connection, but don't actually connect yet."""
-    cf = config.irc["frontend"]
-    connection = Connection(cf["host"], cf["port"], cf["nick"], cf["ident"],
-            cf["realname"], logger)
-    return connection
-
-def startup(conn):
-    """Accept a single arg, a Connection() object, and set our global variable
-    'connection' to it. Load all command classes in irc/commands with
-    command_handler, and then establish a connection with the IRC server."""
-    global connection
-    connection = conn
-    commands.load(connection)
-    connection.connect()
-
-def main():
-    """Main loop for the frontend component.
-
-    get_connection() and startup() should have already been called before this.
-    """
-    read_buffer = str()
-
-    while 1:
-        try:
-            read_buffer = read_buffer + connection.get()
-        except BrokenSocketException:
-            logger.warn("Socket has broken on front-end; restarting bot")
-            return
-
-        lines = read_buffer.split("\n")
-        read_buffer = lines.pop()
-        for line in lines:
-            ret = _process_message(line)
-            if ret:
-                return
-
-def _process_message(line):
-    """Process a single message from IRC."""
-    line = line.strip().split()
-    data = Data(line)  # new Data instance to store info about this line
-
-    if line[1] == "JOIN":
-        data.nick, data.ident, data.host = sender_regex.findall(line[0])[0]
-        data.chan = line[2]
-        # Check for 'join' hooks in our commands:
-        commands.check("join", data)
-
-    elif line[1] == "PRIVMSG":
-        data.nick, data.ident, data.host = sender_regex.findall(line[0])[0]
-        data.msg = ' '.join(line[3:])[1:]
-        data.chan = line[2]
-
-        if data.chan == config.irc["frontend"]["nick"]:
-            # This is a privmsg to us, so set 'chan' as the nick of the, sender
-            # then check for private-only command hooks:
-            data.chan = data.nick
-            commands.check("msg_private", data)
-        else:
-            # Check for public-only command hooks:
-            commands.check("msg_public", data)
-
-        # Check for command hooks that apply to all messages:
-        commands.check("msg", data)
-
-        # Hardcode the !restart command (we can't restart from within an
-        # ordinary command):
-        if data.msg in ["!restart", ".restart"]:
-            if data.host in config.irc["permissions"]["owners"]:
-                logger.info("Restarting bot per owner request")
-                return True
-
-    # If we are pinged, pong back:
-    elif line[0] == "PING":
-        msg = " ".join(("PONG", line[1]))
-        connection.send(msg)
-
-    # On successful connection to the server:
-    elif line[1] == "376":
-        # If we're supposed to auth to NickServ, do that:
-        try:
-            username = config.irc["frontend"]["nickservUsername"]
-            password = config.irc["frontend"]["nickservPassword"]
-        except KeyError:
-            pass
-        else:
-            msg = " ".join(("IDENTIFY", username, password))
-            connection.say("NickServ", msg)
-
-        # Join all of our startup channels:
-        for chan in config.irc["frontend"]["channels"]:
-            connection.join(chan)

earwigbot/classes/__init__.py → earwigbot/irc/__init__.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,8 +20,8 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes.base_command import *
-from earwigbot.classes.base_task import *
-from earwigbot.classes.connection import *
-from earwigbot.classes.data import *
-from earwigbot.classes.rc import *
+from earwigbot.irc.connection import *
+from earwigbot.irc.data import *
+from earwigbot.irc.frontend import *
+from earwigbot.irc.rc import *
+from earwigbot.irc.watcher import *

earwigbot/irc/connection.py

@@ -0,0 +1,228 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import socket
+from threading import Lock
+from time import sleep, time
+
+from earwigbot.exceptions import BrokenSocketError
+
+__all__ = ["IRCConnection"]
+
+class IRCConnection(object):
+    """Interface with an IRC server."""
+
+    def __init__(self, host, port, nick, ident, realname):
+        self._host = host
+        self._port = port
+        self._nick = nick
+        self._ident = ident
+        self._realname = realname
+
+        self._is_running = False
+        self._send_lock = Lock()
+
+        self._last_recv = time()
+        self._last_ping = 0
+
+    def __repr__(self):
+        """Return the canonical string representation of the IRCConnection."""
+        res = "IRCConnection(host={0!r}, port={1!r}, nick={2!r}, ident={3!r}, realname={4!r})"
+        return res.format(self.host, self.port, self.nick, self.ident,
+                          self.realname)
+
+    def __str__(self):
+        """Return a nice string representation of the IRCConnection."""
+        res = "<IRCConnection {0}!{1} at {2}:{3}>"
+        return res.format(self.nick, self.ident, self.host, self.port)
+
+    def _connect(self):
+        """Connect to our IRC server."""
+        self._sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            self._sock.connect((self.host, self.port))
+        except socket.error:
+            self.logger.exception("Couldn't connect to IRC server; retrying")
+            sleep(8)
+            self._connect()
+        self._send("NICK {0}".format(self.nick))
+        self._send("USER {0} {1} * :{2}".format(self.ident, self.host, self.realname))
+
+    def _close(self):
+        """Completely close our connection with the IRC server."""
+        try:
+            self._sock.shutdown(socket.SHUT_RDWR)  # Shut down connection first
+        except socket.error:
+            pass  # Ignore if the socket is already down
+        self._sock.close()
+
+    def _get(self, size=4096):
+        """Receive (i.e. get) data from the server."""
+        data = self._sock.recv(size)
+        if not data:
+            # Socket isn't giving us any data, so it is dead or broken:
+            raise BrokenSocketError()
+        return data
+
+    def _send(self, msg, hidelog=False):
+        """Send data to the server."""
+        with self._send_lock:
+            try:
+                self._sock.sendall(msg + "\r\n")
+            except socket.error:
+                self._is_running = False
+            else:
+                if not hidelog:
+                    self.logger.debug(msg)
+
+    def _quit(self, msg=None):
+        """Issue a quit message to the server. Doesn't close the connection."""
+        if msg:
+            self._send("QUIT :{0}".format(msg))
+        else:
+            self._send("QUIT")
+
+    def _process_defaults(self, line):
+        """Default process hooks for lines received on IRC."""
+        self._last_recv = time()
+        if line[0] == "PING":  # If we are pinged, pong back
+            self.pong(line[1][1:])
+
+    @property
+    def host(self):
+        """The hostname of the IRC server, like ``"irc.freenode.net"``."""
+        return self._host
+
+    @property
+    def port(self):
+        """The port of the IRC server, like ``6667``."""
+        return self._port
+
+    @property
+    def nick(self):
+        """Our nickname on the server, like ``"EarwigBot"``."""
+        return self._nick
+
+    @property
+    def ident(self):
+        """Our ident on the server, like ``"earwig"``.
+
+        See http://en.wikipedia.org/wiki/Ident.
+        """
+        return self._ident
+
+    @property
+    def realname(self):
+        """Our realname (gecos field) on the server."""
+        return self._realname
+
+    def say(self, target, msg, hidelog=False):
+        """Send a private message to a target on the server."""
+        msg = "PRIVMSG {0} :{1}".format(target, msg)
+        self._send(msg, hidelog)
+
+    def reply(self, data, msg, hidelog=False):
+        """Send a private message as a reply to a user on the server."""
+        msg = "\x02{0}\x0f: {1}".format(data.nick, msg)
+        self.say(data.chan, msg, hidelog)
+
+    def action(self, target, msg, hidelog=False):
+        """Send a private message to a target on the server as an action."""
+        msg = "\x01ACTION {0}\x01".format(msg)
+        self.say(target, msg, hidelog)
+
+    def notice(self, target, msg, hidelog=False):
+        """Send a notice to a target on the server."""
+        msg = "NOTICE {0} :{1}".format(target, msg)
+        self._send(msg, hidelog)
+
+    def join(self, chan, hidelog=False):
+        """Join a channel on the server."""
+        msg = "JOIN {0}".format(chan)
+        self._send(msg, hidelog)
+
+    def part(self, chan, msg=None, hidelog=False):
+        """Part from a channel on the server, optionally using an message."""
+        if msg:
+            self._send("PART {0} :{1}".format(chan, msg), hidelog)
+        else:
+            self._send("PART {0}".format(chan), hidelog)
+
+    def mode(self, target, level, msg, hidelog=False):
+        """Send a mode message to the server."""
+        msg = "MODE {0} {1} {2}".format(target, level, msg)
+        self._send(msg, hidelog)
+
+    def ping(self, target, hidelog=False):
+        """Ping another entity on the server."""
+        msg = "PING {0}".format(target)
+        self._send(msg, hidelog)
+
+    def pong(self, target, hidelog=False):
+        """Pong another entity on the server."""
+        msg = "PONG {0}".format(target)
+        self._send(msg, hidelog)
+
+    def loop(self):
+        """Main loop for the IRC connection."""
+        self._is_running = True
+        read_buffer = ""
+        while 1:
+            try:
+                read_buffer += self._get()
+            except BrokenSocketError:
+                self._is_running = False
+                break
+
+            lines = read_buffer.split("\n")
+            read_buffer = lines.pop()
+            for line in lines:
+                line = line.strip().split()
+                self._process_defaults(line)
+                self._process_message(line)
+            if self.is_stopped():
+                break
+
+        self._close()
+
+    def keep_alive(self):
+        """Ensure that we stay connected, stopping if the connection breaks."""
+        now = time()
+        if now - self._last_recv > 120:
+            if self._last_ping < self._last_recv:
+                log = "Last message was received over 120 seconds ago. Pinging."
+                self.logger.debug(log)
+                self.ping(self.host)
+                self._last_ping = now
+            elif now - self._last_ping > 60:
+                self.logger.debug("No ping response in 60 seconds. Stopping.")
+                self.stop()
+
+    def stop(self, msg=None):
+        """Request the IRC connection to close at earliest convenience."""
+        if self._is_running:
+            self._quit(msg)
+            self._is_running = False
+
+    def is_stopped(self):
+        """Return whether the IRC connection has been (or is to be) closed."""
+        return not self._is_running

earwigbot/irc/data.py

@@ -0,0 +1,211 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import re
+
+__all__ = ["Data"]
+
+class Data(object):
+    """Store data from an individual line received on IRC."""
+
+    def __init__(self, bot, my_nick, line, msgtype):
+        self._bot = bot
+        self._my_nick = my_nick.lower()
+        self._line = line
+
+        self._is_private = self._is_command = False
+        self._msg = self._command = self._trigger = None
+        self._args = []
+        self._kwargs = {}
+
+        self._parse(msgtype)
+
+    def __repr__(self):
+        """Return the canonical string representation of the Data."""
+        res = "Data(bot={0!r}, my_nick={1!r}, line={2!r})"
+        return res.format(self.bot, self.my_nick, self.line)
+
+    def __str__(self):
+        """Return a nice string representation of the Data."""
+        return "<Data of {0!r}>".format(" ".join(self.line))
+
+    def _parse(self, msgtype):
+        """Parse a line from IRC into its components as instance attributes."""
+        sender = re.findall(":(.*?)!(.*?)@(.*?)\Z", self.line[0])[0]
+        self._nick, self._ident, self._host = sender
+        self._chan = self.line[2]
+
+        if msgtype == "PRIVMSG":
+            if self.chan.lower() == self.my_nick:
+                # This is a privmsg to us, so set 'chan' as the nick of the
+                # sender instead of the 'channel', which is ourselves:
+                self._chan = self._nick
+                self._is_private = True
+            self._msg = " ".join(self.line[3:])[1:]
+            self._parse_args()
+            self._parse_kwargs()
+
+    def _parse_args(self):
+        """Parse command arguments from the message.
+
+        self.msg is converted into the string self.command and the argument
+        list self.args if the message starts with a "trigger" ("!", ".", or the
+        bot's name); self.is_command will be set to True, and self.trigger will
+        store the trigger string. Otherwise, is_command will be set to False.
+        """
+        self._args = self.msg.strip().split()
+
+        try:
+            self._command = self.args.pop(0).lower()
+        except IndexError:
+            return
+
+        if self.command.startswith("!") or self.command.startswith("."):
+            # e.g. "!command arg1 arg2"
+            self._is_command = True
+            self._trigger = self.command[0]
+            self._command = self.command[1:]  # Strip the "!" or "."
+        elif self.command.startswith(self.my_nick):
+            # e.g. "EarwigBot, command arg1 arg2"
+            self._is_command = True
+            self._trigger = self.my_nick
+            try:
+                self._command = self.args.pop(0).lower()
+            except IndexError:
+                self._command = ""
+            else:
+                try:
+                    if self.msg[-1] == "." and self.msg[-2] != ".":
+                        if self.args:
+                            self.args[-1] = self.args[-1][:-1]
+                        else:
+                            self._command = self.command[:-1]
+                except IndexError:
+                    pass
+
+    def _parse_kwargs(self):
+        """Parse keyword arguments embedded in self.args.
+
+        Parse a command given as "!command key1=value1 key2=value2..." into a
+        dict, self.kwargs, like {'key1': 'value2', 'key2': 'value2'...}.
+        """
+        for arg in self.args:
+            try:
+                key, value = re.findall("^(.*?)\=(.*?)$", arg)[0]
+            except IndexError:
+                continue
+            if key and value:
+                self.kwargs[key] = value
+
+    @property
+    def my_nick(self):
+        """Our nickname, *not* the nickname of the sender."""
+        return self._my_nick
+
+    @property
+    def line(self):
+        """The full message received on IRC, including escape characters."""
+        return self._line
+
+    @property
+    def chan(self):
+        """Channel the message was sent from.
+
+        This will be equal to :py:attr:`nick` if the message is a private
+        message.
+        """
+        return self._chan
+
+    @property
+    def nick(self):
+        """Nickname of the sender."""
+        return self._nick
+
+    @property
+    def ident(self):
+        """`Ident <http://en.wikipedia.org/wiki/Ident>`_ of the sender."""
+        return self._ident
+
+    @property
+    def host(self):
+        """Hostname of the sender."""
+        return self._host
+
+    @property
+    def msg(self):
+        """Text of the sent message, if it is a message, else ``None``."""
+        return self._msg
+
+    @property
+    def is_private(self):
+        """``True`` if this message was sent to us *only*, else ``False``."""
+        return self._is_private
+
+    @property
+    def is_command(self):
+        """Boolean telling whether or not this message is a bot command.
+
+        A message is considered a command if and only if it begins with the
+        character ``"!"``, ``"."``, or the bot's name followed by optional
+        punctuation and a space (so ``EarwigBot: do something``, ``EarwigBot,
+        do something``, and ``EarwigBot do something`` are all valid).
+        """
+        return self._is_command
+
+    @property
+    def command(self):
+        """If the message is a command, this is the name of the command used.
+
+        See :py:attr:`is_command <self.is_command>` for when a message is
+        considered a command. If it's not a command, this will be set to
+        ``None``.
+        """
+        return self._command
+
+    @property
+    def trigger(self):
+        """If this message is a command, this is what triggered it.
+
+        It can be either "!" (``"!help"``), "." (``".help"``), or the bot's
+        name (``"EarwigBot: help"``). Otherwise, it will be ``None``."""
+        return self._trigger
+
+    @property
+    def args(self):
+        """List of all arguments given to this command.
+
+        For example, the message ``"!command arg1 arg2 arg3=val3"`` will
+        produce the args ``["arg1", "arg2", "arg3=val3"]``. This is empty if
+        the message was not a command or if it doesn't have arguments.
+        """
+        return self._args
+
+    @property
+    def kwargs(self):
+        """Dictionary of keyword arguments given to this command.
+
+        For example, the message ``"!command arg1=val1 arg2=val2"`` will
+        produce the kwargs ``{"arg1": "val1", "arg2": "val2"}``. This is empty
+        if the message was not a command or if it doesn't have keyword
+        arguments.
+        """
+        return self._kwargs

earwigbot/irc/frontend.py

@@ -0,0 +1,88 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from earwigbot.irc import IRCConnection, Data
+
+__all__ = ["Frontend"]
+
+class Frontend(IRCConnection):
+    """
+    **EarwigBot: IRC Frontend Component**
+
+    The IRC frontend runs on a normal IRC server and expects users to interact
+    with it and give it commands. Commands are stored as "command classes",
+    subclasses of :py:class:`~earwigbot.commands.Command`. All command classes
+    are automatically imported by :py:meth:`commands.load()
+    <earwigbot.managers._ResourceManager.load>` if they are in
+    :py:mod:`earwigbot.commands` or the bot's custom command directory
+    (explained in the :doc:`documentation </customizing>`).
+    """
+
+    def __init__(self, bot):
+        self.bot = bot
+        self.logger = bot.logger.getChild("frontend")
+
+        cf = bot.config.irc["frontend"]
+        base = super(Frontend, self)
+        base.__init__(cf["host"], cf["port"], cf["nick"], cf["ident"],
+                      cf["realname"])
+        self._connect()
+
+    def __repr__(self):
+        """Return the canonical string representation of the Frontend."""
+        res = "Frontend(host={0!r}, port={1!r}, nick={2!r}, ident={3!r}, realname={4!r}, bot={5!r})"
+        return res.format(self.host, self.port, self.nick, self.ident,
+                          self.realname, self.bot)
+
+    def __str__(self):
+        """Return a nice string representation of the Frontend."""
+        res = "<Frontend {0}!{1} at {2}:{3}>"
+        return res.format(self.nick, self.ident, self.host, self.port)
+
+    def _process_message(self, line):
+        """Process a single message from IRC."""
+        if line[1] == "JOIN":
+            data = Data(self.bot, self.nick, line, msgtype="JOIN")
+            self.bot.commands.call("join", data)
+
+        elif line[1] == "PRIVMSG":
+            data = Data(self.bot, self.nick, line, msgtype="PRIVMSG")
+            if data.is_private:
+                self.bot.commands.call("msg_private", data)
+            else:
+                self.bot.commands.call("msg_public", data)
+            self.bot.commands.call("msg", data)
+
+        elif line[1] == "376":  # On successful connection to the server
+            # If we're supposed to auth to NickServ, do that:
+            try:
+                username = self.bot.config.irc["frontend"]["nickservUsername"]
+                password = self.bot.config.irc["frontend"]["nickservPassword"]
+            except KeyError:
+                pass
+            else:
+                msg = "IDENTIFY {0} {1}".format(username, password)
+                self.say("NickServ", msg, hidelog=True)
+
+            # Join all of our startup channels:
+            for chan in self.bot.config.irc["frontend"]["channels"]:
+                self.join(chan)

earwigbot/classes/rc.py → earwigbot/irc/rc.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -25,14 +25,26 @@ import re
 __all__ = ["RC"]
 
 class RC(object):
-    """A class to store data on an event received from our IRC watcher."""
+    """Store data from an event received from our IRC watcher."""
     re_color = re.compile("\x03([0-9]{1,2}(,[0-9]{1,2})?)?")
     re_edit = re.compile("\A\[\[(.*?)\]\]\s(.*?)\s(http://.*?)\s\*\s(.*?)\s\*\s(.*?)\Z")
-    re_log = re.compile("\A\[\[(.*?)\]\]\s(.*?)\s\*\s(.*?)\s\*\s(.*?)\Z")
+    re_log = re.compile("\A\[\[(.*?)\]\]\s(.*?)\s\s\*\s(.*?)\s\*\s(.*?)\Z")
 
-    def __init__(self, msg):
+    pretty_edit = "\x02New {0}\x0F: \x0314[[\x0307{1}\x0314]]\x0306 * \x0303{2}\x0306 * \x0302{3}\x0306 * \x0310{4}"
+    pretty_log = "\x02New {0}\x0F: \x0303{1}\x0306 * \x0302{2}\x0306 * \x0310{3}"
+
+    def __init__(self, chan, msg):
+        self.chan = chan
         self.msg = msg
 
+    def __repr__(self):
+        """Return the canonical string representation of the RC."""
+        return "RC(chan={0!r}, msg={1!r})".format(self.chan, self.msg)
+
+    def __str__(self):
+        """Return a nice string representation of the RC."""
+        return "<RC of {0!r} on {1}>".format(self.msg, self.chan)
+
     def parse(self):
         """Parse a recent change event into some variables."""
         # Strip IRC color codes; we don't want or need 'em:
@@ -48,7 +60,7 @@ class RC(object):
             # We're probably missing the http:// part, because it's a log
             # entry, which lacks a URL:
             page, flags, user, comment = self.re_log.findall(msg)[0]
-            url = "".join(("http://en.wikipedia.org/wiki/", page))
+            url = "http://{0}.org/wiki/{1}".format(self.chan[1:], page)
 
             self.is_edit = False  # This is a log entry, not edit
 
@@ -61,35 +73,24 @@ class RC(object):
     def prettify(self):
         """Make a nice, colorful message to send back to the IRC front-end."""
         flags = self.flags
-        # "New <event>:" if we don't know exactly what happened:
-        event_type = flags
-        if "N" in flags:
-            event_type = "page"  # "New page:"
-        elif flags == "delete":
-            event_type = "deletion"  # "New deletion:"
+        if self.is_edit:
+            if "N" in flags:
+                event = "page"  # "New page:"
+            else:
+                event = "edit"  # "New edit:"
+                if "B" in flags:
+                    event = "bot edit"  # "New bot edit:"
+                if "M" in flags:
+                    event = "minor " + event  # "New minor (bot)? edit:"
+            return self.pretty_edit.format(event, self.page, self.user,
+                                           self.url, self.comment)
+
+        if flags == "delete":
+            event = "deletion"  # "New deletion:"
         elif flags == "protect":
-            event_type = "protection"  # "New protection:"
+            event = "protection"  # "New protection:"
         elif flags == "create":
-            event_type = "user"  # "New user:"
-        if self.page == "Special:Log/move":
-            event_type = "move"  # New move:
+            event = "user"  # "New user:"
         else:
-            event_type = "edit"  # "New edit:"
-            if "B" in flags:
-                # "New bot edit:"
-                event_type = "bot {}".format(event_type)
-            if "M" in flags:
-                # "New minor edit:" OR "New minor bot edit:"
-                event_type = "minor {}".format(event_type)
-
-        # Example formatting:
-        # New edit: [[Page title]] * User name * http://en... * edit summary
-        if self.is_edit:
-            return "".join(("\x02New ", event_type, "\x0F: \x0314[[\x0307",
-                            self.page, "\x0314]]\x0306 *\x0303 ", self.user,
-                            "\x0306 *\x0302 ", self.url, "\x0306 *\x0310 ",
-                            self.comment))
-
-        return "".join(("\x02New ", event_type, "\x0F: \x0303", self.user,
-                        "\x0306 *\x0302 ", self.url, "\x0306 *\x0310 ",
-                        self.comment))
+            event = flags  # Works for "move", "block", etc
+        return self.pretty_log.format(event, self.user, self.url, self.comment)

earwigbot/irc/watcher.py

@@ -0,0 +1,125 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import imp
+
+from earwigbot.irc import IRCConnection, RC
+
+__all__ = ["Watcher"]
+
+class Watcher(IRCConnection):
+    """
+    **EarwigBot: IRC Watcher Component**
+
+    The IRC watcher runs on a wiki recent-changes server and listens for
+    edits. Users cannot interact with this part of the bot. When an event
+    occurs, we run it through some rules stored in our config, which can result
+    in wiki bot tasks being started or messages being sent to channels on the
+    IRC frontend.
+    """
+
+    def __init__(self, bot):
+        self.bot = bot
+        self.logger = bot.logger.getChild("watcher")
+
+        cf = bot.config.irc["watcher"]
+        base = super(Watcher, self)
+        base.__init__(cf["host"], cf["port"], cf["nick"], cf["ident"],
+                      cf["realname"])
+        self._prepare_process_hook()
+        self._connect()
+
+    def __repr__(self):
+        """Return the canonical string representation of the Watcher."""
+        res = "Watcher(host={0!r}, port={1!r}, nick={2!r}, ident={3!r}, realname={4!r}, bot={5!r})"
+        return res.format(self.host, self.port, self.nick, self.ident,
+                          self.realname, self.bot)
+
+    def __str__(self):
+        """Return a nice string representation of the Watcher."""
+        res = "<Watcher {0}!{1} at {2}:{3}>"
+        return res.format(self.nick, self.ident, self.host, self.port)
+
+    def _process_message(self, line):
+        """Process a single message from IRC."""
+        if line[1] == "PRIVMSG":
+            chan = line[2]
+
+            # Ignore messages originating from channels not in our list, to
+            # prevent someone PMing us false data:
+            if chan not in self.bot.config.irc["watcher"]["channels"]:
+                return
+
+            msg = " ".join(line[3:])[1:]
+            rc = RC(chan, msg)  # New RC object to store this event's data
+            rc.parse()  # Parse a message into pagenames, usernames, etc.
+            self._process_rc_event(rc)
+
+        # When we've finished starting up, join all watcher channels:
+        elif line[1] == "376":
+            for chan in self.bot.config.irc["watcher"]["channels"]:
+                self.join(chan)
+
+    def _prepare_process_hook(self):
+        """Create our RC event process hook from information in config.
+
+        This will get put in the function self._process_hook, which takes the
+        Bot object and an RC object and returns a list of frontend channels to
+        report this event to.
+        """
+        # Set a default RC process hook that does nothing:
+        self._process_hook = lambda rc: ()
+        try:
+            rules = self.bot.config.data["rules"]
+        except KeyError:
+            return
+        module = imp.new_module("_rc_event_processing_rules")
+        path = self.bot.config.path
+        try:
+            exec compile(rules, path, "exec") in module.__dict__
+        except Exception:
+            e = "Could not compile config file's RC event rules:"
+            self.logger.exception(e)
+            return
+        self._process_hook_module = module
+        try:
+            self._process_hook = module.process
+        except AttributeError:
+            e = "RC event rules compiled correctly, but no process(bot, rc) function was found"
+            self.logger.error(e)
+            return
+
+    def _process_rc_event(self, rc):
+        """Process a recent change event from IRC (or, an RC object).
+
+        The actual processing is configurable, so we don't have that hard-coded
+        here. We simply call our process hook (self._process_hook), created by
+        self._prepare_process_hook() from information in the "rules" section of
+        our config.
+        """
+        chans = self._process_hook(self.bot, rc)
+        with self.bot.component_lock:
+            frontend = self.bot.frontend
+            if chans and frontend and not frontend.is_stopped():
+                pretty = rc.prettify()
+                for chan in chans:
+                    frontend.say(chan, pretty)

earwigbot/main.py

@@ -1,146 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot's Main Module
-
-The core is essentially responsible for starting the various bot components
-(irc, scheduler, etc) and making sure they are all happy. An explanation of the
-different components follows:
-
-EarwigBot has three components that can run independently of each other: an IRC
-front-end, an IRC watcher, and a wiki scheduler.
-* The IRC front-end runs on a normal IRC server and expects users to interact
-  with it/give it commands.
-* The IRC watcher runs on a wiki recent-changes server and listens for edits.
-  Users cannot interact with this part of the bot.
-* The wiki scheduler runs wiki-editing bot tasks in separate threads at
-  user-defined times through a cron-like interface.
-
-There is a "priority" system here:
-1. If the IRC frontend is enabled, it will run on the main thread, and the IRC
-   watcher and wiki scheduler (if enabled) will run on separate threads.
-2. If the wiki scheduler is enabled, it will run on the main thread, and the
-   IRC watcher (if enabled) will run on a separate thread.
-3. If the IRC watcher is enabled, it will run on the main (and only) thread.
-Else, the bot will stop, as no components are enabled.
-"""
-
-import logging
-import threading
-import time
-
-from earwigbot import frontend
-from earwigbot import tasks
-from earwigbot import watcher
-from earwigbot.config import config
-
-logger = logging.getLogger("earwigbot")
-f_conn = None
-w_conn = None
-
-def irc_watcher(f_conn=None):
-    """Function to handle the IRC watcher as another thread (if frontend and/or
-    scheduler is enabled), otherwise run as the main thread."""
-    global w_conn
-    while 1:  # restart the watcher component if it breaks (and nothing else)
-        w_conn = watcher.get_connection()
-        w_conn.connect()
-        try:
-            watcher.main(w_conn, f_conn)
-        except:
-            logger.exception("Watcher had an error")
-        time.sleep(5)  # sleep a bit before restarting watcher
-        logger.warn("Watcher has stopped; restarting component")
-
-def wiki_scheduler():
-    """Function to handle the wiki scheduler as another thread, or as the
-    primary thread if the IRC frontend is not enabled."""
-    while 1:
-        time_start = time.time()
-        now = time.gmtime(time_start)
-
-        tasks.schedule(now)
-
-        time_end = time.time()
-        time_diff = time_start - time_end
-        if time_diff < 60:  # sleep until the next minute
-            time.sleep(60 - time_diff)
-
-def irc_frontend():
-    """If the IRC frontend is enabled, make it run on our primary thread, and
-    enable the wiki scheduler and IRC watcher on new threads if they are
-    enabled."""
-    global f_conn
-    logger.info("Starting IRC frontend")
-    f_conn = frontend.get_connection()
-    frontend.startup(f_conn)
-
-    if "wiki_schedule" in config.components:
-        logger.info("Starting wiki scheduler")
-        tasks.load()
-        t_scheduler = threading.Thread(target=wiki_scheduler)
-        t_scheduler.name = "wiki-scheduler"
-        t_scheduler.daemon = True
-        t_scheduler.start()
-
-    if "irc_watcher" in config.components:
-        logger.info("Starting IRC watcher")
-        t_watcher = threading.Thread(target=irc_watcher, args=(f_conn,))
-        t_watcher.name = "irc-watcher"
-        t_watcher.daemon = True
-        t_watcher.start()
-
-    frontend.main()
-
-    if "irc_watcher" in config.components:
-        w_conn.close()
-    f_conn.close()
-
-def main():
-    if "irc_frontend" in config.components:
-        # Make the frontend run on our primary thread if enabled, and enable
-        # additional components through that function
-        irc_frontend()
-
-    elif "wiki_schedule" in config.components:
-        # Run the scheduler on the main thread, but also run the IRC watcher on
-        # another thread iff it is enabled
-        logger.info("Starting wiki scheduler")
-        tasks.load()
-        if "irc_watcher" in enabled:
-            logger.info("Starting IRC watcher")
-            t_watcher = threading.Thread(target=irc_watcher)
-            t_watcher.name = "irc-watcher"
-            t_watcher.daemon = True
-            t_watcher.start()
-        wiki_scheduler()
-
-    elif "irc_watcher" in config.components:
-        # The IRC watcher is our only enabled component, so run its function
-        # only and don't worry about anything else:
-        logger.info("Starting IRC watcher")
-        irc_watcher()
-
-    else:  # Nothing is enabled!
-        logger.critical("No bot parts are enabled; stopping")
-        exit(1)

earwigbot/managers.py

@@ -0,0 +1,246 @@
+#! /usr/bin/env python
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import imp
+from os import listdir, path
+from re import sub
+from threading import RLock, Thread
+from time import gmtime, strftime
+
+from earwigbot.commands import Command
+from earwigbot.tasks import Task
+
+__all__ = ["CommandManager", "TaskManager"]
+
+class _ResourceManager(object):
+    """
+    **EarwigBot: Resource Manager**
+
+    Resources are essentially objects dynamically loaded by the bot, both
+    packaged with it (built-in resources) and created by users (plugins, aka
+    custom resources). Currently, the only two types of resources are IRC
+    commands and bot tasks. These are both loaded from two locations: the
+    :py:mod:`earwigbot.commands` and :py:mod:`earwigbot.tasks packages`, and
+    the :file:`commands/` and :file:`tasks/` directories within the bot's
+    working directory.
+
+    This class handles the low-level tasks of (re)loading resources via
+    :py:meth:`load`, retrieving specific resources via :py:meth:`get`, and
+    iterating over all resources via :py:meth:`__iter__`.
+    """
+    def __init__(self, bot, name, base):
+        self.bot = bot
+        self.logger = bot.logger.getChild(name)
+
+        self._resources = {}
+        self._resource_name = name  # e.g. "commands" or "tasks"
+        self._resource_base = base  # e.g. Command or Task
+        self._resource_access_lock = RLock()
+
+    def __repr__(self):
+        """Return the canonical string representation of the manager."""
+        res = "{0}(bot={1!r}, name={2!r}, base={3!r})"
+        return res.format(self.__class__.__name__, self.bot,
+                          self._resource_name, self._resource_base)
+
+    def __str__(self):
+        """Return a nice string representation of the manager."""
+        return "<{0} of {1}>".format(self.__class__.__name__, self.bot)
+
+    def __iter__(self):
+        with self.lock:
+            for resource in self._resources.itervalues():
+                yield resource
+
+    def _load_resource(self, name, path, klass):
+        """Instantiate a resource class and add it to the dictionary."""
+        res_type = self._resource_name[:-1]  # e.g. "command" or "task"
+        try:
+            resource = klass(self.bot)  # Create instance of resource
+        except Exception:
+            e = "Error instantiating {0} class in {1} (from {2})"
+            self.logger.exception(e.format(res_type, name, path))
+        else:
+            self._resources[resource.name] = resource
+            self.logger.debug("Loaded {0} {1}".format(res_type, resource.name))
+
+    def _load_module(self, name, path):
+        """Load a specific resource from a module, identified by name and path.
+
+        We'll first try to import it using imp magic, and if that works, make
+        instances of any classes inside that are subclasses of the base
+        (:py:attr:`self._resource_base <_resource_base>`), add them to the
+        resources dictionary with :py:meth:`self._load_resource()
+        <_load_resource>`, and finally log the addition. Any problems along
+        the way will either be ignored or logged.
+        """
+        f, path, desc = imp.find_module(name, [path])
+        try:
+            module = imp.load_module(name, f, path, desc)
+        except Exception:
+            e = "Couldn't load module {0} (from {1})"
+            self.logger.exception(e.format(name, path))
+            return
+        finally:
+            f.close()
+
+        for obj in vars(module).values():
+            if type(obj) is type:
+                isresource = issubclass(obj, self._resource_base)
+                if isresource and not obj is self._resource_base:
+                    self._load_resource(name, path, obj)
+
+    def _load_directory(self, dir):
+        """Load all valid resources in a given directory."""
+        self.logger.debug("Loading directory {0}".format(dir))
+        processed = []
+        for name in listdir(dir):
+            if not name.endswith(".py") and not name.endswith(".pyc"):
+                continue
+            if name.startswith("_") or name.startswith("."):
+                continue
+            modname = sub("\.pyc?$", "", name)  # Remove extension
+            if modname not in processed:
+                self._load_module(modname, dir)
+                processed.append(modname)
+
+    @property
+    def lock(self):
+        """The resource access/modify lock."""
+        return self._resource_access_lock
+
+    def load(self):
+        """Load (or reload) all valid resources into :py:attr:`_resources`."""
+        name = self._resource_name  # e.g. "commands" or "tasks"
+        with self.lock:
+            self._resources.clear()
+            builtin_dir = path.join(path.dirname(__file__), name)
+            plugins_dir = path.join(self.bot.config.root_dir, name)
+            self._load_directory(builtin_dir)  # Built-in resources
+            self._load_directory(plugins_dir)  # Custom resources, aka plugins
+
+        msg = "Loaded {0} {1}: {2}"
+        resources = ", ".join(self._resources.keys())
+        self.logger.info(msg.format(len(self._resources), name, resources))
+
+    def get(self, key):
+        """Return the class instance associated with a certain resource.
+
+        Will raise :py:exc:`KeyError` if the resource (a command or task) is
+        not found.
+        """
+        with self.lock:
+            return self._resources[key]
+
+
+class CommandManager(_ResourceManager):
+    """
+    Manages (i.e., loads, reloads, and calls) IRC commands.
+    """
+    def __init__(self, bot):
+        super(CommandManager, self).__init__(bot, "commands", Command)
+
+    def _wrap_check(self, command, data):
+        """Check whether a command should be called, catching errors."""
+        try:
+            return command.check(data)
+        except Exception:
+            e = "Error checking command '{0}' with data: {1}:"
+            self.logger.exception(e.format(command.name, data))
+
+    def _wrap_process(self, command, data):
+        """process() the message, catching and reporting any errors."""
+        try:
+            command.process(data)
+        except Exception:
+            e = "Error executing command '{0}':"
+            self.logger.exception(e.format(command.name))
+
+    def call(self, hook, data):
+        """Respond to a hook type and a :py:class:`Data` object."""
+        for command in self:
+            if hook in command.hooks and self._wrap_check(command, data):
+                thread = Thread(target=self._wrap_process,
+                                args=(command, data))
+                start_time = strftime("%b %d %H:%M:%S")
+                thread.name = "irc:{0} ({1})".format(command.name, start_time)
+                thread.daemon = True
+                thread.start()
+                return
+
+
+class TaskManager(_ResourceManager):
+    """
+    Manages (i.e., loads, reloads, schedules, and runs) wiki bot tasks.
+    """
+    def __init__(self, bot):
+        super(TaskManager, self).__init__(bot, "tasks", Task)
+
+    def _wrapper(self, task, **kwargs):
+        """Wrapper for task classes: run the task and catch any errors."""
+        try:
+            task.run(**kwargs)
+        except Exception:
+            msg = "Task '{0}' raised an exception and had to stop:"
+            self.logger.exception(msg.format(task.name))
+        else:
+            msg = "Task '{0}' finished successfully"
+            self.logger.info(msg.format(task.name))
+
+    def start(self, task_name, **kwargs):
+        """Start a given task in a new daemon thread, and return the thread.
+
+        kwargs are passed to :py:meth:`task.run() <earwigbot.tasks.Task.run>`.
+        If the task is not found, ``None`` will be returned an an error is
+        logged.
+        """
+        msg = "Starting task '{0}' in a new thread"
+        self.logger.info(msg.format(task_name))
+
+        try:
+            task = self.get(task_name)
+        except KeyError:
+            e = "Couldn't find task '{0}'"
+            self.logger.error(e.format(task_name))
+            return
+
+        task_thread = Thread(target=self._wrapper, args=(task,), kwargs=kwargs)
+        start_time = strftime("%b %d %H:%M:%S")
+        task_thread.name = "{0} ({1})".format(task_name, start_time)
+        task_thread.daemon = True
+        task_thread.start()
+        return task_thread
+
+    def schedule(self, now=None):
+        """Start all tasks that are supposed to be run at a given time."""
+        if not now:
+            now = gmtime()
+        # Get list of tasks to run this turn:
+        tasks = self.bot.config.schedule(now.tm_min, now.tm_hour, now.tm_mday,
+                                         now.tm_mon, now.tm_wday)
+
+        for task in tasks:
+            if isinstance(task, list):          # They've specified kwargs,
+                self.start(task[0], **task[1])  # so pass those to start
+            else:  # Otherwise, just pass task_name
+                self.start(task)

earwigbot/rules.py

@@ -1,85 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot's IRC Watcher Rules
-
-This file contains (configurable!) rules that EarwigBot's watcher uses after it
-recieves an event from IRC.
-"""
-
-import re
-
-from earwigbot import tasks
-
-afc_prefix = "wikipedia( talk)?:(wikiproject )?articles for creation"
-
-# compile some regexps used when finding specific events
-r_page = re.compile(afc_prefix)
-r_ffu = re.compile("wikipedia( talk)?:files for upload")
-r_move1 = re.compile("moved \[\[{}".format(afc_prefix))
-r_move2 = re.compile("moved \[\[(.*?)\]\] to \[\[{}".format(afc_prefix))
-r_moved_pages = re.compile("^moved \[\[(.*?)\]\] to \[\[(.*?)\]\]")
-r_delete = re.compile("deleted \"\[\[{}".format(afc_prefix))
-r_deleted_page = re.compile("^deleted \"\[\[(.*?)\]\]")
-r_restore = re.compile("restored \"\[\[{}".format(afc_prefix))
-r_restored_page = re.compile("^restored \"\[\[(.*?)\]\]")
-r_protect = re.compile("protected \"\[\[{}".format(afc_prefix))
-
-def process(rc):
-    """Given an RC() object, return a list of channels to report this event to.
-    Also, start any wiki bot tasks within this function if necessary."""
-    chans = set()  # channels to report this message to
-    page_name = rc.page.lower()
-    comment = rc.comment.lower()
-
-    if "!earwigbot" in rc.msg.lower():
-        chans.update(("##earwigbot", "#wikipedia-en-afc-feed"))
-
-    if r_page.search(page_name):
-        #tasks.start("afc_copyvios", page=rc.page)
-        chans.add("#wikipedia-en-afc-feed")
-
-    elif r_ffu.match(page_name):
-        chans.add("#wikipedia-en-afc-feed")
-
-    elif page_name.startswith("template:afc submission"):
-        chans.add("#wikipedia-en-afc-feed")
-
-    elif rc.flags == "move" and (r_move1.match(comment) or
-                                 r_move2.match(comment)):
-        p = r_moved_pages.findall(rc.comment)[0]
-        chans.add("#wikipedia-en-afc-feed")
-
-    elif rc.flags == "delete" and r_delete.match(comment):
-        p = r_deleted_page.findall(rc.comment)[0]
-        chans.add("#wikipedia-en-afc-feed")
-
-    elif rc.flags == "restore" and r_restore.match(comment):
-        p = r_restored_page.findall(rc.comment)[0]
-        #tasks.start("afc_copyvios", page=p)
-        chans.add("#wikipedia-en-afc-feed")
-
-    elif rc.flags == "protect" and r_protect.match(comment):
-        chans.add("#wikipedia-en-afc-feed")
-
-    return chans

earwigbot/runner.py

@@ -1,65 +0,0 @@
-#! /usr/bin/env python
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot Runner
-
-This is a very simple script that can be run from anywhere. It will add the
-'earwigbot' package to sys.path if it's not already in there (i.e., it hasn't
-been "installed"), accept a root_dir (the directory in which bot.py is located)
-and a decryption key from raw_input (if passwords are encrypted), then call
-config.load() and decrypt any passwords, and finally call the main() function
-of earwigbot.main.
-"""
-
-from os import path
-import sys
-
-def run():
-    pkg_dir = path.split(path.dirname(path.abspath(__file__)))[0]
-    if pkg_dir not in sys.path:
-        sys.path.insert(0, pkg_dir)
-
-    from earwigbot.config import config
-    from earwigbot import main
-
-    root_dir = raw_input()
-    config_path = path.join(root_dir, "config.json")
-    log_dir = path.join(root_dir, "logs")
-    is_encrypted = config.load(config_path, log_dir)
-    if is_encrypted:
-        config._decryption_key = raw_input()
-        config.decrypt(config.wiki, "password")
-        config.decrypt(config.wiki, "search", "credentials", "key")
-        config.decrypt(config.wiki, "search", "credentials", "secret")
-        config.decrypt(config.irc, "frontend", "nickservPassword")
-        config.decrypt(config.irc, "watcher", "nickservPassword")
-
-    try:
-        main.main()
-    except KeyboardInterrupt:
-        main.logger.critical("KeyboardInterrupt: stopping main bot loop")
-        exit(1)
-
-if __name__ == "__main__":
-    run()

earwigbot/tasks/__init__.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,119 +20,124 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-"""
-EarwigBot's Wiki Task Manager
-
-This package provides the wiki bot "tasks" EarwigBot runs. Here in __init__,
-you can find some functions used to load and run these tasks.
-"""
-
-import logging
-import os
-import sys
-import threading
-import time
-
-from earwigbot.classes import BaseTask
-from earwigbot.config import config
-
-__all__ = ["load", "schedule", "start", "get", "get_all"]
-
-# Base directory when searching for tasks:
-base_dir = os.path.dirname(os.path.abspath(__file__))
-
-# Store loaded tasks as a dict where the key is the task name and the value is
-# an instance of the task class:
-_tasks = {}
-
-# Logger for this module:
-logger = logging.getLogger("earwigbot.commands")
-
-def _load_task(filename):
-    """Try to load a specific task from a module, identified by file name."""
-    global _tasks
-
-    # Strip .py from the end of the filename and join with our package name:
-    name = ".".join(("tasks", filename[:-3]))
-    try:
-         __import__(name)
-    except:
-        logger.exception("Couldn't load file {0}:".format(filename))
-        return
-
-    task = sys.modules[name].Task()
-    task._setup_logger()
-    if not isinstance(task, BaseTask):
-        return
-
-    _tasks[task.name] = task
-    logger.debug("Added task {0}".format(task.name))
-
-def _wrapper(task, **kwargs):
-    """Wrapper for task classes: run the task and catch any errors."""
-    try:
-        task.run(**kwargs)
-    except:
-        error = "Task '{0}' raised an exception and had to stop"
-        logger.exception(error.format(task.name))
-    else:
-        logger.info("Task '{0}' finished without error".format(task.name))
-
-def load():
-    """Load all valid tasks from bot/tasks/, into the _tasks variable."""
-    files = os.listdir(base_dir)
-    files.sort()
-
-    for filename in files:
-        if filename.startswith("_") or not filename.endswith(".py"):
-            continue
-        try:
-            _load_task(filename)
-        except AttributeError:
-            pass  # The file is doesn't contain a task, so just move on
-
-    logger.info("Found {0} tasks: {1}".format(len(_tasks), ', '.join(_tasks.keys())))
-
-def schedule(now=time.gmtime()):
-    """Start all tasks that are supposed to be run at a given time."""
-    # Get list of tasks to run this turn:
-    tasks = config.schedule(now.tm_min, now.tm_hour, now.tm_mday, now.tm_mon,
-                            now.tm_wday)
-
-    for task in tasks:
-        if isinstance(task, list):     # they've specified kwargs
-            start(task[0], **task[1])  # so pass those to start_task
-        else:  # otherwise, just pass task_name
-            start(task)
+from earwigbot import exceptions
+from earwigbot import wiki
 
-def start(task_name, **kwargs):
-    """Start a given task in a new thread. Pass args to the task's run()
-    function."""
-    logger.info("Starting task '{0}' in a new thread".format(task_name))
+__all__ = ["Task"]
 
-    try:
-        task = _tasks[task_name]
-    except KeyError:
-        error = "Couldn't find task '{0}': bot/tasks/{0}.py does not exist"
-        logger.error(error.format(task_name))
-        return
-
-    task_thread = threading.Thread(target=lambda: _wrapper(task, **kwargs))
-    start_time = time.strftime("%b %d %H:%M:%S")
-    task_thread.name = "{0} ({1})".format(task_name, start_time)
-
-    # Stop bot task threads automagically if the main bot stops:
-    task_thread.daemon = True
+class Task(object):
+    """
+    **EarwigBot: Base Bot Task**
 
-    task_thread.start()
+    This package provides built-in wiki bot "tasks" EarwigBot runs. Additional
+    tasks can be installed as plugins in the bot's working directory.
 
-def get(task_name):
-    """Return the class instance associated with a certain task name.
+    This class (import with ``from earwigbot.tasks import Task``) can be
+    subclassed to create custom bot tasks.
 
-    Will raise KeyError if the task is not found.
+    To run a task, use :py:meth:`bot.tasks.start(name, **kwargs)
+    <earwigbot.managers.TaskManager.start>`. ``**kwargs`` get passed to the
+    Task's :meth:`run` method.
     """
-    return _tasks[task_name]
+    name = None
+    number = 0
+
+    def __init__(self, bot):
+        """Constructor for new tasks.
+
+        This is called once immediately after the task class is loaded by
+        the task manager (in :py:meth:`tasks.load()
+        <earwigbot.managers._ResourceManager.load>`). Don't override this
+        directly; if you do, remember to place ``super(Task, self).__init()``
+        first. Use :py:meth:`setup` for typical task-init/setup needs.
+        """
+        self.bot = bot
+        self.config = bot.config
+        self.logger = bot.tasks.logger.getChild(self.name)
+        self.setup()
+
+    def __repr__(self):
+        """Return the canonical string representation of the Task."""
+        res = "Task(name={0!r}, number={1!r}, bot={2!r})"
+        return res.format(self.name, self.number, self.bot)
+
+    def __str__(self):
+        """Return a nice string representation of the Task."""
+        res = "<Task {0} ({1}) of {2}>"
+        return res.format(self.name, self.number, self.bot)
+
+    def setup(self):
+        """Hook called immediately after the task is loaded.
+
+        Does nothing by default; feel free to override.
+        """
+        pass
+
+    def run(self, **kwargs):
+        """Main entry point to run a given task.
+
+        This is called directly by :py:meth:`tasks.start()
+        <earwigbot.managers.TaskManager.start>` and is the main way to make a
+        task do stuff. *kwargs* will be any keyword arguments passed to
+        :py:meth:`~earwigbot.managers.TaskManager.start`, which are entirely
+        optional.
+        """
+        pass
+
+    def make_summary(self, comment):
+        """Make an edit summary by filling in variables in a config value.
+
+        :py:attr:`config.wiki["summary"] <earwigbot.config.BotConfig.wiki>` is
+        used, where ``$2`` is replaced by the main summary body, given by the
+        *comment* argument, and ``$1`` is replaced by the task number.
+
+        If the config value is not found, we'll just return *comment* as-is.
+        """
+        try:
+            summary = self.bot.config.wiki["summary"]
+        except KeyError:
+            return comment
+        return summary.replace("$1", str(self.number)).replace("$2", comment)
+
+    def shutoff_enabled(self, site=None):
+        """Return whether on-wiki shutoff is enabled for this task.
+
+        We check a certain page for certain content. This is determined by
+        our config file: :py:attr:`config.wiki["shutoff"]["page"]
+        <earwigbot.config.BotConfig.wiki>` is used as the title, with any
+        embedded ``$1`` replaced by our username and ``$2``  replaced by the
+        task number; and :py:attr:`config.wiki["shutoff"]["disabled"]
+        <earwigbot.config.BotConfig.wiki>` is used as the content.
+
+        If the page has that exact content or the page does not exist, then
+        shutoff is "disabled", meaning the bot is supposed to run normally, and
+        we return  ``False``. If the page's content is something other than
+        what we expect, shutoff is enabled, and we return ``True``.
+
+        If a site is not provided, we'll try to use :py:attr:`self.site <site>`
+        if it's set. Otherwise, we'll use our default site.
+        """
+        if not site:
+            if hasattr(self, "site"):
+                site = getattr(self, "site")
+            else:
+                site = self.bot.wiki.get_site()
 
-def get_all():
-    """Return our dict of all loaded tasks."""
-    return _tasks
+        try:
+            cfg = self.config.wiki["shutoff"]
+        except KeyError:
+            return False
+        title = cfg.get("page", "User:$1/Shutoff/Task $2")
+        username = site.get_user().name
+        title = title.replace("$1", username).replace("$2", str(self.number))
+        page = site.get_page(title)
+
+        try:
+            content = page.get()
+        except exceptions.PageNotFoundError:
+            return False
+        if content == cfg.get("disabled", "run"):
+            return False
+
+        self.logger.warn("Emergency task shutoff has been enabled!")
+        return True

earwigbot/tasks/afc_catdelink.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,14 +20,14 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseTask
+from earwigbot.tasks import Task
 
-class Task(BaseTask):
+class AFCCatDelink(Task):
     """A task to delink mainspace categories in declined [[WP:AFC]]
     submissions."""
     name = "afc_catdelink"
 
-    def __init__(self):
+    def setup(self):
         pass
 
     def run(self, **kwargs):

earwigbot/tasks/afc_copyvios.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -26,18 +26,16 @@ from threading import Lock
 
 import oursql
 
-from earwigbot import wiki
-from earwigbot.classes import BaseTask
-from earwigbot.config import config
+from earwigbot.tasks import Task
 
-class Task(BaseTask):
+class AFCCopyvios(Task):
     """A task to check newly-edited [[WP:AFC]] submissions for copyright
     violations."""
     name = "afc_copyvios"
     number = 1
 
-    def __init__(self):
-        cfg = config.tasks.get(self.name, {})
+    def setup(self):
+        cfg = self.config.tasks.get(self.name, {})
         self.template = cfg.get("template", "AfC suspected copyvio")
         self.ignore_list = cfg.get("ignoreList", [])
         self.min_confidence = cfg.get("minConfidence", 0.5)
@@ -63,20 +61,20 @@ class Task(BaseTask):
         if self.shutoff_enabled():
             return
         title = kwargs["page"]
-        page = wiki.get_site().get_page(title)
+        page = self.bot.wiki.get_site().get_page(title)
         with self.db_access_lock:
             self.conn = oursql.connect(**self.conn_data)
             self.process(page)
 
     def process(self, page):
         """Detect copyvios in 'page' and add a note if any are found."""
-        title = page.title()
+        title = page.title
         if title in self.ignore_list:
             msg = "Skipping page in ignore list: [[{0}]]"
             self.logger.info(msg.format(title))
             return
 
-        pageid = page.pageid()
+        pageid = page.pageid
         if self.has_been_processed(pageid):
             msg = "Skipping check on already processed page [[{0}]]"
             self.logger.info(msg.format(title))
@@ -89,9 +87,9 @@ class Task(BaseTask):
 
         if result.violation:
             content = page.get()
-            template = "\{\{{0}|url={1}|confidence={2}\}\}"
+            template = "\{\{{0}|url={1}|confidence={2}\}\}\n"
             template = template.format(self.template, url, confidence)
-            newtext = "\n".join((template, content))
+            newtext = template + content
             if "{url}" in self.summary:
                 page.edit(newtext, self.summary.format(url=url))
             else:
@@ -140,10 +138,10 @@ class Task(BaseTask):
         be) retained for one day; this task does not remove old entries (that
         is handled by the Toolserver component).
 
-        This will only be called if "cache_results" == True in the task's,
+        This will only be called if "cache_results" == True in the task's
         config, which is False by default.
         """
-        pageid = page.pageid()
+        pageid = page.pageid
         hash = sha256(page.get()).hexdigest()
         query1 = "SELECT 1 FROM cache WHERE cache_id = ?"
         query2 = "DELETE FROM cache WHERE cache_id = ?"

earwigbot/tasks/afc_dailycats.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,14 +20,14 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseTask
+from earwigbot.tasks import Task
 
-class Task(BaseTask):
+class AFCDailyCats(Task):
     """ A task to create daily categories for [[WP:AFC]]."""
     name = "afc_dailycats"
     number = 3
 
-    def __init__(self):
+    def setup(self):
         pass
 
     def run(self, **kwargs):

earwigbot/tasks/afc_history.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -32,16 +32,9 @@ from numpy import arange
 import oursql
 
 from earwigbot import wiki
-from earwigbot.classes import BaseTask
-from earwigbot.config import config
+from earwigbot.tasks import Task
 
-# Valid submission statuses:
-STATUS_NONE = 0
-STATUS_PEND = 1
-STATUS_DECLINE = 2
-STATUS_ACCEPT = 3
-
-class Task(BaseTask):
+class AFCHistory(Task):
     """A task to generate charts about AfC submissions over time.
 
     The main function of the task is to work through the "AfC submissions by
@@ -57,8 +50,14 @@ class Task(BaseTask):
     """
     name = "afc_history"
 
-    def __init__(self):
-        cfg = config.tasks.get(self.name, {})
+    # Valid submission statuses:
+    STATUS_NONE = 0
+    STATUS_PEND = 1
+    STATUS_DECLINE = 2
+    STATUS_ACCEPT = 3
+
+    def setup(self):
+        cfg = self.config.tasks.get(self.name, {})
         self.num_days = cfg.get("days", 90)
         self.categories = cfg.get("categories", {})
 
@@ -73,10 +72,10 @@ class Task(BaseTask):
         self.db_access_lock = Lock()
 
     def run(self, **kwargs):
-        self.site = wiki.get_site()
+        self.site = self.bot.wiki.get_site()
         with self.db_access_lock:
             self.conn = oursql.connect(**self.conn_data)
-            
+
             action = kwargs.get("action")
             try:
                 num_days = int(kwargs.get("days", self.num_days))
@@ -90,9 +89,9 @@ class Task(BaseTask):
     def update(self, num_days):
         self.logger.info("Updating past {0} days".format(num_days))
         generator = self.backwards_cat_iterator()
-        for d in xrange(num_days):
+        for i in xrange(num_days):
             category = generator.next()
-            date = category.title().split("/")[-1]
+            date = category.title.split("/")[-1]
             self.update_date(date, category)
             sleep(10)
         self.logger.info("Update complete")
@@ -101,9 +100,9 @@ class Task(BaseTask):
         self.logger.info("Generating chart for past {0} days".format(num_days))
         data = OrderedDict()
         generator = self.backwards_cat_iterator()
-        for d in xrange(num_days):
+        for i in xrange(num_days):
             category = generator.next()
-            date = category.title().split("/")[-1]
+            date = category.title.split("/")[-1]
             data[date] = self.get_date_counts(date)
 
         data = OrderedDict(reversed(data.items()))  # Oldest to most recent
@@ -122,14 +121,14 @@ class Task(BaseTask):
             current -= timedelta(1)  # Subtract one day from date
 
     def update_date(self, date, category):
-        msg = "Updating {0} ([[{1}]])".format(date, category.title())
+        msg = "Updating {0} ([[{1}]])".format(date, category.title)
         self.logger.debug(msg)
 
         q_select = "SELECT page_date, page_status FROM page WHERE page_id = ?"
         q_delete = "DELETE FROM page WHERE page_id = ?"
         q_update = "UPDATE page SET page_date = ?, page_status = ? WHERE page_id = ?"
         q_insert = "INSERT INTO page VALUES (?, ?, ?)"
-        members = category.members(use_sql=True)
+        members = category.get_members()
 
         with self.conn.cursor() as cursor:
             for title, pageid in members:
@@ -137,7 +136,7 @@ class Task(BaseTask):
                 stored = cursor.fetchall()
                 status = self.get_status(title, pageid)
 
-                if status == STATUS_NONE:
+                if status == self.STATUS_NONE:
                     if stored:
                         cursor.execute(q_delete, (pageid,))
                     continue
@@ -152,17 +151,17 @@ class Task(BaseTask):
 
     def get_status(self, title, pageid):
         page = self.site.get_page(title)
-        ns = page.namespace()
+        ns = page.namespace
 
         if ns == wiki.NS_FILE_TALK:  # Ignore accepted FFU requests
-            return STATUS_NONE
+            return self.STATUS_NONE
 
         if ns == wiki.NS_TALK:
             new_page = page.toggle_talk()
             sleep(2)
-            if new_page.is_redirect():
-                return STATUS_NONE  # Ignore accepted AFC/R requests
-            return STATUS_ACCEPT
+            if new_page.is_redirect:
+                return self.STATUS_NONE  # Ignore accepted AFC/R requests
+            return self.STATUS_ACCEPT
 
         cats = self.categories
         sq = self.site.sql_query
@@ -170,16 +169,16 @@ class Task(BaseTask):
         match = lambda cat: list(sq(query, (cat.replace(" ", "_"), pageid)))
 
         if match(cats["pending"]):
-            return STATUS_PEND
+            return self.STATUS_PEND
         elif match(cats["unsubmitted"]):
-            return STATUS_NONE
+            return self.STATUS_NONE
         elif match(cats["declined"]):
-            return STATUS_DECLINE
-        return STATUS_NONE
+            return self.STATUS_DECLINE
+        return self.STATUS_NONE
 
     def get_date_counts(self, date):
         query = "SELECT COUNT(*) FROM page WHERE page_date = ? AND page_status = ?"
-        statuses = [STATUS_PEND, STATUS_DECLINE, STATUS_ACCEPT]
+        statuses = [self.STATUS_PEND, self.STATUS_DECLINE, self.STATUS_ACCEPT]
         counts = {}
         with self.conn.cursor() as cursor:
             for status in statuses:
@@ -193,9 +192,9 @@ class Task(BaseTask):
         plt.xlabel(self.graph.get("xaxis", "Date"))
         plt.ylabel(self.graph.get("yaxis", "Submissions"))
 
-        pends = [d[STATUS_PEND] for d in data.itervalues()]
-        declines = [d[STATUS_DECLINE] for d in data.itervalues()]
-        accepts = [d[STATUS_ACCEPT] for d in data.itervalues()]
+        pends = [d[self.STATUS_PEND] for d in data.itervalues()]
+        declines = [d[self.STATUS_DECLINE] for d in data.itervalues()]
+        accepts = [d[self.STATUS_ACCEPT] for d in data.itervalues()]
         pends_declines = [p + d for p, d in zip(pends, declines)]
         ind = arange(len(data))
         xsize = self.graph.get("xsize", 1200)

earwigbot/tasks/afc_statistics.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -21,7 +21,6 @@
 # SOFTWARE.
 
 from datetime import datetime
-import logging
 import re
 from os.path import expanduser
 from threading import Lock
@@ -29,20 +28,11 @@ from time import sleep
 
 import oursql
 
+from earwigbot import exceptions
 from earwigbot import wiki
-from earwigbot.classes import BaseTask
-from earwigbot.config import config
-
-# Chart status number constants:
-CHART_NONE = 0
-CHART_PEND = 1
-CHART_DRAFT = 2
-CHART_REVIEW = 3
-CHART_ACCEPT = 4
-CHART_DECLINE = 5
-CHART_MISPLACE = 6
-
-class Task(BaseTask):
+from earwigbot.tasks import Task
+
+class AFCStatistics(Task):
     """A task to generate statistics for WikiProject Articles for Creation.
 
     Statistics are stored in a MySQL database ("u_earwig_afc_statistics")
@@ -53,8 +43,17 @@ class Task(BaseTask):
     name = "afc_statistics"
     number = 2
 
-    def __init__(self):
-        self.cfg = cfg = config.tasks.get(self.name, {})
+    # Chart status number constants:
+    CHART_NONE = 0
+    CHART_PEND = 1
+    CHART_DRAFT = 2
+    CHART_REVIEW = 3
+    CHART_ACCEPT = 4
+    CHART_DECLINE = 5
+    CHART_MISPLACE = 6
+
+    def setup(self):
+        self.cfg = cfg = self.config.tasks.get(self.name, {})
 
         # Set some wiki-related attributes:
         self.pagename = cfg.get("page", "Template:AFC statistics")
@@ -83,22 +82,30 @@ class Task(BaseTask):
         (self.save()). We will additionally create an SQL connection with our
         local database.
         """
-        self.site = wiki.get_site()
-        with self.db_access_lock:
-            self.conn = oursql.connect(**self.conn_data)
+        action = kwargs.get("action")
+        if not self.db_access_lock.acquire(False):  # Non-blocking
+            if action == "sync":
+                self.logger.info("A sync is already ongoing; aborting")
+                return
+            self.logger.info("Waiting for database access lock")
+            self.db_access_lock.acquire()
 
-            action = kwargs.get("action")
+        try:
+            self.site = self.bot.wiki.get_site()
+            self.conn = oursql.connect(**self.conn_data)
             try:
                 if action == "save":
-                    self.save(**kwargs)
+                    self.save(kwargs)
                 elif action == "sync":
-                    self.sync(**kwargs)
+                    self.sync(kwargs)
                 elif action == "update":
-                    self.update(**kwargs)
+                    self.update(kwargs)
             finally:
                 self.conn.close()
+        finally:
+            self.db_access_lock.release()
 
-    def save(self, **kwargs):
+    def save(self, kwargs):
         """Save our local statistics to the wiki.
 
         After checking for emergency shutoff, the statistics chart is compiled,
@@ -107,7 +114,7 @@ class Task(BaseTask):
         """
         self.logger.info("Saving chart")
         if kwargs.get("fromIRC"):
-            summary = " ".join((self.summary, "(!earwigbot)"))
+            summary = self.summary + " (!earwigbot)"
         else:
             if self.shutoff_enabled():
                 return
@@ -117,17 +124,18 @@ class Task(BaseTask):
 
         page = self.site.get_page(self.pagename)
         text = page.get().encode("utf8")
-        newtext = re.sub("(<!-- stat begin -->)(.*?)(<!-- stat end -->)",
-                         statistics.join(("\\1\n", "\n\\3")), text,
-                         flags=re.DOTALL)
+        newtext = re.sub("<!-- stat begin -->(.*?)<!-- stat end -->",
+                         "<!-- stat begin -->\n" + statistics + "\n<!-- stat end -->",
+                         text, flags=re.DOTALL)
         if newtext == text:
             self.logger.info("Chart unchanged; not saving")
             return  # Don't edit the page if we're not adding anything
 
-        newtext = re.sub("(<!-- sig begin -->)(.*?)(<!-- sig end -->)",
-                         "\\1~~~ at ~~~~~\\3", newtext)
+        newtext = re.sub("<!-- sig begin -->(.*?)<!-- sig end -->",
+                         "<!-- sig begin -->~~~ at ~~~~~<!-- sig end -->",
+                         newtext)
         page.edit(newtext, summary, minor=True, bot=True)
-        self.logger.info("Chart saved to [[{0}]]".format(page.title()))
+        self.logger.info(u"Chart saved to [[{0}]]".format(page.title))
 
     def compile_charts(self):
         """Compile and return all statistics information from our local db."""
@@ -142,10 +150,10 @@ class Task(BaseTask):
         """Compile and return a single statistics chart."""
         chart_id, chart_title, special_title = chart_info
 
-        chart = "|".join((self.tl_header, chart_title))
+        chart = self.tl_header + "|" + chart_title
         if special_title:
-            chart += "".join(("|", special_title))
-        chart = "".join(("{{", chart, "}}"))
+            chart += "|" + special_title
+        chart = "{{" + chart + "}}"
 
         query = "SELECT * FROM page JOIN row ON page_id = row_id WHERE row_chart = ?"
         with self.conn.cursor(oursql.DictCursor) as cursor:
@@ -153,7 +161,7 @@ class Task(BaseTask):
             for page in cursor:
                 chart += "\n" + self.compile_chart_row(page).decode("utf8")
 
-        chart += "".join(("\n{{", self.tl_footer, "}}"))
+        chart += "\n{{" + self.tl_footer + "}}"
         return chart
 
     def compile_chart_row(self, page):
@@ -163,31 +171,22 @@ class Task(BaseTask):
         table, where keys are column names and values are their cell contents.
         """
         row = "{0}|s={page_status}|t={page_title}|h={page_short}|z={page_size}|"
-        row += "sr={page_special_user}|sh={page_special_hidden}|sd={page_special_time}|si={page_special_oldid}|"
-        row += "mr={page_modify_user}|mh={page_modify_hidden}|md={page_modify_time}|mi={page_modify_oldid}"
+        row += "sr={page_special_user}|sd={page_special_time}|si={page_special_oldid}|"
+        row += "mr={page_modify_user}|md={page_modify_time}|mi={page_modify_oldid}"
 
-        page["page_special_hidden"] = self.format_hidden(page["page_special_time"])
-        page["page_modify_hidden"] = self.format_hidden(page["page_modify_time"])
         page["page_special_time"] = self.format_time(page["page_special_time"])
         page["page_modify_time"] = self.format_time(page["page_modify_time"])
 
         if page["page_notes"]:
             row += "|n=1{page_notes}"
 
-        return "".join(("{{", row.format(self.tl_row, **page), "}}"))
+        return "{{" + row.format(self.tl_row, **page) + "}}"
 
     def format_time(self, dt):
         """Format a datetime into the standard MediaWiki timestamp format."""
         return dt.strftime("%H:%M, %d %b %Y")
 
-    def format_hidden(self, dt):
-        """Convert a datetime into seconds since the epoch.
-
-        This is used by the template as a hidden sortkey.
-        """
-        return int((dt - datetime(1970, 1, 1)).total_seconds())
-
-    def sync(self, **kwargs):
+    def sync(self, kwargs):
         """Synchronize our local statistics database with the site.
 
         Syncing involves, in order, updating tracked submissions that have
@@ -205,7 +204,7 @@ class Task(BaseTask):
         replag = self.site.get_replag()
         self.logger.debug("Server replag is {0}".format(replag))
         if replag > 600 and not kwargs.get("ignore_replag"):
-            msg = "Sync canceled as replag ({0} secs) is greater than ten minutes."
+            msg = "Sync canceled as replag ({0} secs) is greater than ten minutes"
             self.logger.warn(msg.format(replag))
             return
 
@@ -239,18 +238,23 @@ class Task(BaseTask):
                 self.untrack_page(cursor, pageid)
                 continue
 
+            title = title.decode("utf8")  # SQL gives strings; we want Unicode
             real_oldid = result[0][0]
             if oldid != real_oldid:
-                msg = "Updating page [[{0}]] (id: {1}) @ {2}"
+                msg = u"Updating page [[{0}]] (id: {1}) @ {2}"
                 self.logger.debug(msg.format(title, pageid, oldid))
                 self.logger.debug("  {0} -> {1}".format(oldid, real_oldid))
-                body = result[0][1].replace("_", " ")
+                base = result[0][1].decode("utf8").replace("_", " ")
                 ns = self.site.namespace_id_to_name(result[0][2])
                 if ns:
-                    real_title = ":".join((str(ns), body))
+                    real_title = u":".join((ns, base))
                 else:
-                    real_title = body
-                self.update_page(cursor, pageid, real_title)
+                    real_title = base
+                try:
+                    self.update_page(cursor, pageid, real_title)
+                except Exception:
+                    e = u"Error updating page [[{0}]] (id: {1})"
+                    self.logger.exception(e.format(real_title, pageid))
 
     def add_untracked(self, cursor):
         """Add pending submissions that are not yet tracked.
@@ -265,15 +269,17 @@ class Task(BaseTask):
         tracked = [i[0] for i in cursor.fetchall()]
 
         category = self.site.get_category(self.pending_cat)
-        pending = category.members(use_sql=True)
-
-        for title, pageid in pending:
-            if title.decode("utf8") in self.ignore_list:
+        for title, pageid in category.get_members():
+            if title in self.ignore_list:
                 continue
             if pageid not in tracked:
-                msg = "Tracking page [[{0}]] (id: {1})".format(title, pageid)
+                msg = u"Tracking page [[{0}]] (id: {1})".format(title, pageid)
                 self.logger.debug(msg)
-                self.track_page(cursor, pageid, title)
+                try:
+                    self.track_page(cursor, pageid, title)
+                except Exception:
+                    e = u"Error tracking page [[{0}]] (id: {1})"
+                    self.logger.exception(e.format(title, pageid))
 
     def delete_old(self, cursor):
         """Remove old submissions from the database.
@@ -285,9 +291,9 @@ class Task(BaseTask):
         query = """DELETE FROM page, row USING page JOIN row
                    ON page_id = row_id WHERE row_chart IN (?, ?)
                    AND ADDTIME(page_special_time, '36:00:00') < NOW()"""
-        cursor.execute(query, (CHART_ACCEPT, CHART_DECLINE))
+        cursor.execute(query, (self.CHART_ACCEPT, self.CHART_DECLINE))
 
-    def update(self, **kwargs):
+    def update(self, kwargs):
         """Update a page by name, regardless of whether anything has changed.
 
         Mainly intended as a command to be used via IRC, e.g.:
@@ -297,17 +303,17 @@ class Task(BaseTask):
         if not title:
             return
 
-        title = title.replace("_", " ")
+        title = title.replace("_", " ").decode("utf8")
         query = "SELECT page_id, page_modify_oldid FROM page WHERE page_title = ?"
         with self.conn.cursor() as cursor:
             cursor.execute(query, (title,))
             try:
                 pageid, oldid = cursor.fetchall()[0]
             except IndexError:
-                msg = "Page [[{0}]] not found in database".format(title)
+                msg = u"Page [[{0}]] not found in database".format(title)
                 self.logger.error(msg)
 
-            msg = "Updating page [[{0}]] (id: {1}) @ {2}"
+            msg = u"Updating page [[{0}]] (id: {1}) @ {2}"
             self.logger.info(msg.format(title, pageid, oldid))
             self.update_page(cursor, pageid, title)
 
@@ -326,14 +332,14 @@ class Task(BaseTask):
         """
         content = self.get_content(title)
         if content is None:
-            msg = "Could not get page content for [[{0}]]".format(title)
+            msg = u"Could not get page content for [[{0}]]".format(title)
             self.logger.error(msg)
             return
 
-        namespace = self.site.get_page(title).namespace()
+        namespace = self.site.get_page(title).namespace
         status, chart = self.get_status_and_chart(content, namespace)
-        if chart == CHART_NONE:
-            msg = "Could not find a status for [[{0}]]".format(title)
+        if chart == self.CHART_NONE:
+            msg = u"Could not find a status for [[{0}]]".format(title)
             self.logger.warn(msg)
             return
 
@@ -346,10 +352,8 @@ class Task(BaseTask):
         query1 = "INSERT INTO row VALUES (?, ?)"
         query2 = "INSERT INTO page VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
         cursor.execute(query1, (pageid, chart))
-        cursor.execute(query2, (pageid, status, title.decode("utf8"),
-                                short.decode("utf8"), size, notes,
-                                m_user.decode("utf8"), m_time, m_id,
-                                s_user.decode("utf8"), s_time, s_id))
+        cursor.execute(query2, (pageid, status, title, short, size, notes,
+                                m_user, m_time, m_id, s_user, s_time, s_id))
 
     def update_page(self, cursor, pageid, title):
         """Update hook for when page is already in our database.
@@ -360,13 +364,13 @@ class Task(BaseTask):
         """
         content = self.get_content(title)
         if content is None:
-            msg = "Could not get page content for [[{0}]]".format(title)
+            msg = u"Could not get page content for [[{0}]]".format(title)
             self.logger.error(msg)
             return
 
-        namespace = self.site.get_page(title).namespace()
+        namespace = self.site.get_page(title).namespace
         status, chart = self.get_status_and_chart(content, namespace)
-        if chart == CHART_NONE:
+        if chart == self.CHART_NONE:
             self.untrack_page(cursor, pageid)
             return
 
@@ -377,17 +381,25 @@ class Task(BaseTask):
 
         size = self.get_size(content)
         m_user, m_time, m_id = self.get_modify(pageid)
-        notes = self.get_notes(chart, content, m_time, result["page_special_user"])
 
-        if title != result["page_title"]:
+        if title != result["page_title"].decode("utf8"):
             self.update_page_title(cursor, result, pageid, title)
 
         if m_id != result["page_modify_oldid"]:
-            self.update_page_modify(cursor, result, pageid, size, m_user, m_time, m_id)
+            self.update_page_modify(cursor, result, pageid, size, m_user,
+                                    m_time, m_id)
 
         if status != result["page_status"]:
-            self.update_page_status(cursor, result, pageid, status, chart)
+            special = self.update_page_status(cursor, result, pageid, status,
+                                              chart)
+            s_user = special[0]
+        else:
+            try:
+                s_user = result["page_special_user"].decode("utf8")
+            except AttributeError:  # Happens if page_special_user is None
+                s_user = result["page_special_user"]
 
+        notes = self.get_notes(chart, content, m_time, s_user)
         if notes != result["page_notes"]:
             self.update_page_notes(cursor, result, pageid, notes)
 
@@ -395,21 +407,21 @@ class Task(BaseTask):
         """Update the title and short_title of a page in our database."""
         query = "UPDATE page SET page_title = ?, page_short = ? WHERE page_id = ?"
         short = self.get_short_title(title)
-        cursor.execute(query, (title.decode("utf8"), short.decode("utf8"),
-                               pageid))
-        msg = "  {0}: title: {1} -> {2}"
-        self.logger.debug(msg.format(pageid, result["page_title"], title))
+        cursor.execute(query, (title, short, pageid))
+
+        msg = u"  {0}: title: {1} -> {2}"
+        old_title = result["page_title"].decode("utf8")
+        self.logger.debug(msg.format(pageid, old_title, title))
 
     def update_page_modify(self, cursor, result, pageid, size, m_user, m_time, m_id):
         """Update the last modified information of a page in our database."""
         query = """UPDATE page SET page_size = ?, page_modify_user = ?,
                    page_modify_time = ?, page_modify_oldid = ?
                    WHERE page_id = ?"""
-        cursor.execute(query, (size, m_user.decode("utf8"), m_time, m_id,
-                               pageid))
+        cursor.execute(query, (size, m_user, m_time, m_id, pageid))
 
-        msg = "  {0}: modify: {1} / {2} / {3} -> {4} / {5} / {6}"
-        msg = msg.format(pageid, result["page_modify_user"],
+        msg = u"  {0}: modify: {1} / {2} / {3} -> {4} / {5} / {6}"
+        msg = msg.format(pageid, result["page_modify_user"].decode("utf8"),
                          result["page_modify_time"],
                          result["page_modify_oldid"], m_user, m_time, m_id)
         self.logger.debug(msg)
@@ -428,16 +440,17 @@ class Task(BaseTask):
                                      result["row_chart"], status, chart))
 
         s_user, s_time, s_id = self.get_special(pageid, chart)
-
         if s_id != result["page_special_oldid"]:
-            cursor.execute(query2, (s_user.decode("utf8"), s_time, s_id,
-                                    pageid))
-            msg = "{0}: special: {1} / {2} / {3} -> {4} / {5} / {6}"
-            msg = msg.format(pageid, result["page_special_user"],
+            cursor.execute(query2, (s_user, s_time, s_id, pageid))
+            msg = u"{0}: special: {1} / {2} / {3} -> {4} / {5} / {6}"
+            msg = msg.format(pageid,
+                             result["page_special_user"].decode("utf8"),
                              result["page_special_time"],
                              result["page_special_oldid"], s_user, s_time, s_id)
             self.logger.debug(msg)
 
+        return s_user, s_time, s_id
+
     def update_page_notes(self, cursor, result, pageid, notes):
         """Update the notes (or warnings) of a page in our database."""
         query = "UPDATE page SET page_notes = ? WHERE page_id = ?"
@@ -454,36 +467,34 @@ class Task(BaseTask):
         """
         query = "SELECT page_latest FROM page WHERE page_title = ? AND page_namespace = ?"
         try:
-            namespace, base = title.decode("utf8").split(":", 1)
+            namespace, base = title.split(":", 1)
         except ValueError:
-            base = title.decode("utf8")
+            base = title
             ns = wiki.NS_MAIN
         else:
             try:
                 ns = self.site.namespace_name_to_id(namespace)
-            except wiki.NamespaceNotFoundError:
-                base = title.decode("utf8")
+            except exceptions.NamespaceNotFoundError:
+                base = title
                 ns = wiki.NS_MAIN
 
         result = self.site.sql_query(query, (base.replace(" ", "_"), ns))
-        revid = int(list(result)[0][0])
-
+        try:
+            revid = int(list(result)[0][0])
+        except IndexError:
+            return None
         return self.get_revision_content(revid)
 
-    def get_revision_content(self, revid):
+    def get_revision_content(self, revid, tries=1):
         """Get the content of a revision by ID from the API."""
         res = self.site.api_query(action="query", prop="revisions",
                                   revids=revid, rvprop="content")
         try:
             return res["query"]["pages"].values()[0]["revisions"][0]["*"]
         except KeyError:
-            sleep(5)
-            res = self.site.api_query(action="query", prop="revisions",
-                                      revids=revid, rvprop="content")
-            try:
-                return res["query"]["pages"].values()[0]["revisions"][0]["*"]
-            except KeyError:
-                return None
+            if tries > 0:
+                sleep(5)
+                return self.get_revision_content(revid, tries=tries - 1)
 
     def get_status_and_chart(self, content, namespace):
         """Determine the status and chart number of an AFC submission.
@@ -498,23 +509,23 @@ class Task(BaseTask):
         statuses = self.get_statuses(content)
 
         if "R" in statuses:
-            status, chart = "r", CHART_REVIEW
+            status, chart = "r", self.CHART_REVIEW
         elif "H" in statuses:
-            status, chart = "p", CHART_DRAFT
+            status, chart = "p", self.CHART_DRAFT
         elif "P" in statuses:
-            status, chart = "p", CHART_PEND
+            status, chart = "p", self.CHART_PEND
         elif "T" in statuses:
-            status, chart = None, CHART_NONE
+            status, chart = None, self.CHART_NONE
         elif "D" in statuses:
-            status, chart = "d", CHART_DECLINE
+            status, chart = "d", self.CHART_DECLINE
         else:
-            status, chart = None, CHART_NONE
+            status, chart = None, self.CHART_NONE
 
         if namespace == wiki.NS_MAIN:
             if not statuses:
-                status, chart = "a", CHART_ACCEPT
+                status, chart = "a", self.CHART_ACCEPT
             else:
-                status, chart = None, CHART_MISPLACE
+                status, chart = None, self.CHART_MISPLACE
 
         return status, chart
 
@@ -579,7 +590,7 @@ class Task(BaseTask):
         """
         short = re.sub("Wikipedia(\s*talk)?\:Articles\sfor\screation\/", "", title)
         if len(short) > 50:
-            short = "".join((short[:47], "..."))
+            short = short[:47] + "..."
         return short
 
     def get_size(self, content):
@@ -596,7 +607,8 @@ class Task(BaseTask):
                    JOIN page ON rev_id = page_latest WHERE page_id = ?"""
         result = self.site.sql_query(query, (pageid,))
         m_user, m_time, m_id = list(result)[0]
-        return m_user, datetime.strptime(m_time, "%Y%m%d%H%M%S"), m_id
+        timestamp = datetime.strptime(m_time, "%Y%m%d%H%M%S")
+        return m_user.decode("utf8"), timestamp, m_id
 
     def get_special(self, pageid, chart):
         """Return information about a page's "special" edit.
@@ -611,25 +623,25 @@ class Task(BaseTask):
         its revision ID. If the page's status is not something that involves
         "special"-ing, we will return None for all three. The same will be
         returned if we cannot determine when the page was "special"-ed, or if
-        it was "special"-ed more than 250 edits ago.
+        it was "special"-ed more than 100 edits ago.
         """
-        if chart ==CHART_NONE:
+        if chart == self.CHART_NONE:
             return None, None, None
-        elif chart == CHART_MISPLACE:
+        elif chart == self.CHART_MISPLACE:
             return self.get_create(pageid)
-        elif chart == CHART_ACCEPT:
+        elif chart == self.CHART_ACCEPT:
             search_for = None
             search_not = ["R", "H", "P", "T", "D"]
-        elif chart == CHART_DRAFT:
+        elif chart == self.CHART_DRAFT:
             search_for = "H"
             search_not = []
-        elif chart == CHART_PEND:
+        elif chart == self.CHART_PEND:
             search_for = "P"
             search_not = []
-        elif chart == CHART_REVIEW:
+        elif chart == self.CHART_REVIEW:
             search_for = "R"
             search_not = []
-        elif chart == CHART_DECLINE:
+        elif chart == self.CHART_DECLINE:
             search_for = "D"
             search_not = ["R", "H", "P", "T"]
 
@@ -641,11 +653,16 @@ class Task(BaseTask):
         last = (None, None, None)
         for user, ts, revid in result:
             counter += 1
-            if counter > 100:
-                msg = "Exceeded 100 content lookups while determining special for page (id: {0}, chart: {1})"
+            if counter > 50:
+                msg = "Exceeded 50 content lookups while determining special for page (id: {0}, chart: {1})"
                 self.logger.warn(msg.format(pageid, chart))
                 return None, None, None
-            content = self.get_revision_content(revid)
+            try:
+                content = self.get_revision_content(revid)
+            except exceptions.APIError:
+                msg = "API error interrupted SQL query in get_special() for page (id: {0}, chart: {1})"
+                self.logger.exception(msg.format(pageid, chart))
+                return None, None, None
             statuses = self.get_statuses(content)
             matches = [s in statuses for s in search_not]
             if search_for:
@@ -654,7 +671,8 @@ class Task(BaseTask):
             else:
                 if any(matches):
                     return last
-            last = (user, datetime.strptime(ts, "%Y%m%d%H%M%S"), revid)
+            timestamp = datetime.strptime(ts, "%Y%m%d%H%M%S")
+            last = (user.decode("utf8"), timestamp, revid)
 
         return last
 
@@ -669,7 +687,8 @@ class Task(BaseTask):
                    (SELECT MIN(rev_id) FROM revision WHERE rev_page = ?)"""
         result = self.site.sql_query(query, (pageid,))
         c_user, c_time, c_id = list(result)[0]
-        return c_user, datetime.strptime(c_time, "%Y%m%d%H%M%S"), c_id
+        timestamp = datetime.strptime(c_time, "%Y%m%d%H%M%S")
+        return c_user.encode("utf8"), timestamp, c_id
 
     def get_notes(self, chart, content, m_time, s_user):
         """Return any special notes or warnings about this page.
@@ -683,19 +702,21 @@ class Task(BaseTask):
         """
         notes = ""
 
-        ignored_charts = [CHART_NONE, CHART_ACCEPT, CHART_DECLINE]
+        ignored_charts = [self.CHART_NONE, self.CHART_ACCEPT, self.CHART_DECLINE]
         if chart in ignored_charts:
             return notes
 
         statuses = self.get_statuses(content)
-        if "D" in statuses and chart != CHART_MISPLACE:
+        if "D" in statuses and chart != self.CHART_MISPLACE:
             notes += "|nr=1"  # Submission was resubmitted
 
         if len(content) < 500:
             notes += "|ns=1"  # Submission is short
 
-        if not re.search("\<ref\s*(.*?)\>(.*?)\</ref\>", content, re.I|re.S):
-            if re.search("https?:\/\/(.*?)\.", content, re.I|re.S):
+        if not re.search("\<ref\s*(.*?)\>(.*?)\</ref\>", content, re.I | re.S):
+            regex = "(https?:)|\[//(?!{0})([^ \]\\t\\n\\r\\f\\v]+?)"
+            sitedomain = re.escape(self.site.domain)
+            if re.search(regex.format(sitedomain), content, re.I | re.S):
                 notes += "|ni=1"  # Submission has no inline citations
             else:
                 notes += "|nu=1"  # Submission is completely unsourced
@@ -705,12 +726,12 @@ class Task(BaseTask):
         if time_since_modify > max_time:
             notes += "|no=1"  # Submission hasn't been touched in over 4 days
 
-        if chart in [CHART_PEND, CHART_DRAFT]:
+        if chart in [self.CHART_PEND, self.CHART_DRAFT] and s_user:
             submitter = self.site.get_user(s_user)
             try:
-                if submitter.blockinfo():
+                if submitter.blockinfo:
                     notes += "|nb=1"  # Submitter is blocked
-            except wiki.UserNotFoundError:  # Likely an IP
+            except exceptions.UserNotFoundError:  # Likely an IP
                 pass
 
         return notes

earwigbot/tasks/afc_undated.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,13 +20,13 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseTask
+from earwigbot.tasks import Task
 
-class Task(BaseTask):
+class AFCUndated(Task):
     """A task to clear [[Category:Undated AfC submissions]]."""
     name = "afc_undated"
 
-    def __init__(self):
+    def setup(self):
         pass
 
     def run(self, **kwargs):

earwigbot/tasks/blptag.py → earwigbot/tasks/blp_tag.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,14 +20,14 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseTask
+from earwigbot.tasks import Task
 
-class Task(BaseTask):
-    """A task to add |blp=yes to {{WPB}} or {{WPBS}} when it is used along with
-    {{WP Biography}}."""
-    name = "blptag"
+class BLPTag(Task):
+    """A task to add |blp=yes to ``{{WPB}}`` or ``{{WPBS}}`` when it is used
+    along with ``{{WP Biography}}``."""
+    name = "blp_tag"
 
-    def __init__(self):
+    def setup(self):
         pass
 
     def run(self, **kwargs):

earwigbot/tasks/image_display_resize.py

@@ -0,0 +1,33 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from earwigbot.tasks import Task
+
+class ImageDisplayResize(Task):
+    """A task to resize upscaled portraits in infoboxes."""
+    name = "image_display_resize"
+
+    def setup(self):
+        pass
+
+    def run(self, **kwargs):
+        pass

earwigbot/tasks/feed_dailycats.py → earwigbot/tasks/wikiproject_tagger.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,13 +20,13 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseTask
+from earwigbot.tasks import Task
 
-class Task(BaseTask):
-    """A task to create daily categories for [[WP:FEED]]."""
-    name = "feed_dailycats"
+class WikiProjectTagger(Task):
+    """A task to tag talk pages with WikiProject Banners."""
+    name = "wikiproject_tagger"
 
-    def __init__(self):
+    def setup(self):
         pass
 
     def run(self, **kwargs):

earwigbot/tasks/wrongmime.py → earwigbot/tasks/wrong_mime.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -20,14 +20,14 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from earwigbot.classes import BaseTask
+from earwigbot.tasks import Task
 
-class Task(BaseTask):
+class WrongMIME(Task):
     """A task to tag files whose extensions do not agree with their MIME
     type."""
-    name = "wrongmime"
+    name = "wrong_mime"
 
-    def __init__(self):
+    def setup(self):
         pass
 
     def run(self, **kwargs):

earwigbot/tests/test_blowfish.py

@@ -1,96 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-import unittest
-import random
-import string
-
-from earwigbot import blowfish
-
-class TestBlowfish(unittest.TestCase):
-
-    def test_key_sizes(self):
-        b = blowfish.Blowfish
-        e = blowfish.KeyLengthError
-
-        self.assertRaisesRegexp(e, "no key given", b, None)
-        self.assertRaisesRegexp(e, "no key given", b, "")
-        self.assertRaisesRegexp(e, "at least", b, " " * 3)
-        self.assertRaisesRegexp(e, "at least", b, "1234567")
-        self.assertRaisesRegexp(e, "less than", b, " " * 57)
-        self.assertRaisesRegexp(e, "less than", b, "x" * 60)
-        self.assertRaisesRegexp(e, "less than", b, "1" * 128)
-
-        b("These keys should be valid!")
-        b("'!\"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'")
-        b(" " * 8)
-        b(" " * 56)
-
-    def test_symmetry(self):
-        def _test_symmetry():
-            key_length = random.randint(8, 56)
-            msg_length = random.randint(0, 4096)
-            key = "".join([random.choice(chars) for i in xrange(key_length)])
-            msg = "".join([random.choice(chars) for i in xrange(msg_length)])
-            
-            enc = blowfish.encrypt(key, msg)
-            dec = blowfish.decrypt(key, enc)
-            self.assertEqual(dec, msg)
-
-        chars = string.letters + string.digits + string.punctuation
-        for i in xrange(8):
-            _test_symmetry()
-
-    def test_encryption(self):
-        tests = [
-            ("example_key", "Hello, I'm a message!", "8411a21574431176cdff9a549d27962c616014a9fe2a1fe3b0c7a823e8a1e635"),
-            ("another random key", "Another random message! :(", "2cdcdf4e53145897ed9d4cc2433aa4bf59b087b14d0ac76a13eff12dec00e60c40857109da3c7bc4"),
-            ("HEY LET'S TRY |°|_J|\|C7|_J/-\\710|\|", "Yes, that was my fail attempt at 1337SP33K >_>", "d4901c7c0956da3b9507cd81cd3c880d7cda25ec6c5336deb9280ce67c099eeddf7c7e052f3a946afbd92c32ae0ab8dbdd875bc5a3f0d686")
-        ]
-
-        for test in tests:
-            self.assertEquals(blowfish.encrypt(test[0], test[1]), test[2])
-
-    def test_decryption(self):
-        tests = [
-            ("blah blah blah", "ab35274c66bb8b3b03c9bd26ab477f3de06857e1d369ad35", "Blah, blah, blah!"),
-            ("random key", "eb2fe950c5c12bca9534ffdd27631f33d3e4bcae53a634b4aaa09f9fe14c4386", "Random message as well!"),
-            ("Okay, now I'm just desperate", "0da74e1cec41e8323da93d0c05bcf3919084130cef93021991da174fd97f8e1c9b125ed5263b41a8", "Unit testing is SO FUN ISN'T IT.")
-        ]
-
-        for test in tests:
-            self.assertEquals(blowfish.decrypt(test[0], test[1]), test[2])
-
-    def test_decryption_exceptions(self):
-        d = blowfish.decrypt
-        e = blowfish.BlowfishError
-
-        e1 = "could not be decoded"
-        e2 = "cannot be broken into 8-byte blocks"
-        e3 = "key is incorrect"
-
-        self.assertRaisesRegexp(e, e1, d, "some_key", "arr!")
-        self.assertRaisesRegexp(e, e2, d, "some_key", "abcd")
-        self.assertRaisesRegexp(e, e3, d, "some_key", "abcdabcdabcdabcd")
-
-if __name__ == "__main__":
-    unittest.main(verbosity=2)

earwigbot/util.py

@@ -0,0 +1,113 @@
+#! /usr/bin/env python
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+"""
+usage: :command:`earwigbot [-h] [-v] [-d] [-q] [-t NAME] [PATH]`
+
+This is EarwigBot's command-line utility, enabling you to easily start the bot
+or run specific tasks.
+
+.. glossary::
+
+``PATH``
+    path to the bot's working directory, which will be created if it doesn't
+    exist; current directory assumed if not specified
+``-h``, ``--help``
+    show this help message and exit
+``-v``, ``--version``
+    show program's version number and exit
+``-d``, ``--debug``
+    print all logs, including ``DEBUG``-level messages
+``-q``, ``--quiet``
+    don't print any logs except warnings and errors
+``-t NAME``, ``--task NAME``
+    given the name of a task, the bot will run it instead of the main bot and
+    then exit
+"""
+
+from argparse import ArgumentParser
+import logging
+from os import path
+from time import sleep
+
+from earwigbot import __version__
+from earwigbot.bot import Bot
+
+__all__ = ["main"]
+
+def main():
+    """Main entry point for the command-line utility."""
+    version = "EarwigBot v{0}".format(__version__)
+    desc = """This is EarwigBot's command-line utility, enabling you to easily
+              start the bot or run specific tasks."""
+    parser = ArgumentParser(description=desc)
+    parser.add_argument("path", nargs="?", metavar="PATH", default=path.curdir,
+                        help="""path to the bot's working directory, which will
+                                be created if it doesn't exist; current
+                                directory assumed if not specified""")
+    parser.add_argument("-v", "--version", action="version", version=version)
+    parser.add_argument("-d", "--debug", action="store_true",
+                        help="print all logs, including DEBUG-level messages")
+    parser.add_argument("-q", "--quiet", action="store_true",
+                        help="don't print any logs except warnings and errors")
+    parser.add_argument("-t", "--task", metavar="NAME",
+                        help="""given the name of a task, the bot will run it
+                                instead of the main bot and then exit""")
+    args = parser.parse_args()
+
+    level = logging.INFO
+    if args.debug and args.quiet:
+        parser.print_usage()
+        print "earwigbot: error: cannot show debug messages and be quiet at the same time"
+        return
+    if args.debug:
+        level = logging.DEBUG
+    elif args.quiet:
+        level = logging.WARNING
+    print version
+    print
+
+    bot = Bot(path.abspath(args.path), level=level)
+    if args.task:
+        thread = bot.tasks.start(args.task)
+        if not thread:
+            return
+        try:
+            while thread.is_alive():  # Keep it alive; it's a daemon
+                sleep(1)
+        except KeyboardInterrupt:
+            pass
+        finally:
+            if thread.is_alive():
+                bot.tasks.logger.warn("The task is will be killed")
+    else:
+        try:
+            bot.run()
+        except KeyboardInterrupt:
+            pass
+        finally:
+            if bot._keep_looping:  # Indicates bot hasn't already been stopped
+                bot.stop()
+
+if __name__ == "__main__":
+    main()

earwigbot/watcher.py

@@ -1,114 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot's IRC Watcher Component
-
-The IRC watcher runs on a wiki recent-changes server and listens for edits.
-Users cannot interact with this part of the bot. When an event occurs, we run
-it through rules.py's process() function, which can result in wiki bot tasks
-being started (located in tasks/) or messages being sent to channels on the IRC
-frontend.
-"""
-
-import logging
-
-from earwigbot import rules
-from earwigbot.classes import Connection, RC, BrokenSocketException
-from earwigbot.config import config
-
-frontend_conn = None
-logger = logging.getLogger("earwigbot.watcher")
-
-def get_connection():
-    """Return a new Connection() instance with connection information.
-
-    Don't actually connect yet.
-    """
-    cf = config.irc["watcher"]
-    connection = Connection(cf["host"], cf["port"], cf["nick"], cf["ident"],
-            cf["realname"], logger)
-    return connection
-
-def main(connection, f_conn=None):
-    """Main loop for the Watcher IRC Bot component.
-    
-    get_connection() should have already been called and the connection should
-    have been started with connection.connect(). Accept the frontend connection
-    as well as an optional parameter in order to send messages directly to
-    frontend IRC channels.
-    """
-    global frontend_conn
-    frontend_conn = f_conn
-    read_buffer = str()
-
-    while 1:
-        try:
-            read_buffer = read_buffer + connection.get()
-        except BrokenSocketException:
-            return
-
-        lines = read_buffer.split("\n")
-        read_buffer = lines.pop()
-
-        for line in lines:
-            _process_message(connection, line)
-
-def _process_message(connection, line):
-    """Process a single message from IRC."""
-    line = line.strip().split()
-
-    if line[1] == "PRIVMSG":
-        chan = line[2]
-
-        # Ignore messages originating from channels not in our list, to prevent
-        # someone PMing us false data:
-        if chan not in config.irc["watcher"]["channels"]:
-            return
-
-        msg = ' '.join(line[3:])[1:]
-        rc = RC(msg)  # new RC object to store this event's data
-        rc.parse()  # parse a message into pagenames, usernames, etc.
-        process_rc(rc)  # report to frontend channels or start tasks
-
-    # If we are pinged, pong back to the server:
-    elif line[0] == "PING":
-        msg = " ".join(("PONG", line[1]))
-        connection.send(msg)
-
-    # When we've finished starting up, join all watcher channels:
-    elif line[1] == "376":
-        for chan in config.irc["watcher"]["channels"]:
-            connection.join(chan)
-
-def process_rc(rc):
-    """Process a recent change event from IRC (or, an RC object).
-
-    The actual processing is configurable, so we don't have that hard-coded
-    here. We simply call rules's process() function and expect a list of
-    channels back, which we report the event data to.
-    """
-    chans = rules.process(rc)
-    if chans and frontend_conn:
-        pretty = rc.prettify()
-        for chan in chans:
-            frontend_conn.say(chan, pretty)

earwigbot/wiki/__init__.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -21,24 +21,31 @@
 # SOFTWARE.
 
 """
-EarwigBot's Wiki Toolset
+**EarwigBot: Wiki Toolset**
 
 This is a collection of classes and functions to read from and write to
-Wikipedia and other wiki sites. No connection whatsoever to python-wikitools
-written by Mr.Z-man, other than a similar purpose. We share no code.
+Wikipedia and other wiki sites. No connection whatsoever to `python-wikitools
+<http://code.google.com/p/python-wikitools/>`_ written by `Mr.Z-man
+<http://en.wikipedia.org/wiki/User:Mr.Z-man>`_, other than a similar purpose.
+We share no code.
 
-Import the toolset with `from earwigbot import wiki`.
+Import the toolset directly with ``from earwigbot import wiki``. If using the
+built-in integration with the rest of the bot, :py:class:`~earwigbot.bot.Bot`
+objects contain a :py:attr:`~earwigbot.bot.Bot.wiki` attribute, which is a
+:py:class:`~earwigbot.wiki.sitesdb.SitesDB` object tied to the :file:`sites.db`
+file located in the same directory as :file:`config.yml`. That object has the
+principal methods :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.get_site`,
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.add_site`, and
+:py:meth:`~earwigbot.wiki.sitesdb.SitesDB.remove_site` that should handle all
+of your :py:class:`~earwigbot.wiki.site.Site` (and thus,
+:py:class:`~earwigbot.wiki.page.Page`,
+:py:class:`~earwigbot.wiki.category.Category`, and
+:py:class:`~earwigbot.wiki.user.User`) needs.
 """
 
-import logging as _log
-logger = _log.getLogger("earwigbot.wiki")
-logger.addHandler(_log.NullHandler())
-
+from earwigbot.wiki.category import *
 from earwigbot.wiki.constants import *
-from earwigbot.wiki.exceptions import *
-from earwigbot.wiki.functions import *
-
-from earwigbot.wiki.category import Category
-from earwigbot.wiki.page import Page
-from earwigbot.wiki.site import Site
-from earwigbot.wiki.user import User
+from earwigbot.wiki.page import *
+from earwigbot.wiki.site import *
+from earwigbot.wiki.sitesdb import *
+from earwigbot.wiki.user import *

earwigbot/wiki/category.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,60 +22,184 @@
 
 from earwigbot.wiki.page import Page
 
+__all__ = ["Category"]
+
 class Category(Page):
     """
-    EarwigBot's Wiki Toolset: Category Class
+    **EarwigBot: Wiki Toolset: Category**
 
-    Represents a Category on a given Site, a subclass of Page. Provides
-    additional methods, but Page's own methods should work fine on Category
-    objects. Site.get_page() will return a Category instead of a Page if the
-    given title is in the category namespace; get_category() is shorthand,
-    because it accepts category names without the namespace prefix.
+    Represents a category on a given :py:class:`~earwigbot.wiki.site.Site`, a
+    subclass of :py:class:`~earwigbot.wiki.page.Page`. Provides additional
+    methods, but :py:class:`~earwigbot.wiki.page.Page`'s own methods should
+    work fine on :py:class:`Category` objects. :py:meth:`site.get_page()
+    <earwigbot.wiki.site.Site.get_page>` will return a :py:class:`Category`
+    instead of a :py:class:`~earwigbot.wiki.page.Page` if the given title is in
+    the category namespace; :py:meth:`~earwigbot.wiki.site.Site.get_category`
+    is shorthand, accepting category names without the namespace prefix.
 
-    Public methods:
-    members -- returns a list of page titles in the category
+    *Attributes:*
+
+    - :py:attr:`size`:    the total number of members in the category
+    - :py:attr:`pages`:   the number of pages in the category
+    - :py:attr:`files`:   the number of files in the category
+    - :py:attr:`subcats`: the number of subcategories in the category
+
+    *Public methods:*
+
+    - :py:meth:`get_members`: iterates over Pages in the category
     """
 
     def __repr__(self):
-        """Returns the canonical string representation of the Category."""
+        """Return the canonical string representation of the Category."""
         res = "Category(title={0!r}, follow_redirects={1!r}, site={2!r})"
         return res.format(self._title, self._follow_redirects, self._site)
 
     def __str__(self):
-        """Returns a nice string representation of the Category."""
-        return '<Category "{0}" of {1}>'.format(self.title(), str(self._site))
+        """Return a nice string representation of the Category."""
+        return '<Category "{0}" of {1}>'.format(self.title, str(self.site))
 
-    def members(self, limit=50, use_sql=False):
-        """Returns a list of page titles in the category.
+    def _get_members_via_api(self, limit, follow):
+        """Iterate over Pages in the category using the API."""
+        params = {"action": "query", "list": "categorymembers",
+                  "cmtitle": self.title}
 
-        If `limit` is provided, we will provide this many titles, or less if
-        the category is too small. `limit` defaults to 50; normal users can go
-        up to 500, and bots can go up to 5,000 on a single API query.
+        while 1:
+            params["cmlimit"] = limit if limit else "max"
+            result = self.site.api_query(**params)
+            for member in result["query"]["categorymembers"]:
+                title = member["title"]
+                yield self.site.get_page(title, follow_redirects=follow)
 
-        If `use_sql` is True, we will use a SQL query instead of the API. The
-        limit argument will be ignored, and pages will be returned as tuples
-        of (title, pageid) instead of just titles.
-        """
-        if use_sql:
-            query = """SELECT page_title, page_namespace, page_id FROM page
-                       JOIN categorylinks ON page_id = cl_from
-                       WHERE cl_to = ?"""
-            title = self.title().replace(" ", "_").split(":", 1)[1]
-            result = self._site.sql_query(query, (title,))
-            members = []
-            for row in result:
-                body = row[0].replace("_", " ")
-                namespace = self._site.namespace_id_to_name(row[1])
-                if namespace:
-                    title = ":".join((str(namespace), body))
-                else:  # Avoid doing a silly (albeit valid) ":Pagename" thing
-                    title = body
-                members.append((title, row[2]))
-            return members
+            if "query-continue" in result:
+                qcontinue = result["query-continue"]["categorymembers"]
+                params["cmcontinue"] = qcontinue["cmcontinue"]
+                if limit:
+                    limit -= len(result["query"]["categorymembers"])
+            else:
+                break
 
+    def _get_members_via_sql(self, limit, follow):
+        """Iterate over Pages in the category using SQL."""
+        query = """SELECT page_title, page_namespace, page_id FROM page
+                   JOIN categorylinks ON page_id = cl_from
+                   WHERE cl_to = ?"""
+        title = self.title.replace(" ", "_").split(":", 1)[1]
+
+        if limit:
+            query += " LIMIT ?"
+            result = self.site.sql_query(query, (title, limit))
+        else:
+            result = self.site.sql_query(query, (title,))
+
+        members = list(result)
+        for row in members:
+            base = row[0].replace("_", " ").decode("utf8")
+            namespace = self.site.namespace_id_to_name(row[1])
+            if namespace:
+                title = u":".join((namespace, base))
+            else:  # Avoid doing a silly (albeit valid) ":Pagename" thing
+                title = base
+            yield self.site.get_page(title, follow_redirects=follow,
+                                      pageid=row[2])
+
+    def _get_size_via_api(self, member_type):
+        """Return the size of the category using the API."""
+        result = self.site.api_query(action="query", prop="categoryinfo",
+                                     titles=self.title)
+        info = result["query"]["pages"].values()[0]["categoryinfo"]
+        return info[member_type]
+
+    def _get_size_via_sql(self, member_type):
+        """Return the size of the category using SQL."""
+        query = "SELECT COUNT(*) FROM categorylinks WHERE cl_to = ?"
+        title = self.title.replace(" ", "_").split(":", 1)[1]
+        if member_type == "size":
+            result = self.site.sql_query(query, (title,))
         else:
-            params = {"action": "query", "list": "categorymembers",
-                "cmlimit": limit, "cmtitle": self._title}
-            result = self._site._api_query(params)
-            members = result['query']['categorymembers']
-            return [member["title"] for member in members]
+            query += " AND cl_type = ?"
+            result = self.site.sql_query(query, (title, member_type[:-1]))
+        return list(result)[0][0]
+
+    def _get_size(self, member_type):
+        """Return the size of the category."""
+        services = {
+            self.site.SERVICE_API: self._get_size_via_api,
+            self.site.SERVICE_SQL: self._get_size_via_sql
+        }
+        return self.site.delegate(services, (member_type,))
+
+    @property
+    def size(self):
+        """The total number of members in the category.
+
+        Includes pages, files, and subcats. Equal to :py:attr:`pages` +
+        :py:attr:`files` + :py:attr:`subcats`. This will use either the API or
+        SQL depending on which are enabled and the amount of lag on each. This
+        is handled by :py:meth:`site.delegate()
+        <earwigbot.wiki.site.Site.delegate>`.
+        """
+        return self._get_size("size")
+
+    @property
+    def pages(self):
+        """The number of pages in the category.
+
+        This will use either the API or SQL depending on which are enabled and
+        the amount of lag on each. This is handled by :py:meth:`site.delegate()
+        <earwigbot.wiki.site.Site.delegate>`.
+        """
+        return self._get_size("pages")
+
+    @property
+    def files(self):
+        """The number of files in the category.
+
+        This will use either the API or SQL depending on which are enabled and
+        the amount of lag on each. This is handled by :py:meth:`site.delegate()
+        <earwigbot.wiki.site.Site.delegate>`.
+        """
+        return self._get_size("files")
+
+    @property
+    def subcats(self):
+        """The number of subcategories in the category.
+
+        This will use either the API or SQL depending on which are enabled and
+        the amount of lag on each. This is handled by :py:meth:`site.delegate()
+        <earwigbot.wiki.site.Site.delegate>`.
+        """
+        return self._get_size("subcats")
+
+    def get_members(self, limit=None, follow_redirects=None):
+        """Iterate over Pages in the category.
+
+        If *limit* is given, we will provide this many pages, or less if the
+        category is smaller. By default, *limit* is ``None``, meaning we will
+        keep iterating over members until the category is exhausted.
+        *follow_redirects* is passed directly to :py:meth:`site.get_page()
+        <earwigbot.wiki.site.Site.get_page>`; it defaults to ``None``, which
+        will use the value passed to our :py:meth:`__init__`.
+
+        This will use either the API or SQL depending on which are enabled and
+        the amount of lag on each. This is handled by :py:meth:`site.delegate()
+        <earwigbot.wiki.site.Site.delegate>`.
+
+        .. note::
+           Be careful when iterating over very large categories with no limit.
+           If using the API, at best, you will make one query per 5000 pages,
+           which can add up significantly for categories with hundreds of
+           thousands of members. As for SQL, note that *all page titles are
+           stored internally* as soon as the query is made, so the site-wide
+           SQL lock can be freed and unrelated queries can be made without
+           requiring a separate connection to be opened. This is generally not
+           an issue unless your category's size approaches several hundred
+           thousand, in which case the sheer number of titles in memory becomes
+           problematic.
+        """
+        services = {
+            self.site.SERVICE_API: self._get_members_via_api,
+            self.site.SERVICE_SQL: self._get_members_via_sql
+        }
+        if follow_redirects is None:
+            follow_redirects = self._follow_redirects
+        return self.site.delegate(services, (limit, follow_redirects))

earwigbot/wiki/constants.py

@@ -21,19 +21,23 @@
 # SOFTWARE.
 
 """
-EarwigBot's Wiki Toolset: Constants
+**EarwigBot: Wiki Toolset: Constants**
 
 This module defines some useful constants:
-* USER_AGENT  - our default User Agent when making API queries
-* NS_*        - default namespace IDs for easy lookup
 
-Import with `from earwigbot.wiki import constants` or `from earwigbot.wiki.constants import *`.
+- :py:const:`USER_AGENT`: our default User Agent when making API queries
+- :py:const:`NS_*`:       default namespace IDs for easy lookup
+
+Import directly with ``from earwigbot.wiki import constants`` or
+``from earwigbot.wiki.constants import *``. These are also available from
+:py:mod:`earwigbot.wiki` directly (e.g. ``earwigbot.wiki.USER_AGENT``).
 """
 
 # Default User Agent when making API queries:
 from earwigbot import __version__ as _v
 from platform import python_version as _p
-USER_AGENT = "EarwigBot/{0} (Python/{1}; https://github.com/earwig/earwigbot)".format(_v, _p())
+USER_AGENT = "EarwigBot/{0} (Python/{1}; https://github.com/earwig/earwigbot)"
+USER_AGENT = USER_AGENT.format(_v, _p())
 del _v, _p
 
 # Default namespace IDs:

earwigbot/wiki/copyvios/__init__.py

@@ -30,12 +30,12 @@ try:
 except ImportError:
     oauth = None
 
+from earwigbot import exceptions
 from earwigbot.wiki.copyvios.markov import *
 from earwigbot.wiki.copyvios.parsers import *
 from earwigbot.wiki.copyvios.search import *
-from earwigbot.wiki.exceptions import *
 
-__all__ = ["CopyvioCheckResult", "CopyvioMixin"]
+__all__ = ["CopyvioCheckResult", "CopyvioMixIn"]
 
 class CopyvioCheckResult(object):
     def __init__(self, violation, confidence, url, queries, article, chains):
@@ -52,7 +52,7 @@ class CopyvioCheckResult(object):
         return r.format(self.violation, self.confidence, self.url, self.queries)
 
 
-class CopyvioMixin(object):
+class CopyvioMixIn(object):
     """
     EarwigBot's Wiki Toolset: Copyright Violation Mixin
 

earwigbot/wiki/exceptions.py

@@ -1,124 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot's Wiki Toolset: Exceptions
-
-This module contains all exceptions used by the wiki package. There are a lot:
-
-        -- SiteNotFoundError
-        -- SiteAPIError
-        -- LoginError
-        -- NamespaceNotFoundError
-        -- PageNotFoundError
-        -- InvalidPageError
-        -- RedirectError
-        -- UserNotFoundError
-        -- EditError
-                -- PermissionsError
-                -- EditConflictError
-                -- NoContentError
-                -- ContentTooBigError
-                -- SpamDetectedError
-                -- FilteredError
-        -- SQLError
-        -- CopyvioCheckError
-                -- UnknownSearchEngineError
-                -- UnsupportedSearchEngineError
-                -- SearchQueryError
-"""
-
-class WikiToolsetError(Exception):
-    """Base exception class for errors in the Wiki Toolset."""
-
-class SiteNotFoundError(WikiToolsetError):
-    """A site matching the args given to get_site() could not be found in the
-    config file."""
-
-class SiteAPIError(WikiToolsetError):
-    """We couldn't connect to a site's API, perhaps because the server doesn't
-    exist, our URL is wrong or incomplete, or they're having temporary
-    problems."""
-
-class LoginError(WikiToolsetError):
-    """An error occured while trying to login. Perhaps the username/password is
-    incorrect."""
-
-class NamespaceNotFoundError(WikiToolsetError):
-    """A requested namespace name or namespace ID does not exist."""
-
-class PageNotFoundError(WikiToolsetError):
-    """Attempting to get certain information about a page that does not
-    exist."""
-
-class InvalidPageError(WikiToolsetError):
-    """Attempting to get certain information about a page whose title is
-    invalid."""
-
-class RedirectError(WikiToolsetError):
-    """Page's get_redirect_target() method failed because the page is either
-    not a redirect, or it is malformed."""
-
-class UserNotFoundError(WikiToolsetError):
-    """Attempting to get certain information about a user that does not
-    exist."""
-
-class EditError(WikiToolsetError):
-    """We got some error while editing. Sometimes, a subclass of this exception
-    will be used, like PermissionsError or EditConflictError."""
-
-class PermissionsError(EditError):
-    """We tried to do something we don't have permission to, like a non-admin
-    trying to delete a page, or trying to edit a page when no login information
-    was provided."""
-
-class EditConflictError(EditError):
-    """We've gotten an edit conflict or a (rarer) delete/recreate conflict."""
-
-class NoContentError(EditError):
-    """We tried to create a page or new section with no content."""
-
-class ContentTooBigError(EditError):
-    """The edit we tried to push exceeded the article size limit."""
-
-class SpamDetectedError(EditError):
-    """The spam filter refused our edit."""
-
-class FilteredError(EditError):
-    """The edit filter refused our edit."""
-
-class SQLError(WikiToolsetError):
-    """Some error involving SQL querying occurred."""
-
-class CopyvioCheckError(WikiToolsetError):
-    """An error occured when checking a page for copyright violations."""
-
-class UnknownSearchEngineError(CopyvioCheckError):
-    """CopyrightMixin().copyvio_check() called with an unknown engine."""
-
-class UnsupportedSearchEngineError(CopyvioCheckError):
-    """The engine requested is not available, e.g., because a required package
-    is missing."""
-
-class SearchQueryError(CopyvioCheckError):
-    """Some error ocurred while doing a search query."""

earwigbot/wiki/functions.py

@@ -1,219 +0,0 @@
-# -*- coding: utf-8  -*-
-#
-# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
-# furnished to do so, subject to the following conditions:
-# 
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-# 
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot's Wiki Toolset: Misc Functions
-
-This module, a component of the wiki package, contains miscellaneous functions
-that are not methods of any class, like get_site().
-
-There's no need to import this module explicitly. All functions here are
-automatically available from earwigbot.wiki.
-"""
-
-from cookielib import LWPCookieJar, LoadError
-import errno
-from getpass import getpass
-from os import chmod, path
-import platform
-import stat
-
-import earwigbot
-from earwigbot.config import config
-from earwigbot.wiki.exceptions import SiteNotFoundError
-from earwigbot.wiki.site import Site
-
-__all__ = ["get_site", "add_site", "del_site"]
-
-_cookiejar = None
-
-def _load_config():
-    """Called by a config-requiring function, such as get_site(), when config
-    has not been loaded. This will usually happen only if we're running code
-    directly from Python's interpreter and not the bot itself, because
-    earwigbot.py or core/main.py will already call these functions.
-    """
-    is_encrypted = config.load()
-    if is_encrypted:  # Passwords in the config file are encrypted
-        key = getpass("Enter key to unencrypt bot passwords: ")
-        config._decryption_key = key
-        config.decrypt(config.wiki, "password")
-
-def _get_cookiejar():
-    """Returns a LWPCookieJar object loaded from our .cookies file. The same
-    one is returned every time.
-
-    The .cookies file is located in the project root, same directory as
-    config.json and earwigbot.py. If it doesn't exist, we will create the file
-    and set it to be readable and writeable only by us. If it exists but the
-    information inside is bogus, we will ignore it.
-
-    This is normally called by _get_site_object_from_dict() (in turn called by
-    get_site()), and the cookiejar is passed to our Site's constructor, used
-    when it makes API queries. This way, we can easily preserve cookies between
-    sites (e.g., for CentralAuth), making logins easier.
-    """
-    global _cookiejar
-    if _cookiejar is not None:
-        return _cookiejar
-
-    cookie_file = path.join(config.root_dir, ".cookies")
-    _cookiejar = LWPCookieJar(cookie_file)
-
-    try:
-        _cookiejar.load()
-    except LoadError:
-        pass  # File contains bad data, so ignore it completely
-    except IOError as e:
-        if e.errno == errno.ENOENT:  # "No such file or directory"
-            # Create the file and restrict reading/writing only to the owner,
-            # so others can't peak at our cookies:
-            open(cookie_file, "w").close()
-            chmod(cookie_file, stat.S_IRUSR|stat.S_IWUSR)
-        else:
-            raise
-
-    return _cookiejar
-
-def _get_site_object_from_dict(name, d):
-    """Return a Site object based on the contents of a dict, probably acquired
-    through our config file, and a separate name.
-    """
-    project = d.get("project")
-    lang = d.get("lang")
-    base_url = d.get("baseURL")
-    article_path = d.get("articlePath")
-    script_path = d.get("scriptPath")
-    sql = d.get("sql", {})
-    namespaces = d.get("namespaces", {})
-    login = (config.wiki.get("username"), config.wiki.get("password"))
-    cookiejar = _get_cookiejar()
-    user_agent = config.wiki.get("userAgent")
-    assert_edit = config.wiki.get("assert")
-    maxlag = config.wiki.get("maxlag")
-    search_config = config.wiki.get("search")
-
-    if user_agent:
-        user_agent = user_agent.replace("$1", earwigbot.__version__)
-        user_agent = user_agent.replace("$2", platform.python_version())
-
-    for key, value in namespaces.items():  # Convert string keys to integers
-        del namespaces[key]
-        try:
-            namespaces[int(key)] = value
-        except ValueError:  # Data is broken, ignore it
-            namespaces = None
-            break
-
-    return Site(name=name, project=project, lang=lang, base_url=base_url,
-                article_path=article_path, script_path=script_path, sql=sql,
-                namespaces=namespaces, login=login, cookiejar=cookiejar,
-                user_agent=user_agent, assert_edit=assert_edit, maxlag=maxlag,
-                search_config=search_config)
-
-def get_site(name=None, project=None, lang=None):
-    """Returns a Site instance based on information from our config file.
-
-    With no arguments, returns the default site as specified by our config
-    file. This is default = config.wiki["defaultSite"];
-    config.wiki["sites"][default].
-
-    With `name` specified, returns the site specified by
-    config.wiki["sites"][name].
-
-    With `project` and `lang` specified, returns the site specified by the
-    member of config.wiki["sites"], `s`, for which s["project"] == project and
-    s["lang"] == lang.
-
-    We will attempt to login to the site automatically
-    using config.wiki["username"] and config.wiki["password"] if both are
-    defined.
-
-    Specifying a project without a lang or a lang without a project will raise
-    TypeError. If all three args are specified, `name` will be first tried,
-    then `project` and `lang`. If, with any number of args, a site cannot be
-    found in the config, SiteNotFoundError is raised.
-    """
-    # Check if config has been loaded, and load it if it hasn't:
-    if not config.is_loaded():
-        _load_config()
-
-    # Someone specified a project without a lang (or a lang without a project)!
-    if (project is None and lang is not None) or (project is not None and
-                                                  lang is None):
-        e = "Keyword arguments 'lang' and 'project' must be specified together."
-        raise TypeError(e)
-
-    # No args given, so return our default site (project is None implies lang
-    # is None, so we don't need to add that in):
-    if name is None and project is None:
-        try:
-            default = config.wiki["defaultSite"]
-        except KeyError:
-            e = "Default site is not specified in config."
-            raise SiteNotFoundError(e)
-        try:
-            site = config.wiki["sites"][default]
-        except KeyError:
-            e = "Default site specified by config is not in the config's sites list."
-            raise SiteNotFoundError(e)
-        return _get_site_object_from_dict(default, site)
-
-    # Name arg given, but don't look at others unless `name` isn't found:
-    if name is not None:
-        try:
-            site = config.wiki["sites"][name]
-        except KeyError:
-            if project is None:  # Implies lang is None, so only name was given
-                e = "Site '{0}' not found in config.".format(name)
-                raise SiteNotFoundError(e)
-            for sitename, site in config.wiki["sites"].items():
-                if site["project"] == project and site["lang"] == lang:
-                    return _get_site_object_from_dict(sitename, site)
-            e = "Neither site '{0}' nor site '{1}:{2}' found in config."
-            e.format(name, project, lang)
-            raise SiteNotFoundError(e)
-        else:
-            return _get_site_object_from_dict(name, site)
-
-    # If we end up here, then project and lang are both not None:
-    for sitename, site in config.wiki["sites"].items():
-        if site["project"] == project and site["lang"] == lang:
-            return _get_site_object_from_dict(sitename, site)
-    e = "Site '{0}:{1}' not found in config.".format(project, lang)
-    raise SiteNotFoundError(e)
-
-def add_site():
-    """STUB: config editing is required first.
-
-    Returns True if the site was added successfully or False if the site was
-    already in our config. Raises ConfigError if saving the updated file failed
-    for some reason."""
-    pass
-
-def del_site(name):
-    """STUB: config editing is required first.
-
-    Returns True if the site was removed successfully or False if the site was
-    not in our config originally. Raises ConfigError if saving the updated file
-    failed for some reason."""
-    pass

earwigbot/wiki/page.py

@@ -25,57 +25,83 @@ import re
 from time import gmtime, strftime
 from urllib import quote
 
-from earwigbot.wiki.copyvios import CopyvioMixin
-from earwigbot.wiki.exceptions import *
+try:
+    import mwparserfromhell
+except ImportError:
+    mwparserfromhell = None
+
+from earwigbot import exceptions
+from earwigbot.wiki.copyvios import CopyvioMixIn
+
+__all__ = ["Page"]
 
 class Page(CopyvioMixin):
     """
-    EarwigBot's Wiki Toolset: Page Class
-
-    Represents a Page on a given Site. Has methods for getting information
-    about the page, getting page content, and so on. Category is a subclass of
-    Page with additional methods.
-
-    Public methods:
-    title               -- returns the page's title, or pagename
-    exists              -- returns whether the page exists
-    pageid              -- returns an integer ID representing the page
-    url                 -- returns the page's URL
-    namespace           -- returns the page's namespace as an integer
-    protection          -- returns the page's current protection status
-    creator             -- returns the page's creator (first user to edit)
-    is_talkpage         -- returns True if the page is a talkpage, else False
-    is_redirect         -- returns True if the page is a redirect, else False
-    toggle_talk         -- returns a content page's talk page, or vice versa
-    get                 -- returns page content
-    get_redirect_target -- if the page is a redirect, returns its destination
-    edit                -- replaces the page's content or creates a new page
-    add_section         -- adds a new section at the bottom of the page
-    copyvio_check       -- checks the page for copyright violations
+    **EarwigBot: Wiki Toolset: Page**
+
+    Represents a page on a given :py:class:`~earwigbot.wiki.site.Site`. Has
+    methods for getting information about the page, getting page content, and
+    so on. :py:class:`~earwigbot.wiki.category.Category` is a subclass of
+    :py:class:`Page` with additional methods.
+
+    *Attributes:*
+
+    - :py:attr:`site`:        the page's corresponding Site object
+    - :py:attr:`title`:       the page's title, or pagename
+    - :py:attr:`exists`:      whether or not the page exists
+    - :py:attr:`pageid`:      an integer ID representing the page
+    - :py:attr:`url`:         the page's URL
+    - :py:attr:`namespace`:   the page's namespace as an integer
+    - :py:attr:`protection`:  the page's current protection status
+    - :py:attr:`is_talkpage`: ``True`` if this is a talkpage, else ``False``
+    - :py:attr:`is_redirect`: ``True`` if this is a redirect, else ``False``
+
+    *Public methods:*
+
+    - :py:meth:`reload`:      forcibly reloads the page's attributes
+    - :py:meth:`toggle_talk`: returns a content page's talk page, or vice versa
+    - :py:meth:`get`:         returns the page's content
+    - :py:meth:`get_redirect_target`: returns the page's destination if it is a
+      redirect
+    - :py:meth:`get_creator`: returns a User object representing the first
+      person to edit the page
+    - :py:meth:`parse`:       parses the page content for templates, links, etc
+    - :py:meth:`edit`:        replaces the page's content or creates a new page
+    - :py:meth:`add_section`: adds a new section at the bottom of the page
+    - :py:meth:`check_exclusion`: checks whether or not we are allowed to edit
+      the page, per ``{{bots}}``/``{{nobots}}``
+
+    - :py:meth:`~earwigbot.wiki.copyvios.CopyrightMixIn.copyvio_check`:
+      checks the page for copyright violations
+    - :py:meth:`~earwigbot.wiki.copyvios.CopyrightMixIn.copyvio_compare`:
+      checks the page like :py:meth:`copyvio_check`, but against a specific URL
     """
 
-    re_redirect = "^\s*\#\s*redirect\s*\[\[(.*?)\]\]"
+    PAGE_UNKNOWN = 0
+    PAGE_INVALID = 1
+    PAGE_MISSING = 2
+    PAGE_EXISTS = 3
 
-    def __init__(self, site, title, follow_redirects=False):
+    def __init__(self, site, title, follow_redirects=False, pageid=None):
         """Constructor for new Page instances.
 
-        Takes three arguments: a Site object, the Page's title (or pagename),
-        and whether or not to follow redirects (optional, defaults to False).
+        Takes four arguments: a Site object, the Page's title (or pagename),
+        whether or not to follow redirects (optional, defaults to False), and
+        a page ID to supplement the title (optional, defaults to None - i.e.,
+        we will have to query the API to get it).
 
-        As with User, site.get_page() is preferred. Site's method has support
-        for a default `follow_redirects` value in our config, while __init__
-        always defaults to False.
+        As with User, site.get_page() is preferred.
 
-        __init__ will not do any API queries, but it will use basic namespace
+        __init__() will not do any API queries, but it will use basic namespace
         logic to determine our namespace ID and if we are a talkpage.
         """
         super(Page, self).__init__(site)
         self._site = site
         self._title = title.strip()
         self._follow_redirects = self._keep_following = follow_redirects
+        self._pageid = pageid
 
-        self._exists = 0
-        self._pageid = None
+        self._exists = self.PAGE_UNKNOWN
         self._is_redirect = None
         self._lastrevid = None
         self._protection = None
@@ -93,8 +119,8 @@ class Page(CopyvioMixin):
         prefix = self._title.split(":", 1)[0]
         if prefix != title:  # ignore a page that's titled "Category" or "User"
             try:
-                self._namespace = self._site.namespace_name_to_id(prefix)
-            except NamespaceNotFoundError:
+                self._namespace = self.site.namespace_name_to_id(prefix)
+            except exceptions.NamespaceNotFoundError:
                 self._namespace = 0
         else:
             self._namespace = 0
@@ -107,15 +133,15 @@ class Page(CopyvioMixin):
             self._is_talkpage = self._namespace % 2 == 1
 
     def __repr__(self):
-        """Returns the canonical string representation of the Page."""
+        """Return the canonical string representation of the Page."""
         res = "Page(title={0!r}, follow_redirects={1!r}, site={2!r})"
         return res.format(self._title, self._follow_redirects, self._site)
 
     def __str__(self):
-        """Returns a nice string representation of the Page."""
-        return '<Page "{0}" of {1}>'.format(self.title(), str(self._site))
+        """Return a nice string representation of the Page."""
+        return '<Page "{0}" of {1}>'.format(self.title, str(self.site))
 
-    def _force_validity(self):
+    def _assert_validity(self):
         """Used to ensure that our page's title is valid.
 
         If this method is called when our page is not valid (and after
@@ -124,24 +150,24 @@ class Page(CopyvioMixin):
         Note that validity != existence. If a page's title is invalid (e.g, it
         contains "[") it will always be invalid, and cannot be edited.
         """
-        if self._exists == 1:
-            e = "Page '{0}' is invalid.".format(self._title)
-            raise InvalidPageError(e)
+        if self._exists == self.PAGE_INVALID:
+            e = u"Page '{0}' is invalid.".format(self._title)
+            raise exceptions.InvalidPageError(e)
 
-    def _force_existence(self):
+    def _assert_existence(self):
         """Used to ensure that our page exists.
 
         If this method is called when our page doesn't exist (and after
         _load_attributes() has been called), PageNotFoundError will be raised.
-        It will also call _force_validity() beforehand.
+        It will also call _assert_validity() beforehand.
         """
-        self._force_validity()
-        if self._exists == 2:
-            e = "Page '{0}' does not exist.".format(self._title)
-            raise PageNotFoundError(e)
+        self._assert_validity()
+        if self._exists == self.PAGE_MISSING:
+            e = u"Page '{0}' does not exist.".format(self._title)
+            raise exceptions.PageNotFoundError(e)
 
-    def _load_wrapper(self):
-        """Calls _load_attributes() and follows redirects if we're supposed to.
+    def _load(self):
+        """Call _load_attributes() and follows redirects if we're supposed to.
 
         This method will only follow redirects if follow_redirects=True was
         passed to __init__() (perhaps indirectly passed by site.get_page()).
@@ -164,21 +190,21 @@ class Page(CopyvioMixin):
             self._load_attributes()
 
     def _load_attributes(self, result=None):
-        """Loads various data from the API in a single query.
+        """Load various data from the API in a single query.
 
         Loads self._title, ._exists, ._is_redirect, ._pageid, ._fullurl,
         ._protection, ._namespace, ._is_talkpage, ._creator, ._lastrevid,
         ._token, and ._starttimestamp using the API. It will do a query of
-        its own unless `result` is provided, in which case we'll pretend
-        `result` is what the query returned.
+        its own unless *result* is provided, in which case we'll pretend
+        *result* is what the query returned.
 
         Assuming the API is sound, this should not raise any exceptions.
         """
-        if result is None:
-            params = {"action": "query", "rvprop": "user", "intoken": "edit",
-                      "prop": "info|revisions", "rvlimit": 1, "rvdir": "newer",
-                      "titles": self._title, "inprop": "protection|url"}
-            result = self._site._api_query(params)
+        if not result:
+            query = self.site.api_query
+            result = query(action="query", rvprop="user", intoken="edit",
+                           prop="info|revisions", rvlimit=1, rvdir="newer",
+                           titles=self._title, inprop="protection|url")
 
         res = result["query"]["pages"].values()[0]
 
@@ -192,21 +218,19 @@ class Page(CopyvioMixin):
         else:
             self._is_redirect = True
 
-        self._pageid = result["query"]["pages"].keys()[0]
-        if int(self._pageid) < 0:
-            try:
-                res["missing"]
-            except KeyError:
+        self._pageid = int(result["query"]["pages"].keys()[0])
+        if self._pageid < 0:
+            if "missing" in res:
+                # If it has a negative ID and it's missing; we can still get
+                # data like the namespace, protection, and URL:
+                self._exists = self.PAGE_MISSING
+            else:
                 # If it has a negative ID and it's invalid, then break here,
                 # because there's no other data for us to get:
-                self._exists = 1
+                self._exists = self.PAGE_INVALID
                 return
-            else:
-                # If it has a negative ID and it's missing; we can still get
-                # data like the namespace, protection, and URL:
-                self._exists = 2
         else:
-            self._exists = 3
+            self._exists = self.PAGE_EXISTS
 
         self._fullurl = res["fullurl"]
         self._protection = res["protection"]
@@ -231,19 +255,19 @@ class Page(CopyvioMixin):
             pass
 
     def _load_content(self, result=None):
-        """Loads current page content from the API.
+        """Load current page content from the API.
 
-        If `result` is provided, we'll pretend that is the result of an API
+        If *result* is provided, we'll pretend that is the result of an API
         query and try to get content from that. Otherwise, we'll do an API
         query on our own.
 
-        Don't call this directly, ever - use .get(force=True) if you want to
-        force content reloading.
+        Don't call this directly, ever; use reload() followed by get() if you
+        want to force content reloading.
         """
-        if result is None:
-            params = {"action": "query", "prop": "revisions", "rvlimit": 1,
-                      "rvprop": "content|timestamp", "titles": self._title}
-            result = self._site._api_query(params)
+        if not result:
+            query = self.site.api_query
+            result = query(action="query", prop="revisions", rvlimit=1,
+                           rvprop="content|timestamp", titles=self._title)
 
         res = result["query"]["pages"].values()[0]
         try:
@@ -254,14 +278,14 @@ class Page(CopyvioMixin):
             # self._load_attributes(). In that case, some of our attributes are
             # outdated, so force another self._load_attributes():
             self._load_attributes()
-            self._force_existence()
+            self._assert_existence()
 
     def _edit(self, params=None, text=None, summary=None, minor=None, bot=None,
               force=None, section=None, captcha_id=None, captcha_word=None,
               tries=0):
         """Edit the page!
 
-        If `params` is given, we'll use it as our API query parameters.
+        If *params* is given, we'll use it as our API query parameters.
         Otherwise, we'll build params using the given kwargs via
         _build_edit_params().
 
@@ -274,10 +298,10 @@ class Page(CopyvioMixin):
             self._load_attributes()
         if not self._token:
             e = "You don't have permission to edit this page."
-            raise PermissionsError(e)
+            raise exceptions.PermissionsError(e)
 
         # Weed out invalid pages before we get too far:
-        self._force_validity()
+        self._assert_validity()
 
         # Build our API query string:
         if not params:
@@ -288,8 +312,8 @@ class Page(CopyvioMixin):
 
         # Try the API query, catching most errors with our handler:
         try:
-            result = self._site._api_query(params)
-        except SiteAPIError as error:
+            result = self.site.api_query(**params)
+        except exceptions.APIError as error:
             if not hasattr(error, "code"):
                 raise  # We can only handle errors with a code attribute
             result = self._handle_edit_errors(error, params, tries)
@@ -298,7 +322,7 @@ class Page(CopyvioMixin):
         if result["edit"]["result"] == "Success":
             self._content = None
             self._basetimestamp = None
-            self._exists = 0
+            self._exists = self.PAGE_UNKNOWN
             return
 
         # If we're here, then the edit failed. If it's because of AssertEdit,
@@ -306,7 +330,7 @@ class Page(CopyvioMixin):
         try:
             assertion = result["edit"]["assert"]
         except KeyError:
-            raise EditError(result["edit"])
+            raise exceptions.EditError(result["edit"])
         self._handle_assert_edit(assertion, params, tries)
 
     def _build_edit_params(self, text, summary, minor, bot, force, section,
@@ -332,7 +356,7 @@ class Page(CopyvioMixin):
             params["starttimestamp"] = self._starttimestamp
             if self._basetimestamp:
                 params["basetimestamp"] = self._basetimestamp
-            if self._exists == 2:
+            if self._exists == self.PAGE_MISSING:
                 # Page does not exist; don't edit if it already exists:
                 params["createonly"] = "true"
         else:
@@ -349,43 +373,43 @@ class Page(CopyvioMixin):
         """
         if error.code in ["noedit", "cantcreate", "protectedtitle",
                           "noimageredirect"]:
-            raise PermissionsError(error.info)
+            raise exceptions.PermissionsError(error.info)
 
         elif error.code in ["noedit-anon", "cantcreate-anon",
                             "noimageredirect-anon"]:
-            if not all(self._site._login_info):
+            if not all(self.site._login_info):
                 # Insufficient login info:
-                raise PermissionsError(error.info)
+                raise exceptions.PermissionsError(error.info)
             if tries == 0:
                 # We have login info; try to login:
-                self._site._login(self._site._login_info)
+                self.site._login(self.site._login_info)
                 self._token = None  # Need a new token; old one is invalid now
                 return self._edit(params=params, tries=1)
             else:
                 # We already tried to log in and failed!
                 e = "Although we should be logged in, we are not. This may be a cookie problem or an odd bug."
-                raise LoginError(e)
+                raise exceptions.LoginError(e)
 
         elif error.code in ["editconflict", "pagedeleted", "articleexists"]:
             # These attributes are now invalidated:
             self._content = None
             self._basetimestamp = None
-            self._exists = 0
-            raise EditConflictError(error.info)
+            self._exists = self.PAGE_UNKNOWN
+            raise exceptions.EditConflictError(error.info)
 
         elif error.code in ["emptypage", "emptynewsection"]:
-            raise NoContentError(error.info)
+            raise exceptions.NoContentError(error.info)
 
         elif error.code == "contenttoobig":
-            raise ContentTooBigError(error.info)
+            raise exceptions.ContentTooBigError(error.info)
 
         elif error.code == "spamdetected":
-            raise SpamDetectedError(error.info)
+            raise exceptions.SpamDetectedError(error.info)
 
         elif error.code == "filtered":
-            raise FilteredError(error.info)
+            raise exceptions.FilteredError(error.info)
 
-        raise EditError(": ".join((error.code, error.info)))
+        raise exceptions.EditError(": ".join((error.code, error.info)))
 
     def _handle_assert_edit(self, assertion, params, tries):
         """If we can't edit due to a failed AssertEdit assertion, handle that.
@@ -394,179 +418,174 @@ class Page(CopyvioMixin):
         log in. Otherwise, raise PermissionsError with details.
         """
         if assertion == "user":
-            if not all(self._site._login_info):
+            if not all(self.site._login_info):
                 # Insufficient login info:
                 e = "AssertEdit: user assertion failed, and no login info was provided."
-                raise PermissionsError(e)
+                raise exceptions.PermissionsError(e)
             if tries == 0:
                 # We have login info; try to login:
-                self._site._login(self._site._login_info)
+                self.site._login(self.site._login_info)
                 self._token = None  # Need a new token; old one is invalid now
                 return self._edit(params=params, tries=1)
             else:
                 # We already tried to log in and failed!
                 e = "Although we should be logged in, we are not. This may be a cookie problem or an odd bug."
-                raise LoginError(e)
+                raise exceptions.LoginError(e)
 
         elif assertion == "bot":
             e = "AssertEdit: bot assertion failed; we don't have a bot flag!"
-            raise PermissionsError(e)
+            raise exceptions.PermissionsError(e)
 
         # Unknown assertion, maybe "true", "false", or "exists":
         e = "AssertEdit: assertion '{0}' failed.".format(assertion)
-        raise PermissionsError(e)
+        raise exceptions.PermissionsError(e)
+
+    @property
+    def site(self):
+        """The page's corresponding Site object."""
+        return self._site
 
-    def title(self, force=False):
-        """Returns the Page's title, or pagename.
+    @property
+    def title(self):
+        """The page's title, or "pagename".
 
-        This won't do any API queries on its own unless force is True, in which
-        case the title will be forcibly reloaded from the API (normalizing it,
-        and following redirects if follow_redirects=True was passed to
-        __init__()). Any other methods that do API queries will reload title on
-        their own, however, like exists() and get().
+        This won't do any API queries on its own. Any other attributes or
+        methods that do API queries will reload the title, however, like
+        :py:attr:`exists` and :py:meth:`get`, potentially "normalizing" it or
+        following redirects if :py:attr:`self._follow_redirects` is ``True``.
         """
-        if force:
-            self._load_wrapper()
         return self._title
 
-    def exists(self, force=False):
-        """Returns information about whether the Page exists or not.
+    @property
+    def exists(self):
+        """Whether or not the page exists.
 
-        The returned "information" is a tuple with two items. The first is a
-        bool, either True if the page exists or False if it does not. The
-        second is a string giving more information, either "invalid", (title
-        is invalid, e.g. it contains "["), "missing", or "exists".
+        This will be a number; its value does not matter, but it will equal
+        one of :py:attr:`self.PAGE_INVALID <PAGE_INVALID>`,
+        :py:attr:`self.PAGE_MISSING <PAGE_MISSING>`, or
+        :py:attr:`self.PAGE_EXISTS <PAGE_EXISTS>`.
 
-        Makes an API query if force is True or if we haven't already made one.
+        Makes an API query only if we haven't already made one.
         """
-        cases = {
-            0: (None, "unknown"),
-            1: (False, "invalid"),
-            2: (False, "missing"),
-            3: (True, "exists"),
-        }
-        if self._exists == 0 or force:
-            self._load_wrapper()
-        return cases[self._exists]
-
-    def pageid(self, force=False):
-        """Returns an integer ID representing the Page.
-
-        Makes an API query if force is True or if we haven't already made one.
-
-        Raises InvalidPageError or PageNotFoundError if the page name is
+        if self._exists == self.PAGE_UNKNOWN:
+            self._load()
+        return self._exists
+
+    @property
+    def pageid(self):
+        """An integer ID representing the page.
+
+        Makes an API query only if we haven't already made one and the *pageid*
+        parameter to :py:meth:`__init__` was left as ``None``, which should be
+        true for all cases except when pages are returned by an SQL generator
+        (like :py:meth:`category.get_members()
+        <earwigbot.wiki.category.Category.get_members>`).
+
+        Raises :py:exc:`~earwigbot.exceptions.InvalidPageError` or
+        :py:exc:`~earwigbot.exceptions.PageNotFoundError` if the page name is
         invalid or the page does not exist, respectively.
         """
-        if self._exists == 0 or force:
-            self._load_wrapper()
-        self._force_existence()  # missing pages do not have IDs
+        if self._pageid:
+            return self._pageid
+        if self._exists == self.PAGE_UNKNOWN:
+            self._load()
+        self._assert_existence()  # Missing pages do not have IDs
         return self._pageid
 
-    def url(self, force=False):
-        """Returns the page's URL.
+    @property
+    def url(self):
+        """The page's URL.
 
-        Like title(), this won't do any API queries on its own unless force is
-        True. If the API was never queried for this page, we will attempt to
-        determine the URL ourselves based on the title.
+        Like :py:meth:`title`, this won't do any API queries on its own. If the
+        API was never queried for this page, we will attempt to determine the
+        URL ourselves based on the title.
         """
-        if force:
-            self._load_wrapper()
-        if self._fullurl is not None:
+        if self._fullurl:
             return self._fullurl
         else:
-            slug = quote(self._title.replace(" ", "_"), safe="/:")
-            path = self._site._article_path.replace("$1", slug)
-            return ''.join((self._site._base_url, path))
-
-    def namespace(self, force=False):
-        """Returns the page's namespace ID (an integer).
-
-        Like title(), this won't do any API queries on its own unless force is
-        True. If the API was never queried for this page, we will attempt to
-        determine the namespace ourselves based on the title.
+            encoded = self._title.encode("utf8").replace(" ", "_")
+            slug = quote(encoded, safe="/:")
+            path = self.site._article_path.replace("$1", slug)
+            return ''.join((self.site.url, path))
+
+    @property
+    def namespace(self):
+        """The page's namespace ID (an integer).
+
+        Like :py:meth:`title`, this won't do any API queries on its own. If the
+        API was never queried for this page, we will attempt to determine the
+        namespace ourselves based on the title.
         """
-        if force:
-            self._load_wrapper()
         return self._namespace
 
-    def protection(self, force=False):
-        """Returns the page's current protection status.
+    @property
+    def protection(self):
+        """The page's current protection status.
 
-        Makes an API query if force is True or if we haven't already made one.
+        Makes an API query only if we haven't already made one.
 
-        Raises InvalidPageError if the page name is invalid. Will not raise an
-        error if the page is missing because those can still be protected.
+        Raises :py:exc:`~earwigbot.exceptions.InvalidPageError` if the page
+        name is invalid. Won't raise an error if the page is missing because
+        those can still be create-protected.
         """
-        if self._exists == 0 or force:
-            self._load_wrapper()
-        self._force_validity()  # invalid pages cannot be protected
+        if self._exists == self.PAGE_UNKNOWN:
+            self._load()
+        self._assert_validity()  # Invalid pages cannot be protected
         return self._protection
 
-    def creator(self, force=False):
-        """Returns the page's creator (i.e., the first user to edit the page).
-
-        Makes an API query if force is True or if we haven't already made one.
-        Normally, we can get the creator along with everything else (except
-        content) in self._load_attributes(). However, due to a limitation in
-        the API (can't get the editor of one revision and the content of
-        another at both ends of the history), if our other attributes were only
-        loaded from get(), we'll have to do another API query. This is done
-        by calling ourselves again with force=True.
+    @property
+    def is_talkpage(self):
+        """``True`` if the page is a talkpage, otherwise ``False``.
 
-        Raises InvalidPageError or PageNotFoundError if the page name is
-        invalid or the page does not exist, respectively.
+        Like :py:meth:`title`, this won't do any API queries on its own. If the
+        API was never queried for this page, we will attempt to determine
+        whether it is a talkpage ourselves based on its namespace.
         """
-        if self._exists == 0 or force:
-            self._load_wrapper()
-        self._force_existence()
-        if not self._creator and not force:
-            self.creator(force=True)
-        return self._creator
-
-    def is_talkpage(self, force=False):
-        """Returns True if the page is a talkpage, else False.
-
-        Like title(), this won't do any API queries on its own unless force is
-        True. If the API was never queried for this page, we will attempt to
-        determine the talkpage status ourselves based on its namespace ID.
-        """
-        if force:
-            self._load_wrapper()
         return self._is_talkpage
 
-    def is_redirect(self, force=False):
-        """Returns True if the page is a redirect, else False.
+    @property
+    def is_redirect(self):
+        """``True`` if the page is a redirect, otherwise ``False``.
 
-        Makes an API query if force is True or if we haven't already made one.
+        Makes an API query only if we haven't already made one.
 
-        We will return False even if the page does not exist or is invalid.
+        We will return ``False`` even if the page does not exist or is invalid.
         """
-        if self._exists == 0 or force:
-            self._load_wrapper()
+        if self._exists == self.PAGE_UNKNOWN:
+            self._load()
         return self._is_redirect
 
-    def toggle_talk(self, force=False, follow_redirects=None):
-        """Returns a content page's talk page, or vice versa.
+    def reload(self):
+        """Forcibly reload the page's attributes.
+
+        Emphasis on *reload*: this is only necessary if there is reason to
+        believe they have changed.
+        """
+        self._load()
+        if self._content is not None:
+            # Only reload content if it has already been loaded:
+            self._load_content()
+
+    def toggle_talk(self, follow_redirects=None):
+        """Return a content page's talk page, or vice versa.
 
         The title of the new page is determined by namespace logic, not API
-        queries. We won't make any API queries on our own unless force is True,
-        and the only reason then would be to forcibly update the title or
-        follow redirects if we haven't already made an API query.
+        queries. We won't make any API queries on our own.
 
-        If `follow_redirects` is anything other than None (the default), it
-        will be passed to the new Page's __init__(). Otherwise, we'll use the
-        value passed to our own __init__().
+        If *follow_redirects* is anything other than ``None`` (the default), it
+        will be passed to the new :py:class:`~earwigbot.wiki.page.Page`
+        object's :py:meth:`__init__`. Otherwise, we'll use the value passed to
+        our own :py:meth:`__init__`.
 
-        Will raise InvalidPageError if we try to get the talk page of a special
-        page (in the Special: or Media: namespaces), but we won't raise an
-        exception if our page is otherwise missing or invalid.
+        Will raise :py:exc:`~earwigbot.exceptions.InvalidPageError` if we try
+        to get the talk page of a special page (in the ``Special:`` or
+        ``Media:`` namespaces), but we won't raise an exception if our page is
+        otherwise missing or invalid.
         """
-        if force:
-            self._load_wrapper()
         if self._namespace < 0:
-            ns = self._site.namespace_id_to_name(self._namespace)
-            e = "Pages in the {0} namespace can't have talk pages.".format(ns)
-            raise InvalidPageError(e)
+            ns = self.site.namespace_id_to_name(self._namespace)
+            e = u"Pages in the {0} namespace can't have talk pages.".format(ns)
+            raise exceptions.InvalidPageError(e)
 
         if self._is_talkpage:
             new_ns = self._namespace - 1
@@ -578,85 +597,116 @@ class Page(CopyvioMixin):
         except IndexError:
             body = self._title
 
-        new_prefix = self._site.namespace_id_to_name(new_ns)
+        new_prefix = self.site.namespace_id_to_name(new_ns)
 
         # If the new page is in namespace 0, don't do ":Title" (it's correct,
         # but unnecessary), just do "Title":
         if new_prefix:
-            new_title = ':'.join((new_prefix, body))
+            new_title = u":".join((new_prefix, body))
         else:
             new_title = body
 
         if follow_redirects is None:
             follow_redirects = self._follow_redirects
-        return Page(self._site, new_title, follow_redirects)
+        return Page(self.site, new_title, follow_redirects)
 
-    def get(self, force=False):
-        """Returns page content, which is cached if you try to call get again.
-
-        Use `force` to forcibly reload page content even if we've already
-        loaded some. This is good if you want to edit a page multiple times,
-        and you want to get updated content before you make your second edit.
+    def get(self):
+        """Return page content, which is cached if you try to call get again.
 
         Raises InvalidPageError or PageNotFoundError if the page name is
         invalid or the page does not exist, respectively.
         """
-        if force or self._exists == 0:
+        if self._exists == self.PAGE_UNKNOWN:
             # Kill two birds with one stone by doing an API query for both our
             # attributes and our page content:
-            params = {"action": "query", "rvlimit": 1, "titles": self._title,
-                      "prop": "info|revisions", "inprop": "protection|url",
-                      "intoken": "edit", "rvprop": "content|timestamp"}
-            result = self._site._api_query(params)
+            query = self.site.api_query
+            result = query(action="query", rvlimit=1, titles=self._title,
+                           prop="info|revisions", inprop="protection|url",
+                           intoken="edit", rvprop="content|timestamp")
             self._load_attributes(result=result)
-            self._force_existence()
+            self._assert_existence()
             self._load_content(result=result)
 
             # Follow redirects if we're told to:
             if self._keep_following and self._is_redirect:
                 self._title = self.get_redirect_target()
-                self._keep_following = False  # don't follow double redirects
-                self._content = None  # reset the content we just loaded
-                self.get(force=True)
+                self._keep_following = False  # Don't follow double redirects
+                self._exists = self.PAGE_UNKNOWN  # Force another API query
+                self.get()
 
             return self._content
 
         # Make sure we're dealing with a real page here. This may be outdated
         # if the page was deleted since we last called self._load_attributes(),
         # but self._load_content() can handle that:
-        self._force_existence()
+        self._assert_existence()
 
         if self._content is None:
             self._load_content()
 
         return self._content
 
-    def get_redirect_target(self, force=False):
-        """If the page is a redirect, returns its destination.
-
-        Use `force` to forcibly reload content even if we've already loaded
-        some before. Note that this method calls get() for page content.
+    def get_redirect_target(self):
+        """If the page is a redirect, return its destination.
 
-        Raises InvalidPageError or PageNotFoundError if the page name is
-        invalid or the page does not exist, respectively. Raises RedirectError
-        if the page is not a redirect.
+        Raises :py:exc:`~earwigbot.exceptions.InvalidPageError` or
+        :py:exc:`~earwigbot.exceptions.PageNotFoundError` if the page name is
+        invalid or the page does not exist, respectively. Raises
+        :py:exc:`~earwigbot.exceptions.RedirectError` if the page is not a
+        redirect.
         """
-        content = self.get(force)
+        re_redirect = "^\s*\#\s*redirect\s*\[\[(.*?)\]\]"
+        content = self.get()
         try:
-            return re.findall(self.re_redirect, content, flags=re.I)[0]
+            return re.findall(re_redirect, content, flags=re.I)[0]
         except IndexError:
             e = "The page does not appear to have a redirect target."
-            raise RedirectError(e)
+            raise exceptions.RedirectError(e)
+
+    def get_creator(self):
+        """Return the User object for the first person to edit the page.
+
+        Makes an API query only if we haven't already made one. Normally, we
+        can get the creator along with everything else (except content) in
+        :py:meth:`_load_attributes`. However, due to a limitation in the API
+        (can't get the editor of one revision and the content of another at
+        both ends of the history), if our other attributes were only loaded
+        through :py:meth:`get`, we'll have to do another API query.
+
+        Raises :py:exc:`~earwigbot.exceptions.InvalidPageError` or
+        :py:exc:`~earwigbot.exceptions.PageNotFoundError` if the page name is
+        invalid or the page does not exist, respectively.
+        """
+        if self._exists == self.PAGE_UNKNOWN:
+            self._load()
+        self._assert_existence()
+        if not self._creator:
+            self._load()
+            self._assert_existence()
+        return self.site.get_user(self._creator)
+
+    def parse(self):
+        """Parse the page content for templates, links, etc.
+
+        Actual parsing is handled by :py:mod:`mwparserfromhell`. Raises
+        :py:exc:`ImportError` if :py:mod:`mwparserfromhell` isn't installed,
+        and :py:exc:`~earwigbot.exceptions.InvalidPageError` or
+        :py:exc:`~earwigbot.exceptions.PageNotFoundError` if the page name is
+        invalid or the page does not exist, respectively.
+        """
+        if not mwparserfromhell:
+            raise ImportError("mwparserfromhell")
+        return mwparserfromhell.parse(self.get())
 
     def edit(self, text, summary, minor=False, bot=True, force=False):
-        """Replaces the page's content or creates a new page.
+        """Replace the page's content or creates a new page.
 
-        `text` is the new page content, with `summary` as the edit summary.
-        If `minor` is True, the edit will be marked as minor. If `bot` is true,
-        the edit will be marked as a bot edit, but only if we actually have a
-        bot flag.
+        *text* is the new page content, with *summary* as the edit summary.
+        If *minor* is ``True``, the edit will be marked as minor. If *bot* is
+        ``True``, the edit will be marked as a bot edit, but only if we
+        actually have a bot flag.
 
-        Use `force` to push the new content even if there's an edit conflict or
+        Use *force* to push the new content even if there's an edit conflict or
         the page was deleted/recreated between getting our edit token and
         editing our page. Be careful with this!
         """
@@ -664,15 +714,66 @@ class Page(CopyvioMixin):
                    force=force)
 
     def add_section(self, text, title, minor=False, bot=True, force=False):
-        """Adds a new section to the bottom of the page.
-
-        The arguments for this are the same as those for edit(), but instead of
-        providing a summary, you provide a section title.
+        """Add a new section to the bottom of the page.
 
-        Likewise, raised exceptions are the same as edit()'s.
+        The arguments for this are the same as those for :py:meth:`edit`, but
+        instead of providing a summary, you provide a section title. Likewise,
+        raised exceptions are the same as :py:meth:`edit`'s.
 
         This should create the page if it does not already exist, with just the
         new section as content.
         """
         self._edit(text=text, summary=title, minor=minor, bot=bot, force=force,
                    section="new")
+
+    def check_exclusion(self, username=None, optouts=None):
+        """Check whether or not we are allowed to edit the page.
+
+        Return ``True`` if we *are* allowed to edit this page, and ``False`` if
+        we aren't.
+
+        *username* is used to determine whether we are part of a specific list
+        of allowed or disallowed bots (e.g. ``{{bots|allow=EarwigBot}}`` or
+        ``{{bots|deny=FooBot,EarwigBot}}``). It's ``None`` by default, which
+        will swipe our username from :py:meth:`site.get_user()
+        <earwigbot.wiki.site.Site.get_user>`.\
+        :py:attr:`~earwigbot.wiki.user.User.name`.
+
+        *optouts* is a list of messages to consider this check as part of for
+        the purpose of opt-out; it defaults to ``None``, which ignores the
+        parameter completely. For example, if *optouts* is ``["nolicense"]``,
+        we'll return ``False`` on ``{{bots|optout=nolicense}}`` or
+        ``{{bots|optout=all}}``, but `True` on
+        ``{{bots|optout=orfud,norationale,replaceable}}``.
+        """
+        def parse_param(template, param):
+            value = template.get_param(param).value
+            return [item.strip().lower() for item in value.split(",")]
+
+        if not username:
+            username = self.site.get_user().name
+
+        # Lowercase everything:
+        username = username.lower()
+        optouts = [optout.lower() for optout in optouts] if optouts else []
+
+        re_bots = "\{\{\s*(no)?bots\s*(\||\}\})"
+        filter = self.parse().filter_templates(matches=re_bots, recursive=True)
+        for template in filter:
+            if template.has_param("deny"):
+                denies = parse_param(template, "deny")
+                if "all" in denies or username in denies:
+                    return False
+            if template.has_param("allow"):
+                allows = parse_param(template, "allow")
+                if "all" in allows or username in allows:
+                    continue
+            if optouts and template.has_param("optout"):
+                tasks = parse_param(template, "optout")
+                matches = [optout in tasks for optout in optouts]
+                if "all" in tasks or any(matches):
+                    return False
+            if template.name.strip().lower() == "nobots":
+                return False
+
+        return True

earwigbot/wiki/site.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,11 +23,13 @@
 from cookielib import CookieJar
 from gzip import GzipFile
 from json import loads
+from logging import getLogger, NullHandler
 from os.path import expanduser
 from re import escape as re_escape, match as re_match
 from StringIO import StringIO
-from time import sleep
-from urllib import unquote_plus, urlencode
+from threading import Lock
+from time import sleep, time
+from urllib import quote_plus
 from urllib2 import build_opener, HTTPCookieProcessor, URLError
 from urlparse import urlparse
 
@@ -36,53 +38,72 @@ try:
 except ImportError:
     oursql = None
 
-from earwigbot.wiki import logger
+from earwigbot import exceptions
+from earwigbot.wiki import constants
 from earwigbot.wiki.category import Category
-from earwigbot.wiki.constants import *
-from earwigbot.wiki.exceptions import *
 from earwigbot.wiki.page import Page
 from earwigbot.wiki.user import User
 
+__all__ = ["Site"]
+
 class Site(object):
     """
-    EarwigBot's Wiki Toolset: Site Class
-
-    Represents a Site, with support for API queries and returning Pages, Users,
-    and Categories. The constructor takes a bunch of arguments and you probably
-    won't need to call it directly, rather tools.get_site() for returning Site
-    instances, tools.add_site() for adding new ones to config, and
-    tools.del_site() for removing old ones from config, should suffice.
-
-    Public methods:
-    name                 -- returns our name (or "wikiid"), like "enwiki"
-    project              -- returns our project name, like "wikipedia"
-    lang                 -- returns our language code, like "en"
-    domain               -- returns our web domain, like "en.wikipedia.org"
-    api_query            -- does an API query with the given kwargs as params
-    sql_query            -- does an SQL query and yields its results
-    get_replag           -- returns the estimated database replication lag
-    namespace_id_to_name -- given a namespace ID, returns associated name(s)
-    namespace_name_to_id -- given a namespace name, returns associated id
-    get_page             -- returns a Page object for the given title
-    get_category         -- returns a Category object for the given title
-    get_user             -- returns a User object for the given username
+    **EarwigBot: Wiki Toolset: Site**
+
+    Represents a site, with support for API queries and returning
+    :py:class:`~earwigbot.wiki.page.Page`,
+    :py:class:`~earwigbot.wiki.user.User`,
+    and :py:class:`~earwigbot.wiki.category.Category` objects. The constructor
+    takes a bunch of arguments and you probably won't need to call it directly,
+    rather :py:meth:`wiki.get_site() <earwigbot.wiki.sitesdb.SitesDB.get_site>`
+    for returning :py:class:`Site`
+    instances, :py:meth:`wiki.add_site()
+    <earwigbot.wiki.sitesdb.SitesDB.add_site>` for adding new ones to our
+    database, and :py:meth:`wiki.remove_site()
+    <earwigbot.wiki.sitesdb.SitesDB.remove_site>` for removing old ones from
+    our database, should suffice.
+
+    *Attributes:*
+
+    - :py:attr:`name`:    the site's name (or "wikiid"), like ``"enwiki"``
+    - :py:attr:`project`: the site's project name, like ``"wikipedia"``
+    - :py:attr:`lang`:    the site's language code, like ``"en"``
+    - :py:attr:`domain`:  the site's web domain, like ``"en.wikipedia.org"``
+    - :py:attr:`url`:     the site's URL, like ``"https://en.wikipedia.org"``
+
+    *Public methods:*
+
+    - :py:meth:`api_query`:            does an API query with kwargs as params
+    - :py:meth:`sql_query`:            does an SQL query and yields its results
+    - :py:meth:`get_maxlag`:           returns the internal database lag
+    - :py:meth:`get_replag`:           estimates the external database lag
+    - :py:meth:`namespace_id_to_name`: returns names associated with an NS id
+    - :py:meth:`namespace_name_to_id`: returns the ID associated with a NS name
+    - :py:meth:`get_page`:             returns a Page for the given title
+    - :py:meth:`get_category`:         returns a Category for the given title
+    - :py:meth:`get_user`:             returns a User object for the given name
+    - :py:meth:`delegate`:             controls when the API or SQL is used
     """
+    SERVICE_API = 1
+    SERVICE_SQL = 2
 
     def __init__(self, name=None, project=None, lang=None, base_url=None,
                  article_path=None, script_path=None, sql=None,
                  namespaces=None, login=(None, None), cookiejar=None,
-                 user_agent=None, assert_edit=None, maxlag=None,
+                 user_agent=None, use_https=False, assert_edit=None,
+                 maxlag=None, wait_between_queries=3, logger=None,
                  search_config=(None, None)):
         """Constructor for new Site instances.
 
         This probably isn't necessary to call yourself unless you're building a
         Site that's not in your config and you don't want to add it - normally
-        all you need is tools.get_site(name), which creates the Site for you
-        based on your config file. We accept a bunch of kwargs, but the only
-        ones you really "need" are `base_url` and `script_path` - this is
-        enough to figure out an API url. `login`, a tuple of
-        (username, password), is highly recommended. `cookiejar` will be used
-        to store cookies, and we'll use a normal CookieJar if none is given.
+        all you need is wiki.get_site(name), which creates the Site for you
+        based on your config file and the sites database. We accept a bunch of
+        kwargs, but the only ones you really "need" are *base_url* and
+        *script_path*; this is enough to figure out an API url. *login*, a
+        tuple of (username, password), is highly recommended. *cookiejar will
+        be used to store cookies, and we'll use a normal CookieJar if none is
+        given.
 
         First, we'll store the given arguments as attributes, then set up our
         URL opener. We'll load any of the attributes that weren't given from
@@ -99,25 +120,32 @@ class Site(object):
         self._script_path = script_path
         self._namespaces = namespaces
 
-        # Attributes used for API queries: 
+        # Attributes used for API queries:
+        self._use_https = use_https
         self._assert_edit = assert_edit
         self._maxlag = maxlag
-        self._max_retries = 5
+        self._wait_between_queries = wait_between_queries
+        self._max_retries = 6
+        self._last_query_time = 0
+        self._api_lock = Lock()
+        self._api_info_cache = {"maxlag": 0, "lastcheck": 0}
 
         # Attributes used for SQL queries:
         self._sql_data = sql
         self._sql_conn = None
+        self._sql_lock = Lock()
+        self._sql_info_cache = {"replag": 0, "lastcheck": 0, "usable": None}
 
-        # Attribute used in copyright violation checks (see CopyrightMixin):
+        # Attribute used in copyright violation checks (see CopyrightMixIn):
         self._search_config = search_config
 
         # Set up cookiejar and URL opener for making API queries:
-        if cookiejar is not None:
+        if cookiejar:
             self._cookiejar = cookiejar
         else:
             self._cookiejar = CookieJar()
-        if user_agent is None:
-            user_agent = USER_AGENT  # Set default UA from wiki.constants
+        if not user_agent:
+            user_agent = constants.USER_AGENT  # Set default UA
         self._opener = build_opener(HTTPCookieProcessor(self._cookiejar))
         self._opener.addheaders = [("User-Agent", user_agent),
                                    ("Accept-Encoding", "gzip")]
@@ -125,22 +153,29 @@ class Site(object):
         # Get all of the above attributes that were not specified as arguments:
         self._load_attributes()
 
+        # Set up our internal logger:
+        if logger:
+            self._logger = logger
+        else:  # Just set up a null logger to eat up our messages:
+            self._logger = getLogger("earwigbot.wiki")
+            self._logger.addHandler(NullHandler())
+
         # If we have a name/pass and the API says we're not logged in, log in:
         self._login_info = name, password = login
-        if name is not None and password is not None:
+        if name and password:
             logged_in_as = self._get_username_from_cookies()
-            if logged_in_as is None or name != logged_in_as:
+            if not logged_in_as or name != logged_in_as:
                 self._login(login)
 
     def __repr__(self):
-        """Returns the canonical string representation of the Site."""
+        """Return the canonical string representation of the Site."""
         res = ", ".join((
             "Site(name={_name!r}", "project={_project!r}", "lang={_lang!r}",
             "base_url={_base_url!r}", "article_path={_article_path!r}",
-            "script_path={_script_path!r}", "assert_edit={_assert_edit!r}",
-            "maxlag={_maxlag!r}", "sql={_sql!r}", "login={0}",
-            "user_agent={2!r}", "cookiejar={1})"
-        ))
+            "script_path={_script_path!r}", "use_https={_use_https!r}",
+            "assert_edit={_assert_edit!r}", "maxlag={_maxlag!r}",
+            "sql={_sql_data!r}", "login={0}", "user_agent={2!r}",
+            "cookiejar={1})"))
         name, password = self._login_info
         login = "({0}, {1})".format(repr(name), "hidden" if password else None)
         cookies = self._cookiejar.__class__.__name__
@@ -152,49 +187,45 @@ class Site(object):
         return res.format(login, cookies, agent, **self.__dict__)
 
     def __str__(self):
-        """Returns a nice string representation of the Site."""
+        """Return a nice string representation of the Site."""
         res = "<Site {0} ({1}:{2}) at {3}>"
-        return res.format(self.name(), self.project(), self.lang(),
-                          self.domain())
-
-    def _api_query(self, params, tries=0, wait=5):
-        """Do an API query with `params` as a dict of parameters.
-
-        This will first attempt to construct an API url from self._base_url and
-        self._script_path. We need both of these, or else we'll raise
-        SiteAPIError.
-
-        We'll encode the given params, adding format=json along the way, as
-        well as &assert= and &maxlag= based on self._assert_edit and _maxlag.
-        We make the request through self._opener, which has built-in cookie
-        support via self._cookiejar, a User-Agent (wiki.constants.USER_AGENT),
-        and Accept-Encoding set to "gzip".
-
-        Assuming everything went well, we'll gunzip the data (if compressed),
-        load it as a JSON object, and return it.
-
-        If our request failed for some reason, we'll raise SiteAPIError with
-        details. If that reason was due to maxlag, we'll sleep for a bit and
-        then repeat the query until we exceed self._max_retries.
-
-        There's helpful MediaWiki API documentation at
-        <http://www.mediawiki.org/wiki/API>.
+        return res.format(self.name, self.project, self.lang, self.domain)
+
+    def _unicodeify(self, value, encoding="utf8"):
+        """Return input as unicode if it's not unicode to begin with."""
+        if isinstance(value, unicode):
+            return value
+        return unicode(value, encoding)
+
+    def _urlencode_utf8(self, params):
+        """Implement urllib.urlencode() with support for unicode input."""
+        enc = lambda s: s.encode("utf8") if isinstance(s, unicode) else str(s)
+        args = []
+        for key, val in params.iteritems():
+            key = quote_plus(enc(key))
+            val = quote_plus(enc(val))
+            args.append(key + "=" + val)
+        return "&".join(args)
+
+    def _api_query(self, params, tries=0, wait=5, ignore_maxlag=False):
+        """Do an API query with *params* as a dict of parameters.
+
+        See the documentation for :py:meth:`api_query` for full implementation
+        details.
         """
-        if self._base_url is None or self._script_path is None:
-            e = "Tried to do an API query, but no API URL is known."
-            raise SiteAPIError(e)
-
-        url = ''.join((self._base_url, self._script_path, "/api.php"))
-
-        params["format"] = "json"  # This is the only format we understand
-        if self._assert_edit:  # If requested, ensure that we're logged in
-            params["assert"] = self._assert_edit
-        if self._maxlag:  # If requested, don't overload the servers
-            params["maxlag"] = self._maxlag
-
-        data = urlencode(params)
-
-        logger.debug("{0} -> {1}".format(url, data))
+        since_last_query = time() - self._last_query_time  # Throttling support
+        if since_last_query < self._wait_between_queries:
+            wait_time = self._wait_between_queries - since_last_query
+            log = "Throttled: waiting {0} seconds".format(round(wait_time, 2))
+            self._logger.debug(log)
+            sleep(wait_time)
+        self._last_query_time = time()
+
+        url, data = self._build_api_query(params, ignore_maxlag)
+        if "lgpassword" in params:
+            self._logger.debug("{0} -> <hidden>".format(url))
+        else:
+            self._logger.debug("{0} -> {1}".format(url, data))
 
         try:
             response = self._opener.open(url, data)
@@ -206,7 +237,7 @@ class Site(object):
                 e = e.format(error.code)
             else:
                 e = "API query failed."
-            raise SiteAPIError(e)
+            raise exceptions.APIError(e)
 
         result = response.read()
         if response.headers.get("Content-Encoding") == "gzip":
@@ -214,30 +245,51 @@ class Site(object):
             gzipper = GzipFile(fileobj=stream)
             result = gzipper.read()
 
+        return self._handle_api_query_result(result, params, tries, wait)
+
+    def _build_api_query(self, params, ignore_maxlag):
+        """Given API query params, return the URL to query and POST data."""
+        if not self._base_url or self._script_path is None:
+            e = "Tried to do an API query, but no API URL is known."
+            raise exceptions.APIError(e)
+
+        url = ''.join((self.url, self._script_path, "/api.php"))
+        params["format"] = "json"  # This is the only format we understand
+        if self._assert_edit:  # If requested, ensure that we're logged in
+            params["assert"] = self._assert_edit
+        if self._maxlag and not ignore_maxlag:
+            # If requested, don't overload the servers:
+            params["maxlag"] = self._maxlag
+
+        data = self._urlencode_utf8(params)
+        return url, data
+
+    def _handle_api_query_result(self, result, params, tries, wait):
+        """Given the result of an API query, attempt to return useful data."""
         try:
-            res = loads(result)  # Parse as a JSON object
+            res = loads(result)  # Try to parse as a JSON object
         except ValueError:
             e = "API query failed: JSON could not be decoded."
-            raise SiteAPIError(e)
+            raise exceptions.APIError(e)
 
         try:
             code = res["error"]["code"]
             info = res["error"]["info"]
-        except (TypeError, KeyError):
-            return res
+        except (TypeError, KeyError):  # Having these keys indicates a problem
+            return res  # All is well; return the decoded JSON
 
-        if code == "maxlag":
+        if code == "maxlag":  # We've been throttled by the server
             if tries >= self._max_retries:
                 e = "Maximum number of retries reached ({0})."
-                raise SiteAPIError(e.format(self._max_retries))
+                raise exceptions.APIError(e.format(self._max_retries))
             tries += 1
-            msg = 'Server says: "{0}". Retrying in {1} seconds ({2}/{3}).'
-            logger.info(msg.format(info, wait, tries, self._max_retries))
+            msg = 'Server says "{0}"; retrying in {1} seconds ({2}/{3})'
+            self._logger.info(msg.format(info, wait, tries, self._max_retries))
             sleep(wait)
-            return self._api_query(params, tries=tries, wait=wait*3)
-        else:
+            return self._api_query(params, tries=tries, wait=wait*2)
+        else:  # Some unknown error occurred
             e = 'API query failed: got error "{0}"; server says: "{1}".'
-            error = SiteAPIError(e.format(code, info))
+            error = exceptions.APIError(e.format(code, info))
             error.code, error.info = code, info
             raise error
 
@@ -248,25 +300,24 @@ class Site(object):
         was not given as a keyword argument. We'll do an API query to get the
         missing data, but only if there actually *is* missing data.
 
-        Additionally, you can call this with `force=True` to forcibly reload
-        all attributes.
+        Additionally, you can call this with *force* set to True to forcibly
+        reload all attributes.
         """
         # All attributes to be loaded, except _namespaces, which is a special
         # case because it requires additional params in the API query:
         attrs = [self._name, self._project, self._lang, self._base_url,
             self._article_path, self._script_path]
 
-        params = {"action": "query", "meta": "siteinfo"}
+        params = {"action": "query", "meta": "siteinfo", "siprop": "general"}
 
         if not self._namespaces or force:
-            params["siprop"] = "general|namespaces|namespacealiases"
-            result = self._api_query(params)
+            params["siprop"] += "|namespaces|namespacealiases"
+            result = self.api_query(**params)
             self._load_namespaces(result)
         elif all(attrs):  # Everything is already specified and we're not told
             return        # to force a reload, so do nothing
         else:  # We're only loading attributes other than _namespaces
-            params["siprop"] = "general"
-            result = self._api_query(params)
+            result = self.api_query(**params)
 
         res = result["query"]["general"]
         self._name = res["wikiid"]
@@ -279,7 +330,7 @@ class Site(object):
     def _load_namespaces(self, result):
         """Fill self._namespaces with a dict of namespace IDs and names.
 
-        Called by _load_attributes() with API data as `result` when
+        Called by _load_attributes() with API data as *result* when
         self._namespaces was not given as an kwarg to __init__().
         """
         self._namespaces = {}
@@ -326,44 +377,42 @@ class Site(object):
         If we didn't get any matches, we'll return None. Our goal here isn't to
         return the most likely username, or what we *want* our username to be
         (for that, we'd do self._login_info[0]), but rather to get our current
-        username without an unnecessary ?action=query&meta=userinfo API query. 
+        username without an unnecessary ?action=query&meta=userinfo API query.
         """
-        domain = self.domain()
         name = ''.join((self._name, "Token"))
-        cookie = self._get_cookie(name, domain)
+        cookie = self._get_cookie(name, self.domain)
 
-        if cookie is not None:
+        if cookie:
             name = ''.join((self._name, "UserName"))
-            user_name = self._get_cookie(name, domain)
-            if user_name is not None:
+            user_name = self._get_cookie(name, self.domain)
+            if user_name:
                 return user_name.value
 
         name = "centralauth_Token"
-        for cookie in self._cookiejar:            
-            if cookie.domain_initial_dot is False or cookie.is_expired():
+        for cookie in self._cookiejar:
+            if not cookie.domain_initial_dot or cookie.is_expired():
                 continue
             if cookie.name != name:
                 continue
             # Build a regex that will match domains this cookie affects:
             search = ''.join(("(.*?)", re_escape(cookie.domain)))
-            if re_match(search, domain):  # Test it against our site
+            if re_match(search, self.domain):  # Test it against our site
                 user_name = self._get_cookie("centralauth_User", cookie.domain)
-                if user_name is not None:
+                if user_name:
                     return user_name.value
 
     def _get_username_from_api(self):
         """Do a simple API query to get our username and return it.
-        
+
         This is a reliable way to make sure we are actually logged in, because
         it doesn't deal with annoying cookie logic, but it results in an API
         query that is unnecessary in some cases.
-        
+
         Called by _get_username() (in turn called by get_user() with no
         username argument) when cookie lookup fails, probably indicating that
         we are logged out.
         """
-        params = {"action": "query", "meta": "userinfo"}
-        result = self._api_query(params)
+        result = self.api_query(action="query", meta="userinfo")
         return result["query"]["userinfo"]["name"]
 
     def _get_username(self):
@@ -378,7 +427,7 @@ class Site(object):
         single API query for our username (or IP address) and return that.
         """
         name = self._get_username_from_cookies()
-        if name is not None:
+        if name:
             return name
         return self._get_username_from_api()
 
@@ -411,17 +460,19 @@ class Site(object):
         Raises LoginError on login errors (duh), like bad passwords and
         nonexistent usernames.
 
-        `login` is a (username, password) tuple. `token` is the token returned
-        from our first request, and `attempt` is to prevent getting stuck in a
+        *login* is a (username, password) tuple. *token* is the token returned
+        from our first request, and *attempt* is to prevent getting stuck in a
         loop if MediaWiki isn't acting right.
         """
         name, password = login
-        params = {"action": "login", "lgname": name, "lgpassword": password}
-        if token is not None:
-            params["lgtoken"] = token
-        result = self._api_query(params)
-        res = result["login"]["result"]
+        if token:
+            result = self.api_query(action="login", lgname=name,
+                                    lgpassword=password, lgtoken=token)
+        else:
+            result = self.api_query(action="login", lgname=name,
+                                    lgpassword=password)
 
+        res = result["login"]["result"]
         if res == "Success":
             self._save_cookiejar()
         elif res == "NeedToken" and attempt == 0:
@@ -438,7 +489,7 @@ class Site(object):
                 e = "The given password is incorrect."
             else:
                 e = "Couldn't login; server says '{0}'.".format(res)
-            raise LoginError(e)
+            raise exceptions.LoginError(e)
 
     def _logout(self):
         """Safely logout through the API.
@@ -447,18 +498,16 @@ class Site(object):
         cookiejar (which probably contains now-invalidated cookies) and try to
         save it, if it supports that sort of thing.
         """
-        params = {"action": "logout"}
-        self._api_query(params)
+        self.api_query(action="logout")
         self._cookiejar.clear()
         self._save_cookiejar()
 
     def _sql_connect(self, **kwargs):
         """Attempt to establish a connection with this site's SQL database.
 
-        oursql.connect() will be called with self._sql_data as its kwargs,
-        which is usually config.wiki["sites"][self.name()]["sql"]. Any kwargs
-        given to this function will be passed to connect() and will have
-        precedence over the config file.
+        oursql.connect() will be called with self._sql_data as its kwargs.
+        Any kwargs given to this function will be passed to connect() and will
+        have precedence over the config file.
 
         Will raise SQLError() if the module "oursql" is not available. oursql
         may raise its own exceptions (e.g. oursql.InterfaceError) if it cannot
@@ -466,7 +515,7 @@ class Site(object):
         """
         if not oursql:
             e = "Module 'oursql' is required for SQL queries."
-            raise SQLError(e)
+            raise exceptions.SQLError(e)
 
         args = self._sql_data
         for key, value in kwargs.iteritems():
@@ -475,85 +524,206 @@ class Site(object):
         if "read_default_file" not in args and "user" not in args and "passwd" not in args:
             args["read_default_file"] = expanduser("~/.my.cnf")
 
+        if "autoping" not in args:
+            args["autoping"] = True
+
+        if "autoreconnect" not in args:
+            args["autoreconnect"] = True
+
         self._sql_conn = oursql.connect(**args)
 
+    def _get_service_order(self):
+        """Return a preferred order for using services (e.g. the API and SQL).
+
+        A list is returned, starting with the most preferred service first and
+        ending with the least preferred one. Currently, there are only two
+        services. SERVICE_API will always be included since the API is expected
+        to be always usable. In normal circumstances, self.SERVICE_SQL will be
+        first (with the API second), since using SQL directly is easier on the
+        servers than making web queries with the API. self.SERVICE_SQL will be
+        second if replag is greater than three minutes (a cached value updated
+        every two minutes at most), *unless* API lag is also very high.
+        self.SERVICE_SQL will not be included in the list if we cannot form a
+        proper SQL connection.
+        """
+        now = time()
+        if now - self._sql_info_cache["lastcheck"] > 120:
+            self._sql_info_cache["lastcheck"] = now
+            try:
+                self._sql_info_cache["replag"] = sqllag = self.get_replag()
+            except (exceptions.SQLError, oursql.Error):
+                self._sql_info_cache["usable"] = False
+                return [self.SERVICE_API]
+            self._sql_info_cache["usable"] = True
+        else:
+            if not self._sql_info_cache["usable"]:
+                return [self.SERVICE_API]
+            sqllag = self._sql_info_cache["replag"]
+
+        if sqllag > 180:
+            if not self._maxlag:
+                return [self.SERVICE_API, self.SERVICE_SQL]
+            if now - self._api_info_cache["lastcheck"] > 120:
+                self._api_info_cache["lastcheck"] = now
+                try:
+                    self._api_info_cache["maxlag"] = apilag = self.get_maxlag()
+                except exceptions.APIError:
+                    self._api_info_cache["maxlag"] = apilag = 0
+            else:
+                apilag = self._api_info_cache["maxlag"]
+            if sqllag / (180.0 / self._maxlag) < apilag:
+                return [self.SERVICE_SQL, self.SERVICE_API]
+            return [self.SERVICE_API, self.SERVICE_SQL]
+
+        return [self.SERVICE_SQL, self.SERVICE_API]
+
+    @property
     def name(self):
-        """Returns the Site's name (or "wikiid" in the API), like "enwiki"."""
+        """The Site's name (or "wikiid" in the API), like ``"enwiki"``."""
         return self._name
 
+    @property
     def project(self):
-        """Returns the Site's project name in lowercase, like "wikipedia"."""
+        """The Site's project name in lowercase, like ``"wikipedia"``."""
         return self._project
 
+    @property
     def lang(self):
-        """Returns the Site's language code, like "en" or "es"."""
+        """The Site's language code, like ``"en"`` or ``"es"``."""
         return self._lang
 
+    @property
     def domain(self):
-        """Returns the Site's web domain, like "en.wikipedia.org"."""
+        """The Site's web domain, like ``"en.wikipedia.org"``."""
         return urlparse(self._base_url).netloc
 
+    @property
+    def url(self):
+        """The Site's full base URL, like ``"https://en.wikipedia.org"``."""
+        url = self._base_url
+        if url.startswith("//"):  # Protocol-relative URLs from 1.18
+            if self._use_https:
+                url = "https:" + url
+            else:
+                url = "http:" + url
+        return url
+
     def api_query(self, **kwargs):
         """Do an API query with `kwargs` as the parameters.
 
-        See _api_query()'s documentation for details.
+        This will first attempt to construct an API url from
+        :py:attr:`self._base_url` and :py:attr:`self._script_path`. We need
+        both of these, or else we'll raise
+        :py:exc:`~earwigbot.exceptions.APIError`. If
+        :py:attr:`self._base_url` is protocol-relative (introduced in MediaWiki
+        1.18), we'll choose HTTPS only if :py:attr:`self._user_https` is
+        ``True``, otherwise HTTP.
+
+        We'll encode the given params, adding ``format=json`` along the way, as
+        well as ``&assert=`` and ``&maxlag=`` based on
+        :py:attr:`self._assert_edit` and :py:attr:`_maxlag` respectively.
+        Additionally, we'll sleep a bit if the last query was made fewer than
+        :py:attr:`self._wait_between_queries` seconds ago. The request is made
+        through :py:attr:`self._opener`, which has cookie support
+        (:py:attr:`self._cookiejar`), a ``User-Agent``
+        (:py:const:`earwigbot.wiki.constants.USER_AGENT`), and
+        ``Accept-Encoding`` set to ``"gzip"``.
+
+        Assuming everything went well, we'll gunzip the data (if compressed),
+        load it as a JSON object, and return it.
+
+        If our request failed for some reason, we'll raise
+        :py:exc:`~earwigbot.exceptions.APIError` with details. If that
+        reason was due to maxlag, we'll sleep for a bit and then repeat the
+        query until we exceed :py:attr:`self._max_retries`.
+
+        There is helpful MediaWiki API documentation at `MediaWiki.org
+        <http://www.mediawiki.org/wiki/API>`_.
         """
-        return self._api_query(kwargs)
+        with self._api_lock:
+            return self._api_query(kwargs)
 
     def sql_query(self, query, params=(), plain_query=False, dict_cursor=False,
                   cursor_class=None, show_table=False):
         """Do an SQL query and yield its results.
 
-        If `plain_query` is True, we will force an unparameterized query.
-        Specifying both params and plain_query will cause an error.
-
-        If `dict_cursor` is True, we will use oursql.DictCursor as our cursor,
-        otherwise the default oursql.Cursor. If `cursor_class` is given, it
-        will override this option.
-
-        If `show_table` is True, the name of the table will be prepended to the
-        name of the column. This will mainly affect a DictCursor.
-
-        Example:
-        >>> query = "SELECT user_id, user_registration FROM user WHERE user_name = ?"
-        >>> params = ("The Earwig",)
-        >>> result1 = site.sql_query(query, params)
-        >>> result2 = site.sql_query(query, params, dict_cursor=True)
-        >>> for row in result1: print row
-        (7418060L, '20080703215134')
-        >>> for row in result2: print row
-        {'user_id': 7418060L, 'user_registration': '20080703215134'}
-
-        See _sql_connect() for information on how a connection is acquired.
-
-        <http://packages.python.org/oursql> has helpful documentation on the
-        oursql module.
-
-        This may raise SQLError() or one of oursql's exceptions
-        (oursql.ProgrammingError, oursql.InterfaceError, ...) if there were
-        problems with the query.
+        If *plain_query* is ``True``, we will force an unparameterized query.
+        Specifying both *params* and *plain_query* will cause an error. If
+        *dict_cursor* is ``True``, we will use :py:class:`oursql.DictCursor` as
+        our cursor, otherwise the default :py:class:`oursql.Cursor`. If
+        *cursor_class* is given, it will override this option. If *show_table*
+        is True, the name of the table will be prepended to the name of the
+        column. This will mainly affect an :py:class:`~oursql.DictCursor`.
+
+        Example usage::
+
+            >>> query = "SELECT user_id, user_registration FROM user WHERE user_name = ?"
+            >>> params = ("The Earwig",)
+            >>> result1 = site.sql_query(query, params)
+            >>> result2 = site.sql_query(query, params, dict_cursor=True)
+            >>> for row in result1: print row
+            (7418060L, '20080703215134')
+            >>> for row in result2: print row
+            {'user_id': 7418060L, 'user_registration': '20080703215134'}
+
+        This may raise :py:exc:`~earwigbot.exceptions.SQLError` or one of
+        oursql's exceptions (:py:exc:`oursql.ProgrammingError`,
+        :py:exc:`oursql.InterfaceError`, ...) if there were problems with the
+        query.
+
+        See :py:meth:`_sql_connect` for information on how a connection is
+        acquired. Also relevant is `oursql's documentation
+        <http://packages.python.org/oursql>`_ for details on that package.
         """
-        if not self._sql_conn:
-            self._sql_connect()
-
         if not cursor_class:
             if dict_cursor:
                 cursor_class = oursql.DictCursor
             else:
                 cursor_class = oursql.Cursor
-
-        with self._sql_conn.cursor(cursor_class, show_table=show_table) as cur:
-            cur.execute(query, params, plain_query)
-            for result in cur:
-                yield result
+        klass = cursor_class
+
+        with self._sql_lock:
+            if not self._sql_conn:
+                self._sql_connect()
+            with self._sql_conn.cursor(klass, show_table=show_table) as cur:
+                cur.execute(query, params, plain_query)
+                for result in cur:
+                    yield result
+
+    def get_maxlag(self, showall=False):
+        """Return the internal database replication lag in seconds.
+
+        In a typical setup, this function returns the replication lag *within*
+        the WMF's cluster, *not* external replication lag affecting the
+        Toolserver (see :py:meth:`get_replag` for that). This is useful when
+        combined with the ``maxlag`` API query param (added by config), in
+        which queries will be halted and retried if the lag is too high,
+        usually above five seconds.
+
+        With *showall*, will return a list of the lag for all servers in the
+        cluster, not just the one with the highest lag.
+        """
+        params = {"action": "query", "meta": "siteinfo", "siprop": "dbrepllag"}
+        if showall:
+            params["sishowalldb"] = 1
+        with self._api_lock:
+            result = self._api_query(params, ignore_maxlag=True)
+        if showall:
+            return [server["lag"] for server in result["query"]["dbrepllag"]]
+        return result["query"]["dbrepllag"][0]["lag"]
 
     def get_replag(self):
-        """Return the estimated database replication lag in seconds.
-        
+        """Return the estimated external database replication lag in seconds.
+
         Requires SQL access. This function only makes sense on a replicated
         database (e.g. the Wikimedia Toolserver) and on a wiki that receives a
         large number of edits (ideally, at least one per second), or the result
-        may be larger than expected. 
+        may be larger than expected, since it works by subtracting the current
+        time from the timestamp of the latest recent changes event.
+
+        This may raise :py:exc:`~earwigbot.exceptions.SQLError` or one of
+        oursql's exceptions (:py:exc:`oursql.ProgrammingError`,
+        :py:exc:`oursql.InterfaceError`, ...) if there were problems.
         """
         query = """SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(rc_timestamp) FROM
                    recentchanges ORDER BY rc_timestamp DESC LIMIT 1"""
@@ -563,14 +733,15 @@ class Site(object):
     def namespace_id_to_name(self, ns_id, all=False):
         """Given a namespace ID, returns associated namespace names.
 
-        If all is False (default), we'll return the first name in the list,
-        which is usually the localized version. Otherwise, we'll return the
-        entire list, which includes the canonical name.
+        If *all* is ``False`` (default), we'll return the first name in the
+        list, which is usually the localized version. Otherwise, we'll return
+        the entire list, which includes the canonical name. For example, this
+        returns ``u"Wikipedia"`` if *ns_id* = ``4`` and *all* is ``False`` on
+        ``enwiki``; returns ``[u"Wikipedia", u"Project", u"WP"]`` if *ns_id* =
+        ``4`` and *all* is ``True``.
 
-        For example, returns u"Wikipedia" if ns_id=4 and all=False on enwiki;
-        returns [u"Wikipedia", u"Project", u"WP"] if ns_id=4 and all=True.
-
-        Raises NamespaceNotFoundError if the ID is not found.
+        Raises :py:exc:`~earwigbot.exceptions.NamespaceNotFoundError` if the ID
+        is not found.
         """
         try:
             if all:
@@ -579,15 +750,16 @@ class Site(object):
                 return self._namespaces[ns_id][0]
         except KeyError:
             e = "There is no namespace with id {0}.".format(ns_id)
-            raise NamespaceNotFoundError(e)
+            raise exceptions.NamespaceNotFoundError(e)
 
     def namespace_name_to_id(self, name):
         """Given a namespace name, returns the associated ID.
 
-        Like namespace_id_to_name(), but reversed. Case is ignored, because
-        namespaces are assumed to be case-insensitive.
+        Like :py:meth:`namespace_id_to_name`, but reversed. Case is ignored,
+        because namespaces are assumed to be case-insensitive.
 
-        Raises NamespaceNotFoundError if the name is not found.
+        Raises :py:exc:`~earwigbot.exceptions.NamespaceNotFoundError` if the
+        name is not found.
         """
         lname = name.lower()
         for ns_id, names in self._namespaces.items():
@@ -596,41 +768,80 @@ class Site(object):
                 return ns_id
 
         e = "There is no namespace with name '{0}'.".format(name)
-        raise NamespaceNotFoundError(e)
+        raise exceptions.NamespaceNotFoundError(e)
 
-    def get_page(self, title, follow_redirects=False):
-        """Returns a Page object for the given title (pagename).
+    def get_page(self, title, follow_redirects=False, pageid=None):
+        """Return a :py:class:`Page` object for the given title.
 
-        Will return a Category object instead if the given title is in the
-        category namespace. As Category is a subclass of Page, this should not
-        cause problems.
+        *follow_redirects* is passed directly to
+        :py:class:`~earwigbot.wiki.page.Page`'s constructor. Also, this will
+        return a :py:class:`~earwigbot.wiki.category.Category` object instead
+        if the given title is in the category namespace. As
+        :py:class:`~earwigbot.wiki.category.Category` is a subclass of
+        :py:class:`~earwigbot.wiki.page.Page`, this should not cause problems.
 
         Note that this doesn't do any direct checks for existence or
-        redirect-following - Page's methods provide that.
+        redirect-following: :py:class:`~earwigbot.wiki.page.Page`'s methods
+        provide that.
         """
-        prefixes = self.namespace_id_to_name(NS_CATEGORY, all=True)
+        title = self._unicodeify(title)
+        prefixes = self.namespace_id_to_name(constants.NS_CATEGORY, all=True)
         prefix = title.split(":", 1)[0]
         if prefix != title:  # Avoid a page that is simply "Category"
             if prefix in prefixes:
-                return Category(self, title, follow_redirects)
-        return Page(self, title, follow_redirects)
+                return Category(self, title, follow_redirects, pageid)
+        return Page(self, title, follow_redirects, pageid)
 
-    def get_category(self, catname, follow_redirects=False):
-        """Returns a Category object for the given category name.
+    def get_category(self, catname, follow_redirects=False, pageid=None):
+        """Return a :py:class:`Category` object for the given category name.
 
-        `catname` should be given *without* a namespace prefix. This method is
-        really just shorthand for get_page("Category:" + catname).
+        *catname* should be given *without* a namespace prefix. This method is
+        really just shorthand for :py:meth:`get_page("Category:" + catname)
+        <get_page>`.
         """
-        prefix = self.namespace_id_to_name(NS_CATEGORY)
-        pagename = ':'.join((prefix, catname))
-        return Category(self, pagename, follow_redirects)
+        catname = self._unicodeify(catname)
+        prefix = self.namespace_id_to_name(constants.NS_CATEGORY)
+        pagename = u':'.join((prefix, catname))
+        return Category(self, pagename, follow_redirects, pageid)
 
     def get_user(self, username=None):
-        """Returns a User object for the given username.
+        """Return a :py:class:`User` object for the given username.
 
-        If `username` is left as None, then a User object representing the
-        currently logged-in (or anonymous!) user is returned.
+        If *username* is left as ``None``, then a
+        :py:class:`~earwigbot.wiki.user.User` object representing the currently
+        logged-in (or anonymous!) user is returned.
         """
-        if username is None:
+        if username:
+            username = self._unicodeify(username)
+        else:
             username = self._get_username()
         return User(self, username)
+
+    def delegate(self, services, args=None, kwargs=None):
+        """Delegate a task to either the API or SQL depending on conditions.
+
+        *services* should be a dictionary in which the key is the service name
+        (:py:attr:`self.SERVICE_API <SERVICE_API>` or
+        :py:attr:`self.SERVICE_SQL <SERVICE_SQL>`), and the value is the
+        function to call for this service. All functions will be passed the
+        same arguments the tuple *args* and the dict **kwargs**, which are both
+        empty by default. The service order is determined by
+        :py:meth:`_get_service_order`.
+
+        Not every service needs an entry in the dictionary. Will raise
+        :py:exc:`~earwigbot.exceptions.NoServiceError` if an appropriate
+        service cannot be found.
+        """
+        if not args:
+            args = ()
+        if not kwargs:
+            kwargs = {}
+
+        order = self._get_service_order()
+        for srv in order:
+            if srv in services:
+                try:
+                    return services[srv](*args, **kwargs)
+                except exceptions.ServiceError:
+                    continue
+        raise exceptions.NoServiceError(services)

earwigbot/wiki/sitesdb.py

@@ -0,0 +1,405 @@
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from cookielib import LWPCookieJar, LoadError
+import errno
+from os import chmod, path
+from platform import python_version
+import stat
+import sqlite3 as sqlite
+
+from earwigbot import __version__
+from earwigbot.exceptions import SiteNotFoundError
+from earwigbot.wiki.site import Site
+
+__all__ = ["SitesDB"]
+
+class SitesDB(object):
+    """
+    **EarwigBot: Wiki Toolset: Sites Database Manager**
+
+    This class controls the :file:`sites.db` file, which stores information
+    about all wiki sites known to the bot. Three public methods act as bridges
+    between the bot's config files and :py:class:`~earwigbot.wiki.site.Site`
+    objects:
+
+    - :py:meth:`get_site`:    returns a Site object corresponding to a site
+    - :py:meth:`add_site`:    stores a site in the database
+    - :py:meth:`remove_site`: removes a site from the database
+
+    There's usually no need to use this class directly. All public methods
+    here are available as :py:meth:`bot.wiki.get_site`,
+    :py:meth:`bot.wiki.add_site`, and :py:meth:`bot.wiki.remove_site`, which
+    use a :file:`sites.db` file located in the same directory as our
+    :file:`config.yml` file. Lower-level access can be achieved by importing
+    the manager class (``from earwigbot.wiki import SitesDB``).
+    """
+
+    def __init__(self, bot):
+        """Set up the manager with an attribute for the base Bot object."""
+        self.config = bot.config
+        self._logger = bot.logger.getChild("wiki")
+        self._sites = {}  # Internal site cache
+        self._sitesdb = path.join(bot.config.root_dir, "sites.db")
+        self._cookie_file = path.join(bot.config.root_dir, ".cookies")
+        self._cookiejar = None
+
+    def __repr__(self):
+        """Return the canonical string representation of the SitesDB."""
+        res = "SitesDB(config={0!r}, sitesdb={1!r}, cookie_file={2!r})"
+        return res.format(self.config, self._sitesdb, self._cookie_file)
+
+    def __str__(self):
+        """Return a nice string representation of the SitesDB."""
+        return "<SitesDB at {0}>".format(self._sitesdb)
+
+    def _get_cookiejar(self):
+        """Return a LWPCookieJar object loaded from our .cookies file.
+
+        The same .cookies file is returned every time, located in the project
+        root, same directory as config.yml and bot.py. If it doesn't exist, we
+        will create the file and set it to be readable and writeable only by
+        us. If it exists but the information inside is bogus, we'll ignore it.
+
+        This is normally called by _make_site_object() (in turn called by
+        get_site()), and the cookiejar is passed to our Site's constructor,
+        used when it makes API queries. This way, we can easily preserve
+        cookies between sites (e.g., for CentralAuth), making logins easier.
+        """
+        if self._cookiejar:
+            return self._cookiejar
+
+        self._cookiejar = LWPCookieJar(self._cookie_file)
+
+        try:
+            self._cookiejar.load()
+        except LoadError:
+            pass  # File contains bad data, so ignore it completely
+        except IOError as e:
+            if e.errno == errno.ENOENT:  # "No such file or directory"
+                # Create the file and restrict reading/writing only to the
+                # owner, so others can't peak at our cookies:
+                open(self._cookie_file, "w").close()
+                chmod(self._cookie_file, stat.S_IRUSR|stat.S_IWUSR)
+            else:
+                raise
+
+        return self._cookiejar
+
+    def _create_sitesdb(self):
+        """Initialize the sitesdb file with its three necessary tables."""
+        script = """
+        CREATE TABLE sites (site_name, site_project, site_lang, site_base_url,
+                            site_article_path, site_script_path);
+        CREATE TABLE sql_data (sql_site, sql_data_key, sql_data_value);
+        CREATE TABLE namespaces (ns_site, ns_id, ns_name, ns_is_primary_name);
+        """
+        with sqlite.connect(self._sitesdb) as conn:
+            conn.executescript(script)
+
+    def _get_site_object(self, name):
+        """Return the site from our cache, or create it if it doesn't exist.
+
+        This is essentially just a wrapper around _make_site_object that
+        returns the same object each time a specific site is asked for.
+        """
+        try:
+            return self._sites[name]
+        except KeyError:
+            site = self._make_site_object(name)
+            self._sites[name] = site
+            return site
+
+    def _load_site_from_sitesdb(self, name):
+        """Return all information stored in the sitesdb relating to given site.
+
+        The information will be returned as a tuple, containing the site's
+        name, project, language, base URL, article path, script path, SQL
+        connection data, and namespaces, in that order. If the site is not
+        found in the database, SiteNotFoundError will be raised. An empty
+        database will be created before the exception is raised if none exists.
+        """
+        query1 = "SELECT * FROM sites WHERE site_name = ?"
+        query2 = "SELECT sql_data_key, sql_data_value FROM sql_data WHERE sql_site = ?"
+        query3 = "SELECT ns_id, ns_name, ns_is_primary_name FROM namespaces WHERE ns_site = ?"
+        error = "Site '{0}' not found in the sitesdb.".format(name)
+        with sqlite.connect(self._sitesdb) as conn:
+            try:
+                site_data = conn.execute(query1, (name,)).fetchone()
+            except sqlite.OperationalError:
+                self._create_sitesdb()
+                raise SiteNotFoundError(error)
+            if not site_data:
+                raise SiteNotFoundError(error)
+            sql_data = conn.execute(query2, (name,)).fetchall()
+            ns_data = conn.execute(query3, (name,)).fetchall()
+
+        name, project, lang, base_url, article_path, script_path = site_data
+        sql = dict(sql_data)
+        namespaces = {}
+        for ns_id, ns_name, ns_is_primary_name in ns_data:
+            try:
+                if ns_is_primary_name:  # "Primary" name goes first in list
+                    namespaces[ns_id].insert(0, ns_name)
+                else:  # Ordering of the aliases doesn't matter
+                    namespaces[ns_id].append(ns_name)
+            except KeyError:
+                namespaces[ns_id] = [ns_name]
+
+        return (name, project, lang, base_url, article_path, script_path, sql,
+                namespaces)
+
+    def _make_site_object(self, name):
+        """Return a Site object associated with the site *name* in our sitesdb.
+
+        This calls _load_site_from_sitesdb(), so SiteNotFoundError will be
+        raised if the site is not in our sitesdb.
+        """
+        cookiejar = self._get_cookiejar()
+        (name, project, lang, base_url, article_path, script_path, sql,
+         namespaces) = self._load_site_from_sitesdb(name)
+
+        config = self.config
+        login = (config.wiki.get("username"), config.wiki.get("password"))
+        user_agent = config.wiki.get("userAgent")
+        use_https = config.wiki.get("useHTTPS", False)
+        assert_edit = config.wiki.get("assert")
+        maxlag = config.wiki.get("maxlag")
+        wait_between_queries = config.wiki.get("waitTime", 3)
+        logger = self._logger.getChild(name)
+        search_config = config.wiki.get("search")
+
+        if user_agent:
+            user_agent = user_agent.replace("$1", __version__)
+            user_agent = user_agent.replace("$2", python_version())
+
+        return Site(name=name, project=project, lang=lang, base_url=base_url,
+                    article_path=article_path, script_path=script_path,
+                    sql=sql, namespaces=namespaces, login=login,
+                    cookiejar=cookiejar, user_agent=user_agent,
+                    use_https=use_https, assert_edit=assert_edit,
+                    maxlag=maxlag, wait_between_queries=wait_between_queries,
+                    logger=logger, search_config=search_config)
+
+    def _get_site_name_from_sitesdb(self, project, lang):
+        """Return the name of the first site with the given project and lang.
+
+        If the site is not found, return None. An empty sitesdb will be created
+        if none exists.
+        """
+        query = "SELECT site_name FROM sites WHERE site_project = ? and site_lang = ?"
+        with sqlite.connect(self._sitesdb) as conn:
+            try:
+                site = conn.execute(query, (project, lang)).fetchone()
+                return site[0] if site else None
+            except sqlite.OperationalError:
+                self._create_sitesdb()
+
+    def _add_site_to_sitesdb(self, site):
+        """Extract relevant info from a Site object and add it to the sitesdb.
+
+        Works like a reverse _load_site_from_sitesdb(); the site's project,
+        language, base URL, article path, script path, SQL connection data, and
+        namespaces are extracted from the site and inserted into the sites
+        database. If the sitesdb doesn't exist, we'll create it first.
+        """
+        name = site.name
+        sites_data = (name, site.project, site.lang, site._base_url,
+                      site._article_path, site._script_path)
+        sql_data = [(name, key, val) for key, val in site._sql_data.iteritems()]
+        ns_data = []
+        for ns_id, ns_names in site._namespaces.iteritems():
+            ns_data.append((name, ns_id, ns_names.pop(0), True))
+            for ns_name in ns_names:
+                ns_data.append((name, ns_id, ns_name, False))
+
+        with sqlite.connect(self._sitesdb) as conn:
+            check_exists = "SELECT 1 FROM sites WHERE site_name = ?"
+            try:
+                exists = conn.execute(check_exists, (name,)).fetchone()
+            except sqlite.OperationalError:
+                self._create_sitesdb()
+            else:
+                if exists:
+                    conn.execute("DELETE FROM sites WHERE site_name = ?", (name,))
+                    conn.execute("DELETE FROM sql_data WHERE sql_site = ?", (name,))
+                    conn.execute("DELETE FROM namespaces WHERE ns_site = ?", (name,))
+            conn.execute("INSERT INTO sites VALUES (?, ?, ?, ?, ?, ?)", sites_data)
+            conn.executemany("INSERT INTO sql_data VALUES (?, ?, ?)", sql_data)
+            conn.executemany("INSERT INTO namespaces VALUES (?, ?, ?, ?)", ns_data)
+
+    def _remove_site_from_sitesdb(self, name):
+        """Remove a site by name from the sitesdb and the internal cache."""
+        try:
+            del self._sites[name]
+        except KeyError:
+            pass
+
+        with sqlite.connect(self._sitesdb) as conn:
+            cursor = conn.execute("DELETE FROM sites WHERE site_name = ?", (name,))
+            if cursor.rowcount == 0:
+                return False
+            else:
+                conn.execute("DELETE FROM sql_data WHERE sql_site = ?", (name,))
+                conn.execute("DELETE FROM namespaces WHERE ns_site = ?", (name,))
+                return True
+
+    def get_site(self, name=None, project=None, lang=None):
+        """Return a Site instance based on information from the sitesdb.
+
+        With no arguments, return the default site as specified by our config
+        file. This is ``config.wiki["defaultSite"]``.
+
+        With *name* specified, return the site with that name. This is
+        equivalent to the site's ``wikiid`` in the API, like *enwiki*.
+
+        With *project* and *lang* specified, return the site whose project and
+        language match these values. If there are multiple sites with the same
+        values (unlikely), this is not a reliable way of loading a site. Call
+        the function with an explicit *name* in that case.
+
+        We will attempt to login to the site automatically using
+        ``config.wiki["username"]`` and ``config.wiki["password"]`` if both are
+        defined.
+
+        Specifying a project without a lang or a lang without a project will
+        raise :py:exc:`TypeError`. If all three args are specified, *name* will
+        be first tried, then *project* and *lang* if *name* doesn't work. If a
+        site cannot be found in the sitesdb,
+        :py:exc:`~earwigbot.exceptions.SiteNotFoundError` will be raised. An
+        empty sitesdb will be created if none is found.
+        """
+        # Someone specified a project without a lang, or vice versa:
+        if (project and not lang) or (not project and lang):
+            e = "Keyword arguments 'lang' and 'project' must be specified together."
+            raise TypeError(e)
+
+        # No args given, so return our default site:
+        if not name and not project and not lang:
+            try:
+                default = self.config.wiki["defaultSite"]
+            except KeyError:
+                e = "Default site is not specified in config."
+                raise SiteNotFoundError(e)
+            return self._get_site_object(default)
+
+        # Name arg given, but don't look at others unless `name` isn't found:
+        if name:
+            try:
+               return self._get_site_object(name)
+            except SiteNotFoundError:
+                if project and lang:
+                    name = self._get_site_name_from_sitesdb(project, lang)
+                    if name:
+                        return self._get_site_object(name)
+                raise
+
+        # If we end up here, then project and lang are the only args given:
+        name = self._get_site_name_from_sitesdb(project, lang)
+        if name:
+            return self._get_site_object(name)
+        e = "Site '{0}:{1}' not found in the sitesdb.".format(project, lang)
+        raise SiteNotFoundError(e)
+
+    def add_site(self, project=None, lang=None, base_url=None,
+                 script_path="/w", sql=None):
+        """Add a site to the sitesdb so it can be retrieved with get_site().
+
+        If only a project and a lang are given, we'll guess the *base_url* as
+        ``"//{lang}.{project}.org"`` (which is protocol-relative, becoming
+        ``"https"`` if *useHTTPS* is ``True`` in config otherwise ``"http"``).
+        If this is wrong, provide the correct *base_url* as an argument (in
+        which case project and lang are ignored). Most wikis use ``"/w"`` as
+        the script path (meaning the API is located at
+        ``"{base_url}{script_path}/api.php"`` ->
+        ``"//{lang}.{project}.org/w/api.php"``), so this is the default. If
+        your wiki is different, provide the script_path as an argument. The
+        only other argument to :py:class:`~earwigbot.wiki.site.Site` that we
+        can't get from config files or by querying the wiki itself is SQL
+        connection info, so provide a dict of kwargs as *sql* and Site will
+        pass it to :py:func:`oursql.connect(**sql) <oursql.connect>`, allowing
+        you to make queries with :py:meth:`site.sql_query
+        <earwigbot.wiki.site.Site.sql_query>`.
+
+        Returns ``True`` if the site was added successfully or ``False`` if the
+        site is already in our sitesdb (this can be done purposefully to update
+        old site info). Raises :py:exc:`~earwigbot.exception.SiteNotFoundError`
+        if not enough information has been provided to identify the site (e.g.
+        a *project* but not a *lang*).
+        """
+        if not base_url:
+            if not project or not lang:
+                e = "Without a base_url, both a project and a lang must be given."
+                raise SiteNotFoundError(e)
+            base_url = "//{0}.{1}.org".format(lang, project)
+        cookiejar = self._get_cookiejar()
+
+        config = self.config
+        login = (config.wiki.get("username"), config.wiki.get("password"))
+        user_agent = config.wiki.get("userAgent")
+        use_https = config.wiki.get("useHTTPS", False)
+        assert_edit = config.wiki.get("assert")
+        maxlag = config.wiki.get("maxlag")
+        wait_between_queries = config.wiki.get("waitTime", 5)
+        search_config = config.wiki.get("search")
+
+        # Create a Site object to log in and load the other attributes:
+        site = Site(base_url=base_url, script_path=script_path, sql=sql,
+                    login=login, cookiejar=cookiejar, user_agent=user_agent,
+                    use_https=use_https, assert_edit=assert_edit,
+                    maxlag=maxlag, wait_between_queries=wait_between_queries,
+                    search_config=search_config)
+
+        self._add_site_to_sitesdb(site)
+        self._sites[site.name] = site
+        return site
+
+    def remove_site(self, name=None, project=None, lang=None):
+        """Remove a site from the sitesdb.
+
+        Returns ``True`` if the site was removed successfully or ``False`` if
+        the site was not in our sitesdb originally. If all three args (*name*,
+        *project*, and *lang*) are given, we'll first try *name* and then try
+        the latter two if *name* wasn't found in the database. Raises
+        :py:exc:`TypeError` if a project was given but not a language, or vice
+        versa. Will create an empty sitesdb if none was found.
+        """
+        # Someone specified a project without a lang, or vice versa:
+        if (project and not lang) or (not project and lang):
+            e = "Keyword arguments 'lang' and 'project' must be specified together."
+            raise TypeError(e)
+
+        if name:
+            was_removed = self._remove_site_from_sitesdb(name)
+            if not was_removed:
+                if project and lang:
+                    name = self._get_site_name_from_sitesdb(project, lang)
+                    if name:
+                        return self._remove_site_from_sitesdb(name)
+            return was_removed
+
+        if project and lang:
+            name = self._get_site_name_from_sitesdb(project, lang)
+            if name:
+                return self._remove_site_from_sitesdb(name)
+
+        return False

earwigbot/wiki/user.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -22,31 +22,42 @@
 
 from time import gmtime, strptime
 
-from earwigbot.wiki.constants import *
-from earwigbot.wiki.exceptions import UserNotFoundError
+from earwigbot.exceptions import UserNotFoundError
+from earwigbot.wiki import constants
 from earwigbot.wiki.page import Page
 
+__all__ = ["User"]
+
 class User(object):
     """
-    EarwigBot's Wiki Toolset: User Class
-
-    Represents a User on a given Site. Has methods for getting a bunch of 
-    information about the user, such as editcount and user rights, methods for
-    returning the user's userpage and talkpage, etc.
-
-    Public methods:
-    name         -- returns the user's username
-    exists       -- returns True if the user exists, False if they do not
-    userid       -- returns an integer ID representing the user
-    blockinfo    -- returns information about a current block on the user
-    groups       -- returns a list of the user's groups
-    rights       -- returns a list of the user's rights
-    editcount    -- returns the number of edits made by the user
-    registration -- returns the time the user registered as a time.struct_time
-    emailable    -- returns True if you can email the user, False if you cannot
-    gender       -- returns the user's gender ("male", "female", or "unknown")
-    get_userpage -- returns a Page object representing the user's userpage
-    get_talkpage -- returns a Page object representing the user's talkpage
+    **EarwigBot: Wiki Toolset: User**
+
+    Represents a user on a given :py:class:`~earwigbot.wiki.site.Site`. Has
+    methods for getting a bunch of information about the user, such as
+    editcount and user rights, methods for returning the user's userpage and
+    talkpage, etc.
+
+    *Attributes:*
+
+    - :py:attr:`site`:         the user's corresponding Site object
+    - :py:attr:`name`:         the user's username
+    - :py:attr:`exists`:       ``True`` if the user exists, else ``False``
+    - :py:attr:`userid`:       an integer ID representing the user
+    - :py:attr:`blockinfo`:    information about any current blocks on the user
+    - :py:attr:`groups`:       a list of the user's groups
+    - :py:attr:`rights`:       a list of the user's rights
+    - :py:attr:`editcount`:    the number of edits made by the user
+    - :py:attr:`registration`: the time the user registered
+    - :py:attr:`emailable`:    ``True`` if you can email the user, or ``False``
+    - :py:attr:`gender`:       the user's gender ("male"/"female"/"unknown")
+
+    *Public methods:*
+
+    - :py:meth:`reload`:       forcibly reloads the user's attributes
+    - :py:meth:`get_userpage`: returns a Page object representing the user's
+      userpage
+    - :py:meth:`get_talkpage`: returns a Page object representing the user's
+      talkpage
     """
 
     def __init__(self, site, name):
@@ -66,27 +77,26 @@ class User(object):
         self._name = name
 
     def __repr__(self):
-        """Returns the canonical string representation of the User."""
+        """Return the canonical string representation of the User."""
         return "User(name={0!r}, site={1!r})".format(self._name, self._site)
 
     def __str__(self):
-        """Returns a nice string representation of the User."""
-        return '<User "{0}" of {1}>'.format(self.name(), str(self._site))
+        """Return a nice string representation of the User."""
+        return '<User "{0}" of {1}>'.format(self.name, str(self.site))
 
-    def _get_attribute(self, attr, force):
+    def _get_attribute(self, attr):
         """Internally used to get an attribute by name.
 
         We'll call _load_attributes() to get this (and all other attributes)
-        from the API if it is not already defined. If `force` is True, we'll
-        re-load them even if they've already been loaded.
+        from the API if it is not already defined.
 
         Raises UserNotFoundError if a nonexistant user prevents us from
         returning a certain attribute.
         """
-        if not hasattr(self, attr) or force:
+        if not hasattr(self, attr):
             self._load_attributes()
-        if self._exists is False:
-            e = "User '{0}' does not exist.".format(self._name)
+        if not self._exists:
+            e = u"User '{0}' does not exist.".format(self._name)
             raise UserNotFoundError(e)
         return getattr(self, attr)
 
@@ -96,9 +106,9 @@ class User(object):
         Normally, this is called by _get_attribute() when a requested attribute
         is not defined. This defines it.
         """
-        params = {"action": "query", "list": "users", "ususers": self._name,
-                  "usprop": "blockinfo|groups|rights|editcount|registration|emailable|gender"}
-        result = self._site._api_query(params)
+        props = "blockinfo|groups|rights|editcount|registration|emailable|gender"
+        result = self.site.api_query(action="query", list="users",
+                                     ususers=self._name, usprop=props)
         res = result["query"]["users"][0]
 
         # normalize our username in case it was entered oddly
@@ -145,118 +155,136 @@ class User(object):
 
         self._gender = res["gender"]
 
-    def name(self, force=False):
-        """Returns the user's name.
+    @property
+    def site(self):
+        """The user's corresponding Site object."""
+        return self._site
 
-        If `force` is True, we will load the name from the API and return that.
-        This could potentially return a "normalized" version of the name - for
-        example, without a "User:" prefix or without underscores. Unlike other
-        attribute getters, this will never make an API query without `force`.
+    @property
+    def name(self):
+        """The user's username.
 
-        Note that if another attribute getter, like exists(), has already been
-        called, then the username has already been normalized.
+        This will never make an API query on its own, but if one has already
+        been made by the time this is retrieved, the username may have been
+        "normalized" from the original input to the constructor, converted into
+        a Unicode object, with underscores removed, etc.
         """
-        if force:
-            self._load_attributes()
         return self._name
 
-    def exists(self, force=False):
-        """Returns True if the user exists, or False if they do not.
+    @property
+    def exists(self):
+        """``True`` if the user exists, or ``False`` if they do not.
 
-        Makes an API query if `force` is True or if we haven't made one
-        already.
+        Makes an API query only if we haven't made one already.
         """
-        if not hasattr(self, "_exists") or force:
+        if not hasattr(self, "_exists"):
             self._load_attributes()
         return self._exists
 
-    def userid(self, force=False):
-        """Returns an integer ID used by MediaWiki to represent the user.
+    @property
+    def userid(self):
+        """An integer ID used by MediaWiki to represent the user.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_userid", force)
+        return self._get_attribute("_userid")
 
-    def blockinfo(self, force=False):
-        """Returns information about a current block on the user.
+    @property
+    def blockinfo(self):
+        """Information about any current blocks on the user.
 
-        If the user is not blocked, returns False. If they are, returns a dict
-        with three keys: "by" is the blocker's username, "reason" is the reason
-        why they were blocked, and "expiry" is when the block expires.
+        If the user is not blocked, returns ``False``. If they are, returns a
+        dict with three keys: ``"by"`` is the blocker's username, ``"reason"``
+        is the reason why they were blocked, and ``"expiry"`` is when the block
+        expires.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_blockinfo", force)
+        return self._get_attribute("_blockinfo")
 
-    def groups(self, force=False):
-        """Returns a list of groups this user is in, including "*".
+    @property
+    def groups(self):
+        """A list of groups this user is in, including ``"*"``.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_groups", force)
+        return self._get_attribute("_groups")
 
-    def rights(self, force=False):
-        """Returns a list of this user's rights.
+    @property
+    def rights(self):
+        """A list of this user's rights.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_rights", force)
+        return self._get_attribute("_rights")
 
-    def editcount(self, force=False):
+    @property
+    def editcount(self):
         """Returns the number of edits made by the user.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_editcount", force)
+        return self._get_attribute("_editcount")
 
-    def registration(self, force=False):
-        """Returns the time the user registered as a time.struct_time object.
+    @property
+    def registration(self):
+        """The time the user registered as a :py:class:`time.struct_time`.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_registration", force)
+        return self._get_attribute("_registration")
 
-    def emailable(self, force=False):
-        """Returns True if the user can be emailed, or False if they cannot.
+    @property
+    def emailable(self):
+        """``True`` if the user can be emailed, or ``False`` if they cannot.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_emailable", force)
+        return self._get_attribute("_emailable")
 
-    def gender(self, force=False):
-        """Returns the user's gender.
+    @property
+    def gender(self):
+        """The user's gender.
 
-        Can return either "male", "female", or "unknown", if they did not
-        specify it.
+        Can return either ``"male"``, ``"female"``, or ``"unknown"``, if they
+        did not specify it.
 
-        Raises UserNotFoundError if the user does not exist. Makes an API query
-        if `force` is True or if we haven't made one already.
+        Raises :py:exc:`~earwigbot.exceptions.UserNotFoundError` if the user
+        does not exist. Makes an API query only if we haven't made one already.
         """
-        return self._get_attribute("_gender", force)
+        return self._get_attribute("_gender")
+
+    def reload(self):
+        """Forcibly reload the user's attributes.
+
+        Emphasis on *reload*: this is only necessary if there is reason to
+        believe they have changed.
+        """
+        self._load_attributes()
 
     def get_userpage(self):
-        """Returns a Page object representing the user's userpage.
-        
+        """Return a Page object representing the user's userpage.
+
         No checks are made to see if it exists or not. Proper site namespace
         conventions are followed.
         """
-        prefix = self._site.namespace_id_to_name(NS_USER)
+        prefix = self.site.namespace_id_to_name(constants.NS_USER)
         pagename = ':'.join((prefix, self._name))
-        return Page(self._site, pagename)
+        return Page(self.site, pagename)
 
     def get_talkpage(self):
-        """Returns a Page object representing the user's talkpage.
-        
+        """Return a Page object representing the user's talkpage.
+
         No checks are made to see if it exists or not. Proper site namespace
         conventions are followed.
         """
-        prefix = self._site.namespace_id_to_name(NS_USER_TALK)
+        prefix = self.site.namespace_id_to_name(constants.NS_USER_TALK)
         pagename = ':'.join((prefix, self._name))
-        return Page(self._site, pagename)
+        return Page(self.site, pagename)

setup.py

@@ -0,0 +1,65 @@
+#! /usr/bin/env python
+# -*- coding: utf-8  -*-
+#
+# Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from setuptools import setup, find_packages
+
+from earwigbot import __version__
+
+with open("README.rst") as fp:
+    long_docs = fp.read()
+
+setup(
+    name = "earwigbot",
+    packages = find_packages(exclude=("tests",)),
+    entry_points = {"console_scripts": ["earwigbot = earwigbot.util:main"]},
+    install_requires = ["GitPython >= 0.3.2.RC1",  # Interfacing with git
+                        "PyYAML >= 3.10",  # Config parsing
+                        "mwparserfromhell >= 0.1",  # Wikicode parsing
+                        "oursql >= 0.9.3",  # Talking with MediaWiki databases
+                        "oauth2 >= 1.5.211",  # Talking with Yahoo BOSS Search
+                        "py-bcrypt >= 0.2",  # Password hashing in config
+                        "pycrypto >= 2.5",  # Storing bot passwords and keys
+                        "pytz >= 2012c",  # Timezone handling
+                        ],
+    test_suite = "tests",
+    version = __version__,
+    author = "Ben Kurtovic",
+    author_email = "ben.kurtovic@verizon.net",
+    url = "https://github.com/earwig/earwigbot",
+    description = "EarwigBot is a Python robot that edits Wikipedia and interacts with people over IRC.",
+    long_description = long_docs,
+    download_url = "https://github.com/earwig/earwigbot/tarball/v{0}".format(__version__),
+    keywords = "earwig earwigbot irc wikipedia wiki mediawiki",
+    license = "MIT License",
+    classifiers = [
+        "Development Status :: 3 - Alpha",
+        "Environment :: Console",
+        "Intended Audience :: Developers",
+        "License :: OSI Approved :: MIT License",
+        "Natural Language :: English",
+        "Operating System :: OS Independent",
+        "Programming Language :: Python :: 2.7",
+        "Topic :: Communications :: Chat :: Internet Relay Chat",
+        "Topic :: Internet :: WWW/HTTP"
+    ],
+)

earwigbot/tests/__init__.py → tests/__init__.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,29 +23,44 @@
 """
 EarwigBot's Unit Tests
 
-This module __init__ file provides some support code for unit tests.
+This __init__ file provides some support code for unit tests.
+
+Test cases:
+  -- CommandTestCase provides setUp() for creating a fake connection, plus
+     some other helpful methods for testing IRC commands.
+
+Fake objects:
+  -- FakeBot implements Bot, using the Fake* equivalents of all objects
+     whenever possible.
+  -- FakeBotConfig implements BotConfig with silent logging.
+  -- FakeIRCConnection implements IRCConnection, using an internal string
+     buffer for data instead of sending it over a socket.
 
-CommandTestCase is a subclass of unittest.TestCase that provides setUp() for
-creating a fake connection and some other helpful methods. It uses 
-FakeConnection, a subclass of classes.Connection, but with an internal string
-instead of a socket for data.
 """
 
+import logging
+from os import path
 import re
+from threading import Lock
 from unittest import TestCase
 
-from earwigbot.classes import Connection, Data
+from earwigbot.bot import Bot
+from earwigbot.commands import CommandManager
+from earwigbot.config import BotConfig
+from earwigbot.irc import IRCConnection, Data
+from earwigbot.tasks import TaskManager
+from earwigbot.wiki import SitesDB
 
 class CommandTestCase(TestCase):
     re_sender = re.compile(":(.*?)!(.*?)@(.*?)\Z")
 
     def setUp(self, command):
-        self.connection = FakeConnection()
-        self.connection.connect()
-        self.command = command(self.connection)
+        self.bot = FakeBot(path.dirname(__file__))
+        self.command = command(self.bot)
+        self.command.connection = self.connection = self.bot.frontend
 
     def get_single(self):
-        data = self.connection.get().split("\n")
+        data = self.connection._get().split("\n")
         line = data.pop(0)
         for remaining in data[1:]:
             self.connection.send(remaining)
@@ -92,16 +107,42 @@ class CommandTestCase(TestCase):
         line = ":Foo!bar@example.com JOIN :#channel".strip().split()
         return self.maker(line, line[2][1:])
 
-class FakeConnection(Connection):
-    def connect(self):
+
+class FakeBot(Bot):
+    def __init__(self, root_dir):
+        self.config = FakeBotConfig(root_dir)
+        self.logger = logging.getLogger("earwigbot")
+        self.commands = CommandManager(self)
+        self.tasks = TaskManager(self)
+        self.wiki = SitesDB(self)
+        self.frontend = FakeIRCConnection(self)
+        self.watcher = FakeIRCConnection(self)
+
+        self.component_lock = Lock()
+        self._keep_looping = True
+
+
+class FakeBotConfig(BotConfig):
+    def _setup_logging(self):
+        logger = logging.getLogger("earwigbot")
+        logger.addHandler(logging.NullHandler())
+
+
+class FakeIRCConnection(IRCConnection):
+    def __init__(self, bot):
+        self.bot = bot
+        self._is_running = False
+        self._connect()
+
+    def _connect(self):
         self._buffer = ""
 
-    def close(self):
-        pass
+    def _close(self):
+        self._buffer = ""
 
-    def get(self, size=4096):
+    def _get(self, size=4096):
         data, self._buffer = self._buffer, ""
         return data
 
-    def send(self, msg):
+    def _send(self, msg):
         self._buffer += msg + "\n"

earwigbot/tests/test_calc.py → tests/test_calc.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,7 +23,7 @@
 import unittest
 
 from earwigbot.commands.calc import Command
-from earwigbot.tests import CommandTestCase
+from tests import CommandTestCase
 
 class TestCalc(CommandTestCase):
 

earwigbot/tests/test_test.py → tests/test_test.py

@@ -1,17 +1,17 @@
 # -*- coding: utf-8  -*-
 #
 # Copyright (C) 2009-2012 by Ben Kurtovic <ben.kurtovic@verizon.net>
-# 
+#
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
 # in the Software without restriction, including without limitation the rights
 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is 
+# copies of the Software, and to permit persons to whom the Software is
 # furnished to do so, subject to the following conditions:
-# 
+#
 # The above copyright notice and this permission notice shall be included in
 # all copies or substantial portions of the Software.
-# 
+#
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -23,7 +23,7 @@
 import unittest
 
 from earwigbot.commands.test import Command
-from earwigbot.tests import CommandTestCase
+from tests import CommandTestCase
 
 class TestTest(CommandTestCase):
 
@@ -38,12 +38,12 @@ class TestTest(CommandTestCase):
         self.assertTrue(self.command.check(self.make_msg("TEST", "foo")))
 
     def test_process(self):
-        def _test():
+        def test():
             self.command.process(self.make_msg("test"))
             self.assertSaidIn(["Hey \x02Foo\x0F!", "'sup \x02Foo\x0F?"])
 
         for i in xrange(64):
-            _test()
+            test()
 
 if __name__ == "__main__":
     unittest.main(verbosity=2)