Restructure, switch to pyproject.toml, pytest, update docs

CHANGELOG

@@ -1,6 +1,7 @@
 v0.4 (unreleased):
 
 - Migrated to Python 3 (3.11+). Substantial code cleanup.
+- Migrated to pyproject.toml and pytest.
 - Migrated from oursql to pymysql.
 - Copyvios: Configurable proxy support for specific domains.
 - Copyvios: Parser-directed URL redirection.

README.rst

@@ -1,75 +1,84 @@
 EarwigBot
 =========
 
-EarwigBot_ is a Python_ robot that edits Wikipedia_ and interacts with people
-over IRC_. This file provides a basic overview of how to install and setup the
-bot; more detailed information is located in the ``docs/`` directory (available
-online at PyPI_).
+EarwigBot_ is a Python bot that edits Wikipedia_ and interacts over IRC_.
+This README provides a basic overview of how to install and setup the bot;
+more detailed information is located in the ``docs/`` directory
+(`available online_`).
 
 History
 -------
 
 Development began, based on `Pywikibot`_, in early 2009. Approval for its
 first task, a `copyright violation detector`_, was carried out in May, and the
-bot has been running consistently ever since (with the exception of Jan/Feb 2011).
-It currently handles `several ongoing tasks`_ ranging from statistics generation
-to category cleanup, and on-demand tasks such as WikiProject template tagging.
-Since it started running, the bot has made over 250,000 edits.
+bot has been running consistently ever since. It currently handles
+`several ongoing tasks`_ ranging from statistics generation to category
+cleanup, and on-demand tasks such as WikiProject template tagging. Since it
+started running, the bot has made over 300,000 edits.
 
-A project to rewrite it from scratch began in early April 2011, thus moving
-away from the Pywikibot framework and allowing for less overall code, better
-integration between bot parts, and easier maintenance.
+The current version of its codebase began development in April 2011, moving
+away from Pywikibot to a custom framework.
 
 Installation
 ------------
 
-This package contains the core ``earwigbot``, abstracted enough that it should
-be usable and customizable by anyone running a bot on a MediaWiki site. Since
-it is component-based, the IRC components can be disabled if desired. IRC
-commands and bot tasks specific to `my instance of EarwigBot`_ that I don't
-feel the average user will need are available from the repository
-`earwigbot-plugins`_.
-
-It's recommended to run the bot's unit tests before installing. Run ``python
-setup.py test`` from the project's root directory. Note that some
-tests require an internet connection, and others may take a while to run.
-Coverage is currently rather incomplete.
+This package contains the core ``earwigbot``, abstracted to be usable and
+customizable by anyone running a bot on a MediaWiki site. Since it is modular,
+the IRC components can be disabled if desired. IRC commands and bot tasks
+specific to `my instance of EarwigBot`_ that I don't feel the average user
+will need are available from the repository `earwigbot-plugins`_.
 
 Latest release
 ~~~~~~~~~~~~~~
 
-EarwigBot is available from the `Python Package Index`_, so you can install the
-latest release with ``pip install earwigbot``.
+EarwigBot is available from the `Python Package Index`_, so you can install
+the latest release with:
+
+    pip install earwigbot
+
+There are a few sets of optional dependencies:
+
+- ``crypto``: Allows encrypting bot passwords and secrets in the config
+- ``sql``: Allows interfacing with MediaWiki databases (e.g. on Toolforge_)
+- ``copyvios``: Includes parsing libraries for checking copyright violations
+- ``dev``: Installs development dependencies (e.g. test runners)
 
-If you get an error while pip is installing dependencies, you may be missing
-some header files. For example, on Ubuntu, see `this StackOverflow post`_.
+For example, to install all non-dev dependencies:
+
+    pip install 'earwigbot[crypto,sql,copyvios]'
+
+Errors while pip is installing dependencies may be due to missing header
+files. For example, on Ubuntu, see `this StackOverflow post`_.
 
 Development version
 ~~~~~~~~~~~~~~~~~~~
 
-You can install the development version of the bot from ``git`` by using
-setuptools's ``develop`` command::
+You can install the development version of the bot::
 
-    git clone git://github.com/earwig/earwigbot.git earwigbot
+    git clone https://github.com/earwig/earwigbot.git
     cd earwigbot
-    python setup.py develop
+    python3 -m venv venv
+    . venv/bin/activate
+    pip install -e '.[crypto,sql,copyvios,dev]'
+
+To run the bot's unit tests, run ``pytest`` (requires the ``dev``
+dependencies). Coverage is currently rather incomplete.
 
 Setup
 -----
 
-The bot stores its data in a "working directory", including its config file and
-databases. This is also the location where you will place custom IRC commands
-and bot tasks, which will be explained later. It doesn't matter where this
-directory is, as long as the bot can write to it.
+The bot stores its data in a "working directory", including its config file
+and databases. This is also the location where you will place custom IRC
+commands and bot tasks, which will be explained later. It doesn't matter where
+this directory is, as long as the bot can write to it.
 
 Start the bot with ``earwigbot path/to/working/dir``, or just ``earwigbot`` if
 the working directory is the current directory. It will notice that no
 ``config.yml`` file exists and take you through the setup process.
 
 There is currently no way to edit the ``config.yml`` file from within the bot
-after it has been created, but YAML is a very straightforward format, so you
-should be able to make any necessary changes yourself. Check out the
-`explanation of YAML`_ on Wikipedia for help.
+after it has been created, but you should be able to make any necessary
+changes yourself.
 
 After setup, the bot will start. This means it will connect to the IRC servers
 it has been configured for, schedule bot tasks to run at specific times, and
@@ -86,8 +95,8 @@ Customizing
 The bot's working directory contains a ``commands`` subdirectory and a
 ``tasks`` subdirectory. Custom IRC commands can be placed in the former,
 whereas custom wiki bot tasks go into the latter. Developing custom modules is
-explained below, and in more detail through the bot's documentation on PyPI_
-(or in the ``docs/`` dir).
+explained below, and in more detail through the bot's documentation_ or in the
+``docs/`` dir.
 
 Note that custom commands will override built-in commands and tasks with the
 same name.
@@ -101,11 +110,11 @@ because it is the main way to communicate with other parts of the bot. A
 ``Bot`` object is accessible as an attribute of commands and tasks (i.e.,
 ``self.bot``).
 
-`earwigbot.config.BotConfig`_ stores configuration information for the bot. Its
-docstring explains what each attribute is used for, but essentially each "node"
-(one of ``config.components``, ``wiki``, ``irc``, ``commands``, ``tasks``, and
-``metadata``) maps to a section of the bot's ``config.yml`` file. For example,
-if ``config.yml`` includes something like::
+`earwigbot.config.BotConfig`_ stores configuration information for the bot.
+Its docstring explains what each attribute is used for, but essentially each
+"node" (one of ``config.components``, ``wiki``, ``irc``, ``commands``,
+``tasks``, and ``metadata``) maps to a section of the bot's ``config.yml``
+file. For example, if ``config.yml`` includes something like::
 
     irc:
         frontend:
@@ -115,7 +124,7 @@ if ``config.yml`` includes something like::
                 - "#channel"
                 - "#other-channel"
 
-...then ``config.irc["frontend"]["nick"]`` will be ``"MyAwesomeBot"`` and
+then ``config.irc["frontend"]["nick"]`` will be ``"MyAwesomeBot"`` and
 ``config.irc["frontend"]["channels"]`` will be ``["##earwigbot", "#channel",
 "#other-channel"]``.
 
@@ -133,8 +142,8 @@ afc_status_ for some more complicated scripts.
 Custom bot tasks
 ~~~~~~~~~~~~~~~~
 
-Custom tasks are subclasses of `earwigbot.tasks.Task`_ that override ``Task``'s
-``run()`` (and optionally ``setup()`` or ``unload()``) methods.
+Custom tasks are subclasses of `earwigbot.tasks.Task`_ that override
+``Task``'s ``run()`` (and optionally ``setup()`` or ``unload()``) methods.
 
 See the built-in wikiproject_tagger_ task for a relatively straightforward
 task, or the afc_statistics_ plugin for a more complicated one.
@@ -142,10 +151,10 @@ task, or the afc_statistics_ plugin for a more complicated one.
 The Wiki Toolset
 ----------------
 
-EarwigBot's answer to the `Pywikipedia framework`_ is the Wiki Toolset
-(``earwigbot.wiki``), which you will mainly access through ``bot.wiki``.
+EarwigBot's answer to the Pywikibot_ is the Wiki Toolset (``earwigbot.wiki``),
+which you will mainly access through ``bot.wiki``.
 
-``bot.wiki`` provides three methods for the management of Sites -
+``bot.wiki`` provides three methods for the management of Sites:
 ``get_site()``, ``add_site()``, and ``remove_site()``. Sites are objects that
 simply represent a MediaWiki site. A single instance of EarwigBot (i.e. a
 single *working directory*) is expected to relate to a single site or group of
@@ -160,25 +169,25 @@ docstrings`_ to learn how to use it in a more hands-on fashion. For reference,
 ``sites.db`` file in the bot's working directory.
 
 .. _EarwigBot:                      https://en.wikipedia.org/wiki/User:EarwigBot
-.. _Python:                         https://python.org/
 .. _Wikipedia:                      https://en.wikipedia.org/
 .. _IRC:                            https://en.wikipedia.org/wiki/Internet_Relay_Chat
-.. _PyPI:                           https://packages.python.org/earwigbot
+.. _available online:               https://pythonhosted.org/earwigbot/
 .. _Pywikibot:                      https://www.mediawiki.org/wiki/Manual:Pywikibot
 .. _copyright violation detector:   https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/EarwigBot_1
 .. _several ongoing tasks:          https://en.wikipedia.org/wiki/User:EarwigBot#Tasks
 .. _my instance of EarwigBot:       https://en.wikipedia.org/wiki/User:EarwigBot
 .. _earwigbot-plugins:              https://github.com/earwig/earwigbot-plugins
 .. _Python Package Index:           https://pypi.python.org/pypi/earwigbot
+.. _Toolforge:                      https://wikitech.wikimedia.org/wiki/Portal:Toolforge
 .. _this StackOverflow post:        https://stackoverflow.com/questions/6504810/how-to-install-lxml-on-ubuntu/6504860#6504860
-.. _explanation of YAML:            https://en.wikipedia.org/wiki/YAML
-.. _earwigbot.bot.Bot:              https://github.com/earwig/earwigbot/blob/develop/earwigbot/bot.py
-.. _earwigbot.config.BotConfig:     https://github.com/earwig/earwigbot/blob/develop/earwigbot/config.py
-.. _earwigbot.commands.Command:     https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/__init__.py
-.. _test:                           https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/test.py
-.. _chanops:                        https://github.com/earwig/earwigbot/blob/develop/earwigbot/commands/chanops.py
-.. _afc_status:                     https://github.com/earwig/earwigbot-plugins/blob/develop/commands/afc_status.py
-.. _earwigbot.tasks.Task:           https://github.com/earwig/earwigbot/blob/develop/earwigbot/tasks/__init__.py
-.. _wikiproject_tagger:             https://github.com/earwig/earwigbot/blob/develop/earwigbot/tasks/wikiproject_tagger.py
-.. _afc_statistics:                 https://github.com/earwig/earwigbot-plugins/blob/develop/tasks/afc_statistics.py
-.. _its code and docstrings:        https://github.com/earwig/earwigbot/tree/develop/earwigbot/wiki
+.. _documentation:                  https://pythonhosted.org/earwigbot/
+.. _earwigbot.bot.Bot:              https://github.com/earwig/earwigbot/blob/main/earwigbot/bot.py
+.. _earwigbot.config.BotConfig:     https://github.com/earwig/earwigbot/blob/main/earwigbot/config.py
+.. _earwigbot.commands.Command:     https://github.com/earwig/earwigbot/blob/main/earwigbot/commands/__init__.py
+.. _test:                           https://github.com/earwig/earwigbot/blob/main/earwigbot/commands/test.py
+.. _chanops:                        https://github.com/earwig/earwigbot/blob/main/earwigbot/commands/chanops.py
+.. _afc_status:                     https://github.com/earwig/earwigbot-plugins/blob/main/commands/afc_status.py
+.. _earwigbot.tasks.Task:           https://github.com/earwig/earwigbot/blob/main/earwigbot/tasks/__init__.py
+.. _wikiproject_tagger:             https://github.com/earwig/earwigbot/blob/main/earwigbot/tasks/wikiproject_tagger.py
+.. _afc_statistics:                 https://github.com/earwig/earwigbot-plugins/blob/main/tasks/afc_statistics.py
+.. _its code and docstrings:        https://github.com/earwig/earwigbot/tree/main/earwigbot/wiki

docs/api/earwigbot.rst

@@ -15,6 +15,13 @@ earwigbot Package
     :members:
     :undoc-members:
 
+:mod:`cli` Module
+------------------
+
+.. automodule:: earwigbot.cli
+    :members:
+    :undoc-members:
+
 :mod:`exceptions` Module
 ------------------------
 
@@ -38,13 +45,6 @@ earwigbot Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`util` Module
-
-.. automodule:: earwigbot.util
-    :members:
-    :undoc-members:
-
 Subpackages
 -----------
 

docs/conf.py

@@ -226,7 +226,7 @@ texinfo_documents = [
         "EarwigBot Documentation",
         "Ben Kurtovic",
         "EarwigBot",
-        "EarwigBot is a Python robot that edits Wikipedia and interacts with people over IRC.",
+        "EarwigBot is a bot that edits Wikipedia and interacts over IRC",
         "Miscellaneous",
     ),
 ]

docs/customizing.rst

@@ -76,8 +76,8 @@ includes something like::
                 - "#channel"
                 - "#other-channel"
 
-...then :py:attr:`config.irc["frontend"]["nick"]` will be ``"MyAwesomeBot"``
-and :py:attr:`config.irc["frontend"]["channels"]` will be
+then :py:attr:`config.irc["frontend"]["nick"]` will be ``"MyAwesomeBot"`` and
+:py:attr:`config.irc["frontend"]["channels"]` will be
 ``["##earwigbot", "#channel", "#other-channel"]``.
 
 Custom IRC commands

docs/index.rst

@@ -1,25 +1,22 @@
 EarwigBot v0.4 Documentation
 ============================
 
-EarwigBot_ is a Python_ robot that edits Wikipedia_ and interacts with people
-over IRC_.
+EarwigBot_ is a Python bot that edits Wikipedia_ and interacts over IRC_.
 
 History
 -------
 
 Development began, based on `Pywikibot`_, in early 2009. Approval for its
 first task, a `copyright violation detector`_, was carried out in May, and the
-bot has been running consistently ever since (with the exception of Jan/Feb 2011).
-It currently handles `several ongoing tasks`_ ranging from statistics generation
-to category cleanup, and on-demand tasks such as WikiProject template tagging.
-Since it started running, the bot has made over 250,000 edits.
+bot has been running consistently ever since. It currently handles
+`several ongoing tasks`_ ranging from statistics generation to category
+cleanup, and on-demand tasks such as WikiProject template tagging. Since it
+started running, the bot has made over 300,000 edits.
 
-A project to rewrite it from scratch began in early April 2011, thus moving
-away from the Pywikibot framework and allowing for less overall code, better
-integration between bot parts, and easier maintenance.
+The current version of its codebase began development in April 2011, moving
+away from Pywikibot to a custom framework.
 
 .. _EarwigBot:                      https://en.wikipedia.org/wiki/User:EarwigBot
-.. _Python:                         https://python.org/
 .. _Wikipedia:                      https://en.wikipedia.org/
 .. _IRC:                            https://en.wikipedia.org/wiki/Internet_Relay_Chat
 .. _PyPI:                           https://packages.python.org/earwigbot

docs/installation.rst

@@ -1,38 +1,50 @@
 Installation
 ============
 
-This package contains the core :py:mod:`earwigbot`, abstracted enough that it
-should be usable and customizable by anyone running a bot on a MediaWiki site.
-Since it is component-based, the IRC components can be disabled if desired. IRC
-commands and bot tasks specific to `my instance of EarwigBot`_ that I don't
-feel the average user will need are available from the repository
-`earwigbot-plugins`_.
-
-It's recommended to run the bot's unit tests before installing. Run
-:command:`python setup.py test` from the project's root directory. Note that
-some tests require an internet connection, and others may take a while to run.
-Coverage is currently rather incomplete.
+This package contains the core :py:mod:`earwigbot`, abstracted to be usable
+and customizable by anyone running a bot on a MediaWiki site. Since it is
+modular, the IRC components can be disabled if desired. IRC commands and bot
+tasks specific to `my instance of EarwigBot`_ that I don't feel the average
+user will need are available from the repository `earwigbot-plugins`_.
 
 Latest release
 --------------
 
-EarwigBot is available from the `Python Package Index`_, so you can install the
-latest release with :command:`pip install earwigbot`.
+EarwigBot is available from the `Python Package Index`_, so you can install
+the latest release with:
 
-If you get an error while pip is installing dependencies, you may be missing
-some header files. For example, on Ubuntu, see `this StackOverflow post`_.
+    pip install earwigbot
+
+There are a few sets of optional dependencies:
+
+- ``crypto``: Allows encrypting bot passwords and secrets in the config
+- ``sql``: Allows interfacing with MediaWiki databases (e.g. on Toolforge_)
+- ``copyvios``: Includes parsing libraries for checking copyright violations
+- ``dev``: Installs development dependencies (e.g. test runners)
+
+For example, to install all non-dev dependencies:
+
+    pip install 'earwigbot[crypto,sql,copyvios]'
+
+Errors while pip is installing dependencies may be due to missing header
+files. For example, on Ubuntu, see `this StackOverflow post`_.
 
 Development version
 -------------------
 
-You can install the development version of the bot from :command:`git` by using
-setuptools's :command:`develop` command::
+You can install the development version of the bot::
 
-    git clone git://github.com/earwig/earwigbot.git earwigbot
+    git clone https://github.com/earwig/earwigbot.git
     cd earwigbot
-    python setup.py develop
+    python3 -m venv venv
+    . venv/bin/activate
+    pip install -e '.[crypto,sql,copyvios,dev]'
+
+To run the bot's unit tests, run :command:`pytest` (requires the ``dev``
+dependencies). Coverage is currently rather incomplete.
 
 .. _my instance of EarwigBot: https://en.wikipedia.org/wiki/User:EarwigBot
 .. _earwigbot-plugins:        https://github.com/earwig/earwigbot-plugins
 .. _Python Package Index:     https://pypi.python.org/pypi/earwigbot
+.. _Toolforge:                https://wikitech.wikimedia.org/wiki/Portal:Toolforge
 .. _this StackOverflow post:  https://stackoverflow.com/questions/6504810/how-to-install-lxml-on-ubuntu/6504860#6504860

docs/setup.rst

@@ -1,20 +1,19 @@
 Setup
 =====
 
-The bot stores its data in a "working directory", including its config file and
-databases. This is also the location where you will place custom IRC commands
-and bot tasks, which will be explained later. It doesn't matter where this
-directory is, as long as the bot can write to it.
+The bot stores its data in a "working directory", including its config file
+and databases. This is also the location where you will place custom IRC
+commands and bot tasks, which will be explained later. It doesn't matter where
+this directory is, as long as the bot can write to it.
 
 Start the bot with :command:`earwigbot path/to/working/dir`, or just
-:command:`earwigbot` if the working directory is the current directory. It will
-notice that no :file:`config.yml` file exists and take you through the setup
-process.
+:command:`earwigbot` if the working directory is the current directory. It
+will notice that no :file:`config.yml` file exists and take you through the
+setup process.
 
 There is currently no way to edit the :file:`config.yml` file from within the
-bot after it has been created, but YAML is a very straightforward format, so
-you should be able to make any necessary changes yourself. Check out the
-`explanation of YAML`_ on Wikipedia for help.
+bot after it has been created, but you should be able to make any necessary
+changes yourself.
 
 After setup, the bot will start. This means it will connect to the IRC servers
 it has been configured for, schedule bot tasks to run at specific times, and
@@ -24,5 +23,3 @@ then wait for instructions (as commands on IRC). For a list of commands, say
 You can stop the bot at any time with :kbd:`Control-c`, same as you stop a
 normal Python program, and it will try to exit safely. You can also use the
 "``!quit``" command on IRC.
-
-.. _explanation of YAML: https://en.wikipedia.org/wiki/YAML

docs/toolset.rst

@@ -6,7 +6,7 @@ EarwigBot's answer to `Pywikibot`_ is the Wiki Toolset
 :py:attr:`bot.wiki <earwigbot.bot.Bot.wiki>`.
 
 :py:attr:`bot.wiki <earwigbot.bot.Bot.wiki>` provides three methods for the
-management of Sites - :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.get_site`,
+management of Sites: :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.get_site`,
 :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.add_site`, and
 :py:meth:`~earwigbot.wiki.sitesdb.SitesDB.remove_site`. Sites are objects that
 simply represent a MediaWiki site. A single instance of EarwigBot (i.e. a

pyproject.toml

@@ -1,3 +1,63 @@
+[project]
+name = "earwigbot"
+version = "0.4.dev0"
+authors = [
+  {name = "Ben Kurtovic", email = "ben@benkurtovic.com"},
+]
+description = "EarwigBot is a bot that edits Wikipedia and interacts over IRC"
+readme = "README.rst"
+requires-python = ">=3.11"
+dependencies = [
+    "PyYAML >= 5.4.1",  # Parsing config files
+    "mwparserfromhell >= 0.6",  # Parsing wikicode for manipulation
+    "requests >= 2.25.1",  # Wiki API requests
+    "requests_oauthlib >= 1.3.0",  # API authentication via OAuth
+]
+keywords = ["earwig", "earwigbot", "irc", "wikipedia", "wiki", "mediawiki"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Natural Language :: English",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Topic :: Communications :: Chat :: Internet Relay Chat",
+    "Topic :: Internet :: WWW/HTTP",
+]
+
+[project.optional-dependencies]
+crypto = [
+    "cryptography >= 3.4.7",  # Storing bot passwords + keys in the config file
+]
+sql = [
+    "pymysql >= 1.1.0",  # Interfacing with MediaWiki databases
+]
+copyvios = [
+    "beautifulsoup4 >= 4.9.3",  # Parsing/scraping HTML
+    "charset_normalizer >= 3.3.2",  # Encoding detection for BeautifulSoup
+    "lxml >= 4.6.3",  # Faster parser for BeautifulSoup
+    "nltk >= 3.6.1",  # Parsing sentences to split article content
+    "pdfminer >= 20191125",  # Extracting text from PDF files
+    "tldextract >= 3.1.0",  # Getting domains for the multithreaded workers
+]
+dev = [
+    "pytest >= 8.3.1"
+]
+
+[project.urls]
+Homepage = "https://github.com/earwig/earwigbot"
+Documentation = "https://pythonhosted.org/earwigbot/"
+Issues = "https://github.com/earwig/earwigbot/issues"
+Changelog = "https://github.com/earwig/earwigbot/blob/main/CHANGELOG"
+
+[project.scripts]
+earwigbot = "earwigbot.cli:main"
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
 [tool.ruff]
 target-version = "py311"
 

setup.py

@@ -1,84 +0,0 @@
-#! /usr/bin/env python
-# Copyright (C) 2009-2024 Ben Kurtovic <ben.kurtovic@gmail.com>
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-from setuptools import find_packages, setup
-
-from earwigbot import __version__
-
-required_deps = [
-    "PyYAML >= 5.4.1",  # Parsing config files
-    "mwparserfromhell >= 0.6",  # Parsing wikicode for manipulation
-    "requests >= 2.25.1",  # Wiki API requests
-    "requests_oauthlib >= 1.3.0",  # API authentication via OAuth
-]
-
-extra_deps = {
-    "crypto": [
-        "cryptography >= 3.4.7",  # Storing bot passwords + keys in the config file
-    ],
-    "sql": [
-        "pymysql >= 1.1.0",  # Interfacing with MediaWiki databases
-    ],
-    "copyvios": [
-        "beautifulsoup4 >= 4.9.3",  # Parsing/scraping HTML
-        "charset_normalizer >= 3.3.2",  # Encoding detection for BeautifulSoup
-        "lxml >= 4.6.3",  # Faster parser for BeautifulSoup
-        "nltk >= 3.6.1",  # Parsing sentences to split article content
-        "pdfminer >= 20191125",  # Extracting text from PDF files
-        "tldextract >= 3.1.0",  # Getting domains for the multithreaded workers
-    ],
-    "time": [
-        "pytz >= 2021.1",  # Handling timezones for the !time IRC command
-    ],
-}
-
-dependencies = required_deps + sum(extra_deps.values(), [])
-
-with open("README.rst") as fp:
-    long_docs = fp.read()
-
-setup(
-    name="earwigbot",
-    packages=find_packages(exclude=("tests",)),
-    entry_points={"console_scripts": ["earwigbot = earwigbot.util:main"]},
-    install_requires=dependencies,
-    test_suite="tests",
-    version=__version__,
-    author="Ben Kurtovic",
-    author_email="ben.kurtovic@gmail.com",
-    url="https://github.com/earwig/earwigbot",
-    description="EarwigBot is a Python robot that edits Wikipedia and interacts with people over IRC.",
-    long_description=long_docs,
-    download_url=f"https://github.com/earwig/earwigbot/tarball/v{__version__}",
-    keywords="earwig earwigbot irc wikipedia wiki mediawiki",
-    license="MIT License",
-    classifiers=[
-        "Development Status :: 3 - Alpha",
-        "Environment :: Console",
-        "Intended Audience :: Developers",
-        "License :: OSI Approved :: MIT License",
-        "Natural Language :: English",
-        "Operating System :: OS Independent",
-        "Programming Language :: Python :: 3",
-        "Topic :: Communications :: Chat :: Internet Relay Chat",
-        "Topic :: Internet :: WWW/HTTP",
-    ],
-)

earwigbot/__init__.py → src/earwigbot/__init__.py

@@ -61,23 +61,23 @@ importer = lazy.LazyImporter()
 if typing.TYPE_CHECKING:
     from earwigbot import (
         bot,
+        cli,
         commands,
         config,
         exceptions,
         irc,
         managers,
         tasks,
-        util,
         wiki,
     )
 
 else:
     bot = importer.new("earwigbot.bot")
+    cli = importer.new("earwigbot.cli")
     commands = importer.new("earwigbot.commands")
     config = importer.new("earwigbot.config")
     exceptions = importer.new("earwigbot.exceptions")
     irc = importer.new("earwigbot.irc")
     managers = importer.new("earwigbot.managers")
     tasks = importer.new("earwigbot.tasks")
-    util = importer.new("earwigbot.util")
     wiki = importer.new("earwigbot.wiki")

earwigbot/bot.py → src/earwigbot/bot.py

@@ -59,7 +59,7 @@ class Bot:
     :py:meth:`bot.wiki.get_site() <earwigbot.wiki.sitesdb.SitesDB.get_site>`.
     """
 
-    def __init__(self, root_dir, level=logging.INFO):
+    def __init__(self, root_dir: str, level=logging.INFO):
         self.config = BotConfig(self, root_dir, level)
         self.logger = logging.getLogger("earwigbot")
         self.commands = CommandManager(self)

earwigbot/util.py → src/earwigbot/cli.py

@@ -64,6 +64,7 @@ class _StoreTaskArg(Action):
     def __call__(self, parser, namespace, values, option_string=None):
         kwargs = {}
         name = None
+        assert isinstance(values, list | tuple), values
         for value in values:
             if value.startswith("-") and "=" in value:
                 key, value = value.split("=", 1)

earwigbot/commands/time_command.py → src/earwigbot/commands/time_command.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2009-2015 Ben Kurtovic <ben.kurtovic@gmail.com>
+# Copyright (C) 2009-2024 Ben Kurtovic <ben.kurtovic@gmail.com>
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -18,14 +18,13 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from datetime import datetime
-from math import floor
-from time import time
+import math
+import time
+from datetime import datetime, timezone
+from zoneinfo import ZoneInfo
 
-from earwigbot import importer
 from earwigbot.commands import Command
-
-pytz = importer.new("pytz")
+from earwigbot.irc import Data
 
 
 class Time(Command):
@@ -35,12 +34,12 @@ class Time(Command):
     name = "time"
     commands = ["time", "beats", "swatch", "epoch", "date"]
 
-    def process(self, data):
+    def process(self, data: Data) -> None:
         if data.command in ["beats", "swatch"]:
             self.do_beats(data)
             return
         if data.command == "epoch":
-            self.reply(data, time())
+            self.reply(data, time.time())
             return
         if data.args:
             timezone = data.args[0]
@@ -51,20 +50,16 @@ class Time(Command):
         else:
             self.do_time(data, timezone)
 
-    def do_beats(self, data):
-        beats = ((time() + 3600) % 86400) / 86.4
-        beats = int(floor(beats))
+    def do_beats(self, data: Data) -> None:
+        beats = ((time.time() + 3600) % 86400) / 86.4
+        beats = int(math.floor(beats))
         self.reply(data, f"@{beats:0>3}")
 
-    def do_time(self, data, timezone):
+    def do_time(self, data: Data, tzname: str) -> None:
         try:
-            tzinfo = pytz.timezone(timezone)
-        except ImportError:
-            msg = "This command requires the 'pytz' package: https://pypi.org/project/pytz/"
-            self.reply(data, msg)
-            return
-        except pytz.exceptions.UnknownTimeZoneError:
-            self.reply(data, f"Unknown timezone: {timezone}.")
+            tzinfo = ZoneInfo(tzname)
+        except LookupError:
+            self.reply(data, f"Unknown timezone: {timezone}")
             return
-        now = pytz.utc.localize(datetime.utcnow()).astimezone(tzinfo)
+        now = datetime.now(tz=tzinfo)
         self.reply(data, now.strftime("%Y-%m-%d %H:%M:%S %Z"))

earwigbot/managers.py → src/earwigbot/managers.py

@@ -263,7 +263,9 @@ class TaskManager(_ResourceManager):
             msg = "Task '{0}' finished successfully"
             self.logger.info(msg.format(task.name))
         if kwargs.get("fromIRC"):
-            kwargs.get("_IRCCallback")()
+            callback = kwargs.get("_IRCCallback")
+            assert callable(callback), callback
+            callback()
 
     def start(self, task_name, **kwargs):
         """Start a given task in a new daemon thread, and return the thread.
@@ -299,7 +301,9 @@ class TaskManager(_ResourceManager):
         )
 
         for task in tasks:
-            if isinstance(task, list):  # They've specified kwargs,
-                self.start(task[0], **task[1])  # so pass those to start
-            else:  # Otherwise, just pass task_name
+            if isinstance(task, list):
+                # They've specified kwargs, so pass those to start
+                self.start(task[0], **task[1])
+            else:
+                # Otherwise, just pass task_name
                 self.start(task)

tests/__init__.py

@@ -1,147 +0,0 @@
-# Copyright (C) 2009-2015 Ben Kurtovic <ben.kurtovic@gmail.com>
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-"""
-EarwigBot's Unit Tests
-
-This __init__ file provides some support code for unit tests.
-
-Test cases:
-  -- CommandTestCase provides setUp() for creating a fake connection, plus
-     some other helpful methods for testing IRC commands.
-
-Fake objects:
-  -- FakeBot implements Bot, using the Fake* equivalents of all objects
-     whenever possible.
-  -- FakeBotConfig implements BotConfig with silent logging.
-  -- FakeIRCConnection implements IRCConnection, using an internal string
-     buffer for data instead of sending it over a socket.
-
-"""
-
-import logging
-import re
-from os import path
-from threading import Lock
-from unittest import TestCase
-
-from earwigbot.bot import Bot
-from earwigbot.commands import CommandManager
-from earwigbot.config import BotConfig
-from earwigbot.irc import Data, IRCConnection
-from earwigbot.tasks import TaskManager
-from earwigbot.wiki import SitesDB
-
-
-class CommandTestCase(TestCase):
-    re_sender = re.compile(":(.*?)!(.*?)@(.*?)\Z")
-
-    def setUp(self, command):
-        self.bot = FakeBot(path.dirname(__file__))
-        self.command = command(self.bot)
-        self.command.connection = self.connection = self.bot.frontend
-
-    def get_single(self):
-        data = self.connection._get().split("\n")
-        line = data.pop(0)
-        for remaining in data[1:]:
-            self.connection.send(remaining)
-        return line
-
-    def assertSent(self, msg):
-        line = self.get_single()
-        self.assertEqual(line, msg)
-
-    def assertSentIn(self, msgs):
-        line = self.get_single()
-        self.assertIn(line, msgs)
-
-    def assertSaid(self, msg):
-        self.assertSent(f"PRIVMSG #channel :{msg}")
-
-    def assertSaidIn(self, msgs):
-        msgs = [f"PRIVMSG #channel :{msg}" for msg in msgs]
-        self.assertSentIn(msgs)
-
-    def assertReply(self, msg):
-        self.assertSaid(f"\x02Foo\x0f: {msg}")
-
-    def assertReplyIn(self, msgs):
-        msgs = [f"\x02Foo\x0f: {msg}" for msg in msgs]
-        self.assertSaidIn(msgs)
-
-    def maker(self, line, chan, msg=None):
-        data = Data(line)
-        data.nick, data.ident, data.host = self.re_sender.findall(line[0])[0]
-        if msg is not None:
-            data.msg = msg
-        data.chan = chan
-        data.parse_args()
-        return data
-
-    def make_msg(self, command, *args):
-        line = f":Foo!bar@example.com PRIVMSG #channel :!{command}"
-        line = line.strip().split()
-        line.extend(args)
-        return self.maker(line, line[2], " ".join(line[3:])[1:])
-
-    def make_join(self):
-        line = ":Foo!bar@example.com JOIN :#channel".strip().split()
-        return self.maker(line, line[2][1:])
-
-
-class FakeBot(Bot):
-    def __init__(self, root_dir):
-        self.config = FakeBotConfig(root_dir)
-        self.logger = logging.getLogger("earwigbot")
-        self.commands = CommandManager(self)
-        self.tasks = TaskManager(self)
-        self.wiki = SitesDB(self)
-        self.frontend = FakeIRCConnection(self)
-        self.watcher = FakeIRCConnection(self)
-
-        self.component_lock = Lock()
-        self._keep_looping = True
-
-
-class FakeBotConfig(BotConfig):
-    def _setup_logging(self):
-        logger = logging.getLogger("earwigbot")
-        logger.addHandler(logging.NullHandler())
-
-
-class FakeIRCConnection(IRCConnection):
-    def __init__(self, bot):
-        self.bot = bot
-        self._is_running = False
-        self._connect()
-
-    def _connect(self):
-        self._buffer = ""
-
-    def _close(self):
-        self._buffer = ""
-
-    def _get(self, size=4096):
-        data, self._buffer = self._buffer, ""
-        return data
-
-    def _send(self, msg):
-        self._buffer += msg + "\n"

tests/conftest.py

@@ -0,0 +1,151 @@
+# Copyright (C) 2009-2024 Ben Kurtovic <ben.kurtovic@gmail.com>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+"""
+EarwigBot's Unit Tests
+
+This conftest provides some support code for unit tests.
+
+Fixtures:
+  -- ``command`` creates a mock connection and provides some helpful methods for
+     testing IRC commands.
+
+Mock objects:
+  -- ``MockBot`` implements ``Bot``, using the ``Mock`` equivalents of all objects
+     whenever possible.
+  -- ``MockBotConfig`` implements ``BotConfig`` with silent logging.
+  -- ``MockIRCConnection`` implements ``IRCConnection``, using an internal string
+     buffer for data instead of sending it over a socket.
+"""
+
+import logging
+import os.path
+import re
+from collections.abc import Iterable, Sequence
+from threading import Lock
+
+import pytest
+from earwigbot.bot import Bot
+from earwigbot.commands import Command
+from earwigbot.config import BotConfig
+from earwigbot.irc import Data, IRCConnection
+from earwigbot.managers import CommandManager, TaskManager
+from earwigbot.wiki import SitesDB
+
+
+@pytest.fixture
+def command():
+    return MockCommand()
+
+
+class MockCommand:
+    re_sender = re.compile(r":(.*?)!(.*?)@(.*?)\Z")
+
+    def setup(self, command: type[Command]) -> None:
+        self.bot = MockBot(os.path.dirname(__file__))
+        self.command = command(self.bot)
+
+    def get_single(self) -> str:
+        data = self.bot.frontend._get().split("\n")
+        line = data.pop(0)
+        for remaining in data[1:]:
+            self.bot.frontend._send(remaining)
+        return line
+
+    def assert_sent(self, msg: str) -> None:
+        line = self.get_single()
+        assert line == msg
+
+    def assert_sent_in(self, msgs: Iterable[str]) -> None:
+        line = self.get_single()
+        assert line in msgs
+
+    def assert_said(self, msg: str) -> None:
+        self.assert_sent(f"PRIVMSG #channel :{msg}")
+
+    def assert_said_in(self, msgs: Iterable[str]) -> None:
+        msgs = [f"PRIVMSG #channel :{msg}" for msg in msgs]
+        self.assert_sent_in(msgs)
+
+    def assert_reply(self, msg: str) -> None:
+        self.assert_said(f"\x02Foo\x0f: {msg}")
+
+    def assert_reply_in(self, msgs: Iterable[str]) -> None:
+        msgs = [f"\x02Foo\x0f: {msg}" for msg in msgs]
+        self.assert_said_in(msgs)
+
+    def _make(self, line: Sequence[str]) -> Data:
+        return Data(self.bot.frontend.nick, line, line[1])
+
+    def make_msg(self, command, *args):
+        line = f":Foo!bar@example.com PRIVMSG #channel :!{command}"
+        line = line.strip().split()
+        line.extend(args)
+        return self._make(line)
+
+    def make_join(self):
+        line = ":Foo!bar@example.com JOIN :#channel".strip().split()
+        return self._make(line)
+
+
+class MockBot(Bot):
+    def __init__(self, root_dir: str, level=logging.INFO) -> None:
+        self.config = MockBotConfig(self, root_dir, level)
+        self.logger = logging.getLogger("earwigbot")
+        self.commands = CommandManager(self)
+        self.tasks = TaskManager(self)
+        self.wiki = SitesDB(self)
+        self.frontend = MockIRCConnection(self)
+        self.watcher = MockIRCConnection(self)
+
+        self.component_lock = Lock()
+        self._keep_looping = True
+
+
+class MockBotConfig(BotConfig):
+    def _setup_logging(self) -> None:
+        logger = logging.getLogger("earwigbot")
+        logger.addHandler(logging.NullHandler())
+
+
+class MockIRCConnection(IRCConnection):
+    def __init__(self, bot: MockBot) -> None:
+        super().__init__(
+            "localhost",
+            6667,
+            "MockBot",
+            "mock",
+            "Mock Bot",
+            bot.logger.getChild("mock"),
+        )
+        self._buffer = ""
+
+    def _connect(self) -> None:
+        self._buffer = ""
+
+    def _close(self) -> None:
+        self._buffer = ""
+
+    def _get(self, size: int = 4096) -> str:
+        data, self._buffer = self._buffer, ""
+        return data
+
+    def _send(self, msg: str, hidelog: bool = False) -> None:
+        self._buffer += msg + "\n"

tests/test_calc.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2009-2015 Ben Kurtovic <ben.kurtovic@gmail.com>
+# Copyright (C) 2009-2024 Ben Kurtovic <ben.kurtovic@gmail.com>
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -18,41 +18,41 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-import unittest
+import pytest
+from conftest import MockCommand
+from earwigbot.commands.calc import Calc
 
-from earwigbot.commands.calc import Command
-from tests import CommandTestCase
 
+def test_check(command: MockCommand):
+    command.setup(Calc)
 
-class TestCalc(CommandTestCase):
-    def setUp(self):
-        super().setUp(Command)
+    assert command.command.check(command.make_msg("bloop")) is False
+    assert command.command.check(command.make_join()) is False
 
-    def test_check(self):
-        self.assertFalse(self.command.check(self.make_msg("bloop")))
-        self.assertFalse(self.command.check(self.make_join()))
+    assert command.command.check(command.make_msg("calc")) is True
+    assert command.command.check(command.make_msg("CALC", "foo")) is True
 
-        self.assertTrue(self.command.check(self.make_msg("calc")))
-        self.assertTrue(self.command.check(self.make_msg("CALC", "foo")))
 
-    def test_ignore_empty(self):
-        self.command.process(self.make_msg("calc"))
-        self.assertReply("what do you want me to calculate?")
+def test_ignore_empty(command: MockCommand):
+    command.setup(Calc)
 
-    def test_maths(self):
-        tests = [
-            ("2 + 2", "2 + 2 = 4"),
-            ("13 * 5", "13 * 5 = 65"),
-            ("80 / 42", "80 / 42 = 40/21 (approx. 1.9047619047619047)"),
-            ("2/0", "2/0 = undef"),
-            ("π", "π = 3.141592653589793238"),
-        ]
+    command.command.process(command.make_msg("calc"))
+    command.assert_reply("What do you want me to calculate?")
 
-        for test in tests:
-            q = test[0].strip().split()
-            self.command.process(self.make_msg("calc", *q))
-            self.assertReply(test[1])
 
+@pytest.mark.parametrize(
+    "expr, expected",
+    [
+        ("2 + 2", "2 + 2 = 4"),
+        ("13 * 5", "13 * 5 = 65"),
+        ("80 / 42", "80 / 42 = 40/21 (approx. 1.9047619047619048)"),
+        ("2/0", "2/0 = undef"),
+        ("π", "π = 3.141592653589793238"),
+    ],
+)
+def test_math(command: MockCommand, expr: str, expected: str):
+    command.setup(Calc)
 
-if __name__ == "__main__":
-    unittest.main(verbosity=2)
+    q = expr.strip().split()
+    command.command.process(command.make_msg("calc", *q))
+    command.assert_reply(expected)

tests/test_test.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2009-2015 Ben Kurtovic <ben.kurtovic@gmail.com>
+# Copyright (C) 2009-2024 Ben Kurtovic <ben.kurtovic@gmail.com>
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -18,31 +18,23 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-import unittest
+from conftest import MockCommand
+from earwigbot.commands.test import Test
 
-from earwigbot.commands.test import Command
-from tests import CommandTestCase
 
+def test_check(command: MockCommand):
+    command.setup(Test)
 
-class TestTest(CommandTestCase):
-    def setUp(self):
-        super().setUp(Command)
+    assert command.command.check(command.make_msg("bloop")) is False
+    assert command.command.check(command.make_join()) is False
 
-    def test_check(self):
-        self.assertFalse(self.command.check(self.make_msg("bloop")))
-        self.assertFalse(self.command.check(self.make_join()))
+    assert command.command.check(command.make_msg("test")) is True
+    assert command.command.check(command.make_msg("TEST", "foo")) is True
 
-        self.assertTrue(self.command.check(self.make_msg("test")))
-        self.assertTrue(self.command.check(self.make_msg("TEST", "foo")))
 
-    def test_process(self):
-        def test():
-            self.command.process(self.make_msg("test"))
-            self.assertSaidIn(["Hey \x02Foo\x0f!", "'sup \x02Foo\x0f?"])
+def test_process(command: MockCommand):
+    command.setup(Test)
 
-        for i in range(64):
-            test()
-
-
-if __name__ == "__main__":
-    unittest.main(verbosity=2)
+    for i in range(64):
+        command.command.process(command.make_msg("test"))
+        command.assert_said_in(["Hey \x02Foo\x0f!", "'Sup \x02Foo\x0f?"])