Personal website https://benkurtovic.com/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2012-01-29-earwigbot-and-toolserver-updates.md 5.0 KiB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990
  1. ---
  2. layout: post
  3. title: EarwigBot and Toolserver Updates
  4. description: More progress on EarwigBot and the Toolserver site rewrite, including dynamic backgrounds.
  5. ---
  6. Haven't really said much in a while, so I felt it appropriate to make a new
  7. blog post. No, I'm not dead, and yes, I _am_ busy working on my Wikipedia
  8. responsibilities, including EarwigBot and my Toolserver site (which I'll go
  9. into detail about in a bit). Progress is not as quick now as it was over the
  10. summer, and you can blame school for the delay in getting things done. My
  11. primary to-do list for Wikipedia right now looks like this:
  12. 1. Finish copyvio detection in the new EarwigBot
  13. 2. Integrate EarwigBot's copyvio detection with the new Toolserver site
  14. \#1 has a good portion of its work done, but I still have to finish the actual
  15. detection. I've isolated the work down to a few methods of
  16. `earwigbot.wiki.copyright.CopyrightMixin`: `_copyvio_strip_html()`, to
  17. extract the "text" (i.e. content inside <p> tags) from an HTML document
  18. (I'll probably use something like
  19. [Beautiful Soup](http://www.crummy.com/software/BeautifulSoup/) for this);
  20. `_copyvio_strip_article()`, to extract the "text" from an article (that is,
  21. stripping templates, quotes, references); and `_copyvio_chunk_article()`, to
  22. divide a stripped article into a list of web-searchable queries. Everything
  23. else, including the Task class for `afc_copyvios` is done.
  24. \#2 is very simple once #1 is done. I've already written the code to load
  25. EarwigBot's wiki toolset from `copyvios.mako`, and the config file is written,
  26. so running the detector is trivial once it works. The only thing left here is
  27. to have the tool produce relatively eye-pleasing output, perhaps with a
  28. "details" section showing the Markov chains formed from the two sources and
  29. comparing them visually. Not necessary at all, but a nice touch.
  30. Unfortunately, there's still a bit more work to do on EarwigBot before he's
  31. ready for his first release (0.1!). Aside from the copyvio stuff above, which
  32. is integrated directly as a function of `Page`, I want to finish porting over
  33. the remaining tasks from old EarwigBot that are still running via cron, improve
  34. the Wiki Toolset such that new sites can be added programmatically, and improve
  35. config such that it can be created by the bot and not only by hand. This is the
  36. main barrier stopping other people from running EarwigBot, and thus the primary
  37. concern before v0.1 is good. Of course, none of this urgent; getting copyvio
  38. detection finished is my primary concern.
  39. ## Dynamic Backgrounds
  40. Now that that's covered, let's look at something (mostly unrelated) I finished
  41. a couple days ago: dynamic backgrounds for
  42. [my new toolserver site](http://toolserver.org/~earwig/rewrite)! You can see it
  43. in action a bit better on [this page](http://toolserver.org/~earwig/earwigbot).
  44. The background is the the [Wikimedia Commons](//commons.wikimedia.org/)
  45. [Picture of the Day](//commons.wikimedia.org/wiki/Commons:Picture_of_the_day),
  46. loaded and displayed with JavaScript.
  47. [Here's the code for it](//github.com/earwig/toolserver/blob/master/static/js/potd.js),
  48. a good deal more code than I had expected to write.
  49. Here's what we have to do:
  50. 1. Query Commons' API for the content of the template \{\{Potd/YYY-MM-DD}}.
  51. 2. Parse that content for the filename of the image, which will be hidden in
  52. something like \{\{Potd filename|1=Foo.png}}.
  53. 3. Query Commons' API again for Foo.png's URL and dimensions.
  54. 4. Since we want the image to "cover" the background (that is, be the smallest
  55. size possible while leaving none of the background color visible), we need
  56. to calculate the image's aspect ratio and our own aspect ratio, then
  57. determine the width of the thumbnail we want. If the image is shorter than
  58. our screen, the necessary width is our screen's width, but if the image is
  59. longer than our screen, the necessary width is our screen's height
  60. multiplied by the image's aspect ratio.
  61. 5. If the width of our desired thumbnail is less than the width of the image,
  62. we'll alter the image's URL (insert a `/thumb/` in the middle somewhere and
  63. add a resolution at the end) and set that as our `body`'s `background-image`
  64. property. This is better than loading the full image and downscaling it,
  65. because less bandwidth is required.
  66. 6. If the width of our desired thumbnail is _greater_ than (or equal to) the
  67. width of the image, we load the full image and upscale it (gasp! horror!)
  68. using the CSS bit `background-size: cover;`.
  69. All of the images I tested looked decent when displayed under this method, some
  70. better than others, but all acceptable. I figured this code provided a nice
  71. touch to an otherwise drab webpage (like the one you're viewing now, it
  72. wouldn't have been very pretty), which is why I did it, but I couldn't help but
  73. wonder if there was an... easier... method that still saved bandwidth and
  74. didn't resort to ugly scaling/cropping/repeating/whatever, but I could come up
  75. with nothing. It was a fun project in a language I almost never use, though, so
  76. worth it in the end.
  77. That's all for now!
  78. :—earwig