earwigbot

Граф коммитов

Автор	SHA1	Сообщение	Дата
Ben Kurtovic	4e8be871b7	Update copyright year for 2015.	9 лет назад
Ben Kurtovic	9ffc3f1bf5	Raise file crawl size limit for PDFs.	9 лет назад
Ben Kurtovic	901192ec18	Handle errors from UnicodeDamnit.	9 лет назад
Ben Kurtovic	699f6e3b17	Seems it will sometimes raise AssertionError.	9 лет назад
Ben Kurtovic	12c5170815	Catch another exception thrown by pdfminer.	9 лет назад
Ben Kurtovic	77514ee925	Add another PDF string substitution.	9 лет назад
Ben Kurtovic	0bdcbca8b0	Rudimentary solution for PDF parsing (closes earwig/copyvios#18 )	9 лет назад
Ben Kurtovic	30f72df470	Refactor parsers; fix empty document behavior.	9 лет назад
Ben Kurtovic	5349179088	Fix parsing of plain text documents (earwig/copyvios#3 )	9 лет назад
Ben Kurtovic	7afb484cea	Refactor a bunch of copyvio internals. Store all sources with a result object.	9 лет назад
Ben Kurtovic	54ddff049f	Make CopyvioSource public; tweaks.	9 лет назад
Ben Kurtovic	9b87e2e5f7	Fix trying to remove a node that was already removed.	9 лет назад
Ben Kurtovic	193f96451e	Also strip <ref>s in ArticleTextParser.strip().	9 лет назад
Ben Kurtovic	c4dede1459	Reorder length check to potentially fix an empty-query bug.	9 лет назад
Ben Kurtovic	5874467ec3	Bugfix, cleanup.	9 лет назад
Ben Kurtovic	ae0c390ceb	Redesign copyvio internals to parallelize URL loading/parsing.	9 лет назад
Ben Kurtovic	3e4dac967d	Remove auto-quotes from queries; add min_query; halve max_query.	9 лет назад
Ben Kurtovic	6b146a397a	Also strip out files and categories in ATP.strip().	9 лет назад
Ben Kurtovic	2dfdf1bd4a	Ensure the text is stripped properly.	9 лет назад
Ben Kurtovic	3dde1c5d60	Correctly handle HTML with no <body> tags.	10 лет назад
Ben Kurtovic	39d5c7c149	Update copyright notices for 2014.	10 лет назад
Ben Kurtovic	ed95c99f0e	Update email address.	10 лет назад
Ben Kurtovic	0b7a13eca5	Update copyright notices for 2013.	11 лет назад
Ben Kurtovic	8862bec3d9	Fix statements assigned to nothing.	11 лет назад
Ben Kurtovic	a4dda89a61	Various fixes for copyvios. - Fix a bug in ExclusionsDB; improve URL regexes. - NLTK's LookupError is actually an IOError. - Fix bug in __repr__ for CopyvioCheckResult. - Rewrite YahooBOSSSearchEngine to actually work with oauth2. - Search engines now take a URL opener in addition to credentials.	11 лет назад
Ben Kurtovic	0ca84ab9bc	Implement lazy-importing of oauth2, nltk, and bs4.	11 лет назад
Ben Kurtovic	4baab6f57c	Implement lazy importing of root-level modules and packages. - Simplify all imports - Update dependency version in setup.py - Change waitTime default from three seconds to two	11 лет назад
Ben Kurtovic	7d52d4558a	Some updates to !lag.	11 лет назад
Ben Kurtovic	33aa1d6744	Collapse extra newlines to avoid distorting trigrams.	11 лет назад
Ben Kurtovic	5ab736723b	Fixing a number of silly mistakes; refactoring (thanks pylint)	12 лет назад
Ben Kurtovic	c260648bdb	Finish chunking algorithm, improve !link, other fixes.	12 лет назад
Ben Kurtovic	569c815d99	Implement NLTK for chunking article content (#5 ).	12 лет назад
Ben Kurtovic	cb87004107	Primitive screen scraper for HTML using BeautifulSoup and LXML. Obviously this can and should be improved significantly later, but it seems good enough for now.	12 лет назад
Ben Kurtovic	1af4217b63	Update copyright notices and some other improvements.	12 лет назад
Ben Kurtovic	d45e342bac	DOCUMENT EVERYTHING (#5 ) Also implementing MWParserFromHell, plus some cleanup.	12 лет назад
Ben Kurtovic	d87c226417	__repr__ and __str__ for everything per #5 and #22 .	12 лет назад
Ben Kurtovic	5ca1d91f3e	Use __all__ within e.w.copyvios and shorter imports	12 лет назад
Ben Kurtovic	86a8440730	Moving parsers to own file.	12 лет назад

38 коммитов (4e8be871b7d57e5512854240baefd1d3bf4268be)