Ben Kurtovic
4e8be871b7
Update copyright year for 2015.
9 роки тому
Ben Kurtovic
9ffc3f1bf5
Raise file crawl size limit for PDFs.
9 роки тому
Ben Kurtovic
901192ec18
Handle errors from UnicodeDamnit.
9 роки тому
Ben Kurtovic
699f6e3b17
Seems it will sometimes raise AssertionError.
9 роки тому
Ben Kurtovic
12c5170815
Catch another exception thrown by pdfminer.
9 роки тому
Ben Kurtovic
77514ee925
Add another PDF string substitution.
9 роки тому
Ben Kurtovic
0bdcbca8b0
Rudimentary solution for PDF parsing ( closes earwig/copyvios#18 )
9 роки тому
Ben Kurtovic
30f72df470
Refactor parsers; fix empty document behavior.
9 роки тому
Ben Kurtovic
5349179088
Fix parsing of plain text documents ( earwig/copyvios#3 )
9 роки тому
Ben Kurtovic
7afb484cea
Refactor a bunch of copyvio internals. Store all sources with a result object.
9 роки тому
Ben Kurtovic
54ddff049f
Make CopyvioSource public; tweaks.
9 роки тому
Ben Kurtovic
9b87e2e5f7
Fix trying to remove a node that was already removed.
9 роки тому
Ben Kurtovic
193f96451e
Also strip <ref>s in ArticleTextParser.strip().
9 роки тому
Ben Kurtovic
c4dede1459
Reorder length check to potentially fix an empty-query bug.
9 роки тому
Ben Kurtovic
5874467ec3
Bugfix, cleanup.
9 роки тому
Ben Kurtovic
ae0c390ceb
Redesign copyvio internals to parallelize URL loading/parsing.
9 роки тому
Ben Kurtovic
3e4dac967d
Remove auto-quotes from queries; add min_query; halve max_query.
9 роки тому
Ben Kurtovic
6b146a397a
Also strip out files and categories in ATP.strip().
9 роки тому
Ben Kurtovic
2dfdf1bd4a
Ensure the text is stripped properly.
9 роки тому
Ben Kurtovic
3dde1c5d60
Correctly handle HTML with no <body> tags.
10 роки тому
Ben Kurtovic
39d5c7c149
Update copyright notices for 2014.
10 роки тому
Ben Kurtovic
ed95c99f0e
Update email address.
10 роки тому
Ben Kurtovic
0b7a13eca5
Update copyright notices for 2013.
11 роки тому
Ben Kurtovic
8862bec3d9
Fix statements assigned to nothing.
11 роки тому
Ben Kurtovic
a4dda89a61
Various fixes for copyvios.
- Fix a bug in ExclusionsDB; improve URL regexes.
- NLTK's LookupError is actually an IOError.
- Fix bug in __repr__ for CopyvioCheckResult.
- Rewrite YahooBOSSSearchEngine to actually work with oauth2.
- Search engines now take a URL opener in addition to credentials.
11 роки тому
Ben Kurtovic
0ca84ab9bc
Implement lazy-importing of oauth2, nltk, and bs4.
11 роки тому
Ben Kurtovic
4baab6f57c
Implement lazy importing of root-level modules and packages.
- Simplify all imports
- Update dependency version in setup.py
- Change waitTime default from three seconds to two
11 роки тому
Ben Kurtovic
7d52d4558a
Some updates to !lag.
11 роки тому
Ben Kurtovic
33aa1d6744
Collapse extra newlines to avoid distorting trigrams.
11 роки тому
Ben Kurtovic
5ab736723b
Fixing a number of silly mistakes; refactoring (thanks pylint)
12 роки тому
Ben Kurtovic
c260648bdb
Finish chunking algorithm, improve !link, other fixes.
12 роки тому
Ben Kurtovic
569c815d99
Implement NLTK for chunking article content ( #5 ).
12 роки тому
Ben Kurtovic
cb87004107
Primitive screen scraper for HTML using BeautifulSoup and LXML.
Obviously this can and should be improved significantly later, but it seems
good enough for now.
12 роки тому
Ben Kurtovic
1af4217b63
Update copyright notices and some other improvements.
12 роки тому
Ben Kurtovic
d45e342bac
DOCUMENT EVERYTHING ( #5 )
Also implementing MWParserFromHell, plus some cleanup.
12 роки тому
Ben Kurtovic
d87c226417
__repr__ and __str__ for everything per #5 and #22 .
12 роки тому
Ben Kurtovic
5ca1d91f3e
Use __all__ within e.w.copyvios and shorter imports
12 роки тому
Ben Kurtovic
86a8440730
Moving parsers to own file.
12 роки тому