 Ben Kurtovic
|
48a14ee3ed
|
Don't log the full debug line when sending a lot of data.
|
9 years ago |
|
Ben Kurtovic
|
eceb4d139a
|
Minor refactor.
|
9 years ago |
|
Ben Kurtovic
|
f92fb34d0e
|
Improve sentence splitting, again.
|
9 years ago |
|
Ben Kurtovic
|
75058997c2
|
Split copyvio queries a bit differently; maybe better on other languages.
|
9 years ago |
|
Ben Kurtovic
|
f52fb06c19
|
Add a debug message when catching ParserExclusionError.
|
9 years ago |
|
Ben Kurtovic
|
c81d1d949d
|
Update global exclusion lists more often than site-specific ones.
|
9 years ago |
|
Ben Kurtovic
|
108eca13ac
|
Finish mirror hinting algorithm.
|
9 years ago |
|
Ben Kurtovic
|
91846ce4fb
|
Refactor out mirror hinting logic in source parsers.
|
9 years ago |
|
Ben Kurtovic
|
147b46f572
|
A couple more fixes and cleanup.
|
9 years ago |
|
Ben Kurtovic
|
03910b6cb5
|
Add mirror detection logic to parsers; fixes.
|
9 years ago |
|
Ben Kurtovic
|
81a090c923
|
Allow content parsers to signal that a source should be excluded.
|
9 years ago |
|
Ben Kurtovic
|
bb819c9306
|
Explicitly include excluded URLs in the result set; mark as excluded.
|
9 years ago |
|
Ben Kurtovic
|
e99e1c1ef1
|
Typo fix.
|
9 years ago |
|
Ben Kurtovic
|
509598d7fc
|
Try merging in templates with parameter values of a certain size (fixes #42)
|
9 years ago |
|
Ben Kurtovic
|
d741667c4c
|
Try using pentagrams rather than trigrams for copyvio Markov chains.
|
9 years ago |
|
Ben Kurtovic
|
b315e9bdc5
|
Default to useHTTPS=True for new Sites.
|
9 years ago |
|
Ben Kurtovic
|
68e3ec2a73
|
Don't pass starttimestamp when None.
This sometimes happens when pages are edited without loading them beforehand.
|
9 years ago |
|
Ben Kurtovic
|
4e8be871b7
|
Update copyright year for 2015.
|
10 years ago |
|
Ben Kurtovic
|
09319b1675
|
Don't die on broken regexes.
|
10 years ago |
|
Ben Kurtovic
|
4cdfafd487
|
Skip site check.
|
10 years ago |
|
Ben Kurtovic
|
4075d887e9
|
Fix return.
|
10 years ago |
|
Ben Kurtovic
|
a2c10650a8
|
Add support for User:EranBot/Copyright/Blacklist (closes #52)
|
10 years ago |
|
Ben Kurtovic
|
9ffc3f1bf5
|
Raise file crawl size limit for PDFs.
|
10 years ago |
|
Ben Kurtovic
|
33107c6a01
|
Add a lastrevid property to pages.
|
10 years ago |
|
Ben Kurtovic
|
b87d5ac673
|
Pass parameter to recursive call.
|
10 years ago |
|
Ben Kurtovic
|
170f810735
|
Allow ExclusionDB to force a sync.
|
10 years ago |
|
Ben Kurtovic
|
6ae3cd6d08
|
Handle interwiki page titles correctly.
|
10 years ago |
|
Ben Kurtovic
|
901192ec18
|
Handle errors from UnicodeDamnit.
|
10 years ago |
|
Ben Kurtovic
|
3f2dd1094f
|
Catch HTTPException in opener.open.
|
10 years ago |
|
Ben Kurtovic
|
9eaad11efb
|
Fix unicode bug in exception.
|
10 years ago |
|
Ben Kurtovic
|
699f6e3b17
|
Seems it will sometimes raise AssertionError.
|
10 years ago |
|
Ben Kurtovic
|
12c5170815
|
Catch another exception thrown by pdfminer.
|
10 years ago |
|
Ben Kurtovic
|
08d02917f2
|
Strange typo.
|
10 years ago |
|
Ben Kurtovic
|
c2a5946874
|
Fix generating -0.0 as a confidence value.
|
10 years ago |
|
Ben Kurtovic
|
106e58b164
|
Update confidence function comments.
|
10 years ago |
|
Ben Kurtovic
|
b8d55973c9
|
Tell Yahoo! it's okay to return PDFs.
|
10 years ago |
|
Ben Kurtovic
|
5194525a32
|
Note when sources might have been missed.
|
10 years ago |
|
Ben Kurtovic
|
065d9ea498
|
Fix; should always return a float.
|
10 years ago |
|
Ben Kurtovic
|
290f81abed
|
Prevent -0.0 from being a confidence value.
|
10 years ago |
|
Ben Kurtovic
|
932b93572a
|
Simplify function.
|
10 years ago |
|
Ben Kurtovic
|
07bbf240f6
|
Support new method for query continuation.
|
10 years ago |
|
Ben Kurtovic
|
459c252fc7
|
Support new CSRF token API.
|
10 years ago |
|
Ben Kurtovic
|
77514ee925
|
Add another PDF string substitution.
|
10 years ago |
|
Ben Kurtovic
|
0bdcbca8b0
|
Rudimentary solution for PDF parsing (closes earwig/copyvios#18)
|
10 years ago |
|
Ben Kurtovic
|
30f72df470
|
Refactor parsers; fix empty document behavior.
|
10 years ago |
|
Ben Kurtovic
|
5349179088
|
Fix parsing of plain text documents (earwig/copyvios#3)
|
10 years ago |
|
Ben Kurtovic
|
f10908e34e
|
Handle struct.error from GzipFile.read() (Python bug?)
|
10 years ago |
|
Ben Kurtovic
|
693cdc302f
|
Catch errors while searching.
|
10 years ago |
|
Ben Kurtovic
|
303c39c8c7
|
Add an option to disable short-circuiting.
|
10 years ago |
|
Ben Kurtovic
|
f8f4669460
|
Remove unnecessary key attribute of sources.
|
10 years ago |
