 Ben Kurtovic
|
01dcbd4394
|
copyvios: Support on-error condition for URL proxying
|
há 3 anos |
|
Ben Kurtovic
|
9d66ebc6b2
|
copyvios: Config-directed URL proxying
|
há 3 anos |
|
Ben Kurtovic
|
fe2e7879e4
|
Fix issues in previous commit
|
há 3 anos |
|
Ben Kurtovic
|
2324a73624
|
copyvios: Refactor some parsing logic and add dynamic Blogger support
|
há 3 anos |
|
Ben Kurtovic
|
abb9403e5d
|
More bug fixes
|
há 3 anos |
|
Ben Kurtovic
|
a49a82e263
|
Fix a few bugs
|
há 3 anos |
|
Ben Kurtovic
|
2b5914b6ae
|
Support parser-directed URL redirecting (for Wayback Machine PDFs)
|
há 3 anos |
|
Ben Kurtovic
|
466d3a42f1
|
copyvios: Minor refactor for cleaner stack frames.
|
há 5 anos |
|
Ben Kurtovic
|
b4b079ffd0
|
Update copyright year for 2016.
|
há 8 anos |
|
Ben Kurtovic
|
4828cbad69
|
Catch possible ValueError when doing opener.open().
|
há 8 anos |
|
Ben Kurtovic
|
f52fb06c19
|
Add a debug message when catching ParserExclusionError.
|
há 8 anos |
|
Ben Kurtovic
|
91846ce4fb
|
Refactor out mirror hinting logic in source parsers.
|
há 8 anos |
|
Ben Kurtovic
|
147b46f572
|
A couple more fixes and cleanup.
|
há 8 anos |
|
Ben Kurtovic
|
03910b6cb5
|
Add mirror detection logic to parsers; fixes.
|
há 8 anos |
|
Ben Kurtovic
|
81a090c923
|
Allow content parsers to signal that a source should be excluded.
|
há 8 anos |
|
Ben Kurtovic
|
bb819c9306
|
Explicitly include excluded URLs in the result set; mark as excluded.
|
há 8 anos |
|
Ben Kurtovic
|
4e8be871b7
|
Update copyright year for 2015.
|
há 9 anos |
|
Ben Kurtovic
|
9ffc3f1bf5
|
Raise file crawl size limit for PDFs.
|
há 9 anos |
|
Ben Kurtovic
|
3f2dd1094f
|
Catch HTTPException in opener.open.
|
há 9 anos |
|
Ben Kurtovic
|
08d02917f2
|
Strange typo.
|
há 9 anos |
|
Ben Kurtovic
|
c2a5946874
|
Fix generating -0.0 as a confidence value.
|
há 9 anos |
|
Ben Kurtovic
|
106e58b164
|
Update confidence function comments.
|
há 9 anos |
|
Ben Kurtovic
|
5194525a32
|
Note when sources might have been missed.
|
há 9 anos |
|
Ben Kurtovic
|
065d9ea498
|
Fix; should always return a float.
|
há 9 anos |
|
Ben Kurtovic
|
290f81abed
|
Prevent -0.0 from being a confidence value.
|
há 9 anos |
|
Ben Kurtovic
|
932b93572a
|
Simplify function.
|
há 9 anos |
|
Ben Kurtovic
|
30f72df470
|
Refactor parsers; fix empty document behavior.
|
há 9 anos |
|
Ben Kurtovic
|
5349179088
|
Fix parsing of plain text documents (earwig/copyvios#3)
|
há 9 anos |
|
Ben Kurtovic
|
f10908e34e
|
Handle struct.error from GzipFile.read() (Python bug?)
|
há 9 anos |
|
Ben Kurtovic
|
303c39c8c7
|
Add an option to disable short-circuiting.
|
há 9 anos |
|
Ben Kurtovic
|
f8f4669460
|
Remove unnecessary key attribute of sources.
|
há 9 anos |
|
Ben Kurtovic
|
9fd145da5c
|
Add some docs; better sorting function.
|
há 9 anos |
|
Ben Kurtovic
|
7afb484cea
|
Refactor a bunch of copyvio internals. Store all sources with a result object.
|
há 9 anos |
|
Ben Kurtovic
|
54ddff049f
|
Make CopyvioSource public; tweaks.
|
há 9 anos |
|
Ben Kurtovic
|
0438766ee4
|
Handle empty URLs better.
|
há 9 anos |
|
Ben Kurtovic
|
2147207388
|
Remove unnecessary variable assign.
|
há 9 anos |
|
Ben Kurtovic
|
f37621e5ec
|
Use a deque for a FIFO instead of the python list LIFO.
|
há 9 anos |
|
Ben Kurtovic
|
8e439e1eea
|
source.join() now blocks when in the middle of processing.
|
há 9 anos |
|
Ben Kurtovic
|
dbb1ae5483
|
Handle empty queues correctly. Remove some log messages.
|
há 9 anos |
|
Ben Kurtovic
|
2fa8aeba5b
|
Fix a blocking issue.
|
há 9 anos |
|
Ben Kurtovic
|
939d8be08f
|
Fix variable.
|
há 9 anos |
|
Ben Kurtovic
|
3ed8837a3e
|
Fix stopping queues in local mode.
|
há 9 anos |
|
Ben Kurtovic
|
de7576728f
|
Fix dequeueing logic a bit.
|
há 9 anos |
|
Ben Kurtovic
|
b939262b11
|
Bugfix.
|
há 9 anos |
|
Ben Kurtovic
|
32ef0fbf1f
|
Add a bunch of temporary debugging code.
|
há 9 anos |
|
Ben Kurtovic
|
c7b3b7bc7f
|
CopyvioSource.workspace should be public.
|
há 9 anos |
|
Ben Kurtovic
|
e73e626994
|
Some locks needed to be tightened.
|
há 9 anos |
|
Ben Kurtovic
|
486c4692ed
|
Remove _workers attr of workspaces.
|
há 9 anos |
|
Ben Kurtovic
|
7c0e98596c
|
Some bugfixes.
|
há 9 anos |
|
Ben Kurtovic
|
361f7709f8
|
Starting work on global workers.
|
há 9 anos |
