 Ben Kurtovic
|
a73f618e0a
|
Initial conversion to Python 3
|
3 jaren geleden |
|
Ben Kurtovic
|
92d43e566e
|
copyvios: Missed a file
|
3 jaren geleden |
|
Ben Kurtovic
|
a49a82e263
|
Fix a few bugs
|
3 jaren geleden |
|
Ben Kurtovic
|
2b5914b6ae
|
Support parser-directed URL redirecting (for Wayback Machine PDFs)
|
3 jaren geleden |
|
Ben Kurtovic
|
774628b34e
|
OAuth support; switch to requests; update login flow
|
5 jaren geleden |
|
Ben Kurtovic
|
7853bcc0f3
|
Fix dependency checking for search engines.
|
8 jaren geleden |
|
Ben Kurtovic
|
04ed5257c7
|
Refactor search engines.
|
8 jaren geleden |
|
Ben Kurtovic
|
977b587e5e
|
Add support for Bing Search
|
8 jaren geleden |
|
Ben Kurtovic
|
69cdb41d07
|
Adjust mirror hints to include direct links back to the article.
|
8 jaren geleden |
|
Ben Kurtovic
|
f92fb34d0e
|
Improve sentence splitting, again.
|
8 jaren geleden |
|
Ben Kurtovic
|
108eca13ac
|
Finish mirror hinting algorithm.
|
8 jaren geleden |
|
Ben Kurtovic
|
91846ce4fb
|
Refactor out mirror hinting logic in source parsers.
|
8 jaren geleden |
|
Ben Kurtovic
|
03910b6cb5
|
Add mirror detection logic to parsers; fixes.
|
8 jaren geleden |
|
Ben Kurtovic
|
4e8be871b7
|
Update copyright year for 2015.
|
9 jaren geleden |
|
Ben Kurtovic
|
5194525a32
|
Note when sources might have been missed.
|
9 jaren geleden |
|
Ben Kurtovic
|
303c39c8c7
|
Add an option to disable short-circuiting.
|
9 jaren geleden |
|
Ben Kurtovic
|
9fd145da5c
|
Add some docs; better sorting function.
|
9 jaren geleden |
|
Ben Kurtovic
|
7afb484cea
|
Refactor a bunch of copyvio internals. Store all sources with a result object.
|
9 jaren geleden |
|
Ben Kurtovic
|
f94a67e0e3
|
Define num_queries in the proper place.
|
9 jaren geleden |
|
Ben Kurtovic
|
12247dd756
|
Add no_links and no_searches to copyvio_check().
|
9 jaren geleden |
|
Ben Kurtovic
|
c56838e742
|
Only spawn one worker for comparisons in local mode.
|
9 jaren geleden |
|
Ben Kurtovic
|
7c0e98596c
|
Some bugfixes.
|
9 jaren geleden |
|
Ben Kurtovic
|
361f7709f8
|
Starting work on global workers.
|
9 jaren geleden |
|
Ben Kurtovic
|
bdcbfa5327
|
Catch errors around response.read().
|
9 jaren geleden |
|
Ben Kurtovic
|
24dd497fd9
|
Catch more general socket.error.
|
9 jaren geleden |
|
Ben Kurtovic
|
5e72e74759
|
Employ new piecewise article-delta confidence function.
|
9 jaren geleden |
|
Ben Kurtovic
|
203c65280c
|
Float delta.
|
9 jaren geleden |
|
Ben Kurtovic
|
6b0f8ad311
|
Fix reference.
|
9 jaren geleden |
|
Ben Kurtovic
|
e2d7c7aef6
|
Update with new confidence function; fix unicode.
|
9 jaren geleden |
|
Ben Kurtovic
|
05010933c7
|
Reorder some URL opening code; zip protection.
|
9 jaren geleden |
|
Ben Kurtovic
|
2bddf79a3d
|
Fix deadlock when calling queue.put() while holding the mutex.
|
9 jaren geleden |
|
Ben Kurtovic
|
7a4fcd7807
|
Fix queue clear call.
|
9 jaren geleden |
|
Ben Kurtovic
|
efae85a1fe
|
Move thread spawning code to worker class.
|
9 jaren geleden |
|
Ben Kurtovic
|
7137dda920
|
Update copyvio checker to not make concurrent requests to a single domain.
|
9 jaren geleden |
|
Ben Kurtovic
|
5874467ec3
|
Bugfix, cleanup.
|
9 jaren geleden |
|
Ben Kurtovic
|
cc7ac52a05
|
Fix query counting.
|
10 jaren geleden |
|
Ben Kurtovic
|
d672e670fa
|
Fix param name.
|
10 jaren geleden |
|
Ben Kurtovic
|
0e28f89466
|
Update logging.
|
10 jaren geleden |
|
Ben Kurtovic
|
ae0c390ceb
|
Redesign copyvio internals to parallelize URL loading/parsing.
|
10 jaren geleden |
|
Ben Kurtovic
|
1501341000
|
Allow even more time for a URL to time out.
|
10 jaren geleden |
|
Ben Kurtovic
|
ccb3c022ca
|
Some servers don't leave a space before the content type parameter list.
|
10 jaren geleden |
|
Ben Kurtovic
|
5e9d4cfa78
|
copyvios: use a different timeout for direct URL comparisons.
|
10 jaren geleden |
|
Ben Kurtovic
|
ea14f39e73
|
Split content type correctly.
|
10 jaren geleden |
|
Ben Kurtovic
|
e0cd174310
|
Refactor out empty chain definitions.
|
10 jaren geleden |
|
Ben Kurtovic
|
0eadf65a09
|
Only accept HTML and plain text for copyvio checks.
|
10 jaren geleden |
|
Ben Kurtovic
|
c3ddc3d35a
|
Return the correct empty chain.
|
10 jaren geleden |
|
Ben Kurtovic
|
39d5c7c149
|
Update copyright notices for 2014.
|
10 jaren geleden |
|
Ben Kurtovic
|
ed95c99f0e
|
Update email address.
|
10 jaren geleden |
 Ben Kurtovic
|
5931f375de
|
Put response.read() in the try:, since that's what throws the timeout.
|
11 jaren geleden |
|
Ben Kurtovic
|
0b7a13eca5
|
Update copyright notices for 2013.
|
11 jaren geleden |
