Ben Kurtovic
4e8be871b7
Update copyright year for 2015.
pirms 9 gadiem
Ben Kurtovic
5194525a32
Note when sources might have been missed.
pirms 9 gadiem
Ben Kurtovic
303c39c8c7
Add an option to disable short-circuiting.
pirms 9 gadiem
Ben Kurtovic
9fd145da5c
Add some docs; better sorting function.
pirms 9 gadiem
Ben Kurtovic
7afb484cea
Refactor a bunch of copyvio internals. Store all sources with a result object.
pirms 9 gadiem
Ben Kurtovic
f94a67e0e3
Define num_queries in the proper place.
pirms 9 gadiem
Ben Kurtovic
12247dd756
Add no_links and no_searches to copyvio_check().
pirms 9 gadiem
Ben Kurtovic
c56838e742
Only spawn one worker for comparisons in local mode.
pirms 9 gadiem
Ben Kurtovic
7c0e98596c
Some bugfixes.
pirms 9 gadiem
Ben Kurtovic
361f7709f8
Starting work on global workers.
pirms 9 gadiem
Ben Kurtovic
bdcbfa5327
Catch errors around response.read().
pirms 9 gadiem
Ben Kurtovic
24dd497fd9
Catch more general socket.error.
pirms 9 gadiem
Ben Kurtovic
5e72e74759
Employ new piecewise article-delta confidence function.
pirms 9 gadiem
Ben Kurtovic
203c65280c
Float delta.
pirms 9 gadiem
Ben Kurtovic
6b0f8ad311
Fix reference.
pirms 9 gadiem
Ben Kurtovic
e2d7c7aef6
Update with new confidence function; fix unicode.
pirms 9 gadiem
Ben Kurtovic
05010933c7
Reorder some URL opening code; zip protection.
pirms 9 gadiem
Ben Kurtovic
2bddf79a3d
Fix deadlock when calling queue.put() while holding the mutex.
pirms 9 gadiem
Ben Kurtovic
7a4fcd7807
Fix queue clear call.
pirms 9 gadiem
Ben Kurtovic
efae85a1fe
Move thread spawning code to worker class.
pirms 9 gadiem
Ben Kurtovic
7137dda920
Update copyvio checker to not make concurrent requests to a single domain.
pirms 9 gadiem
Ben Kurtovic
5874467ec3
Bugfix, cleanup.
pirms 9 gadiem
Ben Kurtovic
cc7ac52a05
Fix query counting.
pirms 10 gadiem
Ben Kurtovic
d672e670fa
Fix param name.
pirms 10 gadiem
Ben Kurtovic
0e28f89466
Update logging.
pirms 10 gadiem
Ben Kurtovic
ae0c390ceb
Redesign copyvio internals to parallelize URL loading/parsing.
pirms 10 gadiem
Ben Kurtovic
1501341000
Allow even more time for a URL to time out.
pirms 10 gadiem
Ben Kurtovic
ccb3c022ca
Some servers don't leave a space before the content type parameter list.
pirms 10 gadiem
Ben Kurtovic
5e9d4cfa78
copyvios: use a different timeout for direct URL comparisons.
pirms 10 gadiem
Ben Kurtovic
ea14f39e73
Split content type correctly.
pirms 10 gadiem
Ben Kurtovic
e0cd174310
Refactor out empty chain definitions.
pirms 10 gadiem
Ben Kurtovic
0eadf65a09
Only accept HTML and plain text for copyvio checks.
pirms 10 gadiem
Ben Kurtovic
c3ddc3d35a
Return the correct empty chain.
pirms 10 gadiem
Ben Kurtovic
39d5c7c149
Update copyright notices for 2014.
pirms 10 gadiem
Ben Kurtovic
ed95c99f0e
Update email address.
pirms 10 gadiem
Ben Kurtovic
5931f375de
Put response.read() in the try:, since that's what throws the timeout.
pirms 11 gadiem
Ben Kurtovic
0b7a13eca5
Update copyright notices for 2013.
pirms 11 gadiem
Ben Kurtovic
4ff7612a27
Fix when we've cached None as a url.
pirms 11 gadiem
Ben Kurtovic
25d75c5d11
Forgot to give process_time param to CopyvioCheckResult.
pirms 11 gadiem
Ben Kurtovic
bcf9b70107
Keep track of how long generating results takes; support 'max_time'.
pirms 11 gadiem
Ben Kurtovic
de23112f43
Bugfix when we can't read HTML.
pirms 11 gadiem
Ben Kurtovic
a4dda89a61
Various fixes for copyvios.
- Fix a bug in ExclusionsDB; improve URL regexes.
- NLTK's LookupError is actually an IOError.
- Fix bug in __repr__ for CopyvioCheckResult.
- Rewrite YahooBOSSSearchEngine to actually work with oauth2.
- Search engines now take a URL opener in addition to credentials.
pirms 11 gadiem
Ben Kurtovic
655624c2cf
Lazy-importing of py-bcrypt and pycrypto; restructured deps in setup.
pirms 11 gadiem
Ben Kurtovic
0ca84ab9bc
Implement lazy-importing of oauth2, nltk, and bs4.
pirms 11 gadiem
Ben Kurtovic
4baab6f57c
Implement lazy importing of root-level modules and packages.
- Simplify all imports
- Update dependency version in setup.py
- Change waitTime default from three seconds to two
pirms 12 gadiem
Ben Kurtovic
8d8703358c
More fixes and tweaks; cleanup; etc.
pirms 12 gadiem
Ben Kurtovic
f993b847ab
Encode URLs as UTF-8 before opening them.
pirms 12 gadiem
Ben Kurtovic
570168ed0e
Institute a timeout so we don't try to open these suspicious URLs forever.
pirms 12 gadiem
Ben Kurtovic
439b855254
Fully implement logging; fix non-unicode log messages.
pirms 12 gadiem
Ben Kurtovic
a074da853b
More work on copyvios, including an exclusions database ( #5 )
* Added exclusions module with a fully implemented ExclusionsDB that can pull
from multiple sources for different sites.
* Moved CopyvioCheckResult to its own module, to be imported by __init__.
* Some other related changes.
pirms 12 gadiem