A copyright violation detector running on Wikimedia Cloud Services https://tools.wmflabs.org/copyvios/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 

306 lines
18 KiB

  1. <%!
  2. from flask import g, request
  3. from copyvios.attribution import get_attribution_info
  4. from copyvios.checker import T_POSSIBLE, T_SUSPECT
  5. from copyvios.misc import cache
  6. %>\
  7. <%include file="/support/header.mako" args="title='Earwig\'s Copyvio Detector'"/>
  8. <%namespace module="copyvios.highlighter" import="highlight_delta"/>\
  9. <%namespace module="copyvios.misc" import="httpsfix, urlstrip"/>\
  10. % if notice:
  11. <div id="notice-box" class="gray-box">
  12. ${notice}
  13. </div>
  14. % endif
  15. % if query.submitted:
  16. % if query.error:
  17. <div id="info-box" class="red-box"><p>
  18. % if query.error == "bad action":
  19. Unknown action: <b><span class="mono">${query.action | h}</span></b>.
  20. % elif query.error == "no search method":
  21. No copyvio search methods were selected. A check can only be made using the search engine, links present in the page, Turnitin, or some combination of these.
  22. % elif query.error == "bad oldid":
  23. The revision ID <code>${query.oldid | h}</code> is invalid. It should be an integer.
  24. % elif query.error == "no URL":
  25. URL comparison mode requires a URL to be entered. Enter one in the text box below, or choose copyvio search mode to look for content similar to the article elsewhere on the web.
  26. % elif query.error == "bad URI":
  27. Unsupported URI scheme: <a href="${query.url | h}">${query.url | h}</a>.
  28. % elif query.error == "no data":
  29. Couldn't find any text in <a href="${query.url | h}">${query.url | h}</a>. <i>Note:</i> only HTML documents, plain text pages, and PDFs are supported, and content generated by JavaScript or found inside iframes is ignored.
  30. % elif query.error == "timeout":
  31. The URL <a href="${query.url | h}">${query.url | h}</a> timed out before any data could be retrieved.
  32. % elif query.error == "search error":
  33. An error occurred while using the search engine (${query.exception}). <i>Note:</i> there is a daily limit on the number of search queries the tool is allowed to make. You may <a href="${request.url | httpsfix, h}&amp;use_engine=0">repeat the check without using the search engine</a>.
  34. % else:
  35. An unknown error occurred.
  36. % endif
  37. </p></div>
  38. % elif not query.site:
  39. <div id="info-box" class="red-box">
  40. <p>The given site (project=<b><span class="mono">${query.project | h}</span></b>, language=<b><span class="mono">${query.lang | h}</span></b>) doesn't seem to exist. It may also be closed or private. <a href="https://${query.lang | h}.${query.project | h}.org/">Confirm its URL.</a></p>
  41. </div>
  42. % elif query.oldid and not result:
  43. <div id="info-box" class="red-box">
  44. <p>The revision ID couldn't be found: <a href="https://${query.site.domain | h}/w/index.php?oldid=${query.oldid | h}">${query.oldid | h}</a>.</p>
  45. </div>
  46. % elif query.title and not result:
  47. <div id="info-box" class="red-box">
  48. <p>The page couldn't be found: <a href="${query.page.url}">${query.page.title | h}</a>.</p>
  49. </div>
  50. % endif
  51. %endif
  52. <p>This tool attempts to detect <a href="https://en.wikipedia.org/wiki/WP:COPYVIO">copyright violations</a> in articles. In <i>search mode</i>, it will check for similar content elsewhere on the web using <a href="https://developers.google.com/custom-search/">Google</a>, external links present in the text of the page, or <a href="https://en.wikipedia.org/wiki/Wikipedia:Turnitin">Turnitin</a> (provided by <a href="https://en.wikipedia.org/wiki/User:EranBot">EranBot</a>), depending on which options are selected. In <i>comparison mode</i>, the tool will compare the article to a specific webpage without making additional searches, like the <a href="https://dupdet.toolforge.org/">Duplication Detector</a>.</p>
  53. <p>Running a full check can take up to a minute if other websites are slow or if the tool is under heavy use. Please be patient. If you get a timeout, wait a moment and refresh the page.</p>
  54. <p>Be aware that other websites can copy from Wikipedia, so check the results carefully, especially for older or well-developed articles. Specific websites can be skipped by being added to the <a href="https://en.wikipedia.org/wiki/User:EarwigBot/Copyvios/Exclusions">excluded URL list</a>.</p>
  55. <form id="cv-form" action="${request.script_root}/" method="get">
  56. <table id="cv-form-outer">
  57. <tr>
  58. <td>Site:</td>
  59. <td colspan="3">
  60. <span class="mono">https://</span>
  61. <select name="lang">
  62. <% selected_lang = query.orig_lang if query.orig_lang else g.cookies["CopyviosDefaultLang"].value if "CopyviosDefaultLang" in g.cookies else cache.bot.wiki.get_site().lang %>\
  63. % for code, name in cache.langs:
  64. % if code == selected_lang:
  65. <option value="${code | h}" selected="selected">${name}</option>
  66. % else:
  67. <option value="${code | h}">${name}</option>
  68. % endif
  69. % endfor
  70. </select>
  71. <span class="mono">.</span>
  72. <select name="project">
  73. <% selected_project = query.project if query.project else g.cookies["CopyviosDefaultProject"].value if "CopyviosDefaultProject" in g.cookies else cache.bot.wiki.get_site().project %>\
  74. % for code, name in cache.projects:
  75. % if code == selected_project:
  76. <option value="${code | h}" selected="selected">${name}</option>
  77. % else:
  78. <option value="${code | h}">${name}</option>
  79. % endif
  80. % endfor
  81. </select>
  82. <span class="mono">.org</span>
  83. </td>
  84. </tr>
  85. <tr>
  86. <td id="cv-col1">Page&nbsp;title:</td>
  87. <td id="cv-col2">
  88. % if query.title:
  89. <input class="cv-text" type="text" name="title" value="${query.page.title if query.page else query.title | h}" />
  90. % else:
  91. <input class="cv-text" type="text" name="title" />
  92. % endif
  93. </td>
  94. <td id="cv-col3">or&nbsp;revision&nbsp;ID:</td>
  95. <td id="cv-col4">
  96. % if query.oldid:
  97. <input class="cv-text" type="text" name="oldid" value="${query.oldid | h}" />
  98. % else:
  99. <input class="cv-text" type="text" name="oldid" />
  100. % endif
  101. </td>
  102. </tr>
  103. <tr>
  104. <td>Action:</td>
  105. <td colspan="3">
  106. <table id="cv-form-inner">
  107. <tr>
  108. <td id="cv-inner-col1">
  109. <input id="action-search" type="radio" name="action" value="search" ${'checked="checked"' if (query.action == "search" or not query.action) else ""} />
  110. </td>
  111. <td id="cv-inner-col2"><label for="action-search">Copyvio&nbsp;search:</label></td>
  112. <td id="cv-inner-col3">
  113. <input type="hidden" name="use_engine" value="0" />
  114. <input id="cv-cb-engine" class="cv-search" type="checkbox" name="use_engine" value="1" ${'checked="checked"' if query.use_engine not in ("0", "false") else ""} />
  115. <label for="cv-cb-engine">Use&nbsp;search&nbsp;engine</label>
  116. <input type="hidden" name="use_links" value="0" />
  117. <input id="cv-cb-links" class="cv-search" type="checkbox" name="use_links" value="1" ${'checked="checked"' if query.use_links not in ("0", "false") else ""} />
  118. <label for="cv-cb-links">Use&nbsp;links&nbsp;in&nbsp;page</label>
  119. <input type="hidden" name="turnitin" value="0" />
  120. <input id="cv-cb-turnitin" class="cv-search" type="checkbox" name="turnitin" value="1" ${'checked="checked"' if query.turnitin in ("1", "true") else ""}/>
  121. <label for="cv-cb-turnitin">Use&nbsp;Turnitin</label>
  122. </td>
  123. </tr>
  124. <tr>
  125. <td>
  126. <input id="action-compare" type="radio" name="action" value="compare" ${'checked="checked"' if query.action == "compare" else ""} />
  127. </td>
  128. <td><label for="action-compare">URL&nbsp;comparison:</label></td>
  129. <td>
  130. <input class="cv-compare cv-text" type="text" name="url"
  131. % if query.url:
  132. value="${query.url | h}"
  133. % endif
  134. />
  135. </td>
  136. </tr>
  137. </table>
  138. </td>
  139. </tr>
  140. % if query.nocache or (result and result.cached):
  141. <tr>
  142. <td><label for="cb-nocache">Bypass&nbsp;cache:</label></td>
  143. <td colspan="3">
  144. <input id="cb-nocache" type="checkbox" name="nocache" value="1" ${'checked="checked"' if query.nocache else ""} />
  145. </td>
  146. </tr>
  147. % endif
  148. <tr>
  149. <td colspan="4">
  150. <button type="submit">Submit</button>
  151. </td>
  152. </tr>
  153. </table>
  154. </form>
  155. % if result:
  156. <div id="generation-time">
  157. Results
  158. % if result.cached:
  159. <a id="cv-cached" href="#">cached<span>To save time (and money), this tool will retain the results of checks for up to 72 hours. This includes the URLs of the checked sources, but neither their content nor the content of the article. Future checks on the same page (assuming it remains unchanged) will not involve additional search queries, but a fresh comparison against the source URL will be made. If the page is modified, a new check will be run.</span></a> from <abbr title="${result.cache_time}">${result.cache_age} ago</abbr>. Originally
  160. % endif
  161. generated in <span class="mono">${round(result.time, 3)}</span>
  162. % if query.action == "search":
  163. seconds using <span class="mono">${result.queries}</span> quer${"y" if result.queries == 1 else "ies"}.
  164. % else:
  165. seconds.
  166. % endif
  167. <a href="${request.script_root | h}/?lang=${query.lang | h}&amp;project=${query.project | h}&amp;oldid=${query.oldid or query.page.lastrevid | h}&amp;action=${query.action | h}&amp;${"use_engine={0}&use_links={1}".format(int(query.use_engine not in ("0", "false")), int(query.use_links not in ("0", "false"))) if query.action == "search" else "" | h}${"url=" if query.action == "compare" else ""}${query.url if query.action == "compare" else "" | u}">Permalink.</a>
  168. </div>
  169. <div id="cv-result" class="${'red' if result.confidence >= T_SUSPECT else 'yellow' if result.confidence >= T_POSSIBLE else 'green'}-box">
  170. <table id="cv-result-head-table">
  171. <colgroup>
  172. <col>
  173. <col>
  174. <col>
  175. </colgroup>
  176. <tr>
  177. <td>
  178. <a href="${query.page.url}">${query.page.title | h}</a>
  179. % if query.oldid:
  180. @<a href="https://${query.site.domain | h}/w/index.php?oldid=${query.oldid | h}">${query.oldid | h}</a>
  181. % endif
  182. % if query.redirected_from:
  183. <br />
  184. <span id="redirected-from">Redirected from <a href="https://${query.site.domain | h}/w/index.php?title=${query.redirected_from.title | u}&amp;redirect=no">${query.redirected_from.title | h}</a>. <a href="${request.url | httpsfix, h}&amp;noredirect=1">Check original.</a></span>
  185. % endif
  186. </td>
  187. <td>
  188. <div>
  189. % if result.confidence >= T_SUSPECT:
  190. Violation&nbsp;Suspected
  191. % elif result.confidence >= T_POSSIBLE:
  192. Violation&nbsp;Possible
  193. % elif result.sources:
  194. Violation&nbsp;Unlikely
  195. % else:
  196. No&nbsp;Violation
  197. % endif
  198. </div>
  199. <div>${round(result.confidence * 100, 1)}%</div>
  200. <div>similarity</div>
  201. </td>
  202. <td>
  203. % if result.url:
  204. <a href="${result.url | h}">${result.url | urlstrip, h}</a>
  205. % else:
  206. <span id="result-head-no-sources">No matches found.</span>
  207. % endif
  208. </td>
  209. </tr>
  210. </table>
  211. </div>
  212. <% attrib = get_attribution_info(query.site, query.page) %>
  213. % if attrib:
  214. <div id="attribution-warning" class="yellow-box">
  215. This article contains an attribution template: <code>{{<a href="${attrib[1]}">${attrib[0] | h}</a>}}</code>. Please verify that any potential copyvios are not from properly attributed sources.
  216. </div>
  217. % endif
  218. % if query.turnitin_result:
  219. <div id="turnitin-container" class="${'red' if query.turnitin_result.reports else 'green'}-box">
  220. <div id="turnitin-title">Turnitin Results</div>
  221. % if query.turnitin_result.reports:
  222. <table id="turnitin-table"><tbody>
  223. % for report in turnitin_result.reports:
  224. <tr><td class="turnitin-table-cell"><a href="https://eranbot.toolforge.org/ithenticate.py?rid=${report.reportid}">Report ${report.reportid}</a> for text added at <a href="https://${query.lang}.wikipedia.org/w/index.php?title=${query.title}&amp;diff=${report.diffid}"> ${report.time_posted.strftime("%H:%M, %d %B %Y (UTC)")}</a>:
  225. <ul>
  226. % for source in report.sources:
  227. <li>${source['percent']}% of revision text (${source['words']} words) found at <a href="${source['url'] | h}">${source['url'] | h}</a></li>
  228. % endfor
  229. </ul></td></tr>
  230. % endfor
  231. </tbody></table>
  232. % else:
  233. <div id="turnitin-summary">No matching sources found.</div>
  234. % endif
  235. </div>
  236. % endif
  237. % if query.action == "search":
  238. <% skips = False %>
  239. <div id="sources-container">
  240. <div id="sources-title">Checked Sources</div>
  241. % if result.sources:
  242. <table id="cv-result-sources">
  243. <colgroup>
  244. <col>
  245. <col>
  246. <col>
  247. </colgroup>
  248. <tr>
  249. <th>URL</th>
  250. <th>Similarity</th>
  251. <th>Compare</th>
  252. </tr>
  253. % for i, source in enumerate(result.sources):
  254. <tr ${'class="source-default-hidden"' if i >= 10 else 'id="source-row-selected"' if i == 0 else ""}>
  255. <td><a ${'id="source-selected"' if i == 0 else ""} class="source-url" href="${source.url | h}">${source.url | h}</a></td>
  256. <td>
  257. % if source.excluded:
  258. <span class="source-excluded">Excluded</span>
  259. % elif source.skipped:
  260. <% skips = True %>
  261. <span class="source-skipped">Skipped</span>
  262. % else:
  263. <span class="source-similarity ${"source-suspect" if source.confidence >= T_SUSPECT else "source-possible" if source.confidence >= T_POSSIBLE else "source-novio"}">${round(source.confidence * 100, 1)}%</span>
  264. % endif
  265. </td>
  266. <td>
  267. <a href="${request.script_root | h}/?lang=${query.lang | h}&amp;project=${query.project | h}&amp;oldid=${query.oldid or query.page.lastrevid | h}&amp;action=compare&amp;url=${source.url | u}">Compare</a>
  268. </td>
  269. </tr>
  270. % endfor
  271. </table>
  272. % else:
  273. <div class="cv-source-footer">
  274. No sources checked.
  275. </div>
  276. % endif
  277. % if len(result.sources) > 10:
  278. <div id="cv-additional" class="cv-source-footer">
  279. ${len(result.sources) - 10} URL${"s" if len(result.sources) > 11 else ""} with lower similarity hidden. <a id="show-additional-sources" href="#">Show them.</a>
  280. </div>
  281. % endif
  282. % if skips or result.possible_miss:
  283. <div class="cv-source-footer">
  284. The search ended early because a match was found with high similarity. <a href="${request.url | httpsfix, h}&amp;noskip=1">Do a complete check.</a>
  285. </div>
  286. % endif
  287. </div>
  288. % endif
  289. <div id="cv-chain-container">
  290. <table id="cv-chain-table">
  291. <tr>
  292. <td class="cv-chain-cell">Article: <div class="cv-chain-detail"><p>${highlight_delta(result.article_chain, result.best.chains[1] if result.best else None)}</p></div></td>
  293. <td class="cv-chain-cell">Source: <div class="cv-chain-detail"><p>${highlight_delta(result.best.chains[0], result.best.chains[1]) if result.best else ""}</p></div></td>
  294. </tr>
  295. </table>
  296. </div>
  297. % endif
  298. <%include file="/support/footer.mako"/>