Ver código fonte

Collapse extra newlines to avoid distorting trigrams.

tags/v0.1^2
Ben Kurtovic 12 anos atrás
pai
commit
33aa1d6744
1 arquivos alterados com 2 adições e 1 exclusões
  1. +2
    -1
      earwigbot/wiki/copyvios/parsers.py

+ 2
- 1
earwigbot/wiki/copyvios/parsers.py Ver arquivo

@@ -70,7 +70,8 @@ class ArticleTextParser(BaseTextParser):
The actual stripping is handled by :py:mod:`mwparserfromhell`.
"""
wikicode = mwparserfromhell.parse(self.text)
self.clean = wikicode.strip_code(normalize=True)
clean = wikicode.strip_code(normalize=True, collapse=True)
self.clean = clean.replace("\n\n", "\n") # Collapse extra newlines.
return self.clean

def chunk(self, nltk_dir, max_chunks, max_query=256):


Carregando…
Cancelar
Salvar