The brWaC Corpus: A New Open Resource for Brazilian Portuguese
In this work, we present the construction process of a large Web corpus for Brazilian Portuguese, aiming to achieve a size comparable
to the state of the art in other languages. We also discuss our updated sentence-level approach for the strict removal of duplicated
content. Following the pipeline methodology, more than 60 million pages were crawled and filtered, with 3.5 million being selected.
The obtained multi-domain corpus, named brWaC, is composed by 2.7 billion tokens, and has been annotated with tagging and parsing
information. The incidence of non-unique long sentences, an indication of replicated content, which reaches 9% in other Web corpora,
was reduced to only 0.5%. Domain diversity was also maximized, with 120,000 different websites contributing content. We are making
our new resource freely available for the research community, both for querying and downloading, in the expectation of aiding in new
advances for the processing of Brazilian Portuguese.
https://www.inf.ufrgs.br/pln/wiki/index.php?title=BrWaC
Federal University of Rio Grande do Sul
A university or research institution
Brazil
Jorge A. Wagner Filho
jawfilho@inf.ufrgs.br
https://www.inf.ufrgs.br/pln/wiki/index.php?title=Presentation
Yes - after signing a user agreement
https://goo.gl/forms/jrAe1HqcLJ39Zrou2
Yes
open license research use
other: Other license