Grants:IdeaLab/Copyvio Detection

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Copyvio Detection
This is a proposal for a simple system to detect copyright infringement, and make it simple to figure out whats wrong and how to fix it.
idea creator
Jeblad
this project needs...
volunteer
join
endorse
created on17:18, 2 November 2014 (UTC)

Project idea[edit]

What is the problem you're trying to solve?[edit]

Editors will from time to time, especially the new one, copy content from other sites without rephrasing the content themselves, thereby infringing copyrights. The copyright is about the form of the prose, not about the facts. Usually the editors want to get the facts, and more or less unwittingly creates a copyright infringement. One thing is that the editors should not copy content content verbatim, but it still happen from time to time.

At Norwegian (bokmål) Wikipedia we had a copyright infringement detector for a time, but it has now been deprecated as the API it used is not available anymore. The old Norwegian help page is at w:no:Bruker:Jeblad/Påvisning av opphavsrettsbrudd, the code at w:no:Bruker:Jeblad/Gadget-copyvio-check.js, and sthe style at w:no:Bruker:Jeblad/Gadget-copyvio-check.css. One possibility could be to reimplement this as a node.js service, but it is probably equally easy to start from scratch and implement it as an extension.

Copyright violations must be interpreted in a legal context; many discussions at Wikipedia tend to center around the idea that some concept is copyrighted, but it is the form of the expression that is copyrightde. This means that there must be some distance between the old expression and the new one to dismiss any claims of copyright violation, and that this distance must be maintained over some larger piece of text.

What is your solution?[edit]

What makes the proposed system possible is that it checks the edits as they are made, before the edits can propagate to other sites.

  1. The user makes an edit
  2. The diff of the edit is used to build query sets
    • The query sets are split on sentence boundaries
    • Filter out sets with to few words
  3. The queries are sent to a search engine
    • Keep results with sufficient number of hits
  4. Accumulate the results for each sentence
    • Filter out pages with more than N number of hits
  5. Check the fragments from the search engines for maximum similarity (edit distance through local sensitive hashing)
    • Filter out pages with similarity above some level
  6. Tag and log the edits with a "high similarity"-tag

Such edits should then be inspected for possible copyright violations. This warning should also be available for the editor so the copyvio can be fixed swiftly. The tag would altough be visible to all, so anyone can fix it.

It should be possible to continue editing of a specific copyvio for the purpose of fixing it, and then the tag should be removed when it no longer apply. If the edit changes the diff, but without solving the issue it should be retained. A new edit that is not a continuation of a previous edit will not remove tags set on the previous edit. It will although be possible for a user with sufficient rights to remove such a tag. The log entry will although be retained.

Changes to external sites will not trigger a reevaluation of the tag, it will only change due to editing of the internal content and matches or non-matches against the original text snippets delivered by the search engines. This makes the system causal on the original time stamp.

Note that from point 3 the processing will be done in a separate worker thread so to not block the delivery of the page.

Goals[edit]

Primary goal will be to remove the easy claim that "Wikipedia is full of copyright violations". It should be non-trivial to add a copyright violation to an article, and it should be trivial to find it and remove it. How it is done, and why, should be clearly visible for everyone.

Get Involved[edit]

Participants[edit]

Endorsements[edit]

  • Fair use is an excuse for stealing. Wikipedia should not be stealing. There is no need for a picture in most cases. Good writing is better. Wowee Zowee public (talk) 17:40, 8 March 2015 (UTC)
https://meta.wikimedia.org/wiki/Grants:IdeaLab/Copyvio_Detection
meta.wikimedia.org
CC-MAIN-2023-40
40731
application/http; msgtype=response
sha1:AXUEVI3ECMJOEXCF3IW3KLLCZD2YF5AB
<urn:uuid:b3db6a4b-c5bc-4209-a34b-ae0aa80d7431>
2023-10-03T14:32:23Z
208.80.154.224
text/html
sha1:QRHSCNCLL4MOXC2SB6HS6XY7PH725GDO
<urn:uuid:38043812-85e5-4c01-a4c5-e2618cc54e1c>
https://meta.wikimedia.org/wiki/Grants:IdeaLab/Copyvio_Detection
response
<urn:uuid:50cc441e-9cfa-4ca9-8144-58c12eaad14c>
isPartOf: CC-MAIN-2023-40
publisher: Common Crawl
description: Wide crawl of the web for September/October 2023
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-82
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/