Skip to Main Content

SoTE: Data Mining

Boston College Attribution

This page was copied and adapted from the Boston College Libraries Text & Data Mining Guide under a Creative Commons Attribution 4.0 License. Our thanks to Boston College for developing this excellent resource and sharing it under the license!

Best Practice Tips

We can help you contact database publishers. If you want access to information that you can’t easily access through a database, we can help you get in touch with the right people at the database publisher and identify what options you might have. Options may include choices of delivery methods and negotiations regarding authorized uses. We can also provide assistance throughout the process of working with the publisher.

  • This may take time. If you are considering a text or data mining project, you should contact us early in your process as publishers may be slow to respond to requests for data and negotiations may take time, particularly since this is not a topic that all publishers have considered.

  • Publishers may charge for access to their data. Some publishers offer access to text and data mining projects only if the user pays an additional fee. Alternatively, some may sell their data separately specifically for this purpose. We can help you to identify the most economical and efficient method of getting the access you need.

  • Open Access alternatives may exist. Depending on the nature of your research, there may be Open Access journals, databases or datasets that you can use. We can help you to identify whether such a source would work for your project.​

Coming Soon! Databases that allow text mining

Model LIcenses

Several libraries and associations have developed model language to be included in content licenses in order to assure the ability to text/data mine content.
https://resources.nu.edu/c.php?g=881992&p=6336863
resources.nu.edu
CC-MAIN-2024-26
37905
application/http; msgtype=response
sha1:YB4JG4V35V4CS7TBJRO46BQSU5TJRRSJ
<urn:uuid:8abc33db-9985-47d6-b4ea-1ea7c4956d13>
2024-06-13T03:18:55Z
54.88.35.99
text/html
sha1:CM5OM2VJU5APKVOMUIKZSDLM5OBKK2RK
<urn:uuid:2bd6229f-4aee-44ee-a94f-def23be8c843>
https://resources.nu.edu/c.php?g=881992&p=6336863
response
<urn:uuid:01a51816-9a7f-4f44-9854-471e657cbaa8>
isPartOf: CC-MAIN-2024-26
publisher: Common Crawl
description: Wide crawl of the web for June 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-105
software: Apache Nutch 1.20 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/