Dataset:

Task Categories: sequence-modeling
Languages: he
Multilinguality: monolingual
Size Categories: 10K<n<100K
Licenses: mit
Language Creators: found
Annotations Creators: expert-generated
Source Datasets: original

Dataset Card for Hebrew Projectbenyehuda

Dataset Summary

This repository contains a dump of thousands of public domain works in Hebrew, from Project Ben-Yehuda, in plaintext UTF-8 files, with and without diacritics (nikkud), and in HTML files. The pseudocatalogue.csv file is a list of titles, authors, genres, and file paths, to help you process the dump.

The Releases tab contains a downloadable ZIP archive of the full release. The git repo can be used to track individual file changes, or for incremenetal updates. In the ZIPs, each format (plaintext, plaintext stripped of diacritics, and HTML) has a ZIP file containing one directory per author, with all the author's works under that directory.

To request changes or improvements to this dump, file an issue against this repository.

All these works are in the public domain, so you are free to make any use of them, and do not need to ask for permission.

If you would like to give credit, please credit "Project Ben-Yehuda volunteers", and include a link to the site. We'd also love to hear about the uses you've made of this dump, as it encourages us to keep producing the dump. E-mail us with a brief description (and links, if/as appropriate) of your re-use, at editor@benyehuda.org.

There are 10078 files, 3181136 lines

Data Annotation:

Supported Tasks and Leaderboards

[More Information Needed]

Languages

Hebrew

Dataset Structure

Sample:

'authors' = {str} 'אחד העם'
'genre' = {str} 'מאמרים ומסות'
'id' = {int} 10
'original_language' = {str} ''
'source_edition' = {str} ''
'text' = {str} '\n\n\n\t\n\tחצי-נחמה\n\t\n\n\n\n1\n\nבין כל הצרות שנתחדשו עלינו בעת האחרונה תעשׂה ביחוד רושם מעציב בלב כל איש ישׂראל התחדשות ‘עלילת־הדם’. העלילה הנתעבה הזאת, בכל יָשנה, היתה ותהיה תמיד בעינינו כחדשה, ומימי הבינים ועד עתה תצטין בפעולתה החזקה על רוח עמנו, לא רק במקום המע
'title' = {str} 'חצי-נחמה'
'translators' = {str} ''
'url' = {str} 'https://raw.githubusercontent.com/projectbenyehuda/public_domain_dump/master/txt/p23/m10.txt'

Data Instances

[More Information Needed]

Data Fields

  • authors
  • genre
  • id
  • original_language
  • source_edition
  • text
  • title
  • translators
  • url

Data Splits

train
corpus 10078

Dataset Creation

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

Who are the source language producers?

[More Information Needed]

Researchers

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Citation Information

@article{, author = {}, title = {Public domain texts from Project Ben-Yehuda}, journal = {}, url = {https://github.com/projectbenyehuda/public_domain_dump}, year = {2020}, }

Contributions

Thanks to @imvladikon for adding this dataset.

Models trained or fine-tuned on hebrew_projectbenyehuda

None yet
https://huggingface.co/datasets/hebrew_projectbenyehuda
huggingface.co
crawl=CC-MAIN-2021-39
84502
application/http; msgtype=response
sha1:MVYYFXQUXOUOTD45LJERTZ5JF4XYU72R
<urn:uuid:38815e38-5a68-44b6-b6a4-3d4e9f9d11cb>
2021-09-25T19:29:16Z
34.200.164.230
text/html
sha1:M7GX7CQ3KXAB6RCGCCY2WAOM4VHOA6AF
<urn:uuid:d6c5b916-9160-49d5-afaa-cc2e7d952551>
https://huggingface.co/datasets/hebrew_projectbenyehuda
response
<urn:uuid:dda53144-a328-40e7-a250-c63c9a0657f8>
isPartOf: CC-MAIN-2021-39
publisher: Common Crawl
description: Wide crawl of the web for September 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-143
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/