6

In what formats can I retrieve texts from wikisource?

The obvious format is the HTML they deliver for reading, but the HTML source is very layout oriented and messy.

Is there a downloadable source format available that is more logical (XML or wiki format)? How can I download it?

3

For automated processing, by far the best alternative is a database dump. They might be a little bit old, but hopefully that won't be a problem.

You can select database dumps by language and wiki, with or without edit history.

At the time of writing, the most recent enwikisource dump is from 26 December 2015. You can also find dumps for dewikisource, frwikisource, etc.

3
  • Age is not a problem, but size matters: I am basically interested in a selection of about 2 dozen texts. – user6083 Feb 2 '16 at 8:15
  • @jknappen That's more of a (mediawiki) technical question. There's probably a forum somewhere on wikimedia to ask it. – gerrit Feb 2 '16 at 10:41
  • I found out that I can export all pages from one category using Special:Export (available in the "special pages" menue). To get at the textual content, I had to check the box "include templates". The format is still crowded, but ways better than the HTML. – user6083 Feb 3 '16 at 16:36
2

If you are just interested in two dozen text I think it is easiest just to request for the raw wikitext, for instance for the 'The Metal Pig' page:

https://en.wikisource.org/wiki/The_Metal_Pig?action=raw

This query returns the raw wikitext, i.e., with unexpanded templates and category links.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy
https://opendata.stackexchange.com/questions/6974/formats-of-texts-from-wikisource
opendata.stackexchange.com
crawl=CC-MAIN-2021-21
169726
application/http; msgtype=response
sha1:YCJUDA47S6FNNQITZBMDGHLFIEAZDZBW
<urn:uuid:20b2f4c4-9596-4967-993c-99b5471088d4>
2021-05-06T06:30:07Z
151.101.65.69
text/html
sha1:K5YBILPLDSOJKSSGICZHV3PCE2DP3S2K
<urn:uuid:0ad953de-799a-46ad-a9c4-aba50cbf6ce9>
https://opendata.stackexchange.com/questions/6974/formats-of-texts-from-wikisource
response
<urn:uuid:ae72c749-dc38-47f9-8825-e2e94f2d8c03>
isPartOf: CC-MAIN-2021-21
publisher: Common Crawl
description: Wide crawl of the web for May 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-59.ec2.internal
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/