Wizzy Africa - Tag - kiwix 2014-09-20T18:11:35+02:00 Andy Rabagliati urn:md5:7502 Dotclear Kiwix - enabling offline copies of wikimedia projects urn:md5:aa8ccc7d63bf4518ba5c3b941a164974 2010-04-27T14:09:00+02:00 Andy Computers educationkiwixofflineubuntuwikibookswikipedia <p>Wikimedia projects, including the flagship English Wikipedia, have been restricted in access to people with internet access. <a href="http://blog.wizzy.com/post/www.kiwix.org" hreflang="en">kiwix</a> is opening that up, via its offline reader.</p> <p>I have <a href="http://blog.wizzy.com/post/Kiwix-install-at-Kwena-Malapo-school-Johannesberg" hreflang="en">blogged before</a> about kiwix - this article is an effort to tell other people how to do the same.</p> <p>Kiwix is a cross-platform reader of <a href="http://openzim.org/Main_Page" hreflang="en">zim files</a>. <strong>Zim</strong> is an open, standardised file format to store Wiki content efficiently for offline usage. It is compressed (LZMA), with fast resolution of inter-article links. It is simple (one file), and optimised to run on really small devices like phones.</p> <h2>Readers available</h2> <p><a href="http://blog.wizzy.com/public/Schools/Kiwix_0.9_alpha1_screenshot_en.png"><img src="http://blog.wizzy.com/public/Schools/.Kiwix_0.9_alpha1_screenshot_en_m.jpg" alt="Kiwix screenshot" style="float:right; margin: 0 0 1em 1em;" title="Kiwix screenshot, Apr 2010" /></a> There are two supported platforms for the reader, Windows and Linux. These are graphical, client programs that appear similar to a web browser - with a search box at the top of a display area. These readers need a zim file to read - there is a standard File-&gt;Open dialog box available top-left to allow you to choose a zim file.</p> <p>On the first time you access a zim file, the reader will offer to index it, allowing you to use the search bar. This can take quite a long time, but the results are saved on disk and you do not need to do it again. On linux, these are stored at <strong>~/.www.kiwix.org/kiwix</strong> in case you are interested. Both readers below can be downloaded from <strong><a href="http://sourceforge.net/projects/kiwix/files/" hreflang="en">http://sourceforge.net/projects/kiwix/files/</a></strong> - which I refer to as <em>Sourceforge</em> below.</p> <h2>Windows</h2> <p>To install the Windows reader, you will need to download it from sourceforge above - where you will find <strong>kiwix-0.9-alpha1-win.zip</strong>. You need to unzip that file under your &quot;Program Files&quot; directory, and create a launcher for your desktop pointing to &quot;<em>Program Files/kiwix/kiwix.exe</em>&quot;</p> <h2>Linux</h2> <p>On sourceforge you will also find a deb for the very latest Ubuntu linux 10.04 (Lucid Lynx), called <strong>kiwix-0.9-alpha2.deb</strong>. If you are running 10.04, download this file and install it with</p> <pre> sudo dpkg -i kiwix-0.9-alpha2.deb </pre> <p>It will undoubtedly complain that some or all of <strong>libicu42, libxapian15, xapian-tools, xulrunner-1.9, zlib1g, libbz2-1.0, liblzma1, libmicrohttpd5</strong> are missing. That is fixable by running this :-</p> <pre> sudo aptitude install libicu42 libxapian15 xapian-tools xulrunner-1.9 zlib1g libbz2-1.0 liblzma1 libmicrohttpd5 </pre> <p>Which will take care of those dependencies, and finish installing kiwix for you. All these packages are in the lucid repositories.</p> <p>If you are running something other than Ubuntu 10.04, you have slightly more work ahead of you. I suggest that you download the source package, <strong>kiwix-0.9-alpha2-src.tar.bz2</strong>, and get the required development packages for your distribution. You will need <strong>libxapian-dev</strong> (for search), <strong>libbz2-dev, libicu-dev, liblzma-dev, xulrunner-dev, zlib1g-dev, libmicrohttpd-dev</strong> in addition to the standard development tools. If you build an RPM or other finished package, let me know and I will get the files up on sourceforge if possible.</p> <p>The reader under linux is called kiwix, and you can use the File -&gt; Open menu to find a zim file to read. It appears in Applications under Education.</p> <h2>Zim files</h2> <p>Finally, you will need a zim file as a target for your reader. Currently, kiwix has these available at <strong><a href="http://tmp.kiwix.org/zim/0.9/" hreflang="en">http://tmp.kiwix.org/zim/0.9/</a></strong> - no doubt they will move to a more permanent place later.</p> <p>The files are <strong>big</strong>. Available now (April 2010) is an excellent collection of english articles, selected for school use and painstakingly checked for vandalism (thanks <a href="http://www2.potsdam.edu/walkerma/" hreflang="en">Martin Walker</a>!!) as <strong>wikipedia_en_wp1_0.7_30000+_05_2009_beta3.zim</strong>. This is 30,000 articles from May 2009 and is a wonderful resource.</p> <p>There are other languages available, but they have not had the same scrutiny as the English archive above. Since the associated wikipedias are smaller, these are a snapshot of the <strong>entire</strong> wikipedia - Arabic, Hebrew, Italian and Persian.</p> <p><a href="http://www.wikimedia.org/" hreflang="en">Wikimedia</a> has other project besides wikipedia - like <a href="http://en.wikibooks.org/wiki/Main_Page" hreflang="en">wikibooks</a>, <a href="http://en.wikiversity.org/wiki/Wikiversity:Main_Page" hreflang="en">wikiversity</a> and <a href="http://www.wiktionary.org/" hreflang="en">wiktionary</a> that all make excellent candidates for zim files.</p> <h2>Server</h2> <p>The linux installations have another mode - a webserver, serving the zim of your choice up via http. That is how I set it up at Kwena Malapo school above. It means that on a network at a school, only one installation is necessary, and all other computers on the network can reach it as if from a website.</p> <p>First, you will need to build the index. I keep the zim files in /opt/kiwix as they are so huge. I put the index files in the same directory, so to build the index for /opt/kiwix/wikipedia_en_wp1_0.7_30000+_05_2009_beta3.zim I run the following :-</p> <pre> export ZIM=/opt/kiwix/wikipedia_en_wp1_0.7_30000+_05_2009_beta3.zim kiwix-index $ZIM ${ZIM%zim}idx </pre> <p>Then you can run the server as follows (please read ―― as double dash ) :-</p> <pre> export ZIM=/opt/kiwix/wikipedia_en_wp1_0.7_30000+_05_2009_beta3.zim kiwix-serve ――index=${ZIM%zim}idx ――port=8080 ――daemon $ZIM </pre> <p>This instructs kiwix to serve the same English wikipedia with its index from port 8080 on this computer. You can then access it as http://192.168.0.254:8080/ - replace the IP address by the IP address of this computer.</p> <p>Kiwix is open-source, and uses freely-available toolkits for compression, search and rendering. The zim format is optimised for use on small devices with low computing power, so you can expect to have a port onto your favourite smartphone OS in the future. And, yes, it handles right-to-left Hebrew, and arabic writing thanks to its xulrunner mozilla heritage.</p> http://blog.wizzy.com/post/Kiwix-enabling-offline-copies-of-wikimedia-projects#comment-form http://blog.wizzy.com/feed/atom/comments/511629 Kiwix install at Kwena Malapo school Johannesberg urn:md5:41f93e63e5ce5bae16fde7f5c54b8242 2010-03-01T12:47:00+02:00 Andy Computers AfricaEducationinternetJohannesbergkiwixRotary FoundationSandtonSouth AfricaWikipedia <p><img src="http://blog.wizzy.com/public/Schools/.Kwena2_t.jpg" alt="Kwena Computer lab" style="float:right; margin: 0 0 1em 1em;" title="Kwena Computer lab, Mar 2010" /> Kiwix, an offline wikipedia selection, is installed at Kwena Malapo school.</p> <p><a href="http://blog.wizzy.com/public/Schools/Kwena1.jpg"><img src="http://blog.wizzy.com/public/Schools/.Kwena1_s.jpg" alt="Kwena Malapo school" style="float:right; margin: 0 0 1em 1em;" title="Kwena Malapo school, Mar 2010" /></a> I have <a href="http://blog.wizzy.com/post/Offline-copies-of-wikipedia">blogged</a> before about creating offline copies of wikipedia for use in school computer labs that do not have internet access.</p> <p>Wikimedia, the umbrella organisation behind wikipedia and other related projects, is also keenly interested in offline uses of wikipedia. From November through January, the <a href="http://strategy.wikimedia.org/wiki/Task_force/Offline" hreflang="en">offline taskforce</a> had a series of IRC meetings where we attempted to answer questions relating to the use of these offline copies. We came up with <a href="http://strategy.wikimedia.org/wiki/Task_force/Recommendations/Offline" hreflang="en">four recommendations</a>, one of which addressed schools, and another cellphones.</p> <p>In conjunction with this effort, Emmanuel Engelhart made progress and could deliver me both a compressed zim database of 30000 articles (2.4 Gigabytes, including pictures) and the software, called <a href="http://www.kiwix.org/index.php/Main_Page" hreflang="en">kiwix</a> that could read and deliver this database served by http - the standard web protocol.</p> <p>There is another tool, kiwix-index, that will read this large zim database and create an index. If this index is supplied to kiwix-serve then a floating javascript search box appears, and the whole article collection can be searched.</p> <p><a href="http://blog.wizzy.com/public/Schools/Kwena2.jpg"><img src="http://blog.wizzy.com/public/Schools/.Kwena2_s.jpg" alt="Kwena Computer lab" style="float:right; margin: 0 0 1em 1em;" title="Kwena Computer lab, Mar 2010" /></a> I was asked by Mark Batchelor of the <a href="http://www.sandtonrotary.co.za/" hreflang="en">Sandton Rotary club</a> to help him with the computer lab they installed at Kwena Malapo secondary school. This school, with principal Michael Maligana, has 800 students and operates out of simple prefab buildings close to Lanseria airport in Johannesberg. The computer lab is a converted shipping container. A new door had been put in, with four windows and roof ventilation. A network of donated computers running Windows XP and Microsoft Office had been installed.</p> <p>My proposal to Mark was to install kiwix as a 'poor mans internet' - an rich information resource navigable via a browser, with hot-linked words that drill down to more articles. We all recognise this as Internet, but the principles of the information age could now be brought to a school that has no internet connectivity.</p> <p>Ricardo, Mark's Windows technician, and myself travelled out to the remote farm school on a saturday, and met Michael Maligana. I commandeered one of the computers, installed Ubuntu Linux, kiwix-serve and its software dependencies, the huge zim database and its associated index for search. We networked all the computers, and set the browser home page on each to the kiwix front page.</p> <p>After some networking and power hassles and four hours in the hot environment we finally got it all working. Kwena had 30000 wikipedia articles selected especially for a school environment, and was one step closer to the information age.</p> <h3>Update:</h3> <p>I wrote a blog post describing how to <a href="http://blog.wizzy.com/post/Kiwix-enabling-offline-copies-of-wikimedia-projects" hreflang="en">get the software and install it</a>.</p> http://blog.wizzy.com/post/Kiwix-install-at-Kwena-Malapo-school-Johannesberg#comment-form http://blog.wizzy.com/feed/atom/comments/490812 Offline copies of wikipedia urn:md5:60cad55b79bed6ff7b7608cd8a9b982a 2009-11-25T09:23:00+02:00 Andy Computers africaEducationInkululekointernetkiwixOLPCSouth AfricaWikipedia <p><a href="http://blog.wizzy.com/public/Mediawiki/489px-Wikipedia-logo-en.png"><img src="http://blog.wizzy.com/public/Mediawiki/.489px-Wikipedia-logo-en_t.jpg" alt="489px-Wikipedia-logo-en.png" style="float:right; margin: 0 0 1em 1em;" title="489px-Wikipedia-logo-en.png, Nov 2009" /></a> I have been involved for a number of years with Hilton Theunissen and the Shuttleworth Foundation and their efforts to bring computers to township schools. A part of that software suite was an offline copy of wikipedia.</p> <h2>Early attempts</h2> <p>I have <a href="http://blog.wizzy.com/post/mediawiki-psa" hreflang="en">blogged before</a> about my own project <a href="http://blog.wizzy.com/post/Habeni-Primary" hreflang="en">Wizzy Digital Courier</a> putting Thin Client labs down in South African classrooms. That also included a copy of the english language wikipedia.</p> <p>Initially in 2003 I took the whole of the then-existing English wikipedia, installed a copy of the mediawiki software in conjunction with mysql and apache as database and webserver respectively. The whole thing was around 18 Gigabytes - quite a handful.</p> <p>It worked well, but I had various complaints on the unsuitability of the material - it was a single snapshot, and had not (could not) be proofread, so it had vandalism, and quite explicit articles around sex. Oops. But - it had search, it had a vast amount of useful information on all manner of subjects.</p> <p>Very soon the English wikipedia bloomed to hundreds of Gigabytes, making it completely unmanageable in terms of size. I couldn't download it, and I couldn't proof it. What to do ?</p> <h2>Wikipedia Version 1.0</h2> <p>Wikipedia has its own community of people, and among them I connected with <a href="http://schools-wikipedia.org/" hreflang="en">Andrew Cates</a>, of the SOS Children website, who came up with a selection of articles (1000 or so) as an HTML dump that he and some others painstakingly proof-read for suitability as a children's educational resource. (He has a larger article collection now). <a href="http://jonathancarter.co.za/" hreflang="en">Jonathan Carter</a> helped package this for the tuXlab installs for the Shuttleworth Foundation.</p> <p>There is a lot of work to do preparing such a collection.</p> <ul> <li>Which articles should be selected ?</li> <li>all the article text in the HTML dump must be stripped of links to articles <em>not</em> in the dump</li> <li>it must be proof-read</li> <li>associated pictures must be incorporated</li> </ul> <p><a href="http://blog.wizzy.com/public/Mediawiki/Screenshot.png"><img src="http://blog.wizzy.com/public/Mediawiki/.Screenshot_m.jpg" alt="Screenshot.png" style="float:right; margin: 0 0 1em 1em;" title="Screenshot.png, Nov 2009" /></a> Other people became involved, in particular Martin Walker from the State University of New York at Potsdam. Systems were put in place to help on article selection. A project was started - the <a href="http://en.wikipedia.org/wiki/Wikipedia:Version_1.0_Editorial_Team" hreflang="en">Wikipedia Version 1.0 Editorial Team</a>. Articles were <a href="http://en.wikipedia.org/wiki/Wikipedia:Version_1.0_Editorial_Team/Assessment" hreflang="en">assessed</a> - both for quality (from Featured Article down to Stub) and for importance (from Top to None). These assessments are placed on the article Talk page, and a robot goes through them all on a regular basis and collects the results on project pages like <a href="http://en.wikipedia.org/wiki/Wikipedia:Version_1.0_Editorial_Team/Index" hreflang="en">this</a> and <a href="http://en.wikipedia.org/wiki/Wikipedia:Version_1.0_Editorial_Team/Africa_articles_by_quality" hreflang="en">this</a> - conveniently doing all the heavy lifting to present sortable tables of the state of all the articles.</p> <p>Thus you can find Top Importance articles of poor quality, and can highlight that page for improvement. You can cherry-pick Featured Articles to add to the collection. These tools made the article selection process far more manageable.</p> <p>Offline wikipedia was <a href="http://strategy.wikimedia.org/wiki/Task_force/Offline_Task_Force" hreflang="en">identified as a priority</a> by the wikimedia foundation.</p> <p>I assisted in the post-processing by writing a script that would search all the chosen articles for <a href="http://en.wikipedia.org/wiki/User:Lupin/badwords" hreflang="en">'bad words'</a> - an indication that the article has been vandalised - and then a cleanup crew has to go through these to check and possibly remove material.</p> <p>Now to package all this conveniently. A French company called <a href="http://www.linterweb.fr/" hreflang="fr">Linterweb</a> came up with <a href="http://www.okawix.com/" hreflang="en">Okawix</a> - all these articles and pictures packaged in a file, with a cross-platform reader to navigate the collection. Why do we need a reader ? To implement search.</p> <h2>Search</h2> <p>For many of the places we put an offline wikipedia down, it became the 'Internet' for the children in the classroom. They had no net connection, but the principles of self-paced learning, hyperlinks for tangential information, and other net paradigms made it the 'killer app' of their little school. For Internet, you need search. For a wikipedia collection of a few thousand articles, you need search. Search needs a computer - you cannot put search on a CD or DVD or USB stick.</p> <p>For a standalone computer, a Reader is needed to perform this function. From a basic HTML dump, navigated with a browser, Javascript can be pressed into service, but I have found it inadequate. In the tuXlab Thin client labs, a small network of old computers is networked to a powerful server - and I want the wikipedia collection to be browsed via HTTP, and the search to be performed server-side.</p> <h2>Categories</h2> <p>A related problem is how to organise all these thousands of articles. They are mushed together in a big web of information, but where is the structure ? Wikipedia proper has categories - useful for grouping similar articles, but the arbitrariness of the invented categories means that it has been a problem incorporating them into the static dump. Martin battled for days to make a river in Poland appear 'automatically' and conveniently under a Poland heirarchy.</p> <h2>Metadata</h2> <p>In computer jargon, this is called <em>metadata</em>. It is structure beyond the mere linking of articles. There is other metadata - like the <em>Importance</em> assessment scale. We need to extract all that metadata and place it alongside the article tree so it can be used for indexes.</p> <p>On the 24th November we had an IRC meeting - an online chat between all interested parties spread around the world. Much of this was discussed - and one thing became clear - the Wikimedia foundation needs to concentrate more on the process of generating a release, rather than the end product like Okawix. That means tools to work with the Metadata, tools to package the pictures and article references in such a way they can be optional. Perhaps targeted article collections, like Mathematics, Chemistry, Africa, Oceans. Let other organisations do the work of packaging and marketing.</p> <p>To allow computers to do the work - we need good metadata. Assess articles. Rugby articles are not Top importance, except in the context of sport.</p> <p>I think the article collection should consist of a number of different pieces, to be incorporated as necessary.</p> <ul> <li>The text of the selected articles</li> <li>Pictures for those articles</li> <li>metadata to support this collection</li> <li>a text search index, like one created by <a href="https://help.ubuntu.com/community/Namazu" hreflang="en">namazu</a>, for those tools that can use it.</li> </ul> <h2>Future efforts</h2> <p>Though a lot of effort on offline wikipedia collections is targeted at schools, and Third World, there are other target markets. One we have not really addressed yet is the cellphone as a wikipedia platform. A cellphone implies connectivity, but these days it is becoming a universal platform - a camera, a music player, a gaming box, GPS. Personally, I would like a text-only wikipedia collection of lots of articles, but only the <em>lede</em> paragraph - the first section of a wikipedia page that introduces the subject. It is a song by Black-eyed Peas. It is a river in Poland. That way, I can carry all of this on my phone without paying airtime.</p> <h3>Cellphones</h3> <p>Cellphones have huge penetration in the Third World. I tell tourists I take to the townships that South Africans spend their money on cellphones and hair. Maybe we should concentrate there, as much as schools ?</p> <h2>Update:</h2> <p>Okawix offline reader <a href="http://blog.wikiwix.com/en/2009/12/07/okawix-et-openzim/" hreflang="en">standardises on OpenZIM</a> data format.</p> http://blog.wizzy.com/post/Offline-copies-of-wikipedia#comment-form http://blog.wizzy.com/feed/atom/comments/462239
http://blog.wizzy.com/feed/tag/kiwix/atom
blog.wizzy.com
crawl=CC-MAIN-2014-41
26252
application/http; msgtype=response
sha1:QDV7MXCRWDPOVBZOTADMF4K67R4FWOVA
<urn:uuid:b5fe43ba-0509-4367-bb51-49b58116a9f8>
2014-09-20T19:59:53Z
217.70.184.34
sha1:6S7YI3WXRQGOAORSDSBUTUAQX3O76Y6G
<urn:uuid:853a89f8-8402-4722-b866-36ed32342f06>
http://blog.wizzy.com/feed/tag/kiwix/atom
response
<urn:uuid:bd77679c-9d39-42cb-a457-433746c6f263>
robots: classic
hostname: ip-10-196-40-205.us-west-1.compute.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2014-41
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for September 2014
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf