This topic is locked.

Oh, so it worked after all? Excellent \o/
Yep, it worked incredibly well, at the beginning it was a bit confusing, but after some fiddling I managed to use it effectively XD

So, there's a huuuuuuuuuuge update coming as I write this post, I'm uploading to the archive 28 extra packs (384 up to 412), and I'm working on the next batch, the catch is that my internet connection only has a 2Mb up link.... soooooo yeah it might take a while XD

But as soon as it's uploaded, I'll edit this post to let you all know =3

EDIT: Upload of the new packs finished, phew that took a while, might have another batch of packs for tuesday/wednesday, but I'm not completely sure XD

Updated the previous post on the archive, and will keep posting the new packs (for this year) on that same post, I'll post when there are any updates to it =P

EDIT 2: Another huge update!!!!!!! New batch of packs is uploaded to the archive (pack 413 up to pack 452), took a while, but I'm closer to putting the packs up to date XD

Enjoy!

https://archive.org/details/Konachan-siterip-2015
The original post has been updated with a link to the 2015 archive.

Keep up the excellent work, and thanks again. \o/

Edit: Once you have everything current uploaded through, say, the end of June (which hasn't quite arrived yet :3 ), since I figure the half-year point would suffice as a demarcation line, could you create a new all-in-one torrent with every sitepack assembled up until then?

And, another suggestion... It seems that each yearly archive is numbered for the year after when the images were uploaded, but you've included a couple months of 2013 images in this "2015" archive. Perhaps you could move sitepacks 374 through 393 to the 2014 archive and start the 2015 archive with pack 394 (which contains post #175631, first image of 2014)? And then continue that through all the 2014 images and begin the "2016" archive with the pack containing the first 2015 image?

Edit 2: Or if you have trouble editing the 2014 archive, then... Could you write a review on the 2014 archive which points out that the 374 through 393 segments from the 2015 archive belong with the 2014 one, and also make a note of that in the description for 2015?
Sure, I'll give it a try and see if I manage to build the remaining packs in order to leave them at a midpoint, actually I was also thinking about doing the packs on a half year basis so you beat me to it XD

About your second point, yeah, the packs were numbered for the year when images were uploaded, but since naikoto left, posts were still being created, and I started building the packs I think about 3 months ago or something like that, so there were A LOT of posts that should've been on the 2014 pack, but they were left out, so I included them in the 2015 since it was the current year.

Also I was thinking of making the packs related to the same year, let me explain, I'll update the 2015 pack until let's say December 27th this year, and then the 2016 pack will start with the posts from the 28th onwards until December 27th, 2016, then the 2017 will start on the 28th and so on.

About editing older packs on the archive, I can't do that, the reason being that older packs belong to a different user of the archive, and therefore cannot be modified by me, but I'll add the note on the metadata info of the pack in order to clarify that, and let people know that 2015 packs have some of the older posts.

About building the torrent, first I might need to clear some space up on my hard drive, because at the moment, all the packs are (I believe) about 180GB, now the main issue becomes my internet connection, at the moment my connection is a 10Mb download with a 2Mb upload (ISP's in Costa Rica are really bad....), so uploading the 180Gb might take, I don't know, 2 years? XD

That's the reason I uploaded them to the archive, and in the archive they create a torrent for each yearly pack, so I might post them around here.

Any other suggestions are welcome XD
Well, the torrent doesn't help anybody unless people seed it. I wouldn't mind seeding it (if I upgrade my hard drive first :3, something I've been meaning to do), but my Internet connection is more often than not disconnected, so it wouldn't be that helpful.

Maybe if that changes, then I could seed it.

As for the upload speed, if by 2MB you mean 2 megabytes per second, then you would finish a 180GB upload in 180x1024 = 184320 seconds = 3072 minutes = 51 hours 12 minutes.
As for the upload speed, if by 2MB you mean 2 megabytes per second, then you would finish a 180GB upload in 180x1024 = 184320 seconds = 3072 minutes = 51 hours 12 minutes.
Nope, it's Mb/s ---> Megabits per second with lowercase b =P

So, 180GB x 1024 = 184320MB x 1024 = 188743680KB x 8 = 1509949440Kb / 1024 = 1474560Mb / 2 = 737280s, roughly 8 days 18 hours of non stop upload, that's without taking into account the new packs, and people using some upload speed (connection is shared with my family XD), so yeah.... let's say it would take about twice that, so nearly 17 days or even more of non stop uploading and no online gaming for me @[email protected]

EDIT: New batch of packs is up. Packs 453 to 486 are now uploaded to the archive post, now as far as it is, the packs are nearly up to date, I'm just waiting for it to be the end of June, and compile the last pack (or packs) in order to leave them up to that date.

EDIT2: The last 2 packs for this half year are up, packs 487 and 488 have pics uploaded until June 30th, so with those 2 packs, we're finally up to date!!
Next bunch of packs will be created and uploaded in December for everyone to download.

Enjoy!!
I'm not sure if you're aware of it or if you can do anything about it, but the 2015 torrent only seems to include up to pack 412. Thanks a lot for the updates.

E: Torrent is indeed fixed now, thanks for your troubles ^^
rayanamukami said:
I'm not sure if you're aware of it or if you can do anything about it, but the 2015 torrent only seems to include up to pack 412. Thanks a lot for the updates.
That's weird, according to the archive FAQ:

Internet Archive BitTorrents are automatically made for community-contributed items in many collections, and automatically updated when item contents or metadata change.
So supposedly, the torrent should be up to date with new files ._.

Will try to ask on forums or land a mail to one of the mods and check if it can be fixed, so for the moment I think you might need to download the individual files ^-^''

EDIT: Managed to post about it on the archive forum, and the issue with the torrent not showing all the files has been fixed, so torrent download of the whole pack is now available.

Enjoy!!
Compiling new packs for upload, these will contain all the images uploaded to the site since July this year (when last pack was uploaded....I think XD) up to the end of the year (will try to get images uploaded on the 31st of December), and I will update the thread as soon as these are uploaded to the archive
Thank you for maintainig this, I'm lookong forward to it. :)
Happy New Year to everyone!!! (/^▽^)/

As promised, here's the new update to the site packs, added packs 489 to 522, with posts made from July 2015 (I think xD) up to December 31st 2015 (last post added to the packs had id: 212519) This batch was added to the same archive post where previous updates were made, the link for this is: https://archive.org/details/Konachan-siterip-2015

The next update to the packs will be made around July 1st to keep the 6 month period between updates, and it will be made on a different post on the archive to keep them organized by year.

In any case someone finds any issues with the packs, or with the archive post, you can notify me on this post, or send me a dmail.

On a different topic, Minabi is there any chance of changing this thread to a sticky (damn that sounds bad xD)?

Enjoy the new packs!! (≧∇≦)/
Thanks again for your work here \o/

IMO there are already too many stickies =), but having the availability of the sitepacks immediately visible to anyone who visits the forum for the first time could be useful for maximizing the utilization of this feature of Konachan. I wonder though about the effectiveness of the thread name for advertising its function; I, for one, wouldn't intuitively understand the concept of a "sitepack" from just the word/without prior knowledge of the matter. I'll update the original post with a description, and I have another idea as well (eh heh heh...), but further suggestions are welcome if anyone has them.

I'll go ahead and set it to sticky for now, and if anyone objects then they can undo it if they wish, although I would request that they at least explain their reasoning for doing so.

Edit: Update of the original post complete, now with a functional description and some minabi humour.
minabiStrikesAgain said:
IMO there are already too many stickies =), but having the availability of the sitepacks immediately visible to anyone who visits the forum for the first time could be useful for maximizing the utilization of this feature of Konachan
The idea of making the post sticky was exactly to make it visible to users, since there are times in which I've seen the post go to the 2nd page and some users (specially new ones i think) don't go past the first page XD

minabiStrikesAgain said:
I wonder though about the effectiveness of the thread name for advertising its function; I, for one, wouldn't intuitively understand the concept of a "sitepack" from just the word/without prior knowledge of the matter. I'll update the original post with a description......

.....Edit: Update of the original post complete, now with a functional description and some minabi humour.
Description is great!!! XD And it explains in a clear way the purpose of the packs

minabiStrikesAgain said:
and I have another idea as well (eh heh heh...), but further suggestions are welcome if anyone has them.
I'm open to suggestions in order to improve the packs =P

Thanks Minabi n.n
Is there any fixed urls for this, I would love to include this in the Konachan fan app I built, if there is a page, url or api or so on to pull all the known links that would be great, thanks guys and thanks for the packs :)
At the moment the links to the sitepacks are all located on the Internet Archive pages, from there you can access each of the packs individually, as each page for a specific year, contains a set of packs

There's no mirror of the packs created (for now, kinda thinking of giving it a go), nor exists a torrent file with them

=P
Shaggy410 said:
At the moment the links to the sitepacks are all located on the Internet Archive pages, from there you can access each of the packs individually, as each page for a specific year, contains a set of packs

There's no mirror of the packs created (for now, kinda thinking of giving it a go), nor exists a torrent file with them

=P
Archive offers a torrent file, also if there always going to be on there that's cool I can just include the page that shows them all that will be as useful as a user would need :)
Yeah forgot about the torrent created at the archive xD

What I meant is that there's no torrent with all the packs included, the ones at the archive are specific for packs of a certain period, therefore, there are several individual torrents.

Was trying to work on building the torrent, but the HDD with all my files decided to die, and now I'm waiting for the spare to fix it.

As far as I know, the Archive files will stay there as long as the Archive project stays alive, so sure, you can add the individual links of the Archive pages
Would it be possible to add a current Json dump to the 2015 pack? I am trying to filtering out the questionable and explicit images but that's only possible with a current konachan.json
Percy23 said:
Would it be possible to add a current Json dump to the 2015 pack? I am trying to filtering out the questionable and explicit images but that's only possible with a current konachan.json
It is possible, would need to get my hands on the dump though or get directed as how to build it, sry for the late answer n.n'''
Shaggy410 said:
... would need to get my hands on the dump though or get directed as how to build it...
??? Are you not using a json or xml dump to create the packs in the first place? Or are you using the DOM? You're not parsing the html, are you?

This will return all non-deleted safe rated posts from 2015 (1000 per page).

1000 posts per page is the limit. If you want xml, just replace the "json" in the url with "xml".
Nope. In order to create the packs I'm using a program called DanbooruDownloader, which does something similar, as it uses xml, and it lets me download posts based on the ID of each image, skipping deleted posts and letting me specify whatever parameters I want
It's that time of the year again, as half a year has passed since the last update (holy cow time sure flies lol) here's the new update to the sitepacks, so I present to you:

Part 1 of the 2016 Konachan sitepacks!!!! *public acclaims*

This new Archive link contains packs 523 to 553

https://archive.org/details/Konachan-siterip-2016

In case someone finds any mistake with the packs, or the torrent provided by the Archive, just let me know and I'll fix it!

Next update will be made on the end of December this year / begin of January 2017, so for the mean time, enjoy!! =P

EDIT: Heh...heh...I think I misread my calendar as it's only June and the update was supposed to be made on July..... will give a little update at the end of the month with the remaining packs....and that kids is what happens when you don't sleep well for several days xD
Thank you very much!!!
Finally my internet connection let me update the pack for 2016....

So now live on the Internet Archive post, packs 553 up to pack 587 containing all the images posted from June 2016 up to January 1st 2017

Enjoy the new packs =P
Nice, thank you!
http://konachan.com/forum/show/22483?page=3
konachan.com
crawl=CC-MAIN-2018-17
66341
application/http; msgtype=response
sha1:FAPRHO76GXTYG5GNOOS5WABRCX7W3UJA
<urn:uuid:18546df4-3cba-40f8-9cdf-ef841f471496>
2018-04-24T20:46:22Z
104.28.2.54
text/html
sha1:VDFJ3EJJIYGB5YUKPILQCEPEMCPHYSQS
<urn:uuid:e9edca2e-05cd-4f08-9d02-4b2b4f387402>
http://konachan.com/forum/show/22483?page=3
length
response
<urn:uuid:053c424d-99cf-494c-adf8-133d0c29c3f7>
robots: classic
hostname: ip-10-109-231-228.ec2.internal
software: Nutch 1.6 (CC)
isPartOf: CC-MAIN-2018-17
operator: Common Crawl Admin
description: Wide crawl of the web for April 2018
publisher: Common Crawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf