Speeding up Jigdo

What is Jigdo
Jigdo is a new tool that has been accepted by Debian as THE way to distribute their ISOs. The reasons are:
1. Its a simple tool that can run making use of some very basic linux utilities that are available on most linux distros
2. Can make use of partially downloaded files
3. Can make use of the previous release of an ISO and download only the packages that have changed and recreate the latest release of the ISO locally.
4. Reduces time/resources for both the user and the server that is hosting the ISO

Normally the speed is limited to the number of connections that we can make to the particular server. So more the number of connections that I use to download a particular file, the faster is the download. For example...if I download a file using wget and I get 20KBps then if I use a tool that supports splitting up the file into multiple parts, say I split the file into 5 parts, I can get atleast a 100KBps. Now thats FAST...dont you think?

Okay...so what tool can be used and how can we use it with jigdo?

My choice is aria2c. The reason is that I have used it for generic downloads and have had good performance. It supports mirrors, splitting the download, proxy, torrents, metalinks, https etc... For a complete feature list see: http://aria2.sourceforge.net/

Lets try to understand how jigdo works so we can modify it suitable:

jigdo-file is the tool used to prepare the ISO for distribution. It creates a .template extensioned file that contains the following:
1. The md5 sum of each file that is in the ISO
2. The directory structure for the file in the ISO
3. In case some files were passed to jigdo-file other than ISO, then those
4. The entire directory structure and padding information from ISO

Another file that is created by jigdo-file has an extension of .jigdo. This file uses the same format as windows ini files and has the following information
1. The [Parts] section contains md5 sum of file in ISO mapped to the URL from which it can be downloaded.
2. The [Servers] section contains URL for Core and Updates folders. These can be modified by us to point to the closest mirrors that we have.
3. [Image] gives a list of images on those servers. This is used to let the user choose which ISO he wants to download-say DVD or CD etc.

This file is pretty self explanatory.

How to go about making the download:

We need to download the .template file to the local machine and run jigdo as follows:
jigdo-lite FC-20070401-6-x86_64.jigdo
Lets assume Im trying to get the latest FC6 respin while having a copy of the first released FC6 disk.

1. jigdo-lite checks if the .template file is present in the directory. If not then it downloads it.
2. I mount the iso of the old FC6 disk in some mount point and ask jigdo to scan it.
3. Jigdo scans the files present in that and checks that against the md5 sums it has in the template file.
4. Based on this information it generates a .list file that contains all the files that need to downloaded.
5. It downloads files into .tmpdir folder. After each 10 file, the files are written into an iso.tmp file in the same structure that is mentioned in the .template file.
6. The downloaded and existing files are merged to recreate the new iso with the updated files.

So by downloading just the changed files, the entire new release iso is recreated on our local machine.

Now coming back to where we started:
jigdo uses wget which supports a lot of features except especially splitting of the download into multiple parts. This is a major disadvantage. So let us use aria2c which supports that and a lot of other features.

jigdo-lite as it turns out is a simple shell script that calls wget in turn to get the files. So all we need to do is find this line in jigdo-lite

wget --user-agent="$userAgent" $wgetOpts "$@" || return 1


and replace with

aria2c $ariaOpts "$@" || return 1

At the top of the file set ariaOpts as follows:
ariaOpts="-s4 --ftp-pasv --http-proxy=PROXY:PROXYPORT --http-proxy-user=ADUNAME --http-proxy-passwd=ADPASSWD"

This will make the script split the file into 4 parts.

Now run jigdo-lite and you stand to get atleast a 4x speedup :)


Useful Links:

http://atterer.net/jigdo/
http://aria2.sourceforge.net/

vmware converter

as part of fun at work ;) today im moving my entire windows installation into a vmware image which i intend to run from linux fc6 custom build through vmware server.

initially my c drive showed around 7+ gb used...later after moving some files and uninstalling all that i didnt like i got to around 5.2 gigs. am currently making an image of 6gigs out of my cdrive. if everything goes on fine...will be reporting back from my windows under linux session....

looks like it would another hour atleast before the image is created on my very slow machine.

1hr and 15mins and i still have 43 mins to go :( boy is it slow....but im ready to wait as this would help me in getting rid of the devil :)

2hrs and still 24 mins to go :(

by now you would be wondering 'what the hell is the config he is using?'....the answer to that would be 1.7 amd with 1.2gb RAM.

2hrs and 22 minutes later its over...so now let me go for a reboot...bye...


finally after 2.5 hrs and a long time waiting for it to bo0t(gave it only 256 mb ram...didnt want to waste mem giving it to 'doze ;))

It works like a charm...but a very slow charm. but ofcourse it is a lot better than doing all the manual editing that needs to be done to the registry to get it working on vmware from linux and in addition to that we get it all in a file so if needed we can increase the size of the partition.

okie people...its time to fiddle with 'doze under linux...catch u later :)

Forgetting

Forgetting is being explained by wikipedia as "retention loss-a process by which old memories are replaced with new ones in an delicately balanced optimized way".

Funny is'nt it...your inability to remember to pick up that pack of bread from the grocery store is in actuality nature's own algorithm to cleanup your mind. More over the algorithm is said to be optimized.

So is there a good way to forget something? It turns out there is. It is called priorities and preoccupation.

Though the human mind is an amazing piece of machine...it runs on some simple rules that are set or rather tuned into us by the people around us, the education that we have had and the experiences that we have shared. A mind that is on the constant lookout for ways to make money for the next meal cannot think about the war going on in some far off oil producing country though it directly affects the cost of the food that he is currently trying to earn for. Here the priorities are not the same. Now a well fed politician who has most of the luxuries of life will certainly reflect his thoughts on the war covering not just the fuel price hike but also about the injustice happening to those people, the pollution, the money wasted all though these are in effect felt by him only at the mind's level.

Forgetting does help the common man in one thing. He gets to forgive people. Forgetting makes it easier to forgive.

So friends dont forget to forget ;)

lesson for the day :)

1. google for it
2. the tools used make a lot of difference
3. if a solution looks wrong...then there should be one that is much better and saner

summa nachuku

nanbargalae....en vanakkangal...
remba bore adikkarathala ethavathu urupadiya pannalamnu tv pottu partha intha maku payaluga blr'la 2 channel mattum thaan kudukkaranuga...enna panruthu?

Okie people...all these days I had been thinking that linux is one very complete operating system. Complete in the sense...I believed that there wasnt much more to be done in that. But after having through a fair portion of the kernel code I have found that there is a lot more to be done...a lot of cleaning has to be done. A lot of performance changes can be achieved. Some of the biggest minds are working on this. And Im helping in a very small way.

It is probably very late to join the community but as everyone says-better late than never. Even then this is pretty much a good time to join the linux community-the reason being Linus has kinda frozen work on 2.7 and has asked the community to do the cleanup and performance patches necessary for 2.6

So people...what are u waiting for? Come in and make it happen :) There is a lot to do and a lot more to get done.

extended todo :)

1. What are Syslets
2. What are Threadlets
3. What is the difference between Syslets/System calls and Threadlets/Threads
4. What is advantageous under which situation
5. Maybe a whitepaper on these :)

Todays ToDo

1. Must finish reading up on syslets and threadlets

new assignment

Combined with my usual laziness....the following had eaten up my time...
1. FC6 install, reinstall, reinstall, reinstall again
2. Xen, VMWare
3. New project at my company
I hope to make more entries here...
Lets see how things work out.

one more blog...

Hmmm...so that's one more blogger getting online :)
Lets see how Im going change this world ;)