rsync vs scp vs ssh + tar

Many will agree that copying large directory trees with many files of varying sizes and compressibility from one linux box to another can be such a caterpillar; whereas surprisingly copying huge files is relatively faster. The primary reason being that as the number of files (actual files+directories) increases, the file system and kernel overheads add up to the network slowness.

Here are some interesting results from a small experiment I performed -

The same data was copied between same machine with similar network conditions and loads using different methods with and without compression. Both fedora installations are new and home partitions almost empty (I have no idea at the moment what would happen if they were somewhat filled). 2 dry runs were carried out before actual tests to give cache benefit to all methods.

Data being copied : 1.6GB worth matlab2008a(unix) installation - contains bunch of avi videos in megabytes, moderate number of jar files and many small .m files.

source machine:
Pentium D 820@2.8GHz, 2GB DDR2 dual channel@667, Intel 946 mobo, on board SATA controller; Samsung 160GB SATA hard disk drive@7200rpm and 8M cache, Linux f10 2.6.27.12-170.2.5.fc10.i686,
35GB ext3 source partition (fresh).

target machine:
Pentium core2 duo E4500@2.2GHz, 2GB DDR2 dual channel@667, Intel 965 mobo, on board SATA controller; Samsung 160GB SATA hard disk drive@7200rpm and 2M cache, Linux f10 2.6.27.12-170.2.5.fc10.i686,
100GB ext3 copy-to partition (fresh).

Results

Method command	Timing	avg. CPU util.
uncompressed recursive scp scp -rq matlab_install prashant@10.105.41.19:matlab_install_regular_scp_unc	real 9m54.554s user 0m23.204s sys 0m15.103s	20
compressed recursive scp scp -Crq matlab_install prashant@10.105.41.19:matlab_install_regular_scp	real 11m8.391s user 3m48.508s sys 0m25.200s	85
uncompressed recursive rsync rsync -a matlab_install prashant@10.105.41.19:matlab_install_regular_rsync_unc	real 3m3.604s user 0m26.709s sys 0m21.664s	40
compressed recursive rsync rsync -az matlab_install prashant@10.105.41.19:matlab_install_regular_rsync	real 4m11.651s user 3m11.847s sys 0m31.892s	90
uncompressed tar+ssh tar -cf- matlab_install | ssh prashant@10.105.41.19 'tar -xf- -C ~/matlab_install_hack_unc'	real 2m59.706s user 0m21.428s sys 0m14.020s	20
compressed tar+ssh tar -cf- matlab_install | gzip -f1 | ssh prashant@10.105.41.19 'tar -xzf- -C ~/matlab_install_hack_compr'	real 2m44.349s user 2m7.709s sys 0m18.114s	60

Conclusion

as seen from the timings, rsync and tar+ssh perform close-up; though tar+ssh beats rsync here.
On the other hand though, when updating the huge tree; rsync wins hands down - insane speedups!
scp is not to be used with more than a hundred files. period.

UUID business

Use *nix? Use many distributions at a time? Use many distributions on many disks?

Managing distributions installed on multiple disks can be a real pain in the a**, say you have fedora10 installed on one SATA disk, and Debian on other disk. Now, can you confidently tell which is the system drive? It gets really messy.

Have a portable hard drive you want to be mounted at a fixed place when attached? If there is another USB drive already attached, there is no way you can tell whether the new one is sdc or sdd.

UUID comes to rescue. Though not supported nor understood by the kernel, many distributions provide tools in initrd which can work with UUIDs. Every partition has a Universally Unique IDentifier (UUID), which can be used as a globally unique name for a partition.

Finding UUID of a partition

---

This command needs 'udev' package. It comes pre-installed on many modern Linux distributions. If not, please install it following your distribution's guidelines.

/lib/udev/vol_id --uuid /dev/sdaXX

replacing XX by the partition number.

Using UUIDs

---

1. Painless booting
   UUID comes in handy when you have a separate boot and / partitions. In this case, the kernel and initrd resides on the boot partition, while the root, is on / partition. Naturally, the kernel needs to be told which partition to use as root.
   
   say the root partition is sda8, then
   
   # /lib/udev/vol_id --uuid /dev/sda8
   b036863a-2846-4f57-a6db-e7716f5d903c
   
   
   or use
   
   
   # udevadm info --query=all --name=/dev/sda8 | grep UUID | sed q | cut -d'=' -f2
   b036863a-2846-4f57-a6db-e7716f5d903c
   
   
   This UUID can be used in the kernel parameter. Open up menu.lst of grub (or lilo.conf if you still use LiLo), and make the kernel entry look something like
   
   kernel vmlinuz-2.6.27.9-159.fc10.i686 ro root=UUID=b036863a-2846-4f57-a6db-e7716f5d903c
   
   Now you can safely boot without worrying about which drive has the boot partition.
   
   
2. Pseudo-permanent mountpoint settings
   
   Continuing from the USB disk example, say the partition on the USB drive has UUID 51e3a299-68f3-466f-86ac-428c60420621, an entry can be added into /etc/fstab which looks like
   
   UUID=ba92ef0d-bb4e-4632-bf22-133d9d3fa1f4 / ext3 defaults 1 1
   UUID=b036863a-2846-4f57-a6db-e7716f5d903c /home ext3 defaults 1 2
   UUID=51e3a299-68f3-466f-86ac-428c60420621 /media/thumbdrive auto defaults 0 0
   
   Now, the USB thumbdrive will always be mounted at /media/thumbdrive :)

Adios.
--
prashant

Directx 11 (10) benchmark - comparison with Windows server STD 2008's directx 10

I am trying out both the beta of Windows 7 (build 7000) and release of Windows server 2008 (standard edition). Since it is on the same machine, naturally benchmark comparison was possible and made sense.

Specs :
Windows 7 - Installed on a defragmented 40gig logical partition of 7200SATA disk, which still has other stuff and has 12 gigs free. Regular install, default drivers for sound (doesn't matter in this case).

Windows server 2008 - Installed on a defragmented 20gig primary partition of same disk, which has 5 gigs free. Regular install, Google's drivers for sound (again, doesn't matter in this case).

PC config :
Pentium D 820 (2.8GHz) (I know it is crappy... not a gamer's processor and all, but that's what I've got)
Intel 946GZIS mobo (PCIE 16x), 667MHz FSB.
2Gig (dual channel) RAM.
NVidia 8800GS, factory OC at 650MHz/mem 950MHz, a dx10 card.
Driver version 181.20 (used the same binary on both installations).
(again, that's a budget card; but it is sufficient for me)

3DMark Vantage settings - res 1280x1024, no AA
everything else Extreme, all tests were conducted.

Windows 7 (dx11)
GPU score = 4337.75
CPU score = 17495.47
3DMarks = 5342.16

Windows server 2008 std (dx10)
GPU score = 3975.24
CPU score = 17995.19
3DMarks = 0 (!!)

The demos visually ran faster on DX11 (though I do not have the FPS numbers at the moment).

The GPU scores for DX11 show 9.1% improvement over DX10. In all, windows 7 definitely is worth giving a try IMO (provided I can afford it OR my school has MSDNAA agreement for it). It has other cool 'features' too, of which I doubt the usefulness.

Please drop a comment if you have any suggestions; since as you can see, I am new to graphics benchmarks :)

Writing mathematical equations in OpenOffice

I was writing a report which contained some quite intensive mathematical equations. Naturally, my first options would have been the good old LaTeX. But this time my team-mates decided to go with OpenOffice, as they initially thought it would take less time (because of whatever reasons - GUI etc.). Now here I am, writing a part of a report that is filled with some pretty tough-to-write-in-tex equations, and trying to write it in OpenOffice. After a little Googling, I figured that using Lyx to convert my part of the document to OO format would not be easy.

Just for sake of it, I visited the OpenOffice extensions website and searched for latex. Found OOoLatex !!

After downloading from sf.net, installation was simple. In OpenOffice word:

tools -> Extension manager
choose add and add the downloaded file (version 4.0 beta2 at the time of writing).
and restart ooword.

A new toolbar was added.

Since I am working on Linux, I already have latex and ghostscript installed, so I can freely use the 'Equation' mode which makes a png of the equation and saves the LaTeX in the image properties. Although not as easy as plain LaTeX, it certainly saved me from the pain of inserting all the weird Greek symbols and dealing with subscripts and superscripts, with summations and big matrices :D

Nice utility (or better put - bunch of macros) to keep!

Installing Picasa on Windows Server 2008

Normally, if you try to install picasa on winows server 2008, you would get "current user not authorized to install" error. This happens because of some strange reason and seems to work when these steps are done-

Start->Administrative Tools->Server Manager (as admin)
Select "Features" from the list and then click the "Add feature" link.
Select "Desktop Experience" and restart.

After this, I could install picasa and also got a bunch of other regular desktop utilities.

Sun VirtualBox problem

I just upgraded to VirtualBox-2.0.2_36488_fedora9-1.i386 [not the OSE] and it wouldn't start.
The error message was

Failed to create the VirtualBox COM object.
The application will now terminate.
Callee RC: NS_ERROR_FACTORY_NOT_REGISTERED (0x80040154)


I don't know what it means, but probably looks like some interprocess communication issue.

Ahh ! I removed the [root owned, something I find very weird] directory /tmp/.vbox-prashant-ipc and vbox started very well; but now somehow it can not connect to the host sound.

CPU about to vaporize !!

One fine day, I suddenly felt the urge to OC my CPU for no apparent reason. I had not looked at the temps since a long time, so I just checked the temps before OC'ing, just to get the feel how worse it could get when I actually OC.

I happen to own a powerhouse CPU (~100W), the infamous Pentium D 820. On top of that, I still use the stock cooling, and the room temperature is around ~30 degree Celsius, making board ambient temp~50. I have not even OC'd the chip.
When I run my experimental DES cracker on this dual core boiler-plate, this is what happens - CPU temperature is 84 degrees. The fun part is that this is not the accurate temperature as Intel did some cheap cost-cutting. Some say that the actual could be higher by 10 degrees than what's reported.



Then I realized that I had not cleaned the box for quite some time. I took it apart and get the reason -- ~2mm thick dust layer on the heat sink baffles.


The sad part is that the thermal paste has disappeared and I can no longer get it for free. So the worst case temps are still around 75-80. Hope things will improve when I apply it.

The spurious OC (s)urge went down as fast as it rose.