Linux / Gentoo Linux

Ubuntu Kernel 3.8 and NVidia Drivers

While playing around with various mainline kernels I found out that the vanilla NVidia driver (310.xx and 313.xx) do not compile on the linux 3.8.0-rc3 kernel. Testing the xorg-edgers and ubuntu-x-swat versions of the NVidia driver package did not help either.

Until the NVidia driver officially supports the 3.8.0 kernel, this workaround will allow you to fix the NVidia driver using the DKMS source installation. This might also still apply for kernel 3.7.x. Begin by adding the ubuntu-x-swat repository to get at least version 304.60 of the NVidia driver:

sudo apt-add-repository ppa:ubuntu-x-swat/x-updates
sudo apt-get update
sudo apt-get install nvidia-current

Patch (by hand if you have to) the following 2 files: /usr/src/nvidia-current-304.64/ and /usr/src/nvidia-current-304.64/nv-mmap.c. These patches are based on 310.xx but can be used for manual modification:

---    2012-10-11 19:18:22.704848496 -0400
+++ 2012-10-12 20:35:55.707213868 -0400
@@ -20,6 +20,7 @@
 ISYSTEM=`$CC -print-file-name=include 2> /dev/null`
@@ -118,7 +119,7 @@
@@ -146,10 +147,10 @@
     if [ "$ARCH" = "i386" -o "$ARCH" = "x86_64" ]; then
-        CFLAGS="$CFLAGS -I$SOURCES/arch/x86/include -I$OUTPUT/arch/x86/include/generated"
+        CFLAGS="$CFLAGS -I$SOURCES/arch/x86/include -I$SOURCES/arch/x86/include/uapi -I$OUTPUT/arch/x86/include/generated -I$OUTPUT/arch/x86/include/generated/uapi"
     elif [ "$ARCH" = "arm" ]; then
         CFLAGS="$CFLAGS -I$SOURCES/arch/arm/include -I$OUTPUT/arch/arm/include/generated"
--- nv-mmap.c.dist 2012-08-08 22:52:53.000000000 -0400
+++ nv-mmap.c 2012-08-14 23:52:41.257235863 -0400
@@ -450,7 +450,7 @@
nv_vm_list_page_count(&at->page_table[i], pages);

- vma->vm_flags |= (VM_IO | VM_LOCKED | VM_RESERVED);
+ vma->vm_flags |= (VM_IO | VM_LOCKED | (VM_DONTEXPAND | VM_DONTDUMP));

Patches are taken from here After patching, the NVidia kernel module will not be built unless you install a new kernel or force an update using DKMS. To rebuild the kernel module for your currently running kernel, run this:

sudo dkms autoinstall -k $(uname -r)

Linux / Gentoo Linux

Backing up/cloning a linux system over a network

I had to perform this emergency procedure every now and then and I seem to reinvent the wheel every time so I reckoned it was time to document it.

This time, its a failing master drive in a simple home server – without a RAID. The SMART monitoring has not yet caught on but the self-tests fail and some files are read-only as writes to the drive fail. Also the logs are full with SATA errors like: Apr 29 10:23:57 rivenscryr smartd[3621]: Device: /dev/sda, 18 Currently unreadable (pending) sectors

To save the installation and all the files, I want to clone or backup the entire root filesystem (which is ext3, but that is not relevant) to a new drive (formatted ext4) in another machine. The reason to do this between machines can be numerous: perhaps you are migrating a live machine or (in this case) perhaps you worry that the current drive may fail when you power-cycle the system when the replacement drive is installed.

Backup/Clone Over Network Recipe

  1. Mount the root filesystem of the original machine on another location in order to have a clean copy:
      mkdir /mnt/rootfs && mount / /mnt/rootfs -o bind

    Check if it worked by listing the /mnt/rootfs/ folder. Note that /mnt/rootfs/proc will be empty and dev will be empty or very sparse.

  2. Use netcat and tar on the receiving machine (my case: random system with Ubuntu on USB stick and new HDD plugged in) to receive and write the data.
      nc -l 7777 | tar -xz -C /mnt/targetdir
  3. Use tar and netcat on the original machine to stream-copy the root filesystem over the network.
    Big fat warning: My cloning is done on an internal, trusted network. You will be cloning using an unsecured stream over this network.

      tar -czp --atime-preserve --numeric-owner \
        --xattrs --recursion /mnt/rootfs | nc remotehost 7777

Here is what all these options do:

  • -c: Create a new archive
  • -z: Use GZip compression to speed up the transfer
  • -p: Preserve file permissions
  • –atime-preserve: Do not update the accessed timestamp
  • –numeric-owner: Copy using numeric ids only – useful when cloning into a foreign system
  • –xattrs: Include ACL attributes and SElinux attributes
  • –recursion: Recurse down into sub-folders
  • –ignore-failed-read: Useful when a drive is heavily damaged and you only care to extract as much data as possible. Note: do not enable this by default as you might end up with an unusable clone if some files are missing. (Aka, this is a last resort option for non-functional clones)
Linux / Gentoo Linux

Clearing GPT from harddisk without destroying MBR

For some reason, my attempts to use GPT with Windows 7 ran aground and I ended up with a working Master Boot Record (MBR) and an empty GPT. While Windows 7 was quite happy with this, the Ubuntu installer detected the GPT and attempts to use it – which effectively prevents me from installing it unless I wipe the entire hard drive for Ubuntu.

In this little article I will clear the GPT from the disk so only the MBR remains. This is an advanced technique which WILL destroy your partition layout and thus your data when it fails. Make very very sure you understand each step and have backups.

Start by making a backup of your MBR, which is 512 bytes long at the start of the disk:
sudo dd if=/dev/sda of=disk_mbr.raw bs=512 count=1
I made a backup of the MBR and the GPT as well, just in case (GPT is 512 bytes for the header followed by 16kB of content, total with MBR is 17kB):
sudo dd if=/dev/sda of=disk_mbr_gpt.raw bs=1024 count=17
The GPT is present at the start and end of the disk. Lets start by erasing the GPT from the start:
sudo dd if=/dev/zero of=/dev/sda bs=512 seek=1 count=33
Now find the end of your disk, use fdisk /dev/sda and note the number of bytes on the disk. In my case it is: 240,057,409,536 bytes, since I want to use kilobytes with dd, lets convert it to kB: 234,431,064 kB. We need to erase the last 16.5 kB, but 16kB will suffice (as the header is in the last 512 bytes).

So subtracts 16 from the drive size to generate the GPT offset for the end of the disk and zero-fill it to the end: sudo dd if=/dev/zero of=/dev/sda seek=234431048 bs=1024 count=17

All done! Now fdisk will no longer complain the drive has a GPT table and the Ubuntu installer will only find the MBR and happily install using that.

Linux / Gentoo Linux

Networking for KVM with CentOS 5.5

I’ve found numerous tutorials to set up KVM with CentOS 5.5. Some refer to the Red Hat guides, which are useless in my opinion if you are not using KVM in a large server park, others come up with scripts to set up networking.

I kept searching and trying configurations until I finally found one that works. Note that among problems you might encounter is the one that plagued me for months: CentOS is attempting to bring up the virtual device for LAN access on the host before bringing up the physical interface, this fails and prevents all networking and dependencies from starting properly.

To set up the bridge devices for KVM and the host (something which is done only once to ‘wire’ your virtual network instide the bridge) look here. Most of the guide is still relevant even if they refer to the Red Hat documentation (which is about Red Hats cloud virtualisation platform and not a single server running KVM).

Use a configuration like this for proper KVM networking under CentOS, Red Hat and probably even Fedora.

1. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 to contain something like this:

# /etc/sysconfig/network-scripts/ifcfg-eth0 - the physical NIC which will not be used directly

The “NM_CONTROLLED” bit seems to originate from Fedora 12/13, CentOS might ignore it altogether so you could leave it out.

2. Edit /etc/sysconfig/network-scripts/ifcfg-br0 to restore network access for your host machine:

# /etc/sysconfig/network-scripts/ifcfg-br0 - Bridged network access for host

The crucial parts in this configuration is that eth0 is started on boot and is assigned a bridge device: ‘br0’. The ‘br0’ device is marked as a bridge device which completes the dependencies between them. So make sure if you deviate from the examples you leave the types and ‘bridge=br0’ in tact.

Linux / Gentoo Linux

Determining the UUID of a partition

From Wikipedia:

A Universally Unique Identifier (UUID) is an identifier standard used in software construction, standardized by the Open Software Foundation (OSF) as part of the Distributed Computing Environment (DCE).

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. Thus, anyone can create a UUID and use it to identify something with reasonable confidence that the identifier will never be unintentionally used by anyone for anything else. Information labeled with UUIDs can therefore be later combined into a single database without needing to resolve name conflicts.

Since CentOS (and Ubuntu and others) use the UUID of partitions to mount or perform other operations on them, I needed to find the UUID of my storage partition. The downside to using UUIDs is obvious: I know the device name (/dev/sdb1) but I have no clue what the UUID of the partition is.

The upside is that if I move the drives around, so my /dev/sdb1 becomes /dev/sda1 for example, everything will still work because the identifiers remained unchanged.

Enough talking, I found 3 ways of finding the UUID of a drive on Linux. Note that this is not all of them though, just enough for me.

Option 1: Use /dev

Since the UUID to device mapping needs to be done somewhere for the kernel and system tools, why not use that ourselves?
ls -l /dev/disk/by-uuid

Option 2: Use udev

The vol_id program pulls the information from udev and presents it all in a fairly readable form. For some reason, my Ubuntu 10.10 does not have it anymore but possibly others still do.
vol_id /dev/sdb1

Option 3: Use blkid

The option I tend to use it to use the program called blkid. Which as sole purpose (as its name suggests) is to show the block device ID of a block device *gasp*.
blkid /dev/sdb1

Linux / Gentoo Linux

Change SVN repository location

For some reason, repositories can be moved around. If you are using a working copy to develop a program or web-application or whatever, the move usually only implies a ‘check-out’ and you are set.

However, sometimes the repository is part of a production environment. To be precise, the production version is prepped in a special branch and then deployed using Subversion. Upgrading a website to a newer version only requires a ‘svn up’ and if you automate the database upgrades as well, web management becomes a breeze.

You might imagine that moving the repository in such an environment is less than desirable: each site or application which is part of that repository becomes un-upgradable and checking out everything by hand and setting it all up (plus the verification that every part is still functional) becomes a time consuming task.

However, the SVN information is all stored in plain text files. The following snippet switches the repository from to svn:// This means you can quickly switch protocol, host names or even the location within a certain host. Simply copy-paste the code below into a file in the directory holding one or more projects for that repository, change the URLs (do not forget to escape by replacing ‘/’ with ‘\/’) and run the script.

DIRS=`find . -name ".svn"`
for dir in ${DIRS}
  echo "Processing $dir"
  FILES=`find $dir`
  for file in ${FILES}
    if [ -f ${file} ]; then
      echo "Fixing '${file}'..."
      sed 's/http:\/\/\/\/' "${file}" > "${file}.tmp"
      mv "${file}.tmp" "${file}"
Linux / Gentoo Linux

Fixing Vmware mouse flicker

After upgrading a VM from Ubuntu 9.04 to Xubuntu 9.10 I noticed that the mouse cursor works fine in a certain region (say 640×480) but outside that region, it flickers between the actual mouse position and the edge of the region.

After reinstalling X a few times and upgrading the VMware tools I found the solution: it is a bug in the newer GTK versions. Since I amd running Gentoo (unstable aka bleeding edge) I am one of the lucky owners of this new bug.

The solution seems to be to instruct VMWare Player or Workstation (I use 6.5.x) to use its bundled GTK instead of the system one:

echo "VMWARE_USE_SHIPPED_GTK=yes" >> /etc/vmware/bootstrap
Linux / Gentoo Linux

KDE4: Launching screensaver from commandline or shortcut

Remember the good old days of KDE3? It was so easy to start your favorite screensaver by hand: simply invoke dcop and there we go. But since the dcop framework is no longer part of KDE4 I found myself without a hint of how to create a shortcut to trigger the screensaver.

Until now… use the following commands to lock the screen with the screensaver running or display the screensaver without locking:

qdbus org.freedesktop.ScreenSaver /ScreenSaver Lock

qdbus org.freedesktop.ScreenSaver /ScreenSaver SetActive 1

I hope this helps others as it took me too long to find this…

Linux / Gentoo Linux

Migrating from VMware Server to KVM on Ubuntu

After being let down by VMware one too many times (this time after upgrading Ubuntu to the latest beta and finding out that kernel 2.6.30+ is not supported), I decided to move on to KVM. After all, my server has full visualization support in the CPU so why not put that to good use.

I’ve started off by making sure visualization support is enabled in the BIOS. In my case it was nowhere to be found and a quick search for the MSI KA780G (MS-7551) showed that users had varying experience finding the feature but everyone seemed to agree that a BIOS upgrade should help (most of the time). A quick ‘lshw’ showed that I indeed was using the 16.0 version of the BIOS which is more than a year old.

Next, how to flash the BIOS? I do not (like the guide from MSI suggest) have a floppy drive installed in my server nor do I have Windows installed. Luckily, updating the BIOS of the KA780G is pretty easy as it is supported by the Linux programmer ‘flashrom’:

apt-get install flashrom
flashrom -r backup.bin
flashrom -w newbios.bin

Warning: the last command actually programs the new BIOS into your computer. Make sure you have the right BIOS for your motherboard and do not interrupt the flashing.

Linux / Gentoo Linux

VMware Server 2.x on Ubuntu 9.10 (Karmic Koala)

Many have taken the plunge and updated to the new Ubuntu due for release late October: Ubuntu 9.10 code-name Karmic Koala. And even on my my 64-bit system pretty much everything updated without a hitch. Sure, I had to reinstall the nVidia drivers again for the new kernel but besides that – all was well.

Until I noticed my virtual machines were down. A manual start gave the dreaded ‘vmware is not configured for your system’ message and after invoking I found myself looking at some compile errors about the ‘vmmon’ module.

As usual, the guys at VMware haven’t released a single update for their free flag ship in half a year now. Hence they have no support for newer kernels found on Ubuntu 9.10 but also for example in Gentoo Linux.

Luckily, someone has put together a fix that you can find here. Basically it is a patch set derived from a long thread on a forum somewhere but hey – it does the job.

It does make me seriously consider moving to an alternative visualization platform like Xen or KVM which seems to be part of the kernels and as such will always work after an upgrade…

Update: this did not work for me. Either the virtual machine does not start at all and the control panel is broken as well or the entire system goes down with a kernel crash after a few minutes… Guess I am migrating to KVM after all, good bye VMware, it was fun while it lasted…