Categories
Gentoo Linux

Kill all waiting backup jobs in Bacula

If you are a Bacula user, you most likely had the problem of a Director clogging up on one or more jobs getting stuck while others keep piling up. With the new directives for managing duplicate jobs, this should not happen anymore but last night I found out that 2 of my backup servers managed to dead-lock.

The resulting job queue was over 300 jobs long and restarting the Director did not seem to help. So I threw a little shell script together to use bconsole to cancel all jobs in a waiting state.

Us usual, use on your own risk, I only tested it on my servers and it worked fine…

#!/bin/bash
jobIds=`echo 'status dir running' | bconsole | fgrep 'is waiting' | awk '{print $1}'`
for i in $jobIds
do
  if [ -z `echo "$i" | grep '^[0-9]\+$'` ]
  then
    echo "Error: job ID $i is not a number!"
  else
    echo "Killing waiting Bacula job $i"
    echo "cancel jobid=$i" | bconsole
  fi
done
Categories
Linux / Gentoo Linux

Wine sound errors

After a while, the sound errors from Wine started to annoy me as they seem to indicate something is wrong while it is not. At the start of each program I see the following errors:

err:alsa:ALSA_CheckSetVolume Could not find 'PCM Playback Volume' element
ALSA lib pcm_dmix.c:935:(snd_pcm_dmix_open) The dmix plugin supports 
 only playback stream
err:alsa:ALSA_CheckSetVolume Could not find 'PCM Playback Volume' element
ALSA lib seq_hw.c:457:(snd_seq_hw_open) open /dev/snd/seq failed: No such 
 file or directory

The first errors are related to my Intel HD audio card which has multiple PCM channels. The result is that none of them are called ‘PCM Playback Volume’ and Wine seems to be hard-coded to search for that specific channel to check upon starting.

Harmless but annoying none the less. A bit of searching seems to indicate that this is something I can’t fix.

The second error about dmix seems to mean that the dmix soundcard that Wine is using does not support recording sound. This is correct as it is a softmix device used to let multiple programs ‘claim’ the soundcard without locking each other out. Beats the hell out of the mess called PulseAudio and was a breeze to set up.

The 3rd error is MIDI related: Wine is trying to access the sequencer device. I can’t figure out why Wine has no option to disable midi support altogether but without that device you will keep getting those errors.

To fix this, enable the MIDI drivers in ALSA and load them on boot, the driver will be called ‘snd-seq-device’. In my case this was found in (kernel 2.6.30):

Device Drivers
<*> Sound card support ---->
    <*> Advanced Linux Sound Architecture --->
        <*> Sequencer support
Categories
Linux / Gentoo Linux

Restoring Exchange 2003/2008 using Bacula

Because the instructions in the Bacula documentation left me hanging on how to actually restore the Exchange data from a backup, I am writing this little summary after extracting all information from the mailing lists.

In my case, I have one server called ‘axmail-fd’ running Exchange 2003 and another server called ‘axemail-fd’ running Exchange 2003 SP2. The following steps are needed to restore into the Recovery Storage Group on the new server in order to migrate the mail from the old to the new server.

To prepare, please create the Recovery Storage Group on the target Exchange server. Create the database name you want to recover, for example “Mailbox Store (AXMAIL)”, which will generate a “Mailbox Store (AXMAIL).edb” in the Recovery Storage Group folder. Clear out any log files or other remnants of previous restores as Exchange tends to get confused if data from multiple databases are in there.

Start bconsole, select the restore mode and select the Exchange backup to restore:

*restore
First you select one or more JobIds that contain files to be restored. 
You will be presented several methods of specifying the JobIds. Then 
you will be allowed to select which files from those JobIds are to be restored.
To select the JobIds, you have the following choices:     
1: List last 20 Jobs run                             
2: List Jobs where a given File is saved             
3: Enter list of comma separated JobIds to select    
4: Enter SQL list command                            
5: Select the most recent backup for a client        
6: Select backup for a client before a specified time     
7: Enter a list of files to restore                       
8: Enter a list of files to restore before a specified time     
9: Find the JobIds of the most recent backup for a client      
10: Find the JobIds for a backup for a client before a specified time    
11: Enter a list of directories to restore for found JobIds              
12: Cancel                                                           
Select item:  (1-12): 5                                                 
 Defined Clients:                                                              
...     
4: axmail-fd     
...    
10: axemail-fd
Select the Client (1-10): 4

The defined FileSet resources are:
1: AXMAIL Full Data Set
2: Exchange
Select FileSet resource (1-2): 2
+-------+-------+----------+---------------+---------------------+-------------------------------+
| JobId | Level | JobFiles | JobBytes      | StartTime           | VolumeName                    |
+-------+-------+----------+---------------+---------------------+-------------------------------+
|    90 | F     |       13 | 5,313,968,371 | 2009-06-24 15:36:10 | Deventer_Exchange_Backup_0013 |
|    90 | F     |       13 | 5,313,968,371 | 2009-06-24 15:36:10 | Deventer_Exchange_Backup_0014 |
|    91 | I     |        5 |     2,671,174 | 2009-06-24 17:28:25 | Deventer_Exchange_Backup_0014 |
|    92 | I     |        5 |       233,882 | 2009-06-24 18:00:01 | Deventer_Exchange_Backup_0014 |
|   118 | I     |       17 |    40,099,025 | 2009-06-25 18:00:02 | Deventer_Exchange_Backup_0014 |
+-------+-------+----------+---------------+---------------------+-------------------------------+

You have selected the following JobIds: 90,91,92,118
Building directory tree for JobId(s) 90,91,92,118 ...
24 files inserted into the tree.

Now we want to select the entire First Storage Group to restore, except for the Public Folders store.

Note: If you have the mailbox store defined, perhaps the restoration will work. An old mailing list conversation in 2008 stated that it was only possible to restore one database at a time – hence we now unmark the Public Folder.

You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.                           
Enter "done" to leave this mode.                                          
cwd is: /$ mark *                                       
29 files marked.                               
$ cd "@EXCHANGE/Microsoft Information Store/First Storage Group"
cwd is: /@EXCHANGE/Microsoft Information Store/First Storage Group/
$ unmark Public*                                                   
4 files unmarked.                                                  
$ lsmark                                                           
*C:\Program Files\Exchsrvr\mdbdata\E0002FC5.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FC6.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FC7.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FC8.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FC9.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FCA.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FCB.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FCC.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FCD.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FCE.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FCF.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FD0.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FD1.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FD2.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FD3.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FD4.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FD5.log                    
*C:\Program Files\Exchsrvr\mdbdata\E0002FD6.log                    
*Mailbox Store (AXMAIL)/                                           
*C:\Program Files\Exchsrvr\mdbdata\priv1.edb                       
*C:\Program Files\Exchsrvr\mdbdata\priv1.stm                       
*DatabaseBackupInfo                                                
$ done

Bootstrap records written to /var/bacula/axnet-dir.restore.23.bsr  
The job will require the following   
Volume(s)                 Storage(s)                SD Device(s)
===========================================================================
Deventer_Exchange_Backup_ File                      FileStorage

25 files selected to be restored.

Run Restore job
JobName:         RestoreFiles
Bootstrap:       /var/bacula/axnet-dir.restore.23.bsr
Where:           /tmp/bacula-restores
Replace:         always
FileSet:         Empty FileSet
Backup Client:   axmail-fd
Restore Client:  axmail-fd
Storage:         File
When:            2009-06-26 00:44:50
Catalog:         MyCatalog
Priority:        10
Plugin Options:  *None*

We now need to change the target to the new server and clear out the Where setting:

OK to run? (yes/mod/no): m
Parameters to modify:    
1: Level    
2: Storage    
3: Job    
4: FileSet    
5: Restore Client    
6: When    
7: Priority    
8: Bootstrap    
9: Where   
10: File Relocation   
11: Replace   
12: JobId   
13: Plugin Options
Select parameter to modify (1-13): 5                

The defined Client resources are:    
1: bartje-fd    
2: nakor-fd    
3: hermione-fd
4: snape-fd
5: hagrid-fd
6: axnet-fd
7: axweb-fd
8: axmail-fd
9: axexact-fd
10: axklant-fd
11: axemail-fd
Select Client (File daemon) resource (1-11): 11

Run Restore job
JobName:         RestoreFiles
Bootstrap:       /var/bacula/axnet-dir.restore.23.bsr
Where:           /tmp/bacula-restores
Replace:         always
FileSet:         Empty FileSet
Backup Client:   axmail-fd
Restore Client:  axemail-fd
Storage:         File
When:            2009-06-26 00:44:50
Catalog:         MyCatalog
Priority:        10
Plugin Options:  *None*
OK to run? (yes/mod/no): m

Parameters to modify:    
1: Level    
2: Storage    
3: Job    
4: FileSet    
5: Restore Client    
6: When    
7: Priority    
8: Bootstrap    
9: Where   
10: File Relocation   
11: Replace   
12: JobId   
13: Plugin Options
Select parameter to modify (1-13): 9
Please enter path prefix for restore (/ for none): /

Run Restore job
JobName:         RestoreFiles
Bootstrap:       /var/bacula/axnet-dir.restore.23.bsr
Where:
Replace:         always
FileSet:         Empty FileSet
Backup Client:   axmail-fd
Restore Client:  axemail-fd
Storage:         File
When:            2009-06-26 00:44:50
Catalog:         MyCatalog
Priority:        10
Plugin Options:  *None*
OK to run? (yes/mod/no): y

Job queued. JobId=132
You have messages.

Notice how we did not use any renaming on the database paths: if Exchange 2003 has a Recovery Storage Group defined, that group will receive any restores, making manual adjustment of the paths unneeded.

Troubleshooting

Error 0x7fe1f42

If you get this error: “Error: HrESERestoreAddDatabase failed with error 0xc7fe1f42 – Database not found. Check that the Database you are trying to restore actually exists in the Storage Group you are restoring to”. You have not created the database in the Exchange manager in the Recovery Storage Group. Make sure you created the database in the manager and check the name.

Error 0xc7ff1004

I ran into this message after figuring out how to restore the data: “Error: HrESERestoreComplete failed with error 0xc7ff1004 – Unknown error”. The error is given by the FD after the data has been restored and the FD crashed after that.

You can run the eseutil.exe against the .edb file to check the state (use eseutil /mh filename.edb) and you will probably see the state as “Dirty Shutdown”.

The cause is a problem during the backup and is making the restore fail. If you have all the log files (E00xxxx.log in ‘restore’ in the Recovery Storage Group folder) you can use the eseutil /cc restore command to replay the log files and fix the database.

After replaying the logs, the database should mount fine and all the mail should be there.

Categories
Linux / Gentoo Linux

Growing LVM storage

After adding some new storage to the backup system at work, I needed a way to transfer the roughly 400GB of backups to the new installation – without the use of a USB disk or another server (would take way too long). So after installing CentOS 5.3 with LVM2 enabled, I dedicated the better part of 1TB to the new backup storage on the new hard drive.

After mounting the old storage, I copied the old archives to the new LVM storage and did the following.

Prepping partition for LVM

Use fdisk to change the partition type to ‘Linux LVM’, hex code ‘8e’: fdisk /dev/sdb

Then create the structure needed for use with LVM on the new partition: pvcreate /dev/sdb1

Finally, add the new LVM partition to the LVM caches by letting the automated scan picking it up: vgscan

Now to actually append the new partition to the storage pool: vgextend VolGroup00 /dev/sdb1

Adding to LVM

If you run vgdisplay, you should now see the space you just added as free space in the LVM storage pool.

You can now add the free space to one or more storage volumes that need it: lvextend -L +353.5G /dev/VolGroup00/LogVolBackups

Finally, let the filesystem on the drive resize to fill the new space: resize2fs /dev/VolGroup00/LogVolBackups

And there we go, 353.5GiB added to the LVM2 storage. Now next time I won’t think twice to use LVM2.

Categories
General blog entries

HTC Magic and Android 1.5: epic fail…

After following the news around Google Android 1.0, I decided to wait for the next generation of phones so the small problems would be ironed out. Besides, the HTC Dream or G1 had some small disadvantages – for example the short battery life and the ugly design.

After the release of Cupcake (Google Android 1.5), HTC quickly announced the release date for their 2nd generation Android phone: the HTC Magic which would be sold by Vodafone in June 2009 (in the Netherlands). I was pleasantly surprised that I got an email from a webshop telling me they are now selling the HTC Magic, even without a subscription!

After toiling about it for 2 days I decided to give in and order my first Android phone. The next day it arrived and after taking care to open the box without damaging anything (so it could be returned if needed) I ran into the first bump on the road: it is locked to Vodafone.

A quick call to the shop I ordered it from didn’t help either: “…but it is obvious that it is locked to Vodafone, the brand is stamped on it!”. Apparently, making it possible to order without a subscription is not a hint that it can be used with any sim card. But I even called them before placing an order and the bloke that ‘helped’ me did not seem to find it worth mentioning that if I asked if it was sim locked (which it wasn’t) I probably would not want to know that it was provider locked instead…

After borrowing a Vodafone sim card I finally managed to start the phone and the first thing I tried was testing if the Bluetooth voice-dial and file transfer were fixed. File transfer did not work but I can live with that.

Next came the voice dial: I paired the phone to my car kit and pressed the button. The car kit showed “No voicedial” and broke the connection. So my new shiny, 400 euro costing phone that can play music over Bluetooth with a press of a button can’t voice-dial anybody…

After searching on the internet, it became obvious that the current voice dialer in Android 1.5 is cripple (it requires interaction with the screen) making actual voice dialing (while in a car for example) impossible. So Vodafone just stripped it out completely…

Needless to say, I called the shop where I bought it, explained why I did not want the phone any more and sent it back the next day by mail.

Shame on you Google for not even supporting basic Bluetooth features in the ‘big’ update for Android and shame on you Vodafone/HTC for not clearly pointing out that the would-be ‘iPhone killer’ is in fact crippled…

Categories
Programming

Adding Gravatar to Joomla!

After stumbling upon Gravatar, a site serving your avatar globally, I looked for an easy way to add this functionality to Joomla!. After ditching Gravatar/Avatar for needing some other plugin to work in the first place, I installed gAvatar.

This plugin worked without dependencies and gAvatar is dead simple to set up. Just enable the plugin, change the rendering mode and you are done.

The only thing that can go wrong (and did for me) was the fact that you can choose for a floating avatar (which ended up floating in the text), a table avatar (using the default Joomla header style – which this site doesn’t use) or advanced templating.

That last option seemed like a good solution except that nowhere was described what it did or how it worked. A quick dive in the source showed me everything I needed to know: gAvatar adds a ‘avatar’ parameter to the article object. To use it, find the classes generating the article layouts and modify those.

To find those class files, I used the following in the template folder:

fgrep "article->text" * -R

And add the following (for example) to insert the avatar:

<?php if (isset($this->article->avatar)) : ?>     <div style="float: right"><?php echo $this->article->avatar; ?></div> <?php endif; ?>
Categories
How-To's

Debugging upload problems on JoomGallery

After installing JoomGallery and writing a migration script to migrate from Menalto Gallery2 (G2) , I finally got to uploading photos. But for some mysterious reason, every attempt to upload images would fail.

The manual upload did not work (blank page), the FTP upload did not work (blank page), the batch upload did not work (blank page or session timeout sending me back to the login) and the Java upload did not work either: 

wjhk.jupload2.exception.JUploadExceptionUploadFailed: wjhk.jupload2.policies.PictureUploadPolicy.checkUploadSuccess(): The string "JOOMGALLERYUPLOADSUCCESS" was not found in the response body

Categories
Linux / Gentoo Linux

Fixing XML-RPC’s in KDE4

After posting that Bilbo worked out of the box, I was bound to run into a world of trouble – and so I did

You see, when I was trying out KBlogger, I ran into the “Invalid request payload xmlrpc element STRING cannot be child of DATA” error you find in some KBlogger related posts. After playing around a little I realised that the error prevented me from posting anything and I ditched KBlogger as being buggy as hell (which it actually still is).

But when I installed Bilbo, I ran into the same problem! Only this time I knew that it should work as I posted before using Bilbo. After investigating a bit I found the culprit: KBlog. This backend library handles the communication with the server over the XML-RPC protocol. The error is produced on the server side and to be exact it is PHP XML-RPC that generates the error.

Because the XML-RPC library on the server is very strict (and probably error free), I turned to see what Bilbo actually sent to the server. The problem appears to be in the category structure that is generated. In my first posting, I did not select a category to post to. But when you select one or more categories, a QStringList object is populated with the category names. This is converted into XML somewhere in KDE (I haven’t figured out where yet).

After talking to the Bilbo devs for a bit they pointed me to the WordPress API in KBlog. For some reason, that one generates a string holding the XML for the server. And in that class, the metaWeblog.newPost function is called using a properly constructed category array.

Alas, switching to the WordPress API did not help: “STRUCT cannot be child of PARAM” and whether I used categories or not, the error remained. A quick inspection showed where that error comes from: the ‘struct’ element should not be directly put inside a ‘param’ element. The XML-RPC specifications state that the ‘struct’ should be placed in a ‘value’ element first. After fixing this and recompiling Bilbo, everything is working now!

Well, not everything as the metaWeblog API is still broken. I have filed a bug report with KDE upstream so they can add my fixes to the WordPress API and hopefully someone with more KDE programming experience can fix the metaWeblog API. Or rather, fix the conversion from a QStringList to proper XML-RPC compatible XML.

On a side note: while debugging the XML-RPC protocol I discovered that the Joomler XML-RPC plugin for Joomla! did not apply the categories that are sent along. I’ve modified my (already heavely patched) version of Joomler and once I’ve tested it a bit more, I’ll send the patches to the Joomler devs. Hooray for open source I guess ^^.

Categories
General blog entries

Bilbo, the new blogging program for KDE4

The experimental ebuild for Gentoo worked like a charm: I am typing this on Bilbo. The editor seems to Just Work ™, the previews are functioning (both rich text and the website preview) and the account creation worked out of the box.

The only thing that took some time to figure out is the fact that the auto-config option did not configure my site. In fact it started looking for a xmlrpc.php file to talk to, which results in a 404 as Joomla! does not have a file but rather a folder with that name. So after setting it to the correct location and switching to the metaWeblog API in the Advanced settings I hit ‘Fetch blog ID’ and voila, we are up and running.

Well, kind of: the plugin I use to provide the blog API on the site is incomplete. I have been tweaking it for months now, adding missing functions as I went. So when I wanted to select a category, I got an error stating that the XMLRPC function was not found.

After enabling the debug info on my server, I saw that Bilbo tried to access metaWeblog.getCategories – which is unimplemented in the original version of the plugin. So after adding the missing function, everything came to live and here we are…

Now if you excuse me, I have a virtual machine to dispose of as I can retire Windows Live Writer now ^^

Categories
Linux / Gentoo Linux

Bilbo: Finally a decent blogging program for KDE?

Bilbo interface While the development on KBlogger has somewhat picked up and some features have been pushed back to allow for a semi-working release, I just found a little gem hidden on the big wide web.

In a mailing list post on the development on KDE, someone is suggesting to take a look at Bilbo instead of KBlogger for the ‘extra-gear’ package. Of course, I was expecting a simple blog editor similar to all the other projects out there that never quite made it. Boy, was I in for a surprise!

Bilbo has the features I need in a blogging program (similar to the Windows Live Writer interface) and even better, it should be in working order!

I will report back once I get it to install on KDE 4.3 using Gentoo.