Tag Archives: Storage & Backup

Understanding (some) Firebird’s nbackup error messages (“Error (xxx) opening database file”)

Leave a reply

Because on the server where the application is running I’m unable to do backup using regular gbak tool, I turned my attention to nbackup. In fact I’m doing just file copy and using alter database begin backup/alter database end backup (more info). But doing the backup is only part of the story. You have also know you know how to restore it and whether you can restore it (aka whether the backup is not corrupted).

If you have done backup using process described above and you have the file, you can’t just start using it. You need to “restore it”, which means changing a flag inside the database file. This is where the nbackup‘s -F switch comes to play. But no matter what I was doing, I was getting: 

Failure: Error (5) opening database file: <some>.fdb

This error message is, well, completely utterly useless. While waiting for reply from firebird-support list I played with Process Monitor (of course) to see what’s wrong. But I haven’t seen any disk activity with errors. Changing paths, trying invalid paths (this produced just error 3), running 32bit version of nbackup, using one from Firebird 2.1, … And nothing. Then I decided to have a look into the ultimate documentation aka sources. It was not difficult to find piece of code:

b_error::raise(uSvc, "Error (%d) opening database file: %s", GetLastError(), dbname.c_str());

Great GetLastError. Time to jump into System Error Codes. And in no time I know it’s ERROR_ACCESS_DENIED. That’s a progress. Quick check of permissions and yes; the user I was running under had no permission to write to the file. Quickly changing that and everything was working fine.

Hope it helps somebody.

AWS Simple Storage Service (S3) pain

Leave a reply

When we worked together with Aleš Roubíček using Windows Azure cloud, we faced quite a few surprises with applications in cloud. Especially when the application isn’t small and is using a lot of “cloud” services. Sometimes we used hashtag on Twitter for it: #cloudlife.

Last week I was creating fairly simple tool that allows (or should allow) me to copy some files to S3 bucket. I had some special needs because the folder where the files were was heavily changed and the tool needed to handle that. Because AWS has SDK for .NET I was not expecting that to be extremely difficult. How wrong I was.

Every file in S3 has ETag associated. And you will get it with almost every request, i.e. when you’re listing bucket. Good candidate for decision whether to copy the file (because it was changed) or not. The SDK contains AmazonS3Util class which allows you to compute the checksum/ETag. Great, should be piece of cake. Not so fast. The SDK and S3 itself has some gotchas – #cloudlife.

First the ETag returned from SDK is inside double quotes. But the value from AmazonS3Util is not. I can fix that. Could be worse. Let’s move forward. For big or huge files the new multipart upload (you are uploading the file in smaller chunks) is recommended. Boom. Checksum/ETag is then different. For same file uploaded “normal” way and multipart the ETag differs. Sadly the AmazonS3Util can compute only the checksum for “normal” type of upload. Never mind. Next. Every file can have some metadata associated, I can store my own checksum there. I will need to request metadata in separate call, but I can live with that. So let’s use WithMetadata method to add some key/value. Good no problem so far. Retrieving metadata. Suspicious method GetObjectMetadata (and the BeginGetObjectMetadata/EndGetObjectMetadata counterparts) returns NameValueCollection, I can probably find the same key as I stored there. Not so fast. My key originally stored as i.e. foobar is now x-amz-meta-foobar. Maybe the WithMetadata could enforce that in input, so I know there’s always this prefix. I don’t like hidden magic. Almost there. Every request is also signed (that’s, also, why we have the key and secret) to be sure the request was not changed on the way etc. (headers, parameters etc. are part or the input for signature). SDK handles the signature for me. Badly. As long as you use US-ASCII characters in filenames/paths or better to say keys for storing the data, you’re fine. Try to put there some “special” characters, like in my case diacritics. Suddenly the signature doesn’t match what’s expected and server will not allow me proceed further. Yes you’re right, the encoding issue. Sadly I don’t know currently workaround. Hope it’ll be fixed soon. Hitting wall every while.

And you know what? Some limitations and challenges I like (ETag issue. These are reasons why I like development and creating software. Some I don’t like (encoding issue), I shows somebody wasn’t doing own work properly. And why #cloudlife? From marketing we’re told the cloud is so cool and so … best, it’ll solve all our problems and we expect it to be perfect. But it’s not. It’s piece of architecture as every other component in your solution. You should expect issues.

Have fun, develop for fun.

Off-site initial upload for Synology Amazon S3 backup

Leave a reply

I recently started thinking about backing up also “less” [1] important data to Amazon S3 from my Synology NAS, which is my primary backup location. The problem was, that the amount was about 150GB. Far more than I can upload through my home connection in reasonable time. It would take weeks. When I first started using S3 backup on my NAS, I checked, what’s exactly being copied to the bucket, in case disaster happens and I’ll be force to restore without any Synology box around (or at least restore critical data sooner than I’ll have it available).

Luckily it’s almost the same structure as on disk. There’s a @tmp folder added and whole backup is in particularly named folder, but the name is same for all backups from the NAS. If you checked also Enable metadata backup, there’s one file per folder, with metadata in it (guessing, I haven’t tried to reverse it), but I’m not using this option.

I wanted to upload the initial data from my office, where the connection is better and, because the data isn’t changed too often, just backup the differences directly from box. With the above knowledge I was pretty confident, the off-site upload is going to work. :) And it did. After I uploaded the data, I triggered the backup on box and after a while it was done. Like a glove.

And remember, backup is one part. Restore is the key other!

[1] No matter what you think, every data is important. You’ll find out, when you will loose the least important data (you thought it’s least important). But everybody needs to learn that the hard way, before really believing.

Synology’s support is exceptional

Leave a reply

I’m not writing often about services I use etc. But from time to time I’m extremely pleased with some and that’s worth mentioning. This time it’s Synology. I’m using Synology NAS boxes for a many years and in my opinion these are the best. But that’s not what I like to talk about.

Recently one box started behaving very slowly. Both internal processes, Windows share access and web interface. I did couple of usual checks, and checked health of disks. Nothing. At that time also new firmware came out so I decided it will not hurt to update it, maybe it will solve the problem. The download and installation was very slow, of course and I actually needed more attempts to install it. But no specific error, except second to last try when it was showing me problem with free space on system partition. That brought me to suspicion that the slow behavior is related to it. Couple of quick attempts with this and that, but no luck. Before I go to SSH and look and it from inside, I decided to contact support. Maybe they have so tried-and-true solution. The responses were reasonably fast and I end up sending debug dump of system as asked. Synology investigated the dump and it turned out one of the disks in RAID 1 was dying. Sure when I was doing the initial checks, the S.M.A.R.T. was without errors (who knows why). When I ran it after getting the reply, errors were there. Bad timing.

Synology not only – for free – analyzed the dump, but also provided me with description of what disk is failing, serial number etc. They could tell me something like “Your disk is wrong.” and I would not believe it, because I was focused on the system and system partition. Maybe later I would found the errors myself when again checking everything that could go wrong. But that might be too late (well probably not with RAID 1, but why running on thin ice).

With that being said. Great service Synology!

Solved Samsung i900 Omnia’s freezing

Leave a reply

I had problem with my Samsung i900 Omnia phone being stuck in stand by mode for no obvious reason, except I discovered applications doing this. I was living with that some time until I finally jumped into the problem.

I was, from old ages, used to install almost all applications to storage card. Only some critical ones (like today screen items) into internal memory. I did the same with Omnia, although it has another storage already build in. The symptoms were simple. I ran application, put phone into stand by mode and (sometimes) later I wasn’t able to wake it up. Pretty annoying. I spotted these “problematical” applications and get used to close these after the work.

But few days ago I was sick of this. Mainly because you have to later wait when you’re starting the application again. So I did some observations and thinking. After some false (and tedious) attempts I inferred the problem might be Omnia’s handling of storage card. And yes, it was. After I move all applications from storage card to internal storage, all problems disappeared. Well, at least the phone isn’t freezing. Sometimes it takes 2-5 seconds to wake it up (light up the display), but that’s minor for me (probably the phone is doing something with the storage).

So if you’re experiencing same problem, install/move all (or the “problematic” ones) applications from storage card to internal memory of internal storage.

Note: If you’re using internal storage as USB drive and you connect phone to the computer some applications may stuck or behave weird as during this time the storage is not accessible to phone itself.


My terminology: :)
internal memory = couple of MBs where system is
internal storage = x GBs where you can put your files
storage card = SD card you can put into your phone

Second feelings about the new SSD

2 Replies

Let’s call it a day. I’m now running my new Intel X25-M SSD for a little bit more than a day, but more importantly one working day. Shortly, I’m pleased.

I don’t know, maybe my expectations yesterday were too high or maybe I wasn’t just observing (or had no time) the useful stuff. And as correctly this morning my friend Petr Kaleta pointed, I’m not only comparing the disk itself, but also the rest of the system (Dell Latitude D620), whether it’s able to keep up with the disk.

Anyway new observation from today is seeking. Simply forgot about it. First you not hear it – no moving heads, no sound. So somewhere in background you feel something is different. And other fact is that the “seeking” is so fast, you’ll waste other time checking email etc. My Opera, right now with 30 tabs, starts with same speed as empty Chromium was doing on old disk. And it’s not only about the start, when you see the application it’s ready to work with. The post-start processes are already done. Same with Outlook. It behaves really instantly when switching folders, searching, checking calendar etc.

Other piece I noticed is working with virtual machines. You know, I run all my development environment in VMs, hence I stress it a lot. The start up is roughly two times faster as I wrote yesterday. The suspend is even faster. But when you suspend one and start another it’s significant. Normally I was waiting a long time, the disk was overloaded. Now, it doesn’t matter. Same speed.

Another chapter is working inside the VM. The applications there are working faster, thanks to the good seeking behavior. The Visual Studio 2010 RC there starts in under 10 seconds (loading solution with one DAL and BL library and one console application). The first compilation is about four times faster, subsequent ones are not such a big improvement as good piece of work is CPU bound. I like this improvement a lot.

The battery life seems to be about 50% better. It depends very on what you’re doing (and in what shape your battery is), and I even don’t remember exactly how quickly was my battery discharging before. So it’s based on my estimate how many tasks I was able to solve before need to recharge. :) Bear with me.

Last stuff I was carefully observing was installation of Opera 10.50, which came out today. After 15-20 seconds the installation was done. The progress bar flew couple times from left to right. I put this down again to great seeking times (the installation pack is around 10MB, no huge data).

The rest of work is more pleasant. It’s not lightning fast (as I expected for the first time – I know it’s not RAM, beat me :) ), but your not waiting for the disk too much. And maybe it’s just me, but I feel more relaxed when the machine waits for me not vice versa. :D

If you have any questions or you want me to test something, let me know, I’ll try my best. Oh, for true geeks, I’m running SSDSA2M160G2GC (model code) or SSDSA2MH160G2XX (product code) Intel X25-M 34nm SATA SSD.