Vlad Ioan Topan

My playground

Archive for the ‘system’ Category

Switching to Linux (1)

leave a comment »

Since my first Linux “experience” (which happened some eight years ago during a CS lab at UTCN), every couple of years I spend somewhere between a few days to several weeks trying to switch over to “the other side”. The reasons behind these (as of yet futile) attempts revolve mostly around the concept of “freedom”, and are beyond the scope of this writing. What I find of greater interest is the evolution of Linux-based OSes, and in particular of their target audience, which shifts more and more from hardcore enthusiasts willing to spend countless hours setting up a new machine toward average computer users (even the ones of the “point-and-clicky” variety).

The “sparse” look

My first Linux (Mandriva) was text-only; X-Window was in an “almost-working” state (crashing often, and even more often not being able to start at all) on most Linux machines I touched back then. Some time later most distributions had GUIs, but all the relevant work was still done beyond the scenes by console programs, which is still the case today. The myriad of “flavors” (and window managers) has made it practically impossible to write even remotely portable GUI interfaces for Linux, so graphic interfaces get “strapped” (pardon the low-brow hint; it truly feels like le mot juste) on console programs.

And that’s precisely how most Linux GUIs look and feel like: painful. Most of the window space is simply wasted: text is larger than necessary (for most people) and controls are separated by vast amounts of empty space, giving the interfaces a very “sparse” look. And deeper on the causality chain of problems is the fact that there simply isn’t that much exposed functionality in most Linux GUIs. Although to a much lesser extent than it was the case years ago, you still have to drill down way beyond the graphical interface in order to accomplish most non-trivial tasks. And then there’s responsiveness. After being spoiled by native graphical interfaces (some optimized to the point of writing machine assembly-level code) with excellent responsiveness (as it is more often than not the case on Windows), the sensible lag experienced on most interactions with Linux GUIs tends to annoy me in a very subtle manner. Then there’s Java and Java-based GUIs, which make Linux GUIs feel lightning fast, but I won’t go there.

Along came Ubuntu

My most recent attempt at Linux started a couple of months ago, but was interrupted by even-more-work-than-usual at the job, and would have been completely forgotten and abandoned if not for a Linux-vs-Windows themed conversation with my coworkers. I complained about most of the things that annoyed me about Linux (no decent music player is near the top of my list), and after getting many answers along the main theme of “software X has come a long way since then“, I decided to actually give it (yet) another chance.

My previous experience with Ubuntu (8.04 I think) had been almost pleasant, by far more so than any other previously-tried flavor (most notable being Mandriva back when it was called Mandrake and Red Hat at home and CentOS at work), so Ubuntu 10.10 felt like the way to go. After some research regarding “the most popular Linux”, Linux Mint popped up as a tempting Ubuntu/Debian-based alternative. The sheer volume of documentation / user-assistance available for the vanilla Ubuntu convinced me to stick with it, and so far it has been the right decision: as I’ve become accustomed when setting up a Linux OS, I’ve had problems requiring “workarounds” from the first day.

The good

In spite of the minor technical “misadventures” during setup, the Ubuntu 10.10 GUI finally feels mature (and almost responsive *enough*). The themes look good, the fonts are readable even at smaller sizes etc. And then there’s the repositories: thanks to my recently-acquired 100Mbit Internet connection, in a few hours after the installation I was already playing a pretty good-looking FPS (Assault Cube) and enjoying it. I’m not much of a gamer, but on one hand I was curious how far free games have come along, and on the other I had a lot of blood-spill-requiring-frustration left over from working out what should have been minor kinks and turned into major research themes.

I actually managed to set up both my PPP Internet connection and a VPN to my workplace without much hassle, which is a notable first. The VPN actually works better than on Windows because I have a convenient checkbox option to only route traffic going towards the VPN server’s network through it (as opposed to manually deleting the route through the VPN server on Windows, because some clever bloke figured I *must* want *all* my Internet traffic to be routed through a gateway which only knows private addresses).

The bad

The NTFS driver (ntfs-3g). It’s not bad per se, in fact it also has “come a long way” and when it works, it works fine. But in one instance it *chose* not to work for me, which I found very frustrating and annoying. My problem (and it seems to be a rather common one) is that on a recently-acquired USB hard-disk Windows appears to have messed up either the partition table or the NTFS filesystem; the problem is that it only appears that way to the ntfs-3g driver. Which is not to say that it’s wrong (from what I could gather, the size of the filesystem is set to a larger value than there actually is room on the disk, the difference being of a few sectors = a few KB); it’s just that Windows doesn’t seem to mind and reads from/writes to the disk without problems. I imagine that if I were to write to those last few KB on the disk the data would be lost, but at least I can access the data on the disk, which ntfs-3g won’t allow, because it wouldn’t mount the disk even in read-only mode. Adding insult to injury, the “Tuxera CTO” (an otherwise friendly and helpful person) suggests (here) that the only solution to ignore the warning is to “change the sourcecode”. Booting back into Windows, backing up the data and reformatting the drive to a smaller size fixed the problem, but it shouldn’t have been necessary, and the “I know what’s right for you better than you ’cause I’m the pro” attitude was somewhat disappointing.

Another problem is the lack of a decent file manager. After using all the “commanders” (Norton Commander, then the amazing Dos Navigator and nowadays Total Commander), I’m used to having a software which can handle all file-related operations (and I do a lot of them for my day job) easily and efficiently. TC, which I wholeheartedly recommend on Windows, handles everything just fine. On Linux, so far I haven’t even been able to find a (GUI) file manager with an actual “brief” view mode; all of them insist on giving me a long line of information about each file, whether I actually need it or not, and waste about two thirds of the available screen space in the process. All the features offered by TC, not to mention the plethora of plugins available for it, are still far, far way. And since we’ve hit the sensitive point of software equivalents for Linux, here’s what I’ve managed to find so far.

Software alternatives for Linux

File manager

As mentioned above, I’m profoundly dissatisfied with what I’ve found so far. MC is a must, but lacks many features. Double Commander seems be the best contender, and is built to be similar to TC (going as far as plugin interchangeability, if only there were any ELF plugins for TC…), which is a plus.

Music player

After finding a decent music player (i.e. one which is stable and has a compact interface like WinAMP and the other *AMPs on Windows) has been a seemingly impossible feat for years, along came Audacious, and all became well.

Image viewer

If good file managers and music players are hard to come by in Linux, image viewers are far more challenging. Neither one seems to grasp the basic concept of viewing a folder of images; all of them insist on “organizing my collection of photos” (’cause it’s trendy to index collections of stuff), and offer either very cumbersome methods of simply browsing image folders, or simply no way at all (except for, of course, indexing/organising the folders into a collection). The excellent XnView image viewer for Windows has a multi-platform version aptly-called XnView MP, with the downside that development is favored for the Windows version and Linux builds don’t come for each version.


I’m still looking into options for a development IDE (for C and Python in particular), with no luck as of yet.

As far as web browsing is concerned, all the relevant alternatives for Windows (Opera, Firefox and Chrome) are present on Linux, and from the order of the above enumeration my browser of choice should be obvious enough.

For an office suite I use OpenOffice on Windows, which is also available on most platforms.

I strongly recommend Guake as a terminal and Gnome Do as a generic application/document opening method.

[To be continued]

Written by vtopan

November 11, 2010 at 2:56 AM

Recovering data from a dead Windows (NTFS) disk using Linux

with one comment

At some point in your IT-enthusiast life you must’ve had at least one dead HDD, off of which Windows wouldn’t boot anymore. Up until a while ago, particularly if the partitions were formatted with NTFS, the situation was pretty much hopeless. Nowadays, with very-much-improved NTFS support under Linux (and rather tolerant to faults compared to its native counterpart under Windows), it isn’t always so. If the HDD is in a “coma” (i.e. almost dead, but not still “sort of” kicking), booting off a Linux live CD might still help recover (some of) the data. Basic steps:

  1. Get a Linux live CD distribution which has good built in NTFS support (most of them have basic support by now) and ddrescue
  2. Boot off the live CD and use ddrescue to get a binary image of each partition or mount the partition(s) and copy the files to a safe place
  3. [If using the dd(rescue) approach] mount the images as drives under Windows and copy the files or be brave and mount the partition in a VM and try to actually boot it, at least as far as a command prompt (safe mode) or use a backup/partitioning tool to write the images to another disk

If you’re not paranoid about security (by nature or by job description), i.e. you don’t use EFS for your most sensitive data, you’re pretty much off the hook. If you’ve made the punishable-by-huge-amounts-of-pain mistake of using EFS and your disk crashed, as is my case, hope is as dimmed as the foresight of the folks who designed NTFS and used more than the actual user password to encrypt the data. As it turns out, to decrypt the files you need a certificate which can only be generated on the machine which encrypted the files, which is

Linux live CDs with NTFS support

I’ve tried SystemRescueCd, Trinity Rescue Kit, RIP Linux and plain vanilla Knoppix, and Trinity Rescue Kit appears to be the best: it has ntfstools / Linux-NTFS installed, and it didn’t hang on boot because of the failing HDD (other distros did). As a sidenote, I haven’t managed to boot the GUI (X) of any of the distros, as my laptop monitor/graphics card seems to be uncooperative with the standard drivers/VESA mode, but apart from the visual partition manager, everything works fine from the console anyway.

When choosing a distro, the main points to check are if it has the ntfs-3g driver (as recent a version as possible, as it keeps getting better at a fast pace) and the ntfstools / Linux-NTFS suite I mentioned earlier, especially if you’ve used EFS to encrypt your data (in which case the only viable solution appears to be ntfsdecrypt from that suite, which needs the certificate with which the files were encrypted, which in it’s turn needs you to boot the (dead) machine, but it appears to be the only way to get the data back).

Using dd/ddrescue to recover (NTFS) partitions

dd / ddrescue

The tool to move binary data from one place to another under Linux is dd. It also has a data-recovery-oriented cousin called ddrescue, which basically does the same thing, but is more fault-tolerant.
Basic dd usage:

dd if=/source of=/destination

if stands for input file and of for output file, and neither of them has to be an actual file (in the Windows sense); in the above example, /dev/sda1 is the first partition on the sda disk.
To back up just the MBR of the disk (the first 512 bytes) use:

dd if=/dev/sda of=/mnt/sdb1/saved/mbr.bin bs=512 count=1

This assumes that source disk is sda and that sdb1 is the partition to which you want to back up the data, so in your particular case they may need to be changed. See the next section if you’re not sure which disk is mapped to which name.
ddrescue uses fixed-position input (first) and output (second) arguments:

ddrescue -v /source /destination

The -v option makes ddrescue verbose (i.e. periodically print progress).
Note: by default, dd prints no progress/info until it’s job is finished. To check up on it’s progress, open another console (the terminals are mapped to Alt+N shortcuts in Linux, N >= 1, usually up to 4) and send it the USR1 signal. To do that, first you need to find it’s PID using ps:

ps -A|grep dd

Then, assuming the PID of the dd process is 3456, use kill:

kill -USR1 3456

That won’t actually kill the process, in spite of it’s name; it will just send it the USR1 signal, which makes dd print it’s current status (switch to the dd terminal to see it). The command’s name (“kill”) comes from it’s most frequent usage, which is to send a process the KILL signal (i.e. “kill” it).

Linux drive mapping

Linux maps your disks under /dev with names following the (“regex-like”) pattern [hs]d[abcd]. An h prefix means an (older) IDE disk, meanwhile an s prefix means a serial disk (usually an internal SATA or external USB disk). The individual partitions follow the disk naming + a digit to designate the partition number. So, for example, if you have a SATA disk with two partitions, the disk would be /dev/sda, the first partition would be /dev/sda1 and the second partition /dev/sda2.
To see the available disks/partitions, use ls (the Linux equivalent of dir):

ls /dev/sd*
ls /dev/hd*

To get extended disk info, use hdparm:

hdparm -I /dev/sda

The disks (actually the partitions) found under /dev need to be mounted before the files on them can be read/written; up until that point they are just huge blobs of binary data.

Note: for the rest of this writing, for simplicity’s sake, I’ll assume that sda is the broken disk and it has wto partitions, and that the recovered files/image go to sdb.

There are two ways to mount NTFS partitions: either using the default NTFS driver which comes with mount (ignores many problems, doesn’t care if Windows was improperly stopped & the drive was left “unclean”, read-only mode by default) or the ntfs-3g driver (more sensitive, read-write by default). Use the plain mount for the broken disk and the ntfs-3g version for the drives to which you need read-write access.
First off, you need to make appropriate folders for the partitions to be mounted under; standard practice is to do it under the /mnt folder. e.g.:

mkdir /mnt/sda1
mkdir /mnt/sda2
mkdir /mnt/sdb1

Note that the /mnt folder may not exist, in which case it must be created first: mkdir /mnt
Next, mount partitions from the broken disk (read-only):

mount /dev/sda1 /mnt/sda1
mount /dev/sda2 /mnt/sda2

The syntax of the mount command is straight-forward: mount /what /where; /what is the device, /where is the mount point in the filesystem. It takes other arguments, such as -t type to set the filesystem type, but NTFS is (nowadays) recognized automatically. The naming convention for the mount points is at your choice (you could mount the thing on something like /my/broken/disk/partition/number/1), but sticking to the “standard” /mnt path and using the original device’s name (or the partition letter if you’re more accustomed to that and a lazy typist, such as /mnt/c) is easier, and the help you find on the net will make more sense.
Last step in the mount process: mounting the destination disk in read-write mode (default for ntfs-3g):

ntfs-3g /dev/sdb1 /mnt/sdb1


mount -t ntfs-3g /dev/sdb1 /mnt/sdb1

The syntax is similar to the mount command; to check if the distro you chose has the ntfs-3g command built in, simply try to run it. If it doesn’t, choose another distro.

Copying the data

Run either dd or ddrescue (the latter is preferred if the disk is only partially readable):

dd if=/dev/sda1 of=/mnt/sdb1/saved/part1.bin


ddrescue -n /dev/sdb1 /mnt/sdb1/saved/part1.bin

WARNING: pay attention not to pass as the destination to dd/ddrescue entire disks unless you actually want their contents overwritten (which will be the case when you restore the saved image to a new disk); be sure to add a file name otherwise. The -n option prevents ddrescue from retrying error areas, which is usually what you want. If you have a disk which does yield data after enough retries, don’t use it.

Mounting the (NTFS) partition(s) from Linux/Windows

You can mount the newly backed-up partitions from Linux using the loop feature:

mount -o loop /mnt/sdb1/saved/part1.bin /mnt/part1

The partition can also be mounted directly from Windows using the ImDisk Virtual Disk Driver (free) or using some rather expensive commercial tools (google for alternatives).

Backing up/restoring partitions/whole disks

Alternatively, you can use the dd command to copy the entire disk and write the image to a fresh (identical) disk. Writing an image to a partition/disk using dd simply requires passing the disk as the of argument:
Restoring a partition:

dd if=/mnt/sdb1/saved/part1.bin of=/dev/sdc1

Restoring an entire disk:

dd if=/mnt/sdb1/saved/whole-disk.bin of=/dev/sdc

WARNING: be careful when overwriting raw partition/disk contents; choose other recovery methods unless you understand exactly what you’re doing.

Recovering files from raw data/deleted files: data carving

If the partition table/NTFS structure is broken and you can’t mount the partitions but you can read the binary data, you can use TestDisk to recover some of the files (the ones with a specific structure, such as images and music, are more likely to be found as opposed to, say, plain text files). This is basically the same thing that file recovery programs (such as Recuva) do on the unused space of a disk to recover deleted files.

Recovering EFS encrypted files

As I’ve mentioned in the opening paragraph, to recover EFS encrypted files, even under Linux, you need a recovery certificate. If you don’t have one, EFS file recovery software might help, but I’ve had little luck using them. I know of no open source/free software which does that, so you’ll probably have to use commercial software such as Advanced EFS Data Recovery from ElcomSoft (demo version available). The link called “encrypted file system recovery” from the following section details the process of manually extracting the required information for EFS recovery.

Further reading

Moral of the story

  • ALWAYS BACK UP YOUR IMPORTANT DATA. Seriously. Now. Go get some storage space (USB flash drive, external hard disk, even DVDs if you make a new one often enough, as they tend not to last very long) and copy your data on it. GO!
  • Don’t use EFS under NTFS. Use an alternative encryption solution, e.g. TrueCrypt. There are portable (i.e. works-from-flash-drive) editions of most encryption tools should the need arise, and they are reliable (I’ve used TrueCrypt without problems for quite a while now).
  • If you MUST use EFS, create a recovery certificate using CIPHER /R:filename (details here) and store it in a safe place.

Written by vtopan

November 15, 2009 at 11:51 PM