Sunday, February 23, 2014

Tooling Around With Midnight Commander

If you followed my simple SSH guide, you'll be logging in to your OMV installation and probably coming across a BASH prompt.  For the unitiated, BASH can be a bit intimidating.  And, even once you are fairly comfortable with it, a command line prompt can be a bit painful for some tasks.  So on most linux systems, I like to install midnight commander.  On Debian, this can be done just via apt;

sudo apt-get install mc

Then type mc to run midnight commander.  Basically, this is a text-based file manager that can be used from a linux shell, such as via a PuTTY session.  Most common commands like copying, renaming, etc have been given short cuts.  And the two-window approach allows you to navigate through directories in one window, while keeping your path in another window fixed.  Switching from window to window is done via the tab key.  I find this particularly useful when re-organising files, where I may need to search through multiple directories to find various files before copying and consolidating them.

Midnight Commander vs Rsync

An occasional question in linux forums is what is the best and quickest way to copy files from one directory to another, across different hard drives.  And the most common answer is rsync. 

I'll put together a short tutorial on rsync a little bit later, but basically it is a brilliant little application for synchronising the contents of two directories, potentially across different file systems and across networks.  It really is one of the gems in the linux toolbox.  And the real power of rync is the ability to work incrementally.  That is, it only copies files as required.  So you can set up rsync to copy the contents of one directory across to another directory, but it will only copy the files needed -- if a copy already exists on the destination, copying won't occur (although, there are settings to override this).  Furthermore, rsync checks at the block level, so even if a file is present at the destination, if it is out of date it can be updated with the required blocks.  This is really useful if you have large files, with small changes.  Most backup software will require you to backup the entire file whereas rsync only requires you to back up the differential parts.

So rsync is pretty brilliant.  But my own experience is that it has a few limitations.  If you are transferring files from one directory to another, empty directory, then rsync doesn't have any differential functionality to perform.  And in these instances, I found rsync to actually be slower than a standard linux copy command. 

Recently I needed to consolidate all my movies, scattered across a couple of different directories on two different 2TB hard drives, into a new, single directory on a 4TB hard drive.  Most of these files are between 1GB and 7GB in size, but there are also hundreds of other small resource files such as thumbnails and nfo files which are used by XBMC.  Anecdotally, I found the small files copied at a similar pace whether using copy of rsync.  But the larger media files were noticably slower. 

In the end, I ran a copy across each directory in a single pass (using midnight commander).  Then, after all the files were copied, I ran rsync just to make sure midnight commander hadn't missed anything.  Over 3TB-3.5TB of data, shaving a minute or two per file can make a reasonably significant difference in overall time.

No comments:

Post a Comment