FreeBSD partition resizing requiredGetting Wake On Lan (WOL) working

Dealing with Unicode in filenames

I've recently acquired a portable WD 250GB USB external disk drive, based on a 2.5" drive, very small and light. Since I could carry it around with me, it seemed like a good way to start having an 'off-site' backup. Even copying stuff from Debian was easy, but I had to install the ntfs-3g package as the disk is NTFS formatted.

However, I encounted a problem when trying to copy my mp3 collection on to it via thunar in xfce4. The copy failed, complaining that the file system did not support the characters in some of the filenames: "Invalid or incomplete multibyte or wide character". On inspection, these turned out to be characters from the Unicode Latin-1 Supplement set in UTF-8, as my Debian locale is set to en_GB.utf8. Trawling Google indicated that the version of ntfs-3g in Debian lenny might have limited Unicode support. Hmm, looks like I needed to figure out how to rename the files to eliminate the funny European characters. Let's face it, 26 characters and a bit of punctuation should be enough for any language.

I'd never had to deal with non-ascii characters in ernest before, so I had a perform a little research on UTF-8 and the Python capabilities. There are a lot of resources (e.g. an article by Markus Kuhn and by anonymous).

Armed with this info, I was ready. First, many files did I have to deal with? I produced a list of all the files and directories under the two directories containing mp3 files using:

  find classical mp3 -name -print >flist

Next, into the python code. I made use of the codecs module, which provides a host of capabilities for dealing with Unicode data. To process the list of mp3 pathnames, I used, which returns a file-like object that can handle Unicode strings. The following code (contained in a file called, hence module xlate) will read in the list of pathnames from flist and return a list of those pathnames with non-ascii characters.

  import codecs
  def collect(fn):
  """return list of lines in file fn that contain non-ascii characters."""
      tlist = list()
      flist =,'rb','utf-8','replace').readlines()
      for f in flist:
          for c in f:
              if ord(c) > 127:
                  tlist.append(f.strip()) # remove trailing newline
      return tlist

Let's run this in the interpreter:

  [mark@amber:/rep/music] python 
  Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) 
  [GCC 4.3.2] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import xlate
  >>> tlist = xlate.collect("flist")
  >>> tlist[:1]

OK, a good start. Now, how many different characters are we dealing with? Let's add another function to that will return a list of unique characters found:

  def ident_chars(ls):
    """return list of non-ascii chars in unicode strings found in list ls."""
      uchars = list()
        for f in ls:
            for c in f:
                if ord(c) > 127 and uchars.count(c) == 0:
      return sorted(uchars)

Let's test it:

  >>> reload(xlate)
  <module 'xlate' from 'xlate.pyc'>
  >>> clist = xlate.ident_chars(tlist)
  >>> clist
  [u'\xab', u'\xbb', u'\xc9', u'\xe0', u'\xe1', u'\xe2', u'\xe4', u'\xe8', 
   u'\xe9', u'\xef', u'\xf1', u'\xf3', u'\xf4', u'\xf7', u'\xfa', u'\xfc', 

Not too many. Still, it would be useful to have a function which automatically wrote a dictionary to make it easy to define the mappings of "weird" european characters to standard anglo-saxon. Such a function, to write the unicode code point, an empty replacement and the original character glyph, might look something like the following:

  def create_mapping(clist,fn): 
      """Write python dictionary definition code to filename fn, using
      unicode characters in list clist as keys."""

      with,'wb','utf-8','replace') as f:
          f.write("mapping = {\n")
          for c in clist:
              f.write(u'\tu\'\\x%x\' : \'\', # %s\n '%(ord(c),c))
          f.write('\tu\'\\x00\' : \'\' }\n')

What does the output of running this function look like?

  >>> create_mapping(clist,"")
  >>> with open("") as f:
  ...     print
  mapping = {
      u'\xab' : '', # «
      u'\xbb' : '', # »
      u'\xc9' : '', # É
      u'\xe0' : '', # à
      u'\xe1' : '', # á
      u'\xe2' : '', # â
      u'\xe4' : '', # ä
      u'\xe8' : '', # è
      u'\xe9' : '', # é
      u'\xef' : '', # ï
      u'\xf1' : '', # ñ
      u'\xf3' : '', # ó
      u'\xf4' : '', # ô
      u'\xf7' : '', # ÷
      u'\xfa' : '', # ú
      u'\xfc' : '', # ü
      u'\xfffd' : '', # �
      u'\x00' : '' }

That makes things easy to setup my preferred non-ascii to ascii character mappings. I just insert the ascii characters into the appropriate dictionary element definition. The default for the dictionary mapping is to nothing.

  mapping = {
      u'\xab' : '', # «
      u'\xbb' : '', # »
      u'\xc9' : 'E', # É
      u'\xe0' : 'a', # à
      u'\xe1' : 'a', # á
      u'\xe2' : 'a', # â
      u'\xe4' : 'a', # ä
      u'\xe8' : 'e', # è
      u'\xe9' : 'e', # é
      u'\xef' : 'i', # ï
      u'\xf1' : 'n', # ñ
      u'\xf3' : 'o', # ó
      u'\xf4' : 'o', # ô
      u'\xf7' : '', # ÷
      u'\xfa' : 'u', # ú
      u'\xfc' : 'u', # ü
      u'\xfffd' : '', # �
      u'\x00' : '' }

This file can then be copied wholesale into and used as the basis for the next function we have to write: actually translate filenames. Gird your loins. We are going to take a peek into the ugly world of hacking at the first thing you thought of. Those of a nervous disposition should look away now...

OK, that first thought was: I have a list of pathnames to translate, so I just need to run through them, deriving the plain anglo-saxon name for each path, then rename the file. Sounds simple enough, let's write some code:

  def xlate(ls):
      newls = list()
      for f in ls:
          s = f
          for m in mapping:
              s = s.replace(m,mapping[m])
      return newls

  def rename_files(old,new):
      map(os.rename, old, new)

So, the function xlate takes a list of pathnames with non-ascii characters, translates the odd characters, then returns a list of the translated pathnames. The rename_files function will actually perform the renaming. Hang on a moment, that's not going to work. If you look again at the snippet from tlist (above), the parent directory comes before the contained files. Once the containing directory is renamed, all the other renames in that directory are going to fail because the original (non-ascii) pathname no longer exists. Bugger, now what?

OK, I need a process that visits the files before the containing directory. One easy solution is find with the -depth option. Combined with -execdir a simple python script would do the job, say something like this, where the code resides in

  def xlate_str(s):
    """Return copy of s with utf-8 characters, as defined in dictionary 
    mapping, mapped to ascii."""
    n = s
    for m in mapping:
        n = n.replace(m,mapping[m])
    return n

  # main code
  if __name__ == '__main__':
      pname = sys.argv[0]
      if len(sys.argv) == 2:
          old = unicode(sys.argv[1],'utf-8','replace')
          new = xlate_str(old)
          if old != new:
                  print "> %s %s"% (old,new)
              except OSError,e:
                  print >>sys.stderr,"%s: error in %s at %s: %s"%\
          print >>sys.stderr, "%s: one argument expected."%(pname,)

One could run this with a command such as:

  find mp3 classical -execdir python /rep/music/ {} \;

However, this has an overhead in that it invokes python for every file. A more efficient way would be to perform the filesystem walk (ala find) within python itself, if there is an simple method. And there is: the os modules offers a ready solution, os.walk. The key element is the topdown option, which allows us to invoke a depth-first directory walk, that is we can deal with the filenames before the containing directory. With this function, we can perform everything within python.

  def walk(mod_name,top,do_move): 
  """Walk directory tree (depth first) with root at top, renaming
     contained files/directories that contain non-ascii characters to
     use only ascii characters.

     Error messages are prefixed with the sring mod_name.

     If do_move is false, the renames are not actually performed, but
     a messaage indicating what would have been done is issued.

      for root, dirs, files in os.walk(top,topdown=False):
          eroot = unicode(root,'utf-8','replace');
          for fd in files+dirs:
              old = unicode(fd,'utf-8','replace')
              new = xlate_str(old)
              if old != new:
                  if os.path.exists(os.path.join(eroot,new)):
                      print >>sys.stderr,\
                      "%s: at %s: xlated file %s already exists."%\
                          if do_move: os.rename(os.path.join(eroot,old),
                          print "%s: > %s %s"%(eroot,old,new)
                      except OSError,e:
                          print >>sys.stderr,"%s: error in %s at %s: %s"%\

  # main code
  if __name__ == '__main__':
      mod_name = os.path.basename(sys.argv[0])
      top = os.getcwd()
      do_move = True
          opts,args = getopt.getopt(sys.argv[1:],'np:')
          for o,v in opts:
              if o == '-n': do_move = False
              elif o == '-p': top = v
      except getopt.GetoptError,e:
          print >>sys.stderr,"%s: unknown argument: -%s"%(mod_name,e.opt)


I've added a couple of command arguments to the main code that are passed directly to the xlate.walk function, that is the starting directory and a flag to indicate the the move should actually be performed.

Once this rename process had been run, I was able to copy all my mp3 files to the WD external harddrive. On the other hand, if I were a true European, perhaps I should have just waited until Debian was upated with a version of ntfs-3g which had better unicode support.

Addendum 7th May, 2010

I discovered, in a post of the comp.lang.python newsgroup by Peter Otten (in response to a question from coldpizza), that I had wasted my time. The python unicodedata module already includes a means of eliminating non-ascii characters. This is using unicode.normalize. Here's an example of the usage:

  astr = unicodedata.normalize('NFD',ustr).encode('ascii','ignore')

which essentially replaces the xlate_str function. So, no lookup table needs to be created at all. I hate reinventing the wheel...

FreeBSD partition resizing requiredGetting Wake On Lan (WOL) working