{"id":270,"date":"2013-03-10T22:56:22","date_gmt":"2013-03-11T05:56:22","guid":{"rendered":"http:\/\/galencharlton.com\/blog\/?p=270"},"modified":"2013-03-20T17:02:59","modified_gmt":"2013-03-21T00:02:59","slug":"bestiary-of-monstrous-data-1-a-trip-through-the-character-mangler","status":"publish","type":"post","link":"https:\/\/galencharlton.com\/blog\/2013\/03\/bestiary-of-monstrous-data-1-a-trip-through-the-character-mangler\/","title":{"rendered":"Bestiary of Monstrous Data #1: a trip through the character mangler"},"content":{"rendered":"

This is the first part in an occasional series on how good data can go bad.<\/em>\"bestiary_viper_thumbnail\"<\/p>\n

Consider the following snippets of a MARC21 record for the Spanish edition of the fourth Harry Potter book.<\/p>\n

\r\n00998nam  2200313 c 4500\r\n...\r\n240 10 $a Harry Potter and the goblet of fire $l Espa\u00f1ol\r\n245 10 $a Harry Potter y el c\u00e1liz de fuego \/ $c J.K. Rowling ; [traducci\u00f3n, Adolfo Mu\u00f1oz Garc\u00eda y Nieves Mart\u00edn Azofra]\r\n<\/pre>\n
The original record uses the Unicode character set with the UTF-8 character encoding.  However, If you load this record into a modern ILS, e.g. Koha or Evergreen, the title is likely to end up displayed as:<\/p>\n
\r\nHarry Potter y el c\u00a9\u0141liz de fuego \/ J.K. Rowling ; [traducci\u00a9\u0111n, Adolfo Mu\u00a9\u0142oz Garc\u00a9\u01afa y Nieves Mart\u00a9\u01afn Azofra]\r\n<\/pre>\n
Too much copyright! This isn’t an electronic course reserves blog!<\/p>\n
What happened?  Look at the 9th position of the leader (counting from zero), and you’ll see that it is blank.  In MARC21, blank means that the record uses the MARC-8 character set, while ‘a’ means that it uses Unicode.  Many, if not most, modern MARC tools will go by the Leader\/09 to decide if a character conversion is needed.  If the leader position is wrong, poor, defenseless diacritics will get mangled.<\/p>\n
Why are there so many copyright signs in the mistreated title?  As it happens, the UTF-8 representation of many common characters with Western European diacritics starts with byte 195 (or C3 in hexadecimal).  What does C3 mean in the MARC-8 character encoding?  You’ve guessed it: the copyright symbol.<\/p>\n
There are a couple lessons to draw from this.  First, using a good character encoding isn’t enough; you must also say what you’re up to.  Second, if you look at enough bad data, you will start to recognize patterns on sight.  If you deal with a lot of data, that “second sight” is an arcane but useful skill to develop.<\/p>\n
CC-BY image of a woodcut of a viper<\/a> courtesy of the Penn Provenance Project<\/a>.<\/em><\/small><\/p>\n
Share this:<\/h3>