Tagged: YAML

Day 9: Processing MARC with Catmandu

09_librecatIn the previous days we learned how we can use the catmandu command to process structured data like JSON. Today we will use the same command to process MARC metadata records. In this process we will see that MARC can be processed using JSON paths but this is a bit cumbersome. We will introduce MARCspec as an easier way to point to parts of a MARC record.

As always, you need to startup your Virtual Catmandu (hint: see our day 1 tutorial) and start up the UNIX prompt (hint: see our day 2 tutorial).

In the Virtual Catmandu installation we provided a couple of example MARC files that we can inspect with the UNIX command cat or less. In the UNIX prompt inspect the file Documents/camel.usmarc, for instance, with cat:

$ cat Documents/camel.usmarc

You should see something like this:


Like JSON the MARC file contains structured data but the format is different. All the data is on one line, but there isn’t at first sight a clear separation between fields and values. The field/value structure there but you need to use a MARC parser to extract this information. Catmandu contains a MARC parser which can be used to interpret this file. Type the following command to transform the MARC data into YAML (which we introduced in the previous posts):

$ catmandu convert MARC to YAML < Documents/camel.usmarc

You will see something like this:


When transforming MARC into YAML it looks like something with a simple top level field _id containing the identifier of the MARC record and a record field with a deeper array structure (or more correct an array-of-an-array structure).

We can use catmandu to read the _id fields of the MARC record with the retain_field fix we learned in the Day 6 post:

$ catmandu convert MARC --fix 'retain_field(_id)' to YAML < Documents/camel.usmarc

You will see:

_id: 'fol05731351 '
_id: 'fol05754809 '
_id: 'fol05843555 '
_id: 'fol05843579 '

What is happening here? The MARC file Documents/camel.usmarc contains more than one MARC record. For every MARC record catmandu extracts the _id field.

Extracting data out of the MARC record itself is a bit more difficult. MARC is an array-an-array, you need indexes to extract the data. For instance the MARC leader is usually in the first field of a MARC record. In the previous posts we learned that you need to use the 0 index to extract the first field out of an array:

$ catmandu convert MARC --fix 'retain_field(record.0)' to YAML < Documents/camel.usmarc
_id: 'fol05731351 '
- - LDR
  - ~
  - ~
  - _
  - 00755cam  22002414a 4500

The leader value itself is the fifth entry in the resulting array. So, we need index 4 to extract it:

$ catmandu convert MARC --fix 'copy_field(record.0.4,leader); retain_field(leader)' to YAML < Documents/camel.usmarc

We used here a copy_field fix to extract the value into a field called leader. The retain_field fix is used to keep only this leader field in the result. To process MARC data this way would be very verbose, plus you need to know at which index position the fields are that you are interested in. This is something you usually don’t know.

Catmandu introduces Carsten Klee’s MARCspec to ease the extraction of MARC values out of a record. With the marc_map fix the command above would read:


I skipped here writing the catmandu commands (they will be the same everytime). You can put these fixes into a file using nano (see the Day 5 post) and execute it as:

catmandu convert MARC --fix myfixes.txt to YAML < Documents/camel.usmarc

Where myfixes.txt contains the fixes above.

To extract the title fields, the field 245 remember? ;), you can write:


Or, if you are only interested in the $a subfield you could write:


More elaborate mappings are possible. I’ll show you more complete examples in the next posts. As a warming up, here is some code to extract all the record identifiers, titles and isbn numbers in a MARC file into a CSV listing (which you can open in Excel).

Step 1, create a fix file myfixes.txt containing:


Step 2, execute this command:

$ catmandu convert MARC --fix myfixes.txt to CSV < Documents/camel.usmarc

You will see this as output:

"fol05731351 ","0471383147 (paper/cd-rom : alk. paper)","ActivePerl with ASP and ADO /Tobias Martinsson."
"fol05754809 ",1565926994,"Programming the Perl DBI /Alligator Descartes and Tim Bunce."
"fol05843555 ",,"Perl :programmer's reference /Martin C. Brown."
"fol05843579 ",0072120002,"Perl :the complete reference /Martin C. Brown."
"fol05848297 ",1565924193,"CGI programming with Perl /Scott Guelich, Shishir Gundavaram & Gunther Birznieks."
"fol05865950 ",0596000138,"Proceedings of the Perl Conference 4.0 :July 17-20, 2000, Monterey, California."
"fol05865956 ",1565926099,"Perl for system administration /David N. Blank-Edelman."
"fol05865967 ",0596000278,"Programming Perl /Larry Wall, Tom Christiansen & Jon Orwant."
"fol05872355 ",013020868X,"Perl programmer's interactive workbook /Vincent Lowe."
"fol05882032 ","0764547291 (alk. paper)","Cross-platform Perl /Eric F. Johnson.

In the fix above we mapped the 245-field to the title. The ISBN is in the 020-field. Because MARC records can contain one or more 020 fields we created an isbn array using the isbn.$append syntax. Next we turned the isbn array back into a comma separated string using the join_field fix. As last step we deleted all the fields we didn’t need in the output with the remove_field syntax.

In this post we demonstrated how to process MARC data. In the next post we will show some examples how catmandu typically can be used to process library data.

Continue with Day 10: Working with CSV and Excel files >>

Day 7: Catmandu JSON paths

07_librecatrojectYesterday we learned the command catmandu and how it can be used to parse structured information. Today we will go deeper into catmandu and describe how to pluck data out of structured information. As always, you need to startup your Virtual Catmandu (hint: see our day 1 tutorial) and start up the UNIX prompt (hint: see our day 2 tutorial).

Today will we fetch a new weather report and store it in a new file weather2.json. Lets try to download Tokyo:

$ curl http://api.openweathermap.org/data/2.5/weather?q=Tokyo,jp > weather2.json

From the previous tutorials we know many commands how to examine this data set. For instance, to get a quick overview of the content of weather2.json we can use the cat command:

$ cat weather2.json

Or, we could use the less command:

$ less weather2.json

Remember to type the ‘q’ key to exit less.

We could also use nano to inspect the data, but we skip that for now. Nano is a text editor and is not particularly suited for data.

To count the number of lines, words and characters in weather2.json we can use the wc command:

$ wc weather2.json
1 3 463

This output shows that weather2.json contains 1 line , 3 words and 463 characters. The 1 line is indeed correct: the file contains one big line of JSON. The 463 characters is also correct: when you count every character including spaces you get to 463. But 3 words is obviously wrong. Generic UNIX programs like wc have trouble with counting words in structured information. The command doesn’t know this file is in the JSON format which contains fields and values. You need to use specialized tools like catmandu to make sense of this output.

We also saw in the previous post how you can use catmandu to transform the JSON format into the YAML format which is easier to read and contains the same information:

$ catmandu convert JSON to YAML < weather2.json



We also learned some fixes to retrieve information out of the JSON file like retain_field(main.temp).

In this post we delve a bit deeper into ways how to point to fields in a JSON file.

This main.temp is called a JSON Path and points to a part of the JSON data you are interested in. The data, as shown above, is structured like a tree. There are top level simple fields like: base,cod,dt,id which contain only text values or numbers. There are also fields like coord that contain a deeper structure like lat and lon.

Using a JSON path you can point to every part of the JSON file using a dot-notation. For simple top level fields the path is just the name of the field:

  • base
  • cod
  • dt
  • id
  • name

For the fields with deeper structure you add a dot ‘.’ to point to the leaves:

  • clouds.all
  • coord.lat
  • coord.lon
  • main.temp
  • etc…

So for example. If you would have a deeply nested structure like:


Then you would point to the c field with the JSON Path x.y.z.a.b.c.

There is one extra path structure I would like to explain and that is the when a field can have more than one value. This is called an array and looks like this in YAML:


In the example above you see a field my which contains a deeper field colors which has 3 values. To point to one of the colors you need to use an index. The first index in a array has value 0, the second the value 1, the third the value 2. So, the JSON path of the color red would be:

  • my.color.2

In almost all programming languages things get counted starting with 0. An old programming joke is:

There are 10 types of people in the world:
Those who understand binary,
Those who don’t,
And those who count from zero.

(hint: this is a double joke, 10 in binary == 2 if you count from 0, or 3 when you count from 1).

There is one array type in our JSON report and that is the weather field. To point to the description of the weather you need the JSON Path weather.0.description.

In this post we learned the JSON Path syntax and how it can be used to point to parts of a JSON data set want to manipulate. We explained the JSON path using a YAML transformation as example, because this is easier to read. YAML and JSON are two formats that contain the same informational content (and thus both can work with JSON Path) but look different when written into a file.

Continue to Day 8: Processing JSON data from webservices >>

Day 6: Introduction into Catmandu

06_librecatprojectIn the previous days we learned the UNIX commands grep, nano, ls and less. Today we will introduce you to a UNIX command we have created in the LibreCat project called catmandu. The catmandu command is used to process structured information.  To demo this command, as always, you need to startup your Virtual Catmandu (hint: see our day 1 tutorial) and start up the UNIX prompt (hint: see our day 2 tutorial).

In this tutorial we are going to process structured information. We call data structured when it organised in such a way is that it easy processable by computers. Previously we processed text documents like War and Peace which is structured only in words and sentences, but a computer doesn’t know which words are part of the title or which words contain names. We had to tell the computer that. Today we will download a weather report in a structured format called JSON and inspect it with the command catmandu.

At the UNIX prompt type in this command:

$ curl http://api.openweathermap.org/data/2.5/weather?q=Gent,be

[Update: as of end 2015 the OpenWeatherMap API requires an API key. Use this link to download a copy of the Ghent weather report :

$ curl https://gist.githubusercontent.com/phochste/7673781b19690f66cada/raw/67050da98a7e04b3c56bb4a8bc8261839af57e35/weather.json


You will see a JSON output like:

"sunrise":1417159365,"sunset":1417189422},"weather":[{"id":500,"main":"Rain","description":"light rain",
"icon":"10d"}],"base":"cmc stations","main":{"temp":281.15,"pressure":1006,"humidity":87,"temp_min":281.15,

All these fields tell something about the current weather in Gent, Belgium. You can recognise that there is a light rain and the temperature is 281.15 degrees Kelvin (about 8 degrees Celsius).  Write the output of this command to a file weather.json (using the ‘>’ sign we learned in the day 5 tutorial) so that we can use it in the next examples.

$ curl https://gist.githubusercontent.com/phochste/7673781b19690f66cada/raw/67050da98a7e04b3c56bb4a8bc8261839af57e35/weather.json > weather.json

When you type the ls command you should see the new file name weather.json appearing.

With the catmandu command you can process this file to make it a bit easier readable. For instance type:

$ catmandu convert JSON to YAML < weather.json

YAML is another format for structured information which is a bit easier to read for human eyes. Our weather report should now look like this:


Catmandu can be used to process structured information like the UNIX grep command can process unstructured information. For instance lets try to filter out the name of this report. Type in this command:

$ catmandu convert JSON --fix 'retain_field(name)' to YAML < weather.json

You should end up with something like:

name: Gent

The –fix option in Catmandu is used to ‘massage’ the input weather.json filtering fields we would like to see. Only one fix was used ‘retain_field’, which throws away all the data from the input except the ‘name’ field. By the way, the file weather.json wasn’t changed! We only read the file and displayed the output of catmandu command.

The temperature in Gent is the in ‘temp’ part of the ‘main’ section in weather.json. To filter this out we need two retain_field fixes: one for the main section, one for the temp section:

$ catmandu convert JSON --fix 'retain_field(main); retain_field(main.temp)' to YAML < weather.json

You should now see something like this:

  temp: 281.15

When massaging data you often need to create many fixes to process a data file in the format you need. With the nano command you can write all the fixes in a file. Start the nano editor with the command:

$ nano weather.fix

In nano type now the two fixes above:


To exit nano type Ctrl-X, press Y to confirm the changes and press Enter to confirm the file name.

With this file it will be a bit easier to create many fixes. The name of the fix file can be used to repeat the commands above:

$ catmandu convert JSON --fix weather.fix to YAML < weather.json

To add more fixes we can again edit the weather.fix file. Type:

$ nano weather.fix

And add these lines after the two previous lines:

prepend(main.temp,"The temperature is")
append(main.temp," degrees Kelvin")

Save the changes with Ctrl-X, Y, Enter and execute catmandu  again:

$ catmandu convert JSON --fix weather.fix to YAML < weather.json

You should now see as ouput:

  temp: The weather is 281.15 degrees Kelvin

Catmandu contains many fixes to manipulate data. Check the documentation to get a complete list. This post only presented a short introduction into catmandu. In the next posts we will go deeper into its capabilities.

Continue to Day 7: Catmandu JSON paths >>