Thank you all for joining our Catmandu advent calendar this month. We hope that you enjoyed our daily posts. Catmandu is a very rich programming environment which provides command line tools and even an API. In these blogposts we provided only a short introduction into all these modules. Hopefully we will see you next year again with more examples!
The Catmandu community consists of all people involved in the project, no matter if they do programming, documentation, or drawing cats. We want to thank them all for a wonderful year!
- Christian Pietsch
- Dave Sherohman
- Friedrich Summann
- Jakob Voss
- Johann Rolschewski
- Jörgen Eriksson
- Maria Hedberg
- Mathias Lösch
- Najko Jahn
- Nicolas Franck
- Nicolas Steenlant
- Patrick Hochstenbach
- Petra Kohorst
- Snorri Briem
- Vitali Peil
And a big round of applause for our contributors who kept us sending bug reports and ideas for new features. If you would like to contribute, then please take a look at the contributions section of Catmandu documentation. Don’t be shy to contact us with questions, feature requests, bug fixes, documentation and cat cartoons!
This advent calendar will stay online for you reference.
As a special gift we have still some catmandu USB sticks available that we can send to you. Please send a line to “patrick dot hochstenbach at ugent dot be”. The first 5 emailers will get a free USB!
Yesterday we learned how to import RDF data with Catmandu. Exporting RDF can be as easy as this:
catmandu convert RDF --url http://d-nb.info/1001703464 to RDF
By default, the RDF exporter Catmandu::Exporter::RDF emits RDF/XML, an ugly and verbose serialization format of RDF. Let’s configure catmandu to use the also verbose but less ugly NTriples. This can either by done by appending
--type ntriple on command line or by adding the following to config file
exporter: RDF: package: RDF options: type: ntriples
The NTriples format illustrates the “true” nature of RDF data as a set of RDF triples or statements, each consisting of three parts (subject, predicate, object).
Catmandu can be used for converting between one RDF serialization format to another, but more specialized RDF tools, such as such rapper are more performant especially for large data sets. Catmandu can better help to process RDF data to JSON, YAML, CSV etc. and vice versa.
Let’s proceed with a more complex workflow and with what we’ve learned at day 13 about OAI-PMH and another popular repository: http://arxiv.org. There is a dedicated Catmandu module Catmandu::ArXiv for searching the repository, but ArXiv also supports OAI-PMH for bulk download. We could specify all options at command line, but putting the following into
catmandu.yml will simplify each call:
importer: arxiv-cs: package: OAI options: url: http://export.arxiv.org/oai2 metadataPrefix: oai_dc set: cs
Now we can harvest all computer science papers (
set: cs) for a selected day (e.g.
$ catmandu convert arxiv --from 2014-12-19 --to 2014-12-19 to YAML
The repository may impose a delay of 20 seconds, so be patient. For more precise data, we better use the original data format from ArXiV:
$ catmandu convert arxiv --set cs --from 2014-12-19 --to 2014-12-19 --metadataPrefix arXiv to YAML > arxiv.yaml
The resulting format is based on XML. Have a look at the original data (requires module Catmandu::XML):
$ catmandu convert YAML to XML --field _metadata --pretty 1 < arxiv.yaml $ catmandu convert YAML --fix 'xml_simple(_metadata)' to YAML < arxiv.yaml
Now we’ll transform this XML data to RDF. This is done with the following fix script, saved in file
The following command generates one RDF triple per record, consisting of an arXiv article identifier, the property
http://purl.org/dc/elements/1.1/title and the article title:
$ catmandu convert YAML to RDF --fix arxiv2rdf.fix < arxiv.yaml
To better understand what’s going on, convert to YAML instead of RDF, so the internal aREF data structure is shown:
$ catmandu convert YAML to YAML --fix arxiv2rdf.fix < arxiv.yaml
dc_title: On Conditional Decomposability
This record looks similar to the records imported from RDF at day 13. The special field
_id refers to the subject in RDF triples: a handy feature for small RDF graphs that share the same subject in all RDF triples. Nevertheless, the same RDF graph could have been encoded like this:
--- http://arxiv.org/abs/1201.1733: dc_title: On Conditional Decomposability ...
To transform more parts of the original record to RDF, we only need to map field names to prefixed RDF property names. Here is a more complete version of
xml_simple(_metadata) retain_field(_metadata) move_field(_metadata,m) move_field(m.id,_id) prepend(_id,"http://arxiv.org/abs/") move_field(m.title,dc_title) move_field(m.abstract,bibo_abstract) move_field(m.doi,bibo_doi) copy_field(bibo_doi,owl_sameAs) prepend(owl_sameAs,"http://dx.doi.org/") move_field(m.license,cc_license) move_field(m.authors.author,dc_creator) unless exists(dc_creator.0) move_field(dc_creator,dc_creator.0) end do list(path=>dc_creator) add_field(a,foaf_Person) copy_field(forenames,foaf_name.0) copy_field(keyname,foaf_name.$append) join_field(foaf_name,' ') move_field(forenames,foaf_givenName) move_field(keyname,foaf_familyName) move_field(suffix,schema_honoricSuffix) remove_field(affiliation) end remove_field(m)
The result is one big RDF graph for all records:
$ catmandu convert YAML to RDF --fix arxiv2rdf.fix < arxiv.yaml
Have a look at the internal aREF format by using the same fix with
convert to YAML and try conversion to other RDF serialization forms. The most important part of transformation to RDF is to find matching RDF properties from existing ontologies. The example above uses properties from Dublin Core, Creative Commons, Friend of a Friend, Schema.org, and Bibliographic Ontology.
Continue to Day 18: Merry Christmas! >>
A common problem of data processing is the large number of data formats, dialects, and conceptions. For instance the
author field in one record format may differ from a similar field another format in its meaning or name. As shown in the previous articles, Catmandu can help to bridge such differences, but it can also help to map from and to data structured in a completely different paradigm. This article will show how to process data expressed in RDF, the language of Semantic Web and Linked Open Data.
- There are no records and fields: RDF data instead is a graph structure, build of nodes (“resources” or “values”) and directed links.
- Link types (“properties”) are identified by URI and defined in “ontologies”. In theory this removes the introductory common problem of data processing.
Because graph structures are fundamentally different to record structures, there is no obvious mapping between RDF and records in Catmandu. For this reason you better use dedicated RDF technology as long as your data is RDF. Catmandu, however, can help to process from RDF and to RDF, as shown today and tomorrow, respectively. Let’s first install the Catmandu module Catmandu::RDF for RDF processing:
$ cpanm –sudo Catmandu::RDF
If you happen to use this on a virtual machine from the Catmandu USB stick, you may first have to update another module to remove a nasty bug (the password is “catmandu”):
$ cpanm –sudo List::Util
You can now retrieve RDF data from any Linked Open Data URI like this:
$ catmandu convert RDF –url http://dx.doi.org/10.2474/trol.7.147 to YAML
We could also download RDF data into a file and parse the file with Catmandu afterwards:
$ curl -L -H 'Accept: application/rdf+xml' http://dx.doi.org/10.2474/trol.7.147 > rdf.xml $ catmandu convert RDF --type rdfxml to YAML < rdf.xml $ catmandu convert RDF --file rdf.xml to YAML # alternatively
Downloading RDF with Catmandu::RDF option
--url, however, is shorter and adds an
_url field that contains the original source. The RDF data converted to YAML with Catmandu looks like this (I removed some parts to keep it shorter). The format is called another RDF Encoding Form (aREF) because it can be transformed from and to other RDF encodings:
--- _url: http://dx.doi.org/10.2474/trol.7.147 http://dx.doi.org/10.2474/trol.7.147: dct_title: Frictional Coefficient under Banana Skin@ dct_creator: - <http://id.crossref.org/contributor/daichi-uchijima-y2ol1uygjx72> - <http://id.crossref.org/contributor/kensei-tanaka-y2ol1uygjx72> - <http://id.crossref.org/contributor/kiyoshi-mabuchi-y2ol1uygjx72> - <http://id.crossref.org/contributor/rina-sakai-y2ol1uygjx72> dct_date:- 2012^xs_gYear dct_isPartOf: <http://id.crossref.org/issn/1881-2198> http://id.crossref.org/issn/1881-2198: a: bibo_Journal bibo_issn: 1881-2198@ dct_title: Tribology Online@ http://id.crossref.org/contributor/daichi-uchijima-y2ol1uygjx72: a: foaf_Person foaf_name:Daichi Uchijima@ http://id.crossref.org/contributor/kensei-tanaka-y2ol1uygjx72: foaf_name: Kensei Tanaka@ http://id.crossref.org/contributor/kiyoshi-mabuchi-y2ol1uygjx72: foaf_name: Kiyoshi Mabuchi@ http://id.crossref.org/contributor/rina-sakai-y2ol1uygjx72: foaf_name: Rina Sakai@ ...
The sample record contains a special field
_url with the original source URL and six fields with URLs (or URIs), each corresponding to an RDF resource. The field with the original source URL (http://dx.doi.org/10.2474/trol.7.147) can be used as starting point. Each subfield (
dct_isPartOf) corresponds to an RDF property, abbreviated with namespace prefix. To fetch data from these fields, we could use normal fix functions and JSON path expressions, as shown at day 7 but there is a better way:
Catmandu::RDF provides the fix function
aref_query to map selected parts of the RDF graph to another field. Try to get the the title field with this command:
$ catmandu convert RDF –url http://dx.doi.org/10.2474/trol.7.147 –fix ‘aref_query(dct_title,title)’ to YAML
More complex transformations should better be put into a fix file, so create file
rdf.fix with the following content:
aref_query(dct_title,title) aref_query(dct_date,date); aref_query(dct_creator.foaf_name,author) aref_query(dct_isPartOf.dct_title,journal)
If you apply the fix, there are four additional fields with data extracted from the RDF graph:
$ catmandu convert RDF –url http://dx.doi.org/10.2474/trol.7.147 –fix rdf.fix to YAML
aref_query function also accepts a language, similar to JSON path, but the path is applied to an RDF graph instead of a simple hierarchy. Moreover one can limit results to plain strings or to URIs. For instance the author URIs can be accessed with
aref_query(dct_creator.,author). This feature is useful especially if RDF data contains a property with multiple types of objects, literal strings, and other resources. We can aggregate both with the following fixes:
Before proceeding you should add the following option to config file
importer: RDF: package: RDF options: ns: 2014091
This makes sure that RDF properties are always abbreviated with the same prefixes, for instance
dct for http://purl.org/dc/terms/.
Continue to Day 17: Exporting RDF data with Catmandu >>
Today we will look a bit further into MARC processing with Catmandu. By now you should already know how to startup the Virtual Catmandu (hint: see day 1) and start up the UNIX command prompt (hint: see day 2). We already saw a bit of MARC processing in day 9 and today we will show you how to transform MARC records into Dublin Core. This as a preparation to create RDF and Linked Data in the later posts.
First I’m going to teach you how to process different types of MARC files. On the Virtual Catmandu system we provided five example MARC files. You can find them in your Documents folder:
When you examine these files with the UNIX less command you will see that all the files have a bit different format:
$ less Documents/camel.mrk
$ less Documents/camel.usmarc
$ less Documents/marc.xml
$ less Documents/rug01.sample
There are many ways in which MARC data can be written into a file. Every vendor likes to use its own format. You can compare this with the different ways a text document can be stored: as Word, as Open Office, as PDF and plain text. If we are going to process these files with catmandu, then we need to tell the system what the exact format is.
We will work today with the last example rug01.sample which is a small export out of the Aleph catalog from Ghent University Library. Ex Libris uses a special MARC format to structure their data which is called Aleph sequential. We need to tell catmandu not only that our input file is in MARC but also in this special Aleph format. Let’s try to create YAML to see what it gives:
$ catmandu convert MARC --type ALEPHSEQ to YAML < Documents/rug01.sample
To transform this MARC file into Dublin Core we need to create a fix file. You can use the UNIX command nano for this (hint: see day 5 how to create files with nano). Create a file dublin.fix:
$ nano dublin.fix
And type into nano the following fixes:
Every MARC record contains in the 245-field the title of a record. In the first line we map the MARC-245 field to new field in the record called title:
In the second and third line we map authors to a field creator. In the rug01.sample file the authors are stored in the MARC-100 and MARC-700 field. Because there is usually more than one author in a record, we need to $append them to create an array (a list) of one or more creator-s.
In line 4 and line 5 we do the same trick to filter out the ISBN and ISSN number out of the record which we store in separate fields isbn and issn (indeed these are not Dublin Core fields, we will process them later).
In line 6 and line 7 we read the MARC-260 field which contains publisher and date information. Here we don’t need the $append trick because there is usually only one 260-field in a MARC record.
In line 8 the subjects are extracted from the 260-field using the same $append trick as above. Notice that we only extracted the $a subfields? If you want to add more subfields you can list them as in marc_map(650abcdefgh,subject.$append)
Given the dublin.txt file above we can execute the filtering command like this:
$ catmandu convert MARC --type ALEPHSEQ to YAML --fix dublin.fix < Documents/rug01.sample
As always you can type | less at the end of this command to slow down the screen output, or store the results into a file with > results.txt. Hint:
$ catmandu convert MARC --type ALEPHSEQ to YAML --fix dublin.fix < Documents/rug01.sample | less
$ catmandu convert MARC --type ALEPHSEQ to YAML --fix dublin.fix < Documents/rug01.sample > results.txt
The results should look like this:
- Katz, Jerrold J.
- '0855275103 :'
publisher: Harvester press,
- Proposition (Logic)
- Speech acts (Linguistics)
- Generative grammar.
- Competence and performance (Linguistics)
title: Propositional structure and illocutionary force :a study of the contribution of sentence meaning to speech acts /Jerrold J. Katz.
Congratulations, you’ve created your first mapping file to transform library data from MARC to Dublin Core! We need to add a bit more cleaning to delete some periods and commas here and there but as is we already have our first mapping.
Below you’ll find a complete example. You can read more about our Fix language online.
marc_map(245,title, -join => " ")
Continue to Day 16: Importing RDF data with Catmandu >>
In the last days you have learned how to store data with Catmandu. Storing data is a cool thing, but sharing data is awesome. Interoperability is important as other people may use your data (and you will profit from other people’s interoperable data)
In the day 13 tutorial we’ve learned the basic principle of metadata harvesting via OAI-PMH.
$ cpanm Dancer
$ cpanm Dancer::Plugin::Catmandu::OAI
and you also might need
$ cpanm Template
Let’s start and index some data with Elasticsearch as learned in the previous post:
$ catmandu import OAI --url http://pub.uni-bielefeld.de/oai --metadataPrefix oai_dc --from 2014-12-01 --handler oai_dc to Elasticsearch --index_name oai --bag publication
$ catmandu import OAI --url http://ds.ub.uni-bielefeld.de/viewer/oai --metadataPrefix oai_dc --from 2014-12-01T00:00:00Z --handler oai_dc to Elasticsearch --index_name oai --bag publication
After this, you should have some data in your Elasticsearch index. Run the following command to check this:
$ catmandu export Elasticsearch --index_name oai --bag publication
Everything is fine, so let’s create a simple webservice which exposes to collected data via OAI-PMH. The following code can be downloaded from this gist.
What’s going on here? Well, the script oai-app.pl defines a route /oai via the plugin Dancer::Plugin::Catmandu::OAI.
The template oai_dc.tt defines the xml output of the records. And finally the configuration file catmandu.yml handles the settings for the Dancer plugin as well as for the Elasticsearch indexing and querying.
Run the following command to start a local webserver
$ perl oai-app.pl
and point your browser to
https://localhost:3000/oai?verb=Identify. To get some records go to
Yes, it’s that easy. You can extend this simple example by adding fixes to transform the data as you need it.
Continue to Day 15: MARC to Dublin Core >>
The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a protocol to harvest metadata records from OAI compliant repositories. It was developed by the Open Archives Initiative as a low-barrier mechanism for repository interoperability. The Open Archives Initiative maintains a registry of OAI data providers.
Every OAI server must provide metadata records in Dublin Core, other (bibliographic) formats like MARC may be supported additionally. Available metadata formats can be detected with “ListMetadataFormats“. You can set the metadata format for the Catmandu OAI client via the --metadataPrefix parameter.
The OAI server may support selective harvesting, so OAI clients can get only subsets of records from a repository. The client requests could be limited via datestamps (--from, --until) or set membership (--set).
To get some Dublin Core records from the digital collection of the University of Michigan and convert them to JSON (default) run the following catmandu command:
$ catmandu convert OAI --url http://quod.lib.umich.edu/cgi/o/oai/oai --metadataPrefix oai_dc --from 2014-12-01T07:00:00Z --until 2014-12-01T07:04:00Z --handler oai_dc
You can also harvest MARC data and store it in a file:
$ catmandu convert OAI --url http://quod.lib.umich.edu/cgi/o/oai/oai --metadataPrefix marc21 --from 2014-12-01T07:00:00Z --until 2014-12-01T07:04:00Z --handler marcxml to MARC --type USMARC > umich.mrc
Instead of harvesting the whole metadata you can get the record identifiers (--listIdentifiers) only:
$ catmandu convert OAI --url http://quod.lib.umich.edu/cgi/o/oai/oai --from 2014-12-01T07:00:00Z --until 2014-12-01T07:04:00Z --listIdentifiers 1 to YAML
You can also transform incoming data and immediately store/index it with MongoDB or Elasticsearch. For the transformation you need to create a fix (see Day 6):
$ nano simple.fix
Add the following fixes to the file:
Now you can run an ETL process (extract, transform, load) with one command:
$ catmandu import OAI --url http://quod.lib.umich.edu/cgi/o/oai/oai --metadataPrefix marc21 --from 2014-12-01T07:00:00Z --until 2014-12-01T07:04:00Z --handler marcxml --fix simple.fix to Elasticsearch --index_name oai --bag umich
$ catmandu import OAI --url http://quod.lib.umich.edu/cgi/o/oai/oai --metadataPrefix marc21 --from 2014-12-01T07:00:00Z --until 2014-12-01T07:04:00Z --handler marcxml --fix simple.fix to MongoDB --database_name oai --bag umich
The Catmandu OAI client provides special handler (--handler) for Dublin Core (oai_dc) and MARC (marcxml). For other metadata formats use the default handler (raw) or implement your own. See documentation for further details.
Continue to Day 14: Set up your own OAI data service >>
ElasticSearch is a flexible and powerful open source, distributed, real-time search and analytics engine. You can store structured JSON documents and by default ElasticSearch will try to detect the data structure and index the data. ElasticSearch uses Lucene to provide full text search capabilities with a powerful query language. Install guides for various platforms are available at the ElasticSearch reference. To install the corresponding Catmandu module run:
$ cpanm Catmandu::Store::ElasticSearch
[For those of you running the Catmandu VirtualBox this installation is not required. ElasticSearch is by default installed]
Now get some JSON data to work with:
$ wget -O banned_books.json http://www.berlin.de/rubrik/hauptstadt/verbannte_buecher/verbannte-buecher.json
First index the data with ElasticSearch. You have to specify an index (–index_name) and type (–bag):
$ catmandu import -v JSON --multiline 1 to ElasticSearch --index_name books --bag banned < banned_books.json
Now you can export all items from an index to different formats, like XLSX, YAML and XML:
$ catmandu export ElasticSearch --index_name books --bag banned to YAML
$ catmandu export ElasticSearch --index_name books --bag banned to XML
$ catmandu export -v ElasticSearch --index_name books --bag banned to XLSX --file banned_books.xlsx
You can count all indexed items or those which match a query:
$ catmandu count ElasticSearch --index_name books --bag banned
$ catmandu count ElasticSearch --index_name books --bag banned --query 'firstEditionPublicationYear: "1937"'
$ catmandu count ElasticSearch --index_name books --bag banned --query 'firstEditionPublicationPlace: "Berlin"'
You can search an index for a specific value and export all matching items:
$ catmandu export ElasticSearch --index_name books --bag banned --query 'firstEditionPublicationYear: "1937"' to JSON
$ catmandu export ElasticSearch --index_name books --bag banned --query 'firstEditionPublicationPlace: "Berlin"' to CSV --fields '_id,authorFirstname,authorLastname,title,firstEditionPublicationPlace'
Collections and items can be moved within ElasticSearch or even to other stores or search engines:
$ catmandu move ElasticSearch --index_name books -b-ag banned --query 'firstEditionPublicationPlace: "Berlin"' to Elasticsearch --index_name books --bag berlin
$ catmandu move ElasticSearch --index_name books --bag banned to MongoDB --database_name books --bag banned
You can delete whole collections from a database or just items which match a query:
$ catmandu delete ElasticSearch --index_name books --bag banned --query 'firstEditionPublicationPlace: "Berlin"'
$ catmandu delete ElasticSearch --index_name books --bag banned
Continue to Day 13: Harvest data with OAI-PMH >>