Ever had a need for passing a lot of configuration parameters to a Catmandu Fix script or needing to write repetitive code over and over again in large scripts? There is a neat Bash trick you can use to preprocess your scripts.

For instance image you have a large JSON file that needs to be processed for many customers. To each record in the file you need to include the URL of their homepage:


add_field("homepage","http://library.edu")

You could do this my creating a Fix script for every customer and run the convert command for every customer:


$ catmandu convert --fix customer1.fix < data.json
$ catmandu convert --fix customer2.fix < data.json
$ catmandu convert --fix customer3.fix < data.json


There is another way to do this by using named pipe redirects in Bash. Instead of writing one Fix script for each customer you can write one Fix scipt for all customers that includes preprocessing handles:


add_field("homepage",HOMEPAGE)


With this script customer.fix you can use a preprocessor to populate the HOMEPAGE field:


$ catmandu convert --fix <(m4 -DHOMEPAGE=\"http://customer1.edu\" customer.fix) < data.json

$ catmandu convert --fix <(m4 -DHOMEPAGE=\"http://customer3.edu\" customer.fix) < data.json


Bash is creating a temporary named pipe that is given as input to the catmandu command while in the background a m4 processor is processing the customer.fix file.

You can enter any command into the named pipe redirects. There are plenty of interesting preprocessors available that can be used to process fix files such as: cpp, m4 and the even Template Toolkit tpage command).

By Patrick Hochstenbach

This is part one of a two part overview of extending the Catmandu Fix language. As you might know already, the Fix language is used in Catmandu to transform JSON data. Our rationale for is is to ease the manipulation of library data formats. Fix is to JSON data what XSLT is for XML.

If you have a JSON input file containing name/value pairs you can use fixes to add or delete fields:

$ echo '{ "name": "Patrick" }' | catmandu convert
{"name":"Patrick"}

$ echo '{ "name": "Patrick" }' | catmandu convert --fix 'add_field("age","42")'
{"age":42,"name":"Patrick"}

You can write fixes on the command line or in a file:

$ cat myfixes.txt
add_field("age","42");
add_field("my.favorite.color","blue");
$ echo '{ "name": "Patrick" }' | catmandu convert --fix myfixes.txt
{"name":"Patrick","my":{"favorite":{"color":"blue"}},"age":42}

Check out our Cheat Sheet for more examples of possible fixes.

Fix code can also be integrated into Perl scripts. We can repeat the experiments above with the Perl code below.

use Catmandu::Fix;
use DDP;

my $fixer = Catmandu::Fix->new(fixes => ['add_field("age","42")']);
my $hash  = $fixer->fix({ name => 'Patrick '});

p $hash;

# Read the fixes from a file
my $fixer = Catmandu::Fix->new(fixes => ['myfixes.txt']);
my $hash  = $fixer->fix({ name => 'Patrick '});

p $hash;

Starting from Catmandu version 0.8006 it is also possible to inline all the fixes.

use Catmandu::Fix::add_field as => 'add_field';
use DDP;

my $hash  = { name => 'Patrick ' };

add_field($hash,'age','42');
add_field($hash,'my.favorite.color','blue');

p $hash;

To extend the Fix language you need to create a new Perl package in the Catmandu::Fix namespace which implements a ‘fix’ method. This method gets as input a Perl hash and should return the changed Perl hash. Here is an trivial example of an ‘invert_field’ Fix which transforms a name/value-pair into a value/name pair:

package Catmandu::Fix::invert_field;

sub new {
    my ($class,$path) = @_;
    return bless { path => $path } , $class;
}

sub fix {
    my ($self,$data) = @_;
    my $name  = $self->{path};
    my $value = $data->{$name};

    delete $data->{$name};

    $data->{$value} = $name;

    $data;
}

1;

With this fix installed we can invert name/value pairs in our input data:

$ echo '{ "name": "Patrick" }' | catmandu convert --fix 'invert_field("name")'
 {"Patrick":"name"}

If you know exactly where in the Perl hash you need to make data changes, then this method of creating Fix functions is quite easy and straight forward. Things get complicated when you want to manipulate deeply nested hashes and arrays. For instance, almost all of the Catmandu provided fixes can manipulate very deeply nested structures:

add_field("my.very.deep.field.13.subfield.1.name","Patrick");

This ‘add_field’ fix above will operate on a very deep hash of arrays. To support this in your own created Fix functions you need to be able to parse JSON paths. In a next blog post we will go into more details of this process.

By Patrick Hochstenbach

Recently we had in our mailing list a question by Jakob Voss if Catmandu supports versioning of records. Indeed we do! In this post I’ll give an example what is possible using the Catmandu::Plugin modules developed by Nicolas Steenlant.

First, lets create a sample application in Perl to store some records in an in-memory hash for test purposes:

#!/usr/bin/env perl

use Catmandu::Store::Hash;
use Data::Dumper;

my $store = Catmandu::Store::Hash->new();

$store->bag->add({
    '_id'  => '001',
    'name' => 'John Doe'
});

$store->bag->add({
    '_id'  => '001',
    'name' => 'John Moo'
});

$store->each(sub { print Dumper($_[0]) });

First we store in a record with id ‘001’ a ‘John Doe’ and then we overwrite this record with ‘John Moo’. As result you should see only the version ‘John Moo’ printed on screen.

To add versioning we have to edit the initiation of the Hash and add a ‘Versioning’ plugin to the ‘data’ bag. As you may recall ‘data’ is the default name of the storage container in your Catmandu::Store. Using the get_history method all the versions of your records can be retrieved.

#!/usr/bin/env perl

use Catmandu::Store::Hash;
use Data::Dumper;

my $store = Catmandu::Store::Hash->new(
    bags => {
        'data' => {
            plugins => [qw(Versioning)]
        }
});

$store->bag->add({
    '_id'  => '001',
    'name' => 'John Doe'
});

$store->bag->add({
    '_id'  => '001',
    'name' => 'John Moo'
});

print "Versions:\n";

for (@{$store->bag->get_history('001')}) {
    print Dumper($_);
}

Or, you can get one particular version with the ‘get_version’ method:

my $obj = $store->bag->get_version('001',2);

Or, the previous version with ‘get_previous_version’:

my $obj = $store->bag->get_previous_version('001');

Sometimes you don’t want to create a new version records when nothing changed to the data except for some date field for instance. This can be set using the ‘version_compare_ignore’ field when creating a store:

my $store = Catmandu::Store::Hash->new(
        bags => {
            'data' => {
                plugins => [qw(Versioning)] ,
                version_compare_ignore => [qw(access_date)],
            }
});

Using this setting, new versions will only be created when any field except ‘access_date’ will change in value.

By Patrick Hochstenbach

With Catmandu we create ETL-pipelines for library workflows. Read data from OAI, SRU, Z39.50, PubMed, arXive, transform it with Catmandu Fixes and load the results into Solr, MongoDB, CouchDB or serialize into YAML, CSV, XML whatever you like. Read my blog post about the Catmandu Cheat Sheet to get a quick recap.

Today I want to show you how you can create your own Fix routines in any programming language using the Catmandu::Fix::cmd which Nicolas Steenlant created.

First we create a small Perl script to generate some sample JSON we will use in our examples (you can use your own JSON file or translate this trivial script into Python, Ruby, Java, C, Clojore, Go …).

Here is our little JSON generator:

#!/usr/bin/env perl
# file: generate.pl

use JSON;

for (1...1000) {
    print encode_json({ random => rand }) , "\n";
}

When we execute the script we’ll get one thousand lines of JSON in our terminal:

$ ./generate.pl
{"random":0.721613357218615}
{"random":0.491180438229559}
{"random":0.868290266595814}
.
.
.

It is now easy to use Catmandu Fixes to transform these JSON records. E.g. we can add a new field ‘title’ with content ‘test’:

$ ./generate.pl | catmandu convert JSON --fix 'add_field("title","test")'
{"random":0.611390470122803,"title":"test"}
{"random":0.915937067437753,"title":"test"}
{"random":0.461684127836374,"title":"test"}
.
.
.

This add_field() Fix was written in Perl. What if you need to write a new complicated Fix-routine and don’t want to use Perl? Well, we have Catmandu::Fix::Cmd to the rescue! You can create fixes in any language you like, as long as your program can read JSON records from the standard input and can write JSON records to the standard output you are cool. Lets try that out.

As example we create a Python script to read JSON from the stdin, add a title field and write the JSON back to stdout.

#!/usr/bin/env python
# file: catjson.py
import sys
import json

while 1:
    line = sys.stdin.readline()
    if not line: break
    data = json.loads(line.strip())
    data['title'] = "test";
    print json.dumps(data)

If we run this we can see the expected result.

$ ./generate.pl | ./catjson.py
{"random": 0.530965947974309, "title": "test"}
{"random": 0.371021223752646, "title": "test"}
{"random": 0.0907161737840951, "title": "test"}
.
.
.

With the Catmandu Fix ‘cmd’ we can make this Python program part of an ETL-pipeline. In the simple example below we will repeat the previous test:

$ ./generate.pl | catmandu convert JSON --fix 'cmd("./catjson.py")'
{"random":0.554686750713572,"title":"test"}
{"random":0.275637603863029,"title":"test"}
{"random":0.318374223918873,"title":"test"}
.
.
.

Now this is working you can add the whole Catmandu stack to this pipeline. Add different importers, new fixes, store into ElasticSearch or MongoDB. E.g. we can do an SRU query and use our Python and Perl fixes simultaneously:

$ catmandu convert SRU --base http://www.unicat.be/sru --query dna --fix 'cmd("./catjson.py");remove_field("recordData")'
{"recordPacking":"xml","recordPosition":"1","title":"test","recordSchema":"info:srw/schema/1/dc-schema"}
{"recordPacking":"xml","recordPosition":"2","title":"test","recordSchema":"info:srw/schema/1/dc-schema"}
{"recordPacking":"xml","recordPosition":"3","title":"test","recordSchema":"info:srw/schema/1/dc-schema"}
.
.
.

Here is how the same program might look like in Lua

#!/usr/bin/env luajit
# file: catjson.lua
-- requires dkjson http://chiselapp.com/user/dhkolf/repository/dkjson/home
local json = require ("dkjson")

for line in io.lines() do
    local obj, pos, err = json.decode (line, 1, nil)
    obj['title'] = 'test'
    print(json.encode(obj))
end

With the same expected results:

$ ./generate.pl | catmandu convert JSON --fix 'cmd("./catjson.lua")'
{"random":0.54868770433573,"title":"test"}
{"random":0.26483418097243,"title":"test"}
{"random":0.15708750198151,"title":"test"}
.
.
.

Using Catmandu::Fix::cmd you can create complicated fix routines to extend your data crunching needs.