Preprocessing Catmandu fixes

May 20, 2015

Ever had a need for passing a lot of configuration parameters to a Catmandu Fix script or needing to write repetitive code over and over again in large scripts? There is a neat Bash trick you can use to preprocess your scripts.

For instance image you have a large JSON file that needs to be processed for many customers. To each record in the file you need to include the URL of their homepage:


add_field("homepage","http://library.edu")

You could do this my creating a Fix script for every customer and run the convert command for every customer:


$ catmandu convert --fix customer1.fix < data.json
$ catmandu convert --fix customer2.fix < data.json
$ catmandu convert --fix customer3.fix < data.json


There is another way to do this by using named pipe redirects in Bash. Instead of writing one Fix script for each customer you can write one Fix scipt for all customers that includes preprocessing handles:


add_field("homepage",HOMEPAGE)


With this script customer.fix you can use a preprocessor to populate the HOMEPAGE field:


$ catmandu convert --fix <(m4 -DHOMEPAGE=\"http://customer1.edu\" customer.fix) < data.json

$ catmandu convert --fix <(m4 -DHOMEPAGE=\"http://customer2.edu\" customer.fix) < data.json

$ catmandu convert --fix <(m4 -DHOMEPAGE=\"http://customer3.edu\" customer.fix) < data.json


Bash is creating a temporary named pipe that is given as input to the catmandu command while in the background a m4 processor is processing the customer.fix file.

You can enter any command into the named pipe redirects. There are plenty of interesting preprocessors available that can be used to process fix files such as: cpp, m4 and the even Template Toolkit tpage command).

Written by hochstenbach Posted in Intermediate