Day 8: Processing JSON data from webservices

08_librecatDuring the last two days we got an introduction into Catmandu and learned how to transform structured JSON data. The JSON data in these examples was first fetched from a an URL with command curl. Today we will learn how to simplify fetching more data from web services.

In short, a web service is a server that can be queried by HTTP requests. Most web services return JSON data if queried with an URL. For instance the weather web service used during the last two days is documented at openweathermap.org/api. To retrieve current weather data from selected cities, we used commands and URLs like this:


$ curl http://api.openweathermap.org/data/2.5/weather?q=Gent,be
$ curl http://api.openweathermap.org/data/2.5/weather?q=Tokyo,jp

The URLs only differ in its query parameter q, so we can construct a so called URL template. The form of an URL template is defined in RFC 6570, so our template is:

http://api.openweathermap.org/data/2.5/weather?q

Catmandu supports URL templates to retrieve JSON data with its getJSON Importer. Let’s use it to fetch weather data for Toyko:

$ echo '{"q":"Tokyo,jp"}' | catmandu convert getJSON --url 'http://api.openweathermap.org/data/2.5/weather{?q}'

URL templates make most sense, if applied with multiple values, so let’s create a list of cities. We could use a text editor, such as learned at day 5 but here is an alternative way to learn something new:

$ echo q > cities.csv
$ echo Ghent,be >> cities.csv
$ echo Tokyo,jp >> cities.csv
$ echo Berlin,de >> cities.csv
$ catmandu convert CSV --sep_char _ to JSON < cities.csv > cities.json

We first created the CSV file cities.csv by appending one line after another. The > character is used to pipe output to a file and >> can be used to append to a file instead of overwriting it. You will learn more about processing CSV files in a later article. The last command converts the CSV file to line-separated JSON. Have a look at both files with cat:

$ cat cities.csv
q
Ghent,be
Tokyo,jp
Berlin,de

$ cat cities.json
{"q":"Ghent,be"}
{"q":"Tokyo,jp"}
{"q":"Berlin,de"}

Now we can finally use this list of cities to retrieve weather data in one call:

$ cat cities.json | catmandu convert getJSON --url 'http://api.openweathermap.org/data/2.5/weather{?q}'

Try to append to YAML or to JSON --pretty 1 to this command to get a better view of the data, as described in introduction into catmandu (day 6)!

To better see what’s going on we can skip retrieving data and just get the full URLs instead. This is done by setting the option --dry to 1:

$ catmandu convert getJSON --dry 1 --url 'http://api.openweathermap.org/data/2.5/weather{?q}' < cities.json

With the knowledge from previous days we can extract some information. Here is an improved fix to get both name, and temperature:

retain_field(main.temp)
move_field(name,main.name)
retain_field(main)

Save this fix as file weather2.fix and get temperate of cities of your choice:

$ cat cities.json | catmandu convert getJSON --url 'http://api.openweathermap.org/data/2.5/weather{?q}' --fix weather2.fix

The getJSON Importer get be used to retrieve JSON data from various web services. Catmandu further includes specialized importers for selected web services, for instance:

Continue to Day 9: Processing MARC with Catmandu >>

Advertisements

6 comments

  1. Joh

    Hmm, running:
    echo ‘{“q”:”Tokyo,jp”}’ | catmandu convert getJSON –url ‘http://api.openweathermap.org/data/2.5/weather{?q}’
    results in:
    “Oops! Can not find the importer ‘getJSON’ in your configuration file or Catmandu::Importer::getJSON is not installed.”

    I tried to install it with:
    cpanm Catmandu::Importer::getJSON

    but
    Building and testing Catmandu-Importer-getJSON-0.41 … FAIL

    Also, It does not work after –force install

    I’m on Catmandu ver (0.9209)
    Any hints what to do now?

    Like

  2. Pingback: Day 7: Catmandu JSON paths | LibreCat
    • Király Péter

      The API does no longer work without API Key, so you should provide an API key via the APPID paramterer.

      for example:
      export APPID=your API key
      echo “q;APPID” > cities.csv
      echo “Ghent,be;$APPID” >> cities.csv
      echo “Tokyo,jp;$APPID” >> cities.csv
      echo “Berlin,de;$APPID” >> cities.csv
      catmandu convert CSV –sep_char “;” to JSON cities.json

      then
      cat cities.json | catmandu convert getJSON –url ‘http://api.openweathermap.org/data/2.5/weather{?q,APPID}’

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s