|||

Rake Routes

Need to parse some JSON data? Of course you do! JSON is practically the lifeblood of API calls these days.

So great. What if you want to use that data? If you want to use it quickly, easily, and efficiently? From the command line?

Do you need to write your own bespoke JSON parsers in your programming language of choice? No! jq exists!

jq reads JSON and accepts a format that it will apply to the JSON data.

An example is worth more than more words here.

$ curl -s https://api.github.com/users/sdball | jq '.'
{
  "login": "sdball",
  "id": 45837,
  "node_id": "MDQ6VXNlcjQ1ODM3",
  "avatar_url": "https://avatars3.githubusercontent.com/u/45837?v=4",
  "gravatar_id": "",
  "url": "https://api.github.com/users/sdball",
  "html_url": "https://github.com/sdball",
  "followers_url": "https://api.github.com/users/sdball/followers",
  "following_url": "https://api.github.com/users/sdball/following{/other_user}",
  .
  .
  .

Here we’re pulling the JSON data from https:/[/api.github.com/users/sdball](//api.github.com/users/sdball) and sending it straight through to jq and a "." format. That . there is jq’s identity function. The identity function returns its input unchanged. Hooray! And, bonus, by default jq pretty-prints its output so as a nice side effect jq outputs nicely pretty printed JSON. Let’s see that with a smaller piece of JSON real quick.

$ echo "[1,2,3,4,5]" | jq '.'
[
  1,
  2,
  3,
  4,
  5,
  6
]

Nice.

From here we can ask jq to do ALL KINDS of awesome data extractions and transforms! More examples!

What if we want to get the first three elements of an array? The .[] syntax is how jq parses array values.

$ echo "[1,2,3,4,5,6]" | jq '.[0:3]'
[
  1,
  2,
  3
]

Or pull some key information from the GitHub user API data? In this example we use the {} syntax to tell jq to trim the input object down to specific keys.

$ curl -s https://api.github.com/users/sdball | jq '{ created_at, followers, following, blog }'
{
  "created_at": "2009-01-11T22:44:09Z",
  "followers": 75,
  "following": 29,
  "blog": "http://rakeroutes.com"
}

We can get really fancy!

$ curl -s https://api.github.com/users/sdball | jq -r '. |
  to_entries |
  .[] |
  select(.value) |
  select(.key | endswith("url")) |
  "\(.key) --- \(.value)"'

avatar_url --- https://avatars3.githubusercontent.com/u/45837?v=4
url --- https://api.github.com/users/sdball
html_url --- https://github.com/sdball
followers_url --- https://api.github.com/users/sdball/followers
following_url --- https://api.github.com/users/sdball/following{/other_user}
gists_url --- https://api.github.com/users/sdball/gists{/gist_id}
starred_url --- https://api.github.com/users/sdball/starred{/owner}{/repo}
subscriptions_url --- https://api.github.com/users/sdball/subscriptions
organizations_url --- https://api.github.com/users/sdball/orgs
repos_url --- https://api.github.com/users/sdball/repos
events_url --- https://api.github.com/users/sdball/events{/privacy}
received_events_url --- https://api.github.com/users/sdball/received_events

There’s a lot to that one, let’s unpack it.

jq -r '. |
  to_entries |
  .[] |
  select(.value) |
  select(.key | endswith("url")) |
  "\(.key) --- \(.value)"'
  1. . The identity function
  2. to_entries This function transforms objects into key/value pairs. Like so

    $ echo '{"a": 1, "b": 2}' | jq '. | to_entries'
    [
      {
        "key": "a",
        "value": 1
      },
      {
        "key": "b",
        "value": 2
      }
    ]
  3. .[] unpacks the array
  4. select allows us to filter, e.g. select(.value) filters out any elements with a null .value
  5. Another select filter out any elements that don’t have a key ending in url
  6. And finally the "\(.key) --- \(.value)" tells jq to format the output

Fun times! jq is an extremely powerful tool for parsing and transforming JSON. Add it to your toolbox!

Up next Friday Lunch Links — #1 Hello! Here’s a collection of interesting links I’ve found in the last week. Enjoy! The Little Handbook of Statistical Practice by Gerard E. Dallal, Programming with jq
Latest posts Shrink your data into bitfields (and out again) Every “if” statement is an object waiting to be extracted Choose Generic Tools Hyperlinks you might find interesting — #4 Running bundle install on rails master Use tldr for command line examples Friday Lunch Links — #3 Friday Lunch Links — #2 Logical Solver: Turn facts into conclusions Programming with jq Command line tools - jq Friday Lunch Links — #1 Why diversity matters Music for coding - October 2019 Code puzzles are a poor way to gauge technical candidates Add vim to a pipeline with vipe Connecting Objects with Observable Let’s write a shell script What’s a $PATH anyway? Let’s Use Hwacha to Scan URLs Deliberate Git Customize Your IRB Program Like a Videogamer Gem Spotlight: interactive_editor Things Most Interviewees Fail to Discover Rails isn’t for beginners How to use bundler instead of rvm gemsets How to write (and test) a gem to serve static files on the Rails asset pipeline A Taste of Metaprogramming Fun with Rock, Paper, Scissors Let’s Write a Gem: Part 2