Rake Routes

by Stephen Ball

Programming with jq

So jq is awesome. When you need to process JSON then jq is your factory line transforming the raw JSON ingredients into delicious ice cream sandwiches.

A machine rapidly producing ice cream sandwichesA machine rapidly producing ice cream sandwiches

But jq is even more awesome than that! We can actually program using jq! Think of it like AWK. AWK is mostly thought of for easily parsing out fields of data but you can write programs with it as well.

Here’s AWK the data parser

$ echo "key1 value1\nkey2 value2\nkey3 value3"
key1 value1
key2 value2
key3 value3

$ echo "key1 value1\nkey2 value2\nkey3 value3" | awk '{ print $2 }'

And AWK with some programming in it

$ awk 'function add(a, b) { return a + b } BEGIN { print add(3,4) }'

Like AWK, jq can do more than it might seem at first use. One of the keys is that jq can take a -n argument which tells it not to even expect any input.

$ jq -n "[1, 2, 3]"

$ jq -n "[11, 22, 33] | .[1]"

Now throw in that jq allows you to define functions, has pretty solid standard library of functions, and control flow operators and BAM we have some programming going!

Let’s knock it out the park right away, 99 bottles in jq?! YES! 99 Bottles of Beer in jq

def sing:
  def s: if . == 1 then "" else "s" end;
  def bottles:
    if . == 0 then "No more"
    else "\(.)"
    end + " bottle\(s)";
  (. - range(0;.+1) )
  | "
\(bottles) of beer on the wall
\(bottles) of beer
Take one down, pass it around
\(bottles) of beer on the wall."

$bottles | tonumber | sing

Seriously. The trick is that to run this script we have to tell jq a few things.

We need to pass -n to tell jq not to expect input. We need to pass -r to tell it to not pretty format the output into JSON. We need to pass it an --arg to give the $bottles variable an initial value. And we need to pass it -f with the script’s filename.

$ jq -r -n --arg bottles 3 -f 99_bottles.jq

3 bottles of beer on the wall
3 bottles of beer
Take one down, pass it around
3 bottles of beer on the wall.

2 bottles of beer on the wall
2 bottles of beer
Take one down, pass it around
2 bottles of beer on the wall.

1 bottle of beer on the wall
1 bottle of beer
Take one down, pass it around
1 bottle of beer on the wall.

No more bottles of beer on the wall
No more bottles of beer
Take one down, pass it around
No more bottles of beer on the wall.

Pretty wild eh? I for one did not expect jq to have such power. But it does!

Let’s dive through more of Rosetta Code’s jq category. How about some Prime decomposition?

def factors:
  . as $in
  | [2, $in, false]
  | recurse( .[0] as $p |.[1] as $q | .[2] as $valid | .[3] as $s
             | if $q == 1        then empty
           elif $q % $p == 0 then [$p, $q/$p, true]
               elif $p == 2      then [3, $q, false, $s]
             ($s // ($q | sqrt)) as $s
             | if $p + 2 <= $s then [$p + 2, $q, false, $s]
                   else [$q, 1, true]
           end )
   | if .[2] then .[0] else empty end ;
$ jq -r -n 'include "factors"; 12 | factors'

$ jq -r -n 'include "factors"; 25 | factors'

$ jq -r -n 'include "factors"; 65535 | factors'

$ jq -r -n 'include "factors"; 7783840458009841 | factors'

Pretty great eh? Check out how factors.jq is a module that jq scripts can include (or import) directly. And jq’s performance isn’t bad! That huge factor at the end there takes about two seconds on my MacBook Air.

How about a fun word exercise? Phrase reversals in jq

def reverse_string: explode | reverse | implode;

"jq does more than you might think"
| split(" ") as $words
| "0. input:               \(.)",
  "1. string reversed:     \(reverse_string)",
  "2. each word reversed:  \($words | map(reverse_string) | join(" "))",
  "3. word-order reversed: \($words | reverse | join(" "))"
$ jq -r -n -f phrase-reversals.jq
0. input:               jq does more than you might think
1. string reversed:     kniht thgim uoy naht erom seod qj
2. each word reversed:  qj seod erom naht uoy thgim kniht
3. word-order reversed: think might you than more does jq

Pretty cool! jq is an amazing programming tool that does even more than parse JSON! The next time you think you need to combine jq with another programming language to build up whatever you’re trying to parse from some JSON you might able to do it all in jq itself in one call.

Shout out to Rosetta Code for being an awesome resource as always.

Up next Command line tools - jq Need to parse some JSON data? Of course you do! JSON is practically the lifeblood of API calls these days. GitHub Google Cloud Storage Lots of Logical Solver: Turn facts into conclusions
Latest posts A simple language spec isn’t a feature when you’re building applications The Fastest Possible Tests Shrink your data into bitfields (and out again) Every “if” statement is an object waiting to be extracted Choose Generic Tools Hyperlinks you might find interesting — #4 Running bundle install on rails master Use tldr for command line examples Friday Lunch Links — #3 Friday Lunch Links — #2 Logical Solver: Turn facts into conclusions Programming with jq Command line tools - jq Friday Lunch Links — #1 Why diversity matters Music for coding - October 2019 Code puzzles are a poor way to gauge technical candidates Add vim to a pipeline with vipe Connecting Objects with Observable Let’s write a shell script What’s a $PATH anyway? Let’s Use Hwacha to Scan URLs Deliberate Git Customize Your IRB Program Like a Videogamer Gem Spotlight: interactive_editor Things Most Interviewees Fail to Discover Rails isn’t for beginners How to use bundler instead of rvm gemsets How to write (and test) a gem to serve static files on the Rails asset pipeline A Taste of Metaprogramming