json

• basic

  • syntax for jq

  • dealing with json objects

  • slicing and filtering

  • mapping and transforming

• join

  • join with getOrDefault

• split

• replacing

• builtin operators

  • debug

  • select

  • contains

  • inside

  • toUpperCase : ascii_upcase

  • to_entries

  • try to_entries

  • from_entries

  • with_entries

  • as

• tricky

  • space in the key

  • get urlencode

  • string to json

  • map

json online player:

• Online JSON Formatter & Validator

jq online player:

• https://jqplay.org

reference:

• jq manual (development version)
• Parsing JSON with jq
• olih/jq-cheetsheet.md
• Use jq to filter objects list with regex
• jq: Cannot iterate over number / string and number cannot be added
• Guide to Linux jq Command for JSON Processing
• Reshaping JSON with jq
• * jq cheat sheet

  • Replacing a missing or null property
  • Replacing substrings in a string

basic

syntax for jq

SYNTAX	DESCRIPTION
,	Filters separated by a comma will produce multiple independent outputs
?	Will ignores error if the type is unexpected
[]	Array construction
{}	Object construction
+	Concatenate or Add
-	Difference of sets or Substract
length	Size of selected element
⎮	Pipes are used to chain commands in a similar fashion than bash

dealing with json objects

DESCRIPTION	COMMAND
Display all keys	jq 'keys'
Adds + 1 to all items	jq 'map_values(.+1)'
Delete a key	jq 'del(.foo)'
Convert an object to array	to_entries ⎮ map([.key, .value])

slicing and filtering

[!NOTE|label:references:]

• select by type: with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

DESCRIPTION	COMMAND
All	jq .[]
First	jq '.[0]'
Range	jq '.[2:4]'
First 3	jq '.[:3]'
Last 2	jq '.[-2:]'
Before Last	jq '.[-2]'
split	jq '.[] ⎮ split("/")[1]'
Select array of int by value	jq 'map(select(. >= 2))'
Select array of objects by value	jq '.[] ⎮ select(.id == "second")'
Select by type	jq '.[] ⎮ numbers'

mapping and transforming

DESCRIPTION	COMMAND
Add + 1 to all items	jq 'map(.+1)'
Delete 2 items	jq 'del(.[1, 2])'
Concatenate arrays	jq 'add'
Flatten an array	jq 'flatten'
Create a range of numbers	jq '[range(2;4)]'
Display the type of each item	jq 'map(type)'
Sort an array of basic type	jq 'sort'
Sort an array of objects	jq 'sort_by(.foo)'
Group by a key - opposite to flatten	jq 'group_by(.foo)'
Minimun value of an array	jq 'min' See also min, max, min_by(path_exp), max_by(path_exp)
Remove duplicates	jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)'
Reverse an array	jq 'reverse'

join

[!TIP] references:

• to output multiple values on a single line
• join multiple values

$ echo '{ "some": "thing", "json": "like" }' |
       jq -r '[.some, .json] | join(":")'
thing:like

• or

  $ echo '{ "some": "thing", "json": "like" }' |
         jq -r '[.some, .json] | "\(.[0]) \(.[1])"'
  thing like

• or

  $ echo '{ "some": "thing", "json": "like" }' |
         jq -r '[.some, .json] | "\(.[0]): \(.[1])"'
  thing: like

• or with reduce

  $ echo '{ "some": "thing", "json": "like" }' |
         jq -r '[.some, .json] | reduce .[1:][] as $i ("\(.[0])"; . + ",\($i)")'
  thing,like

• or

  $ echo '{ "k1": "v1", "k2": "v2", "k3": "v3", "k4": "v4" }' |
         jq -r '[.k1, .k2, .k3] | reduce .[1:][] as $i ("\(.[0])"; . + ",\($i)")'
  v1,v2,v3

• or with .first and .last

  $ echo '{ "users": [ { "first": "Stevie", "last": "Wonder" }, { "first": "Michael", "last": "Jackson" } ] }' |
         jq -r '.users[] | .first + " " + .last'
  Stevie Wonder
  Michael Jackson

• or

  $ echo '{ "users": [ { "first": "Stevie", "last": "Wonder" }, { "first": "Michael", "last": "Jackson" } ] }' |
         jq -r '.users[] | .first + " " + (.last|tostring)'

• or

  $ echo '{ "users": [ { "first": "Stevie", "last": "Wonder", "number": 1 }, { "first": "Michael", "last": "Jackson", "number": 2 } ] }' |
         jq -r '.users[] | .first + " " + (.number|tostring)'
  Stevie 1
  Michael 2

join with getOrDefault

[!TIP|label:references:]

• orignal wihtout getOrDefault

  $ echo '[
          {"id": "1", "version": "v1"},
          {"id": "2", "version": "v2"},
          {"id": "3", "version": null}
          ]
         ' |
    jq -r '.[] | .id + " -> " + .version'
  1 -> v1
  2 -> v2
  3 ->

• using //

  [!TIP|label:references:]

  • Get or default function in JQ?

  $ echo '[
            {"id": "1", "version": "v1"},
            {"id": "2", "version": "v2"},
            {"id": "3", "version": null}
          ]' |
    jq -r '.[] | .id + " -> " + ( .version // "UNKNOWN" )'
  1 -> v1
  2 -> v2
  3 -> UNKNOWN

• using if then else end

  [!TIP|label:references:]

  • Filter empty and/or null values with jq
  • Get or default function in JQ?

  $ echo '[
            { "id": "1", "version": "v1" } ,
            { "id": "2", "version": "v2" } ,
            { "id": "3", "version": null } ,
            { "id": "4", "version": ""   }
          ]' |
    jq -r '.[] |
           .id + " -> " +
           if (.version != null and .version != "") then .version
           else "UNKNOWN"
           end'
  
  # or
  $ echo '[
            { "id": "1", "version": "v1" } ,
            { "id": "2", "version": "v2" } ,
            { "id": "3", "version": null } ,
            { "id": "4", "version": ""   }
          ]' |
    jq -r '.[] |
           .id + " -> " +
           if .version == null or .version == "" then "UNKNOWN"
           else .version
           end'
  1 -> v1
  2 -> v2
  3 -> UNKNOWN
  4 -> UNKNOWN
  
  # or using `empty`
  #                           `empty` will not output anything for id==3 and id==4
  #                                                     v
  $ echo '[
            { "id": "1", "version": "v1" } ,
            { "id": "2", "version": "v2" } ,
            { "id": "3", "version": null } ,
            { "id": "4", "version": ""   }
          ]
         ' |
    jq -r '.[] |
           .id + " -> " +
           if .version == null or .version == "" then empty
           else .version
           end
          '
  1 -> v1
  2 -> v2
  // `empty` will not output anything for id==3 and id==4
  
  # or including non-exists key
  $ echo '[
            { "id": "1", "version": "v1" } ,
            { "id": "2", "version": "v2" } ,
            { "id": "3", "version": null } ,
            { "id": "4", "version": ""   } ,
            { "id": "5"                  }
          ]
         ' |
    jq -r '.[] |
           .id + " -> " +
           if ( has("version") and .version != null and .version != "" ) then
             .version
           else "UNKNOWN"
           end
          '
  1 -> v1
  2 -> v2
  3 -> UNKNOWN
  4 -> UNKNOWN
  5 -> UNKNOWN

split

[!TIP] references:

• remove a substring from a string
• split(str)

  • Splits an input string on the separator argument.

  • example

    jq 'split(", ")'
    Input "a, b,c,d, e, "
    Output  ["a","b,c,d","e",""]

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri'
/1
/2
/3

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri | split("/")[1]'
1
2
3

• try online

  $ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
          jq '.[].uri | split("/")[]'
  ""
  "1"
  ""
  "2"
  ""
  "3"
  
  $ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
          jq -r '.[].uri | split("/")'
  [
    "",
    "1"
  ]
  [
    "",
    "2"
  ]
  [
    "",
    "3"
  ]

replacing

[!TIP|label:references:]

• Replacing substrings in a string
• sub(regex; tostring), sub(regex; string; flags)

  • Emit the string obtained by replacing the first match of regex in the input string with tostring, after interpolation. tostring should be a jq string, and may contain references to named captures. The named captures are, in effect, presented as a JSON object (as constructed by capture) to tostring, so a reference to a captured variable named "x" would take the form: "(.x)".

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri'
/1
/2
/3

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri | sub("/"; "")'
1
2
3

builtin operators

[!NOTE|label:references:]

• jq manual - Builtin operators and functions

debug

• without debug

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
         jq '.[] | select( .val == (2, 3) )'

• with debug:

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
         jq '.[] | select( .val == (2, 3) | debug )'
  ["DEBUG:",false]
  ["DEBUG:",false]
  ["DEBUG:",true]
  {
    "id": "second",
    "val": 2
  }
  ["DEBUG:",false]
  ["DEBUG:",false]
  ["DEBUG:",true]
  {
    "id": "SECOND",
    "val": 3
  }

select

[!NOTE|label:refrences:]

• imarslo: example on jenkins api analysis
• imarslo: example on gerrit api analysis
• jq tips : remove emtpy line
• Remove all null values
• Filter empty and/or null values with jq

  to_entries[]
  | select(.value | . == null or . == "")
  | if .value == "" then .value |= "\"\(.)\"" else . end
  | "\(.key): \(.value)"

$ echo '[1,5,3,0,7]' |
       jq 'map(select(. >= 2))'
[
  5,
  3,
  7
]

• or

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2}]''' |
         jq '.[] | select(.id == "second")'
  {
    "id": "second",
    "val": 2
  }

• select with any

  [!NOTE|label:references:]

  • any(.attributes; .enabled == true) == any(.attributes; .enabled) == .attributes.enabled == true == .attributes.enabled
  • try online

  // sample.json
  [
    {
      "attributes": {
        "created": "2021-10-18T12:02:39+00:00",
        "enabled": true,
        "expires": null,
        "notBefore": null
      },
      "contentType": null,
      "id": "https://kjkljk./secrets/-/1",
      "managed": null,
      "name": "pw",
      "tags": {}
    },
    {
      "attributes": {
        "created": "2021-10-18T12:06:16+00:00",
        "enabled": false,
        "expires": null,
        "notBefore": null
      },
      "contentType": "",
      "id": "https://kjklj./secrets/-/2",
      "managed": null,
      "name": "pw",
      "tags": {}
    }
  ]

  # get [{.id}] format
  $ cat sample.json |
        jq -r 'map(select(any(.attributes; .enabled)) | {id})'
  [
    {
      "id": "https://kjkljk./secrets/-/1"
    }
  ]
  
  # get [.id] format
  $ cat sample.json |
        jq -r 'map(select(any(.attributes; .enabled)) | .id)'
  [
    "https://kjkljk./secrets/-/1"
  ]
  
  # get .id format ( without `map()` )
  $ cat sample.json |
        jq -r '.[] | select(any(.attributes; .enabled)) | .id'
  https://kjkljk./secrets/-/1
  
  # re-format
  $ cat sample.json |
        jq -r '{"ids": .[] | select(any(.attributes; .enabled)) | .id}'
  {
    "ids": "https://kjkljk./secrets/-/1"
  }

contains

[!NOTE|label:reference:]

• imarslo: example on artifactory api analysis
• imarslo: example on list Error pods in kuberetnes

$ echo '[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "second-one", "val": 3}]' |
       jq '.[] | select( .id | contains("second") )'
{
  "id": "second",
  "val": 2
}
{
  "id": "second-one",
  "val": 3
}

• by using test

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "second.one", "val": 3}]'''  |
         jq '.[] | select( .id | test("sec*") )'
  {
    "id": "second",
    "val": 2
  }
  {
    "id": "second.one",
    "val": 3
  }

inside

$ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "second-one", "val": 3}]'''  |
       jq '.[] | select( [.val] | inside([2,3]) )'
{
  "id": "second",
  "val": 2
}
{
  "id": "second-one",
  "val": 3
}

• or

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
         jq '.[] | select( .val == (2, 3) )'
  {
    "id": "second",
    "val": 2
  }
  {
    "id": "SECOND",
    "val": 3
  }

toUpperCase : ascii_upcase

$ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
       jq '.[] | select(.id | ascii_upcase == "SECOND")'
{
  "id": "second",
  "val": 2
}
{
  "id": "SECOND",
  "val": 3
}

to_entries

[!NOTE] references:

• jq: filter input based on if key ends with specified string

# original
$ echo '{ "name" : "marslo" }' | jq -r
{
  "name": "marslo"
}

# `to_entries[]`
$ echo '{ "name" : "marslo" }' | jq -r 'to_entries[]'
{
  "key": "name",
  "value": "marslo"
}

# or `to_entries`
$ echo '{"a": 1, "b": 2}' | jq -r to_entries
[
  {
    "key": "a",
    "value": 1
  },
  {
    "key": "b",
    "value": 2
  }
]

• example to get key and value

  $ echo '{ "some": "thing", "json": "like" }' |
         jq -r 'to_entries[] | "\(.key)\t\(.value)"'
  some    thing
  json    like

  • or output format to @csv

    $ echo '{ "some": "thing", "json": "like" }' |
           jq -r '[.some, .json] | @csv'
    "thing","like"

  • or output format to @tsv

    $ echo '{ "some": "thing", "json": "like" }' |
           jq -r '[.some, .json] | @tsv'
    thing like

• to_entries and select

  # orignal
  $ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' | jq -r
  {
    "name/": "marslo",                 # wants value if key ends with '/'
    "age/": "18",                      # wants value if key ends with '/'
    "citizenship": "china"
  }
  
  # select
  $ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' |
         jq -r 'to_entries[] | select(.key|endswith("/")) '
  {
    "key": "name/",
    "value": "marslo"
  }
  {
    "key": "age/",
    "value": "18"
  }
  
  # get `.value` after selected
  $ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' |
         jq -r 'to_entries[] | select(.key|endswith("/")) | .value'
  marslo
  18

try to_entries

// sample.json
[
  {
    "name": "x",
    "hobby": [                     // < has hobby
      "baseball",
      "baseketball"
     ]
  },
  {
    "name": "y"                    // < no hobby
  }
]

# to_entries[] directly will cause issue
$ cat sample.json |
      jq '.[] | .name as $n | .hobby | to_entries[] | [$n, .value]'
[
  "x",
  "baseball"
]
[
  "x",
  "baseketball"
]
jq: error (at <stdin>:12): null (null) has no keys

# try to_entries[]
#                                      try
#                                       v
$ cat sample.json |
      jq '.[] | .name as $n | .hobby | try to_entries[] | [$n, .value]'
[
  "x",
  "baseball"
]
[
  "x",
  "baseketball"
]

from_entries

$ echo '[{"key":"a", "value":1}, {"key":"b", "value":2}]' |
       jq -r from_entries
{
  "a": 1,
  "b": 2
}

with_entries

[!NOTE|label:references:]

• JQ: Filtering for keys

$ echo '{"a": 1, "b": 2}' |
       jq 'with_entries(.key |= "KEY_" + .)'
{
  "KEY_a": 1,
  "KEY_b": 2
}

• to get particular branch permissions from Gerrit

  # with_entries
  $ curl -fsSL https://gerrit.domain.com/a/projects/$(printf %s "path/to/sandbox" | jq -sRr @uri)/access |
         tail -n+2 |
         jq -r '.local | with_entries( if(.key | test("^.+sandbox/marslo.+$")) then ( {key: .key, value: .value } ) else empty end )'
  
  # try to_entries[]
  $ curl -fsSL https://gerrit.domain.com/a/projects/$(printf %s "path/to/sandbox" | jq -sRr @uri)/access |
         tail -n+2 |
         jq -r '.local | try to_entries[] | select(.key | test("^.+sandbox/marslo.+$")) | .value'

as

[!NOTE|label:references]

• jq: filter input based on if key ends with specified string

$ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' |
       jq -r '. as $o | keys_unsorted[] | select(endswith("/")) | $o[.]'
marslo
18

tricky

space in the key

$ echo '{ "k1 name": "v1", "k2 name": "v2", "k3": "v3", "k4": "v4" }' |
       jq -r '."k1 name"'
v1

get urlencode

[!TIP|label:references:]

• Percent-encoding

$ printf %s 'input text' | jq -sRr @uri
input%20text

$ printf %s 'input=(text)' | jq -sRr @uri
input%3D%28text%29

$ printf %s '(=)&' | jq -sRr @uri
%28%3D%29%26

string to json

$ echo [
  "foo1:         xxxxxx",
  "foo2:    xxxxxx",
  "foo3:     xxxxxx",
  "foo4:         xxxxxx",
  "foo5:   xxxxxx",
  "foo6:       xxxxxx"
] | jq -r 'map(capture("^(?<key>.*):\\s*(?<value>.*)$")) | from_entries'
{
  "foo1": "xxxxxx",
  "foo2": "xxxxxx",
  "foo3": "xxxxxx",
  "foo4": "xxxxxx",
  "foo5": "xxxxxx",
  "foo6": "xxxxxx"
}

map

[!NOTE|label:references:]

• semantics of map on a sequence of objects in jq

  • map( ... ) ≡ [ .[] | ... ], which map( . ) ≡ [ .[] | . ] ≡ [ .[] ]

  • for array: map( . ) ≡ [ .[0], .[1], ... ] ≡ .

  • for object: map( . ) ≡ [ .["key1"], .["key2"], ... ]