Code Movement

JQ - a tool for working with JSON

3 mins

JQ is a lightweight and flexible command-line JSON processor. JSON files are common-place and it is important to be able to work with them effectively.

On mac you can install JQ using Homebrew with the following command

brew install jq

With JQ you can pipe json data into JQ and use the DSL to perform filters. The single dot is the simplest command and will print the json neatly formatted to the console.

Below illustrates a decently involved json file that is the output of running terraform show. Using JQ we can format the output and make sense of it.

$ terraform show -json | jq '.'
{
  "format_version": "0.1",
  "terraform_version": "0.14.0",
  "values": {
    "root_module": {
      "resources": [
        {
          "address": "aws_s3_bucket.terraform_course",
          "mode": "managed",
          "type": "aws_s3_bucket",
          "name": "terraform_course",
          "provider_name": "registry.terraform.io/hashicorp/aws",
          "schema_version": 0,
          "values": {
            "acceleration_status": "",
            "acl": "private",
            "arn": "arn:aws:s3:::tf-course",
            "bucket": "tf-course",
            "bucket_domain_name": "tf-course.s3.amazonaws.com",
            "bucket_prefix": null,
            "bucket_regional_domain_name": "tf-course.s3.us-west-2.amazonaws.com",
            "cors_rule": [],
            "force_destroy": false,
            "grant": [],
            "hosted_zone_id": "AOCMQWPR291SJ",
            "id": "tf-course",
            "lifecycle_rule": [],
            "logging": [],
            "object_lock_configuration": [],
            "policy": null,
            "region": "us-west-2",
            "replication_configuration": [],
            "request_payer": "BucketOwner",
            "server_side_encryption_configuration": [],
            "tags": null,
            "versioning": [
              {
                "enabled": false,
                "mfa_delete": false
              }
            ],
            "website": [],
            "website_domain": null,
            "website_endpoint": null
          }
        }
      ]
    }
  }
}

If we want to extract a particular value, such as “terraform-course” from the ’name’ key of the first resource we can do the following

$ terraform show -json | jq '.values.root_module.resources[0].name'
"terraform_course"

When querying for a single value, you can use -r or --raw-output which will write the value directly to standard out without quotations.

$ terraform show -json | jq --raw-output '.values.root_module.resources[0].name'
terraform_course

This can be useful for making jq filters talk to systems that do not use JSON.

The following shows an example of using the raw output combined with join to create a MY_NAME variable with the value from the JSON.

$ MY_NAME=$(echo '{ "firstName": "Josh", "lastName": "Johansen" }' | jq -r '[.firstName, .lastName] | join(" ")')
$ echo $MY_NAME
Josh Johansen

Get unique number of dependencies from package.json

$ jq -r '[(.dependencies, .devDependencies) | keys] | flatten | length' package.json
22

The following shows all available jq paths for an object.

$ jq 'select(objects)|=[.] | map( paths(scalars) ) | map( map(select(numbers)="[]") | join(".")) | unique' package.json
[
  "dependencies.classnames",
  "dependencies.date-fns",
  "dependencies.gray-matter",
  "dependencies.next",
  "dependencies.react",
  "dependencies.react-dom",
  "dependencies.remark",
  "dependencies.remark-html",
  "devDependencies.@applitools/eyes-cypress",
  "devDependencies.@testing-library/jest-dom",
  "devDependencies.@testing-library/react",
  "devDependencies.@types/node",
  "devDependencies.@types/react",
  "devDependencies.babel-jest",
  "devDependencies.cypress",
  "devDependencies.eslint",
  "devDependencies.husky",
  "devDependencies.jest",
  "devDependencies.lint-staged",
  "devDependencies.prettier",
  "devDependencies.start-server-and-test",
  "devDependencies.typescript",
  "husky.hooks.pre-commit",
  "lint-staged.*.{js,jsx,ts,tsx,json}.[]",
  "lint-staged.*.{js,jsx,ts,tsx}.[]",
  "name",
  "private",
  "scripts.build",
  "scripts.cy:run:chrome",
  "scripts.cy:run:edge",
  "scripts.cy:run:firefox",
  "scripts.cypress",
  "scripts.deploy",
  "scripts.dev",
  "scripts.e2e",
  "scripts.start",
  "scripts.test",
  "scripts.test:chrome",
  "scripts.test:cypress",
  "scripts.test:edge",
  "scripts.test:ff",
  "version"
]

If you use JSON files then consider giving JQ a try.

Happy Coding.

Resources: