Parsing JSON and YAML Files with jq and yq in Shell Scripts

jq only reads JSON. Run it on a YAML file and it fails: jq '.name' data.yaml prints jq: parse error: Invalid numeric literal at line 1, column 5 (tested with jq 1.8.2). To query YAML in a shell script, use yq, which uses jq-like syntax. This guide covers both tools in shell scripts.

What is jq?

jq is a powerful command-line tool for parsing and manipulating JSON data. It’s like sed for JSON data, allowing you to extract, filter, and transform JSON in a flexible and readable manner.

What is yq?

yq is a command-line processor for YAML (and JSON, XML and more) that uses jq-like syntax. Two different tools share the name. The Go version, mikefarah/yq, is what brew install yq installs, and its README says it does not yet support everything jq does. The Python version, kislyuk/yq (pip install yq), is a wrapper that converts YAML to JSON and pipes it to jq, so it needs jq installed separately. A filter that works in one may not work in the other, so check which one you have before copying commands.

1. Installing jq and yq

Before diving into examples, you’ll need to install jq and yq. Here’s how you can do that:

# Install jq
sudo apt-get install jq  # For Debian/Ubuntu
brew install jq          # For macOS

# Install yq
snap install yq          # For Linux
brew install yq          # For macOS

The yq README says its Debian package is no longer maintained, so use Snap, Homebrew or a release binary instead of apt.

2. Parsing JSON Files with jq

Example JSON File (data.json):

{
  "name": "John Doe",
  "age": 30,
  "address": {
    "street": "123 Main St",
    "city": "Anytown"
  },
  "hobbies": ["reading", "hiking", "coding"]
}

Extracting Values

Let’s extract the name and age from this JSON file:

# Extract the name and age
jq '.name, .age' data.json

Output:

"John Doe"
30

Filtering Data

You can filter the JSON data based on specific criteria. For example, extracting only the city:

# Extract the city from the address
jq '.address.city' data.json

Output:

"Anytown"

Transforming JSON

You can transform JSON data into a different format. For example, converting the list of hobbies into a single comma-separated string:

# Convert hobbies array to a comma-separated string
jq -r '.hobbies | join(", ")' data.json

Output:

reading, hiking, coding

3. Parsing YAML Files with yq

Example YAML File (data.yaml):

name: John Doe
age: 30
address:
  street: 123 Main St
  city: Anytown
hobbies:
  - reading
  - hiking
  - coding

Extracting Values

Extracting values from YAML files is similar to JSON:

# Extract the name and age
yq '.name, .age' data.yaml

Output:

John Doe
30

Filtering Data

You can also filter YAML data, for example, extracting the city:

# Extract the city from the address
yq '.address.city' data.yaml

Output:

Anytown

Transforming YAML

Similar to jq, you can transform YAML data. For example, converting the list of hobbies into a single string:

# Convert hobbies array to a comma-separated string
yq '.hobbies | join(", ")' data.yaml

Output:

reading, hiking, coding

4. Combining jq and yq

You can combine jq and yq in a script to work with both JSON and YAML files. Here’s a simple script that reads both file types and prints out the name and hobbies:

#!/bin/bash

# Parse JSON file
echo "JSON Data:"
jq '.name, (.hobbies | join(", "))' data.json

# Parse YAML file
echo "YAML Data:"
yq '.name, (.hobbies | join(", "))' data.yaml

Output:

JSON Data:
"John Doe"
"reading, hiking, coding"

YAML Data:
John Doe
reading, hiking, coding

The parentheses matter. Without them, jq reads .name, .hobbies | join(", ") as (.name, .hobbies) | join(", ") and fails with Cannot iterate over string ("John Doe") (tested with jq 1.8.2 and yq v4.54.3).

Conclusion

Using jq and yq simplifies the process of parsing and manipulating JSON and YAML files in shell scripts. Whether you’re working with configuration files or API responses, these tools are invaluable for efficiently handling structured data. You can easily integrate them into your scripts to automate and streamline your workflows.

Frequently Asked Questions

Can jq read YAML files?

No. jq only parses JSON, so running it on a YAML file fails with a parse error. Use yq to query YAML, or convert the file to JSON first.

Is yq the same thing as jq?

No. yq is a separate tool that uses jq-like syntax for YAML. The Go version (mikefarah/yq) says it does not yet support everything jq does, and the Python version (kislyuk/yq) is a wrapper that converts YAML to JSON and pipes it to jq.

How do I install yq?

Run brew install yq or snap install yq for the Go version, or pip install yq for the Python wrapper, which also needs jq installed. The Go version’s README says its Debian package is no longer maintained.

Related guides