
jq only reads JSON. Run it on a YAML file and it fails: jq '.name' data.yaml prints jq: parse error: Invalid numeric literal at line 1, column 5 (tested with jq 1.8.2). To query YAML in a shell script, use yq, which uses jq-like syntax. This guide covers both tools in shell scripts.
What is jq?
jq is a powerful command-line tool for parsing and manipulating JSON data. It’s like sed for JSON data, allowing you to extract, filter, and transform JSON in a flexible and readable manner.
What is yq?
yq is a command-line processor for YAML (and JSON, XML and more) that uses jq-like syntax. Two different tools share the name. The Go version, mikefarah/yq, is what brew install yq installs, and its README says it does not yet support everything jq does. The Python version, kislyuk/yq (pip install yq), is a wrapper that converts YAML to JSON and pipes it to jq, so it needs jq installed separately. A filter that works in one may not work in the other, so check which one you have before copying commands.
1. Installing jq and yq
Before diving into examples, you’ll need to install jq and yq. Here’s how you can do that:
# Install jq
sudo apt-get install jq # For Debian/Ubuntu
brew install jq # For macOS
# Install yq
snap install yq # For Linux
brew install yq # For macOS
The yq README says its Debian package is no longer maintained, so use Snap, Homebrew or a release binary instead of apt.
2. Parsing JSON Files with jq
Example JSON File (data.json):
{
"name": "John Doe",
"age": 30,
"address": {
"street": "123 Main St",
"city": "Anytown"
},
"hobbies": ["reading", "hiking", "coding"]
}
Extracting Values
Let’s extract the name and age from this JSON file:
# Extract the name and age
jq '.name, .age' data.json
Output:
"John Doe"
30
Filtering Data
You can filter the JSON data based on specific criteria. For example, extracting only the city:
# Extract the city from the address
jq '.address.city' data.json
Output:
"Anytown"
Transforming JSON
You can transform JSON data into a different format. For example, converting the list of hobbies into a single comma-separated string:
# Convert hobbies array to a comma-separated string
jq -r '.hobbies | join(", ")' data.json
Output:
reading, hiking, coding
3. Parsing YAML Files with yq
Example YAML File (data.yaml):
name: John Doe
age: 30
address:
street: 123 Main St
city: Anytown
hobbies:
- reading
- hiking
- coding
Extracting Values
Extracting values from YAML files is similar to JSON:
# Extract the name and age
yq '.name, .age' data.yaml
Output:
John Doe
30
Filtering Data
You can also filter YAML data, for example, extracting the city:
# Extract the city from the address
yq '.address.city' data.yaml
Output:
Anytown
Transforming YAML
Similar to jq, you can transform YAML data. For example, converting the list of hobbies into a single string:
# Convert hobbies array to a comma-separated string
yq '.hobbies | join(", ")' data.yaml
Output:
reading, hiking, coding
4. Combining jq and yq
You can combine jq and yq in a script to work with both JSON and YAML files. Here’s a simple script that reads both file types and prints out the name and hobbies:
#!/bin/bash
# Parse JSON file
echo "JSON Data:"
jq '.name, (.hobbies | join(", "))' data.json
# Parse YAML file
echo "YAML Data:"
yq '.name, (.hobbies | join(", "))' data.yaml
Output:
JSON Data:
"John Doe"
"reading, hiking, coding"
YAML Data:
John Doe
reading, hiking, coding
The parentheses matter. Without them, jq reads .name, .hobbies | join(", ") as (.name, .hobbies) | join(", ") and fails with Cannot iterate over string ("John Doe") (tested with jq 1.8.2 and yq v4.54.3).
Conclusion
Using jq and yq simplifies the process of parsing and manipulating JSON and YAML files in shell scripts. Whether you’re working with configuration files or API responses, these tools are invaluable for efficiently handling structured data. You can easily integrate them into your scripts to automate and streamline your workflows.
Frequently Asked Questions
No. jq only parses JSON, so running it on a YAML file fails with a parse error. Use yq to query YAML, or convert the file to JSON first.
No. yq is a separate tool that uses jq-like syntax for YAML. The Go version (mikefarah/yq) says it does not yet support everything jq does, and the Python version (kislyuk/yq) is a wrapper that converts YAML to JSON and pipes it to jq.
Run brew install yq or snap install yq for the Go version, or pip install yq for the Python wrapper, which also needs jq installed. The Go version’s README says its Debian package is no longer maintained.