Parsing JSON and YAML Files with jq and yq in Shell Scripts

Parsing and manipulating JSON and YAML files is a common task in DevOps and scripting. Tools like jq for JSON and yq for YAML make it easier to handle these formats in shell scripts. Below is a guide on how to use these tools, complete with examples that you can use for your blog.

What is `jq`?

jq is a powerful command-line tool for parsing and manipulating JSON data. It’s like sed for JSON data, allowing you to extract, filter, and transform JSON in a flexible and readable manner.

What is `yq`?

yq is similar to jq, but for YAML files. It supports all jq syntax and functions, making it an excellent choice for dealing with YAML data in shell scripts.

1. Installing `jq` and `yq`

Before diving into examples, you’ll need to install jq and yq. Here’s how you can do that:

# Install jq
sudo apt-get install jq  # For Debian/Ubuntu
brew install jq          # For macOS

# Install yq
sudo apt-get install yq  # For Debian/Ubuntu
brew install yq          # For macOS

2. Parsing JSON Files with `jq`

Example JSON File (`data.json`):

{
  "name": "John Doe",
  "age": 30,
  "address": {
    "street": "123 Main St",
    "city": "Anytown"
  },
  "hobbies": ["reading", "hiking", "coding"]
}

Extracting Values

Let’s extract the name and age from this JSON file:

# Extract the name and age
jq '.name, .age' data.json

Output:

"John Doe"
30

Filtering Data

You can filter the JSON data based on specific criteria. For example, extracting only the city:

# Extract the city from the address
jq '.address.city' data.json

Output:

"Anytown"

Transforming JSON

You can transform JSON data into a different format. For example, converting the list of hobbies into a single comma-separated string:

# Convert hobbies array to a comma-separated string
jq -r '.hobbies | join(", ")' data.json

Output:

reading, hiking, coding

3. Parsing YAML Files with `yq`

Example YAML File (`data.yaml`):

name: John Doe
age: 30
address:
  street: 123 Main St
  city: Anytown
hobbies:
  - reading
  - hiking
  - coding

Extracting Values

Extracting values from YAML files is similar to JSON:

# Extract the name and age
yq '.name, .age' data.yaml

Output:

John Doe
30

Filtering Data

You can also filter YAML data, for example, extracting the city:

# Extract the city from the address
yq '.address.city' data.yaml

Output:

Anytown

Transforming YAML

Similar to jq, you can transform YAML data. For example, converting the list of hobbies into a single string:

# Convert hobbies array to a comma-separated string
yq '.hobbies | join(", ")' data.yaml

Output:

reading, hiking, coding

4. Combining `jq` and `yq`

You can combine jq and yq in a script to work with both JSON and YAML files. Here’s a simple script that reads both file types and prints out the name and hobbies:

#!/bin/bash

# Parse JSON file
echo "JSON Data:"
jq '.name, .hobbies | join(", ")' data.json

# Parse YAML file
echo "YAML Data:"
yq '.name, .hobbies | join(", ")' data.yaml

Output:

JSON Data:
"John Doe"
"reading, hiking, coding"

YAML Data:
John Doe
reading, hiking, coding

Conclusion

Using jq and yq simplifies the process of parsing and manipulating JSON and YAML files in shell scripts. Whether you’re working with configuration files or API responses, these tools are invaluable for efficiently handling structured data. You can easily integrate them into your scripts to automate and streamline your workflows.

DevToolHub