DevOps Done Right: Secure Terraform on Azure

In modern cloud infrastructure management, Terraform has become a popular choice due to its flexibility and ease of use. Azure DevOps offers powerful automation capabilities, making it an ideal platform for orchestrating Terraform deployments. In this guide, we’ll explore how to leverage Azure DevOps to automate Terraform deployments efficiently.

Overview

The automation process involves three main stages:

Terraform Plan: Generates an execution plan to preview infrastructure changes.
Manual Approval: Requires manual review and approval before applying changes.
Terraform Apply: Executes the approved changes to provision or update infrastructure.

Pipeline Structure

We organize our pipeline into three stages:

Terraform Plan: Initializes Terraform, validates configuration, performs static analysis, and generates a plan.
Manual Approval: Pauses the pipeline for manual review and approval of the Terraform plan.
Terraform Apply: Executes the Terraform plan to deploy infrastructure changes.

Pipeline Configuration

Why Build a Secure Azure DevOps Pipeline with Terraform?

Streamlined Workflows: Automate infrastructure deployments, saving time and reducing manual errors.
Granular Control: Implement manual approval stages for controlled and secure changes.
Template Reusability: Modularize your pipeline with templates for efficient management.
Enhanced Security: Store sensitive information in secure variable groups, keeping it out of code.
Improved Collaboration: Facilitate seamless collaboration between developers and infrastructure teams.

Prerequisites

Before you begin, make sure you have the following prerequisites in place:

Creating Service Connection from Azure DevOps to Azure portal (Authenticate Azure Portal with Azure DevOps).
Install required Extensions under Organization Settings – > extensions (Terraform , tfsec)

Creating Variable Group under Pipelines – > Library – > Click on Variable group.

Provide necessary details like Variable group name, Description and under Variables click Add
Under name add variable name, Under Value add the desired value, for example name: resource_group, value: test-group.

Build the Azure DevOps Pipeline:

Import the provided YAML template: Adjust parameters to your specific workflow.
Customize stages, templates, and parameters: Tailor the pipeline to your needs.
Define triggers and manual approval: Set when the pipeline runs and approval requirements.

Terraform Plan Stage

In this stage, we use a template (terraform_plan.yml) to define the tasks required for planning:

parameters:
  - name: backendServiceArm
    displayName: backend service connection name
    type: string
  - name: backendAzureRmResourceGroupName
    displayName: backend Resouce Group name
    type: string
  - name: backendAzureRmStorageAccountName
    displayName: backend Storage Account name
    type: string
  - name: backendAzureRmContainerName
    displayName: backend Container name
    type: string
  - name: backendAzureRmKey
    displayName: backend state file name
    type: string 
  - name: environmentServiceNameAzureRM
    displayName: service connection name
    type: string
  - name: tfsecVersion
    displayName: tfsec version
    type: string
  - name: tfsecArguments
    displayName: tfsec Arguments
    type: string
  - name: tfsecDirectory
    displayName: tfsec directory path
    type: string

jobs:
  - job: terrform_plan
    displayName: "Terraform Plan"
    steps:
      - task: TerraformTaskV4@4
        displayName: Terraform init
        inputs:
          provider: 'azurerm'
          command: 'init'
          backendServiceArm: ${{ parameters.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ parameters.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ parameters.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ parameters.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ parameters.backendAzureRmKey }}

      - task: TerraformTaskV4@4
        displayName: terraform validate
        inputs:
          provider: 'azurerm'
          command: 'validate'
          environmentServiceNameAzureRM: ${{ parameters.environmentServiceNameAzureRM }}

      - task: tfsec@1
        displayName: tfsec
        inputs:
          version: ${{ parameters.tfsecVersion }}
          args: ${{ parameters.tfsecArguments }}
          dir: ${{ parameters.tfsecDirectory }}

      - task: TerraformTaskV4@4
        displayName: Terraform plan
        inputs:
          provider: 'azurerm'
          command: 'plan'
          environmentServiceNameAzureRM: ${{ parameters.environmentServiceNameAzureRM }}

Manual Approval and Terraform Apply Stage

We use a template (terraform_apply.yml)This stage ensures human oversight before applying changes:

parameters:
  - name: backendServiceArm
    displayName: backend service connection name
    type: string
  - name: backendAzureRmResourceGroupName
    displayName: backend Resouce Group name
    type: string
  - name: backendAzureRmStorageAccountName
    displayName: backend Storage Account name
    type: string
  - name: backendAzureRmContainerName
    displayName: backend Container name
    type: string
  - name: backendAzureRmKey
    displayName: backend state file name
    type: string 
  - name: environmentServiceNameAzureRM
    displayName: service connection name
    type: string

jobs:
  - job: manual_approval
    displayName: "Manual Approval"
    # dependsOn: terrform_plan
    pool: server
    timeoutInMinutes: 4320
    steps:
      - task: ManualValidation@0
        timeoutInMinutes: 1440
        inputs:
          instructions: 'Please review the Terraform plan and manually approve or reject.'

  - job: terrform_deployment
    displayName: "Terraform deployment"
    dependsOn: manual_approval
    steps:
      - task: TerraformTaskV4@4
        displayName: Terraform init
        inputs:
          provider: 'azurerm'
          command: 'init'
          backendServiceArm: ${{ parameters.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ parameters.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ parameters.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ parameters.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ parameters.backendAzureRmKey }}

      - task: TerraformTaskV4@4
        displayName: Terraform apply
        inputs:
          provider: 'azurerm'
          command: 'apply'
          environmentServiceNameAzureRM: ${{ parameters.environmentServiceNameAzureRM }}

Main yaml file

Main yaml file (terraform.yml) where we refer the above templates and run pipeline to plan and deploy the resources :

trigger:
  batch: true
  branches:
    include:
      - "master"

pool:
  vmImage: ubuntu-latest

variables:
  - group: terraform_backend

stages:
  - stage: terraform_plan
    displayName: Terraform Plan
    jobs:
    - template: ./common/terraform_plan.yml
      parameters:
        backendServiceArm: $(service_connection_name)
        backendAzureRmResourceGroupName: $(backend_ResourceGroupName)
        backendAzureRmStorageAccountName: $(backend_StorageAccountName)
        backendAzureRmContainerName: $(backend_ContainerName)
        backendAzureRmKey: $(backend_Key)
        environmentServiceNameAzureRM: $(service_connection_name)
        tfsecVersion: 'v1.26.0'
        tfsecArguments: '-s'
        tfsecDirectory: '.'
  - stage: terraform_apply
    displayName: Terraform Deploy
    dependsOn: terraform_plan
    jobs:
      - template: ./common/terraform_apply.yml
        parameters:
          backendServiceArm: $(service_connection_name)
          backendAzureRmResourceGroupName: $(backend_ResourceGroupName)
          backendAzureRmStorageAccountName: $(backend_StorageAccountName)
          backendAzureRmContainerName: $(backend_ContainerName)
          backendAzureRmKey: $(backend_Key)
          environmentServiceNameAzureRM: $(service_connection_name)

Once the code is ready in your repo we need to set the Pipelines, from the left menu click on Pipelines Click on New Pipeline – > under connect “Azure Repo Git” -> Select a repo -> under configuration “Existing Azure Pipelines YAML file” -> Select path to the yaml file -> click on Continue and click Run.

Once the Pipeline is set need to add permission for variable group.

Parameters

We define parameters to customize the pipeline configuration:

backendServiceArm: Azure service connection for backend storage.
backendAzureRmResourceGroupName: Resource group name for backend state.
backendAzureRmStorageAccountName: Storage account name for backend state.
backendAzureRmContainerName: Container name for backend state file.
backendAzureRmKey: State file name within the container.
environmentServiceNameAzureRM: Azure service connection for Terraform execution.
tfsecVersion: Optional tfsec tool version.
tfsecArguments: Optional arguments for tfsec scan.
tfsecDirectory: Optional directory containing Terraform code.

How It Works

Developers push Terraform code changes to the master branch.
The pipeline is triggered automatically.
Terraform initializes, validates, and generates a plan.
Manual approval is required for the generated plan.
Upon approval, the pipeline deploys the changes to the infrastructure.
Link for the demo pipeline

Conclusion

By automating Terraform deployments with Azure DevOps, teams can streamline their infrastructure provisioning processes, enforce best practices, and ensure consistency across environments. This approach enhances collaboration, reduces manual errors, and accelerates time-to-market for cloud-based applications and services.

DevToolHub