Create an Azure DevOps Self-Hosted Agent Pool using Terraform

azure windows iac terraform coding devops

When studying Azure DevOps, I found that using IaC for building and tearing down resources is useful. This post will show you how to create an Azure DevOps self-hosted agent pool using Terraform.

Terraform1001

Introduction

Infrastructure as Code (IaC) is a great way to manage resources in the cloud. It allows you to create, update, and delete resources in a programmatic way. While studying for my AZ-400 certification, I found that building Azure resources manually each time running exercises was time-consuming. I decided to use Terraform to automate the process.

In this post, I'll share what I learned, creating the necessary Azure resources to run a self-hosted agent pool using Terraform. This will allow you to run your CI/CD pipelines on your own infrastructure.

Prerequisites

Before you start, you need to have the following:

  • Azure subscription
  • Azure DevOps account
  • Azure DevOps organization
  • Azure DevOps project
  • Azure DevOps PAT (Personal Access Token)
  • Azure DevOps agent pool
  • Terraform installed on your machine, in a docker container or on a VM

Create a Service Principal

To create resources in Azure, you need to create a service principal. Terraform needs the following rights:

  • Contributor role on the subscription

Azure CLI command to create a service principal:

az ad sp create-for-rbac --name tfporvider --role="Contributor" --scopes="/subscriptions/<subscription_id>" --sdk-auth

This script will output a JSON object with the service principal credentials. Save this JSON object safely.

{
  "clientId": "",
  "clientSecret": "",
  "subscriptionId": "",
  "tenantId": "",
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
  "resourceManagerEndpointUrl": "https://management.azure.com/",
  "activeDirectoryGraphResourceId": "https://graph.windows.net/",
  "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
  "galleryEndpointUrl": "https://gallery.azure.com/",
  "managementEndpointUrl": "https://management.core.windows.net/"
}

Some of the values, you need for your own terraform.tfvars later on.

Create a Terraform Configuration

There are many Azure resources involved in automating the creation of a VM to run a self-hosted agent. Here is a list of the resources:

  • Resource Group
  • Virtual Network
  • Subnet
  • Network Security Group
  • Public IP Address
  • Network Interface
  • Virtual Machine
  • Azure DevOps Agent

I learned that putting everything in one file like main.tf is not a good practice. It's better to split the configuration into multiple files. Here is an example of how to structure the files:

.
├── variables.tf
├── main.tf
├── net.tf
├── vm.tf
├── blob.tf
├── shagent.tf

variables.tf

It is a good practice to put all the variables in a separate file. This makes it easier to manage and update the variables. Here are my variables:

variable "rg_name" {
  type        = string
  default     = "app-grp"
  description = "Resource Group name."
}
 
variable "location" {
  type        = string
  default     = "norwayeast"
  description = "Location of all resources deployed."
}
 
variable "subscription_id" {
  description = "Azure Subscription ID"
  type        = string
}
 
variable "client_id" {
  description = "Azure Client ID"
  type        = string
}
 
variable "client_secret" {
  description = "Azure Client Secret"
  type        = string
  sensitive   = true
}
 
variable "tenant_id" {
  description = "Azure Tenant ID"
  type        = string
}
 
variable "vnet_name" {
  type        = string
  default     = "vnet"
  description = "The Virtual Network name."
}
 
variable "vnet_range" {
  type        = string
  default     = "10.0.0.0/24"
  description = "The Virtual Network range."
}
 
variable "subnet_name" {
  type        = string
  default     = "subnet-vm"
  description = "The Virtual Network Subnet for Virtual Machine name."
}
 
variable "subnet_range" {
  type        = string
  default     = "10.0.0.0/26"
  description = "The Virtual Network Subnet for Virtual Machine range."
}
 
variable "vm_nic_name" {
  type        = string
  default     = "vm-nic"
  description = "The Virtual Machine Network interface name."
}
 
variable "vm_name" {
  type        = string
  default     = "app-vm"
  description = "Virtual Machine name."
}
 
variable "vm_size" {
  type        = string
  default     = "Standard_B2s"
  description = "Virtual Machine size."
}
 
variable "admin_username" {
  type        = string
  description = "Admin username for the Virtual Machine."
  sensitive   = true
}
 
variable "admin_password" {
  type        = string
  description = "Admin password for the Virtual Machine."
  sensitive   = true
}
 
variable "agent-name" {
  type        = string
  default     = "self-hosted-agent"
  description = "The name of the agent."
}
 
variable "url" {
  type        = string
  description = "The URL of the Azure DevOps."
}
 
variable "pat" {
  type        = string
  description = "The Personal Access Token of the Azure DevOps."
}
 
variable "pool" {
  type        = string
  description = "The Pool name of the Azure DevOps."
}
 
variable "agent_version" {
  description = "Agent version (default: latest)"
  type        = string
  default     = ""
}

main.tf

This file is the entry point for Terraform. It contains the provider and module blocks. I also put in the azure Resource Group in this file. The highlighted lines are the parameters for the service principal.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "4.8.0"
    }
  }
}
 
provider "azurerm" {
  subscription_id = var.subscription_id
  client_id       = var.client_id
  client_secret   = var.client_secret
  tenant_id       = var.tenant_id
  features {}
}
 
resource "azurerm_resource_group" "rg" {
  name     = var.rg_name
  location = var.location
}

net.tf

Security warning: You are exposing port 3389 to the Internet. It is OK for a lab, but be aware on this for production environments

Before we can create a VM, we need to make sure it will be accessible. This code creates a Virtual Network, Subnet, and Network Security Group, allowing RDP traffic.

resource "azurerm_network_security_group" "app_network_nsg" {
  name                = "${var.subnet_name}-nsg"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
 
  security_rule {
    name                       = "allowRDP"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "3389"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
 
}
 
resource "azurerm_virtual_network" "app_network" {
  name                = var.vnet_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = [var.vnet_range]
 
}
 
resource "azurerm_subnet" "SubnetA" {
  name                 = var.subnet_name
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.app_network.name
  address_prefixes     = [var.subnet_range]
}
 
resource "azurerm_subnet_network_security_group_association" "SubnetA_nsg_link" {
  subnet_id                 = azurerm_subnet.SubnetA.id
  network_security_group_id = azurerm_network_security_group.app_network_nsg.id
}

vm.tf

This file builds the Azure VM for us. It creates a Public IP Address, Network Interface, and the Virtual Machine running Windows Server 2022. The size "Standard_B2s" is a small VM size, but it's enough for a self-hosted agent.

resource "azurerm_network_interface" "vm_nic" {
  name                = var.vm_nic_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
 
  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.SubnetA.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id          = azurerm_public_ip.app_public_ip.id
  }
}
 
resource "azurerm_windows_virtual_machine" "vm" {
  name                  = var.vm_name
  resource_group_name   = azurerm_resource_group.rg.name
  location              = azurerm_resource_group.rg.location
  size                  = "Standard_B2s"
  admin_username        = var.admin_username
  admin_password        = var.admin_password
  network_interface_ids = [azurerm_network_interface.vm_nic.id]
  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }
  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2022-Datacenter"
    version   = "latest"
  }
  identity {
    type = "SystemAssigned"
  }
}
 
resource "azurerm_public_ip" "app_public_ip" {
  name                = "${var.vm_name}-pip"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  allocation_method   = "Static"
  sku                 = "Standard"
}

blob.tf

We need a place to store the script used to install the Azure DevOps agent. This code creates a Storage Account, storage container and a storage blob. The highlighted line is defining the source of the script we need to store online.

resource "azurerm_storage_account" "eshoponweb" {
  name                     = "eshoponweb987987"
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}
 
resource "azurerm_storage_container" "data" {
  name                  = "data"
  storage_account_name  = azurerm_storage_account.eshoponweb.name
  container_access_type = "blob"
}
 
resource "azurerm_storage_blob" "windows-agent-install" {
  name                   = "windows-agent-install.ps1"
  storage_account_name   = azurerm_storage_account.eshoponweb.name
  storage_container_name = azurerm_storage_container.data.name
  type                   = "Block"
  source                 = "./windows-agent-install.ps1"
}

shagent.tf

This file creates the Azure DevOps agent. It uses the azurerm_virtual_machine_extension resource to run a script on the VM. The script installs the Azure DevOps agent. The highlighted lines are the block downloading the script for installing the agent and running the script.

# Powershell Script to Install Azure DevOps Agent
data "local_file" "windows_agent_script" {
  filename = "windows-agent-install.ps1"
}
 
resource "azurerm_virtual_machine_extension" "install_devops_agent" {
  name                 = "install-devops-agent"
  virtual_machine_id   = azurerm_windows_virtual_machine.vm.id
  publisher            = "Microsoft.Compute"
  type                 = "CustomScriptExtension"
  type_handler_version = "1.10"
  depends_on           = [azurerm_storage_blob.windows-agent-install]
 
  # Pass the necessary parameters to the Powershell script
 
  settings = <<SETTINGS
  {
      "fileUris": ["https://${azurerm_storage_account.eshoponweb.name}.blob.core.windows.net/data/windows-agent-install.ps1"],
      "commandToExecute": "powershell.exe -ExecutionPolicy Unrestricted -File ./windows-agent-install.ps1 -URL ${var.url} -PAT ${var.pat} -POOL ${var.pool} -AGENT ${var.agent-name}",
      "timestamp" : "12"
  }
  SETTINGS
}

Powershell Script: windows-agent-install.ps1

This script installs the Azure DevOps agent on the VM. It takes the URL, PAT, Pool, and Agent name as parameters from the settings block highlighted in the shagent.tf file. I found it here: https://melcher.dev/2019/03/self-hosted-azure-devops-build/release-agent-with-terraform-windows-edition/

param (
    [string]$URL,
    [string]$PAT,
    [string]$POOL,
    [string]$AGENT
)
 
Start-Transcript
Write-Host "start"
 
#test if an old installation exists, if so, delete the folder
if (test-path "c:\agent")
{
    Remove-Item -Path "c:\agent" -Force -Confirm:$false -Recurse
}
 
#create a new folder
new-item -ItemType Directory -Force -Path "c:\agent"
set-location "c:\agent"
 
$env:VSTS_AGENT_HTTPTRACE = $true
 
#github requires tls 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
 
#get the latest build agent version
$wr = Invoke-WebRequest https://api.github.com/repos/Microsoft/azure-pipelines-agent/releases/latest -UseBasicParsing
$tag = ($wr | ConvertFrom-Json)[0].tag_name
$tag = $tag.Substring(1)
 
write-host "$tag is the latest version"
#build the url
$download = "https://vstsagentpackage.azureedge.net/agent/$tag/vsts-agent-win-x64-$tag.zip"
 
#download the agent
Invoke-WebRequest $download -Out agent.zip
 
#expand the zip
Expand-Archive -Path agent.zip -DestinationPath $PWD
 
#run the config script of the build agent
.\config.cmd --unattended --url "$URL" --auth pat --token "$PAT" --pool "$POOL" --agent "$AGENT" --acceptTeeEula --runAsService
 
#exit
Stop-Transcript
exit 0

terraform.tfvars

Security warning: Keep sensitive information out of your repo.

This file contains the values for the variables. Keep this out of your repo for security and privacy reasons. .gitignore this file.

Fill in the values from the JSON object you saved earlier, as well as the Azure DevOps URL, PAT, and Pool name. You can set other variables as well, according to your needs.

subscription_id = ""
client_id       = ""
client_secret   = ""
tenant_id       = ""
rg_name         = "rg-eshoponweb-agentpool"
vm_name         = "eshoponweb-vm"
admin_username  = "sysadmin"
url             = "https://dev.azure.com/{YOUR_ORGANIZATION}/"
pat             = ""
pool            = "eShopOnWebSelfPool"

Running Terraform

After you have created the files, you can run Terraform. Here are the steps:

  1. Initialize Terraform
terraform init
  1. Plan the changes
terraform plan --out=main.tfplan

  1. Review the plan

  2. Apply the changes

terraform apply "main.tfplan"

Verify

After the Terraform run is complete, you can verify the resources in the Azure portal. You should see the Resource Group, Virtual Machine, and the Azure DevOps agent in the Azure DevOps portal.

Terraform1002

Tear down the resources

When you are done with the resources, you can tear them down using Terraform.

terraform destroy

Repository

You are free to use the code in your own projects. You can find the code in my GitHub repository: https://github.com/Jihillestad/terraform-selfhostedagentpool

Conclusion

It was a lot of work to create the Terraform configuration, but it was worth it. I learned a lot about Azure resources and how to automate the creation of resources. I hope this post helps you to move on with your IaC and DevOps journey.

In this experience, I only scratched the surface of what you can do with Terraform. There are many more resources and possibilities to explore. The next steps for me are to learn creating a CI/CD pipeline in Azure DevOps or GitHub Actions to deploy the resources automatically. I also need to work on setting Terraform variables securely in my repos, as well as finding a more secure way to store TFSTATE files.

For me, it was a great learning experience. I hope you enjoyed it too.