Infrastructure as code (IaC): Using Azure Resource Manager (ARM) templates
In my previous article I was talking about the usage of infrastructure within the code. Now I am going to take a few steps back and look into the depths of the azure cloud.
I would like to cover in this article all azure resources using Azure Resource Manager templates (ARM templates). There are multiple ways how to create at all resources in Azure cloud.
These resources can be created manually (clicking in portal) or automatically (ARM templates, Bicep templates, Blueprints), and best approach for automation of this process are ARM templates.
ARM overview
Azure Resource Manager templates (ARM templates). Definition from documentation of Microsoft:
The template is a JavaScript Object Notation (JSON) file that defines the infrastructure and configuration for your project The template uses declarative syntax, which lets you state what you intend to deploy without having to write the sequence of programming commands to create it. In the template, you specify the resources to deploy and the properties for those resources. Sources
And ARM can be separated into templates and parameters files. Empty Azure Resource Manager template looks like:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"functions": [],
"variables": {},
"resources": [],
"outputs": {}
}Each part of that file has another functionality, parameters: provide values during deployment, for different values is recommended to use separated parameters file. variables are used in templates and can be reused.
resources is list of definition for all resources.
And deployment parameters. Where can be stored different values for resources list in templates. E.g. When I want to have same resources in different environments. So I can keep one template file and multiple parameters files where I defined changes across environments.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {}
}Another alternative is to use bicep files. In current example I focus on json version of azure templates. In general bicep can be converted from .json with command az bicep build main.bicep. For more information about bicep in general "bicep overview," and also I recommended learn everything about the next generation of ARM templates
Pipeline overview
Before I go deep into templates themselves I take a look at pipeline which are using during deployment. ARM templates are basically only json files, so first you need to check if every templates what I am using is valid json file, next you need to check if it is valid for Azure deployment and contains are necessary stuff. Next you should provide some tests, if all resources will be deployed.
I implemented the test part through Azure Resource Manager Template Toolkit (arm-ttk)
Validation is done by redeploy the resources in the sandbox with a few exceptions. All resources are created there for verification only. Once verified, all resources are deleted and thus will not interfere for further deployments.
I am using multiple environments. Development (dev) for testing and development purpose, shared which is shared by all others environments. Integration (INT) or nonprod (NONprod) which is basically close to the production, but still not accessible for end consumers. Here should be tested all connection and stuff. If everything is fine, all changes can be merged into production.
This approach can bring more clarity to whole process. On the other hand, all the additional environmental costs consume more money and more resources need to be maintained, so decisions should be made carefully.
Automated check and validation
Tests of ARM templates with tool validation of ARM in sandbox environment (pre-deploy ). Validation is done by powershell command Test-AzResourceGroupDeployment. This cmdlet determines whether an Azure resource group deployment template and its parameter values are valid.
- task: AzurePowerShell@5
displayName: Validate ARM templates
inputs:
azureSubscription: "$(dev-environment-subscription)" # ideally it should be validate in same subscription as dev, nonproduction or production
ScriptType: "FilePath"
ScriptPath: "$(System.DefaultWorkingDirectory)/validate_templates.ps1" # script basically run Test-AzResourceGroupDeployment for all templates and their parameters
ScriptArguments: '-resourceGroupName $(validation-sandbox) -location $(location) -templateDirectoryPath $(System.DefaultWorkingDirectory)\arm_templates'
azurePowerShellVersion: "LatestVersion"
FailOnStandardError: trueSmall example how that validation validate_templates.ps1 looks like:
Test-AzResourceGroupDeployment `
-ResourceGroupName $resourceGroupName `
-TemplateFile $templateFilePath `
-TemplateParameterFile $parameterFilePathResources created in Azure and covered by ARM templates
List of main resources which are covered by ARM templates. For purpose of this article I do not delve into everything. In most cases you can find good examples and documentation for all these resources in microsoft docs.
-
Azure firewall,
-
Virtual network,
-
Key-vault,
-
AKS cluster (Azure Kubernetes Service cluster),
-
Route Tables,
-
Public Ip address,
-
Role assignment
Public Ip address
This resource is responsible for create public address for entire communication between outside world and apps. Between all these stuff is firewall, virtual networks, routing tables, load balancer and then running containers inside kubernetes cluster. So this resource must be created first and then others can be created that will point to this public IP address.
Arm template looks very straightforward:
{
"apiVersion": "2020-08-01",
"type": "Microsoft.Network/publicIPAddresses",
"name": "[parameters('publicIpAddressName')]",
"location": "[parameters('location')]",
"sku": {
"name": "[parameters('sku')]"
},
"properties": {
"publicIPAllocationMethod": "Static",
"publicIPAddressVersion": "[parameters('publicIPAddressVersion')]",
"idleTimeoutInMinutes": "[parameters('idleTimeoutInMinutes')]"
}
}More information about "name": "[parameters('sku')]" public-ip-addresses#sku.
The most interesting part is the return of the created ip address, which can be useful in a later process.
{
"publicIpAddress": {
"type": "string",
"value": "[reference(resourceId('Microsoft.Network/publicIPAddresses',parameters('publicIpAddressName'))).IpAddress]"
},
"publicIpResourceId": {
"type": "string",
"value": "[resourceId('Microsoft.Network/publicIPAddresses', parameters('publicIpAddressName'))]"
}
}Route tables
Routing tables can be generated automatically during AKS building, however I want to have all configurations steps in hands. So I created them first, and then I say to AKS (Azure Kubernetes Service), "I have these routing tables already in place, can you please use them."
Parameters for addressPrefix is taken from previous public ip address resource "addressPrefix": "[concat(parameters('publicIpAddress'),'/32')]",.
And "nextHopIpAddress": "[variables('firewallPrivateIPAddress')]", is static value for firewall resource and that ip address is used only internally.
{
"type": "Microsoft.Network/routeTables",
"apiVersion": "2020-11-01",
"name": "[parameters('routeTablesAksEgressFwrtName')]",
"location": "[parameters('location')]",
"properties": {
"disableBgpRoutePropagation": false,
"routes": [
{
"name": "aks-egress-forward-firewall",
"properties": {
"addressPrefix": "0.0.0.0/0",
"nextHopType": "VirtualAppliance",
"nextHopIpAddress": "[variables('firewallPrivateIPAddress')]",
"hasBgpOverride": false
}
},
{
"name": "aks-egress-forward-internet",
"properties": {
"addressPrefix": "[concat(parameters('publicIpAddress'),'/32')]",
"nextHopType": "Internet",
"hasBgpOverride": false
}
}
]
}
}Azure Firewall
First complicated resource is azure firewall. Azure Firewall depends on virtual network, subnet and public ip address. And also depends on firewall policies which needs to be created before Firewall itself.
{
"apiVersion": "2020-05-01",
"type": "Microsoft.Network/azureFirewalls",
"name": "[parameters('azureFirewallName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/firewallPolicies', parameters('firewallPolicyName'))]",
"[resourceId('Microsoft.Network/firewallPolicies/ruleCollectionGroups', parameters('firewallPolicyName'), 'DefaultDnatRuleCollectionGroup')]",
"[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworksAksName'), 'azure-firewall-subnet')]"
],
"properties": {
"ipConfigurations": [
{
"name": "[parameters('azureFirewallName')]",
"properties": {
"subnet": {
"id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworksAksName'), 'azure-firewall-subnet')]"
},
"publicIpAddress": {
"id": "[parameters('publicIpResourceId')]"
}
}
}
],
"sku": {
"tier": "Standard"
},
"firewallPolicy": {
"id": "[resourceId('Microsoft.Network/firewallPolicies', parameters('firewallPolicyName'))]"
}
}
}In these firewall policies I can specify inbound and outbound rules, I can allow or block specific ports, ip etc. All these policies can be reused for another firewall or subnet.
{
"apiVersion": "2020-11-01",
"type": "Microsoft.Network/firewallPolicies",
"name": "[parameters('firewallPolicyName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworksAksName'), 'azure-firewall-subnet')]"
],
"properties": {
"sku": {
"tier": "Standard"
},
"dnsSettings": {
"enableProxy": true
},
"threatIntelWhitelist": {
"fqdns": [],
"ipAddresses": []
}
},
"resources": [
{
"apiVersion": "2020-11-01",
"type": "ruleCollectionGroups",
"name": "DefaultDnatRuleCollectionGroup",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/firewallPolicies',parameters('firewallPolicyName'))]"
],
"properties": {
"priority": 100,
"ruleCollections": [
{
"name": "inboundlbrules",
"priority": 101,
"action": {
"type": "Dnat"
},
"rules": [
{
"name": "allow_specific_ip_location",
"sourceAddresses": [
"11.222.100.224/27"
],
"translatedAddress": "[variable('privateIpAddressFor')]",
"translatedPort": "443",
"ruleType": "NatRule",
"ipProtocols": [
"TCP"
],
"destinationAddresses": [
"[parameters('publicIpAddress')]"
],
"destinationIpGroups": [],
"destinationPorts": [
"443"
],
"sourceIpGroups": []
}
],
"ruleCollectionType": "FirewallPolicyNatRuleCollection"
}
]
}
}
]
}Virtual network and subnets
Virtual network is used as backbone for all communication between resources. On virtual network I use three subnets.
One subnet aks-subnet for all kubernetes stuff, such as nodes and pods. Another subnet for azure-firewall-subnet for firewall activity.
And the last subnet for ingress-internal-subnet for internal communication between Azure Firewall and ingress point.
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2020-05-01",
"name": "[parameters('virtualNetworksAksName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/routeTables', parameters('routeTablesAksEgressFwrtName'))]"
],
"properties": {
"addressSpace": {
"addressPrefixes": [
"172.28.12.0/24",
"172.28.13.0/24",
"172.29.0.0/16"
]
},
"subnets": [
{
"name": "aks-subnet",
"properties": {
"addressPrefix": "172.29.2.0/23",
"routeTable": {
"id": "[resourceId('Microsoft.Network/routeTables', parameters('routeTablesAksEgressFwrtName'))]"
},
"networkSecurityGroup": {
"id": "[variables('nsgId')]"
},
"serviceEndpoints": [
{
"service": "Microsoft.KeyVault",
"locations": [
"*"
]
},
{
"service": "Microsoft.Sql",
"locations": [
"westeurope"
]
}
],
"delegations": [],
"privateEndpointNetworkPolicies": "Disabled",
"privateLinkServiceNetworkPolicies": "Enabled"
}
},
{
"name": "azure-firewall-subnet",
"properties": {
"addressPrefix": "172.29.1.0/26",
"serviceEndpoints": [],
"delegations": [],
"privateEndpointNetworkPolicies": "Enabled",
"privateLinkServiceNetworkPolicies": "Enabled"
}
},
{
"name": "ingress-internal-subnet",
"properties": {
"addressPrefix": "172.28.12.0/28",
"serviceEndpoints": [],
"delegations": [],
"privateEndpointNetworkPolicies": "Enabled",
"privateLinkServiceNetworkPolicies": "Enabled"
}
}
],
"virtualNetworkPeerings": [],
"enableDdosProtection": false
}
}Key-vault
Another interesting resources is key vault for storing all secure, important information which should be accessible only in specific way and should be hidden from outside world.
The template specifies which ip addresses are allowed to access the keyvault, and also specifies which subnets (aks-subnet) can access the keyvault. Other sources, subnets should be able to access this keyvault.
{
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "2020-04-01-preview",
"name": "[parameters('keyVaultName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworksAksName'), 'aks-subnet')]"
],
"properties": {
"enabledForDeployment": false,
"enabledForTemplateDeployment": true,
"enabledForDiskEncryption": false,
"enabledForVolumeEncryption": false,
"tenantId": "[subscription().tenantId]",
"enablePurgeProtection": true,
"enableSoftDelete": true,
"softDeleteRetentionInDays": 90,
"enableRbacAuthorization": "[parameters('enableRbacAuthorization')]",
"accessPolicies": "[parameters('accessPolicies')]",
"createMode": "default",
"sku": {
"family": "A",
"name": "standard"
},
"networkAcls": {
"bypass": "AzureServices",
"defaultAction": "Deny",
"copy": [
{
"name": "ipRules",
"count": "[length(variables('ipWhiteList'))]",
"input": {
"value": "[variables('ipWhiteList')[copyIndex('ipRules')]]"
}
}
],
"virtualNetworkRules": [
{
"action": "Allow",
"id": "[resourceId(variables('mainResourceGroupName'), 'Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworksAksName'), 'aks-subnet')]"
}
]
}
}
}AKS (Azure Kubernetes Service) Cluster
The most complex part of the puzzle, which I will divide into several parts for better understanding. First part as always contains apiVersion, name and location for this kubernetes cluster and dependsOn part, where:
-
aks-subnetis subnet which was created with virtual network in first place. -
nameUniqueRoleGuidKeyVaultAdminis regarding KeyVault (where all https certificates are stored) and particular role for that.
There is also information on where the images for kubernetes should be downloaded from.
For this purpose there is a shared resource group with Azure Container Registry (acr) and it is defined by variables('registryName') and acrResourceGroupName.
There is also an agent pool profile. There may be more than one, but for the purposes of the demo only one is shown.
{
"apiVersion": "2021-02-01",
"type": "Microsoft.ContainerService/managedClusters",
"location": "[parameters('location')]",
"name": "[parameters('managedClustersName')]",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworksAksName'), 'aks-subnet')]",
"[variables('nameUniqueRoleGuidKeyVaultAdmin')]"
],
"properties": {
"kubernetesVersion": "[parameters('kubernetesVersion')]",
"enableRBAC": "[parameters('enableRBAC')]",
"dnsPrefix": "[parameters('dnsPrefix')]",
"acrName": "[variables('registryName')]",
"acrResourceGroupName": "[variables('sharedResourceGroupName')]",
"agentPoolProfiles": [
{
"name": "nodepool1",
"count": "[parameters('agentCount')]",
"vmSize": "[parameters('agentVMSize')]",
"osDiskSizeGB": "[parameters('osDiskSizeGB')]",
"osDiskType": "Managed",
"vnetSubnetID": "[variables('vnetSubnetID')]",
"maxPods": "[parameters('maxPods')]",
"type": "VirtualMachineScaleSets",
"orchestratorVersion": "[parameters('kubernetesVersion')]",
"enableNodePublicIP": "[parameters('enableNodePublicIP')]",
"nodeLabels": {},
"mode": "System",
"osType": "[parameters('osType')]"
}
]
}
}I continue with specified linux and network profile. There are few approach how to access and use AKS cluster. There is limitation with security and support. If you increase security to the cluster, you complicate way to fix and support application running inside that particular cluster. I strongly recommend taking a look aks network-policies.
{
"linuxProfile": {
"adminUsername": "[parameters('adminUsername')]",
"ssh": {
"publicKeys": [
{
"keyData": "[parameters('sshRSAPublicKey')]"
}
]
}
},
"networkProfile": {
"loadBalancerSku": "standard",
"networkPlugin": "kubenet",
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16",
"dnsServiceIP": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16",
"outboundType": "userDefinedRouting",
"networkPolicy": "calico"
},
"apiServerAccessProfile": {
"enablePrivateCluster": false
}
}Next part is associated with logAnalyticsWorkspaceResourceID, it is connected to log analytics where are all logs generated by this resource.
It is also enabled azure-policy. This Azure Kubernetes Service use managed identity to identify.
Managed identities are essentially a wrapper around service principals, and make their management simpler. This managed identity was created before (together with public ip address).
"addonProfiles":{
"omsagent": {
"enabled": true,
"config": {
"logAnalyticsWorkspaceResourceID": "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('workspacesLoganalyticsName'))]"
}
},
"azurepolicy": {
"enabled": "[parameters('azurePolicyEnabled')]",
"config": {
"version": "v2"
}
}
},
"servicePrincipalProfile": {
"clientId": "msi"
}
},
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('userAssignedIdentities_aks_egress_name'))]": {}
}
}User assigned identity was created by previous task and returns by output "userAssignedIdentitiesObjectId": { "value": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('userAssignedIdentities_aks_egress_name')), '2018-11-30').principalId]" }. It is same user identity which is used by AKS.
And last but not least is the role assignment that was mentioned at the beginning. It is part of many role assignments that are used to allow Azure Kubernetes Service access to key-vault, storage account, azure registry account, etc.
The same role assignment can be done via Powershell: az role assignment create --assignee $APPID --scope $KEY_VAULT --role "Key Vault Administrator"
{
"type": "Microsoft.Authorization/roleAssignments",
"apiVersion": "2020-04-01-preview",
"name": "[guid(subscription().id, variables('keyVaultAdministrator'))]",
"scope": "[concat('Microsoft.KeyVault/vaults', '/', parameters('keyVaultName'))]",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults', parameters('keyVaultName'))]"
],
"properties": {
"roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('keyVaultAdministrator'))]",
"principalId": "[parameters('userAssignedIdentitiesObjectId')]"
}
}Conclusion
In this article, I showed you an example of using an Azure Resource Manager Templates and deploying it to the infrastructure using DevOps pipeline. I focused on resources that can be used in any infrastructure or environment. All these resources can be deployed, tested and configured manually and then exported as templates, so they can be used in automation. These templates can provide a great advantage when you need to recreate a new environment with similar resources.
Michal Slovík
Java developer and Cloud DevOps
My job interests include devops, java development and docker / kubernetes technologies.