Creating the AKS Cluster
Follow the instructions below to define the AKS cluster resource requirements and then create the cluster based on your specifications.
Define the AKS Cluster Requirements
The first step in creating the K8s cluster is to define the infrastructure specifications. The configuration file to use for defining the specifications is called k8s_cluster.conf. Multiple sample k8s_cluster.conf files are included in the az directory. Any of them can be copied and used as templates, or the files can be edited directly.
Sample k8s_cluster.conf Files
To help guide you in choosing the appropriate template for your use case, this section describes each of the sample files. Details about the parameters in the sample files are included in Cluster Parameters below.
There are several sample use case files because there is an example for each type of AKS-supported identity and authentication management option. You can use a combination of settings from different sample files to configure your cluster, but you can only choose one type of authentication. For example, you cannot configure Service Principals and enable Azure Active Directory.
az/conf.d/k8s_cluster.conf
This file is a non-specific use case. It includes sample values for all of the available cluster parameters.
az/sample_use_cases/1_azureManagedIdentity_private_cluster/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - Azure Managed Identity is enabled (
ENABLE_MANAGED_IDENTITY="true"
) so that Azure manages identity creation and management. Using Azure Managed Identity is highly recommended.
az/sample_use_cases/2_createServicePrincipal_public_cluster/k8s_cluster.conf
This file includes sample values for a use case where:
- A public AKS cluster is deployed (
PRIVATE_CLUSTER="false"
). - A Service Principal is created (
SP=${SP:-"<service-principal>"}
) that must be renewed and managed by you. - Public access to the cluster can be limited to certain IP ranges by specifying the approved ranges in the
API_SERVER_AUTHORIZED_IP_RANGES
parameter.
az/sample_use_cases/3_useServicePrincipal/k8s_cluster.conf
This file includes sample values for a use case where:
- A public AKS cluster is deployed (
PRIVATE_CLUSTER="false"
). - An existing Service Principal is used for identity and access management. The
SP_ID
andSP_SECRET
parameters are used to specify the ID and secret for the existing Service Principal. - Public access to the cluster can be limited to certain IP ranges by specifying the approved ranges in the
API_SERVER_AUTHORIZED_IP_RANGES
parameter.
az/sample_use_cases/4_userManagedAAD/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - An existing Azure Active Directory (AAD) server is used for identity and authorization management. Details about the existing AAD client and server applications as well as the tenet ID need to be specified in the
AAD_CLIENT_APP_ID
,AAD_SERVER_APP_ID
,AAD_SERVER_APP_SECRET
, andAAD_TENANT_ID
parameters.
az/sample_use_cases/5_azureManagedAAD/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - An Azure-managed Active Directory (AAD) server is enabled (
ENABLE_AAD="true"
). - The AKS resource provider manages the AAD client and server applications.
az/sample_use_cases/6_attachACR/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - The cluster is configured to retrieve images from an existing private Azure Container Registry (ACR) by specifying the name of the ACR in the
ATTACH_ACR
parameter.
az/sample_use_cases/7_clusterAutoscalerSupport/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - The Cluster Autoscaler service is enabled (
ENABLE_CLUSTER_AUTOSCALER="true"
) so that nodes are automatically added to the node pool when demand increases and removed from the node pool when demand decreases. - The parameter
CLUSTER_AUTOSCALER_PROFILE
parameter is used to configure the autoscaler.
az/sample_use_cases/8_MonitoringEnabled/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - The Monitoring service is enabled (
AKS_ENABLE_ADDONS="monitoring"
) for the cluster.
az/sample_use_cases/9_RBACSupport/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - Azure Role-Based Access Control (RBAC) is enabled (
DISABLE_RBAC="false"
).
az/sample_use_cases/10_useExistingResources/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - You deploy the cluster into your existing Resource Group, Virtual Network, and Subnetwork by specifying the values for those resources in the
RESOURCE_GROUP=${RESOURCE_GROUP:-"<resource-group>"}
,VNET_NAME=${VNET_NAME:-"<name>"}
, andSUBNET_NAME="<subnet-name>"
parameters.
az/sample_use_cases/11_useProximityPlacementGroups/k8s_cluster.conf
This file includes sample values for a use case where:
- A private AKS cluster is deployed (
PRIVATE_CLUSTER="true"
) so that the cluster is only accessible from within the Virtual Network or a connected network. - You define a Proximity Placement Group (PPG) so that Azure deploys compute resources into a logical grouping where they are physically located close to each other to reduce latency. You specify the PPG name and type of group in the
PPG=${PPG:-"<name>"}
andPPG_TYPE=${PPG_TYPE:-"<type>"}
parameters.
Cluster Parameters
The contents of k8s_cluster.conf
are shown below. Descriptions of the cluster parameters follow the contents.
ENABLE_MANAGED_IDENTITY="<enable-managed-identity>" SP=${SP:-"<service-principal>"}
SP_VALIDITY_YEARS="<years>"
SP_ID="<id>"
SP_SECRET="<client-secret>"
RESOURCE_GROUP=${RESOURCE_GROUP:-"<resource-group>"}
RESOURCE_GROUP_TAGS="<tags>"
LOCATION=${LOCATION:-"<location>"}
SUBSCRIPTION_ID="<subscription-id>"
VNET_NAME=${VNET_NAME:-"<name>"}
VNET_CIDR="<vnet-cidr>"
VNET_TAGS="<tags>"
VNET_VM_PROTECTION="<vm-protection>"
SUBNET_NAME="<subnet-name>"
SUBNET_CIDR="<subnet-cidr>"
NODE_ZONES="<zones>"
NODEPOOL_NAME="<name>"
NODEPOOL_TAGS="<tags>"
MACHINE_TYPE="<machine-type>"
K8S_CLUSTER_NAME=${K8S_CLUSTER_NAME:-"<name>"}
K8S_CLUSTER_VERSION=${K8S_CLUSTER_VERSION:-"<kubernetes-version>"}
K8S_CLUSTER_NODE_COUNT="<node-count>"
K8S_NODE_ADMIN_USER="<admin-username>"
AKS_TAGS="<tags>"
AKS_ENABLE_ADDONS="<addons>"
PRIVATE_CLUSTER="<enable-private-cluster>"
LOAD_BALANCER_SKU="<load-balancer-sku>"
LB_BALANCER_IDLE_TIMEOUT=<load-balancer-idle-timeout>
LB_OUTBOUND_IP_PREFIXES="<load-balancer-outbound-ip-prefixes>"
LB_OUTBOUND_IPS="<load-balancer-outbound-ips>"
LB_OUTBOUND_PORTS=<load-balancer-outbound-ports>
LB_MANAGED_OUTBOUND_IP_COUNT=<load-balancer-managed-outbound-ip-count>
VM_SET_TYPE="<vm-set-type>"
NETWORK_PLUGIN="<network-plugin>"
NETWORK_POLICY="<network-policy>"
DOCKER_BRIDGE_ADDRESS="<docker-bridge-address>"
DNS_SERVICE_IP="<dns-service-ip>"
DNS_NAME_PREFIX="<dns-name-prefix>"
SERVICE_CIDR="<service-cidr>"
MIN_NODES="<min-count>"
MAX_NODES="<max-count>"
MAX_PODS_PER_NODE="<max-pods>"
DISK_SIZE="<node-osdisk-size>"
AZURE_CLI_VERSION="<azure-cli-version>"
NODE_OSDISK_TYPE="<node-osdisk-type>"
OS_DISK_ENCRYPTIONSET_ID="<node-osdisk-diskencryptionset-id>"
ENABLE_CLUSTER_AUTOSCALER="<enable-cluster-autoscaler>"
CLUSTER_AUTOSCALER_PROFILE="<cluster-autoscaler-profile>"
ATTACH_ACR="<attach-acr>"
ENABLE_AAD="<enable-aad>"
AAD_ADMIN_GROUP_OBJECT_IDS="<aad-admin-group-object-ids>"
AAD_CLIENT_APP_ID="<aad-client-app-id>"
AAD_SERVER_APP_ID="<aad-server-app-id>"
AAD_SERVER_APP_SECRET="<aad-server-app-secret>"
AAD_TENANT_ID="<tenant-id>"
ENABLE_POD_SECURITY_POLICY="<enable-pod-security-policy>"
DISABLE_RBAC="<disable-rbac>"
ENABLE_NODE_PUBLIC_IP="<enable-node-public-ip>"
SSH_PUB_KEY_VALUE="<ssh-key-value>"
API_SERVER_AUTHORIZED_IP_RANGES="<api-server-authorized-ip-ranges>"
NODEPOOL_LABELS="<nodepool-labels>"
PPG=${PPG:-"<name>"}
PPG_TYPE=${PPG_TYPE:-"<type>"}
UPTIME_SLA="<uptime-sla>"
OUTBOUND_TYPE="<outbound-type>"
ENABLE_MANAGED_IDENTITY
Indicates whether to use a system-assigned managed identity for cluster resource management. When enabled, this identity is used to create the K8s cluster resources. In addition, if Managed Identity is enabled, the Service Principal parameters (SP, SP_VALIDITY_YEARS, SP_ID, and SP_SECRET) are not required.
SP
The Service Principal to use for the AKS cluster. If you want to use an existing Service Principal, specify the name for that principal. If you want to create a new Service Principal, specify a new name, and the new Service Principal will be created when the cluster is created. For example, aks-service-principal.
SP_VALIDITY_YEARS
The number of years for which the Service Principal credentials should be valid. For example, 2.
SP_ID
The ID for the existing Service Principal. Leave this value blank if you chose to create a new principal.
SP_SECRET
The secret for the existing Service Principal. Leave this value blank if you chose to create a new principal.
RESOURCE_GROUP
The name of the Azure Resource Group to allocate the AKS cluster resources to. You can specify the name of an existing group, or you can specify a new name if you want the K8s scripts to create a new Resource Group.
RESOURCE_GROUP_TAGS
A space-separated list of any tags (key=value pairs) to add to the Resource Group.
LOCATION
The Region code for the location where the AKS cluster will be deployed. For example, eastus.
SUBSCRIPTION_ID
The ID for your Azure subscription.
VNET_NAME
The name of the Virtual Network to provision the AKS cluster in. This value should match the name of the network that Anzo is deployed in.
VNET_CIDR
The IP address prefix in CIDR format to use for the Virtual Network.
Supply this value even if VNET_NAME is not set and a new Virtual Network will be created.
VNET_TAGS
A space-separated list of any tags (in key=value format) to add to the Virtual Network.
VNET_VM_PROTECTION
A true or false value that indicates whether to enable VM protection for the subnets in the Virtual Network.
SUBNET_NAME
The name of the new subnetwork to create in the Virtual Network.
SUBNET_CIDR
The IP address prefix in CIDR format for the new subnetwork.
NODE_ZONES
The number of Availability Zones to place the agent nodes in. Valid values are 1, 2, or 3.
NODEPOOL_NAME
The name to give the default node pool that is created in the AKS cluster.
NODEPOOL_TAGS
A space-separated list of any tags (in key=value format) to add to resources in the default node pool.
MACHINE_TYPE
The Virtual Machine Type to use for the nodes in the AKS cluster.
K8S_CLUSTER_NAME
The name to give the AKS cluster.
K8S_CLUSTER_VERSION
The version of Kubernetes to use for creating the cluster.
Kubernetes versions 1.18 and 1.19 are supported. See the AKS Engine Release Notes for details about the available versions.
K8S_CLUSTER_NODE_COUNT
The number of nodes to deploy in the default node pool.
K8S_NODE_ADMIN_USER
The user account to create on the K8s cluster nodes for SSH access.
AKS_TAGS
A space-separated list of any tags (in key=value format) to add to the cluster.
AKS_ENABLE_ADDONS
A comma-separated list of addons to enable for the AKS cluster. Cambridge Semantics recommends that you include the monitoring addon.
PRIVATE_CLUSTER
Indicates whether to make the AKS cluster a private cluster. If the cluster is private, network traffic between the K8s API server and node pools remains on the private network.
When deciding whether to configure the cluster as a private cluster, you may want to review the Limitations described in "Create a private Azure Kubernetes Service cluster" in the Azure AKS documentation.
LOAD_BALANCER_SKU
The Azure Load Balancer SKU selection for your cluster. The options are basic or standard. The standard SKU is recommended for AKS clusters. For information about the SKUs, see Azure Load Balancer SKUs in the Azure documentation.
LB_BALANCER_IDLE_TIMEOUT
This optional parameter specifies the number of minutes to wait before dropping idle connections to the Load Balancer. For example, a value of 5 means that idle connections are dropped after 5 minutes. If this parameter is not specified, the default value is 30 minutes.
For more information about configuring the Load Balancer, including details about the idle timeout parameter as well as the outbound IP address and port parameters, see Configure the Public Standard Load Balancer in the Azure AKS documentation.
LB_OUTBOUND_IP_PREFIXES
This optional parameter specifies a comma-separated list of outbound IP prefix resource IDs.
LB_OUTBOUND_IPS
This optional parameter specifies a comma-separated list of outbound IP resource IDs.
LB_OUTBOUND_PORTS
This optional parameter specifies the number of outbound ports to allocate for the Load Balancer. For example, 8000.
LB_MANAGED_OUTBOUND_IP_COUNT
This optional parameter specifies the number of AKS-managed outbound IP addresses to allocate for the Load Balancer. For example, 10.
VM_SET_TYPE
The Agent pool VM set type. Valid values are VirtualMachineScaleSets or AvailabilitySet. Cambridge Semantics recommends that you set this value to VirtualMachineScaleSets.
NETWORK_PLUGIN
The type of Kubernetes network plugin to use, i.e. whether to use basic (kubenet) networking or advanced CNI (azure) networking. Valid values are kubenet or azure.
NETWORK_POLICY
The type of the network policy (Azure Network Policies or Calico Network Policies) to apply to the pods in the AKS cluster. The network policy defines the rules for ingress and egress traffic between pods in the cluster. Valid values are azure or calico. For information about the policies, see Network Policy Options in AKS in the Azure AKS documentation.
DOCKER_BRIDGE_ADDRESS
The CIDR block to use for the Docker bridge. The Docker bridge is not used by the AKS cluster or pods but does need to be set up since Docker is configured as part of the Kubernetes setup. Choose an address space that does not collide with any other CIDRs on your networks, including the cluster's service CIDR and pod CIDR. For example, 172.17.0.1/16.
DNS_SERVICE_IP
The IP address to assign to the Kubernetes DNS service.
DNS_NAME_PREFIX
This optional parameter specifies the prefix to use for hostnames that are created for the DNS service. If not specified, a hostname is generated using the managed cluster and resource group names.
SERVICE_CIDR
The IP address range in CIDR notation from which to assign the Kubernetes DNS service IP addresses.
MIN_NODES
The minimum number of nodes in the default node pool.
MAX_NODES
The maximum number of nodes in the default node pool.
MAX_PODS_PER_NODE
The maximum number of pods deployable to a node in the default node pool.
DISK_SIZE
The size in GB of the OS disk for each node in the default node pool.
AZURE_CLI_VERSION
The version of the Azure CLI on the workstation. For example, 2.25.0.
NODE_OSDISK_TYPE
The type of OS disk to use for machines in the cluster. The options are Ephemeral or Managed.
OS_DISK_ENCRYPTIONSET_ID
Specifies the Resource ID of the disk encryption set to use for encryption at rest on the agent node OS disk.
ENABLE_CLUSTER_AUTOSCALER
Indicates whether to enable the cluster autoscaler for the default node pool.
CLUSTER_AUTOSCALER_PROFILE
A space-separated list of any key=value pairs to use for configuring the Cluster Autoscaler. For example, scan-interval=10s scale-down-delay-after-delete=10s. For information about all of the configuration options, see Using the Autoscaler Profile in the Azure AKS documentation.
ATTACH_ACR
The name or resource ID of the Azure Container Registry to grant the acrpull
role assignment to.
ENABLE_AAD
Indicates whether to enable managed Azure Active Directory (AAD) for the cluster. When AAD is enabled, the Admin Group Object IDs, AAD Client ID, Server ID, Server Secret, and Tenet ID parameters (AAD_ADMIN_GROUP_OBJECT_IDS, AAD_CLIENT_APP_ID, AAD_SERVER_APP_ID, AAD_SERVER_APP_SECRET, and AAD_TENANT_ID) are not required.
AAD_ADMIN_GROUP_OBJECT_IDS
This parameter specifies the comma-separated list of AAD group object IDs to set as cluster admin.
AAD_CLIENT_APP_ID
The ID of a "Native" type Azure Active Directory client application. This application is for user logins via kubectl.
AAD_SERVER_APP_ID
The ID of a "Web app/API" Azure Active Directory server application. This application represents the managed cluster's API Server (apiserver application).
AAD_SERVER_APP_SECRET
The secret for the Azure Active Directory server application.
AAD_TENANT_ID
The ID of the Azure Active Directory tenant.
ENABLE_POD_SECURITY_POLICY
Indicates whether to enable the pod security policy for the AKS cluster.
Azure will deprecate this feature in June 2021. For information, see Secure your cluster using pod security policies in Azure Kubernetes Service (AKS) in the Azure AKS documentation.
DISABLE_RBAC
Indicates whether to disable Kubernetes Role-Based Access Control (RBAC).
ENABLE_NODE_PUBLIC_IP
Indicates whether to enable a public IP address for the Virtual Machine Scale Set (VMSS) node.
SSH_PUB_KEY_VALUE
The public key path or key contents to install on the K8s cluster nodes for SSH access. If not specified, the default value is ~\.ssh\id_rsa.pub.
API_SERVER_AUTHORIZED_IP_RANGES
The list of IP address ranges in CIDR notation that are authorized to access the AKS cluster.
NODEPOOL_LABELS
A space-separated list (in key=value format) of labels to add to the nodes in the default node pool. For information about using labels in Kubernetes clusters, see Labels and Selectors in the Kubernetes documentation.
PPG
This optional parameter specifies the name of the Proximity Placement Group (PPG) to use for the cluster. For information about using proximity placement groups, see Use Proximity Placement Groups in the Azure AKS documentation.
PPG_TYPE
If using a Proximity Placement Group (PPG), this parameter specifies the type of PPG to use. The only valid value is Standard.
UPTIME_SLA
Indicates whether to enable a paid managed cluster service with a financially backed SLA.
OUTBOUND_TYPE
Specifies how to configure outbound traffic for the cluster. Valid values are loadBalancer and userDefinedRouting.
Example Configuration File
An example completed k8s_cluster.conf file is shown below.
ENABLE_MANAGED_IDENTITY="true" #SP=${SP:-"aks-service-principal"} #SP_VALIDITY_YEARS="2" #SP_ID="291bba3f-e0a5-47bc-a099-3bdcb2a50a05" #SP_SECRET="ValidServicePrincipalSecretIfPresent" RESOURCE_GROUP=${RESOURCE_GROUP:-"aks-resource-group"} RESOURCE_GROUP_TAGS="description=aks-cluster" LOCATION=${LOCATION:-"eastus"} SUBSCRIPTION_ID="ValidSubscriptionId" VNET_NAME=${VNET_NAME:-"anzo-vnet"} VNET_CIDR="20.20.0.0/16" VNET_TAGS="description=aks-virtual-network" VNET_VM_PROTECTION="true" SUBNET_NAME="k8s-subnet" SUBNET_CIDR="20.20.0.0/19" #NODE_ZONES="" NODEPOOL_NAME="defaultpool" NODEPOOL_TAGS="description=default-nodepool" MACHINE_TYPE="Standard_DS1_v2" K8S_CLUSTER_NAME=${K8S_CLUSTER_NAME:-"k8s-cluster"} K8S_CLUSTER_VERSION=${K8S_CLUSTER_VERSION:-"1.18"} K8S_CLUSTER_NODE_COUNT="2" K8S_NODE_ADMIN_USER="azureuser" AKS_TAGS="description=aks-cluster" AKS_ENABLE_ADDONS="monitoring" PRIVATE_CLUSTER="false" LOAD_BALANCER_SKU="standard" #LB_BALANCER_IDLE_TIMEOUT=5 #LB_OUTBOUND_IP_PREFIXES="<ip-prefix-resource-id-1,ip-prefix-resource-id-2>" #LB_OUTBOUND_IPS="<ip-resource-id-1,ip-resource-id-2>" #LB_OUTBOUND_PORTS=8000 #LB_MANAGED_OUTBOUND_IP_COUNT=10 VM_SET_TYPE="VirtualMachineScaleSets" NETWORK_PLUGIN="azure" NETWORK_POLICY="azure" DOCKER_BRIDGE_ADDRESS="172.17.0.1/16" DNS_SERVICE_IP="10.0.0.10" #DNS_NAME_PREFIX="k8stest" SERVICE_CIDR="10.0.0.0/16" MIN_NODES="1" MAX_NODES="8" MAX_PODS_PER_NODE="16" DISK_SIZE="100" AZURE_CLI_VERSION="2.19.1" NODE_OSDISK_TYPE="Ephemeral" #OS_DISK_ENCRYPTIONSET_ID="" ENABLE_CLUSTER_AUTOSCALER="true" CLUSTER_AUTOSCALER_PROFILE="scan-interval=10s scale-down-delay-after-delete=10s" ATTACH_ACR="ContainerRegistry" ENABLE_AAD="true" AAD_ADMIN_GROUP_OBJECT_IDS="5d24455a-1111-3333-4444-5dv77afa27aed" #AAD_CLIENT_APP_ID="ValidAADClientAppId" #AAD_SERVER_APP_ID="ValidAADServerAppId" #AAD_SERVER_APP_SECRET="ValidAADServerAppSecret" #AAD_TENANT_ID="8f70baf1-1f6e-46a2-a1ff-238dac1ebfb7" ENABLE_POD_SECURITY_POLICY="true" ENABLE_MANAGED_IDENTITY="false" DISABLE_RBAC="false" SSH_PUB_KEY_VALUE="" API_SERVER_AUTHORIZED_IP_RANGES="10.107.1.0/24" NODEPOOL_LABELS="description=k8scluster" #PPG=${PPG:-"csippg"} #PPG_TYPE=${PPG_TYPE:-"Standard"} UPTIME_SLA="false" OUTBOUND_TYPE="loadBalancer"
Create the AKS Cluster
After defining the cluster requirements, run the create_k8s.sh script in the az
directory to create the cluster. Run the script with the following command. The arguments are described below.
./create_k8s.sh -c <config_file_name> [ -d <config_file_directory> ] [ -f | --force ] [ -h | --help ]
For example, the following command runs the create_k8s script, using k8s_cluster.conf as input to the script. Since k8s_cluster.conf is in the conf.d directory, the -d argument is excluded:
./create_k8s.sh -c k8s_cluster.conf
The script validates that the required software packages, such as the Azure CLI and kubectl, are installed and that the versions are compatible with the script. It also displays an overview of the deployment details based on the values in the specified configuration file.
The script then prompts you to proceed with deploying each component of the AKS cluster infrastructure. Type y and press Enter to proceed with each step in creating the specified Service Principal, Virtual Network, subnetwork, and Load Balancer components. All components are created according to the specifications in the configuration file.
When cluster creation is complete, proceed to Creating the Required Node Pools to add the required node pools to the cluster.