Creating the AKS Cluster

Follow the instructions below to define the AKS cluster resource requirements and then create the cluster based on your specifications.

• Define the AKS Cluster Requirements
• Create the AKS Cluster

Define the AKS Cluster Requirements

The first step in creating the K8s cluster is to define the infrastructure specifications. The k8s_cluster.conf file in the az/conf.d directory is a sample cluster configuration file that you can use as a template, or you can edit the file directly. The contents of k8s_cluster.conf are shown below. Descriptions of the cluster parameters follow the contents.

ENABLE_MANAGED_IDENTITY="<enable-managed-identity>"
SP=${SP:-"<service-principal>"}
SP_VALIDITY_YEARS="<years>"
SP_ID="<id>"
SP_SECRET="<client-secret>"
RESOURCE_GROUP=${RESOURCE_GROUP:-"<resource-group>"}
RESOURCE_GROUP_TAGS="<tags>"
LOCATION=${LOCATION:-"<location>"}
SUBSCRIPTION_ID="<subscription-id>"
VNET_NAME=${VNET_NAME:-"<name>"}
VNET_CIDR="<vnet-cidr>"
VNET_TAGS="<tags>"
VNET_VM_PROTECTION="<vm-protection>"
SUBNET_NAME="<subnet-name>"
SUBNET_CIDR="<subnet-cidr>"
NODEPOOL_NAME="<name>"
NODEPOOL_TAGS="<tags>"
MACHINE_TYPE="<machine-type>"
K8S_CLUSTER_NAME=${K8S_CLUSTER_NAME:-"<name>"}
K8S_CLUSTER_VERSION=${K8S_CLUSTER_VERSION:-"<kubernetes-version>"}
K8S_CLUSTER_NODE_COUNT="<node-count>"
K8S_NODE_ADMIN_USER="<admin-username>"
AKS_TAGS="<tags>"
AKS_ENABLE_ADDONS="<addons>"
PRIVATE_CLUSTER="<enable-private-cluster>"
LOAD_BALANCER_SKU="<load-balancer-sku>"
LB_BALANCER_IDLE_TIMEOUT=<load-balancer-idle-timeout>
LB_OUTBOUND_IP_PREFIXES="<load-balancer-outbound-ip-prefixes>"
LB_OUTBOUND_IPS="<load-balancer-outbound-ips>"
LB_OUTBOUND_PORTS=<load-balancer-outbound-ports>
LB_MANAGED_OUTBOUND_IP_COUNT=<load-balancer-managed-outbound-ip-count>
VM_SET_TYPE="<vm-set-type>"
NETWORK_PLUGIN="<network-plugin>"
NETWORK_POLICY="<network-policy>"
DOCKER_BRIDGE_ADDRESS="<docker-bridge-address>"
DNS_SERVICE_IP="<dns-service-ip>"
DNS_NAME_PREFIX="<dns-name-prefix>"
SERVICE_CIDR="<service-cidr>"
MIN_NODES="<min-count>"
MAX_NODES="<max-count>"
MAX_PODS_PER_NODE="<max-pods>"
DISK_SIZE="<node-osdisk-size>"
AZURE_CLI_VERSION="<azure-cli-version>"
NODE_OSDISK_TYPE="<node-osdisk-type>"
ENABLE_CLUSTER_AUTOSCALER="<enable-cluster-autoscaler>"
CLUSTER_AUTOSCALER_PROFILE="<cluster-autoscaler-profile>"
ATTACH_ACR="<attach-acr>"
ENABLE_AAD="<enable-aad>"
AAD_ADMIN_GROUP_OBJECT_IDS="<aad-admin-group-object-ids>"
AAD_CLIENT_APP_ID="<aad-client-app-id>"
AAD_SERVER_APP_ID="<aad-server-app-id>"
AAD_SERVER_APP_SECRET="<aad-server-app-secret>"
AAD_TENANT_ID="<tenant-id>"
ENABLE_POD_SECURITY_POLICY="<enable-pod-security-policy>"
ENABLE_RBAC="<enable-rbac>"
DISABLE_RBAC="<disable-rbac>"
ENABLE_NODE_PUBLIC_IP="<enable-node-public-ip>"
SSH_PUB_KEY_VALUE="<ssh-key-value>"
AAPI_SERVER_AUTHORIZED_IP_RANGES="<api-server-authorized-ip-ranges>"
NODE_LABELS="<nodepool-labels>"
PPG=${PPG:-"<name>"}
PPG_TYPE=${PPG_TYPE:-"<type>"}
UPTIME_SLA="<uptime-sla>"
OUTBOUND_TYPE="<outbound-type>"

ENABLE_MANAGED_IDENTITY

Indicates whether to use a system-assigned managed identity for cluster resource management. When enabled, this identity is used to create the K8s cluster resources. In addition, if Managed Identity is enabled, the Service Principal parameters (SP, SP_VALIDITY_YEARS, SP_ID, and SP_SECRET) are not required.

SP

The Service Principal to use for the AKS cluster. If you want to use an existing Service Principal, specify the name for that principal. If you want to create a new Service Principal, specify a new name, and the new Service Principal will be created when the cluster is created. For example, aks-service-principal.

SP_VALIDITY_YEARS

The number of years for which the Service Principal credentials should be valid. For example, 2.

SP_ID

The ID for the existing Service Principal. Leave this value blank if you chose to create a new principal.

SP_SECRET

The secret for the existing Service Principal. Leave this value blank if you chose to create a new principal.

RESOURCE_GROUP

The name of the Azure Resource Group to allocate the AKS cluster resources to. You can specify the name of an existing group, or you can specify a new name if you want the K8s scripts to create a new Resource Group.

RESOURCE_GROUP_TAGS

A space-separated list of any tags (key=value pairs) to add to the Resource Group.

LOCATION

The Region code for the location where the AKS cluster will be deployed. For example, eastus.

SUBSCRIPTION_ID

The ID for your Azure subscription.

VNET_NAME

The name of the Virtual Network to provision the AKS cluster in. This value should match the name of the network that Anzo is deployed in.

VNET_CIDR

The IP address prefix in CIDR format to use for the Virtual Network.

VNET_TAGS

A space-separated list of any tags (in key=value format) to add to the Virtual Network.

VNET_VM_PROTECTION

A true or false value that indicates whether to enable VM protection for the subnets in the Virtual Network.

SUBNET_NAME

The name of the new subnetwork to create in the Virtual Network.

SUBNET_CIDR

The IP address prefix in CIDR format for the new subnetwork.

NODEPOOL_NAME

The name to give the default node pool that is created in the AKS cluster.

NODEPOOL_TAGS

A space-separated list of any tags (in key=value format) to add to resources in the default node pool.

MACHINE_TYPE

The Virtual Machine Type to use for the nodes in the AKS cluster.

K8S_CLUSTER_NAME

The name to give the AKS cluster.

K8S_CLUSTER_VERSION

The version of Kubernetes to use for creating the cluster.

K8S_CLUSTER_NODE_COUNT

The number of nodes to deploy in the default node pool.

K8S_NODE_ADMIN_USER

The user account to create on the K8s cluster nodes for SSH access.

AKS_TAGS

A space-separated list of any tags (in key=value format) to add to the cluster.

AKS_ENABLE_ADDONS

A comma-separated list of addons to enable for the AKS cluster. Cambridge Semantics recommends that you include the monitoring addon.

PRIVATE_CLUSTER

Indicates whether to make the AKS cluster a private cluster. If the cluster is private, network traffic between the K8s API server and node pools remains on the private network.

LOAD_BALANCER_SKU

The Azure Load Balancer SKU selection for your cluster. The options are basic or standard. The standard SKU is recommended for AKS clusters. For information about the SKUs, see Azure Load Balancer SKUs in the Azure documentation.

LB_BALANCER_IDLE_TIMEOUT

This optional parameter specifies the number of minutes to wait before dropping idle connections to the Load Balancer. For example, a value of 5 means that idle connections are dropped after 5 minutes. If this parameter is not specified, the default value is 30 minutes.

LB_OUTBOUND_IP_PREFIXES

This optional parameter specifies a comma-separated list of outbound IP prefix resource IDs.

LB_OUTBOUND_IPS

This optional parameter specifies a comma-separated list of outbound IP resource IDs.

LB_OUTBOUND_PORTS

This optional parameter specifies the number of outbound ports to allocate for the Load Balancer. For example, 8000.

LB_MANAGED_OUTBOUND_IP_COUNT

This optional parameter specifies the number of AKS-managed outbound IP addresses to allocate for the Load Balancer. For example, 10.

VM_SET_TYPE

The Agent pool VM set type. Valid values are VirtualMachineScaleSets or AvailabilitySet. Cambridge Semantics recommends that you set this value to VirtualMachineScaleSets.

NETWORK_PLUGIN

The type of Kubernetes network plugin to use, i.e. whether to use basic (kubenet) networking or advanced CNI (azure) networking. Valid values are kubenet or azure.

NETWORK_POLICY

The type of the network policy (Azure Network Policies or Calico Network Policies) to apply to the pods in the AKS cluster. The network policy defines the rules for ingress and egress traffic between pods in the cluster. Valid values are azure or calico. For information about the policies, see Network Policy Options in AKS in the Azure AKS documentation.

DOCKER_BRIDGE_ADDRESS

The CIDR block to use for the Docker bridge. The Docker bridge is not used by the AKS cluster or pods but does need to be set up since Docker is configured as part of the Kubernetes setup. Choose an address space that does not collide with any other CIDRs on your networks, including the cluster's service CIDR and pod CIDR. For example, 172.17.0.1/16.

DNS_SERVICE_IP

The IP address to assign to the Kubernetes DNS service.

DNS_NAME_PREFIX

This optional parameter specifies the prefix to use for hostnames that are created for the DNS service. If not specified, a hostname is generated using the managed cluster and resource group names.

SERVICE_CIDR

The IP address range in CIDR notation from which to assign the Kubernetes DNS service IP addresses.

MIN_NODES

The minimum number of nodes in the default node pool.

MAX_NODES

The maximum number of nodes in the default node pool.

MAX_PODS_PER_NODE

The maximum number of pods deployable to a node in the default node pool.

DISK_SIZE

The size in GB of the OS disk for each node in the default node pool.

AZURE_CLI_VERSION

The version of the Azure CLI on the workstation. For example, 2.19.1.

NODE_OSDISK_TYPE

The type of OS disk to use for machines in the cluster. The options are Ephemeral or Managed.

ENABLE_CLUSTER_AUTOSCALER

Indicates whether to enable the cluster autoscaler for the default node pool.

CLUSTER_AUTOSCALER_PROFILE

A space-separated list of any key=value pairs to use for configuring the Cluster Autoscaler. For example, scan-interval=10s scale-down-delay-after-delete=10s. For information about all of the configuration options, see Using the Autoscaler Profile in the Azure AKS documentation.

ATTACH_ACR

The name or resource ID of the Azure Container Registry to grant the acrpull role assignment to.

ENABLE_AAD

Indicates whether to enable managed Azure Active Directory (AAD) for the cluster. When AAD is enabled and Admin Group Object IDs are provided in AAD_ADMIN_GROUP_OBJECT_IDS, the AAD Client ID, Server ID, Server Secret, and Tenet ID parameters (AAD_CLIENT_APP_ID, AAD_SERVER_APP_ID, AAD_SERVER_APP_SECRET, and AAD_TENANT_ID) are not required.

AAD_ADMIN_GROUP_OBJECT_IDS

When AAD is enabled (ENABLE_AAD="true"), this parameters specifies the comma-separated list of AAD group object IDs to set as cluster admin.

AAD_CLIENT_APP_ID

The ID of a "Native" type Azure Active Directory client application. This application is for user logins via kubectl.

AAD_SERVER_APP_ID

The ID of a "Web app/API" Azure Active Directory server application. This application represents the managed cluster's API Server (apiserver application).

AAD_SERVER_APP_SECRET

The secret for the Azure Active Directory server application.

AAD_TENANT_ID

The ID of the Azure Active Directory tenant.

ENABLE_POD_SECURITY_POLICY

Indicates whether to enable the pod security policy for the AKS cluster.

ENABLE_RBAC

Indicates whether to enable Kubernetes Role-Based Access Control (RBAC). You can include either this parameter or DISABLE_RBAC for enabling or disabling RBAC.

DISABLE_RBAC

Indicates whether to disable Kubernetes Role-Based Access Control (RBAC).

ENABLE_NODE_PUBLIC_IP

Indicates whether to enable a public IP address for the Virtual Machine Scale Set (VMSS) node.

SSH_PUB_KEY_VALUE

The public key path or key contents to install on the K8s cluster nodes for SSH access. If not specified, the default value is ~\.ssh\id_rsa.pub.

AAPI_SERVER_AUTHORIZED_IP_RANGES

The list of IP address ranges in CIDR notation that are authorized to access the AKS cluster.

NODE_LABELS

A space-separated list (in key=value format) of labels to add to the nodes in the default node pool. For information about using labels in Kubernetes clusters, see Labels and Selectors in the Kubernetes documentation.

PPG

This optional parameter specifies the name of the Proximity Placement Group (PPG) to use for the cluster. For information about using proximity placement groups, see Use Proximity Placement Groups in the Azure AKS documentation.

PPG_TYPE

If using a Proximity Placement Group (PPG), this parameter specifies the type of PPG to use. The only valid value is Standard.

UPTIME_SLA

Indicates whether to enable a paid managed cluster service with a financially backed SLA.

OUTBOUND_TYPE

Specifies how to configure outbound traffic for the cluster. Valid values are loadBalancer and userDefinedRouting.

Example Configuration File

An example completed k8s_cluster.conf file is shown below.

ENABLE_MANAGED_IDENTITY="true"
#SP=${SP:-"aks-service-principal"}
#SP_VALIDITY_YEARS="2"
#SP_ID="291bba3f-e0a5-47bc-a099-3bdcb2a50a05"
#SP_SECRET="ValidServicePrincipalSecretIfPresent"
RESOURCE_GROUP=${RESOURCE_GROUP:-"aks-resource-group"}
RESOURCE_GROUP_TAGS="description=aks-cluster"
LOCATION=${LOCATION:-"eastus"}
SUBSCRIPTION_ID="ValidSubscriptionId"
VNET_NAME=${VNET_NAME:-"anzo-vnet"}
VNET_CIDR="20.20.0.0/16"
VNET_TAGS="description=aks-virtual-network"
VNET_VM_PROTECTION="true"
SUBNET_NAME="k8s-subnet"
SUBNET_CIDR="20.20.0.0/19"
NODEPOOL_NAME="defaultpool"
NODEPOOL_TAGS="description=default-nodepool"
MACHINE_TYPE="Standard_DS1_v2"
K8S_CLUSTER_NAME=${K8S_CLUSTER_NAME:-"k8s-cluster"}
K8S_CLUSTER_VERSION=${K8S_CLUSTER_VERSION:-"1.18"}
K8S_CLUSTER_NODE_COUNT="2"
K8S_NODE_ADMIN_USER="azureuser"
AKS_TAGS="description=aks-cluster"
AKS_ENABLE_ADDONS="monitoring"
PRIVATE_CLUSTER="false"
LOAD_BALANCER_SKU="standard"
#LB_BALANCER_IDLE_TIMEOUT=5
#LB_OUTBOUND_IP_PREFIXES="<ip-prefix-resource-id-1,ip-prefix-resource-id-2>"
#LB_OUTBOUND_IPS="<ip-resource-id-1,ip-resource-id-2>"
#LB_OUTBOUND_PORTS=8000
#LB_MANAGED_OUTBOUND_IP_COUNT=10
VM_SET_TYPE="VirtualMachineScaleSets"
NETWORK_PLUGIN="azure"
NETWORK_POLICY="azure"
DOCKER_BRIDGE_ADDRESS="172.17.0.1/16"
DNS_SERVICE_IP="10.0.0.10"
#DNS_NAME_PREFIX="k8stest"
SERVICE_CIDR="10.0.0.0/16"
MIN_NODES="1"
MAX_NODES="8"
MAX_PODS_PER_NODE="16"
DISK_SIZE="100"
AZURE_CLI_VERSION="2.19.1"
NODE_OSDISK_TYPE="Ephemeral"
ENABLE_CLUSTER_AUTOSCALER="true"
CLUSTER_AUTOSCALER_PROFILE="scan-interval=10s scale-down-delay-after-delete=10s"
ATTACH_ACR="ContainerRegistry"
ENABLE_AAD="true"
AAD_ADMIN_GROUP_OBJECT_IDS="5d24455a-1111-3333-4444-5dv77afa27aed"
#AAD_CLIENT_APP_ID="ValidAADClientAppId"
#AAD_SERVER_APP_ID="ValidAADServerAppId"
#AAD_SERVER_APP_SECRET="ValidAADServerAppSecret"
#AAD_TENANT_ID="8f70baf1-1f6e-46a2-a1ff-238dac1ebfb7"
ENABLE_POD_SECURITY_POLICY="true"
ENABLE_MANAGED_IDENTITY="false"
ENABLE_RBAC="true"
DISABLE_RBAC="false"
SSH_PUB_KEY_VALUE=""
AAPI_SERVER_AUTHORIZED_IP_RANGES="10.107.1.0/24"
NODE_LABELS="description=k8scluster"
#PPG=${PPG:-"csippg"}
#PPG_TYPE=${PPG_TYPE:-"Standard"}
UPTIME_SLA="false"
OUTBOUND_TYPE="loadBalancer"

Create the AKS Cluster

After defining the cluster requirements, run the create_k8s.sh script in the az directory to create the cluster. Run the script with the following command. The arguments are described below.

./create_k8s.sh -c <config_file_name> [ -d <config_file_directory> ] [ -f | --force ] [ -h | --help ]

-c <config_file_name>

This is a required argument that specifies the name of the configuration file that supplies the cluster requirements. For example, -c k8s_cluster.conf.

-d <config_file_directory>

This is an optional argument that specifies the path and directory name for the configuration file specified for the -c argument. If you are using the original az directory file structure and the configuration file is in the conf.d directory, you do not need to specify the -d argument. If you created a separate directory structure for different Anzo environments, include the -d option. For example, -d /az/env1/conf.

-f | --force

This is an optional argument that controls whether the script prompts for confirmation before proceeding with each stage involved in creating the cluster. If -f (--force) is specified, the script assumes the answer is "yes" to all prompts and does not display them.

-h | --help

This argument is an optional flag that you can specify to display the help from the create_k8s.sh script.

For example, the following command runs the create_k8s script, using k8s_cluster.conf as input to the script. Since k8s_cluster.conf is in the conf.d directory, the -d argument is excluded:

./create_k8s.sh -c k8s_cluster.conf

The script validates that the required software packages, such as the Azure CLI and kubectl, are installed and that the versions are compatible with the script. It also displays an overview of the deployment details based on the values in the specified configuration file.

The script then prompts you to proceed with deploying each component of the AKS cluster infrastructure. Type y and press Enter to proceed with each step in creating the specified Service Principal, Virtual Network, subnetwork, and Load Balancer components. All components are created according to the specifications in the configuration file.

When cluster creation is complete, proceed to Creating the Required Node Pools to add the required node pools to the cluster.

Related Topics

• Creating and Assigning IAM Roles
• Creating the Required Node Pools