Skip to main content
Service Catalog Version 0.90.0Last updated in version 0.87.0

Amazon EKS Core Services

View SourceRelease Notes

Overview​

This service contains Terraform and Helm code to deploy core administrative services, such as FluentD and the ALB Ingress Controller, onto Elastic Kubernetes Service(EKS).

EKS Core Services architectureEKS Core Services architecture

Features​

  • Deploy FluentD DaemonSet to ship container logs to CloudWatch Logs
  • Deploy ALB Ingress Controller to configure ALBs from within Kubernetes
  • Deploy external-dns to manage Route 53 DNS records from within Kubernetes
  • Deploy Kubernetes cluster-autoscaler to configure auto scaling of ASGs based on Pod demand
  • Deploy AWS CloudWatch Agent to configure container and node level metrics from worker nodes

Learn​

note

This repo is a part of the Gruntwork Service Catalog, a collection of reusable, battle-tested, production ready infrastructure code. If you’ve never used the Service Catalog before, make sure to read How to use the Gruntwork Service Catalog!

Under the hood, this is all implemented using Terraform modules from the Gruntwork terraform-aws-eks repo. If you are a subscriber and don’t have access to this repo, email support@gruntwork.io.

Core concepts​

For information on each of the core services deployed by this service, see the documentation in the terraform-aws-eks repo.

Repo organization​

  • modules: the main implementation code for this repo, broken down into multiple standalone, orthogonal submodules.
  • examples: This folder contains working examples of how to use the submodules.
  • test: Automated tests for the modules and examples.

Deploy​

Non-production deployment (quick start for learning)​

If you just want to try this repo out for experimenting and learning, check out the following resources:

  • examples/for-learning-and-testing folder: The examples/for-learning-and-testing folder contains standalone sample code optimized for learning, experimenting, and testing (but not direct production usage).

Production deployment​

If you want to deploy this repo in production, check out the following resources:

Reference​

Required​

aws_regionstringrequired

The AWS region in which all resources will be created

eks_cluster_namestringrequired

The name of the EKS cluster where the core services will be deployed into.

Configuration for using the IAM role with Service Accounts feature to provide permissions to the applications. This expects a map with two properties: openid_connect_provider_arn and openid_connect_provider_url. The openid_connect_provider_arn is the ARN of the OpenID Connect Provider for EKS to retrieve IAM credentials, while openid_connect_provider_url is the URL. Set to null if you do not wish to use IAM role with Service Accounts.

object({
    openid_connect_provider_arn = string
    openid_connect_provider_url = string
  })

ARN of IAM Role to use as the Pod execution role for Fargate. Required if any of the services are being scheduled on Fargate. Set to null if none of the Pods are being scheduled on Fargate.

vpc_idstringrequired

The ID of the VPC where the EKS cluster is deployed.

worker_vpc_subnet_idslist(string)required

The subnet IDs to use for EKS worker nodes. Used when provisioning Pods on to Fargate. Required if any of the services are being scheduled on Fargate. Set to empty list if none of the Pods are being scheduled on Fargate.

Optional​

The version of the aws-load-balancer-controller helmchart to use.

"1.4.1"

The repository of the aws-load-balancer-controller docker image that should be deployed.

"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller"

The tag of the aws-load-balancer-controller docker image that should be deployed.

"v2.4.1"

Configure affinity rules for the ALB Ingress Controller Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]

Configure tolerations rules to allow the ALB Ingress Controller Pod to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]

Minimum time to wait after a scale up event before any node is considered for scale down.

"10m"

Number for the log level verbosity. Lower numbers are less verbose, higher numbers are more verbose. (Default: 4)

4

Minimum time to wait since the node became unused before the node is considered for scale down by the autoscaler.

"10m"

If true cluster autoscaler will never delete nodes with pods with local storage, e.g. EmptyDir or HostPath

true

The Container repository to use for looking up the cloudwatch-agent Container image when deploying the pods. When null, uses the default repository set in the chart. Only applies to non-fargate workers.

null

Configure affinity rules for the AWS CloudWatch Agent Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]

Pod resource requests and limits to use. Refer to https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more information.

Any types represent complex values of variable type. For details, please consult `variables.tf` in the source repo.
null

Configure tolerations rules to allow the AWS CloudWatch Agent Pods to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]

Which version of amazon/cloudwatch-agent to install. When null, uses the default version set in the chart. Only applies to non-fargate workers.

null

Annotations to apply to the cluster autoscaler pod(s), as key value pairs.

{}

Labels to apply to the cluster autoscaler pod(s), as key value pairs.

{}
cluster_autoscaler_pod_node_affinitylist(object(…))optional

Configure affinity rules for the cluster-autoscaler Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]

Pod resource requests and limits to use. Refer to https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more information. This is most useful for configuring CPU+Memory availability for Fargate, which defaults to 0.25 vCPU and 256MB RAM.

Any types represent complex values of variable type. For details, please consult `variables.tf` in the source repo.
{
  limits = {
    cpu = "250m",
    memory = "1024Mi"
  },
  requests = {
    cpu = "250m",
    memory = "1024Mi"
  }
}

Configure tolerations rules to allow the cluster-autoscaler Pod to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]

The name to use for the helm release for cluster-autoscaler. This is useful to force a redeployment of the cluster-autoscaler component.

"cluster-autoscaler"

Which docker repository to use to install the cluster autoscaler. Check the following link for valid repositories to use https://github.com/kubernetes/autoscaler/releases

"us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler"

Specifies an 'expander' for the cluster autoscaler. This helps determine which ASG to scale when additional resource capacity is needed.

"least-waste"

Which version of the cluster autoscaler to install. This should match the major/minor version (e.g., v1.20) of your Kubernetes Installation. See https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#releases for a list of versions.

"v1.22.2"

Whether or not to enable the AWS LB Ingress controller.

true

Whether to enable the AWS CloudWatch Agent DaemonSet for collecting container and node metrics from worker nodes (self-managed ASG or managed node groups).

true

Whether or not to enable cluster-autoscaler for Autoscaling EKS worker nodes.

true

Whether or not to enable external-dns for DNS entry syncing with Route 53 for Services and Ingresses.

true

Whether or not to enable fluent-bit on EKS Fargate workers for log aggregation.

true
enable_fluent_bitbooloptional

Whether or not to enable fluent-bit for log aggregation.

true

Duration string (e.g. 1m) indicating the interval between making changes to Route 53 by external-dns. When null, use the default defined in the chart (1s).

null

The maximum number of changes that should be applied in a batch by external-dns. When null, use the default defined in the chart (1000).

null

The version of the helm chart to use. Note that this is different from the app/container version.

"6.2.4"
external_dns_pod_node_affinitylist(object(…))optional

Configure affinity rules for the external-dns Pod to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]
external_dns_pod_tolerationslist(map(…))optional

Configure tolerations rules to allow the external-dns Pod to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]

Duration string (e.g. 1m) indicating the polling interval for syncing the domains by external-dns. When null, use the default defined in the chart (1m).

null

Only create records in hosted zones that match the provided domain names. Empty list (default) means match all zones. Zones must satisfy all three constraints (external_dns_route53_hosted_zone_tag_filters, external_dns_route53_hosted_zone_id_filters, and external_dns_route53_hosted_zone_domain_filters).

[]

Only create records in hosted zones that match the provided IDs. Empty list (default) means match all zones. Zones must satisfy all three constraints (external_dns_route53_hosted_zone_tag_filters, external_dns_route53_hosted_zone_id_filters, and external_dns_route53_hosted_zone_domain_filters).

[]

Only create records in hosted zones that match the provided tags. Each item in the list should specify tag key and tag value as a map. Empty list (default) means match all zones. Zones must satisfy all three constraints (external_dns_route53_hosted_zone_tag_filters, external_dns_route53_hosted_zone_id_filters, and external_dns_route53_hosted_zone_domain_filters).

list(object({
    key   = string
    value = string
  }))
[]

Duration string (e.g. 1m) indicating the amount of time the Hosted Zones are cached in external-dns. When null, use the default defined in the chart (0 - no caching).

null
external_dns_sourceslist(string)optional

K8s resources type to be observed for new DNS entries by ExternalDNS.

[
  "ingress",
  "service"
]

When enabled, triggers external-dns run loop on create/update/delete events (optional, in addition of regular interval)

false

List of ARNs of Fargate execution IAM Roles that should get permissions to ship logs using fluent-bit. This must be provided if enable_fargate_fluent_bit is true.

[]

Additional filters that fluent-bit should apply to log output. This string should be formatted according to the Fluent-bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_filter).

""

Additional parsers that fluent-bit should export logs to. This string should be formatted according to the Fluent-bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_output).

""

Prefix string to use for the CloudWatch Log Stream that gets created for each Fargate pod.

"fargate"

A list of availability zones in the region that we CANNOT use to deploy the EKS Fargate workers. You can use this to avoid availability zones that may not be able to provision the resources (e.g ran out of capacity). If empty, will allow all availability zones.

[
  "us-east-1d",
  "us-east-1e",
  "ca-central-1d"
]

Additional filters that fluent-bit should apply to log output. This string should be formatted according to the Fluent-bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_filter).

""

Additional output streams that fluent-bit should export logs to. This string should be formatted according to the Fluent-bit docs (https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_output).

""

The Container repository to use for looking up the aws-for-fluent-bit Container image when deploying the pods. When null, uses the default repository set in the chart. Only applies to non-fargate workers.

null

If set to true, that means that the CloudWatch Log Group fluent-bit should use for streaming logs already exists and does not need to be created.

false

The ARN of the KMS key to use to encrypt the logs in the CloudWatch Log Group used for storing container logs streamed with FluentBit. Set to null to disable encryption.

null

Name of the CloudWatch Log Group fluent-bit should use to stream logs to. When null (default), uses the eks_cluster_name as the Log Group name.

null

number of days to retain log events. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653, and 0. Select 0 to never expire.

0

ARN of the lambda function to trigger when events arrive at the fluent bit log group.

null

Filter pattern for the CloudWatch subscription. Only used if fluent_bit_log_group_subscription_arn is set.

""

Prefix string to use for the CloudWatch Log Stream that gets created for each pod. When null (default), the prefix is set to 'fluentbit'.

null
fluent_bit_pod_node_affinitylist(object(…))optional

Configure affinity rules for the fluent-bit Pods to control which nodes to schedule on. Each item in the list should be a map with the keys key, values, and operator, corresponding to the 3 properties of matchExpressions. Note that all expressions must be satisfied to schedule on the node.

list(object({
    key      = string
    values   = list(string)
    operator = string
  }))
[]
fluent_bit_pod_tolerationslist(map(…))optional

Configure tolerations rules to allow the fluent-bit Pods to schedule on nodes that have been tainted. Each item in the list specifies a toleration rule.

list(map(any))
[]
fluent_bit_versionstringoptional

Which version of aws-for-fluent-bit to install. When null, uses the default version set in the chart. Only applies to non-fargate workers.

null
kubernetes_priority_classesmap(object(…))optional

A map of PriorityClass configurations, with the key as the PriorityClass name. https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass

map(object({
    description    = string
    global_default = bool
    value          = number
  }))
{}

Policy for how DNS records are sychronized between sources and providers (options: sync, upsert-only).

"sync"

When true, the ALB ingress controller pods will be scheduled on Fargate.

false

When true, the cluster autoscaler pods will be scheduled on Fargate. It is recommended to run the cluster autoscaler on Fargate to avoid the autoscaler scaling down a node where it is running (and thus shutting itself down during a scale down event). However, since Fargate is only supported on a handful of regions, we don't default to true here.

false

When true, the external-dns pods will be scheduled on Fargate.

false
service_dns_mappingsmap(object(…))optional

Configure Kubernetes Services to lookup external DNS records. This can be useful to bind friendly internal service names to domains (e.g. the RDS database endpoint).

map(object({
    # DNS record to route requests to the Kubernetes Service to.
    target_dns = string

    # Port to route requests
    target_port = number

    # Namespace to create the underlying Kubernetes Service in.
    namespace = string
  }))
{}

If this variable is set to true, then use an exec-based plugin to authenticate and fetch tokens for EKS. This is useful because EKS clusters use short-lived authentication tokens that can expire in the middle of an 'apply' or 'destroy', and since the native Kubernetes provider in Terraform doesn't have a way to fetch up-to-date tokens, we recommend using an exec-based provider as a workaround. Use the use_kubergrunt_to_fetch_token input variable to control whether kubergrunt or aws is used to fetch tokens.

true

EKS clusters use short-lived authentication tokens that can expire in the middle of an 'apply' or 'destroy'. To avoid this issue, we use an exec-based plugin to fetch an up-to-date token. If this variable is set to true, we'll use kubergrunt to fetch the token (in which case, kubergrunt must be installed and on PATH); if this variable is set to false, we'll use the aws CLI to fetch the token (in which case, aws must be installed and on PATH). Note this functionality is only enabled if use_exec_plugin_for_auth is set to true.

true

When true, all IAM policies will be managed as dedicated policies rather than inline policies attached to the IAM roles. Dedicated managed policies are friendlier to automated policy checkers, which may scan a single resource for findings. As such, it is important to avoid inline policies when targeting compliance with various security standards.

true