Kubernetes Components - Control Plane

Diagram

K8 Control Plane Architecture

About

  • The control plane node provides a running environment for the control plane agents responsible for managing the state of a Kubernetes cluster, and it is the brain behind all operations inside the cluster.

  • The control plane components are agents with very distinct roles in the cluster's management. In order to communicate with the Kubernetes cluster, users send requests to the control plane via a Command Line Interface (CLI) tool, a Web User-Interface (Web UI) Dashboard, or an Application Programming Interface (API).

  • It is important to keep the control plane running at all costs. Losing the control plane may introduce downtime, causing service disruption to clients, with possible loss of business.

    • To ensure the control plane's fault tolerance, control plane node replicas can be added to the cluster, configured in High-Availability (HA) mode.

    • While only one of the control plane nodes is dedicated to actively managing the cluster, the control plane components stay in sync across the control plane node replicas.

    • This type of configuration adds resiliency to the cluster's control plane, should the active control plane node fail.

  • To persist the Kubernetes cluster's state, all cluster configuration data is saved to a distributed key-value store which only holds cluster state related data, no client workload generated data.

    • The key-value store may be configured on the control plane node (stacked topology), or on its dedicated host (external topology) to help reduce the chances of data store loss by decoupling it from the other control plane agents.

    • In the stacked key-value store topology, HA control plane node replicas ensure the key-value store's resiliency as well.

    • However, that is not the case with external key-value store topology, where the dedicated key-value store hosts have to be separately replicated for HA, a configuration that introduces the need for additional hardware, hence additional operational costs.

  • A control plane node runs the following essential control plane components and agents:

    • API Server

    • Scheduler

    • Controller Managers

    • Key-Value Data Store

  • In addition, the control plane node runs:

    • Container Runtime

    • Node Agent (kubelet)

    • Proxy (kube-proxy)

    • Optional add-ons for observability, such as dashboards cluster-level monitoring and logging.

  • Lets dig deep into each of the control plane componenets.

API server

  • The API Server intercepts RESTful calls from users, administrators, developers, operators and external agents, then validates and processes them.

  • During processing the API Server reads the Kubernetes cluster's current state from the key-value store, and after a call's execution, the resulting state of the Kubernetes cluster is saved in the key-value store for persistence.

  • The API Server is the only control plane component to talk to the key-value store, both to read from and to save Kubernetes cluster state information - acting as a middle interface for any other control plane agent inquiring about the cluster's state.

  • The API Server is highly configurable and customizable. It can scale horizontally, but it also supports the addition of custom secondary API Servers, a configuration that transforms the primary API Server into a proxy to all secondary, custom API Servers, routing all incoming RESTful calls to them based on custom defined rules.

Scheduler

  • The role of the kube-scheduler is to assign new workload objects, such as pods encapsulating containers, to nodes - typically worker nodes.

  • During the scheduling process, decisions are made based on current Kubernetes cluster state and new workload object's requirements. The scheduler obtains from the key-value store, via the API Server, resource usage data for each worker node in the cluster.

  • The scheduler also receives from the API Server the new workload object's requirements which are part of its configuration data.

  • The scheduler also takes into account Quality of Service (QoS) requirements, data locality, affinity, anti-affinity, taints, toleration, cluster topology, etc.

  • Once all the cluster data is available, the scheduling algorithm filters the nodes with predicates to isolate the possible node candidates which then are scored with priorities in order to select the one node that satisfies all the requirements for hosting the new workload.

  • The outcome of the decision process is communicated back to the API Server, which then delegates the workload deployment with other control plane agents.

  • The scheduler is highly configurable and customizable through scheduling policies, plugins, and profiles.

  • Additional custom schedulers are also supported, then the object's configuration data should include the name of the custom scheduler expected to make the scheduling decision for that particular object; if no such data is included, the default scheduler is selected instead.

  • A scheduler is extremely important and complex in a multi-node Kubernetes cluster, in single-node cluster the scheduler's job is quite simple.

Controller Managers

  • The controller managers are components of the control plane node running controllers or operator processes to regulate the state of the Kubernetes cluster.

  • Controllers are watch-loop processes continuously running and comparing the cluster's desired state (provided by objects' configuration data) with its current state (obtained from the key-value store via the API Server).

  • In case of a mismatch, corrective action is taken in the cluster until its current state matches the desired state.

Kube Controller Managers

  • Runs controllers or operators responsible to act when nodes become unavailable, to ensure container pod counts are as expected, to create endpoints, service accounts, and API access tokens.

  • There are many different types of controllers. Some examples of them are:

    • Node controller: Responsible for noticing and responding when nodes go down.

    • Job controller: Watches for Job objects that represent one-off tasks, then creates Pods to run those tasks to completion. EndpointSlice controller: Populates EndpointSlice objects (to provide a link between Services and Pods).

    • ServiceAccount controller: Create default ServiceAccounts for new namespaces.

  • The above is not an exhaustive list.

Cloud Controller Managers

  • Runs controllers or operators responsible to interact with the underlying infrastructure of a cloud provider when nodes become unavailable, to manage storage volumes when provided by a cloud service, and to manage load balancing and routing.

  • The following controllers can have cloud provider dependencies:

    • Node controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding.

    • Route controller: For setting up routes in the underlying cloud infrastructure.

    • Service controller: For creating, updating and deleting cloud provider load balancers.

etcd

  • etcd is an open source project under the Cloud Native Computing Foundation (CNCF).

  • etcd is a strongly consistent, distributed key-value data store used to persist a Kubernetes cluster's state.

  • New data is written to the data store only by appending to it, data is never replaced in the data store. Obsolete data is compacted (or shredded) periodically to minimize the size of the data store.

  • Out of all the control plane components, only the API Server is able to communicate with the etcd data store.

  • etcd's CLI management tool - etcdctl, provides snapshot save and restore capabilities which come in handy especially for a single etcd instance Kubernetes cluster - common in Development and learning environments.

  • However, in Stage and Production environments, it is extremely important to replicate the data stores in HA mode, for cluster configuration data resiliency.

  • Some Kubernetes cluster bootstrapping tools, such as kubeadm, by default, provision stacked etcd control plane nodes, where the data store runs alongside and shares resources with the other control plane components on the same control plane node.

  • For data store isolation from the control plane components, the bootstrapping process can be configured for an external etcd topology, where the data store is provisioned on a dedicated separate host, thus reducing the chances of an etcd failure.

  • Both stacked and external etcd topologies support HA configurations.

  • etcd is based on the Raft Consensus Algorithm which allows a collection of machines to work as a coherent group that can survive the failures of some of its members.

    • At any given time, one of the nodes in the group will be the leader, and the rest of them will be the followers.

    • etcd gracefully handles leader elections and can tolerate node failure, including leader node failures. Any node can be treated as a leader.

    • Keep in mind however, that the leader/followers hierarchy is distinct from the primary/secondary hierarchy, meaning that neither node is favored for the leader role, and neither node outranks other nodes.

    • A leader will remain active until it fails, at which point in time a new leader is elected by the group of healthy followers.

  • In Kubernetes, besides storing the cluster state, etcd is also used to store configuration details such as subnets, ConfigMaps, Secrets, etc.

References

PreviousArchitectureNextKubernetes Components - Worker Node

Last updated