Kubernetes Health Check - How-To and Best Practices

Kubernetes health checks are essential to ensure optimal Kubernetes application availability and performance. Health check probes help monitor the health of individual containers and services for detecting issues proactively and avoiding service disruptions or cascading failures.

Kubernetes offers three kinds of health checks, where each probe serves a distinct purpose and can be configured according to the application’s use case. The three types of Kubernetes health checks are:

• Liveness probes to detect a non-responsive application
• Readiness probes to ensure that a service is ready to receive traffic
• Startup probes for identifying and delaying application startup until it’s prepared to handle requests

However, more than implementing health probes alone is needed to determine if your application is healthy. It’s essential to use appropriate protocols and configurations to ensure reliable health checks.

For instance, all of these variables have a significant impact on the effectiveness and efficiency of Kubernetes health checks:

• Probe type protocols
• Timeout settings
• Connection reuse
• Custom scripts for command probes
• HTTP response code selection

In this article, we explore how you can use different protocols to configure the available Kubernetes probe types. We also look at the best practices for minimizing resource consumption and improving the efficiency of Kubernetes health checks.

Summary of Kubernetes health check protocols

Kubernetes supports various protocols for health checks, each with its own benefits and limitations. Selecting the appropriate protocol based on your use case can help ensure the reliability and availability of your applications.

The following table outlines these different protocols and when to use them:

Protocol	Benefits	Limitations	Use case
HTTP	- High-level Kubernetes architecture diagram	– Relies on an application HTTP endpoint	When the application exposes an HTTP endpoint
	- Various components of the Kubernetes planes	– Response may not reflect the actual application state
HTTPS	– Same as HTTP, but uses SSL/TLS encryption to secure the connection between the probe and the server	– Requires a valid TLS certificate	When the application exposes an HTTPS endpoint
	– Includes benefits of the HTTP protocol	– May introduce minor performance overhead due to additional TLS handshake and encryption processes
TCP	– Quickly check if the application is running and listening on a specific TCP port	– Limited to checking if the port is accepting connections	When the application listens on a TCP port that does not support HTTP(S)
	– Can be used to check non-HTTP services such as databases or message queues	– Does not verify data transmission or response
		– Cannot verify the application state beyond the ability to accept TCP connections
Command probes	– Offers flexible health check through the use of any arbitrary command	– Requires more complex setup than other protocols	When the application does not expose an endpoint or needs an extensive probe that cannot be checked via HTTP(S)/TCP
	– Can check application-specific metrics or dependencies	– Requires continuous testing and updating of the command for every application change
gRPC	– Leverages a high-performance remote procedure call (RPC) framework for inter-service communication	– Requires complex setup and configuration	When the application uses gRPC for inter-service communication
	– Supports streaming, bi-directional communication, and authentication	– Incompatible with non-gRPC applications

Using HTTP(S) protocol for Kubernetes health check

HTTP(S) probes can be defined in a Kubernetes deployment to perform readiness and liveness checks on a running application. In the following steps, we define an HTTP/HTTPS probe and perform health checks using curl.

Step 1 - Define an HTTPS probe for the deployment

Create a Kubernetes deployment with an NGINX container running an HTTP/HTTPS server and define an HTTP/HTTPS probe. The YAML configuration of the deployment would look similar to:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: darwin-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: darwin-app
  template:
    metadata:
      labels:
        app: darwin-app
    spec:
      containers:
      - name: darwin-container
        image: nginx:latest
        ports:
        - containerPort: 443
        readinessProbe:
          httpGet:
            path: /darwin-probes
            port: 443
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          httpGet:
            path: /darwin-probes
            port: 443
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 5

Comprehensive Kubernetes cost monitoring & optimization

> Install in 5 mins or less. Get started

In the example above, we use HTTPS protocol for readiness and liveness probes, and the endpoint path for the probe is /darwin-probes. We also set:

• A 5-second delay before the probes start
• A 10-second interval between successive probes
• A 5-second timeout

Note: The container port is set to HTTPS port 443, and is added to the scheme: HTTPS field of both the readiness and liveness probes to specify the use of HTTPS. If you use the HTTP protocol, both the containerPort and port should be set to 80, and the scheme: HTTPS setting should be removed. Removing scheme: HTTPS will make Kubernetes assume HTTP as the default scheme. You can alternately also explicitly set the scheme to be scheme: HTTP.

Key fields used to configure the probes include:

• readinessProbe and livenessProbe: Specify that both the readiness and liveness probes are being defined for the container
• httpGet: Specifies the probe type and whether it is an HTTP/HTTPS GET request
• path: Specifies the endpoint for the probe. In this case, the endpoint is /darwin-probes. It is important to note that the path you specify in the probe configuration matches the path the application serves.
• port: Specifies the port for the probe. In this case, the port is 443.
• scheme: The protocol for the probe. In our case, it is set to HTTPS.
• initialDelaySeconds: The initial delay after the container starts before the first probe is performed
• periodSeconds: The interval between probes
• timeoutSeconds: Time after which the probe is considered failed if no response is received

Step 2 - Use curl to trigger probes

Run the following command on a terminal to check the deployment’s health using curl. The command can be executed on any machine with network access to the Kubernetes cluster where darwin-app is running. :

curl -i https://darwin-app:443/darwin-probes

Note: You can also use the above command with an IP address in case you do not have a resolvable hostname (such as darwin-app). To know the IP and the node port, use the kubectl get services command.

K8s clusters handling 10B daily API calls use Kubecost

Learn More

Step 3 - Analyze application health with response codes

Understanding response codes and response body content helps determine the results of a probe.

Understanding response codes

• For HTTP/HTTPS probes, the default expected success response code is 200, which indicates that the server is available and healthy.
• The failure response codes for HTTP/HTTPS probes in Kubernetes are generally any response code other than 2xx (200-299). You can also customize the failureThreshold field of the probe configuration to contain response codes that can be considered failures. For example, if the failureThreshold is 3 and the probe receives three consecutive non-2xx response codes, the container will be marked as failed.

Understanding response body content

• For response body content, the default value can be an empty string implying Kubernetes is not expecting a specific response body content from the server.
• The expected response body content can either be a string or a regular expression that matches the expected output of the application.
• If the expected response body content is unspecified, the probe is considered successful if the expected response code (e.g., 200 OK) is received.
• If you define the expected response body content, it’s important to ensure that it helps uniquely identify the application’s health and avoid false positives.

In our case, as the expected response code and response body content are not explicitly defined in the probe configuration, the readiness probe will return default response codes and body content, as shown below

HTTP/1.1 200 OK
Server: nginx/1.21.3
Date: Tue, 14 Mar 2021 10:45:30 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive

If either of the probes fails, expect a response similar to:

HTTP/1.1 503 Service Unavailable
Server: nginx/1.21.3
Date: Tue, 14 Mar 2021 11:26:06 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive

Using TCP protocol for Kubernetes health check

Like using HTTP(S) to configure health probes, you can check the health of your application’s TCP ports. TCP probes are particularly helpful for workloads that do not expose a web endpoint or HTTP/HTTPS service but rely on a TCP connection.

Step 1 - Configure TCP probe of a pod

To configure TCP probes in Kubernetes, you can add a tcpSocket field to specify the TCP port the probe will check. The following pod specification configures a TCP readiness probe on port 8080:

apiVersion: v1
kind: Pod
metadata:
  name: darwin-app-pod
spec:
  containers:
  - name: darwin-container
    image: nginx:latest
    ports:
    - containerPort: 8080
    readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20

In this case, we have defined a pod named darwin-app-pod with a single container named darwin-container.

For readinessProbe, we have specified a tcpSocket probe that checks whether the container can accept connections on port 8080. The probe will start after 5 seconds and be executed every 10 seconds. If the probe fails, the pod will be marked as not ready.

For livenessProbe, we have similarly specified a tcpSocket probe that checks whether the container can accept connections on port 8080. In this case, we have set a longer initial delay of 15 seconds and a longer period of 20 seconds. This implies that if the container fails to respond to the probe for 15 seconds, it will be restarted.

Step 2 - Using telnet to check TCP ports

Kubernetes checks if a pod is ready by opening a TCP connection to the specified port. If this connection succeeds, Kubernetes considers the pod ready; otherwise, it will repeatedly attempt to establish another TCP connection to that same port to determine if it’s alive or not. If neither attempt succeeds, Kubernetes assumes the pod is dead and must be restarted.

To verify if the TCP probe is functioning, you can test it using telnet or nc. For instance, to confirm readiness on port 8080 using this configuration, run the following command:

telnet <darwin-app-pod-IP> 8080

If the connection is successful, a message like the following indicates that the connection was established.

Trying <darwin-app-pod-IP>...
Connected to <darwin-app-pod-IP>.
Escape character is '^]'.

Learn how to manage K8s costs via the Kubecost APIs

WATCH 30 MIN YOUTUBE VIDEO

Using Command probes for Kubernetes health check

Command probes check the health of a container by executing commands within it. They’re beneficial when an application doesn’t have an endpoint, or its endpoint cannot be reliably used for health checks.

The first step to setting up a command probe is specifying the command or creating an external script that should run inside your container and return an exit code (usually zero) indicating when everything appears healthy.

Consider a scenario where an administrator wants to confirm if a certain file or directory exists in a container. The following configuration uses a command probe to check if the log file exists in the desired directory:

apiVersion: v1
kind: Pod
metadata:
  name: darwin-app
spec:
  containers:
  - name: darwin-container
    image: nginx:latest
    command:
    - /bin/sh
    - -c
    - 'if [ -f /var/log/darwin-app.log ]; then exit 0; else exit 1; fi'
    volumeMounts:
    - name: log-volume
      mountPath: /var/log/darwin-app.log
  volumes:
  - name: log-volume
    emptyDir: {}
  livenessProbe:
    exec:
      command:
      - /bin/sh
      - -c
      - 'if [ -f /var/log/darwin-app.log ]; then exit 0; else exit 1; fi'
    initialDelaySeconds: 5
    periodSeconds: 10

In this example, the command probe checks for the existence of the file /var/log/darwin-app.log. If the file exists, the command exits with a status code of 0, indicating that the container is healthy.

Besides using the default probes, there may be specific conditions or configurations unique to each application that require custom scripts to be executed as part of health checks. These scripts allow admins to perform checks beyond simple HTTP or TCP probes, enabling more granular health monitoring. Custom scripts can be added to the container image or mounted as a volume to the container at runtime. The script is then executed as part of the liveness or readiness probe.

Using gRPC protocol for Kubernetes health check

With a high-performance Remote Procedure Call (RPC) framework designed for inter-service communications, gRPC can be used as a flexible protocol for probing containerized workloads already used for inter-service communication.

gRPC probes can be seen as an extension of command probes since they also execute custom commands. However, unlike command probes executing arbitrary code, gRPC probes use the gRPC protocol specifically for communicating with applications and checking their health.

gRPC probes are faster and more efficient than command probes since they use a binary protocol and are optimized for performance. They also provide more detailed health check information and can be used to verify the status of individual services within a microservices architecture.

Besides their operational differences, gRPC probes require considerably complex configurations to set up service definitions and establish a connection to the server. In contrast, command probes only require a single command or script to be executed.

Kubernetes health probe best practices and considerations

When setting up health probes, there are several best practices and considerations to ensure Kubernetes deployments are stable and reliable.

Choose the right protocol

Before choosing a protocol for Kubernetes health probes, it is important to consider the specific requirements of your application and platform.

• Because different protocols have different strengths and weaknesses, it requires a diligent analysis before choosing the appropriate one for your cluster’s health check. For instance, TCP probes are simple and quick but may lack comprehensive information, while HTTP probes are more flexible and informative but can be slower.
• Consider the type of application before choosing the health check protocols. For applications that use a web service, HTTP probes may be more appropriate, while a network service may be better suited to TCP probes.
• Consider security requirements before choosing a probe’s protocol. While TCP probes may not require authentication or encryption, HTTP probes may require SSL/TLS encryption.

Use appropriate timeouts

As different protocols consume different amounts of resources, consider the impact on performance and resource usage when choosing a protocol. Using appropriate timeouts is vital to ensure that probes do not consume too many resources or cause unnecessary delays.

• Set a timeout that is shorter than the interval between probes
• Select an interval that provides enough time for each probe to finish successfully but doesn’t cause unnecessary delay for subsequent ones
• Consider how setting timeouts could impact system performance
• Adjust the timeout value as necessary to match the ongoing monitoring and processes of the deployment

Enable connection reuse

Kubernetes can enable connection reuse so that existing connections can be reused instead of creating new ones for each health check, saving time and resources needed to perform them. Some common use cases of connection reuse by Kubernetes involve the connection pooling of pre-established DB connections that can be reused by multiple pods or using HTTP keep-alive to indicate that a server is ready to reuse the same TCP connection for all subsequent requests. Doing so improves performance and reliability throughout the system.

• Use a connection pool to enable connection reuse
• Monitor the connection pool and adjust the configuration as necessary to ensure optimal performance
• Set the maximum number of idle connections to an appropriate value based on expected traffic
• Consider potential impacts on overall system performance when enabling connection reuse

Implement custom scripts for command probes

Using custom scripts for command probes can be useful when standard probes are insufficient. Custom scripts can also perform more complex checks, return detailed custom output, and are considerably more interoperable with non-standard systems.

• Ensure customized scripts return appropriate exit codes to indicate the appropriate status of a health check
• Consider using environment variables or command-line arguments as customizable parameters for the custom script
• Document custom scripts and store them in a version-controlled repository for easy upkeep and sharing

Leverage HTTP/2 for reliability

Kubernetes health probes that leverage HTTP/2 can offer several advantages, such as multiplexing, server push, flow control, and header compression. With HTTP/2’s support for multiplexing and server push, you can also reduce the number of connections required for health checks while improving system efficiency.

• Ensure the HTTP/2 server is configured to handle health check requests with high reliability, such as through load balancing or failover mechanisms
• Utilize HTTP/2 over TLS for added security and dependability when performing health checks
• Consider using a health check tool that supports HTTP/2, such as curl or kubectl, to perform health checks and identify issues

Define appropriate HTTP response codes

Kubernetes health checks rely on HTTP response codes returned by the target container or service to determine whether the container or service is operating optimally. Using appropriate HTTP response codes in health checks helps Kubernetes to accurately monitor the health of the target container or service and take corrective actions such as restarting or rescheduling the container.

• Instead of using generic or ambiguous codes, use appropriate HTTP/2 response codes, such as 200 (OK) or 503 (Service Unavailable) in the health check response to determine the exact status of a target container or service
• Verify that the HTTP response codes generated during health checks are recognized by monitoring and alerting systems of the Kubernetes cluster
• Consider combining HTTP response codes with other metrics, such as response time or error rate, to get a more comprehensive view of the health of your target container or service

Limit resource consumption

Health checks often trigger resource-intensive operations such as making network requests or running custom scripts, potentially impacting the performance of the target container and affecting other services running on the same node. A diligent usage of probes helps Kubernetes minimize the impact of health checks on system resources and ensure that the target container or service can perform its intended function without interference from health checks.

• Utilize liveness probes to detect and recover from container crashes, while reducing the frequency of readiness probes
• Instead of sending GET requests for readiness probes, make HTTP HEAD requests for reduced network traffic and resource usage
• Utilize custom scripts sparingly and, when possible, opt for simpler methods like HTTP probes or TCP socket probes

Managing Kubernetes clusters can be complex, and configuring health checks is just one aspect of ensuring optimal performance and resource utilization. With Kubecost’s comprehensive platform for assessing infrastructure reliability, you can optimize your Kubernetes cluster’s performance, monitor resource utilization, and minimize Kubernetes costs much more efficiently.

In addition to tracking resource utilization and cost, Kubecost allows you to set up cluster health alerts based on specific threshold criteria such as low cluster memory, low cluster CPU, and out-of-memory pods. By utilizing Kubecost alongside carefully configured probes, you can ensure that your cluster is performing optimally while also minimizing unnecessary costs.

Comprehensive Kubernetes cost monitoring & optimization

> Install in 5 mins or less. Get started

Conclusion

As you gain more insight into Kubernetes, exploring different probe types and configurations can significantly enhance your monitoring strategy and enable optimal performance. While Kubernetes offers a robust set of tools and APIs to manage and automate health checks, using additional tools like Kubecost can help optimize your cluster’s performance and resource utilization.