High CPU Utilization on Windows Core Instance with Kubernetes Pods Every Six Hours

Diagnosing high CPU utilization on a Windows Core instance running Kubernetes pods requires a systematic approach to identify the root cause. Here's a step-by-step guide using command-line tools:

Step 1: Monitor CPU Usage

Using typeperf

typeperf is a command-line utility that reports performance data.

1. Open a command prompt on your Windows Core instance.
2. Use typeperf to log CPU utilization data over time.

typeperf "\Processor(_Total)\% Processor Time" -si 1 -sc 3600 -f CSV -o cpu_usage.csv

This command logs the CPU usage every second for an hour. Adjust -sc (sample count) as needed.

Step 2: Identify High CPU Processes

Using tasklist and taskkill

1. Use tasklist to get a list of running processes and their CPU usage.

tasklist /v /fo csv > tasklist.csv

The /v parameter provides verbose information, including CPU time.

2. To check CPU usage at a specific point, you can use wmic:

wmic path Win32_PerfFormattedData_PerfProc_Process get Name,PercentProcessorTime,IDProcess /format:csv > process_cpu_usage.csv

Step 3: Analyze Kubernetes Pods

Since you're running Kubernetes, you should check which pods are consuming resources.

1. Use kubectl to get a list of pods and their resource usage:

kubectl top pods -A

This command shows the CPU and memory usage for all pods across all namespaces.

2. If you identify a pod with high CPU usage, describe the pod to get more details:

kubectl describe pod <pod-name> -n <namespace>

3. Check the logs of the problematic pod to see if there are any clues:

kubectl logs <pod-name> -n <namespace>

Step 4: Scheduled Tasks or Cron Jobs

High CPU utilization spikes every six hours might be due to scheduled tasks or cron jobs.

1. List all scheduled tasks:

schtasks /query /fo LIST /v > scheduled_tasks.txt

2. Check for any cron jobs scheduled in your Kubernetes cluster:

kubectl get cronjobs -A

Step 5: Analyze Network Activity

Sometimes, network activity can cause high CPU usage.

1. Use netstat to check for active network connections:

netstat -ano > netstat.txt

2. Cross-reference the PID in the netstat output with your tasklist output to identify which processes are making network connections.

Step 6: Use Windows Performance Recorder (WPR)

For a more detailed analysis, you can use Windows Performance Recorder (WPR) to capture performance data:

1. Start recording:

wpr -start generalprofile -filemode

2. Reproduce the issue or wait for the next CPU spike.

3. Stop recording:

wpr -stop output.etl

4. Analyze the output.etl file using Windows Performance Analyzer (WPA), which you can run on a machine with a GUI.

Hello Saini,

Here I am providing one by one steps to solve your issue.

1. Identify the culprit pod:

Run the following command to see which pods are currently using the most CPU:

kubectl top pods

This will display a table showing the resource usage of your pods, including CPU. Look for pods with high CPU utilization.

2. Monitor CPU usage over time:

To confirm the periodicity of the spikes, run this command to see the CPU utilization history of your pods:

kubectl top pods --history

This will show you how CPU usage has changed over time for your pods. Look for spikes that occur every six hours.

3. Investigate pod logs:

Once you've identified a suspicious pod, use this command to view its logs:

kubectl logs <pod_name>

The logs might reveal clues about what's causing the CPU spikes. Look for errors, warnings, or any messages indicating high resource usage.

Troubleshooting High CPU Usage on Kubernetes Pods: This guide provides a more in-depth approach to diagnosing CPU spikes in Kubernetes pods, though it may not be specific to Windows: https://github.com/kubernetes/kubernetes/issues/69083

Hello

check these steps to resolve the issue:

Gather Resource Utilization Data:

kubectl top nodes: 

Use this command to get an overview of CPU and memory usage across all nodes in your Kubernetes cluster. Look for spikes in CPU utilization that coincide with the six-hour intervals.

kubectl top pods: 

This command displays CPU and memory usage for each pod in the cluster. Filter by namespace if needed (e.g., kubectl top pods -n my-namespace). Identify pods with high CPU usage during the spike times. 2. Identify Pod Details:

kubectl describe pod <pod_name>: Replace <pod_name>

with the name of a pod showing high CPU usage. This command provides detailed information about the pod, including its container images, resource requests/limits, and events. 3. Analyze Container Processes:

Connect to Pod:

Use kubectl exec -it <pod_name>

bash to establish a bash shell session within the pod. This allows you to examine running processes. Process Monitoring: Use top or ps auxf to view a list of running processes and their CPU utilization. Identify processes consuming significant CPU resources during the spike intervals. Additional Tools: Tools like strace or perf (if available on your Windows Core instance) can provide deeper insights into system calls and function calls made by the CPU-intensive process, helping pinpoint the root cause. However, these require familiarity with system profiling techniques.