Kafka pods gets into crashloop backoff

Problem

In MOSIPv1.1.4 we introduced kafka for processing packets but we see that the kafka pods are gets into crashloop backoff state and keeps on crashing. Even after restarts the pods do not come up.

Root cause

Kalfka takes time to synchronize when it starts hence we need to increase the initialDelaySeconds for liveness and readinesss.

Solution

In order to resolve this issue you need to perform the below changes in your enviornment.

Add readiness and liveness probe for all your kafka services

livenessProbe:
  enabled: true
  initialDelaySeconds: 120
  timeoutSeconds: 5

readinessProbe:
  enabled: true
  initialDelaySeconds: 120
  failureThreshold: 6
  timeoutSeconds: 5

To make sure kafka restarts properly,
1. Remove the kafka helm
2. Remove the pv (persistent volume) of kafka
3. Remove the pvc (persistent volume claim) of kafka
4. Remove the persisted kafka folders
5. Run the kafka playbook

Kafka pods gets into crashloop backoff

Problem

Root cause

Solution

Related articles