Problem

In MOSIPv1.1.4 we introduced kafka for processing packets but we see that the kafka pods are gets into crashloop backoff state and keeps on crashing. Even after restarts the pods do not come up.

Root cause

Kalfka takes time to synchronize when it starts so, it gets into crashloopb backoff state as it did not start on time. Hence, we need to increase the initialDelaySeconds for liveness and readinesss for Kafka services.

Solution

In order to resolve this issue you need to perform the below changes in your enviornment.

  1. Add readiness and liveness probe for all your kafka services

    livenessProbe:
  enabled: true
  initialDelaySeconds: 120
  timeoutSeconds: 5

readinessProbe:
  enabled: true
  initialDelaySeconds: 120
  failureThreshold: 6
  timeoutSeconds: 5

  2. To make sure kafka restarts properly,

    1. Remove the kafka helm

    2. Remove the pv (persistent volume) of kafka

    3. Remove the pvc (persistent volume claim) of kafka

    4. Remove the persisted kafka folders

    5. Run the kafka playbook

Highlight important information in a panel like this one. To edit this panel's color or style, select one of the options in the menu below.

Related articles

The content by label feature displays related articles automatically, based on labels you choose. To edit options for this feature, select the placeholder below and tap the pencil icon.

Related issues