Stuck on selfHealing loop with Presync Jobs that are deleted after completion

[Pauatro]

Hello!

I use a Presync Job to run the migrations of the db before the frontend and backend are updated. The Job is deleted after being finished, which is what I want. ArgoCD detects the missing job and triggers a sync every 3 minutes, so the job runs again. When selfHealing is removed, this loop stops happening and the cluster stays Out of Sync.

I've tried using selective diffing but I can't make it ignore the whole missing Job manifest, and I'm not sure if that's even possible since the documentation only describes ignoring specific fields in manifests, not the entire file. The ignoreDifferences conditions are added properly to the App manifest though.

Is there a common way of doing this? Just knowing whether what I'm trying to do is possible would already go a long way. AFAIK running migrations on a Presync Job is a common practice, so I would expect ArgoCD to include a way to manage it and I'm probably doing something wrong there.

We also use Autopilot to set up ArgoCD.

Some of the important files included here:

• Migration Job

apiVersion: batch/v1
kind: Job
metadata:
  name: migration-job
  labels:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  ttlSecondsAfterFinished: 120
  backoffLimit: 4
  template:
    spec:
      containers:
        - name: migration-job
          image: XXXX
          env:
            - name: XXX
              value: XXX
          volumeMounts:
            - mountPath: XX
              name: XX
      volumes:
      - name: XX
      restartPolicy: Never

• Production AppProject and ApplicationSet

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  annotations:
    argocd-autopilot.argoproj-labs.io/default-dest-server: https://kubernetes.default.svc
    argocd.argoproj.io/sync-options: PruneLast=true
    argocd.argoproj.io/sync-wave: "-2"
  creationTimestamp: null
  name: production
  namespace: argocd
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  description: production project
  destinations:
  - namespace: '*'
    server: '*'
  namespaceResourceWhitelist:
  - group: '*'
    kind: '*'
  sourceRepos:
  - '*'
status: {}

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  creationTimestamp: null
  name: production
  namespace: argocd
spec:
  generators:
  - git:
      files:
      - path: xxx
      repoURL: xxx
      requeueAfterSeconds: 20
      revision: production-cr
      template:
        metadata: {}
        spec:
          destination: {}
          project: ""
  - git:
      files:
      - path: xxx
      repoURL: xxx
      requeueAfterSeconds: 20
      revision: production-cr
      template:
        metadata: {}
        spec:
          destination: {}
          project: ""
          source:
            directory:
              exclude: '{{ exclude }}'
              include: '{{ include }}'
              jsonnet: {}
              recurse: true
            repoURL: ""
  syncPolicy: {}
  template:
    metadata:
      labels:
        app.kubernetes.io/managed-by: argocd-autopilot
        app.kubernetes.io/name: '{{ appName }}'
      name: production-{{ userGivenName }}
      namespace: argocd
    spec:
      destination:
        namespace: '{{ destNamespace }}'
        server: '{{ destServer }}'
      ignoreDifferences:
      - group: argoproj.io
        jsonPointers:
        - /status
        kind: Application
      - group: batch
        kind: Job
        name: migration-job
      project: production
      source:
        path: '{{ srcPath }}'
        repoURL: '{{ srcRepoURL }}'
        targetRevision: '{{ srcTargetRevision }}'
      syncPolicy:
        automated:
          allowEmpty: true
          prune: true
          selfHeal: true
status: {}

[imuneeeb]

@Pauatro
Try using argocd.argoproj.io/hook-delete-policy: BeforeHookCreation in your job. It will delete the job when new hook is created.