add JOB_KUBE_LABELS

[igsaf2]

What

Resolves: airbytehq/airbyte#7790

How

Add JOB_KUBE_LABELS env variable in a similar to JOB_KUBE_ANNOTATIONS way.

The differences are:

• update ENV_VARS_TO_TRANSFER to pass the labels variable to an orchestrator pod
• change KubeProcessFactory.getLabels to accept labels from JOB_KUBE_LABELS (with the lowerst priority if the keys are

There are some limitations in the way how config works with Micronaut as of now. It requires CHECK, DISCOVER, SPEC prefixed variables to run specific jobs and do not fallback to default. This problem exists for other JOB_KUBE_* variables and it seems like some refactoring is ongoing, so I left it out.

I also left out JOB_KUBE_LABELS for normalization jobs. I did it because other JOB_KUBE_* variables behave in the same way and I have no easy way to test this feature.

Recommended reading order

Configs

1. airbyte-config/config-models/src/main/java/io/airbyte/config/Configs.java
2. airbyte-config/config-models/src/main/java/io/airbyte/config/EnvConfigs.java
3. airbyte-commons-worker/src/main/java/io/airbyte/workers/WorkerConfigs.java
4. airbyte-workers/src/main/resources/application.yml
5. airbyte-commons-worker/src/main/java/io/airbyte/workers/config/KubeResourceConfig.java
6. airbyte-commons-worker/src/main/java/io/airbyte/workers/config/WorkerConfigsProvider.java
7. airbyte-commons-temporal/src/main/java/io/airbyte/commons/temporal/sync/OrchestratorConstants.java

Tests

1. airbyte-workers/src/test/java/io/airbyte/workers/config/WorkerConfigProviderMicronautTest.java
2. airbyte-workers/src/test/resources/application-config-test.yaml

Process Creation

1. airbyte-commons-worker/src/main/java/io/airbyte/workers/process/KubeProcessFactory.java
2. airbyte-commons-worker/src/main/java/io/airbyte/workers/sync/LauncherWorker.java

Helm

1. charts/airbyte-worker/values.yaml
2. charts/airbyte-worker/templates/deployment.yaml
3. charts/airbyte/values.yaml.test
4. charts/airbyte/values.yaml
5. charts/airbyte/templates/env-configmap.yaml

Can this PR be safely reverted / rolled back?

If unsure, leave it blank.

• YES 💚
• NO ❌

[igsaf2]

Tested it on minikube with and without orchestration enabled (with E2E Testing connectors). Also checked that there is no crash if this variable not set.

[jackakeller]

Looks good to me! Now testing internally.

[jackakeller]

/create-oss-pr

[jackakeller]

Merged in d52234f after rebasin, fixing merge conflicts, and testing internally.

Thanks for your contribution, @igsaf2!

[chendatadog]

@igsaf2 I'm trying to follow up on this PR as I've upgraded my Airbyte to v0.50.17 which includes this PR.
However, I couldn't make JOB_KUBE_LABELS work as intended in this PR. I set up a team label for my kubenetes pods. I can see JOB_KUBE_LABELS: team=XXX by checking kubectl describe deployment airbyte-worker .
When I check airbyte-read and airbyte-write pods created by worker, I don't find the label under KUBERNETES LABELS even it works for JOB_KUBE_ANNOTATIONS already.
Do you know if I pass in this incorrectly or I missed anything? Thanks

[igsaf2]

@chendatadog
My PR was outdated and the Airbyte team made rebasing. There could be some issues. Or you have misconfigured something. The first step to debug is to see if there is the JOB_KUBE_LABELS env variable inside your airbyte-worker pod. But I think the best is to start a new issue.

Personaly, I switched to my version of Airbyte platform to roll out changes I need faster and I am completely behind the current codebase. So I cannot tell anything useful here. My code already differs a lot and I cannot debug current Airbyte versions without a proper dive-in.

[chendatadog]

@igsaf2 I confirmed that JOB_KUBE_LABELS env variable is set correctly. I can see the team label under "KUBERNETES LABELS" on airbyte-worker pod. However I can't find this label on airbyte-read or airbyte-write pod.
@jackakeller How did you do the testing? Did I miss anything here?
Thanks 

[ilyasemenov84]

I can confirm the same issue on my setup:
JOB_KUBE_LABELS exists in the config and is populated as an env, as well. But transient sync job pods don't pick it up.

[sookeke]

Same here the JOB_KUBE_LABELS does not work!

[kevin-astrafy]

same issue on the latest (0.57.3 today) version of airbyte.