CloudWatch CreateLogGroup error on already existing LogGroup

[jo-gre]

Apache Airflow version

2.1.2

Operating System

Ubuntu Docker

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==2.0.0
apache-airflow-providers-celery==2.0.0
apache-airflow-providers-cncf-kubernetes==2.0.0
apache-airflow-providers-docker==2.0.0
apache-airflow-providers-elasticsearch==2.0.2
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-google==4.0.0
apache-airflow-providers-grpc==2.0.0
apache-airflow-providers-hashicorp==2.0.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-microsoft-azure==3.0.0
apache-airflow-providers-mysql==2.0.0
apache-airflow-providers-neo4j==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-redis==2.0.0
apache-airflow-providers-sendgrid==2.0.0
apache-airflow-providers-sftp==2.0.0
apache-airflow-providers-slack==4.0.0
apache-airflow-providers-sqlite==2.0.0
apache-airflow-providers-ssh==2.0.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

    config:
      core:
        executor: 'CeleryExecutor'
      logging:
        remote_logging: 'True'
        remote_base_log_folder: 'cloudwatch://arn:aws:logs:eu-central-1:<redacted>:log-group:airflow'
        remote_log_conn_id: 'cloudwatch'

The log group is already created in Terraform.

What happened

[2021-09-27 15:03:58,541: ERROR/ForkPoolWorker-15] Failed to execute task An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::<redacted>:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 117, in _execute_in_fork
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 228, in task_run
    ti.init_run_context(raw=args.raw)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2031, in init_run_context
    self._set_context(self)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 54, in _set_context
    set_context(self.log, context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 173, in set_context
    handler.set_context(value)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py", line 81, in set_context
    self.handler = watchtower.CloudWatchLogHandler(
  File "/home/airflow/.local/lib/python3.8/site-packages/watchtower/__init__.py", line 148, in __init__
    _idempotent_create(self.cwl_client, "create_log_group", logGroupName=self.log_group)
  File "/home/airflow/.local/lib/python3.8/site-packages/watchtower/__init__.py", line 15, in _idempotent_create
    method_callable(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts:<redacted>}:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:

What you expected to happen

I would expect Airflow to create a stream and write the logs, not to create a log-group inside a log-group.

How to reproduce

Run any dag with the chart config.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[anaynayak]

@potiuk I think I can try this one.

My current understanding of the code + changes required:

1. CloudwatchTaskHandler delegates to watchtower.CloudWatchLogHandler which provides the option to create a log group if it doesn't exist (true by default) https://github.com/kislyuk/watchtower/blob/develop/watchtower/__init__.py#L84
2. Do we intend to continue with the existing behavior but allow users to override based on config values defined ? e.g. conf.get('logging', 'CLOUDWATCH_CREATE_LOG_GROUP', fallback=True) similar to https://github.com/apache/airflow/blob/main/airflow/config_templates/airflow_local_settings.py#L209 ?

Let me know if thats the approach. Can do the changes accordingly.

[potiuk]

Hard to say. I am not expert on cloud-watch - but maybe others can help :)

[anaynayak]

Was curious more from a config standpoint. I believe the way to specify/override the default config is from airflow_local_settings.py. Will create a PR. Will make it easier to iterate on it.

If anyone else gets to reviewing the approach, please let me know :)

[o-nikolas]

Hey folks,

watchtower does an idempotent create for the log group as you can see here.. If the log group already exists it will catch that error and ignore it silently.

The issue you actually hit is an authentication issue, as seen in your exception message:

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts:}:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1::log-group:airflow:log-stream:

The role you're using doesn't have permission for the log group creation. It's worth double checking that role has all the required permissions for cloudwatch logging (creating groups, streams and uploading records).

[o-nikolas]

Given the discussion in 19022, I think this issue can be resolved/closed?

[jo-gre]

Hey folks,

sorry that I didn't respond for that long.

I also figured out that this looks like an authentication problem but even with all privileges given to airflow the error still occurred. I was wondering if airflow maybe tries to create the log group inside the log stream. The error arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream: looks like the correct path, but I don't see a reason why AWS would require the sub-resouce :log-stream: when calling to create a log-group.

[o-nikolas]

What I'm quite sure of is that the issue is constrained to your configuration. Otherwise we'd have this exception for all users with logging to cloudwatch.

I don't think the boto3 api would return an authentication issue if the log group name was formatted incorrectly. The airflow-service-account-role you're using likely is still missing the correct policies.
As for the strangely formatted loggroup name, check the config you're using to specify it. You can even paste that here to get a second pair of eyes on it.

[ptrhck]

I am facing the very same issue on Airflow 2.2.2

Failed to execute task An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::XXXXXXX:assumed-role/airflow-ecs-task-role-staging/airflow_logs_cloudwatch is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:XXXXXXX:assumed:log-group:airflow-v2-staging-dags:log-stream: because no identity-based policy allows the logs:CreateLogGroup action.

The Cloudwatch log group is already created using Cloudformation and the policies accordingly:

Resources:

  DagsLogs:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub airflow-v2-${Stage}-dags

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      Policies:
        - PolicyName: !Sub ${EnvironmentName}-logs-dags-${Stage}
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource:
                  - !GetAtt DagsLogs.Arn

The Airflow cofiguration variables are the following:

Environment:
  - Name: AIRFLOW_CONN_LOGS_CLOUDWATCH
    Value: !Sub 'cloudwatch://cloudwatch?aws_account_id=XXXXXXX&role_arn=arn%3Aaws%3Aiam%3A%3A919107267526%3Arole%2Fairflow-ecs-task-role-${Stage}'
  - Name: AIRFLOW__LOGGING__REMOTE_LOGGING
    Value: 'true'
  - Name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    Value: !Sub "cloudwatch://arn:aws:logs:eu-central-1:XXXXXXX:log-group:airflow-v2-${Stage}-dags"
  - Name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    Value: logs_cloudwatch

Adding logs:CreateLogGroup to the task role seems to resolve the issue. But why is that necessary?

[eladkal]

Based on discussion in #19022 i'm converting this to github discussion

[tetrabid]

Raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:iam::428122111344:user/awseruser is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:ap-south-1:428122108872:log-group:HouMuch:log-stream: because no identity-based policy allows the logs:CreateLogGroup action

got the same error for an already working application after a EC2 reboot. And it's true the role has full access to cloudwatch