Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GoogleCloudStorageDownloadOperator raises TypeError on Composer #2

Open
ricardolsmendes opened this issue Jul 12, 2021 · 0 comments
Open
Labels
bug Something isn't working

Comments

@ricardolsmendes
Copy link
Owner

ricardolsmendes commented Jul 12, 2021

The GoogleCloudStorageDownloadOperator (used by the CompareXComMapsOperator in wordcount_test.py) works with Python 2.7 but raises a TypeError: Object of type 'bytes' is not JSON serializable when running in Python 3.6.

The error is easily reproducible in the below Cloud Composer Environment:

  1. Composer version: 1.16.8
  2. Airflow version: 1.10.15
  3. Python version: 3.6

Error details:

Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 985, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_download_operator.py", line 96, in execut
    context['ti'].xcom_push(key=self.store_to_xcom_key, value=file_bytes
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 1516, in xcom_pus
    execution_date=execution_date or self.execution_date
  File "/usr/local/lib/airflow/airflow/utils/db.py", line 74, in wrappe
    return func(*args, **kwargs
  File "/usr/local/lib/airflow/airflow/models/xcom.py", line 101, in se
    value = XCom.serialize_value(value
  File "/usr/local/lib/airflow/airflow/models/xcom.py", line 224, in serialize_valu
    return json.dumps(value).encode('UTF-8'
  File "/opt/python3.6/lib/python3.6/json/__init__.py", line 231, in dump
    return _default_encoder.encode(obj
  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 199, in encod
    chunks = self.iterencode(o, _one_shot=True
  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 257, in iterencod
    return _iterencode(o, 0
  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 180, in defaul
    o.__class__.__name__
TypeError: Object of type 'bytes' is not JSON serializabl

GoogleCloudStorageDownloadOperator is deprecated and probably needs to be replaced in order to fix the test workflow.

@ricardolsmendes ricardolsmendes added the bug Something isn't working label Jul 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant