Task DSM_cts_ind.dsm-cts-ind-processing fails

[nesnoj]

...due to non-existing table demand.egon_etrago_electricity_cts.
This table seems to be created in Task electricity_demand.temporal.insert-cts-load here.

The task depends on cts_electricity_demand_annual which is CtsElectricityDemand (see pipeline.py) which includes the insert-cts-load task. This in turn depends on demandregio, so I expect the original task to fail due to a failing upstream task and thus not to be executed. But it is.

Maybe it's just a dependency problem I don't see and it never rose since the electricity demand has always been present when the task started...

Did anyone stumble across this problem? Maybe you @ClaraBuettner ?

Full log:

[2024-01-10 11:13:47,854] {standard_task_runner.py:78} INFO - Job 3: Subtask DSM_cts_ind.dsm-cts-ind-processing
[2024-01-10 11:13:47,951] {logging_mixin.py:120} INFO - Running <TaskInstance: egon-data-processing-pipeline.DSM_cts_ind.dsm-cts-ind-processing 2024-01-10T10:13:36.809164+00:00 [running]> on host ws-02lin.rl-institut.local
[2024-01-10 11:13:48,048] {logging_mixin.py:120} INFO - 
[2024-01-10 11:13:48,048] {logging_mixin.py:120} INFO - CTS per osm-area: cooling, ventilation and air conditioning
[2024-01-10 11:13:48,048] {logging_mixin.py:120} INFO - 
[2024-01-10 11:13:48,191] {taskinstance.py:1150} ERROR - (psycopg2.errors.UndefinedTable) relation "demand.egon_etrago_electricity_cts" does not exist
LINE 2:         demand.egon_etrago_electricity_cts
                ^

[SQL: SELECT bus_id, scn_name, p_set FROM
        demand.egon_etrago_electricity_cts]
(Background on this error at: http://sqlalche.me/e/13/f405)
Traceback (most recent call last):
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "demand.egon_etrago_electricity_cts" does not exist
LINE 2:         demand.egon_etrago_electricity_cts
                ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/__init__.py", line 202, in skip_task
    result = super(type(task), task).execute(*xs, **ks)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/DSM_cts_ind.py", line 1621, in dsm_cts_ind_processing
    dsm_cts_ind()
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/DSM_cts_ind.py", line 1093, in dsm_cts_ind
    dsm = cts_data_import(cts_cool_vent_ac_share)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/DSM_cts_ind.py", line 224, in cts_data_import
    ts = db.select_dataframe(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/db.py", line 188, in select_dataframe
    df = pd.read_sql(sql, engine(), index_col=index_col)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 628, in read_sql
    return pandas_sql.read_query(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 1579, in read_query
    result = self.execute(*args)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 1424, in execute
    return self.connectable.execution_options().execute(*args, **kwargs)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2235, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1003, in execute
    return self._execute_text(object_, multiparams, params)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1172, in _execute_text
    ret = self._execute_context(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "demand.egon_etrago_electricity_cts" does not exist
LINE 2:         demand.egon_etrago_electricity_cts
                ^

[SQL: SELECT bus_id, scn_name, p_set FROM
        demand.egon_etrago_electricity_cts]
(Background on this error at: http://sqlalche.me/e/13/f405)
[2024-01-10 11:13:48,193] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=egon-data-processing-pipeline, task_id=DSM_cts_ind.dsm-cts-ind-processing, execution_date=20240110T101336, start_date=20240110T101347, end_date=20240110T101348
[2024-01-10 11:13:52,682] {local_task_job.py:102} INFO - Task exited with return code 1