Skip to content

Commit

Permalink
Merge pull request #200 from rl-institut/release/v1.0.0
Browse files Browse the repository at this point in the history
Release/v1.0.0
  • Loading branch information
nesnoj authored Nov 7, 2023
2 parents dd3ab30 + 7edf598 commit 9bb33f3
Show file tree
Hide file tree
Showing 430 changed files with 40,090 additions and 125 deletions.
21 changes: 21 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
Fixes #

## Before merging into `dev`-branch, please make sure that the following points are checked:

- [ ] All pre-commit tests passed locally with: `pre-commit run -a`
- [ ] File `CHANGELOG.md` was updated
- [ ] The docs were updated
- [ ] if `dataset.md`s were updated, the dataset docs were regenerated using
`python docs/generate_dataset_mds.py`

If packages were modified:
- [ ] File `poetry.lock` was updated with: `poetry lock`
- [ ] A new env was successfully set up

WARNING: When modifying use snakemake <=7.32.0, cf. #186

If data flow was adjusted:
- [ ] Data pipeline run finished successfully with: `snakemake -jX`
- [ ] Esys appdata was created successfully with: `snakemake -jX make_esys_appdata`

(with `X` = desired number of cores, e.g. 1)
29 changes: 29 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ instance/
# Sphinx documentation
docs/_build/

# Mkdocs documentation
site/

# PyBuilder
target/

Expand Down Expand Up @@ -132,3 +135,29 @@ dmypy.json
.DS_Store
.snakemake
.idea

# Data files
digipipe/store/raw/*/data/*
!digipipe/store/raw/*/data/.gitkeep
!digipipe/store/raw/.TEMPLATE/data/*

digipipe/store/preprocessed/*/data/*
!digipipe/store/preprocessed/*/data/.gitkeep
!digipipe/store/preprocessed/.TEMPLATE/data/*

digipipe/store/datasets/*/data/*
!digipipe/store/datasets/*/data/.gitkeep
!digipipe/store/datasets/.TEMPLATE/data/*

digipipe/store/appdata/esys/*
digipipe/store/appdata/*/data/*
!digipipe/store/appdata/*/data/.gitkeep
!digipipe/store/appdata/datapackage/data/captions/.gitkeep
!digipipe/store/appdata/datapackage/data/geodata/.gitkeep
!digipipe/store/appdata/datapackage/data/scalars/.gitkeep
!digipipe/store/appdata/datapackage/data/settings/.gitkeep
!digipipe/store/appdata/datapackage/data/sequences/.gitkeep
!digipipe/store/appdata/*/.gitkeep

digipipe/store/temp/*
!digipipe/store/temp/.gitkeep
107 changes: 107 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# TODO: Find out if something needs to be changed in here in comparison to digiplan

exclude: 'docs|node_modules|vendors|migrations|.git|.tox'
default_stages: [commit]
fail_fast: true

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
hooks:
- id: check-json
- id: end-of-file-fixer
- id: trailing-whitespace
- id: check-added-large-files

- repo: https://github.com/charliermarsh/ruff-pre-commit
# Ruff version.
rev: 'v0.0.244'
hooks:
- id: ruff
args: ["--fix"]

- repo: https://github.com/pycqa/isort
rev: 5.12.0
hooks:
- id: isort
name: isort (python)
args: ["--line-length", "80", "black"]

- repo: local
hooks:
- id: black
name: black
entry: black
language: python
types: [python]
- id: flake8
name: flake8
args:
- --max-line-length=80
- --ignore=DAR101,DAR201,F821,DAR401,W503,A001,A003,E800,E722,B001,B008,B023,C408,C417,W605,B007,S101
entry: flake8
language: python
types: [python]
- id: pylint
name: pylint
entry: env DATABASE_URL=null env PROJ_LIB=null USE_DOCKER=null pylint
language: system
types: [python]
args:
[
"-rn", # Only display messages
"-sn", # Don't display the score
"--disable=E0602", # Disable the E0602 error
"--disable=C0114", # Disable the C0114 error TODO: To be fixed
"--disable=C0116", # Disable the C0116 error TODO: To be fixed
"--disable=R1729", # Disable the R1729 error TODO: To be fixed
"--disable=C0103", # Disable the C0103 error TODO: To be fixed
"--disable=R0801", # Disable the R0801 error TODO: To be fixed
"--disable=W1514", # Disable the W1514 error TODO: To be fixed
"--disable=R1734", # Disable the R1734 error TODO: To be fixed
"--disable=R1735", # Disable the R1735 error TODO: To be fixed
"--disable=W0612", # Disable the W0612 error TODO: To be fixed
"--disable=W1401", # Disable the W1401 error TODO: To be fixed
"--disable=W0511", # Disable the W0511 error TODO: To be fixed
"--disable=R0913", # Disable the R0913 error TODO: To be fixed
"--disable=R1705", # Disable the R1705 error TODO: To be fixed
"--disable=E0401", # Disable the E0401 error TODO: To be fixed
"--disable=E0611", # Disable the E0611 error TODO: To be fixed
"--disable=W0621", # Disable the W0621 error TODO: To be fixed
"--disable=W0640", # Disable the W0640 error TODO: To be fixed
"--disable=C0115", # Disable the C0115 error TODO: To be fixed
"--disable=R1725", # Disable the R1725 error TODO: To be fixed
"--disable=R0903", # Disable the R0903 error TODO: To be fixed
"--disable=C0302", # Disable the C0302 error TODO: To be fixed
"--disable=W0707", # Disable the W0707 error TODO: To be fixed
"--disable=W0108", # Disable the W0108 error TODO: To be fixed
"--disable=W1203", # Disable the W1203 error TODO: To be fixed
"--disable=W0622", # Disable the W0622 error TODO: To be fixed
"--disable=R1720", # Disable the R1720 error TODO: To be fixed
"--disable=R0914", # Disable the R0914 error TODO: To be fixed
"--disable=R1724", # Disable the R1724 error TODO: To be fixed
"--disable=W0012", # Disable the W0012 error TODO: To be fixed
"--disable=W0611", # Disable the W0611 error TODO: To be fixed
"--disable=C0207", # Disable the C0207 error TODO: To be fixed
"--max-line-length=80", # Set the maximum line length to 80
]
- id: mypy
name: mypy
entry: mypy
language: python
types: [python]
args:
[
"--ignore-missing-imports", # Ignore missing imports
"--warn-unused-ignores", # Warn about unused ignore comments
"--disable-error-code=name-defined", # To suppress Name "snakemake"
# is not defined [name-defined]
"--disable-error-code=var-annotated",
"--disable-error-code=var-annotated",
"--disable-error-code=dict-item",
"--disable-error-code=arg-type",
"--disable-error-code=assignment",
"--disable-error-code=attr-defined",
"--disable-error-code=index",
"--disable-error-code=misc",
]
23 changes: 23 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Read the Docs configuration file for MkDocs projects
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

# Set the version of Python and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.11"
jobs:
pre_install:
- python docs/generate_dataset_mds.py


mkdocs:
configuration: mkdocs.yml

# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/requirements.txt
145 changes: 144 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,155 @@ All notable changes to this project will be documented in this file.
The format is inspired from [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and the versioning aim to respect [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [1.0.0] - 2023-11-07

### Added

- Setup initial structure and files
- Add first bunch of datasets for testing the data flow
- Modularize datasets
- Add first draft of RTD docs
- Add dataset: BNetzA MaStR
- Add Nominatim geocoder
- Add dataset: population
- Clean rule
- Datasets attribute captions
- Create list of region-specific datasets in the docs
- pyproject.toml and poetry.lock file have been added with the conversion to poetry
- Add pre-commit in order to check for errors and linting bugs before commits
- Add types-pyyaml package
- Add dataset: employees and companies
- Add dataset: demandregio electricity demand
- Add dataset: BMWK long-term scenarios
- Add rules to download raw data (zipped) from cloud, extract and copy to 'store/raw'
- Add module 'data_io', containing relevant helper functions (downloading, extracting, copying, cleaning)
- Create metadata
- Add script to generate dataset md files for documentation
- Add dataset: demand_heat_region
- Add dataset: heatpump_cop
- Add dataset: stala_st_energy
- Add dataset: eurostat_lau
- Add dataset: regstat_energy
- Add dataset: dwd_temperature
- Add dataset: ageb_energy_balance
- Add dataset: seenergies_peta5
- Add dataset: renewables.ninja_feedin
- Add dataset: renewable_feedin
- Add dataset: bnetza_mastr_correction_region and correct wrong units
- Integrate building of energy system for appdata in pipeline via dir *esys*
- Update store with dir structure for *esys* data
- Add creation of empty time series for the *esys*
- Add dataset: rpg_abw_regional_plan
- Add dataset: potentialarea_wind_region
- Add writing of default values to *esys* raw scalar data
- Add datasets: rli_pv_wfr and rli_pv_wfr_region
- Add module appdata to workflow
- Add dataset: geodata_infolayers
- Add dataset: potentialarea_pv_ground_region
- Add dataset: app datapackage
- Add dataset: potentialarea_pv_roof_region
- Add dataset: technology_data
- Add dataset: settings
- Calc panel settings from datasets
- Add dataset: osm (Germany)
- Add dataset: osm_buildings and add stats on ground areas
- Add mapping of costs and efficiencies from store/raw to store/datasets
- Add dataset: emissions
- Add captions to app datapackage (here: MaStR, heat, potentialarea_wind)
- Add mapping of time series data in datasets to empty time series according to
the mapping provided in map_ts.yml
- Add build configuration for readthedocs
- Add creation of stats of development over time for bnetza_mastr_wind_region,
bnetza_mastr_pv_ground_region, bnetza_mastr_pv_roof_region
- Add dataset: dbfz_biomass_heat_capacities
- Add the calculation of relative demand of biomass conversion technologies via
their relative capacities
- Add deletion of all data in store/datasets/esys_raw/data
- Add notes on OSM download and run resources
- Add nominal power per wind turbine for 2045
- Add technology data for batteries
- Add technology data for thermal storages
- Add definition of relevant regions (NUTS3) in global configuration file
- Add documentation of energy system of ABW

### Changed

- Move dataset docs from config to md files
- Retain mastr_id in MaStR datasets
- Fix loading of empty yml files
- Fix loading of global config when in workflow dir
- Integrate esys Snakefile in workflow Snakefile and update clean rule
- Fix shapely deprecation warning
- Fix ogr2ogr conversion with recent GDAL version (v3.6.2)
- Fix conda installation by removing python-gdal from environment.yml
- The package management in digipipe has been changed to poetry.
- The installation of a virtual environment is done only from the environment.yml file and via conda.
- Apply linters on repo among others: black, isort, check-json and end-of-file-fixer
- Update population with prognoses from demandregio dataset
- Fix C419 flake8 error
- Switch to mkdocs for documentation (Sphinx deleted)
- Normalize renewable feedin timeseries
- Fix instruction to obtain raw files
- Translate all dataset.md files to German
- Exchange *Test_scenario* with *2045_scenario* in *digipipe/esys/scenarios*
- Split each demand per sector in *esys*
- File .gitignore again includes ignoring of esys appdata
- pv_roof area stats: distinguish between all and non-historic buildings
- storage units: add region-wide values for spec. capacity and power for those
connected to PV roof units
- Add data on installed el. power to bmwk_long_term_scenarios
- Disaggregate PV state targets to region in potentialarea_pv_ground_region
- Adapt osm_filtered to ose osm dataset and extract building data
- Disaggregate PV state targets to region in potentialarea_pv_roof_region
- Changes were applied to the energy system. Among others RoR, small batteries
and biogas were added. A distinction was made between centralized and
decentralized CHPs
- Scenario 2045_scenario needs default_scalars.csv instead of scalars.csv
- By default set costs and efficiencies of esys are written to
default_scalars.csv instead of default_costs_efficiencies.csv
- Default variable_costs are passed with input_parameters for storages
- Pass time series instead of scalar with efficiency for central heat pump
- Fix wind+pv_ground default values in panel settings
- Set all default control values in panel settings
- Kick biogas shortage
- Rename dataset captions to app_captions
- Move app settings to datasets and include in app datapackage
- Adapt 2045_scenario.yml so that time series with values are used instead of
empty ts
- Suppress warning of loosing data in source and comment columns while
unstacking if they are empty
- Change max. installable PV roof capacity in panel settings
- Fix panel settings for large batteries
- Add additional captions to MaStR captions
- Use LTS version of OSM
- The unstacking of time series in esys was fixed so that warning is given if
there is at least one value in columns 'source' or 'comment'
- Minor fix applied reformatting with black
- Only use operating units from mastr for municipality stats and temporal
development
- Heat pump ASHP/GSHP split fixed
- Replace the relative demand of biomass with the relative demand of each
biomass conversion technology
- Fix clean rule
- Update raw datapackage URL
- Restrict snakemake version to v7.32.0
- Add central heat pump targets to slider
- Restrict heat pump sliders to not move under 50%
- Fix pv ground slider values to prevent app to alter SQ value from panel
settings
- Fix PV roof slider values
- Add HP share slider from-max values to prevent 100 % HP share
- Updated technology_data dataset.md and metadata
- Update metadate in store_raw
- Adapt existing dataset rules to use the global region definition (NUTS3)
- Update list of region-specific datasets

### Removed

- setup.py and requirements.txt files are omitted with the conversion to poetry
- sphinx from poetry environment
- Remove dataset: osm_forest
- Obsolete targets from rule all
- Merge dataset costs_efficiencies into technology_data
- Merge dataset costs_efficiencies into technology_data
- Remove values for redundant subsliders from app datapackage
1 change: 0 additions & 1 deletion PIPELINE.md

This file was deleted.

Loading

0 comments on commit 9bb33f3

Please sign in to comment.