🙋Data (gas/elektra) geïmporteerd vanuit domotics wel in database, maar alleen gas is zichtbaar in archief e.d. #1657

fjbosscha · 2022-07-04T16:25:04Z

fjbosscha
Jul 4, 2022

Description

Sinds afgelopen week ben ik overgestapt op DSMR-reader. Omdat ik een aantal jaren historie in domoticz heb te zitten, heb ik deze geïmporteerd binnen DSMR-reader. Ik zie dat data wel in de betreffende tabellen is gekomen (dsmr_consumption_electricityconsumption en dsmr_consumption_gasconsumption) maar via de archief optie, zie ik alleen maar gas-waarden.

Ik heb al in de andere issues gekeken, maar zie niet direct wat waar mis gaat.

DSMR-reader version

5.3

DSMR-reader platform

No response

Debug info dump

DSMR-READER
    App / Python / Database                                                        v5.3 / v3.9.2 / postgresql
    BE sleep / DL sleep / Retention / Override                                     1.0s / 1.0s / 672h / False
    Latest telegram version read / Parser settings                                                 "50" / "4"

DATA
    Telegrams total (est.)                                                                             158839
    Consumption records electricity / gas (est.)                                                103613 / 3230

Answered by dennissiemensma

Jul 4, 2022

Je kunt nog dsmr_stats_daystatistics en dsmr_stats_hourstatistics leeggooien en dan in de URL /admin/dsmr_backend/scheduledprocess/ het proces Generate day and hour statistics zoeken, aanpassen naar vandaag + nu.

Dan hergenereert die als het goed is weer de statistieken. In de debug logs zie je dat dan ook.

Je kunt ook nog wat andere proberen, zoals:

sudo su - dsmr
./manage.py dsmr_stats_reconstruct_missing_day_statistics

Nu ik het weer teruglees, die was hier ook voor gemaakt. Zie #1302 (comment). Probeer die anders eerst maar.

View full answer

dennissiemensma · 2022-07-04T16:42:04Z

dennissiemensma
Jul 4, 2022
Maintainer

Je zou #1630 kunnen bekijken. Daar staat een soortgelijk issue.

#1630 (comment)

0 replies

fjbosscha · 2022-07-04T17:28:21Z

fjbosscha
Jul 4, 2022
Author

Bedankt voor je reactie. Die had ik inderdaad ook al gezien, maar mijn data is, in mijn optiek, correct.

dsmrreader=> select read_at from dsmr_consumption_electricityconsumption;
        read_at         
------------------------
 2019-02-08 00:00:00+01
 2019-02-08 23:59:00+01
 2019-02-09 00:00:00+01
 2019-02-11 23:59:00+01
 2019-02-12 00:00:00+01
 2019-02-14 23:59:00+01
 2019-02-15 00:00:00+01
 2019-02-15 23:59:00+01
 2019-02-16 00:00:00+01
 2019-02-16 23:59:00+01
 2019-02-17 00:00:00+01
 2019-02-17 23:59:00+01
 2019-02-18 00:00:00+01
 2019-02-18 23:59:00+01
 2019-02-19 00:00:00+01
 2019-02-19 23:59:00+01
 2019-02-20 00:00:00+01
 2019-02-20 23:59:00+01
 2019-02-21 00:00:00+01
 2019-02-21 23:59:00+01
 2019-02-22 00:00:00+01
 2019-02-22 23:59:00+01
 2019-02-23 00:00:00+01
 2019-02-23 23:59:00+01
 2019-02-24 00:00:00+01
 2019-02-24 23:59:00+01
 2019-02-25 00:00:00+01
 2019-02-25 23:59:00+01
 2019-02-26 00:00:00+01
 2019-02-26 23:59:00+01
 2019-02-27 00:00:00+01
 2019-02-27 23:59:00+01
 2019-02-28 00:00:00+01

dsmrreader=> select read_at from dsmr_consumption_gasconsumption;
        read_at         
------------------------
 2022-06-28 21:45:08+02
 2022-06-28 21:50:01+02
 2020-01-10 23:59:00+01
 2020-01-11 00:00:00+01
 2020-01-11 23:59:00+01
 2020-01-12 00:00:00+01
 2020-01-12 23:59:00+01
 2020-01-13 00:00:00+01
 2020-01-13 23:59:00+01
 2020-01-14 00:00:00+01
 2020-01-14 23:59:00+01
 2020-01-15 00:00:00+01
 2020-01-15 23:59:00+01
 2020-01-16 00:00:00+01
 2020-01-16 23:59:00+01
 2020-01-17 00:00:00+01
 2020-01-17 23:59:00+01
 2020-01-18 00:00:00+01
 2020-01-18 23:59:00+01
 2020-01-19 00:00:00+01
 2020-01-19 23:59:00+01
 2020-01-20 00:00:00+01
 2020-01-20 23:59:00+01
 2020-01-21 00:00:00+01
 2020-01-21 23:59:00+01
 2020-01-22 00:00:00+01
 2020-01-22 23:59:00+01
 2020-01-23 00:00:00+01
 2020-01-23 23:59:00+01

Heb truncate dsmr_stats_daystatistics gedraaid en de processen weer aangezet.

Debug aangezet en zie nu in de dsmr_backend.log allemaal de volgende meldingen:

2022-07-04 19:26:23,478 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:23,497 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:24.490464+02:00 (ETA 0:00:00.999936)
2022-07-04 19:26:23,509 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:23,513 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:24,521 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:26:24,521 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:24,637 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)
2022-07-04 19:26:24,637 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:24,651 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:25.647331+02:00 (ETA 0:00:00.999938)
2022-07-04 19:26:24,660 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:24,664 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:25,670 DEBUG    schedule     execute_scheduled_processes      32 | SP: 3 backend service(s) ready to run
2022-07-04 19:26:25,671 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:25,760 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)
2022-07-04 19:26:25,761 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:25,778 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:26.772076+02:00 (ETA 0:00:00.999945)
2022-07-04 19:26:25,778 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Calculate quarter hour electricity peaks" (dsmr_consumption.services.run_quarter_hour_peaks)
2022-07-04 19:26:25,783 DEBUG    services     run_quarter_hour_peaks           52 | Quarter hour peaks: Using 2022-07-04 19:26:00+02:00 as fuzzy start
2022-07-04 19:26:25,787 DEBUG    services     run_quarter_hour_peaks           70 | Quarter hour peaks: Ready but awaiting any new readings after 2022-07-04 19:15:00+02:00, postponing for a bit...
2022-07-04 19:26:25,793 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Calculate quarter hour electricity peaks" to 2022-07-04 19:26:30+02:00 (ETA 0:00:04.212378)
2022-07-04 19:26:25,803 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:25,809 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:26,816 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:26:26,817 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:26,916 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)

Weet niet of dit moet kloppen!

0 replies

fjbosscha · 2022-07-04T17:33:13Z

fjbosscha
Jul 4, 2022
Author

Zat wat vervuiling in (had de database op een test-systeem gezet, waarbij de mqtt en pvoutput ook liepen.

logging is nu:

2022-07-04 19:32:52,267 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:52,292 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:53.283032+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:52,316 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:53,325 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:53,325 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:53,349 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:54.341110+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:53,368 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:54,376 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:54,377 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:54,400 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:55.392694+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:54,421 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:55,430 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:32:55,430 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:55,454 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:56.446108+02:00 (ETA 0:00:00.999913)
2022-07-04 19:32:55,454 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Calculate quarter hour electricity peaks" (dsmr_consumption.services.run_quarter_hour_peaks)
2022-07-04 19:32:55,461 DEBUG    services     run_quarter_hour_peaks           52 | Quarter hour peaks: Using 2022-07-04 19:32:00+02:00 as fuzzy start
2022-07-04 19:32:55,468 DEBUG    services     run_quarter_hour_peaks           70 | Quarter hour peaks: Ready but awaiting any new readings after 2022-07-04 19:30:00+02:00, postponing for a bit...
2022-07-04 19:32:55,478 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Calculate quarter hour electricity peaks" to 2022-07-04 19:33:00+02:00 (ETA 0:00:04.530197)
2022-07-04 19:32:55,499 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:56,508 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:56,510 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:56,534 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:57.526560+02:00 (ETA 0:00:00.999912)
2022-07-04 19:32:56,556 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:57,565 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:57,567 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:57,592 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:58.583766+02:00 (ETA 0:00:00.999911)
2022-07-04 19:32:57,609 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:58,618 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:58,620 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:58,647 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:59.638447+02:00 (ETA 0:00:00.999905)
2022-07-04 19:32:58,673 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:59,682 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run

0 replies

dennissiemensma · 2022-07-04T17:38:50Z

dennissiemensma
Jul 4, 2022
Maintainer

Hoe draai je DSMR-reader? Docker of native?

0 replies

dennissiemensma · 2022-07-04T17:43:00Z

dennissiemensma
Jul 4, 2022
Maintainer

Staat er wel data in de dsmr_stats_daystatistics tabel?

0 replies

fjbosscha · 2022-07-04T17:50:14Z

fjbosscha
Jul 4, 2022
Author

Ik draai deze native.
Er staat geen data in dsmr_stats_daystatistics

dsmrreader=> select  * from dsmr_stats_daystatistics;                                                                                                                  id | day | total_cost | electricity1 | electricity2 | electricity1_returned | electricity2_returned | electricity1_cost | electricity2_cost | gas | gas_cost | average_temperature | highest_temperature | lowest_temperature | fixed_cost | electricity1_reading | electricity1_returned_reading | electricity2_reading | electricity2_returned_reading | gas_reading 
----+-----+------------+--------------+--------------+-----------------------+-----------------------+-------------------+-------------------+-----+----------+---------------------+---------------------+--------------------+------------+----------------------+-------------------------------+----------------------+-------------------------------+-------------
(0 rows)

0 replies

fjbosscha · 2022-07-04T18:57:47Z

fjbosscha
Jul 4, 2022
Author

ik heb nog even in de "running" tabellen gekeken en daar staan in dsmr_stats_daystatistics wel waarden bij Electricity1_reading, electricity1_returned_reading, electricity2_reading, electricity2_returned_reading en gas.
Wat mij hierbij opvalt is dat hier geen waarden zijn bij electricity1, electricity2, electricity1_returned en electricity2_returned. Bij gas staan er wel de dagelijkse waarden.

0 replies

fjbosscha · 2022-07-04T19:00:04Z

fjbosscha
Jul 4, 2022
Author

  329 | 2020-01-08 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  0.000 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |             9648.898 |                      2956.777 |             8158.299 |                      6917.227 |            
  330 | 2020-01-09 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  0.000 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |             9651.773 |                      2956.777 |             8164.627 |                      6917.659 |            
  331 | 2020-01-11 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.304 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               14.423 |                         0.120 |                0.000 |                         0.000 |       9.134
  332 | 2020-01-12 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  7.424 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               20.704 |                         0.615 |                0.000 |                         0.000 |      15.438
  333 | 2020-01-13 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  7.257 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               29.230 |                         0.615 |                0.000 |                         0.000 |      22.862
  334 | 2020-01-14 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.747 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               30.535 |                         0.615 |                5.621 |                         0.252 |      30.119
  335 | 2020-01-15 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  5.822 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               32.128 |                         0.615 |               11.657 |                         1.695 |      36.866
  336 | 2020-01-16 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.104 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               35.107 |                         0.615 |               18.425 |                         1.695 |      42.688
  337 | 2020-01-17 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.144 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               36.625 |                         0.615 |               25.185 |                         2.612 |

Er is, op 11 januari 2020 wel een meterwissel geweest. Kan dat invloed hierop hebben. Vanaf 11 januari 2020 is de meter aangepast, zodat ook gas nu met de slimmemeter mee komt

0 replies

dennissiemensma · 2022-07-04T19:21:00Z

dennissiemensma
Jul 4, 2022
Maintainer

Klopt je data in dsmr_consumption_electricityconsumption dan wel? Dat is de tussentabel voordat het naar dsmr_stats_daystatisticsgaat

0 replies

dennissiemensma · 2022-07-04T19:25:27Z

dennissiemensma
Jul 4, 2022
Maintainer

Je kunt nog dsmr_stats_daystatistics en dsmr_stats_hourstatistics leeggooien en dan in de URL /admin/dsmr_backend/scheduledprocess/ het proces Generate day and hour statistics zoeken, aanpassen naar vandaag + nu.

Dan hergenereert die als het goed is weer de statistieken. In de debug logs zie je dat dan ook.

Je kunt ook nog wat andere proberen, zoals:

sudo su - dsmr
./manage.py dsmr_stats_reconstruct_missing_day_statistics

Nu ik het weer teruglees, die was hier ook voor gemaakt. Zie #1302 (comment). Probeer die anders eerst maar.

0 replies

PGTrynwalden · 2022-07-04T21:06:42Z

PGTrynwalden
Jul 4, 2022

Hoi Dennis,
Heb het laatste commando gedraaid en de ontbrekende data staat erin.

Dank voor het meedenken.

0 replies

dennissiemensma · 2022-07-05T18:38:11Z

dennissiemensma
Jul 5, 2022
Maintainer

Fijn om te horen! Ik zal het bij de andere issues zetten.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🙋Data (gas/elektra) geïmporteerd vanuit domotics wel in database, maar alleen gas is zichtbaar in archief e.d. #1657

{{title}}

Replies: 12 comments

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

🙋Data (gas/elektra) geïmporteerd vanuit domotics wel in database, maar alleen gas is zichtbaar in archief e.d. #1657

fjbosscha Jul 4, 2022

Description

DSMR-reader version

DSMR-reader platform

Debug info dump

Replies: 12 comments

dennissiemensma Jul 4, 2022 Maintainer

fjbosscha Jul 4, 2022 Author

fjbosscha Jul 4, 2022 Author

dennissiemensma Jul 4, 2022 Maintainer

dennissiemensma Jul 4, 2022 Maintainer

fjbosscha Jul 4, 2022 Author

fjbosscha Jul 4, 2022 Author

fjbosscha Jul 4, 2022 Author

dennissiemensma Jul 4, 2022 Maintainer

dennissiemensma Jul 4, 2022 Maintainer

PGTrynwalden Jul 4, 2022

dennissiemensma Jul 5, 2022 Maintainer

fjbosscha
Jul 4, 2022

dennissiemensma
Jul 4, 2022
Maintainer

fjbosscha
Jul 4, 2022
Author

fjbosscha
Jul 4, 2022
Author

dennissiemensma
Jul 4, 2022
Maintainer

dennissiemensma
Jul 4, 2022
Maintainer

fjbosscha
Jul 4, 2022
Author

fjbosscha
Jul 4, 2022
Author

fjbosscha
Jul 4, 2022
Author

dennissiemensma
Jul 4, 2022
Maintainer

dennissiemensma
Jul 4, 2022
Maintainer

PGTrynwalden
Jul 4, 2022

dennissiemensma
Jul 5, 2022
Maintainer