v9.02 update mess - redundant information in MQTT

[bcutter]

PROBLEM DESCRIPTION

The update process from v9.01 to v9.02 was very tricky (update to latest version using the web interface, in the first attempt the ping timed out and a power cycle was needed, then switching from a strange partition "reboot to Nuki Hub" and initiating the update once again).

Now after successfully managing to upgrade to v9.02, there's a lot of things mixed up in MQTT looking at what is available in Home Assistant now:

Existing device, fine:

New device, unexpected and with redundant entities:

Is this a desired outcome or the result of a faulty update?
I don't get why one would need to have a sub set of entities redundantly available, one named by the lock (as it was for years) and in addition now one named by the Nuki Hub device.

If it's intentional (looks like as somehow the actual Nuki device is referenced as "linked device" at the bottom), I guess it's safe to disable all redundant entities or even delete the whole device, isn't it?

Quite confusing.

Please clarify.

Aside from this, in general, I'd really love to get back to Nuki Hub updates which a) don't create difficulties on really every single update and b) as a result don't waste a huge amount of time (for the first Nuki Hub now already one hour including this issure report). Stable updates are a long time ago unfortunately. Maybe we shouldn't pack too much changes in a minor release, looking at the quantity of 9.02 seems a bit oversized, just my 2 cents.

REQUESTED INFORMATION

Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!

• Read the Contributing Guide and Policy and the Code of Conduct
• Searched the problem in issues
• Searched the problem in discussions
• Searched the problem in the docs
• Searched the problem in the chat
• Device used (e.g., ESP32-S3): ESP32 (LOLIN32 Lite)
• Nuki Hub binary firmware version number used: v9.02 (build 11995573722.4.1)
  • Pre-compiled
  • Self-compiled
• Provide the output of http://nukihubIP/info:

System Information

------------ NUKI HUB ------------
Version: 9.02
Build: 11995573722.4.1
Build type: Release
Build date: 2024-11-24
Updater version: 9.02
Updater build: 11995573722.4.1
Updater build date: 2024-11-24
Uptime (min): 12
Config version: 902
Last restart reason FW: RequestedViaWebServer
Last restart reason ESP: ESP_RST_SW: Software reset via esp_restart.
Free internal heap: 66976
Total internal heap: 260580
PSRAM Available: No
Network task stack high watermark: 8316
Nuki task stack high watermark: 5088

------------ GENERAL SETTINGS ------------
Network task stack size: 12288
Nuki task stack size: 8192
Check for updates: Yes
Latest version: 9.02
Allow update from MQTT: No
Web configurator username: ***
Web configurator password: ***
Web configurator enabled: Yes
Publish debug information enabled: No
MQTT log enabled: No
Webserial enabled: No
Bootloop protection enabled: No

------------ NETWORK ------------
Network device: Built-in Wi-Fi
Network connected: Yes
IP Address: xxx.xxx.xxx.xxx
SSID: XXX
BSSID of AP: XX:XX:XX:XX:XX:XX
ESP32 MAC address: XX:XX:XX:XX:XX:XX

------------ NETWORK SETTINGS ------------
Nuki Hub hostname: ESP-Nuki-Hub-XXX-XXX
DHCP enabled: Yes
RSSI Publish interval (s): Disabled
Find WiFi AP with strongest signal: Yes
Restart ESP32 on network disconnect enabled: Yes
MQTT Timeout until restart (s): Disabled

------------ MQTT ------------
MQTT connected: Yes
MQTT broker address: xxx.xxx.xxx.xxx
MQTT broker port: 1883
MQTT username: ***
MQTT password: ***
MQTT base topic: xxx
MQTT SSL CA: Not set
MQTT SSL CRT: Not set
MQTT SSL Key: Not set

------------ BLUETOOTH ------------
Bluetooth TX power (dB): 9
Bluetooth command nr of retries: 5
Bluetooth command retry delay (ms): 100
Seconds until reboot when no BLE beacons received: 60

------------ QUERY / PUBLISH SETTINGS ------------
Lock/Opener state query interval (s): 3600
Publish Nuki device authorization log: Yes
Max authorization log entries to retrieve: 5
Battery state query interval (s): 7200
Most non-JSON MQTT topics disabled: No
Publish Nuki device config: Yes
Config query interval (s): 3600
Publish Keypad info: No
Keypad query interval (s): 1800
Enable Keypad control: Yes
Publish Keypad topic per entry: No
Publish Keypad codes: No
Allow checking Keypad codes: No
Max keypad entries to retrieve: 10
Publish timecontrol info: No
Keypad query interval (s): 1800
Enable timecontrol control: No
Publish timecontrol topic per entry: No
Max timecontrol entries to retrieve: 10

------------ HOME ASSISTANT ------------
Home Assistant auto discovery enabled: Yes
Home Assistant auto discovery topic: homeassistant/
Nuki Hub configuration URL for HA: http://xxx.xxx.xxx.xxx

------------ NUKI LOCK ------------
Lock enabled: Yes
Paired: Yes
Nuki Hub device ID: XXXXXXXXX
Nuki device ID: ***
Firmware version: 3.10.7
Hardware version: 4.10
Valid PIN set: Yes
Has door sensor: No
Has keypad: No
Timecontrol highest entries count: 0
Register as: Bridge

------------ HYBRID MODE ------------
Hybrid mode enabled: No

------------ NUKI LOCK ACL ------------
Lock: Allowed
Unlock: Allowed
Unlatch: Allowed
Lock N Go: Allowed
Lock N Go Unlatch: Allowed
Full Lock: Allowed
Fob Action 1: Allowed
Fob Action 2: Allowed
Fob Action 3: Allowed

------------ NUKI LOCK CONFIG ACL ------------
Name: Disallowed
Latitude: Disallowed
Longitude: Disallowed
Auto Unlatch: Disallowed
Pairing enabled: Allowed
Button enabled: Allowed
LED flash enabled: Allowed
LED brightness: Allowed
Timezone offset: Disallowed
DST mode: Disallowed
Fob Action 1: Disallowed
Fob Action 2: Disallowed
Fob Action 3: Disallowed
Single Lock: Allowed
Advertising Mode: Disallowed
Timezone ID: Disallowed
Unlocked Position Offset Degrees: Disallowed
Locked Position Offset Degrees: Disallowed
Single Locked Position Offset Degrees: Disallowed
Unlocked To Locked Transition Offset Degrees: Disallowed
Lock n Go timeout: Allowed
Single button press action: Allowed
Double button press action: Allowed
Detached cylinder: Disallowed
Battery type: Disallowed
Automatic battery type detection: Disallowed
Unlatch duration: Allowed
Auto lock timeout: Disallowed
Auto unlock disabled: Disallowed
Nightmode enabled: Disallowed
Nightmode start time: Disallowed
Nightmode end time: Disallowed
Nightmode auto lock enabled: Disallowed
Nightmode auto unlock disabled: Disallowed
Nightmode immediate lock on start: Disallowed
Auto lock enabled: Disallowed
Immediate auto lock enabled: Disallowed
Auto update enabled: Disallowed
Reboot Nuki: Allowed

------------ NUKI OPENER ------------
Opener enabled: No

------------ GPIO ------------

TO REPRODUCE

Update NH v9.01 to NH v9.02

EXPECTED BEHAVIOUR

No new device in MQTT, no redundant set of entities

SCREENSHOTS

If applicable, add screenshots to help explain your problem.

ADDITIONAL CONTEXT

Add any other context about the problem here.

(Please, remember to close the issue when the problem has been addressed)

[bcutter]

Note:

Same update experience with a 2nd Nuki Hub from 9.01 to 9.02.

After ping times out, a physical power cycle of the ESP is needed.

Then, the web interface presents this (after the initial update tab times out long time ago, like http://esp-nuki-hub-door.domain/autoupdate?release=1&release=1&token=5028):

Reboot to Nuki Hub:

Then: nothing has happened, still on 9.01

So second attempt with the green button:

For half a second a two-liner is visible before the page times out (http://esp-nuki-hub-door.domain/autoupdate?release=1&release=1&token=1737). Reloading the default page after a few minutes now gives the 9.02 update.

Now I had to reboot the Nuki Hub once again to make HA entities show up again at all (all unavailable). Besides that, the same mess in MQTT (HA to be precise) of course.

I decided to simply disable all entities (because all of them are redundant end therefore completely useless) of the new additional Nuki Hub MQTT device.

BUT: the second Nuki Hub MQTT device doesn't even has the "linked device" entry: seems another thing is broken here... I'm pretty done with NH updates for today.

[alexdelprete]

I confirm this new "nukihub" device creation with duplicate diagnostic entities created.

I wouldn't call this a "mess", we have a different view of that word. I have experienced real messes in HA, and this not even close to those. I feel your frustration, but let's not exaggerate, you can simply disable those duplicate entities for now, no real production issues.

[bcutter]

Been there done that, see

I decided to simply disable all entities (because all of them are redundant end therefore completely useless) of the new additional Nuki Hub MQTT device.

Maybe a bit lost in translation. Redundant trash data is what I call a mess. Nevermind, "solved" (worked around) that for the moment.

[tmakowka]

Hi everyone,

i'm not sure whether it's the same issue, but i updated to 9.02 from 9.0 today. I also had to do the ota update process twice, but it worked afterwards.

The problem started after restart of Homeassistant. Now HA is telling me that 'This entity is no longer being provided by the mqtt integration. If the entity is no longer in use, delete it in settings.'

Neither nukihub nor the lock.* entities are active any more. I don't see a duplicate nukihub entity in my setup.

Anyone got an idea how to fix this?

[bcutter]

Luckily I don't see that behavior which would be a real issue. But: I did not restart HA yet.

I guess you should at first check in your MQTT broker if the Nuki Hub topics exist and if - once you restart a Nuki Hub - the corresponding topics in the homeassistant domain are created.

Besides the overall update issues I tend to say this is a separate topic.

We'll see how the maintainers will handle this.

[tmakowka]

Hi @bcutter after downgrading and upgrading HA again (this happended during a bump from 2024.11.1 to 2024.11.3), i restarted nukihub and voila.. it's now working again.

Don't know what caused this, but it seems to be solved by now.

[technyon]

So can this be closed then, or does it need further investigation?

[bcutter]

Of course it requires further investigation.

The actual doubling of a majority of entities is the main issue here (the other minor one the strange update process coming from 9.0/9.01).

Only @tmakowka 's specific issue seems to be solved.

[alexdelprete]

Of course it requires further investigation.

I had only 4 entities duplicated. BTW: they're not really duplicates, after the upgrade the ones on the lock were orphaned, I simply deleted them.

Can you show which entities are "duplicated"?

The logic is that the two devices in HA represent the two physical devices: the esp hub and the Nuki Lock. Problem is that some diagnostic entities were part of the lock and they're now moved to the hub, but they're not updated in both.

[bcutter]

For me all entities remain on the lock, and many have been duplicated to the NH.

Those are the 13 duplicates (another entity ID, same function). Same for another NH:

[alexdelprete]

Those are not duplicate entities, they are orphaned (they're not being updated), since the "live" entities are now on the nukihub device. You simply need to delete them. Difference is I had 4 of them.

You show the nukihub device, please show the same entities also on the lock. The ones on the hub should be "live" the ones on the lock should be unavailable (orphaned).

[bcutter]

No, lock entities are not orphaned. Continue to work just fine afaics. One lock for example:

[alexdelprete]

IP and FW Update sensors, for example, should only be available on the nukihub device, not on the lock device. There are other 2-3 diag. sensors like that. You should see those orphaned on the lock, and active on the nukihub device.

If that's not the case, @iranl / @technyon will probably need more info to understand what's happening in your case.

In my case it was very simple, I just needed to delete 4 orphaned sensors from the lock.

[bcutter]

That's what it is for all my NHs.

Tell me what is needed in terms of information/logs (or: safely purge all MQTT topics once and restart NHs?).

[iranl]

The new NukiHub device separate from the lock and opener devices are there by design.

This is the proper way of providing hub/coordinator etc. devices in Home Assistant and is for example the same as how it is done by Zigbee2MQTT.
As the Nuki Lock, Nuki Opener and NukiHub are separate devices they should be presented in HA as separate devices,

This actually prevents duplicate entities going forward as previously NukiHub IP/Firmware etc. was provided to HA in both lock and opener devices, causing duplicates for users that used both these devices through one NukiHub device.

This was introduced based on a request in #494 in PR #515

It does cause some orphaned and/or duplicate entities in HA for users upgrading from earlier versions.

To fully remove these you need to search for MQTT topics containing "nuki_ using a program like MQTT Explorer and delete all found topics. Reboot NukiHub afterwards and than remove any orphaned/unavailable entities from the Nuki lock and/or opener device from inside Home Assistant.

[bcutter]

I'll have a look at MQTT topics at the weekend. You mean looking for the nuki_ like topics in the homeassistant or the nuki hub root topic?

Curious to see if really only 4 entities are affected. With a bit of entity renaming everything should be back to normal in theory...

---

In general I would like to be made aware of such changes in the release notes. As I read them. Sometimes twice. And then decide if the amount of changes makes this a 2 minutes stressless update or a one hour pain or something in between.

Better a note than discovering strange behavior (not even consistent: one nuki hub has a link to the lock in HA, the other one does not) afterwards with the need of creating issues.

Fundamental changes usually require such changes to be flagged as breaking change or similar. Better safe than sorry (a little extra note is overall probably less struggle than handling issues I guess).

[bcutter]

OK folks @iranl I need a bit more precise instruction on how to check/clean and hopefully finally fix this.

So I repeat:

You mean looking for the nuki_ like topics in the homeassistant or the nuki hub root topic?

Following...

To fully remove these you need to search for MQTT topics containing "nuki_ using a program like MQTT Explorer and delete all found topics.

...blindly would result in huge trouble, as on the broker a lot of topics are found, even one not directly related to NH. So I need to know the root nodes to start searching, making sure to delete the right topics.

Otherwise it's literally impossible to detect which entities should be orphaned on the lock device. As mentioned I have potentially 13 new entities on the NH device. Confusing. Maybe that's the better approach: delete all mqtt topics OF THE LOCK node for which an entity exists in HA in the new NH device (manual deduplication). What do you think?

This is the MQTT/homeassistant/sensor topic:

• above node seems to be the lock, bottom one the new Nuki Hub device

What should be there and what shouldn't? How to safely remove things? Precise instruction please, I don't want to render my locks in my smart home broken because of this annoying upgrade rework.

It gets even more difficult, e. g. looking at the switch domain:

there's no "nuki_" prefix on the topic at all. above lock, below nuki hub (tested with the webserver setting):

---

Edit: something's really broken here. I deleted both webserver topics and restarted the NH. Afterwards, a completely new MQTT device with the NH device default naming got recreated, all entities with new entity ids and enabled. Webserver entity is still "unknown". Checking the topic content it refers to "stat_t":"~/maintenance/webserver/state", which is not existing.

Really starts to annoy me. Never touch a running system, so much wasted time already... another 1.5 hours today! 😠

[bcutter]

To sum up my adventure of the last 3 hours which included fixing a endless reboot issue (#538 (comment)):

• "switch.esp_nuki_hub_door_b_restart_nuki_hub" is still "unknown",
• same for "switch.esp_nuki_hub_door_b_nuki_hub_webserver_enabled"

there's no reset or webserver topic in the mqtt "nuki_door_b/maintenance" section which might be the reason for this. The Nuki Hub simply does not create it. It did not no matter which version of NH was running: 9.02 or 9.03-master6.

Done with NH (for today), gosh it totally broke my sunday evening. for nothing.