Feature Request - Support for User-Defined Payload Definition Dictionary

[nmichaels]

Adding proprietary NMEA extensions appears to have been broken with 1.0.43.

How I added my messages:

#nmea/nmea.py:

from pynmeagps import *

get_dict = nmeatypes_get.NMEA_PAYLOADS_GET
msgids = nmeatypes_core.NMEA_MSGIDS_PROP

msgids["MYMSG"] = "My Special Message"

get_dict["MYMSG"] = {
    "data": IN,
}

# nmea/__init__.py:
from .nmea import *

In a Python shell:

>> import nmea
>> nmea.NMEAReader.parse(b'PMYMSG,0*01').data

What I see with 1.0.41:

0

What I see with 1.0.42:

nmea.NMEAReader.parse(b'PMYMSG,0*01').data
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NMEAMessage' object has no attribute 'data'

With qlxnmea==1.0.42

Environment:

Debian bookworm

$ uname -a
Linux phase 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux
$ pip freeze
pynmeagps==1.0.42

I believe this was introduced by the addition of a default NMEA message type in 126dae2

[nmichaels]

nmea.zip
Here's the directory with the example code so you don't have to type it out.

[semuadmin]

Hi @nmichaels,

I can see how the changes in v1.0.41 & v1.0.42 would have affected your code.

In v1.0.41 a change was introduced to allow pynmeagps to parse any unknown (but structurally valid) NMEA sentence to a nominal NMEAMessage data structure, but the attributes of that nominal data structure will always be field_01, field_02, etc.

In v1.0.42 I basically split out the proprietary message definitions to a separate file nmeatypes_get_prop.py, as the original nmeatypes_get.py file was starting to get a bit unwieldy.

So, to take your example, it is now possible to parse any 'unknown' sentence like b'$PMYMSG,0*01\r\n' as follows:

(remember - a valid NMEA sentence must start with "$" (0x24) and end with "\r\n" (0x0D0A))

from pynmeagps import NMEAReader

msg = NMEAReader.parse(b"$PMYMSG,0*01\r\n")
print(msg)
print(msg.field_01)
msg = NMEAReader.parse(b"$PMYMSG2,0,THISFIELD,THATFIELD,2,3,4,5,*10\r\n")
print(msg)
print(msg.field_03)

<NMEA(PMYMSG, NOMINAL, field_01=0)>
0
<NMEA(PMYMSG2, NOMINAL, field_01=0, field_02=THISFIELD, field_03=THATFIELD, field_04=2, field_05=3, field_06=4, field_07=5, field_08=)>
THATFIELD

But I'm not sure your code was ever an intended use case for pynmeagps. You appear to be trying to dynamically create new 'proprietary' NMEA sentence types and then parse them - I'm not clear what you would be looking to achieve by doing this? If you can help me understand the objective here, I can evaluate whether it's best accomodated as a change to pynmeagps, or a change to your own code.

Thanks

[nmichaels]

nmea.zip

Whoops, there's the zip file with the files actually in it. I didn't change the files though; weird that you couldn't reproduce without changes. Anyway, it sounds like you do see what's going on so I'll explain my use case.

First, why I'm doing this: I'm developing a device that spits out NMEA in its normal course of operation, and I want to include some debug information in-band. This lets me use a single parser for all the messages which simplifies my test scripts.

So to do that, I added some proprietary NMEA sentences and a separate package that depends on pynmeagps. My scripts can just install my-nmea and through its parser they inherit all the standard NMEA stuff plus parsers for my debug messages. I read the extensibility section of the README and it seemed like what it suggested ("Additional message types can be readily added to the appropriate dictionary") was what I did. The only alternatives I see are to try to push my debug messages upstream to pynmeagps, which doesn't make sense, or to fork pynmeagps, which seems like too big a hammer.

Do you have any suggestions?

[nmichaels]

I think I could work with wrapping field_01 etc. It would mean wrapping the pynmeagps parser interface, since I want my tests to have a single interface for parsing messages (that's the half the point of using NMEA).

An externally-defined dictionary that adds temporarily to the core dictionaries would certainly be preferable from my point of view. I suspect it wouldn't have any performance impact unless it was used, but that's the kind of thing I'd need a profiler to prove.

[semuadmin]

@nmichaels some further thoughts on this...

Adding support for a "user-defined" payload definition dictionary is, in itself, fairly straightforward provided each user-defined message type has a unique identifier as dictionary key e.g. "MYMSG": {}.

While this is true for all standard NMEA messages, it's not true for most proprietary messages, which typically use a common manufacturer's mnemonic code ("UBX", "FEC", etc.) with individual message types being identified by a msgId (or similar) attribute within the payload itself (so the unique key is in effect mnemonic+msgId). Some manufacturers (e.g. Trimble) use a combination of unique and non-unique message identifiers, necessitating special processing within pynmeagps.

Implementing a user-defined dictionary facility which catered for all possible proprietary unique message identifier schemes would, imho, be more trouble than its worth.

I'm minded, therefore, to stick with the recommendation to use pynmeagps's 'nominal' payload parsing facilty for unknown message types and/or simply fork the library and merge the new definitions upstream once they're stable.

But open to other opinions.

[nmichaels]

I don't particularly care about allowing for all possible proprietary unique message identifier schemes, and YAGNI says not to do it. I do think it's worth providing a way to extend the existing set though. Maybe an additional extension set that hooked into the default message handler could satisfy your performance and API cleanliness concerns.

[semuadmin]

@nmichaels

How about this...?

#69

A new positional argument userdefined of type dict in NMEAReader and NMEAMessage, defaulting to None. If defined, NMEAMessage will attempt to look in this userdefined dict for any payload definition not already included in the existing NMEA_PAYLOADS_GET or NMEA_PAYLOADS_GET_PROP dictionary constants?

Here's an illustrative test case. The file pygpsdata-userdefined.log contains the following two sentences:

$U1XYZ,102929.00,4427.04024,N,00114.41560,W*10
$U2ZYX,103829.00,somedata,somemoredata*2A

    def testuserdefined(self):  # test user-defined payload definition dictionary
        EXPECTED_RESULTS = (
            "<NMEA(U1XYZ, fixutc=10:29:29, lat=44.4506706667, NS=N, lon=-1.24026, EW=W)>",
            "<NMEA(U2ZYX, fixutc=10:38:29, data1=somedata, data2=somemoredata)>",
        )
        USERDEFINED = {
            "XYZ": {"fixutc": TM, "lat": LA, "NS": LAD, "lon": LN, "EW": LND},
            "ZYX": {"fixutc": TM, "data1": ST, "data2": ST},
        }
        with open(os.path.join(DIRNAME, "pygpsdata-userdefined.log"), "rb") as stream:
            ubr = NMEAReader(
                stream, quitonerror=ERR_RAISE, nmeaonly=True, userdefined=USERDEFINED
            )
            i = 0
            for raw, parsed in ubr:
                self.assertEqual(str(parsed), EXPECTED_RESULTS[i])
                i += 1

Caveats:

• the user-defined dictionary would have to conform to the format used in NMEA_PAYLOADS_GET.
• each 3-letter message ID (XYZ, ZYX, etc.) would have to uniquely define the sentence type - i.e. no reference to any msgId sub-definition field in the payload itself.
• user-defined sentences must be GET (msgmode=0) message types.

Would this meet your needs?

[nmichaels]

That did meet my needs, thanks! Sorry it took so long for me to get back to this, but I wanted to test it thoroughly before answering.