Plugins are not applied to custom directive

[WalkerD243]

Deployment Type

Docker

Version

v2.0.4

Steps to Reproduce

1. Create directives.yaml:
   juniper-bgp-route:
   name: BGP Route Custom
   plugins:
   - "/etc/hyperglass/plugins/transform_plugin.py"
   rules:
   - condition: 0.0.0.0/0
   commands: show route protocol bgp table inet.0 {target} detail | display json
   - condition: ::/0
   commands: show route protocol bgp table inet6.0 {target} detail | display json
   field:
   description: 'IPv4 or IPv6 Address'
   validation: '[0-9a-f.:/]+'
   juniper-received-from-peer:
   name: Received from Peer
   plugins:
   - /etc/hyperglass/custom_plugins/transform_plugin.py
   rules:
   - condition: ''
   commands: show route receive-protocol bgp {target}

   field:
   description: 'Your IPv4 or IPv6 BGP Peer Address'
   validation: '[0-9a-f.:/]+'

2. Create plugin based on https://hyperglass.dev/plugins
   from ipaddress import ip_network
   from hyperglass.plugins import InputPlugin

class TransformCIDR(InputPlugin):
def transform(self, query):
(target := query.query_target)
target_network = ip_network(target)
if target_network.version == 4:
return f"{target_network.network_address!s} {target_network.netmask!s}"
return target

3. put the transform_plugin.py in /etc/hyperglass/plugins and /etc/hyperglass/custom_plugins folder

4. check that the container can see the plugins:
   docker container exec hyperglass-hyperglass-1 cat /etc/hyperglass/plugins/transform_plugin.py
   docker container exec hyperglass-hyperglass-1 cat /etc/hyperglass/custom_plugins/transform_plugin.py

5. enable debug mode like in https://hyperglass.dev/installation/environment-variables

6. enter the hyperglass site and enter "192.0.2.0/24" in the query as in https://hyperglass.dev/plugins example

7. observe the live output from the container and look at the line:
   hyperglass-1 | [DEBUG] 20241030 14:26:43 |129 | queries → Constructed query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24'], 'constructed_query': ['show route protocol bgp table inet.0 192.0.2.0/24 detail | display json']}

8. look at the result in the webinterface, as it should transform 192.0.2.0/24 to 192.0.2.0 255.255.255.0 and the query is executed on a juniper device it should return a syntax error at the 255.255.255.0

Expected Behavior

I would expect that the constructed_query line form step 7. above would be:
hyperglass-1 | [DEBUG] 20241030 14:26:43 |129 | queries → Constructed query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24'], 'constructed_query': ['show route protocol bgp table inet.0 192.0.2.0 255.255.255.0 detail | display json']}

and the result in the webinterface should return a syntax error at the 255.255.255.0 as it is executed on a juniper device in my case

Observed Behavior

I get the line:
hyperglass-1 | [DEBUG] 20241030 14:26:43 |129 | queries → Constructed query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24'], 'constructed_query': ['show route protocol bgp table inet.0 192.0.2.0 255.255.255.0 detail | display json']}

and a normal result in the webinterface as if the query wasn't changed by the plugin

For me it seems like the plugin is not applied at all, even if change the last line in the plugin config to "return 'test'" the query is still successfull. I tried different folders in /etc/hyperglass and giving 777 full rights to the folder and files inside.

The same thing happens with the output plugins, please have a look. this is ssuch a great platform!

Configuration

web:
    logo:
        light: /etc/hyperglass/a.svg
        dark: /etc/hyperglass/a.svg
        favicon: /etc/hyperglass/a.svg
        width: 40%
        height: null
    text:
        title: Looking Glass
        subtitle: provided by hyperglass
        title_mode: all

Devices

devices:
  - name: Oldenburg Test
    address: 
    credential:
      username: 
      password: 
    platform: juniper
    directives:
      - builtins: false
      - juniper-bgp-route
      - juniper-received-from-peer
    attrs:
      source4: 
      source6:

Logs

hyperglass-1  | [INFO] 20241030 14:26:30 |1762 | callHandlers → 80.228.255.62:52285 - "GET / HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:26:31 |1762 | callHandlers → 80.228.255.62:52285 - "GET /_next/static/chunks/webpack-790cb4468e180930.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:26:31 |1762 | callHandlers → 80.228.255.62:52286 - "GET /_next/static/nz0t2HFziAXTmh8bM1vdX/_buildManifest.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:26:31 |1762 | callHandlers → 80.228.255.62:52287 - "GET /_next/static/nz0t2HFziAXTmh8bM1vdX/_ssgManifest.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:26:31 |1762 | callHandlers → 80.228.255.62:52287 - "GET /_next/static/chunks/915.4213c6fa1c4191ba.js HTTP/1.1" 200 {}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |111 | membership → Checking target membership {'target': '192.0.2.0/24', 'network': '0.0.0.0/0'}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |116 | membership → Target membership verified {'target': '192.0.2.0/24', 'network': '0.0.0.0/0'}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |123 | in_range → Target is in range {'target': '192.0.2.0/24', 'range': '0-32'}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |107 | validate_query_target → Validation passed {'query': Query(query_location='oldenburg_test', query_target=['192.0.2.0/24'], query_type='juniper-bgp-route')}
hyperglass-1  | [INFO] 20241030 14:26:43 |79 | query → Starting query execution {'query': Query(query_location='oldenburg_test', query_target=['192.0.2.0/24'], query_type='juniper-bgp-route')}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |97 | query → Cache miss {'query': Query(query_location='oldenburg_test', query_target=['192.0.2.0/24'], query_type='juniper-bgp-route'), 'cache_key': 'hyperglass.query.bd2cab8688de5e3c1830c75380ace8f313a2ef493f56a31e6afefc6840e5135f'}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |51 | execute →  {'query': Query(query_location='oldenburg_test', query_target=['192.0.2.0/24'], query_type='juniper-bgp-route'), 'device': 'oldenburg_test'}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |46 | __init__ → Constructing query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24']}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |129 | queries → Constructed query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24'], 'constructed_query': ['show route protocol bgp table inet.0 192.0.2.0/24 detail | display json']}
hyperglass-1  | [DEBUG] 20241030 14:26:43 |51 | collect → Connecting to device {'device': 'Oldenburg Test', 'address': 'None:None', 'proxy': None}
hyperglass-1  | [DEBUG] 20241030 14:26:52 |115 | query → Runtime: 8.6084 seconds {'query': Query(query_location='oldenburg_test', query_target=['192.0.2.0/24'], query_type='juniper-bgp-route')}
hyperglass-1  | [DEBUG] 20241030 14:26:52 |134 | query → Response cached {'query': Query(query_location='oldenburg_test', query_target=['192.0.2.0/24'], query_type='juniper-bgp-route'), 'cache_timeout': 120}
hyperglass-1  | [INFO] 20241030 14:26:52 |146 | query → Execution completed {'query': Query(query_location='oldenburg_test', query_target=['192.0.2.0/24'], query_type='juniper-bgp-route')}
hyperglass-1  | [INFO] 20241030 14:26:52 |1762 | callHandlers → 80.228.255.62:52292 - "POST /api/query HTTP/1.1" 201 {}
hyperglass-1  | [WARNING] 20241030 14:29:21 |1762 | callHandlers → Invalid HTTP request received. {}
hyperglass-1  | [INFO] 20241030 14:29:32 |1762 | callHandlers → 152.32.245.93:58616 - "GET / HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:37 |1762 | callHandlers → 152.32.245.93:47778 - "GET /images/favicons/favicon-196x196.png HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:38 |1762 | callHandlers → 152.32.245.93:47786 - "GET /robots.txt HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:38 |1762 | callHandlers → 152.32.245.93:47798 - "GET /sitemap.xml HTTP/1.1" 404 {}
hyperglass-1  | [INFO] 20241030 14:29:38 |1762 | callHandlers → 152.32.245.93:47826 - "GET /_next/static/chunks/polyfills-c67a75d1b6f99dc8.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:38 |1762 | callHandlers → 152.32.245.93:47824 - "GET /_next/static/nz0t2HFziAXTmh8bM1vdX/_buildManifest.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:38 |1762 | callHandlers → 152.32.245.93:47802 - "GET /_next/static/chunks/webpack-790cb4468e180930.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:38 |1762 | callHandlers → 152.32.245.93:47816 - "GET /_next/static/chunks/pages/_app-e5cf2d2230ac6396.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:38 |1762 | callHandlers → 152.32.245.93:47840 - "GET /_next/static/chunks/pages/index-a1a889711ef3362f.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:39 |1762 | callHandlers → 152.32.245.93:47866 - "GET /_next/static/chunks/main-113d9e0de0ebdeca.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:39 |1762 | callHandlers → 152.32.245.93:47850 - "GET /_next/static/chunks/framework-f856061fb4a8c9e6.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:39 |1762 | callHandlers → 152.32.245.93:47878 - "GET /_next/static/nz0t2HFziAXTmh8bM1vdX/_ssgManifest.js HTTP/1.1" 200 {}
hyperglass-1  | [INFO] 20241030 14:29:40 |1762 | callHandlers → 152.32.245.93:47892 - "GET /config.json HTTP/1.1" 404 {}

[N0Ball]

I have walked through the code and find out two possiblitiy location that might cause this bug.
at hyperglass/models/directive.py:317 validate_plugins function

316     @field_validator("plugins")
317     def validate_plugins(cls: "Directive", plugins: t.List[str]) -> t.List[str]:
318         """Validate and register configured plugins."""
319         plugin_dir = Settings.app_path / "plugins"
320
321         if plugin_dir.exists():
322             # Path objects whose file names match configured file names, should work
323             # whether or not file extension is specified.
324             matching_plugins = (
325                 f
326                 for f in plugin_dir.iterdir()
327                 if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
328             )
329             return [str(f) for f in matching_plugins]
330         return []

1. I think that () should be [], or else the results are always an empty array
2. According to plugin_dir, I think that currently it only reads from the file /etc/hyperglass/plugins. Instead of any location provided by the derivatives.

Therefore, the current solution is to

1. Change () from line 324 to line 328 to []
2. Put your plugin files in plugins directory which is /etc/hyperglass/plugins

[WalkerD243]

Thank you very much for your time.
I changed directiy.py as to your solution to:
324 matching_plugins = [
325 f
326 for f in plugin_dir.iterdir()
327 if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
328 ]

And put the plugins into the plugins directory but have the same results.
I tried to put a print statement into the function in directive.py (without the arrows):
def validate_plugins(cls: "Directive", plugins: t.List[str]) -> t.List[str]:
"""Validate and register configured plugins."""
plugin_dir = Settings.app_path / "plugins"
--->print("PLUGIN_DIR: " + plugin_dir)<----------
if plugin_dir.exists():
# Path objects whose file names match configured file names, should work
# whether or not file extension is specified.
matching_plugins = [
f
for f in plugin_dir.iterdir()
if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
]
return [str(f) for f in matching_plugins]
return []

And the live output from the container doesnt show PLUGIN_DIR... should that be there or will a print in that function not work?

[N0Ball]

It seems that with docker, the code is actually run at /opt/hyperglass, /etc/hyperglass are only for setting files. My suggestion is to also mount the directory to /opt/hyperglass
e.i.

        volumes:
            - ${HYPERGLASS_APP_PATH-/etc/hyperglass}:/etc/hyperglass
            - ${HYPERGLASS_APP_PATH-/etc/hyperglass}:/opt/hyperglass

if you have done correclty, it should print something out.

[WalkerD243]

Thanks for the suggestion ! I changed the volumes as above and got the error:

hyperglass-1  | /usr/local/bin/python3: Error while finding module specification for 'hyperglass.console' (ModuleNotFoundError: No module named 'hyperglass')
hyperglass-1 exited with code 1

when starting the container. I changed
- ${HYPERGLASS_APP_PATH-/etc/hyperglass}:/opt/hyperglass
to |
- ${HYPERGLASS_APP_PATH-/opt/hyperglass}:/opt/hyperglass

and it works as expected, printing the output directly into the container output on the cli.

With that I could troubleshoot a bit and found a potential solution:
In /opt/hyperglass/hyperglass/models/directive.py I added some prints to show the contents of plugins and plugin_dir.iterdir() beginning in line 316:

@field_validator("plugins")
    def validate_plugins(cls: "Directive", plugins: t.List[str]) -> t.List[str]:
        """Validate and register configured plugins."""
        plugin_dir = Settings.app_path / "plugins"
        if plugin_dir.exists():
            # Path objects whose file names match configured file names, should work
            # whether or not file extension is specified.


----> for p in plugins:
                print(f"ALLOWED PLUGIN: {p.split(".")[0]}")
                for f in plugin_dir.iterdir():
                    print(f"FOUND PLUGIN: {f.name.split(".")[0]}")


            matching_plugins = (
                f
                for f in plugin_dir.iterdir()
                if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
            )
            return [str(f) for f in matching_plugins]
        return []

and got the following result from the container:

hyperglass-1  | ALLOWED PLUGIN: /etc/hyperglass/plugins/transform_plugin
hyperglass-1  | FOUND PLUGIN: transform_plugin
hyperglass-1  | ALLOWED PLUGIN: /etc/hyperglass/plugins/transform_plugin
hyperglass-1  | FOUND PLUGIN: transform_plugin

To fix that problem I have following two solutions:

1: Swap lines 324-328 from

matching_plugins = (
                f
                for f in plugin_dir.iterdir()
                if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
            )

to

matching_plugins = []
           for f in plugin_dir.iterdir():
               for p in plugin_dir.iterdir():
                   if f.name.split(".")[0] == p.name.split(".")[0]:
                       matching_plugins.append(f)

2: Transform the strings in plugins.iterdir() to Path Objects and insert a .name. in the comparison of f and p:

plugins = [Path(p) for p in plugins]
matching_plugins = (
    f
    for f in plugin_dir.iterdir()
    if f.name.split(".")[0] in (p.name.split(".")[0] for p in plugins)
)

But with approach 2. you have to also add
from pathlib import Path
at the beginning.

With these changes applied I am now getting:

hyperglass-1  | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'BGPRoutePluginArista'}
hyperglass-1  | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'BGPRoutePluginJuniper'}
hyperglass-1  | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'MikrotikGarbageOutput'}
hyperglass-1  | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'RemoveCommand'}
hyperglass-1  | [INFO] 20241105 11:57:24 |118 | register → Registered plugin {'type': 'input', 'name': 'TransformCIDR'}

What do you think about it, maybe theres a better solution.

[N0Ball]

Well, since I am not the author/maintainer of this project. I think that we should leave the answer to him; however he is disappeared from the Internet currently. Still, love to see that you have solve the problem.
As for my opinion of the solutions

matching_plugins = (
    f
    for f in plugin_dir.iterdir()
    if f.name.split(".")[0] in (p.name.split(".")[0] for Path(p) in plugins)
)

I think solution 2 with a slightly change might be a better solution, since it have the least change to the original code.
Also I think that you might have some typo in solution 1?

matching_plugins = []
           for f in plugin_dir.iterdir():
               for p in plugins:
                   if f.name.split(".")[0] == p.split(".")[0]:
                       matching_plugins.append(f)

Not tested yet, but I think this is your original intention?

[WalkerD243]

Thanks for your help ! Definitely the cleaner approach but the Path(p) throws errors on my instance. I kept

plugins = [Path(p) for p in plugins] matching_plugins = ( f for f in plugin_dir.iterdir() if f.name.split(".")[0] in (p.name.split(".")[0] for p in plugins) )

for further testing and discovered that my transform_input.py plugin is registered and executed. The line
target_network = ip_network(target)
throws errors as well but this is okay for me as I'm going to create own rules in the plugin.

I created a Output plugin juniper-bgp-route.py:

from hyperglass.plugins import OutputPlugin
 
print("OUTPUT PLUGIN CALLED")        
        
class TestOutput(OutputPlugin):
    def process(self, output, query):
        print("FUNCTION CALLED")
        return f"Plugin Test Successfull: {query}"

which gets registered in the container start but not executed after the query ( yes i bound that in another command in the directives.yaml). For further troubleshooting I did in /opt/hyperglass/hyperglass/plugins/_manager.py:

class OutputPluginManager(PluginManager[OutputPlugin], type="output"):
    """Manage Output Processing Plugins."""

    def execute(self: "OutputPluginManager", *, output: OutputType, query: "Query") -> OutputType:
        """Execute all output parsing plugins.

        The result of each plugin is passed to the next plugin.
        """
        print(f"execute in OutputPluginManager Class in _manager.py called with query id: {query.directive.id} and platform: {query.device.platform}")
        result = output
        for plugin in self.plugins():
            print(f"self plugin: {plugin}")
            print(f"plugin directives: {plugin.directives}")
            print(f"plugin platform: {plugin.platforms}")
        
        directives = (
            plugin
            for plugin in self.plugins()
            if query.directive.id in plugin.directives and query.device.platform in plugin.platforms
        )
        print(f"DIRECTIVES: {directives}")
        common = (plugin for plugin in self.plugins() if plugin.common is True)
        for plugin in (*directives, *common):
            log.bind(plugin=plugin.name, value=result).debug("Output Plugin Starting Value")
            result = plugin.process(output=result, query=query)
            log.bind(plugin=plugin.name, value=result).debug("Output Plugin Ending Value")

            if result is False:
                return result
            # Pass the result of each plugin to the next plugin.
        return result

and did the query as before and now got the output:

hyperglass-1 | [DEBUG] 20241108 07:26:43 |51 | collect → Connecting to device {'device': 'Oldenburg Test', 'address': 'None:None', 'proxy': None}
hyperglass-1 | execute in OutputPluginManager Class in _manager.py called with query id: juniper-received-from-peer and platform: juniper
hyperglass-1 | self plugin: TestOutput
hyperglass-1 | plugin directives: ('juniper-received-from-peer',)
hyperglass-1 | plugin platform: ()
hyperglass-1 | self plugin: BGPRoutePluginArista
hyperglass-1 | plugin directives: ('hyperglass_arista_eos_bgp_route_table', 'hyperglass_arista_eos_bgp_aspath_table', 'hyperglass_arista_eos_bgp_community_table')
hyperglass-1 | plugin platform: ('arista_eos',)
hyperglass-1 | self plugin: BGPRoutePluginJuniper
hyperglass-1 | plugin directives: ('hyperglass_juniper_bgp_route_table', 'hyperglass_juniper_bgp_aspath_table', 'hyperglass_juniper_bgp_community_table')
hyperglass-1 | plugin platform: ('juniper',)
hyperglass-1 | self plugin: MikrotikGarbageOutput
hyperglass-1 | plugin directives: ('hyperglass_mikrotik_bgp_aspath', 'hyperglass_mikrotik_bgp_community', 'hyperglass_mikrotik_bgp_route', 'hyperglass_mikrotik_ping', 'hyperglass_mikrotik_traceroute')
hyperglass-1 | plugin platform: ('mikrotik_routeros', 'mikrotik_switchos')
hyperglass-1 | self plugin: RemoveCommand
hyperglass-1 | plugin directives: ()
hyperglass-1 | plugin platform: ()
hyperglass-1 | DIRECTIVES: <generator object OutputPluginManager.execute.. at 0x7f7229b42740>
hyperglass-1 | [DEBUG] 20241108 07:26:51 |115 | query → Runtime: 8.3 seconds {'query': Query(query_location='oldenburg_test', query_target=['8.8.8.8'], query_type='juniper-received-from-peer')}

so it looks like there's no platform set for the plugin and therefore the plugin fails the check in directives = (
plugin
for plugin in self.plugins()
if query.directive.id in plugin.directives and query.device.platform in plugin.platforms
)

based on the builtin plugins i added the marked line to my plugin juniper-bgp-route.py:

import re, json
from hyperglass.plugins import OutputPlugin
------> from typing import List, Sequence <-----------
 
print("OUTPUT PLUGIN CALLED")        
        
class TestOutput(OutputPlugin):
    -------> platforms: Sequence[str] = ("juniper",) <----------------
    def process(self, output, query):
        print("FUNCTION CALLED")
        return f"Plugin Test Successfull: {query}"

Now the output plugin works as expected and the query output is Plugin Test Successfull...

You mentioned that the creator is currently not responding on this project therefore I'm thinking about forking my changes, so that others with the same problems have a solution until the creator fixes these problems in this original repository .

With that in mind I would also remove the platform check in the /opt/hyperglass/hyperglass/plugins/_manager.py from

directives = (
            plugin
            for plugin in self.plugins()
            if query.directive.id in plugin.directives and query.device.platform in plugin.platforms
        )

to

directives = (
            plugin
            for plugin in self.plugins()
            if query.directive.id in plugin.directives
        ) 

[N0Ball]

I am also thinking of forking one and fix other issues. Perhaps create a hyperglass 2 or something. Do U mind inviting U to create a PR after the new hyperglass is created?

[WalkerD243]

Sure would be glad, after all your help.