ERROR: bad argument type for built-in operation

[ctark]

Hello Team!

Thank you for the constant work on this project. Please let me know if you have any ideas on the below, or would like me to test anything!

I am currently using the develop docker branch, and am receiving the following error:
{"level":"info","ts":"2025-01-24T10:35:41.285-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"ERROR:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: bad argument type for built-in operation"}

Agent.yaml: (with api, username and password changed)

orb:
  config_manager:
    active: local
  backends:
    device_discovery:
    common:
      diode:
        target: grpc://192.168.1.221:8080/diode
        api_key: apikey
        agent_name: agent1
  policies:
    device_discovery:
      discovery_1:
        scope:
          - driver: ros
            hostname: 192.168.99.5
            username: username
            password: password

docker log: (error is 2nd last line)

Contains default config file
Setting default_pcap as visor tap
26 is not running
Installing additional drivers
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: librouteros==3.2.0 in /home/appuser/.local/lib/python3.12/site-packages (from -r /opt/orb/drivers.txt (line 1)) (3.2.0)
Requirement already satisfied: napalm-ros in /home/appuser/.local/lib/python3.12/site-packages (from -r /opt/orb/drivers.txt (line 2)) (1.2.6)
Requirement already satisfied: napalm<5.0.0,>=4.0.0 in /home/appuser/.local/lib/python3.12/site-packages (from napalm-ros->-r /opt/orb/drivers.txt (line 2)) (4.1.0)
....
....
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=13.8->netmiko>=4.1.0->napalm<5.0.0,>=4.0.0->napalm-ros->-r /opt/orb/drivers.txt (line 2)) (0.1.2)
Contains default config file
Setting default_pcap as visor tap
Running with config file parameter
Starting orb-agent : /usr/local/bin/orb-agent with args 3
{"level":"info","ts":"2025-01-24T10:35:37.906-0500","caller":"cmd/main.go:82","msg":"backends loaded","backends":{"common":{"diode":{"agent_name":"agent1","api_key":"c0bfff6561d3463bc0848a5b6c6b52cc12e15582","target":"grpc://192.168.1.221:8080/diode"}},"device_discovery":{"host":"localhost","port":"8072"}}}
{"level":"info","ts":"2025-01-24T10:35:37.906-0500","caller":"agent/agent.go:173","msg":"agent started","version":"v1.0.1","routine":"agentRoutine"}
{"level":"info","ts":"2025-01-24T10:35:37.906-0500","caller":"agent/agent.go:109","msg":"registered backends","values":["pktvisor","otel","device_discovery","network_discovery","worker"]}
{"level":"info","ts":"2025-01-24T10:35:37.906-0500","caller":"agent/agent.go:110","msg":"requested backends","values":{"common":{"diode":{"agent_name":"agent1","api_key":"c0bfff6561d3463bc0848a5b6c6b52cc12e15582","target":"grpc://192.168.1.221:8080/diode"}},"device_discovery":{"host":"localhost","port":"8072"}}}
{"level":"info","ts":"2025-01-24T10:35:37.906-0500","caller":"devicediscovery/device_discovery.go:116","msg":"device-discovery startup","arguments":["--host","localhost","--port","8072","--diode-target","grpc://192.168.1.221:8080/diode","--diode-api-key","********","--diode-app-name-prefix","agent1"]}
{"level":"info","ts":"2025-01-24T10:35:38.907-0500","caller":"devicediscovery/device_discovery.go:170","msg":"device-discovery process started","pid":24}
{"level":"error","ts":"2025-01-24T10:35:38.908-0500","caller":"devicediscovery/utils.go:61","msg":"received error from payload","error":"Get \"http://localhost:8072/api/v1/status\": dial tcp [::1]:8072: connect: connection refused"}
{"level":"info","ts":"2025-01-24T10:35:38.908-0500","caller":"devicediscovery/device_discovery.go:180","msg":"device-discovery is not ready, trying again with backoff","backoff backoffDuration":"0s"}
{"level":"error","ts":"2025-01-24T10:35:38.909-0500","caller":"devicediscovery/utils.go:61","msg":"received error from payload","error":"Get \"http://localhost:8072/api/v1/status\": dial tcp [::1]:8072: connect: connection refused"}
{"level":"info","ts":"2025-01-24T10:35:38.909-0500","caller":"devicediscovery/device_discovery.go:180","msg":"device-discovery is not ready, trying again with backoff","backoff backoffDuration":"1s"}
{"level":"info","ts":"2025-01-24T10:35:39.262-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:     Started server process [24]"}
{"level":"info","ts":"2025-01-24T10:35:39.262-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:     Waiting for application startup."}
{"level":"info","ts":"2025-01-24T10:35:39.262-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:     Application startup complete."}
{"level":"info","ts":"2025-01-24T10:35:39.265-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:     Uvicorn running on http://localhost:8072 (Press CTRL+C to quit)"}
{"level":"info","ts":"2025-01-24T10:35:39.924-0500","caller":"devicediscovery/device_discovery.go:141","msg":"device-discovery stdout","log":"INFO:     ::1:60194 - \"GET /api/v1/status HTTP/1.1\" 200 OK"}
{"level":"info","ts":"2025-01-24T10:35:39.925-0500","caller":"devicediscovery/device_discovery.go:176","msg":"device-discovery readiness ok, got version ","device_discovery_version":"0.4.1"}
{"level":"info","ts":"2025-01-24T10:35:39.925-0500","caller":"policyMgr/manager.go:54","msg":"managing agent policy from core","action":"manage","name":"discovery_1","dataset":"b67d096e-ead5-4c1e-8c80-629d293394ea","backend":"device_discovery","id":"","version":1}
{"level":"info","ts":"2025-01-24T10:35:39.927-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.scheduler:Scheduler started"}
{"level":"info","ts":"2025-01-24T10:35:39.928-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: One-time run"}
{"level":"info","ts":"2025-01-24T10:35:39.928-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.scheduler:Added job \"PolicyRunner.run\" to job store \"default\""}
{"level":"info","ts":"2025-01-24T10:35:39.929-0500","caller":"devicediscovery/device_discovery.go:141","msg":"device-discovery stdout","log":"INFO:     ::1:60194 - \"POST /api/v1/policies HTTP/1.1\" 201 Created"}
{"level":"info","ts":"2025-01-24T10:35:39.929-0500","caller":"policyMgr/manager.go:206","msg":"policy applied successfully","policy_id":"","policy_name":"discovery_1"}
{"level":"info","ts":"2025-01-24T10:35:39.930-0500","caller":"agent/agent.go:199","msg":"heartbeat routine started"}
{"level":"info","ts":"2025-01-24T10:35:40.929-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Running job \"PolicyRunner.run (trigger: date[2025-01-24 10:35:40 EST], next run at: 2025-01-24 10:35:40 EST)\" (scheduled at 2025-01-24 10:35:40.928481-05:00)"}
{"level":"info","ts":"2025-01-24T10:35:40.930-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.scheduler:Removed job d8c267ab-7a50-4d02-be67-a3287d03483c"}
{"level":"info","ts":"2025-01-24T10:35:40.930-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: Get driver 'ros'"}
{"level":"info","ts":"2025-01-24T10:35:40.941-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: Getting information"}
{"level":"info","ts":"2025-01-24T10:35:41.285-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"ERROR:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: bad argument type for built-in operation"}
{"level":"info","ts":"2025-01-24T10:35:41.286-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Job \"PolicyRunner.run (trigger: date[2025-01-24 10:35:40 EST], next run at: 2025-01-24 10:35:40 EST)\" executed successfully"}

[ctark]

Hmm interesting, I tried adding my other 3 mikrotik switches, and they all seem to work fine, just that one isn't.
I can connect via napalm CLI to this switch, just like the others.

    device_discovery:
      discovery_1:
        scope:
          - driver: ros
            hostname: "192.168.99.4"
            username: user
            password: pass
          - driver: ros
            hostname: "192.168.99.5"
            username: user
            password: pass

{"level":"info","ts":"2025-01-24T11:00:43.729-0500","caller":"policyMgr/manager.go:206","msg":"policy applied successfully","policy_id":"","policy_name":"discovery_1"}
{"level":"info","ts":"2025-01-24T11:00:43.729-0500","caller":"agent/agent.go:199","msg":"heartbeat routine started"}
{"level":"info","ts":"2025-01-24T11:00:44.728-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Running job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" (scheduled at 2025-01-24 11:00:44.727062-05:00)"}
{"level":"info","ts":"2025-01-24T11:00:44.729-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.4: Get driver 'ros'"}
{"level":"info","ts":"2025-01-24T11:00:44.730-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.scheduler:Removed job 53ef5975-1819-4396-bcf7-76df1c782f5c"}
{"level":"info","ts":"2025-01-24T11:00:44.731-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Running job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" (scheduled at 2025-01-24 11:00:44.727377-05:00)"}
{"level":"info","ts":"2025-01-24T11:00:44.732-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: Get driver 'ros'"}
{"level":"info","ts":"2025-01-24T11:00:44.735-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.scheduler:Removed job b170ce47-46e6-4726-a217-ca9be542f16d"}
{"level":"info","ts":"2025-01-24T11:00:44.737-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Running job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" (scheduled at 2025-01-24 11:00:44.727558-05:00)"}
{"level":"info","ts":"2025-01-24T11:00:44.739-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.scheduler:Removed job 7b91771c-5841-4181-8bb1-9894e26c0877"}
{"level":"info","ts":"2025-01-24T11:00:44.740-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.6: Get driver 'ros'"}
{"level":"info","ts":"2025-01-24T11:00:44.741-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Running job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" (scheduled at 2025-01-24 11:00:44.727729-05:00)"}
{"level":"info","ts":"2025-01-24T11:00:44.742-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.scheduler:Removed job 35689245-6202-425c-b108-e7fa534767c9"}
{"level":"info","ts":"2025-01-24T11:00:44.743-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.7: Get driver 'ros'"}
{"level":"info","ts":"2025-01-24T11:00:44.746-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.4: Getting information"}
{"level":"info","ts":"2025-01-24T11:00:44.747-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: Getting information"}
{"level":"info","ts":"2025-01-24T11:00:44.748-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.6: Getting information"}
{"level":"info","ts":"2025-01-24T11:00:44.748-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.7: Getting information"}
{"level":"info","ts":"2025-01-24T11:00:45.027-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.client:Hostname 192.168.99.7: Successful ingestion"}
{"level":"info","ts":"2025-01-24T11:00:45.029-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" executed successfully"}
{"level":"info","ts":"2025-01-24T11:00:45.033-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.client:Hostname 192.168.99.6: Successful ingestion"}
{"level":"info","ts":"2025-01-24T11:00:45.034-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" executed successfully"}
{"level":"info","ts":"2025-01-24T11:00:45.096-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"ERROR:device_discovery.policy.runner:Policy discovery_1, Hostname 192.168.99.5: bad argument type for built-in operation"}
{"level":"info","ts":"2025-01-24T11:00:45.096-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" executed successfully"}
{"level":"info","ts":"2025-01-24T11:00:45.415-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:device_discovery.client:Hostname 192.168.99.4: Successful ingestion"}
{"level":"info","ts":"2025-01-24T11:00:45.415-0500","caller":"devicediscovery/device_discovery.go:147","msg":"device-discovery stderr","log":"INFO:apscheduler.executors.default:Job \"PolicyRunner.run (trigger: date[2025-01-24 11:00:44 EST], next run at: 2025-01-24 11:00:44 EST)\" executed successfully"}

[leoparente]

Hi @ctark you can try to run this script locally and compare the router that errors out and one that it working fine to see if there is differences

#!/usr/bin/env python
from napalm import get_network_driver
      
# Define device credentials
device_params = {
    "hostname": "192.168.99.5",
    "username": "admin",
    "password": "<password>",
    "optional_args": {}
}

# Get the ros driver
driver = get_network_driver("ros")

# Connect to the device
device = driver(**device_params)

try:
    print("Connecting to the device...")
    device.open()  # Open a connection

    print(device.get_facts())
    print(device.get_interfaces())
    print(device.get_interfaces_ip())

except Exception as e:
  print(f"ERROR: {e}")

finally:
    # Close the connection
    device.close()
    print("Connection closed.")

[ctark]

Hey @leoparente,

I compared the problematic one and a working one (.5 and .4 respectively), and both outputs look very similar, nothing stands out.
I can post or DM the results if you'd like? Still no idea what is different between the 3 hosts that work and the one that doesn't

[leoparente]

With the above code there is no error for .5? It has same responses patterns as .4?

[ctark]

That's Correct, no error for either, 3 lines of output each.

I performed the test in two ways, the first was by going into the running docker container, and executing it from the same shell that the script is running in.
The other way was from a second host that has napalm installed on it too. I compared the results of both and no issues that I can see.
Then I re-ran the orb-agent and it gave me the same error, so I have absolutely no idea why it's not working