PCIe Corals Issue #918

Justahobby01 · 2021-03-19T14:57:57Z

Justahobby01
Mar 19, 2021

When I enable a second Coral PCIe like this:

 detectors:
  coral_pci1:
    type: edgetpu
    device: pci:0
  coral_pci2:
    type: edgetpu
    device: pci:1

I am getting this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 152, in load_delegate
    delegate = Delegate(library, options)

  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 111, in __init__
    raise ValueError(capture.message)

ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()

  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)

  File "/opt/frigate/frigate/edgetpu.py", line 124, in run_detector
    object_detector = LocalObjectDetector(tf_device=tf_device, num_threads=num_threads)

  File "/opt/frigate/frigate/edgetpu.py", line 63, in __init__
    edge_tpu_delegate = load_delegate('libedgetpu.so.1.0', device_config)

  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 154, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(

ValueError: Failed to load delegate from libedgetpu.so.1.0

Commenting out the second coral gets things back up and running. Should I be specifying which camera should use which detector?

blakeblackshear · 2021-03-19T15:12:38Z

blakeblackshear
Mar 19, 2021
Maintainer

What about this?

 detectors:
  coral_pci1:
    type: edgetpu
    device: pci:1

0 replies

Justahobby01 · 2021-03-19T15:46:15Z

Justahobby01
Mar 19, 2021
Author

Yep, tried that and it is the second one 'pci:1' that isn't working. I figured that would be the easy part since I already got the first one in. The lxc recognizes the second module and I passed through the entire /dev to docker. Not sure what I can or should do differently for the second one. Is Frigate using the pipeline for multiple tpus?

0 replies

Justahobby01 · 2021-03-19T15:57:18Z

Justahobby01
Mar 19, 2021
Author

In case I'm just blind to something obvious....

  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: blakeblackshear/frigate:stable-amd64
    devices:
      - /dev/apex_0:/dev/apex_0
      - /dev/apex_1:/dev/apex_1
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /docker/frigate/config/config.yml:/config/config.yml:ro
      - /docker/frigate/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - '5000:5000'
      - '1935:1935' # RTMP feeds
    environment:
      FRIGATE_RTSP_PASSWORD: '000000'

0 replies

Justahobby01 · 2021-03-19T16:17:13Z

Justahobby01
Mar 19, 2021
Author

Running lspci -v | more

shows something interesting that I don't know what to make of. Flags are different and the memory shows disabled on the second module.

02:00.0 System peripheral: Device 1ac1:089a (prog-if ff)
        Subsystem: Device 1ac1:089a
        **Flags: bus master, fast devsel, latency 0, IRQ 17
        Memory at f0300000 (64-bit, prefetchable) [size=16K]
        Memory at f0200000 (64-bit, prefetchable) [size=1M]**
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable+ Count=128 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
        Capabilities: [108] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [200] Advanced Error Reporting
        Kernel driver in use: apex

03:00.0 System peripheral: Device 1ac1:089a (prog-if ff)
        Subsystem: Device 1ac1:089a
        **Flags: fast devsel, IRQ 19
        Memory at f0100000 (64-bit, prefetchable) [disabled] [size=16K]
        Memory at f0000000 (64-bit, prefetchable) [disabled] [size=1M]**
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable- Count=128 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
        Capabilities: [108] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [200] Advanced Error Reporting
        Kernel driver in use: apex

0 replies

Justahobby01 · 2021-03-19T17:04:52Z

Justahobby01
Mar 19, 2021
Author

I re-did the coral driver install and everything started working again. Walked away, came back, and it was throwing the same error about failed to detect the second one again.

0 replies

Justahobby01 · 2021-03-20T03:34:54Z

Justahobby01
Mar 20, 2021
Author

So, I finally had the bright idea to move the coral device to a new pcie slot and now it is working. Running '''dmesg | grep apex''' was the final clue that it had to be hardware.

0 replies

Justahobby01 · 2021-03-20T18:22:24Z

Justahobby01
Mar 20, 2021
Author

Spoke to soon. It crashed again overnight. To rule out the proxmox complexity I did a fresh install of Linux and then straight to docker and installing frigate. Same issue. At this point I'm throwing in the towel but if anyone else has this issue and figures it out, please let me know. Thanks.

These logs are after the TPUs are found and start working. Detections will work just fine. Then, after some random amount of time (seconds, minutes, or hours) it starts throwing this error:

peewee_migrate                 INFO    : Starting migrations


There is nothing to migrate

peewee_migrate                 INFO    : There is nothing to migrate

frigate.mqtt                   INFO    : MQTT connected


detector.coral_pci1            INFO    : Starting detection process: 42

detector.coral_pci2            INFO    : Starting detection process: 44


frigate.app                    INFO    : Camera processor started for lorex_1: 49


Process detector:coral_pci1:


Process detector:coral_pci2:


frigate.edgetpu                INFO    : Attempting to load TPU as pci:0


frigate.edgetpu                INFO    : Attempting to load TPU as pci:1

frigate.edgetpu                INFO    : No EdgeTPU detected.

frigate.app                    INFO    : Camera processor started for lorex_2: 53


frigate.edgetpu                INFO    : No EdgeTPU detected.

frigate.app                    INFO    : Camera processor started for lorex_3: 57


frigate.app                    INFO    : Camera processor started for w6_backyard: 60

frigate.app                    INFO    : Capture process started for lorex_1: 61

frigate.app                    INFO    : Capture process started for lorex_2: 64


frigate.app                    INFO    : Capture process started for lorex_3: 67

frigate.app                    INFO    : Capture process started for w6_backyard: 70

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 152, in load_delegate

    delegate = Delegate(library, options)


  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 111, in __init__

    raise ValueError(capture.message)


ValueError

During handling of the above exception, another exception occurred:



Traceback (most recent call last):

Traceback (most recent call last):


  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap

    self.run()

  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run

    self._target(*self._args, **self._kwargs)

  File "/opt/frigate/frigate/edgetpu.py", line 124, in run_detector


    object_detector = LocalObjectDetector(tf_device=tf_device, num_threads=num_threads)

  File "/opt/frigate/frigate/edgetpu.py", line 63, in __init__

    edge_tpu_delegate = load_delegate('libedgetpu.so.1.0', device_config)


  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 152, in load_delegate

    delegate = Delegate(library, options)


  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 111, in __init__

    raise ValueError(capture.message)

  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 154, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(


ValueError: Failed to load delegate from libedgetpu.so.1.0


ValueError


During handling of the above exception, another exception occurred


Traceback (most recent call last):


  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()

  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run


    self._target(*self._args, **self._kwargs)

  File "/opt/frigate/frigate/edgetpu.py", line 124, in run_detector


    object_detector = LocalObjectDetector(tf_device=tf_device, num_threads=num_threads)

  File "/opt/frigate/frigate/edgetpu.py", line 63, in __init__

    edge_tpu_delegate = load_delegate('libedgetpu.so.1.0', device_config)


  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 154, in load_delegate

    raise ValueError('Failed to load delegate from {}\n{}'.format(


ValueError: Failed to load delegate from libedgetpu.so.1.0

frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...


frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...

frigate.app                    INFO    : Stopping...

frigate.events                 INFO    : Exiting event processor...


frigate.events                 INFO    : Exiting event cleanup...

frigate.record                 INFO    : Exiting recording maintenance...


frigate.object_processing      INFO    : Exiting object processor...

frigate.watchdog               INFO    : Exiting watchdog...

0 replies

TurboTronix · 2021-09-06T13:39:45Z

TurboTronix
Sep 6, 2021

Was this ever resolved? I never got my pcie coral tpu to work inside my docker. I'm using a Win10 PC.

0 replies

oguime · 2021-09-08T17:51:42Z

oguime
Sep 8, 2021

Hi Justahobby01,
Are you running the Frigate Docker on a LXC? Would you mind listing any special configuration, if needed?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PCIe Corals Issue #918

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

PCIe Corals Issue #918

Justahobby01 Mar 19, 2021

Replies: 9 comments

blakeblackshear Mar 19, 2021 Maintainer

Justahobby01 Mar 19, 2021 Author

Justahobby01 Mar 19, 2021 Author

Justahobby01 Mar 19, 2021 Author

Justahobby01 Mar 19, 2021 Author

Justahobby01 Mar 20, 2021 Author

Justahobby01 Mar 20, 2021 Author

TurboTronix Sep 6, 2021

oguime Sep 8, 2021

Justahobby01
Mar 19, 2021

blakeblackshear
Mar 19, 2021
Maintainer

Justahobby01
Mar 19, 2021
Author

Justahobby01
Mar 19, 2021
Author

Justahobby01
Mar 19, 2021
Author

Justahobby01
Mar 19, 2021
Author

Justahobby01
Mar 20, 2021
Author

Justahobby01
Mar 20, 2021
Author

TurboTronix
Sep 6, 2021

oguime
Sep 8, 2021