RCP fails to recover on transmit timeout

[adamatom]

Hello OpenThread community,

I am working with an ESP32 H2 as my RCP. My host is an ESP32S3. I have an issue where if in my host RCP, I somehow block my transmissions for longer the transmission and trigger the RCP recovery mechanism, it will end up in a cycle of detecting failed transmissions, attempting recovery, declare success, repeat:

2025-02-13 22:50:37 W(373468) OPENTHREAD:[W] P-RadioSpinel-: radio tx timeout
2025-02-13 22:50:37 W(373468) OPENTHREAD:[W] P-RadioSpinel-: RCP failure detected
2025-02-13 22:50:37 W(373468) OPENTHREAD:[W] P-RadioSpinel-: Trying to recover (1/3)
2025-02-13 22:50:37 I(373498) OPENTHREAD:[N] P-RadioSpinel-: RCP recovery is done
2025-02-13 22:50:43 W(379528) OPENTHREAD:[W] P-RadioSpinel-: radio tx timeout
2025-02-13 22:50:43 W(379528) OPENTHREAD:[W] P-RadioSpinel-: RCP failure detected
2025-02-13 22:50:43 W(379528) OPENTHREAD:[W] P-RadioSpinel-: Trying to recover (1/3)
2025-02-13 22:50:43 I(379558) OPENTHREAD:[N] P-RadioSpinel-: RCP recovery is done
2025-02-13 22:50:50 W(386928) OPENTHREAD:[W] P-RadioSpinel-: radio tx timeout
2025-02-13 22:50:50 W(386928) OPENTHREAD:[W] P-RadioSpinel-: RCP failure detected
2025-02-13 22:50:50 W(386928) OPENTHREAD:[W] P-RadioSpinel-: Trying to recover (1/3)
2025-02-13 22:50:50 I(386958) OPENTHREAD:[N] P-RadioSpinel-: RCP recovery is done

I can trigger this on demand by starting an energy scan that requires more than the default tx timeout of 5 seconds, but I don't suspect the energy scan itself.

While debugging, I found that when the SpinelDriver::ResetCoprocessor is invoked (which seems like a reasonable step in recovering the RCP), the reset request is actually aborted due to this line: https://github.com/openthread/openthread/blob/main/src/lib/spinel/spinel_driver.cpp#L128. mIsCoprocessorReady is latched to true at some point in SpinelDriver::HandleInitialFrame.

I have tested a hack that wraps the VerifyOrExit exit point like this:

#if OPENTHREAD_CONFIG_MULTIPAN_RCP_ENABLE
    // Avoid resetting the device twice in a row in Multipan RCP architecture
    VerifyOrExit(!mIsCoprocessorReady, resetDone = true);
#endif

and found that when this function executes the RCP reset, OpenThread and my RCP recover.

Does anyone have any insight why ResetCoprocessor is implemented to ignore the reset request? My understanding is that multipan is for simulatenously using the radio for both Thread and Zigbee. Im new to OpenThread and can not tell if my #if hack has consequences. Is the check for OPENTHREAD_CONFIG_MULTIPAN_RCP_ENABLE enough? Should the reset really be avoided even when I don't want to support mulitpan?

Thanks for reading and any insight.

[zhanglongxia]

Here is the raw commit 9360 that adds the check conditionVerifyOrExit(!sIsReady, resetDone = true);. The check here is used for resolving the reset issue in multi-pan RCP architecture. This commit adds support for multiple spinel interfaces, and it also introduced the new configuration OPENTHREAD_CONFIG_MULTIPAN_RCP_ENABLE . I think it is reasonable to add the #if OPENTHREAD_CONFIG_MULTIPAN_RCP_ENABLE here.

[adamatom]

Okay, thanks. I have submitted a pull request: #11277

[gytxxsy]

@zhanglongxia
I have another thought on this issue: Perhaps we should add a callback in ResetCoprocessor that gets invoked before DieNow when both software and hardware resets fail. This would allow the user to take some actions to handle the situation (e.g., performing an RCP update from the application code).

[zhanglongxia]

@zhanglongxia I have another thought on this issue: Perhaps we should add a callback in ResetCoprocessor that gets invoked before DieNow when both software and hardware resets fail. This would allow the user to take some actions to handle the situation (e.g., performing an RCP update from the application code).

@gytxxsy I think that is reasonable for your case. You can add a callback to notify the user that both resets are failed before OT calls DieNow.

[gytxxsy]

@zhanglongxia Thanks, I have submitted a pull request: #11284.
However, I noticed something during this process: After #11277 was merged into main, OPENTHREAD_SPINEL_CONFIG_RCP_RESTORATION_MAX_COUNT seems to have lost its significance. This is because if both software and hardware resets fail in ResetCoprocessor, the program will crash directly instead of attempting recovery. Unless ResetCoprocessor succeeds in RecoverFromRcpFailure, but the subsequent property settings and restoration fail—I’m not sure if this scenario is expected.