Async Architecture Topics

[ktbyers]

Discussion on high-level Nornir async architecture.

How sync and async worlds interact
Inventory
Runner
Tasks
Error Handling
Tests
Document
A bunch of others.

[ktbyers]

Do NOT comment here--except for adding: 1)new discussion topics, 2)links to the GH issues where the discussion topics are actually addressed

[ktbyers]

Inventory + Async Discussion: #880

[ktbyers]

Goals (this is a discussion point / changeable)

1. Support async for both runners and for inventory.
2. Support having Nornir as part of a broader async event loop (i.e. Nornir is a subpart and invoked in a broader event loop potentially working on other non-Nornir tasks).
3. Maintain compatibilty with current plugin system. In other words, current sync plugins and sync world should not care nor be broken by addition of async support).
4. The nornir-core code base must be easily maintainable with relatively minimal effort (after adding async support).
5. Sufficient test coverage that changes can be reliably made to nornir-core.
6. Sufficient end-user documentation such that async use and async plugin support can be accomplished directly by end-users and plugin maintainers.
7. Async loaded inventory can be used with sync tasks (and vice versa). Note, this does not preclude adding some flag indicating the data source and some config option that could require async inventory-async tasks. # Added 2023-12-05

Questions

1. Do we allow mixing of the sync-world and async world (albeit with blocking)?
2. What Nornir-config elements are needed to support async?
3. Can plugins incrementally add async support in some way, or does the plugin need to register a new endpoint?
4. Are any fundamental Nornir behaviors changed (outside the adding of async inventory and async runners)? The biggest one I see is the sequential nature of multiple nr.run() calls.

[ktbyers]

@dbarrosop @dgarros @ogenstad Adding a few people to goals thread.

[dbarrosop]

3. Maintain compatibilty with current plugin system. In other words, current sync plugins and sync world should not care nor be broken by addition of async support).

I'd extend this point (or add a new one) that users should be able to pick and mix sync/async plugins. For instance, this should work (pseudocode ahead):

def my_grouped_task(task):
    task.run(task=netmiko_send_command)
    task.run(task=async_http_request)


nr = InitNornir(inventory=ShinyNewAsyncInventoryPlugin)
nr.run(task=netmiko_send_command)
nr.run(task=async_http_request)
nr.run(task=my_grouped_task)

The example will probably end up requiring awaits and maybe even using run_async instead of run but that should be fine. My only point is about being able to mix them.

DISCLAIMER: this opinion of mine may change if this means nornir-core becomes too complex and/or unmaintainable.

Do we allow mixing of the sync-world and async world (albeit with blocking)?

I think we should, specially if nornir's API ends up separating sync and async methods (i.e. run vs run_async although I am not suggesting we should). Even if we don't, if the user cares about non blocking code I'd say they need to do some due diligence and make sure plugins are non-blocking too (maybe we could log warnings). The upside to this is that users that don't care about non-blocking code could leverage new and old plugins seamlessly (let's assume "modern" plugins may only come in async fashion).

Can plugins incrementally add async support in some way, or does the plugin need to register a new endpoint?

I'd say new plugins should be needed. I.e. NetBoxInventoryPluginAsync vs NetBoxInventoryPlugin. Mostly because it will probably make things simpler for nornir-core and both Sync and Async plugins can share as much code as they want without us having to make any assumptions (i.e. http calls are different but the whole post-processing of data is shared). It also makes explicit to the user whether they are using Sync or Async code. Although hopefully that'd be properly documented by the plugin and the user would take the time to read the docs. Because that's what users do, don't they? :P

Are any fundamental Nornir behaviors changed (outside the adding of async inventory and async runners)? The biggest one I see is the sequential nature of multiple nr.run() calls.

Probably Processors which are mostly IO-bound and transform_functions.