[@card] Realtime cards (1/N) (#1550)

* [current.card.refresh] refactor cards to support runtime updates

* [card-refresh] implement card reload policy

* [current.card.components] card component refresh
- added interface to MetaflowCardComponent for making it REALTIME_UPDATABLE
- add realtime component rendering capabiity.
- Changed ville's abstraction of CardComponents to CardComponentManager
- introduced the `current.card.components` / `current.card['abc'].components` interface
- `current.card.components` interface helps access/remove components

* [card-creator] decouple decorator code from card creation
- abstraction to handle card creation for decorator (and in future outside code)
- Create CardProcessManager class that helps manage the card processes running
- Remove `_card_proc` logic from code.
- fix card-refresh-bug caused by cardproc instance-method refactor
- the `_card_proc` method is an instance method of a card decorator which is passed to componentCollector. This is done so that ComponentCollector can call the method when a refresh is called for an individual card.
- Since there is only one copy of the ComponentCollector it created an issue when other cards were trying to call refresh (since ComponentCollector was instantiated with a **single card decorator's** `_card_proc`)
- This commit refactored the code to handle multiple cards calling refresh

* Refactor based on comments + bug fixes + handle more edge cases:
1. `card_cli.py`
    - Refactored the `update_card` method to handle many different cases gracefully.
    - The logic of the function is the same, it just gracefully handles the cases where `render_runtime`/`refresh` are not implemented
    - The main logic responsible rendering error cards now robustly handles cases of timeout and missing implementation of the `render_runtime` method.
    - Added more comments that help with the readability of the code.
2. `card.py`
    - rename `MetaflowCard.IS_RUNTIME_CARD` to `MetaflowCard.RUNTIME_UPDATABLE`
    - rename `MetaflowCardComponent.id` to `MetaflowCardComponent.component_id`
3. `card_creator.py`:
    - fix bug for the edge case were async sidecars handling card creation may not have completed execution.
4. `card_decorator.py`:
    - Refactor based on the rename of `MetaflowCard.IS_RUNTIME_CARD` to `MetaflowCard.RUNTIME_UPDATABLE`
5. `component_serializer.py`:
    - Refactor based on the rename of `MetaflowCardComponent.id` to `MetaflowCardComponent.component_id`
    - Remove the `FIXME` in the `__setitem__` function to not allow overwriting to card components via `current.card.components[ID] = NewComponent`
    - Add functionality to the `current.card.refresh` method that allows re-rendering cards when the layout of the card has changed. This mean that when new components are added to a card or when components are removed from a card then a re-render should be triggered. Modified the `ComponentStore` to expose a property that tells when the component array was last modified. The `CardComponentManager` uses this property in the `ComponentStore` to determine if a re-render should be triggered.
    - Add checks to ensure that objects passed to `current.card.refresh(data)` are JSON serializable. If these are not JSON serializable then warn the user that they are not JSON serializable.
    - introduced a `component_update_ts` filed in the return value of `current.card._get_latest_data` method.
    - Added comments to `current.card._get_lastest_data` for readability.
6. `exception.py`
    - Introduce a new exception for when users try to over-write components.

* [card-client][card-datastore] data read methods
- Added methods to card datastore to read data updates
- Add a private method in card client that can help get the latest data.
- Refactored methods in the code to explictly provide all arguments to disambiguate between data reads and HTML reads.

* [card][tests] verify the behavior of `.refresh` / `.components`

* [card-components][core-change][MET-113]
- component-id setter
- safe-access components that don't exist
- warnings for `update` methods of un-implemented UserComponents
- warnings when components don't exist

* [card-decorator][MET-103] refresh-interval

* [card-client][MET-106] Card.get_data as public fn

* [card-cli][MET-105] Simplify timeout logic

* [card-core][chore][MET-111] Remove unsed imports

* [card-datastore][MET-108] simplify path creation logic

* [component-serializer][MET-108] simplify component storage logic

* [card-creator][MET-109] Fix race-condition btw async/sync card create processes
- There is a case where we could have a async process and another non async process run in parallel. This is something we are not okay with because the async one(render_runtime) can override the values of the sync process.
- This commit fixes this race-condition.
- When we call the final render+refresh, we will wait for any async process to finish complete execution and ensure that both methods/processes complete execution.

* [card-client][MET-117] remove from_resumed and rather derive it from origin_pathspec in CardContainer

* [cleanup][MET-107] typo / hygiene

* [card-core][MET-104] revert `MetaflowCard.render` signature to original
- Introduced a `runtime_data` property that has access to the final data object when `render` is called.
- Remove the usage of `inspect`

---------

Co-authored-by: Ville Tuulos <[email protected]>

metaflow/metaflow_config.py

@@ -165,6 +165,8 @@
 
 SKIP_CARD_DUALWRITE = from_conf("SKIP_CARD_DUALWRITE", False)
 
+RUNTIME_CARD_RENDER_INTERVAL = from_conf("RUNTIME_CARD_RENDER_INTERVAL", 60)
+
 # Azure storage account URL
 AZURE_STORAGE_BLOB_SERVICE_ENDPOINT = from_conf("AZURE_STORAGE_BLOB_SERVICE_ENDPOINT")
 

metaflow/plugins/__init__.py

@@ -166,6 +166,8 @@ def get_plugin_cli():
     TestNonEditableCard,
     TestPathSpecCard,
     TestTimeoutCard,
+    TestRefreshCard,
+    TestRefreshComponentCard,
 )
 
 CARDS = [
@@ -182,5 +184,7 @@ def get_plugin_cli():
     TestNonEditableCard,
     BlankCard,
     DefaultCardJSON,
+    TestRefreshCard,
+    TestRefreshComponentCard,
 ]
 merge_lists(CARDS, MF_EXTERNAL_CARDS, "type")