Better error message for artifact serialization errors

[tuulos]

When an artifact fails to serialize, we should show a clear error message highlighting the specific artifact that caused trouble.

Context: https://gitter.im/metaflow_org/community?at=5e29e840348292727916743a

[dvukolov]

I believe Ville refers to https://gitter.im/metaflow_org/community?at=5e2f076da9b30639e2f724a7

[tuulos]

another motivating example by @hugobowne using XGBoost DMatrix:

Metaflow 2.5.0 executing ClassificationFlow for user:hba
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2022-02-24 15:59:14.529 Workflow starting (run-id 1645678754524562):
2022-02-24 15:59:14.537 [1645678754524562/start/1 (pid 24093)] Task is starting.
2022-02-24 15:59:16.145 [1645678754524562/start/1 (pid 24093)] Internal error
2022-02-24 15:59:16.146 [1645678754524562/start/1 (pid 24093)] Traceback (most recent call last):
2022-02-24 15:59:16.146 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/cli.py", line 1110, in main
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] start(auto_envvar_prefix="METAFLOW", obj=state)
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] return self.main(args, kwargs)
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/click/core.py", line 1053, in main
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] rv = self.invoke(ctx)
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] return _process_result(sub_ctx.command.invoke(sub_ctx))
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
2022-02-24 15:59:16.147 [1645678754524562/start/1 (pid 24093)] return ctx.invoke(self.callback, ctx.params)
2022-02-24 15:59:16.293 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/click/core.py", line 754, in invoke
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] return __callback(args, kwargs)
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] return f(get_current_context(), args, kwargs)
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/cli.py", line 544, in step
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] task.run_step(
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/task.py", line 625, in run_step
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] output.persist(self.flow)
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/task_datastore.py", line 29, in method
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] return f(self, args, kwargs)
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/task_datastore.py", line 43, in method
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] return f(self, args, kwargs)
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/task_datastore.py", line 714, in persist
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] self.save_artifacts(artifacts_iter(), len_hint=len(valid_artifacts))
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/task_datastore.py", line 29, in method
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] return f(self, args, kwargs)
2022-02-24 15:59:16.294 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/task_datastore.py", line 43, in method
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] return f(self, args, kwargs)
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/task_datastore.py", line 306, in save_artifacts
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] save_result = self._ca_store.save_blobs(pickle_iter(), len_hint=len_hint)
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/content_addressed_store.py", line 100, in save_blobs
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] self._storage_impl.save_bytes(packing_iter(), overwrite=True, len_hint=len_hint)
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/local_storage.py", line 108, in save_bytes
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] for path, obj in path_and_bytes_iter:
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/content_addressed_store.py", line 78, in packing_iter
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] for blob in blob_iter:
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] File "/Users/hba/opt/anaconda3/envs/mf-demo-dl-3/lib/python3.8/site-packages/metaflow/datastore/task_datastore.py", line 280, in pickle_iter
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] blob = pickle.dumps(obj, protocol=2)
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] ValueError: ctypes objects containing pointers cannot be pickled
2022-02-24 15:59:16.295 [1645678754524562/start/1 (pid 24093)] 
2022-02-24 15:59:16.296 [1645678754524562/start/1 (pid 24093)] Task failed.
2022-02-24 15:59:16.296 Workflow failed.
2022-02-24 15:59:16.296 Terminating 0 active tasks...
2022-02-24 15:59:16.296 Flushing logs...
    Step failure:
    Step start (task-id 1) failed.