Introduce (experimental) DALI proxy

[jantonguirao]

Category:

New feature

Description:

• DALI proxy is a new way to integrate DALI pipelines with existing torch data loading pipelines
• The idea is that torch data processes send data to the main process, where it is processed by DALI before handling it over to the training loop.
• The solution allows for mixing data loading from Pytorch with partial processing on DALI

Co-author: @mdabek-nvidia

Additional information:

Affected modules and functionalities:

• Torch plugin

Key points relevant for the review:

dali/python/nvidia/dali/plugin/pytorch/proxy/__init__.py

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • [?] RST
  • [?] Jupyter
  • Other
• N/A
  TODO

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A

[dali-automaton]

CI MESSAGE: [20865442]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [20865442]: BUILD PASSED

[szkarpinski]

I see there are no tests except for the resnet50 example. I believe we should have normal TL0 tests as well.

[szkarpinski]

Did you test error propagation between the processes? Multiprocessing doesn't automagically propagate exceptions afaik. Maybe we should have tests to check how are the errors in particular processes reported?

[klecki]

Posting few comments as the code started moving.
I also feel like we should limit the scope of the API and hide most of the implementation.

[dali-automaton]

CI MESSAGE: [21050826]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [21050826]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [21052236]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [21052236]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [22445006]: BUILD PASSED

[dali-automaton]

CI MESSAGE: [22484757]: BUILD STARTED

[mzient]

Suggested change

	if not DALIServer._needs_conversion(obj):
	result = obj
	# Named tuple: Reconstruct using `_replace`
	elif hasattr(obj, "_replace") and hasattr(obj, "_fields"):
	# Named tuple: Reconstruct using `_replace`
	if hasattr(obj, "_replace") and hasattr(obj, "_fields"):

[mzient]

Suggested change

	def _need_conversion(obj, need_conversion_cache):
	@staticmethod
	def _need_conversion(obj, need_conversion_cache):

[dali-automaton]

CI MESSAGE: [22487905]: BUILD STARTED

Comments were addressed but I did only partial review.

[jantonguirao]

!build

[dali-automaton]

CI MESSAGE: [22491477]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [22491477]: BUILD PASSED