How would one design an architecture based on channels to modify/process a single document (csv/excel) for a data science dashboard project?

[kalideir]

I am building a dashboard to do some data science tasks on various csv/xlsx files. Each operation requires a varying amount of time. As a normal HTTP-based system might not be suitable for tasks that might have longer processing times than the default timeout, I am intending to utilize this package to build an architecture where each connection get instantiated with an instance of a data frame/file, and the subsequent operations/tasks/requests modify the same file that was opened previously.

Is such a system doable with channels? and is my reasoning about the system reliable? what could be a better alternative to build a backend for long-processing requests for data science projects?

here are the tasks that I would be using inside this project?

- upload dataset
- get all columns
- get datatypes
- get dataset description/info
- modify datatypes
- create new column set
- agg data
- visualize data
- data cleaning and so on

Note that I am doing the processing on the Django/rest side and the interface with Reactjs.
Thanks

[mondwan]

Since I am not quite understand what you need exactly, you may need something more at all. My recommendation is to start dumb first, and advance your project step by step.

• Start dumb

By simple deployment with Daphne + Django, you should be able to implement most of the APIs you mentioned. Although certain http requests are long waiting stuff, they can be handled on both browser side and server side by multi-threading stuff. If performances still acceptable for you, no points to complex the setup

• Next, integrate with Websocket

For certain long running requests, it maybe nice if the server can tell current progress to the UI. Websocket can solve this need nicely. So, in this stage, you need to integrate the websocket stuff in your deployment

• About background tasks with Channels Worker

In certain moments, you really need to run some background tasks. You can first try the Worker stuff provided by Channels. According to the manual, it fits certain simple background tasks requirement. In this stage, you need much more stuff in your deployment to make things work.

By combining solutions from websocket, and Channel worker on certain naive background tasks, it should solve most of your requirements I guess.

• About background tasks with task queue like Celery

This step is similar to the above one except it is using celery, as the task queue, to cater certain non naive background tasks. Although it is complex for setup this stuff, it will be the ultimate modern web solution so far I know.

[kalideir]

Thank you for the insights.
I have actually already implemented and utilized this great package in my work and I have gone very far with it.

The setup was very simple and the work os just as needed.