Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for generating and downloading reports for DAGs #541

Open
pchtsp opened this issue Jun 7, 2024 · 1 comment
Open

Add support for generating and downloading reports for DAGs #541

pchtsp opened this issue Jun 7, 2024 · 1 comment
Labels
triage Issues waiting triage

Comments

@pchtsp
Copy link
Collaborator

pchtsp commented Jun 7, 2024

Is your feature request related to a problem? Please describe.
Building an adhoc web UI (a la cornflow-app) for each optimization problem is very hard and takes a lot of time. Also, once built it's not possible to share the charts/UI with people without access to the web UI.

Describe the solution you'd like
Quarto (https://quarto.org/) reports (*.qmd file) that get compiled with quarto on the server side (in airflow it would seem) into an html, a Word or a Latex pdf.

The file would be self-contained (or not) and be stored in a bucket and served from cornflow as pure html with a download button for a self-contained portable file.

Describe alternatives you've considered
Other report-generation apps such as RMarkdown.

Additional context
Some of the changes required would be [references are for PR 536]:

  • optional report directory in the dag directory with a collection of reports (cornflow-dags/DAG/tsp/report/report.qmd in PR https://github.com/baobabsoluciones/cornflow/pull/536/files).
  • a way to tell airflow to generate a report automatically after solve and store it somewhere (bucket). And store the report address in the execution table. Could be a solve config parameter that asks for a report name to be produced.
  • a way to ask cornflow via the REST API to return the report associated with an execution.
  • a way in cornflow-app to open and show an html/pdf report.

See the TSP example for more detail: #536

@pchtsp pchtsp added the triage Issues waiting triage label Jun 7, 2024
@pchtsp
Copy link
Collaborator Author

pchtsp commented Jun 11, 2024

I'm trying to look for the right place to tell Airflow to generate the report.
Options:

  1. cornflow_client.airflow.dag_utilitied.cf_solve
  2. cornflow_dags.DAG.activate_dags.create_dag

In the first one, we could add an additional piece of code that only runs if the dataset is solved, generates the report and then writes it somewhere.

In the second one, we could create a new airflow task/ operation (PythonOperator?) that reads the execution after the solving part is finished and generates the report as a separate airflow step.

I prefer the second option but I'm not sure how this was done in airflow. I will add a proposal to the TSP PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triage Issues waiting triage
Projects
None yet
Development

No branches or pull requests

1 participant