First commit

.gitignore

@@ -0,0 +1,3 @@
+raw-data/*
+!raw-data/.gitkeep
+.DS_Store

README.md

@@ -0,0 +1,28 @@
+# Video Doodles: Hand-Drawn Animations on Videos with Scene-Aware Canvases
+
+This repository contains source code related to a research paper: [webpage](https://em-yu.github.io/research/videodoodles/), [paper](https://www-sop.inria.fr/reves/Basilic/2023/YBNWKB23/VideoDoodles.pdf), [ACM page](https://dl.acm.org/doi/abs/10.1145/3592413).
+
+The subfolders contain further instructions to run the code for [preprocessing](preprocess) and to set up the [web application](app).
+
+If this code is useful to your research, please consider citing the publication:
+
+```
+@article{videodoodles,
+  author = {Yu, Emilie and Blackburn-Matzen, Kevin and Nguyen, Cuong and Wang, Oliver and Habib Kazi, Rubaiat and Bousseau, Adrien},
+  title = {VideoDoodles: Hand-Drawn Animations on Videos with Scene-Aware Canvases},
+  year = {2023},
+  publisher = {Association for Computing Machinery},
+  doi = {10.1145/3592413},
+  journal = {ACM Trans. Graph.},
+  articleno = {54},
+  numpages = {12},
+}
+```
+
+## License
+
+TODO
+
+## Contact
+
+Emilie Yu: [email protected]

app/README.md

@@ -0,0 +1,92 @@
+# Video Doodles web application
+
+This folder contains the frontend and backend code to run the Video Doodles web application.
+
+* `backend`: a Python websocket server that awaits calls to solve for 3D trajectories. This folder contains all code pertaining to 3D tracking, both in reaction to frontend interactions as well as Python scripts useful to test or play with 3D tracking.
+* `frontend`: a web application based on [Svelte](https://svelte.dev/), served by a simple Python web server.
+
+## Data
+
+Download data for 16 videos:
+
+* Frontend data (1GB): [download here](https://repo-sam.inria.fr/d3/VideoDoodles/video-doodles-preprocessed-data-ui.zip)
+* Backend data (10GB): [download here](https://repo-sam.inria.fr/d3/VideoDoodles/video-doodles-preprocessed-data-backend.zip)
+
+Frontend data must be unzipped under `app/frontend/public/data` and backend data under `app/backend/data`.
+
+## Quick start with Docker
+
+We provide a simple way to run the whole application (locally or on a remote server) via Docker. After installing Docker, run:
+
+```bash
+cd app
+docker compose up --build -d
+```
+
+This will build two Docker images (`video-doodles-app-backend` and `video-doodles-app-ui`), based on the `Dockerfile` of each folder (flag `--build`, it can be ommitted if your images are already built). Each image will have all dependencies installed and the UI webpage content will be built via `npm` based on source code. Then Docker will start running one container for each image (in detached mode, flag `-d`), and launch the Python servers.
+
+At this point, if both containers were started successfully, you should be able to see the app at: http://localhost:80
+
+To see logs from the containers:
+
+```bash
+# This will list all running and exited containers, so you can use it to see if both containers have started fine
+docker ps -a
+# eg:
+CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS          PORTS                    NAMES
+06bf4ac29338   video-doodles-app-backend   "python3 -u app.py"      16 minutes ago   Up 16 minutes   0.0.0.0:8001->8001/tcp   video-doodles-app-backend-1
+848cf6c69995   video-doodles-app-ui        "tiny-http-server --…"   21 minutes ago   Up 21 minutes   0.0.0.0:80->8000/tcp     video-doodles-app-ui-1
+
+# Follow the logs of backend container (Ctrl+C to stop following):
+docker logs -f video-doodles-app-backend-1
+# Stream of logs...
+```
+
+A container that has exited means that an error arose. Checking the logs can help resolve that: `docker logs <container-name>`
+
+Stop containers when you are done:
+
+```bash
+# Stop containers
+docker stop video-doodles-app-backend-1
+docker stop video-doodles-app-ui-1
+# Remove them
+docker container prune
+```
+
+Please refer to `docker-compose.yml` and `Dockerfile` files for more details on the configuration.
+
+## How to use the web application
+
+Please watch the tutorial: https://youtu.be/hD5wZGVG6nQ
+
+We added a feature to export keyframes from a given canvas, in order to use them as offline tracking targets. The button `Export Keyframes` is available on the right-side bar when a canvas is selected in `Edit` mode.
+
+## Expected hardware for interactive rates
+
+The Python backend code needs to read heavy data files for tracking, our current unoptimized implementation will thus work best (ie, fast) with ample CPU memory available. We ran our user evaluations on a VM with 8GB RAM, and performance tests on a Macbook with 16GB RAM. Lower configs should still work, albeit quite slow. With Docker, one might run into lack of available RAM allocated to Docker, so we would encourage people who want to dig deeper to look into each of the individual folders' README to find instructions to run the code directly from their local environment.
+
+## More resources
+
+### Publication
+
+The VideoDoodles system and implementation is described in the associated publication: [webpage](https://em-yu.github.io/research/videodoodles/), [paper](https://www-sop.inria.fr/reves/Basilic/2023/YBNWKB23/VideoDoodles.pdf), [ACM page](https://dl.acm.org/doi/abs/10.1145/3592413).
+
+If this code is useful to your research, please consider citing the publication:
+
+```
+@article{videodoodles,
+  author = {Yu, Emilie and Blackburn-Matzen, Kevin and Nguyen, Cuong and Wang, Oliver and Habib Kazi, Rubaiat and Bousseau, Adrien},
+  title = {VideoDoodles: Hand-Drawn Animations on Videos with Scene-Aware Canvases},
+  year = {2023},
+  publisher = {Association for Computing Machinery},
+  doi = {10.1145/3592413},
+  journal = {ACM Trans. Graph.},
+  articleno = {54},
+  numpages = {12},
+}
+```
+
+### Contact
+
+Emilie Yu: [email protected]

app/backend/.dockerignore

@@ -0,0 +1,7 @@
+keyframe_records/*
+exports/*
+data/*
+**/__pycache__/
+data-new/*
+data-test/*
+.git/*

app/backend/.gitignore

@@ -0,0 +1,20 @@
+__pycache__/
+*.ini
+
+.DS_Store
+
+keyframe_records/*
+!keyframe_records/.gitkeep
+!keyframe_records/train_1kf.json
+!keyframe_records/train_orientations_1kf.json
+!keyframe_records/car-turn_1kf.json
+
+data/*
+!data/.gitkeep
+
+exports/old
+
+exports/trajectory/*
+!exports/trajectory/train_1kf.npz
+!exports/trajectory/train_orientations_1kf.npz
+!exports/trajectory/.gitkeep

app/backend/.gitlab-ci.yml

@@ -0,0 +1,26 @@
+# This file is a template, and might need editing before it works on your project.
+# To contribute improvements to CI/CD templates, please follow the Development guide at:
+# https://docs.gitlab.com/ee/development/cicd/templates.html
+# This specific template is located at:
+# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml
+
+# Build a Docker image with CI/CD and push to the GitLab registry.
+# Docker-in-Docker documentation: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
+#
+# This template uses one generic job with conditional builds
+# for the default branch and all other (MR) branches.
+
+docker-build:
+  # Use the official docker image.
+  image: docker:latest
+  stage: build
+  services:
+    - docker:dind
+  before_script:
+    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
+  script:
+    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
+    - docker push "$CI_REGISTRY_IMAGE"
+  tags:
+    - ci.inria.fr
+    - small

app/backend/Dockerfile

@@ -0,0 +1,14 @@
+
+FROM python:3.11
+
+WORKDIR /app
+
+RUN apt-get update
+RUN apt-get install ffmpeg libsm6 libxext6  -y
+
+COPY requirements.txt requirements.txt
+RUN pip3 install -r requirements.txt
+
+COPY . .
+
+CMD [ "python3", "-u", "app.py"]

app/backend/README.md

@@ -0,0 +1,86 @@
+# Video Doodles backend and tracking code
+
+## Dependencies and required input data
+
+- [x] **Install dependencies**
+
+```bash
+conda create -n video-doodles-backend python=3.10
+conda activate video-doodles-backend
+
+pip install -r requirements.txt
+```
+
+Note: `polyscope` and `opencv-python` are used only for debug visualisation purposes and can be omitted when deploying the backend to be used without visualisation.
+
+- [x] **Provide videos data**
+
+We provide data for 16 videos, please [download the archive](https://repo-sam.inria.fr/d3/VideoDoodles/video-doodles-preprocessed-data-backend.zip) and extract into `data` folder:
+
+```
+backend
+    data
+        blackswan
+        car-roundabout
+        ...
+        train
+```
+
+## Running the server
+
+To run locally:
+
+```bash
+cd app/backend
+python3 app.py
+# This exposes websocket ports at: ws://localhost:8001/
+```
+
+## Running tracking scripts
+
+The tracking scripts can be called in standalone mode, to facilitate testing or evaluation.
+
+Note that to visualize the results, we need to read in color frames and binary masks which you should download separately ([download link](https://repo-sam.inria.fr/d3/VideoDoodles/video-doodles-raw-data.zip), extract in `VideoDoodles/raw-data`). Alternatively, running with the flag `-H` skips visualization.
+
+```bash
+cd app/backend
+
+# Test the motion path solve ("Tracking 3D positions" in Sec. 5.2 of the paper)
+python3 -m scripts.test_find_motion_path --kf train_1kf
+
+# Test the 3D trajectory optimization ("Recovering stable, high-resolution trajectories" of Sec. 5.2.)
+python3 -m scripts.test_optimize_trajectory --kf train_1kf
+
+# Export the trajectory result so that it can be used for orientations estimation (-E flag)
+python3 -m scripts.test_optimize_trajectory --kf train_orientations_1kf -E
+
+# Test orientations optimization
+python3 -m scripts.test_orientations --traj train_orientations_1kf
+```
+
+The keyframe record files (eg `train_1kf`) can be created through the web UI by clicking the `Export Keyframes` button (available on the right-side bar when a canvas is selected in `Edit` mode).
+
+## More resources
+
+### Publication
+
+The VideoDoodles system and implementation is described in the associated publication: [webpage](https://em-yu.github.io/research/videodoodles/), [paper](https://www-sop.inria.fr/reves/Basilic/2023/YBNWKB23/VideoDoodles.pdf), [ACM page](https://dl.acm.org/doi/abs/10.1145/3592413).
+
+If this code is useful to your research, please consider citing the publication:
+
+```
+@article{videodoodles,
+  author = {Yu, Emilie and Blackburn-Matzen, Kevin and Nguyen, Cuong and Wang, Oliver and Habib Kazi, Rubaiat and Bousseau, Adrien},
+  title = {VideoDoodles: Hand-Drawn Animations on Videos with Scene-Aware Canvases},
+  year = {2023},
+  publisher = {Association for Computing Machinery},
+  doi = {10.1145/3592413},
+  journal = {ACM Trans. Graph.},
+  articleno = {54},
+  numpages = {12},
+}
+```
+
+### Contact
+
+Emilie Yu: [email protected]