Helio Video based AI Workout Journal App
- Runs on an Iphone
- Runs on a FastAPI server
- Uses Pytorch slowfast model for action recognition
- Helio is an advanced video analysis application designed to leverage machine learning for real-time action recognition. Built with FastAPI and PyTorch Video, - Helio provides a robust platform for analyzing videos uploaded by users, identifying specific actions, and filtering results based on user-defined labels.
- Video Upload and Processing: Users can upload video files directly through a user-friendly interface. Helio processes these videos to detect and classify various actions.
- Action Recognition: Utilizing the SlowFast R50 model from PyTorch Video, Helio performs sophisticated video analysis to recognize and classify actions present in the video.
- Flexible Label Filtering: Users can specify a list of actions or labels to filter the results. This allows for targeted analysis and retrieval of relevant information.
- Timestamped Results: The app provides timestamped results, indicating when specific actions were detected within the video. This feature helps users pinpoint and review relevant sections of the video.
- Real-time Feedback: The app offers real-time feedback and results, ensuring users receive prompt and accurate analysis.
- FastAPI: A modern, fast (high-performance) web framework for building APIs with Python 3.6+ based on standard Python type hints.
- PyTorch Video: A library that provides video processing functionalities and pre-trained models for video action recognition. Torchvision: A library that includes popular pre-trained models and transforms for video data processing. How It Works
- Upload: Users upload a video file via the FastAPI endpoint.
- Process: The video is processed using PyTorch Video's SlowFast R50 model. This involves applying a series of transformations to prepare the video data for classification.
- Classify: The processed video is analyzed to detect and classify actions. Results are filtered based on user-defined labels.
- Respond: The app returns the classification results, including timestamps for when each action was detected in the video. Example Use Case
- A fitness coach uploads a video of a workout session and specifies labels such as "squat," "jumping jacks," and "archery." Helio analyzes the video, detects when each of these actions occurs, and provides a timestamped list of detected actions. This allows the coach to quickly review and analyze specific segments of the workout.
- Clone the Repository: Clone the Helio repository from GitHub.
- Install Dependencies: Install the required Python packages using pip install -r requirements.txt.
- Run the Application: Start the FastAPI server using uvicorn main:app --host 0.0.0.0 --port 8000.
- Upload and Analyze: Access the API endpoint to upload videos and receive analysis results.
- Future Improvements
- User Interface: Develop a frontend UI to enhance user experience and simplify video uploads.
- Performance Optimization: Implement additional optimizations for faster video processing and analysis.
- Extended Model Support: Integrate more pre-trained models to support a wider range of action categories.
the app is currently running on a google cloud instance at http://34.72.202.95:8000/docs
- create a conda env and then install the following
conda create -n helio python=3.11
conda activate helio
conda install mamba -c conda-forge
conda install pytorch==2.1.0 torchvision==0.16.0 -c pytorch
python3 -m pip install -r requirements.txt- start the app
python3 helio_app.py
- App is currently running at the following endpoint: http://34.72.202.95:8000/docs
- you can filter for certain actions, if you leave the labels empty, it will return all the actions it detects
- You can send a POST request with a video file to get responses for each 5 seconds of the video
- There is a mobile app which is connected to the endpoint and the relative code is in the mobile_app folder