Skip to content

Data Platform for TechLabs to store & analyze Techie engagement

License

Notifications You must be signed in to change notification settings

TechLabs-Berlin/trm

Repository files navigation

Techie Relationship Management

The Techie Relationship Management (TRM) project at TechLabs provides a data platform to store & analyze all information we have about our Techies (participants). We want to know who they are and how they're doing throughout their journey to become a Digital Shaper.

This project aims to provide a standardized platform across TechLabs locations. It integrates with existing tools within the TechLabs organization, such as TypeForm, GSuite, and Slack. The TRM Dashboard is a web application for the local journey teams to view, edit, and analyze the data for their Techies to get actionable insights for improving the TechLabs experience.

Resources

Architecture

TRM project architecture

As for the main components, we have Hasura, the TRM Dashboard, and some serverless functions for integrating external providers.

  • The Hasura GraphQL engine provides a GraphQL API used by the TRM dashboard and provider integrations.
  • The TRM Dashboard is a React single-page web application (SPA) based on the react-admin framework. See /frontend/ for the dashboard source code.
  • The serverless functions in /functions/ integrate the TRM project with external providers, such as for authentification with Google, retrieving form responses from TypeForm, and fetching activity data from edyoucated & Slack.
  • PostgreSQL is used for data storage & retrieval

Development

We're happy that you want to work with us to make the TRM even more awesome! 🎉

Try to work your way to the documentation & setup steps outlined below. If you have any questions or are stuck with the setup, do not hestitate to call for help on Slack! 😌

Technical Onboarding

You should familiarize yourself with the architecture as described in the Architecture section. In particular, have a look what Hasura and react-admin are, and how they integrate with the GraphQL API.

This is a list of relevant resources like documentation, books, and conference talks:

  • Introduction to Hasura with React & Apollo

    The TRM uses this exact stack in combination with react-admin as a user interface & frontend data management framework.

  • Introduction to GraphQL with Hasura

    This introduction to GraphQL and how to use it with Hasura will familiarize yourself with the way React & Hasura play together behind the scenes. You will create a basic todo app and see how Hasura works with foreign key relationships, authorization, authentication and custom business logic.

    The TRM uses Google Auth (provided by the /functions/auth serverless function) instead of Auth0, but the JSON Web Token (JWT) issuance & verification is the same.

    We use remote schemas and event triggers to integrate with Google Sheets, Typeform, and edyoucated.

  • React-Admin 30-minute tutorial

    Marmelab's 30-minute react-admin tutorial will show you how react-admin works in combination with a simple placeholder data source. You'll see how to map API resources to the frontend, and how to customize the frontend.

    The TRM will use Apollo with the Hasura GraphQL API for data storage & retrieval, but working with a simple JSON API first may be an easier way with less moving parts to learn react-admin.

  • YouTube Guides

    If you're more into watching intro videos, have a look at these tutorials on YouTube:

If you feel like you have a general understanding how it works together, proceed to setup the development environment on your local machine as explained below.

Requirements

Must-Have:

May-Have:

  • NodeJS v13.9.0 or higher
  • Yarn v1.22.4 or higher
  • NPM v6.13.7 or higher

Local NodeJS, Yarn and NPM installations are required in case you want to develop on your local machine instead of in a container.

Setup

  1. Clone repository

  2. Build images for Hasura & functions

    docker-compose build --parallel
    
  3. Start Postgres, Hasura & functions:

    docker-compose up -d hasura
    
  4. Start the frontend development server:

    docker-compose up frontend
    

    After 30s or so it should say You can now view trm-frontend in the browser..

  5. Verify everything is running and healthy:

    docker-compose ps
    

    It should say Up (healthy) for every service.

  6. (Only required if developing locally) Setup frontend for development:

    • The following commands are assumed to be run in the /frontend directory (cd frontend)

    • yarn install downloads the dependencies

    • Run cp config.example.js config.js for an example config which is set up to work with the local database & Hasura instance you brought up with docker-compose.

Usage

  • Generate Sample Data

    • It is not much fun to work with an empty database. This is why there is an included data generator which you can use to populate the database with some near-realistic data. This generator is also used on the staging environment.

    • Generate some techies:

      1. Create a Semester in the frontend and copy its UUID (shown in the location bar like http://localhost:3000/#/semesters/UUID)

      2. Run the techie generator:

        docker-compose run --rm data-generator -- generate techies 100 --semester UUID --location LOCATION
        

        (location is BERLIN, for example)

  • Frontend Development

    • yarn start starts a development server on port 3000 with support for live code reloading.

      Run this in the /frontend directory (cd frontend).

      It should open the browser at http://localhost:3000 automatically.

  • Hasura Development

    • Run docker-compose exec hasura hasura-cli console --address 0.0.0.0 --no-browser and open http://localhost:9695 for the Hasura console.

    • You can find the Admin Secret in docker-compose.yml (hasura service, environment variables, HASURA_GRAPHQL_ADMIN_SECRET)

    • Authorization works with the Admin Secret, but you may want to use token authorization (e.g. for testing roles & permissions).

      Generate a token with docker-compose run --rm data-generator -- generate token --location BERLIN --team-member-id ID --functional-team JOURNEY --roles user,hr,journey.

      Then add the Authorization header in the Hasura GraphQL Explorer with Bearer TOKEN as its value (replace TOKEN with the output of the generate token command).

    • (Optional, for setup) Generate new JWT Secret Key: openssl rand 256 | base64

Deployment

The project is deployed on Google Cloud Platform (GCP). Follow the guide for setting up GCP.

The infrastructure (see /infrastructure/) is managed with Terraform in two projects:

The deployment process runs the GitHub Actions deploy workflow for every push to the deployment branches:

  • staging branch for the staging environment
  • production branch for the production environment

The TRM Dashboard is deployed on GitHub pages. GitHub Actions builds the frontend and commits the build in the following repositories:

Contributing

We appreciate any contribution and are happy to get in touch. If you see a problem, please open an issue in this repository. For write access to this repository, please send a message on our Slack Channel

To contribute code, we use a Pull Request workflow:

  1. Create a branch with your proposed changes and open a Pull Request
  2. We work with you for review and merge your code into the main branch.

Setup on Gitpod

Gitpod just recently announced they're now supporting Docker in their workspaces. The following guide will guide you through the setup process. It works quite well, though be aware they may be some quirks left.

Watch a Screencast of @Shark setting up the frontend on Gitpod.

Or follow these steps:

  1. Go to gitpod.io/settings, login and enable "Feature Preview". Choose "Code" as Default IDE.

  2. Go to https://gitpod.io/#https://github.com/TechLabs-Berlin/trm to create a new workspace for the TRM project.

  3. Download and install newest docker-compose (https://docs.docker.com/compose/install/) in the workspace.

  4. Run sudo docker-up. You need to leave the command running, thus open a new terminal by clicking on the plus icon.

  5. In docker-compose.yml, replace the keys under depends_on: with an array of the container names.

    Currently, the healthchecks don't work on Gitpod. This is why you need to tell docker-compose to bring up dependent services even though they are unhealthy. This step probably won't be necessary in the future.

  6. Follow normal Setup guide

    Ignore warnings about services being unhealthy.

  7. In /frontend, copy config.example.js to config.js

  8. Go to Remote Explorer Tab in the Sidebar

    1. For Port 8000: Allow public access, copy URL & set as authFnURL
    2. For Port 8080: Allow public access, copy URL & set as graphqlApiURL
  9. Open Port 3000 in the browser (click on the globe icon in the Remote Explorer Tab)

License

The TechLabs-Berlin/trm project is licensed MIT. Learn more about MIT license on choosealicense.com.

About

Data Platform for TechLabs to store & analyze Techie engagement

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published