diff --git a/docker/initialize_user.sh b/bin/initialize_user.sh similarity index 100% rename from docker/initialize_user.sh rename to bin/initialize_user.sh diff --git a/bin/setup.sh b/bin/setup.sh index e6a3317b3..0404e59a6 100755 --- a/bin/setup.sh +++ b/bin/setup.sh @@ -1,5 +1,8 @@ #!/usr/bin/env bash +# This is the legacy One-Click install script +# It probably doesn't work anymore, but is left here as reference + ##################################### ## Initialization & Helper Functions ##################################### diff --git a/config/environments/production.rb.template b/config/environments/production.rb.template index 835babd4f..d82a9ddf4 100755 --- a/config/environments/production.rb.template +++ b/config/environments/production.rb.template @@ -39,10 +39,7 @@ Autolab3::Application.configure do config.force_ssl = true config.middleware.use Rack::SslEnforcer, :except => [ /log_submit/, /local_submit/ ] end - - - - + # See everything in the log (default is :info) config.log_level = :info @@ -75,23 +72,28 @@ Autolab3::Application.configure do # with SQLite, MySQL, and PostgreSQL) # config.active_record.auto_explain_threshold_in_seconds = 0.5 - # Provide context to the email generator about the host - #config.action_mailer.default_url_options = {protocol: 'http', host: 'example.com' } - - config.action_mailer.delivery_method = :sendmail config.action_mailer.perform_deliveries = true config.action_mailer.raise_delivery_errors = true - # Use a custom smtp server, like Mandrill - #config.action_mailer.smtp_settings = { - # address: 'smtp.mandrillapp.com', - # port: 25, - # enable_starttls_auto: true, - # authentication: 'login', - # user_name: 'MANDRILL_USERNAME', - # password: 'MANDRILL_PASSWORD', - # domain: 'example.com', - #} + # Provide context to the email generator about the host + config.action_mailer.default_url_options = {protocol: 'http', host: 'yourhost.com' } + + # Use a custom smtp server, such as gmail, mailgun, sendgrid + config.action_mailer.smtp_settings = { + address: 'smtp.example.com', + port: 25, + enable_starttls_auto: true, + authentication: 'login', + user_name: 'example', + password: 'example', + domain: 'example.com', + } + + # default from + ActionMailer::Base.default :from => 'something@example.com' + + # option for using sendmail + # config.action_mailer.delivery_method = :sendmail config.middleware.use ExceptionNotification::Rack, email: { diff --git a/docs/api-interface.md b/docs/api-interface.md index 0e460eaa0..b75477aac 100644 --- a/docs/api-interface.md +++ b/docs/api-interface.md @@ -1,6 +1,6 @@ This page details all the endpoints of the Autolab REST API. -The client's access token should be included as a parameter to all endpoints. For details on obtaining access tokens, please see the [API Overview](/api-overview/) +The client's access token should be included as a parameter to all endpoints. For details on obtaining access tokens, please see the [API Overview](/api-overview/). ## Routing diff --git a/docs/api-overview.md b/docs/api-overview.md index 637da2c36..47af8e00e 100644 --- a/docs/api-overview.md +++ b/docs/api-overview.md @@ -18,7 +18,7 @@ V1 of the API allows clients to: All endpoints of the Autolab API requires client authentication in the form of an access token. To obtain this access token, clients must obtain authorization from the user. -Autolab API uses the standard [OAuth2](https://tools.ietf.org/html/rfc6749) [Authorization Code Grant](https://tools.ietf.org/html/rfc6749#section-4.1) for user authorization. For clients with no easy access to web browsers (e.g. console apps), an alternative [device flow](https://tools.ietf.org/html/draft-ietf-oauth-device-flow-07)-based authorization method is provided as well. +Autolab API uses the standard OAuth2 Authorization Code Grant for user authorization. For clients with no easy access to web browsers (e.g. console apps), an alternative device flow-based authorization method is provided as well. To understand how to authorize and unauthorize clients as a user, go to [Managing Authorized Apps](/api-managing-authorized-apps/) @@ -38,9 +38,9 @@ The endpoint for obtaining user authorization is The endpoint for obtaning access tokens and refresh tokens is `oauth/token` -[Section 4.1 of RFC 6749](https://tools.ietf.org/html/rfc6749#section-4.1) details the parameters required and the response clients can expect from these endpoints. +Section 4.1 of RFC 6749 details the parameters required and the response clients can expect from these endpoints. -Autolab API provides a refresh token with every new access token. Once the access token has expired, the client can use the refresh token to obtain a new access token, refresh token pair. Details are also provided in RFC 6749 [here](https://tools.ietf.org/html/rfc6749#section-6). +Autolab API provides a refresh token with every new access token. Once the access token has expired, the client can use the refresh token to obtain a new access token, refresh token pair. Details are also provided in RFC 6749 here. ### Device Flow diff --git a/docs/command-line-interface.md b/docs/command-line-interface.md index 02934cc2d..c5007bfa8 100644 --- a/docs/command-line-interface.md +++ b/docs/command-line-interface.md @@ -1,13 +1,13 @@ # Autolab Command Line Interface -To help showcase the capabilities of the API, we developed autolab-cli: A first-party command line client that serves as both a practical tool for users of Autolab, as well as a reference design for developers intending to use the API in their own programs. The cli includes features like downloading and submitting assignments from the terminal, viewing problems, and getting submission feedback. +To help showcase the capabilities of the [API](/api-overview), we developed autolab-cli: A first-party command line client that serves as both a practical tool for users of Autolab, as well as a reference design for developers intending to use the API in their own programs. The cli includes features like downloading and submitting assignments from the terminal, viewing problems, and getting submission feedback. !!! info "Note to CMU Students:" This cli binary has already been installed on the andrew machines as `autolab`. ## Obtaining authorization -Make sure you have the cli installed by running `autolab` in your terminal. If you see the usage instructions you're good to go. Otherwise, ask your school admin to install the cli from the [Autolab CLI Repository](https://github.com/autolab/autolab-cli). +Make sure you have the cli installed by running `autolab` in your terminal. If you see the usage instructions you're good to go. Otherwise, ask your school admin to install the cli from the Autolab CLI Repository. To setup autolab-cli with your Autolab account, run `autolab setup`. This will initiate a manual activation. @@ -96,4 +96,4 @@ $ autolab -h This will reveal other flags you may be able to use with each command. For example, you can call all of the context-dependent commands outside of an assessment directory by providing the `:` pair. -We hope this speeds up your workflow! If you find any problems, please file an issue on the [Autolab CLI Repository](https://github.com/autolab/autolab-cli). +We hope this speeds up your workflow! If you find any problems, please file an issue on the Autolab CLI Repository. diff --git a/docs/docker-compose.md b/docs/docker-compose.md deleted file mode 100644 index 61f168024..000000000 --- a/docs/docker-compose.md +++ /dev/null @@ -1,227 +0,0 @@ -# Autolab + Tango Docker Compose Installation - -The Autolab Docker Compose installation is a fast and easy production-ready installation and deployment method. It uses a MySQL database for the Autolab deployment, and comes with SSL support. This is now the preferred way of installing Autolab. - -If you are stuck or find issues with the installation process you can either file an issue on our Github repository, or join our Slack [here](https://autolab-slack.herokuapp.com/) and let us know and we will try our best to help. Also see the [debugging](#debugging) section for tips on how to diagnose problems. - -## Installation -First ensure that you have Docker and Docker Compose installed on your machine. See the official [Docker docs](See https://docs.docker.com/install/) for the installation steps. - -1. Clone this repository and its submodules: `git clone --recurse-submodules -j8 git://github.com/autolab/docker.git autolab-docker` -2. Enter the project directory: `cd autolab-docker` -3. Update submodules: `make update` -4. Create initial configs: `make` -5. Build the Dockerfiles: `docker-compose build` -6. Run the containers: `docker-compose up -d`. Note at this point Nginx will still be crash-looping in the Autolab container because SSL has not been configuired/disabled yet. -7. Ensure that the newly created config files have the right permissions: `make set-perms` -8. Perform migrations: `make db-migrate` -9. Create initial root user: `make create-user` -10. Change `DOCKER_TANGO_HOST_VOLUME_PATH` in `docker-compose.yml` to be the absolute path to the Tango `volumes` directory, i.e `//Tango/volumes` -11. Stop all containers: `docker-compose stop` -12. Modify the appropriate Nginx config (either `app.conf` or `no-ssl-app.conf` depending on whether you want to use SSL for the next step) by replacing the placeholders with your real domain -13. Continue with TLS setup as outlined in the next section -14. Build the autograding image(s) that you want to use in Tango (see [the docs](https://docs.autolabproject.com/tango-vmms/#docker-vmms-setup) for more information). For this setup we will stick to the default Ubuntu 18.04 autograding image: `docker build -t autograding_image Tango/vmms/`. Note that we can just run this directly on the host because we are mapping the Docker socket to the Tango container (i.e they are using the same Docker server). -15. Start up everything: `docker-compose up -d`. Autolab should now be accessible on port 80 (and 443 if you configured SSL)! - -## Configuring SSL/TLS -There are three options for TLS: using Let's Encrypt (for free TLS certificates), using your own certificate, and not using TLS (not recommended for production deployment). - -### Option 1: Let's Encrypt -1. Ensure that your DNS record points towards the IP address of your server -2. Ensure that port 443 is exposed on your server (i.e checking your firewall, AWS security group settings, etc) -3. Get initial SSL setup script: `make ssl` -4. In `ssl/init-letsencrypt.sh`, change `domains=(example.com)` to the list of domains that your host is associated with, and change `email` to be your email address so that Let's Encrypt will be able to email you when your certificate is about to expire -5. If necessary, change `staging=0` to `staging=1` to avoid being rate-limited by Let's Encrypt since there is a limit of 20 certificates/week. Setting this is helpful if you have an experimental setup. -6. Run your modified script: `sudo sh ./ssl/init-letsencrypt.sh` - -### Option 2: Using your own TLS certificate -1. Copy your private key to ./ssl/privkey.pem -2. Copy your certificate to ./ssl/fullchain.pem -3. Generate your dhparams, i.e `openssl dhparam -out ./ssl/ssl-dhparams.pem 4096` -4. Uncomment the following lines in `docker-compose.yml`: -``` - # - ./ssl/fullchain.pem:/etc/letsencrypt/live/test.autolab.io/fullchain.pem; - # - ./ssl/privkey.pem:/etc/letsencrypt/live/test.autolab.io/privkey.pem; - # - ./ssl/ssl-dhparams.pem:/etc/letsencrypt/ssl-dhparams.pem -``` - -### Option 3: No TLS (not recommended, only for local development/testing) -1. In `docker-compose.yml` (for all the subsequent steps), comment out the following: - -``` - # Comment the below out to disable SSL (not recommended) - - ./nginx/app.conf:/etc/nginx/sites-enabled/webapp.conf -``` - -Also uncomment the following: - -``` - # Uncomment the below to disable SSL (not recommended) - # - ./nginx/no-ssl-app.conf:/etc/nginx/sites-enabled/webapp.conf -``` - -Lastly set `DOCKER_SSL=false`: -``` - environment: - - DOCKER_SSL=true # set to false for no SSL (not recommended) -``` - -## Updating Autolab/Tango Deployment -1. Stop your running instances: `docker-compose stop` -2. Run `make update` to update your Autolab and Tango repositories -3. Run `docker-compose build` to rebuild the images with the latest code -4. Re-deploy your containers with `docker-compose up` - -## Debugging your Deployment -In the (very likely) event that you run into problems during setup, hopefully these steps will help you to help identify and diagnose the issue. If you continue to face difficulties or believe you discovered issues with the setup process please join our Slack [here](https://autolab-slack.herokuapp.com/) and let us know and we will try our best to help. - -### Better logging output for Docker Compose -By default, `docker-compose up -d` runs in detached state and it is not easy to immediately see errors: - -``` -$ docker-compose up -d -Starting certbot ... done -Starting redis ... done -Starting mysql ... done -Starting tango ... done -Recreating autolab ... done -``` - -Use `docker-compose up` instead to get output from all the containers in real time: - -``` -$ docker-compose up -Starting certbot ... done -Starting mysql ... done -Starting redis ... done -Starting tango ... done -Starting autolab ... done -Attaching to redis, mysql, certbot, tango, autolab -mysql | [Entrypoint] MySQL Docker Image 8.0.22-1.1.18 -tango | 2020-11-11 04:33:19,533 CRIT Supervisor running as root (no user in config file) -redis | 1:C 11 Nov 2020 04:33:19.032 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo -redis | 1:C 11 Nov 2020 04:33:19.032 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started -redis | 1:C 11 Nov 2020 04:33:19.032 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf -mysql | [Entrypoint] Starting MySQL 8.0.22-1.1.18 -redis | 1:M 11 Nov 2020 04:33:19.033 * Running mode=standalone, port=6379. -redis | 1:M 11 Nov 2020 04:33:19.033 # Server initialized -tango | 2020-11-11 04:33:19,539 INFO RPC interface 'supervisor' initialized -tango | 2020-11-11 04:33:19,539 CRIT Server 'unix_http_server' running without any HTTP authentication checking -mysql | 2020-11-11T04:33:19.476749Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.22) starting as process 22 ---- output truncated --- -``` - -### Checking Autolab logs -If the Autolab instance is not working properly, taking a look at both the application logs as well as the Nginx logs in the container will be helpful. - -First, find the name of the container. This should be just `autolab` by default: - -``` -$ docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -765d35962f52 autolab-docker_autolab "/sbin/my_init" 31 minutes ago Up 22 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp autolab -a5b77b5267b1 autolab-docker_tango "/usr/bin/supervisor…" 7 days ago Up 22 minutes 0.0.0.0:3000->3000/tcp tango -438d8e9f73e2 redis:latest "docker-entrypoint.s…" 7 days ago Up 22 minutes 6379/tcp redis -da86acc5a4c3 mysql/mysql-server:latest "/entrypoint.sh mysq…" 7 days ago Up 22 minutes (healthy) 3306/tcp, 33060-33061/tcp mysql -88032e85d669 a2eb12050715 "/bin/bash" 9 days ago Up 2 days compiler -``` - -Next get a shell inside the container: - -``` -$ docker exec -it autolab bash -root@be56be775428:/home/app/webapp# -``` - -By default we are in the project directory. Navigate to the `logs` directory and `cat` or `tail` `production.log`. This contains logs from the Autolab application itself. - -``` -root@be56be775428:/home/app/webapp# cd log -root@be56be775428:/home/app/webapp/log# tail -f -n +1 production.log -``` - -We can also check out our Nginx logs in `/var/log/nginx/`: - -``` -root@be56be775428:/home/app/webapp/log# cd /var/log/nginx/ -root@be56be775428:/var/log/nginx# ls -access.log error.log -``` - -### Accessing the Rails console -Obtain a shell in the `autolab` container as described [previously](#checking-autolab-logs), and do `RAILS_ENV=production bundle exec rails c`: - -``` -root@be56be775428:/home/app/webapp# RAILS_ENV=production bundle exec rails c -Loading production environment (Rails 5.2.0) -2.6.6 :001 > User.all.count - => 1 -``` - -In the example above, if you performed `make create-user` you should have at least one user in your database. If there are errors connecting to a database here it is likely that the database was misconfigured. - -### Checking Tango Logs -Get a shell in the Tango instance, similar to the instructions mentioned [previously](#checking-autolab-logs). The logs are stored in the parent folder (`/opt/TangoService`) of the project directory: - -``` -$ docker exec -it tango bash -root@a5b77b5267b1:/opt/TangoService/Tango# cd .. -root@a5b77b5267b1:/opt/TangoService# ls -Tango tango_job_manager_log.log tango_log.log -root@a5b77b5267b1:/opt/TangoService# tail -f -n +1 tango_job_manager_log.log tango_log.log -``` - -### Troubleshooting Autolab/Tango Connection -In the Autolab container, try to curl Tango: - -``` -root@be56be775428:/home/app/webapp# curl tango:3000 -Hello, world! RESTful Tango here! -``` - -In the Tango container, try to curl Autolab: - -``` -root@a5b77b5267b1:/opt/TangoService/Tango# curl autolab - -301 Moved Permanently - -

301 Moved Permanently

-
nginx/1.14.0 (Ubuntu)
- - -``` - -### Permission issues in Autolab -Run `make set-perms` again - -### Restarting Autolab Passenger Server -This is useful when you might want to test out some code change within the Autolab container without having to rebuild everything again. These changes can be applied by just restarting the Passenger service that is serving Autolab. - -Run `passenger-config restart-app`: - -``` -root@8b56488b3fb6:/home/app/webapp# passenger-config restart-app -Please select the application to restart. -Tip: re-run this command with --help to learn how to automate it. -If the menu doesn't display correctly, press '!' - - ‣ /home/app/webapp (production) - Cancel - -Restarting /home/app/webapp (production) -``` - -## FAQ -### error: unable to unlink old 'db/schema.rb': Permission denied - -If you obtain the following error when attempting to perform `make update`: - -``` -error: unable to unlink old 'db/schema.rb': Permission denied -fatal: Could not reset index file to revision 'HEAD'. -``` - -This is due to the fact that `db/schema.rb` is updated whenever migrations are performed. `db/schema.rb` documents the database schema, which depends on the database that you are using, its version, and when the migrations were run. It is likely that your `db/schema.rb` will diverge from the one generated by the devs. - -You can resolve this by changing the owner of the files to be your current user, and then running `make set-perms` afterwards when you start the containers again. diff --git a/docs/features/embedded-forms.md b/docs/features/embedded-forms.md index adc3b771f..374cbbe1e 100644 --- a/docs/features/embedded-forms.md +++ b/docs/features/embedded-forms.md @@ -3,7 +3,7 @@ This feature allows an instructor to create an assessment which does not require a file submission on the part of the student. Instead, when an assessment is created, the hand-in page for that assessment will display an HTML form of the instructor’s design. When the student submits the form, the information is sent directly in JSON format to the Tango grading server for evaluation. !!! attention "Tango Required" - Tango is needed to use this feature. Please install [Tango](/tango/) and connect it to Autolab before proceeding. + Tango is needed to use this feature. Please install [Tango](/installation/tango/) and connect it to Autolab before proceeding. ![Embedded Form](/images/embedded_form_example.png) @@ -103,7 +103,7 @@ Navigate to the Basic section of editing an assessment (`/courses//asses ### Grading an Embedded Form -When a student submits a form, the form data is sent to [Tango](/tango/) in the form of a JSON string in the file `out.txt.` In your grading script, parse the contents of `out.txt` as a JSON object. The JSON object will be a key-value pair data structure, so you can access the students response string (`value`) by its unique key (the `name` attribute). +When a student submits a form, the form data is sent to [Tango](/installation/tango/) in the form of a JSON string in the file `out.txt.` In your grading script, parse the contents of `out.txt` as a JSON object. The JSON object will be a key-value pair data structure, so you can access the students response string (`value`) by its unique key (the `name` attribute). For the example form shown above, the JSON object will be as follows: @@ -119,4 +119,4 @@ For the example form shown above, the JSON object will be as follows: } ``` -Use this information to do any processing you need in Tango.If you find any problems, please file an issue on the [Autolab Github](https://github.com/autolab/Autolab). +Use this information to do any processing you need in Tango.If you find any problems, please file an issue on the Autolab Github. diff --git a/docs/features/formatted-feedback.md b/docs/features/formatted-feedback.md index 7eef3def6..2c1a6598e 100644 --- a/docs/features/formatted-feedback.md +++ b/docs/features/formatted-feedback.md @@ -1,10 +1,10 @@ # Formatted Feedback -Autograding feedback plays an important role in a student's Autolab experience. Good feedback provided by autograders can really enhance a student's learning. As of Summer 2020, Autolab includes the formatted feedback feature by [Jala Alamin](https://github.com/alamijal). The feature was originally introduced in Washington State University Vancouver's version of Autolab. +Autograding feedback plays an important role in a student's Autolab experience. Good feedback provided by autograders can really enhance a student's learning. As of Summer 2020, Autolab includes the formatted feedback feature by Jala Alamin. The feature was originally introduced in Washington State University Vancouver's version of Autolab. Using formatted feedback requires a prior understanding of how Autolab's autograders work, as per the [Guide for Lab Authors](/lab/#writing-autograders). The formatted feedback feature is an **optional extension** of the default feedback. It comes in a staged fashion, allowing differing levels of adoption. -The next few sections are meant to be read in order, with each following section introducing a more complex usage of the formatted feedback feature than the previous. Experimenting the [hellocat](https://github.com/autolab/Autolab/tree/master/examples/hellocat) example code is another way to familiarize with the formatted feedback. +The next few sections are meant to be read in order, with each following section introducing a more complex usage of the formatted feedback feature than the previous. Experimenting with the hellocat example code is another way to familiarize with the formatted feedback. ### Default Feedback diff --git a/docs/features/metrics.md b/docs/features/metrics.md index ff5258361..f0bee8dbc 100644 --- a/docs/features/metrics.md +++ b/docs/features/metrics.md @@ -2,7 +2,7 @@ Traditional approaches of identifying students who are struggling with class is reactive; course staff wait for students to come to them to provide help, which can often be too late. -The metrics feature seeks to be an proactive approach by actively identifying students who might be struggling in class through tracking of metrics that signify possible risks. Identifying at-risk students early would provide a better chance of getting them back on track in the course. +The metrics feature seeks to be an proactive approach by actively identifying students who might be struggling in class through tracking of metrics that signify possible risks. Identifying students in need of attention early in the course would provide a better chance of getting them back on track in the course. ## Usage Flow @@ -10,20 +10,20 @@ The metrics feature seeks to be an proactive approach by actively identifying st We envision the feature to be used by the instructors in this order. They would: 1. Set up their course and assignments as per usual -1. Set up [Risk Metrics](#risk-metrics) at the start of the course -1. Be notified of new at risk students in their [Watchlist](#watchlist) on their course page +1. Set up [Student Metrics](#student-metrics) at the start of the course +1. Be notified of pending students in need of attention in their [Watchlist](#watchlist) on the course page 1. Visit the [Watchlist](#watchlist), contact students if necessary, using it as a work list at the same time -1. Refine the [Risk Metrics](#risk-metrics) as the course progresses +1. Refine the [Student Metrics](#student-metrics) as the course progresses -## Risk Metrics +## Student Metrics -![Metrics](/images/risk_metrics.png) +![Student Metrics](/images/student_metrics.png) -From our interviews with instructors, we understand that different courses have different measures of whether a student is at risk. As such, a set of risk conditions together will define the course's risk metrics. We intend to add more conditions to the risk metrics in the future. Feel free to suggest them via our [GitHub Issues](https://github.com/autolab/Autolab/issues) page. +From our interviews with instructors, we understand that different courses have different measures of whether a student is in need of attention. As such, a set of conditions together will define the course's student metrics. We intend to add more conditions to the metrics in the future. Feel free to suggest them via our GitHub Issues page. -### Risk Condition Rationale +### Student Metrics Condition Rationale -The conditions are designed to capture different characteristics of a possibly at risk student. In the sections that follow we attempt to explain the rationale behind each condition to aid with selecting the risk conditions. +The conditions are designed to capture different characteristics of a possible student in need of attention. In the sections that follow we attempt to explain the rationale behind each condition to aid with selecting the conditions. #### Students who have used *number* grace days by *date*. @@ -60,13 +60,13 @@ Identify weaker students. We expect this condition to be useful earlier in the c ![Watchlist](/images/watchlist.png) -Once instructors have set up risk metrics for their course, students that are identified as at-risk students based on these metrics will appear in the watchlist. +Once instructors have set up student metrics for their course, students that are identified as in need of attention based on these metrics will appear in the watchlist. ### Watchlist Instance ![Watchlist Instance](/images/watchlist_instance.png) -Every row in the watchlist represents a particular instance of a student who meets one or more of the risk conditions. A single student can appear in multiple watchlist instances if they are identified for new risk conditions on separate occassions of loading the watchlist. +Every row in the watchlist represents a particular instance of a student who meets one or more of the metrics conditions. A single student can appear in multiple watchlist instances if they are identified for new metrics conditions on separate occassions of loading the watchlist. For example, let's look at Jane Doe in the image above. Upon loading the watchlist, Jane appears in a watchlist instance for using `3 grace days` before the instructor-specified date and for having `2 low scores` below the instructor-specified threshold. If Jane later receives another score below the threshold, a new instance will appear for Jane when the instructor reloads the watchlist. Jane now appears twice in the watchlist, once in an instance with `3 grace days` and `2 low scores`, and once in an instance with `3 grace days` and `3 low scores`. @@ -78,11 +78,11 @@ An instructor can also hover over the condition tags to view the specific submis ### Tabs -There are four categories that watchlist instances can fall into: new, contacted, resolved, and archived. +There are four categories that watchlist instances can fall into: pending, contacted, resolved, and archived. -#### New +#### Pending -The `new` tab contains identified students who have not yet been contacted or resolved. The number of `new` instances will appear in a notification badge on the main course page, as shown below. +The `pending` tab contains identified students who have not yet been contacted or resolved. The number of `pending` instances will appear in a notification badge on the main course page, as shown below. ![Metrics Notification](/images/metrics_notification.png) @@ -96,4 +96,4 @@ The `resolved` tab contains all instances that the instructor has marked as reso #### Archived -When an instructor adjusts the risk metrics for a course, all instances that were in `contacted` or `resolved` for the outdated risk metrics are placed into `archived`. All `new` instances for the outdated risk metrics are dropped. As such, all instances in `new`, `contacted`, and `resolved` are consistent with the most up-to-date risk metrics. \ No newline at end of file +When an instructor adjusts the student metrics for a course, all instances that were in `contacted` or `resolved` for the outdated student metrics are placed into `archived`. All `pending` instances for the outdated student metrics are dropped. As such, all instances in `pending`, `contacted`, and `resolved` are consistent with the most up-to-date student metrics. \ No newline at end of file diff --git a/docs/features/moss.md b/docs/features/moss.md index 87d497d6a..a586e1dc3 100644 --- a/docs/features/moss.md +++ b/docs/features/moss.md @@ -1,8 +1,8 @@ # MOSS Plagiarism Detection Installation -[MOSS (Measure Of Software Similarity)](https://theory.stanford.edu/~aiken/moss/) is a system for checking for plagiarism. MOSS can be setup on Autolab as follows: +MOSS (Measure Of Software Similarity) is a system for checking for plagiarism. MOSS can be setup on Autolab as follows: -1. Obtain the script for MOSS based on the instructions given in [https://theory.stanford.edu/~aiken/moss/](https://theory.stanford.edu/~aiken/moss/). +1. Obtain the script for MOSS based on the instructions given here. 2. Create a directory called `vendor` at the root of your Autolab installation, i.e diff --git a/docs/images/autolab_system.png b/docs/images/autolab_system.png new file mode 100644 index 000000000..bfbbcf771 Binary files /dev/null and b/docs/images/autolab_system.png differ diff --git a/docs/images/metrics_notification.png b/docs/images/metrics_notification.png index 8a939d560..547d84b9e 100644 Binary files a/docs/images/metrics_notification.png and b/docs/images/metrics_notification.png differ diff --git a/docs/images/risk_metrics.png b/docs/images/risk_metrics.png deleted file mode 100644 index 4373f31f0..000000000 Binary files a/docs/images/risk_metrics.png and /dev/null differ diff --git a/docs/images/student_metrics.png b/docs/images/student_metrics.png new file mode 100644 index 000000000..176d0bda5 Binary files /dev/null and b/docs/images/student_metrics.png differ diff --git a/docs/images/watchlist.png b/docs/images/watchlist.png index 0d6912b16..0e7b43a5e 100644 Binary files a/docs/images/watchlist.png and b/docs/images/watchlist.png differ diff --git a/docs/images/watchlist_instance.png b/docs/images/watchlist_instance.png index 267b23b80..f012fbedc 100644 Binary files a/docs/images/watchlist_instance.png and b/docs/images/watchlist_instance.png differ diff --git a/docs/index.md b/docs/index.md index 4ad1fef9b..77229a3e3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,15 +1,33 @@ # Welcome to the Autolab Docs -Autolab is a course management platform that enables instructors to offer autograded programming assignments to their students. The two key ideas in Autolab are _autograding_ that is, programs evaluating other programs, and _scoreboards_ that display the latest autograded scores for each student. Autolab also provides gradebooks, rosters, handins/handouts, lab writeups, code annotation, manual grading, late penalties, grace days, cheat checking, meetings, partners, and bulk emails. +##Rationale -You can install Autolab easily by using our recommended [Docker Compose installation](/docker-compose) guide, or by following the steps from a manual install at [Getting Started](#getting-started). +Autolab is a course management platform that enables instructors to offer autograded programming assignments to their students. The two key ideas in Autolab are _autograding_–that is, programs evaluating other programs, and _scoreboards_ that display the latest autograded scores for each student. -For information on how to use Autolab for your course see the [Guide for Instructors](/instructors). To learn how to write an autograded lab see the [Guide for Lab Authors](/lab). +Autograding: The model for a traditional programming class is that students work on their code, hand it in once, and then get feedback a week or two later, at which point they've already moved on to the next assignment. Autograding, on the other hand, allows students to get immediate feedback on their performance and become more motivated to refine their coursework. -## Demonstration Site -If you would like to check out Autolab prior to installation, go over to our Demo Site! Login through `Developer Login` with the email: `admin@foo.bar`. +Scoreboard: The scoreboard is a fun and powerful motivation for students. When coupled with autograding, it creates a sense of community and healthy competition that benefits everyone. Students anonymize themselves on the scoreboard by giving themselves nicknames. A mix of curiosity and competitiveness drives the stronger students to be at the top of the scoreboard, and all students have a clear idea of what they need for full credit. -This is a demonstration website. It refreshes at 0,6,12,18 Hours (UTC) daily, and it is publicly accessible, so please only use it for your exploration. Do not use this site to store important information. +Autolab also provides gradebooks, rosters, handins/handouts, lab writeups, code annotation, manual grading, late penalties, grace days, cheat checking, meetings, partners, and bulk emails. + +For more of the rationale behind Autolab, please check out this blog post. + + +##Components + +Autolab consists of two services: (1) the Autolab frontend which is implemented using Ruby on Rails, and (2) Tango, the RESTful Python autograding server. Either service can run independently without the other. But in order to use all features of Autolab, we highly recommend installing both services. + +While the Autolab frontend supports Autolab's web application framework, the backend Tango is responsible for distributing and completing autograding jobs, which run in virtual machines or containers (we currently support Docker and AWS virtual machines). When Tango is done running a job, it then sends the autograded result back to the frontend. Below is a visualization of the typical workflow of the Autolab system. + +![Autolab System](/images/autolab_system.png) + +As you can see on the left, the Autolab frontend receives handin files from the clients through either traditional browser interaction or [the command line interface (CLI)](/command-line-interface). The frontend then sends the files through http requests to Tango. Tango adds them to a job queue, assigns them to available containers/virtual machines for grading through ssh, and shepherds the jobs through the process. On the right, inside the domain box for Docker at CMU, we show 3 VM pools - rhel, rhel122, and rhel411 - each with potentially different software packages. At the bottom right, we show an example of an AWS VM domain with VM pools rhelPKU and ubuntu. Tango assigns jobs only to the virtual machine instances from their corresponding course’s VM pool. For example, jobs with handin files for the course 122 would only go to rhel122’s instances. Once a job is done, the feedback is copied back to Tango through ssh and sent back to the Autolab frontend through http. The frontend then displays the feedback in the browser or through the CLI. It also updates the scoreboard if applicable and stores the feedback into the database, which is displayed at the bottom left. + +Apart from client usage, both the Autolab frontend and Tango provide application programming interfaces (API) for developers. The specific guides are included in the [Reference](/reference) section. + +##Demonstration Website +Installation instructions can be found in our comprehensive [installation guide](/installation/overview). However, if this is your first experience with Autolab, we encourage you to try out some key features on Autolab's Demo Site. You can login through `Developer Login` with the email: `admin@foo.bar`. The demonstration website refreshes at 0,6,12,18 Hours (UTC) daily, and it is publicly accessible, so please only use it for your exploration. Do not use this site to store important information. Try the following in order: @@ -54,405 +72,4 @@ Because it defaults to accepting `.c` files, we would like to change it to `*.pd 2. Look at your submission using the magnifying glass icon ### Grading submissions -Click on `Grade Submissions`, and then the arrow button to open up student submissions. For details on the relevant features for an Instructor, go to [Guide for Instructor](/instructors) - -## Getting Started - -Autolab consists of two services: (1) the Ruby on Rails frontend, and (2) [Tango](/tango), the RESTful Python autograding server. Either service can run independently without the other. But in order to use all features of Autolab, we highly recommend installing both services. - -Currently, we have support for installing Autolab on [Ubuntu 18.04+](#ubuntu-1804), and [Mac OSX](#mac-osx-1011). - -The following instructions are for manual installation of Autolab. We strongly encourage you to try out the [Docker Compose](/docker-compose) installation method which is much faster and easier. It is production ready and comes deployed with MySQL and TLS. - -### Mac OSX 10.11+ - -Follow the step-by-step instructions below: - -1. Install [rbenv](https://github.com/sstephenson/rbenv) (use the Basic GitHub Checkout method) - -2. Install [ruby-build](https://github.com/sstephenson/ruby-build) as an rbenv plugin: - - :::bash - git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build - - Restart your shell at this point in order to start using your newly installed rbenv - -3. Clone the Autolab repo into home directory and enter it: - - :::bash - cd ~/ - git clone https://github.com/autolab/Autolab.git && cd Autolab - -4. Install the correct version of ruby: - - :::bash - rbenv install $(cat .ruby-version) - - At this point, confirm that `rbenv` is working (you might need to restart your shell): - - :::bash - $ which ruby - ~/.rbenv/shims/ruby - - $ which rake - ~/.rbenv/shims/rake - Note that Mac OSX comes with its own installation of ruby. You might need to switch your ruby from - the system version to the rbenv installed version. One option is to add the following lines to ~/.bash_profile: - - :::bash - export RBENV_ROOT= - eval "$(rbenv init -)" - -5. Install `bundler`: - - :::bash - gem install bundler - rbenv rehash - -6. Install the required gems (run the following commands in the cloned Autolab repo): - - :::bash - cd bin - bundle install - - Refer to the [FAQ](#faq) for issues installing gems - -7. Install one of two database options - - - [SQLite](https://www.tutorialspoint.com/sqlite/sqlite_installation.htm) should **only** be used in development - - [MySQL](https://dev.mysql.com/doc/refman/5.7/en/osx-installation-pkg.html) can be used in development or production - -8. Configure your database: - - :::bash - cp config/database.yml.template config/database.yml - - Edit `database.yml` with the correct credentials for your chosen database. Refer to the [FAQ](#faq) for any issues. - -9. Configure school/organization specific information (new feature): - - :::bash - cp config/school.yml.template config/school.yml - - Edit `school.yml` with your school/organization specific names and emails - -10. Configure the Devise Auth System with a unique key (run these commands exactly - leave `` as it is): - - :::bash - cp config/initializers/devise.rb.template config/initializers/devise.rb - sed -i "s//`bundle exec rails secret`/g" config/initializers/devise.rb - - Fill in `` in the `config/initializers/devise.rb` file. To skip this step for now, fill with `foo.bar`. - -11. Create and initialize the database tables: - - :::bash - bundle exec rails db:create - bundle exec rails db:migrate - - Do not forget to use `bundle exec` in front of every rake/rails command. - -12. Populate dummy data (development only): - - :::bash - bundle exec rails autolab:populate - -13. Start the rails server: - - :::bash - bundle exec rails s -p 3000 - -14. Go to localhost:3000 and login with `Developer Login`: - - :::bash - Email: "admin@foo.bar". - -15. Install [Tango](/tango), the backend autograding service. - -16. Now you are all set to start using Autolab! Visit the [Guide for Instructors](/instructors) and [Guide for Lab Authors](/lab) pages for more info. - -### Ubuntu 18.04+ - -This set of instruction is meant to install of AutoLab v2.40 on Ubuntu 18.04 LTS. - -1. Upgrade system packages and installing prerequisites - - :::bash - sudo apt-get update - sudo apt-get upgrade - sudo apt-get install build-essential git libffi-dev zlib1g-dev autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev libncurses5-dev libgdbm5 libgdbm-dev libmysqlclient-dev libjansson-dev ctags - -2. Cloning Autolab repo from Github to ~/Autolab - - :::bash - cd ~/ - git clone https://github.com/autolab/Autolab.git - cd Autolab - -3. Setting up rbenv and ruby-build plugin - - :::bash - cd ~/ - git clone https://github.com/rbenv/rbenv.git ~/.rbenv - echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc - echo 'eval "$(rbenv init -)"' >> ~/.bashrc - source ~/.bashrc - - ~/.rbenv/bin/rbenv init - git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build - -4. Installing Ruby (Based on ruby version) - - :::bash - cd Autolab - rbenv install `cat .ruby-version` - -5. Installing SQLite - - :::bash - sudo apt-get install sqlite3 libsqlite3-dev - -6. Installing MySQL. (If you would just like to test Autolab, then you can skip this step by using SQLite) -Following instructions from [How to Install MySQL on Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-18-04). - - :::bash - sudo apt install mysql-server - sudo mysql_secure_installation - - > There will be a few questions asked during the MySQL setup. - - * Validate Password Plugin? N - * Remove Annonymous Users? Y - * Disallow Root Login Remotely? Y - * Remove Test Database and Access to it? Y - * Reload Privilege Tables Now? Y - -7. (If you are using MySQL) Create a new user with access to `autolab_test` and `autolab_development` databases. Because a password rather than auth_socket is needed, we need to ensure that user uses `mysql_native_password` - - :::bash - sudo mysql - mysql> CREATE USER 'user1'@'localhost' IDENTIFIED WITH mysql_native_password BY ''; - mysql> FLUSH PRIVILEGES; - mysql> exit; - -8. Installing Rails - - :::bash - cd Autolab - gem install bundler - rbenv rehash - bundle install - -9. Initializing Autolab Configs - - :::bash - cd Autolab - cp config/database.yml.template config/database.yml - cp config/school.yml.template config/school.yml - cp config/initializers/devise.rb.template config/initializers/devise.rb - sed -i "s//`bundle exec rails secret`/g" config/initializers/devise.rb - cp config/autogradeConfig.rb.template config/autogradeConfig.rb - -10. (Using MySQL) Editing Database YML. -Change the and fields in config/database.yml to the username and password that has been set up for the mysql. For example if your username is `user1`, and your password is `123456`, then your yml would be - - :::yml - development: - adapter: mysql2 - database: autolab_development - pool: 5 - username: user1 - password: '123456' - socket: /var/run/mysqld/mysqld.sock - host: localhost - variables: - sql_mode: NO_ENGINE_SUBSTITUTION - - test: - adapter: mysql2 - database: autolab_test - pool: 5 - username: user1 - password: '123456' - socket: /var/run/mysqld/mysqld.sock - host: localhost - variables: - sql_mode: NO_ENGINE_SUBSTITUTION - -11. (Using SQLite) Editing Database YML. -Comment out the configurations meant for MySQL in config/database.yml, and insert the following - - :::yml - development: - adapter: sqlite3 - database: db/autolab_development - pool: 5 - timeout: 5000 - - test: - adapter: sqlite3 - database: db/autolab_test - pool: 5 - timeout: 5000 - -12. Granting permissions on the databases. Setting global sql mode is important to relax the rules of mysql when it comes to group by mode - - :::bash - (access mysql using your root first to grant permissions) - mysql> grant all privileges on autolab_development.* to ''@localhost; - mysql> grant all privileges on autolab_test.* to ''@localhost; - mysql> SET GLOBAL sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY','')); - mysql> exit - -12. Initializing Autolab Database - - :::bash - cd Autolab - bundle exec rails db:create - bundle exec rails db:reset - bundle exec rails db:migrate - -13. Populating sample course & students - - :::bash - cd Autolab - bundle exec rails autolab:populate - -14. Run Autolab! - - :::bash - cd Autolab - bundle exec rails s -p 3000 --binding=0.0.0.0 - -15. Visit localhost:3000 on your browser to view your local deployment of Autolab, and login with `Developer Login` - - Email: "admin@foo.bar" - -16. Install [Tango](/tango), the backend autograding service. - -17. If you would like to deploy the server, you can try out [Phusion Passenger](https://www.phusionpassenger.com/library/walkthroughs/start/ruby.html) - -18. Now you are all set to start using Autolab! Visit the [Guide for Instructors](/instructors) and [Guide for Lab Authors](/lab) pages for more info. - - - -## FAQ - -This is a general list of questions that we get often. If you find a solution to an issue not mentioned here, -please contact us at - -#### Ubuntu Script Bugs - -If you get the following error - -```bash -Failed to fetch http://dl.google.com/linux/chrome/deb/dists/stable/Release -Unable to find expected entry 'main/binary-i386/Packages' in Release file (Wrong sources.list entry or malformed file) -``` - -then follow the solution in [this post](http://askubuntu.com/questions/743814/unable-to-find-expected-entry-main-binary-i386-packages-chrome). - -#### Where do I find the MySQL username and password? -If this is your first time logging into MySQL, your username is 'root'. You may also need to set the root password: - -Start the server: - -```bash -sudo /usr/local/mysql/support-files/mysql.server start -``` - -Set the password: - -```bash -mysqladmin -u root password "[New_Password]" -``` - -If you lost your root password, refer to the [MySQL wiki](http://dev.mysql.com/doc/refman/5.7/en/resetting-permissions.html) - -#### Bundle Install Errors -This happens as gems get updated. These fixes are gem-specific, but two common ones are - -`eventmachine` - -```bash -bundle config build.eventmachine --with-cppflags=-I/usr/local/opt/openssl/include -``` - -`libv8` - -```bash -bundle config build.libv8 --with-system-v8 -``` - -Run `bundle install` again - -If this does not work, another option would be - -```bash -bundle update libv8 -``` - -Because updating libv8 has dependency on other gems, it might fail due to a need to update other gems. Just do - -```bash -bundle update -``` - -according to the error messages until all gems are up to date. - -Run `bundle install` again - -If neither of these works, try exploring [this StackOverflow link](http://stackoverflow.com/questions/23536893/therubyracer-gemextbuilderror-error-failed-to-build-gem-native-extension) - -#### Can't connect to local MySQL server through socket -Make sure you've started the MySQL server and double-check the socket in `config/database.yml` - -The default socket location is `/tmp/mysql.sock`. - -#### I forgot my MySQL root password - -You can reset it following the instructions on [this Stack Overflow post](http://stackoverflow.com/questions/6474775/setting-the-mysql-root-user-password-on-os-x) - -If `mysql` complains that the password is expired, follow the instructions on the second answer on [this post](http://stackoverflow.com/questions/33326065/unable-to-access-mysql-after-it-automatically-generated-a-temporary-password) - -#### MySQL Syntax Error - -If you get the following error - -```bash -Mysql2::Error: You have an error in your SQL syntax -``` - -this may be an issue with using an incompatible version of MySQL. Try switching to MySQL 5.7 if you are currently using a different version. - -### Undefined method 'devise' for User -You most likely missed the step of copying 'config/initializers/devise.rb.template' to 'config/initializers/devise.rb' and setting your secret key in the setup instructions. \ No newline at end of file +Click on `Grade Submissions`, and then the arrow button to open up student submissions. For details on the relevant features for an Instructor, go to [Guide for Instructors](/instructors). diff --git a/docs/installation/docker-compose.md b/docs/installation/docker-compose.md new file mode 100644 index 000000000..f76d9cb61 --- /dev/null +++ b/docs/installation/docker-compose.md @@ -0,0 +1,301 @@ +# Autolab + Tango Docker Compose Installation + +The Autolab Docker Compose installation is a fast and easy production-ready installation and deployment method. It uses a MySQL database for the Autolab deployment, and comes with TLS/SSL support. This is now the preferred way of installing Autolab. + +If you are stuck or find issues with the installation process you can either file an issue on our Github repository, or join our Slack here and let us know and we will try our best to help. Also see the [debugging](#debugging-your-deployment) section for tips on how to diagnose problems and check out the [troubleshooting](#troubleshooting) section if you run into any issues. + +## Installation +First ensure that you have Docker and Docker Compose installed on your machine. See the official Docker docs for the installation steps. + +1. Clone this repository and its Autolab and Tango submodules: + + :::bash + git clone --recurse-submodules -j8 git://github.com/autolab/docker.git autolab-docker + + +2. Enter the project directory: + + :::bash + cd autolab-docker + +3. Update our Autolab and Tango submodules to ensure that you are getting the latest versions: + + :::bash + make update + +4. Create initial default configs: + + :::bash + make + +5. Build the Dockerfiles for both Autolab and Tango: + + :::bash + docker-compose build + +6. Run the Docker containers: + + :::bash + docker-compose up -d + + Note at this point Nginx will still be crash-looping in the Autolab container because TLS/SSL has not been configuired/disabled yet. + +7. Ensure that the newly created config files have the right permissions, as it may have been modified during the building process: + + :::bash + make set-perms + +8. Perform database migrations for Autolab, which will initialize your database schema: + + :::bash + make db-migrate + +9. Create administrative user for Autolab: + + :::bash + make create-user + + This user has full permissions on Autolab and will be able to create other users and designate other admins. + +10. Change `DOCKER_TANGO_HOST_VOLUME_PATH` in `docker-compose.yml` to be the absolute path to the Tango `volumes` directory, i.e `//Tango/volumes`. This is so that Tango knows where to put the output files of its autograded jobs. + + :::bash + # in docker-compose.yml + + # Modify the below to be the path to volumes on your host machine + - DOCKER_TANGO_HOST_VOLUME_PATH=/home/your-user/autolab-docker/Tango/volumes # example path + +11. Stop all containers, as we are going to setup/disable TLS: + + :::bash + docker-compose stop + +12. If you intend to use TLS later, in `nginx/app.conf`, change instances of `` to your real domain name. Otherwise, if you are not using TLS, in `nginx/no-ssl-app.conf`, change `server_name` to your real domain name. + +13. Continue with TLS setup as outlined in the [next section](#configuring-tlsssl) +14. Build the autograding image(s) that you want to use in Tango (see [the docs](/installation/tango/#docker-vmms-setup) for more information). For this setup we will stick to the default Ubuntu 18.04 autograding image: + + :::bash + docker build -t autograding_image Tango/vmms/ + +Note that we can just run this directly on the host because we are mapping the Docker socket to the Tango container (i.e they are using the same Docker server). + +15. Start up everything: + + :::bash + docker-compose up -d + +Autolab should now be accessible on port 80 (and 443 if you configured TLS)! + +## Configuring TLS/SSL +Having TLS/SSL configured is important as it helps to ensure that sensitive information like user credentials and submission information are encrypted instead of being sent over in plaintext across the network when users are using Autolab. We have made setting up TLS as easy and pain-free as possible. Using TLS is strongly recommended if you are using Autolab in a production environment with real students and instructors. + +There are three options for TLS: using Let's Encrypt (for free TLS certificates), using your own certificate, and not using TLS (suitable for local testing/development, but not recommended for production deployment). + +### Option 1: Let's Encrypt +1. Ensure that your DNS record points towards the IP address of your server +2. Ensure that port 443 is exposed on your server (i.e checking your firewall, AWS security group settings, etc) +3. Get initial SSL setup script: `make ssl` +4. In `ssl/init-letsencrypt.sh`, change `domains=(example.com)` to the list of domains that your host is associated with, and change `email` to be your email address so that Let's Encrypt will be able to email you when your certificate is about to expire +5. If necessary, change `staging=0` to `staging=1` to avoid being rate-limited by Let's Encrypt since there is a limit of 20 certificates/week. Setting this is helpful if you have an experimental setup. +6. Run your modified script: `sudo sh ./ssl/init-letsencrypt.sh` + +### Option 2: Using your own TLS certificate +1. Copy your private key to `ssl/privkey.pem` +2. Copy your certificate to `ssl/fullchain.pem` +3. Generate your dhparams: + + :::bash + openssl dhparam -out ssl/ssl-dhparams.pem 4096 + +4. Uncomment the following lines in `docker-compose.yml`: + + :::bash + # - ./ssl/fullchain.pem:/etc/letsencrypt/live/test.autolab.io/fullchain.pem; + # - ./ssl/privkey.pem:/etc/letsencrypt/live/test.autolab.io/privkey.pem; + # - ./ssl/ssl-dhparams.pem:/etc/letsencrypt/ssl-dhparams.pem + +### Option 3: No TLS (not recommended, only for local development/testing) +1. In `docker-compose.yml` (for all the subsequent steps), comment out the following: + + :::bash + # Comment the below out to disable SSL (not recommended) + - ./nginx/app.conf:/etc/nginx/sites-enabled/webapp.conf + + Also uncomment the following: + + :::bash + # Uncomment the below to disable SSL (not recommended) + # - ./nginx/no-ssl-app.conf:/etc/nginx/sites-enabled/webapp.conf + + Lastly set `DOCKER_SSL=false`: + + :::bash + environment: + - DOCKER_SSL=true # set to false for no SSL (not recommended) + +## Updating Your Docker Compose Deployment +1. Stop your running instances: + + :::bash + docker-compose stop + +2. Update your Autolab and Tango repositories: + + :::bash + make update + +3. Rebuild the images with the latest code: + + :::bash + docker-compose build + +4. Re-deploy your containers: + + :::bash + docker-compose up + +## Debugging your Deployment +In the (very likely) event that you run into problems during setup, hopefully these steps will help you to help identify and diagnose the issue. If you continue to face difficulties or believe you discovered issues with the setup process please join our Slack [here](https://autolab-slack.herokuapp.com/) and let us know and we will try our best to help. + +### Better logging output for Docker Compose +By default, `docker-compose up -d` runs in detached state and it is not easy to immediately see errors: + + :::bash + $ docker-compose up -d + Starting certbot ... done + Starting redis ... done + Starting mysql ... done + Starting tango ... done + Recreating autolab ... done + +Use `docker-compose up` instead to get output from all the containers in real time: + + :::bash + $ docker-compose up + Starting certbot ... done + Starting mysql ... done + Starting redis ... done + Starting tango ... done + Starting autolab ... done + Attaching to redis, mysql, certbot, tango, autolab + mysql | [Entrypoint] MySQL Docker Image 8.0.22-1.1.18 + tango | 2020-11-11 04:33:19,533 CRIT Supervisor running as root (no user in config file) + redis | 1:C 11 Nov 2020 04:33:19.032 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo + redis | 1:C 11 Nov 2020 04:33:19.032 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started + redis | 1:C 11 Nov 2020 04:33:19.032 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf + mysql | [Entrypoint] Starting MySQL 8.0.22-1.1.18 + redis | 1:M 11 Nov 2020 04:33:19.033 * Running mode=standalone, port=6379. + redis | 1:M 11 Nov 2020 04:33:19.033 # Server initialized + tango | 2020-11-11 04:33:19,539 INFO RPC interface 'supervisor' initialized + tango | 2020-11-11 04:33:19,539 CRIT Server 'unix_http_server' running without any HTTP authentication checking + mysql | 2020-11-11T04:33:19.476749Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.22) starting as process 22 + --- output truncated --- + +### Checking Autolab logs +If the Autolab instance is not working properly, taking a look at both the application logs as well as the Nginx logs in the container will be helpful. + +First, find the name of the container. This should be just `autolab` by default: + + :::bash + $ docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + 765d35962f52 autolab-docker_autolab "/sbin/my_init" 31 minutes ago Up 22 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp autolab + a5b77b5267b1 autolab-docker_tango "/usr/bin/supervisor…" 7 days ago Up 22 minutes 0.0.0.0:3000->3000/tcp tango + 438d8e9f73e2 redis:latest "docker-entrypoint.s…" 7 days ago Up 22 minutes 6379/tcp redis + da86acc5a4c3 mysql/mysql-server:latest "/entrypoint.sh mysq…" 7 days ago Up 22 minutes (healthy) 3306/tcp, 33060-33061/tcp mysql + 88032e85d669 a2eb12050715 "/bin/bash" 9 days ago Up 2 days compiler + +Next get a shell inside the container: + + :::bash + $ docker exec -it autolab bash + root@be56be775428:/home/app/webapp# + +By default we are in the project directory. Navigate to the `logs` directory and `cat` or `tail` `production.log`. This contains logs from the Autolab application itself. + + :::bash + root@be56be775428:/home/app/webapp# cd log + root@be56be775428:/home/app/webapp/log# tail -f -n +1 production.log + +We can also check out our Nginx logs in `/var/log/nginx/`: + + :::bash + root@be56be775428:/home/app/webapp/log# cd /var/log/nginx/ + root@be56be775428:/var/log/nginx# ls + access.log error.log + +### Accessing the Rails console +Obtain a shell in the `autolab` container as described [previously](#checking-autolab-logs), and do `RAILS_ENV=production bundle exec rails c`: + + :::bash + root@be56be775428:/home/app/webapp# RAILS_ENV=production bundle exec rails c + Loading production environment (Rails 5.2.0) + 2.6.6 :001 > User.all.count + => 1 + +In the example above, if you performed `make create-user` you should have at least one user in your database. If there are errors connecting to a database here it is likely that the database was misconfigured. + +### Checking Tango Logs +Get a shell in the Tango instance, similar to the instructions mentioned [previously](#checking-autolab-logs). The logs are stored in the parent folder (`/opt/TangoService`) of the project directory: + + :::bash + $ docker exec -it tango bash + root@a5b77b5267b1:/opt/TangoService/Tango# cd .. + root@a5b77b5267b1:/opt/TangoService# ls + Tango tango_job_manager_log.log tango_log.log + root@a5b77b5267b1:/opt/TangoService# tail -f -n +1 tango_job_manager_log.log tango_log.log + +### Troubleshooting Autolab/Tango Connection +In the Autolab container, try to curl Tango: + + :::bash + root@be56be775428:/home/app/webapp# curl tango:3000 + Hello, world! RESTful Tango here! + +In the Tango container, try to curl Autolab: + + :::bash + root@a5b77b5267b1:/opt/TangoService/Tango# curl autolab + + 301 Moved Permanently + +

301 Moved Permanently

+
nginx/1.14.0 (Ubuntu)
+ + + +### Permission issues in Autolab +Run the following again: + + :::bash + make set-perms + +### Restarting Autolab Passenger Server +This is useful when you might want to test out some code change within the Autolab container without having to rebuild everything again. These changes can be applied by just restarting the Passenger service that is serving Autolab. + +Run `passenger-config restart-app`: + + :::bash + root@8b56488b3fb6:/home/app/webapp# passenger-config restart-app + Please select the application to restart. + Tip: re-run this command with --help to learn how to automate it. + If the menu doesn't display correctly, press '!' + + ‣ /home/app/webapp (production) + Cancel + + Restarting /home/app/webapp (production) + +## Troubleshooting +### error: unable to unlink old 'db/schema.rb': Permission denied + +If you obtain the following error when attempting to perform `make update`: + + :::bash + error: unable to unlink old 'db/schema.rb': Permission denied + fatal: Could not reset index file to revision 'HEAD'. + +This is due to the fact that `db/schema.rb` is updated whenever migrations are performed. `db/schema.rb` documents the database schema, which depends on the database that you are using, its version, and when the migrations were run. It is likely that your `db/schema.rb` will diverge from the one generated by the devs. + +You can resolve this by changing the owner of the files to be your current user, and then running `make set-perms` afterwards when you start the containers again. diff --git a/docs/installation/mailing.md b/docs/installation/mailing.md new file mode 100644 index 000000000..9d273f327 --- /dev/null +++ b/docs/installation/mailing.md @@ -0,0 +1,36 @@ +# Mailing Setup + +Autolab requires mailing to allow users to register accounts and reset passwords. You will also be able to make announcements through Autolab as well. The recommended approach is to setup Autolab to use a SMTP server, such as [mailgun](https://mailgun.com), [SendGrid](https://sendgrid.com), [Amazon SES](https://aws.amazon.com/ses/) or any other valid SMTP mail servers to send out email. + +We intend this instructions mainly for production usage. To set Autolab up to use a custom SMTP Server, edit the following in `production.rb` that you have created. (If you would like to test it in development, add the following settings into `development.rb`). Both `production.rb` and `development.rb` are located at `config/environments` + +1. Update the host domain of your Autolab frontend instance + + :::ruby + config.action_mailer.default_url_options = {protocol: 'http', host: 'yourhost.com' } + + Host here should be the domain in which Autolab is hosted on. (e.g. `autolab.andrew.cmu.edu`) + +2. Update the custom smtp server settings + + :::ruby + config.action_mailer.smtp_settings = { + address: 'smtp.example.com', + port: 25, + enable_starttls_auto: true, + authentication: 'login', + user_name: 'example', + password: 'example', + domain: 'example.com', + } + + Refer to the SMTP settings instructions that your selected service provides you such as [SendGrid SMTP for Ruby on Rails](https://sendgrid.com/docs/for-developers/sending-email/rubyonrails/), [Amazon SES](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp.html). + +3. Update the "from" setting + + :::ruby + ActionMailer::Base.default :from => 'something@example.com' + + Here the from address **must** be a address that your SMTP service permits you to send from. Oftentimes it is the same as your user_name in the smtp settings. + +Make sure to restart your Autolab client to see the changes \ No newline at end of file diff --git a/docs/installation/manual.md b/docs/installation/manual.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/one-click.md b/docs/installation/one-click.md similarity index 100% rename from docs/one-click.md rename to docs/installation/one-click.md diff --git a/docs/installation/osx.md b/docs/installation/osx.md new file mode 100644 index 000000000..496267072 --- /dev/null +++ b/docs/installation/osx.md @@ -0,0 +1,111 @@ +This page provides instructions on installing Autolab for development on Mac OSX 10.11+. If you encounter any issue along the way, check out [Troubleshooting](/installation/troubleshoot). + +Follow the step-by-step instructions below: + +1. Install rbenv (use the Basic GitHub Checkout method) + +2. Install ruby-build as an rbenv plugin: + + :::bash + git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build + + Restart your shell at this point in order to start using your newly installed rbenv + +3. Clone the Autolab repo into home directory and enter it: + + :::bash + cd ~/ + git clone https://github.com/autolab/Autolab.git && cd Autolab + +4. Install the correct version of ruby: + + :::bash + rbenv install $(cat .ruby-version) + + At this point, confirm that `rbenv` is working (you might need to restart your shell): + + :::bash + $ which ruby + ~/.rbenv/shims/ruby + + $ which rake + ~/.rbenv/shims/rake + Note that Mac OSX comes with its own installation of ruby. You might need to switch your ruby from + the system version to the rbenv installed version. One option is to add the following lines to ~/.bash_profile: + + :::bash + export RBENV_ROOT= + eval "$(rbenv init -)" + +5. Install `bundler`: + + :::bash + gem install bundler + rbenv rehash + +6. Install the required gems (run the following commands in the cloned Autolab repo): + + :::bash + cd bin + bundle install + + Refer to [Troubleshooting](/installation/troubleshoot) for issues installing gems + +7. Install one of two database options + + - SQLite should **only** be used in development + - MySQL can be used in development or production + +8. Configure your database: + + :::bash + cp config/database.yml.template config/database.yml + + Edit `database.yml` with the correct credentials for your chosen database. Refer to [Troubleshooting](/installation/troubleshoot) for any issues. + +9. Configure school/organization specific information (new feature): + + :::bash + cp config/school.yml.template config/school.yml + + Edit `school.yml` with your school/organization specific names and emails + +10. Configure the Devise Auth System with a unique key (run these commands exactly - leave `` as it is): + + :::bash + cp config/initializers/devise.rb.template config/initializers/devise.rb + sed -i "s//`bundle exec rails secret`/g" config/initializers/devise.rb + + Fill in `` in the `config/initializers/devise.rb` file. To skip this step for now, fill with `foo.bar`. + +11. Create and initialize the database tables: + + :::bash + bundle exec rails db:create + bundle exec rails db:migrate + + Do not forget to use `bundle exec` in front of every rake/rails command. + +12. Create initial root user + + :::bash + ./bin/initialize_user.sh + +13. Populate dummy data (for development only): + + :::bash + bundle exec rails autolab:populate + +14. Start the rails server: + + :::bash + bundle exec rails s -p 3000 + +15. Go to localhost:3000 and login with either the credentials of the root user you just created, or choose `Developer Login` with: + + :::bash + Email: "admin@foo.bar". + +16. Install [Tango](/installation/tango), the backend autograding service. + +17. Now you are all set to start using Autolab! Visit the [Guide for Instructors](/instructors) and [Guide for Lab Authors](/lab) pages for more info. \ No newline at end of file diff --git a/docs/installation/overview.md b/docs/installation/overview.md new file mode 100644 index 000000000..ca91b4308 --- /dev/null +++ b/docs/installation/overview.md @@ -0,0 +1,9 @@ +The installation process comprises installing both Autolab and Tango (the autograding backend). MySQL and Redis will also need to be installed in most use cases. + +There are 2 different ways to install Autolab and Tango: + + 1. The simplest and fastest way to get up and running is to use our [Docker Compose installation](/installation/docker-compose/), which is ideal for most workloads. Both production-ready and testing deployments are available. This will set-up containers for Autolab, Tango, and other required services. + 2. You can also install Autolab manually. There are instructions for installing Autolab on [Ubuntu 18.04](/installation/ubuntu) and on [Mac OSX 10.11+](/installation/osx). The instructions for [installing Tango manually](/installation/tango) are the same for both environments. + + +Most of our users prefer the Docker Compose installation method as it is simpler, production-ready, and comes deployed with MySQL and TLS/SSL. \ No newline at end of file diff --git a/docs/installation/tango.md b/docs/installation/tango.md new file mode 100644 index 000000000..42abca2a1 --- /dev/null +++ b/docs/installation/tango.md @@ -0,0 +1,329 @@ +# Tango Installation + +This guide provides instructions for installing Tango on either a [development environment](#development-installation) or a [production environment](#production-installation). + +# Development Installation + +This guide shows how to setup Tango in a **development environment**. Use the [production installation](#production-installation) guide for installing in a **production environment**. + +1. Obtain the source code. + + :::bash + git clone https://github.com/autolab/Tango.git; cd Tango + +2. Install Redis following this guide. By default, Tango uses Redis as a stateless job queue. Learn more here. + +3. Create a `config.py` file from the given template. + + :::bash + cp config.template.py config.py + +4. Create the course labs directory where job's output files will go, organized by key and lab name: + + :::bash + mkdir courselabs + + By default the `COURSELABS` option in `config.py` points to the `courselabs` directory in the Tango directory. + Change this to specify another path if you wish. + +5. Set up a VMMS for Tango to use. + + - [Docker](#docker-vmms-setup) (**recommended**) + - [Amazon EC2](#amazon-ec2-vmms-setup) + - TashiVMMS (deprecated) + + +6. Run the following commands to setup the Tango dev environment inside the Tango directory. Install pip if needed. + + :::bash + $ pip install virtualenv + $ virtualenv . + $ source bin/activate + $ pip install -r requirements.txt + $ mkdir volumes + +7. If you are using Docker, set `DOCKER_VOLUME_PATH` in `config.py` to be the path to the `volumes` directory you just created. + + :::bash + DOCKER_VOLUME_PATH = "/path/to/Tango/volumes/" + + +8. Start Redis by running the following command: + + :::bash + $ redis-server + +9. Run the following command to start the server (producer). If no port is given, the server will run on the port specified in `config.py` (default: 3000): + + :::bash + python restful-tango/server.py + + Open another terminal window and start the job manager (consumer): + + :::bash + python jobManager.py + + For more information on the job producer/consumer model check out our blog post. + +10. Ensure Tango is running: + + :::bash + $ curl localhost: + # Hello, world! RESTful Tango here! + +11. You can test the Tango setup using the [command line client](/tango-cli/). + +12. If you are using Tango with Autolab, you have to configure Autolab to use Tango. Go to your Autolab directory and enter the following commands: + + :::bash + cp config/autogradeConfig.rb.template config/autogradeConfig.rb + + Fill in the correct info for your Tango deployment, mainly the following: + + :::ruby + # Hostname for Tango RESTful API + RESTFUL_HOST = "foo.bar.edu" #(if you are running Tango locally, then it is just "localhost") + + # Port for Tango RESTful API + RESTFUL_PORT = "3000" + + # Key for Tango RESTful API + RESTFUL_KEY = "test" + +13. See below for instructions on how to deploy Tango in a standalone production environment. + + + + +# Production Installation + +This is a guide to setup a fully self-sufficient Tango deployment environment out-of-the-box using Docker. The suggested deployment pattern for Tango uses Nginx as a proxy and Supervisor as a process manager for Tango and all its dependencies. All requests to Nginx are rerouted to a Tango process. + +## Details + +- Nginx default port - 8600 +- Tango ports - 8610, 8611 +- Redis port - 6379 +- You can change any of these in the respective config files in `deployment/config/` before you build the `tango_deployment` image. + +## Steps + +1. Clone the Tango repo + + :::sh + $ git clone https://github.com/autolab/Tango.git; cd Tango + +2. Create a `config.py` file from the given template. + + :::sh + $ cp config.template.py config.py + +3. Modify `DOCKER_VOLUME_PATH` in `config.py` as follows: + + :::sh + DOCKER_VOLUME_PATH = '/opt/TangoService/Tango/volumes/' + +4. Install docker on the host machine by following instructions on the docker installation page. Ensure docker is running: + + :::sh + $ docker ps + # CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + +5. Run the following command to build the Tango deployment image. + + :::sh + $ docker build --tag="tango_deployment" . + +6. Ensure the image was built by running. + + :::sh + $ docker images + # REPOSITORY TAG IMAGE ID CREATED SIZE + # tango_deployment latest 3c0d4f4b4958 2 minutes ago 742.6 MB + # ubuntu 15.04 d1b55fd07600 4 minutes ago 131.3 MB + +7. Run the following command to access the image in a container with a bash shell. The `-p` flag will map `nginxPort` on the docker container to `localPort` (8610 recommended) on your local machine (or on the VM that docker is running in on the local machine) so that Tango is accessible from outside the docker container. + + :::sh + $ docker run --privileged -p : -it tango_deployment /bin/bash + +8. Set up a VMMS for Tango within the Docker container. + + - [Docker](#docker-vmms-setup) (**recommended**) + - [Amazon EC2](#amazon-ec2-vmms-setup) + +9. Run the following command to start supervisor, which will then start Tango and all its dependencies. + + :::sh + $ service supervisor start + +10. Check to see if Tango is responding to requests + + :::sh + $ curl localhost:8610 # Hello, world! RESTful Tango here! + +11. Once you have a VMMS set up, leave the tango_deployment container by typing `exit` and once back in the host shell run the following command to get the name of your production container. + + :::sh + $ docker ps -as + # CONTAINER ID IMAGE COMMAND NAMES SIZE + # c704d45c3737 tango_deployment "/bin/bash" erwin 40.26 MB + + The container created in this example has the name `erwin`. + +12. The name of the production container can be changed by running the following command and will be used to run the container and create services. + + :::sh + $ docker rename + +13. To reopen the container once it has been built use the following command. This will reopen the interactive shell within the container and allow for configuration of the container after its initial run. + + :::sh + $ docker start erwin + $ docker attach erwin + +14. Once the container is set up with the autograding image, and the VMMS configured with any necessary software/environments needed for autograding (java, perl, etc), some configurations need to be changed to make the container daemon ready. Using the `CONTAINER ID` above, use the following commands to modify that containers `config.v2.json` file. + + :::sh + $ sudo ls /var/lib/docker/containers + c704d45c37372a034cb97761d99f6f3f362707cc23d689734895e017eda3e55b + $ sudo vim /var/lib/docker/containers/c704d45c37372a034cb97761d99f6f3f362707cc23d689734895e017eda3e55b/config.v2.json + +15. Edit the "Path" field in the config.v2.json file from "/bin/bash" to "/usr/bin/supervisord" and save the file. Run the following commands to verify the changes were successful. The COMMAND field should now be "/usr/bin/supervisord" + + :::sh + $ service docker restart + $ docker ps -as + # CONTAINER ID IMAGE COMMAND NAMES SIZE + # c704d45c3737 tango_deployment "/usr/bin/supervisord" erwin 40.26 MB + +16. At this point when the container is started, the environment is fully set up and will no longer be an interactive shell. Instead, it will be the supervisor service that starts Tango and all its dependencies. Test this with the following commands and ensure Tango is functioning properly. + + :::sh + $ docker start erwin + # (Test tango environment) + $ docker stop erwin + +17. Test the setup by running sample jobs using [the testing guide](/tango-cli/). + + **The following steps are optional and should only be used if you would like the Tango container to start on system boot.** + +18. To ensure Tango starts with the system in the production environment, the container needs to be configured as a service. Below is a sample service config file that needs to be changed to suit your environment and placed in `/etc/systemd/system/`. The file should be named `.service`. For this example, it is `erwin.service`. + + :::sh + [Unit] + Description=Docker Service Managing Tango Container + Requires=docker.service + After=docker.service + + [Service] + Restart=always + ExecStart=/usr/bin/docker start -a erwin + ExecStop=/usr/bin/docker stop -t 2 erwin + + [Install] + WantedBy=default.target + +19. Test and ensure the service was set up correctly. The service should start successfully and remain running. + + :::sh + $ systemctl daemon-reload + $ service erwin start + $ service erwin status + +20. Enable the service at system startup and reboot and ensure it starts with the host. + + :::sh + $ systemctl enable erwin.service + $ sudo reboot + # (Server Reboots) + $ service erwin status + +# Docker VMMS Setup + +This is a guide to set up Tango to run jobs inside Docker containers. + +1. Install docker on host machine by following instructions on the docker installation page. Ensure docker is running: + + :::bash + $ docker ps # CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + +2. Build base Docker image from root Tango directory. + + :::sh + cd path/to/Tango + docker build -t autograding_image vmms/ + docker images autograding_image # Check if image built + +3. Update `VMMS_NAME` in `config.py`. + + :::python + # in config.py + VMMS_NAME = "localDocker" + +# Amazon EC2 VMMS Setup + +This is a guide to set up Tango to run jobs on an Amazon EC2 VM. + +1. Create an AWS Account or use an existing one. + +2. Obtain your `access_key_id` and `secret_access_key` by following the instructions here. + +3. Add AWS Credentials to a file called `~/.boto` using the following format: + + :::bash + [Credentials] + aws_access_key_id = MYAMAZONTESTKEY12345 + aws_secret_access_key = myawssecretaccesskey12345 + + Tango uses the Boto Python package to interface with Amazon Web Services + +4. In the AWS EC2 console, create an Ubuntu 14.04+ EC2 instance and save the `.pem` file in a safe location. + +5. Copy the directory and contents of `autodriver/` in the Tango repo into the EC2 VM. For more help connecting to the EC2 instance follow this guide. + + :::bash + chmod 400 /path/my-key-pair.pem + scp -i /path/my-key-pair.pem -r autodriver/ ubuntu@.compute-1.amazonaws.com:~/ + + The autodriver is used as a sandbox environment to run the job inside the VM. It limits Disk I/O, Disk Usage, monitors security, and controls other valuable `sudo` level resources. + +6. In the EC2 VM, compile the autodriver. + + :::bash + $ cd autodriver/ + $ make clean; make + $ cp -p autodriver /usr/bin/autodriver + +7. Create the `autograde` Linux user and directory. All jobs will be run under this user. + + :::bash + $ useradd autograde + $ mkdir autograde + $ chown autograde autograde + $ chown :autograde autograde + +8. In the AWS EC2 console, create an AMI image from your EC2 VM. Use this guide to create a custom AMI. + +9. Exit the EC2 instance and edit the following values in `config.py` in the Tango directory. + + :::bash + # VMMS to use. Must be set to a VMMS implemented in vmms/ before + # starting Tango. Options are: "localDocker", "distDocker", + # "tashiSSH", and "ec2SSH" + VMMS_NAME = "ec2SSH" + ###### + # Part 5: EC2 Constants + # + EC2_REGION = 'us-east-1' # EC2 Region + EC2_USER_NAME = 'ubuntu' # EC2 username + DEFAULT_AMI = 'ami-4c99c35b' # Custom AMI Id + DEFAULT_INST_TYPE = 't2.micro' # Instance Type + DEFAULT_SECURITY_GROUP = 'autolab-autograde-ec2' # Security Group with full access to EC2 + SECURITY_KEY_PATH = '/path/to/my-key-pair.pem' # Absolute path to my-key-pair.pem + DYNAMIC_SECURITY_KEY_PATH = '' # Leave blank + SECURITY_KEY_NAME = 'my-key-pair' # Name of the key file. Ex: if file name is 'my-key-pair.pem', fill value with 'my-key-pair' + TANGO_RESERVATION_ID = '1' # Leave as 1 + INSTANCE_RUNNING = 16 # Status code of a running instance, leave as 16 + +10. You should now be ready to run Tango jobs on EC2! Use the [Tango CLI](/tango-cli/) to test your setup. diff --git a/docs/installation/troubleshoot.md b/docs/installation/troubleshoot.md new file mode 100644 index 000000000..ac302a798 --- /dev/null +++ b/docs/installation/troubleshoot.md @@ -0,0 +1,89 @@ +This is a general list of questions that we get often. If you find a solution to an issue not mentioned here, +please contact us at + +#### Ubuntu Script Bugs + +If you get the following error + +```bash +Failed to fetch http://dl.google.com/linux/chrome/deb/dists/stable/Release +Unable to find expected entry 'main/binary-i386/Packages' in Release file (Wrong sources.list entry or malformed file) +``` + +then follow the solution in [this post](http://askubuntu.com/questions/743814/unable-to-find-expected-entry-main-binary-i386-packages-chrome). + +#### Where do I find the MySQL username and password? +If this is your first time logging into MySQL, your username is 'root'. You may also need to set the root password: + +Start the server: + +```bash +sudo /usr/local/mysql/support-files/mysql.server start +``` + +Set the password: + +```bash +mysqladmin -u root password "[New_Password]" +``` + +If you lost your root password, refer to the [MySQL wiki](http://dev.mysql.com/doc/refman/5.7/en/resetting-permissions.html) + +#### Bundle Install Errors +This happens as gems get updated. These fixes are gem-specific, but two common ones are + +`eventmachine` + +```bash +bundle config build.eventmachine --with-cppflags=-I/usr/local/opt/openssl/include +``` + +`libv8` + +```bash +bundle config build.libv8 --with-system-v8 +``` + +Run `bundle install` again + +If this does not work, another option would be + +```bash +bundle update libv8 +``` + +Because updating libv8 has dependency on other gems, it might fail due to a need to update other gems. Just do + +```bash +bundle update +``` + +according to the error messages until all gems are up to date. + +Run `bundle install` again + +If neither of these works, try exploring [this StackOverflow link](http://stackoverflow.com/questions/23536893/therubyracer-gemextbuilderror-error-failed-to-build-gem-native-extension) + +#### Can't connect to local MySQL server through socket +Make sure you've started the MySQL server and double-check the socket in `config/database.yml` + +The default socket location is `/tmp/mysql.sock`. + +#### I forgot my MySQL root password + +You can reset it following the instructions on [this Stack Overflow post](http://stackoverflow.com/questions/6474775/setting-the-mysql-root-user-password-on-os-x) + +If `mysql` complains that the password is expired, follow the instructions on the second answer on [this post](http://stackoverflow.com/questions/33326065/unable-to-access-mysql-after-it-automatically-generated-a-temporary-password) + +#### MySQL Syntax Error + +If you get the following error + +```bash +Mysql2::Error: You have an error in your SQL syntax +``` + +this may be an issue with using an incompatible version of MySQL. Try switching to MySQL 5.7 if you are currently using a different version. + +#### Undefined method 'devise' for User +You most likely missed the step of copying 'config/initializers/devise.rb.template' to 'config/initializers/devise.rb' and setting your secret key in the setup instructions. \ No newline at end of file diff --git a/docs/installation/ubuntu.md b/docs/installation/ubuntu.md new file mode 100644 index 000000000..2194f1779 --- /dev/null +++ b/docs/installation/ubuntu.md @@ -0,0 +1,165 @@ +This page provides instructions on installing Autolab for development on Ubuntu 18.04 LTS. If you encounter any issue along the way, check out [Troubleshooting](/installation/troubleshoot). + +1. Upgrade system packages and installing prerequisites + + :::bash + sudo apt-get update + sudo apt-get upgrade + sudo apt-get install build-essential git libffi-dev zlib1g-dev autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev libncurses5-dev libgdbm5 libgdbm-dev libmysqlclient-dev libjansson-dev ctags + +2. Cloning Autolab repo from Github to ~/Autolab + + :::bash + cd ~/ + git clone https://github.com/autolab/Autolab.git + cd Autolab + +3. Setting up rbenv and ruby-build plugin + + :::bash + cd ~/ + git clone https://github.com/rbenv/rbenv.git ~/.rbenv + echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc + echo 'eval "$(rbenv init -)"' >> ~/.bashrc + source ~/.bashrc + + ~/.rbenv/bin/rbenv init + git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build + +4. Installing Ruby (Based on ruby version) + + :::bash + cd Autolab + rbenv install `cat .ruby-version` + +5. Installing SQLite + + :::bash + sudo apt-get install sqlite3 libsqlite3-dev + +6. Installing MySQL. (If you would just like to test Autolab, then you can skip this step by using SQLite) +Following instructions from How to Install MySQL on Ubuntu. + + :::bash + sudo apt install mysql-server + sudo mysql_secure_installation + + > There will be a few questions asked during the MySQL setup. + + * Validate Password Plugin? N + * Remove Annonymous Users? Y + * Disallow Root Login Remotely? Y + * Remove Test Database and Access to it? Y + * Reload Privilege Tables Now? Y + +7. (If you are using MySQL) Create a new user with access to `autolab_test` and `autolab_development` databases. Because a password rather than auth_socket is needed, we need to ensure that user uses `mysql_native_password` + + :::bash + sudo mysql + mysql> CREATE USER 'user1'@'localhost' IDENTIFIED WITH mysql_native_password BY ''; + mysql> FLUSH PRIVILEGES; + mysql> exit; + +8. Installing Rails + + :::bash + cd Autolab + gem install bundler + rbenv rehash + bundle install + +9. Initializing Autolab Configs + + :::bash + cd Autolab + cp config/database.yml.template config/database.yml + cp config/school.yml.template config/school.yml + cp config/initializers/devise.rb.template config/initializers/devise.rb + sed -i "s//`bundle exec rails secret`/g" config/initializers/devise.rb + cp config/autogradeConfig.rb.template config/autogradeConfig.rb + +10. (Using MySQL) Editing Database YML. +Change the and fields in config/database.yml to the username and password that has been set up for the mysql. For example if your username is `user1`, and your password is `123456`, then your yml would be + + :::yml + development: + adapter: mysql2 + database: autolab_development + pool: 5 + username: user1 + password: '123456' + socket: /var/run/mysqld/mysqld.sock + host: localhost + variables: + sql_mode: NO_ENGINE_SUBSTITUTION + + test: + adapter: mysql2 + database: autolab_test + pool: 5 + username: user1 + password: '123456' + socket: /var/run/mysqld/mysqld.sock + host: localhost + variables: + sql_mode: NO_ENGINE_SUBSTITUTION + +11. (Using SQLite) Editing Database YML. +Comment out the configurations meant for MySQL in config/database.yml, and insert the following + + :::yml + development: + adapter: sqlite3 + database: db/autolab_development + pool: 5 + timeout: 5000 + + test: + adapter: sqlite3 + database: db/autolab_test + pool: 5 + timeout: 5000 + +12. Granting permissions on the databases. Setting global sql mode is important to relax the rules of mysql when it comes to group by mode + + :::bash + (access mysql using your root first to grant permissions) + mysql> grant all privileges on autolab_development.* to ''@localhost; + mysql> grant all privileges on autolab_test.* to ''@localhost; + mysql> SET GLOBAL sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY','')); + mysql> exit + +12. Initializing Autolab Database + + :::bash + cd Autolab + bundle exec rails db:create + bundle exec rails db:reset + bundle exec rails db:migrate + +13. Create initial root user + + :::bash + ./bin/initialize_user.sh + +14. If you are just testing Autolab, you can populate the database with sample course & students + + :::bash + cd Autolab + bundle exec rails autolab:populate + +15. Run Autolab! + + :::bash + cd Autolab + bundle exec rails s -p 3000 --binding=0.0.0.0 + +16. Visit localhost:3000 on your browser to view your local deployment of Autolab, and login with either the credentials of the root user you just created, or choose `Developer Login` with + + Email: "admin@foo.bar" + +17. Install [Tango](/installation/tango), the backend autograding service. + +18. If you would like to deploy the server, you can try out Phusion Passenger. + +19. Now you are all set to start using Autolab! Visit the [Guide for Instructors](/instructors) and [Guide for Lab Authors](/lab) pages for more info. \ No newline at end of file diff --git a/docs/instructors.md b/docs/instructors.md index ad781bcf1..a113819f3 100644 --- a/docs/instructors.md +++ b/docs/instructors.md @@ -49,7 +49,7 @@ Labs can be _autograded_ or not, at your disrcretion. When a student submits to A lab has a _start date_, _due date_, _end date_ and _grading deadline_. The link to a lab becomes visible to students after the start date (it's always visible to instructors). Students can submit until the due date without penalty or consuming grace days. Submission is turned off after the end date. Grades are included in the gradebook's category and course averages only after the grading deadline. -## Handins +## Handins/Submissions Once an assessment is live (past the start date), students can begin submitting handins, where each handin is a single file, which can be either a text file or an archive file (e.g., `mm.c`, `handin.tar`). @@ -66,10 +66,6 @@ Autolab provides support for a late handin policy based on _grace days_. Each st Each lab contains at least one _problem_, defined by the instructor, with some point value. Each problem has a name (e.g., "Prob1", "Style") that is unique for the lab (although different labs can have the same problem names). -## Submissions - -Once an assessment is live (past the start date), students can begin making submissions (handins), where each submission is a single file. - ## Grades _Grades_ come in a number of different forms: diff --git a/docs/lab.md b/docs/lab.md index d5b508735..3e95e555f 100644 --- a/docs/lab.md +++ b/docs/lab.md @@ -25,6 +25,8 @@ By convention, an autograder accepts an optional `-A` command line argument that One of the nice properties of Autolab autograders is that they can be written and tested offline, without requiring any interaction with Autolab. Writing autograders is not easy, but the fact that they can be developed offline allows you to develop and test them in your own familiar computing environment. +To format your autoresult feedback provided to the students, use the [formatted feedback feature](/features/formatted-feedback). + ## Installing Autograded Labs After you've written and tested the autograder, you then use the Autolab web site to create the autograded lab. Autolab supports creating new labs from scratch, or reusing labs from previous semesters. We'll describe each of these in turn. @@ -78,9 +80,7 @@ If you don't have access to the lab directory, another option is to import a lab ## Example: Hello Lab -In this section we'll look at the simplest possible autograded lab we could imagine, called, appropriately enough, the -[Hello Lab](https://github.com/autolab/Autolab/tree/master/examples/hello) -(with [tarball](https://github.com/autolab/Autolab/tree/master/examples/hello.tar)), which is stored in a lab directory called `hello` in the Autolab github repo. While it's trivial, it illustrates all of the aspects of developing an autograded lab, and provides a simple example that you can use for sanity testing on your Autolab installation. +In this section we'll look at the simplest possible autograded lab we could imagine, called, appropriately enough, the Hello Lab (with tarball), which is stored in a lab directory called `hello` in the Autolab github repo. While it's trivial, it illustrates all of the aspects of developing an autograded lab, and provides a simple example that you can use for sanity testing on your Autolab installation. In this lab, students are asked to write a version of the K&R "hello, world" program, called `hello.c`. The autograder simply checks that the submitted `hello.c` program compiles and runs with an exit status of zero. If so, the submission gets 100 points. Otherwise it gets 0 points. @@ -88,7 +88,7 @@ In this lab, students are asked to write a version of the K&R "hello, world" pro Autolab expects to find the `autograde-Makefile`and `autograde.tar` files in the `hello` lab directory, but otherwise places no constraints on the contents and organization of this directory. However, based on our experience, we strongly recommend a directory structure with the following form: -[hello/README](https://github.com/autolab/Autolab/tree/master/examples/hello/README): +hello/README: ```md # Basic files created by the lab author @@ -120,7 +120,7 @@ The key idea with this directory structure is to place _all_ code for the lab in The main makefile creates `hello-handout` by copying files from `src`, and then tars it up: -[hello/Makefile](https://github.com/autolab/Autolab/tree/master/examples/hello/Makefile): +hello/Makefile: ```makefile # @@ -174,10 +174,10 @@ To build the lab, type `make clean; make`. You can do this as often as you like ### Source Directory -The [hello/src/](https://github.com/autolab/Autolab/tree/master/examples/hello/src) directory +The hello/src/ directory contains _all_ of the code files for the Hello Lab, including the files that are handed out to students: -[hello/src/README](https://github.com/autolab/Autolab/tree/master/examples/hello/src/README): +hello/src/README: ``` # Autograder and solution files @@ -194,10 +194,10 @@ hello.c-handout Blank hello.c file handed out to students ### Handout Directory -The [hello/hello-handout/](https://github.com/autolab/Autolab/tree/master/examples/hello/hello-handout/) directory +The hello/hello-handout/ directory contains the files that the students will use to work on the lab. It contains no hard state, and is populated entirely with files from `hello/src`: -[hello/hello-handout/README](https://github.com/autolab/Autolab/tree/master/examples/hello/hello-handout/README): +hello/hello-handout/README: ``` For this lab, you should write a tiny C program, called "hello.c", @@ -217,7 +217,7 @@ driver.sh Autolab autograder hello.c Empty C file that you will edit ``` -[hello/hello-handout/Makefile](https://github.com/autolab/Autolab/tree/master/examples/hello/hello-handout/Makefile) contains the rules that compile the student source code: +hello/hello-handout/Makefile contains the rules that compile the student source code: ```makefile # Student makefile for the Hello Lab @@ -239,7 +239,7 @@ $ ./hello The autograder for the Hello Lab is a trivially simple bash script called `driver.sh` that compiles and runs `hello.c` and verifies that it returns with an exit status of zero: -[hello/src/driver.sh](https://github.com/autolab/Autolab/tree/master/examples/hello/src/driver.sh) +hello/src/driver.sh: ```bash #!/bin/bash @@ -294,7 +294,7 @@ Notice that the autograder expects the `hello` lab on the Autolab front-end to h Autolab requires two _autograding files_ called `autograde.tar`, which contains all of the code required by the autograder, and `autograde-Makefile`, which runs the autograder on the autograding image when each submission is graded. For the Hello Lab, `autograde.tar` is simply a copy of the `hello-handout.tar` file that is handed out to students. And here is the corresponding -[hello/autograde-makefile](https://github.com/autolab/Autolab/tree/master/examples/hello/autograde-Makefile): +hello/autograde-makefile: ```makefile all: @@ -335,12 +335,8 @@ $ cd test-autograder && make clean && make The `hello/writeup` contains the detailed lab writeup, either html or pdf file, that students can download from the Autolab front end. -## FAQ +## Troubleshooting #### Why is Autolab not displaying my stdout output? -Autolab always shows the stdout output of running make, even when the program crashed or timed out. However, when it does crash and the expected autoresult json string is not appended to the output, parsing of the last line will fail. If this happens, any stdout output that is longer than 10,000 lines will be discarded (Note that this limit does not apply when the autoresult json is valid). - -#### Is there a way to formatting the feedback provided to the students - -Yes there is, by using the [formatted feedback feature](/features/#formatted-feedback). \ No newline at end of file +Autolab always shows the stdout output of running make, even when the program crashed or timed out. However, when it does crash and the expected autoresult json string is not appended to the output, parsing of the last line will fail. If this happens, any stdout output that is longer than 10,000 lines will be discarded (Note that this limit does not apply when the autoresult json is valid). \ No newline at end of file diff --git a/docs/reference.md b/docs/reference.md new file mode 100644 index 000000000..761ff67f7 --- /dev/null +++ b/docs/reference.md @@ -0,0 +1,9 @@ +# Reference Overview + +This section includes all the references for Autolab's Application Programming Interfaces (APIs) and is primarily intended for developers. + +| Reference | Description | +| ----------- | ------------------------------------ | +| [Autolab Frontend API](/api-overview) | API that allows developers to create clients that connect to Autolab's web server | +| [Tango API](/tango-rest) | API that is used to manage jobs on Tango | +| [Tango VMMS API](/tango-vmms) | API for creating new tango interfaces with VMMS | \ No newline at end of file diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css new file mode 100644 index 000000000..34ab9c08b --- /dev/null +++ b/docs/stylesheets/extra.css @@ -0,0 +1,6 @@ +[data-md-color-scheme="autolab"] { + --md-primary-fg-color: rgb(153, 0, 0); + --md-primary-fg-color--light: #C87B7E; + --md-primary-fg-color--dark: rgb(120, 0, 0); +} + diff --git a/docs/tango-cli.md b/docs/tango-cli.md index 39c113b1e..2cfc4c40f 100644 --- a/docs/tango-cli.md +++ b/docs/tango-cli.md @@ -1,10 +1,10 @@ # Tango Command Line Client -This is a guide to use the command-line client (`clients/tango-cli.py`) to test and collect other valuable information from Tango. Please [setup Tango](/tango/#installation) before moving forward. This guide assumes an instance of Tango is already up and running. +This is a guide to use the command-line client (`clients/tango-cli.py`) to test and collect other valuable information from Tango. Please [setup Tango](/installation/tango) before moving forward. This guide assumes an instance of Tango is already up and running. ## Running a Sample Job -The CLI supports two ways to run a sample job, [individual steps](/tango-cli/#indivdual-steps) or in a [single all-in-one command](/tango-cli/#single-command). The first option is better for debugging each individual API call, whereas the second option is best for quickly running a job. Other Tango CLI commands are also discussed [below](/tango-cli/#miscallaneous-commands). +The CLI supports two ways to run a sample job, [individual steps](/tango-cli/#individual-steps) or in a [single all-in-one command](/tango-cli/#single-command). The first option is better for debugging each individual API call, whereas the second option is best for quickly running a job. Other Tango CLI commands are also discussed [below](/tango-cli/#miscellaneous-commands). The Tango directory contains various different jobs in the `clients/` directory; `clients/README.md` discusses the function of each job. diff --git a/docs/tango-vmms.md b/docs/tango-vmms.md index a48e72b8d..1b105bdba 100644 --- a/docs/tango-vmms.md +++ b/docs/tango-vmms.md @@ -1,4 +1,9 @@ -This page documents the interface for Tango's Virtual Machine Management Systems' (VMMSs) API and instructions for setting up VMMSs. See [the vmms directory](https://github.com/autolab/Tango/tree/master/vmms) in Tango for example implementations. +This page documents the interface for Tango's Virtual Machine Management Systems' (VMMSs) API and instructions for setting up VMMSs. + +See the vmms directory in Tango for example implementations. + +We currently provide [Docker VMMS](/installation/tango/#docker-vmms-setup) and the [Amazon EC2 VMMS](/installation/tango/#amazon-ec2-vmms-setup) interfaces. The following API is for developers who intend to use a different kind of VMMS for Tango's autograding purposes. + ## API @@ -67,92 +72,3 @@ getVMs(self) ``` Returns a complete list of VMs associated with this Tango system. - -## Docker VMMS Setup - -This is a guide to set up Tango to run jobs inside Docker containers. - -1. Install docker on host machine by following instructions on the [docker installation page](https://docs.docker.com/installation/). Ensure docker is running: - - :::bash - $ docker ps # CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - -2. Build base Docker image from root Tango directory. - - :::sh - cd path/to/Tango - docker build -t autograding_image vmms/ - docker images autograding_image # Check if image built - -3. Update `VMMS_NAME` in `config.py`. - - :::python - # in config.py - VMMS_NAME = "localDocker" - -## Amazon EC2 VMMS Setup - -This is a guide to set up Tango to run jobs on an Amazon EC2 VM. - -1. Create an [AWS Account](https://aws.amazon.com/) or use an existing one. - -2. Obtain your `access_key_id` and `secret_access_key` by following the instructions [here](http://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). - -3. Add AWS Credentials to a file called `~/.boto` using the following format: - - :::bash - [Credentials] - aws_access_key_id = MYAMAZONTESTKEY12345 - aws_secret_access_key = myawssecretaccesskey12345 - - Tango uses the [Boto](http://boto.cloudhackers.com/en/latest/) Python package to interface with Amazon Web Services - -4. In the AWS EC2 console, create an Ubuntu 14.04+ EC2 instance and save the `.pem` file in a safe location. - -5. Copy the directory and contents of `autodriver/` in the Tango repo into the EC2 VM. For more help connecting to the EC2 instance follow [this guide](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html#AccessingInstancesLinuxSCP) - - :::bash - chmod 400 /path/my-key-pair.pem - scp -i /path/my-key-pair.pem -r autodriver/ ubuntu@.compute-1.amazonaws.com:~/ - - The autodriver is used as a sandbox environment to run the job inside the VM. It limits Disk I/O, Disk Usage, monitors security, and controls other valuable `sudo` level resources. - -6. In the EC2 VM, compile the autodriver. - - :::bash - $ cd autodriver/ - $ make clean; make - $ cp -p autodriver /usr/bin/autodriver - -7. Create the `autograde` Linux user and directory. All jobs will be run under this user. - - :::bash - $ useradd autograde - $ mkdir autograde - $ chown autograde autograde - $ chown :autograde autograde - -8. In the AWS EC2 console, create an AMI image from your EC2 VM. Use [this guide](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html#how-to-create-ebs-ami) to create a custom AMI. - -9. Exit the EC2 instance and edit the following values in `config.py` in the Tango directory. - - :::bash - # VMMS to use. Must be set to a VMMS implemented in vmms/ before - # starting Tango. Options are: "localDocker", "distDocker", - # "tashiSSH", and "ec2SSH" - VMMS_NAME = "ec2SSH" - ###### - # Part 5: EC2 Constants - # - EC2_REGION = 'us-east-1' # EC2 Region - EC2_USER_NAME = 'ubuntu' # EC2 username - DEFAULT_AMI = 'ami-4c99c35b' # Custom AMI Id - DEFAULT_INST_TYPE = 't2.micro' # Instance Type - DEFAULT_SECURITY_GROUP = 'autolab-autograde-ec2' # Security Group with full access to EC2 - SECURITY_KEY_PATH = '/path/to/my-key-pair.pem' # Absolute path to my-key-pair.pem - DYNAMIC_SECURITY_KEY_PATH = '' # Leave blank - SECURITY_KEY_NAME = 'my-key-pair' # Name of the key file. Ex: if file name is 'my-key-pair.pem', fill value with 'my-key-pair' - TANGO_RESERVATION_ID = '1' # Leave as 1 - INSTANCE_RUNNING = 16 # Status code of a running instance, leave as 16 - -10. You should now be ready to run Tango jobs on EC2! Use the [Tango CLI](/tango-cli/) to test your setup. diff --git a/docs/tango.md b/docs/tango.md deleted file mode 100644 index 5b6367642..000000000 --- a/docs/tango.md +++ /dev/null @@ -1,114 +0,0 @@ -# Tango - -Tango is a standalone RESTful Web service that runs jobs in virtual machines or containers. It was developed as a distributed grading system for [Autolab](/) and has been extensively used for autograding programming assignments. It is also open source and hosted on [Github](https://www.github.com/autolab/Tango). - -## Getting Started - -A brief overview of the Tango respository: - -- `tango.py` - Main tango server -- `jobQueue.py` - Manages the job queue -- `jobManager.py` - Assigns jobs to free VMs -- `worker.py` - Shepherds a job through its execution -- `preallocator.py` - Manages pools of VMs -- `vmms/` - VMMS library implementations -- `restful-tango/` - HTTP server layer on the main Tango - -Tango runs jobs in VMs using a high level Virtual Memory Management System (VMMS) API. Tango currently has support for running jobs in [Docker](https://www.docker.com/) containers (**recommended**) or [Amazon EC2](https://aws.amazon.com/ec2). - -For more information about the different Tango components, go to the following pages: - -- [REST API docs](/tango-rest/) -- [VMMS API docs](/tango-vmms/) -- [Tango Architecture Overview](https://docs.autolabproject.com/2015/04/making-backend-scalable/) -- [Deploying Tango](/tango-deploy/) - -### Installation - -This guide shows how to setup Tango in a **development environment**. Use the [deploying Tango](/tango-deploy/) guide for installing in a **production environment**. - -1. Obtain the source code. - - :::bash - git clone https://github.com/autolab/Tango.git; cd Tango - -2. Install Redis following [this guide](http://redis.io/topics/quickstart). By default, Tango uses Redis as a stateless job queue. Learn more [here](https://docs.autolabproject.com/2015/04/making-backend-scalable/). - -3. Create a `config.py` file from the given template. - - :::bash - cp config.template.py config.py - -4. Create the course labs directory where job's output files will go, organized by key and lab name: - - :::bash - mkdir courselabs - - By default the `COURSELABS` option in `config.py` points to the `courselabs` directory in the Tango directory. - Change this to specify another path if you wish. - -5. Set up a VMMS for Tango to use. - - - [Docker](/tango-vmms/#docker-vmms-setup) (**recommended**) - - [Amazon EC2](/tango-vmms/#amazon-ec2-vmms-setup) - - TashiVMMS (deprecated) - - -6. Run the following commands to setup the Tango dev environment inside the Tango directory. [Install pip](https://pip.pypa.io/en/stable/installing/) if needed. - - :::bash - $ pip install virtualenv - $ virtualenv . - $ source bin/activate - $ pip install -r requirements.txt - $ mkdir volumes - -7. If you are using Docker, set `DOCKER_VOLUME_PATH` in `config.py` to be the path to the `volumes` directory you just created. - - :::bash - DOCKER_VOLUME_PATH = "/path/to/Tango/volumes/" - - -8. Start Redis by running the following command: - - :::bash - $ redis-server - -9. Run the following command to start the server (producer). If no port is given, the server will run on the port specified in `config.py` (default: 3000): - - :::bash - python restful-tango/server.py - - Open another terminal window and start the job manager (consumer): - - :::bash - python jobManager.py - - For more information on the job producer/consumer model check out our [blog post](https://docs.autolabproject.com/2015/04/making-backend-scalable/) - -10. Ensure Tango is running: - - :::bash - $ curl localhost: - # Hello, world! RESTful Tango here! - -11. You can test the Tango setup using the [command line client](/tango-cli/). - -12. If you are using Tango with Autolab, you have to configure Autolab to use Tango. Go to your Autolab directory and enter the following commands: - - :::bash - cp config/autogradeConfig.rb.template config/autogradeConfig.rb - - Fill in the correct info for your Tango deployment, mainly the following: - - :::ruby - # Hostname for Tango RESTful API - RESTFUL_HOST = "foo.bar.edu" #(if you are running Tango locally, then it is just "localhost") - - # Port for Tango RESTful API - RESTFUL_PORT = "3000" - - # Key for Tango RESTful API - RESTFUL_KEY = "test" - -13. To deploy Tango in a standalone production environment, use this [guide](/tango-deploy/) diff --git a/mkdocs.yml b/mkdocs.yml index fa90c3833..4e92b8163 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,36 +1,48 @@ site_name: Autolab Documentation site_url: "https://docs.autolabproject.com/" +extra_css: + - stylesheets/extra.css nav: - - Autolab: - - Getting Started: index.md - - Guide for Instructors: instructors.md - - Guide for Lab Authors: lab.md - - Features: - - Formatted Feedback: features/formatted-feedback.md - - Scoreboards: features/scoreboards.md - - Embedded Forms: features/embedded-forms.md - - Annotations: features/annotations.md - - MOSS Plagiarism Detection: features/moss.md - - Metrics: features/metrics.md - - Autolab API: - - Overview: api-overview.md - - Reference: api-interface.md - - Command Line Interface: command-line-interface.md - - Managing Authorized Apps: api-managing-authorized-apps.md - - Tango: - - Getting Started: tango.md - - Tango CLI: tango-cli.md - - REST API: tango-rest.md - - VMMS Docs: tango-vmms.md - - Deploying Tango: tango-deploy.md - - Docker Compose Install: docker-compose.md - - (Legacy) OneClick Install: one-click.md + - Overview: index.md + - Installation: + - Installation Overview: installation/overview.md + - Docker Compose Install: installation/docker-compose.md + - Autolab: + - Ubuntu 18.04: installation/ubuntu.md + - Mac OSX 10.11+: installation/osx.md + - Troubleshooting: installation/troubleshoot.md + - Mailing Setup: installation/mailing.md + - Tango: installation/tango.md + # - (Legacy) OneClick Install: installation/one-click.md + - Getting Started: + - Autolab Frontend: + - Guide for Instructors: instructors.md + - Guide for Lab Authors: lab.md + - Features: + - Formatted Feedback: features/formatted-feedback.md + - Scoreboards: features/scoreboards.md + - Embedded Forms: features/embedded-forms.md + - Annotations: features/annotations.md + - MOSS Plagiarism Detection: features/moss.md + - Metrics: features/metrics.md + - Autolab Frontend CLI: command-line-interface.md + - Tango CLI: tango-cli.md + - Reference: + - Reference Overview: reference.md + - Autolab Frontend API: + - Overview: api-overview.md + - Endpoints: api-interface.md + - Managing Authorized Apps: api-managing-authorized-apps.md + - Tango API: tango-rest.md + - Tango VMMS API: tango-vmms.md theme: name: material - palette: - primary: "red" - accent: "blue" logo: "images/logo.svg" + palette: + scheme: autolab + features: + - navigation.tabs + - toc.integrate repo_name: "Autolab" repo_url: "https://github.com/Autolab/autolab" extra: @@ -47,3 +59,5 @@ markdown_extensions: - codehilite #linenums: true strict: true +plugins: + - search