-
Notifications
You must be signed in to change notification settings - Fork 0
Using Celery
It is stated in the xAPI spec that, "If an Activity IRI is an IRL, an LRS SHOULD attempt to GET that IRL, and include in HTTP headers: "Accept: application/json, /". This SHOULD be done as soon as practical after the LRS first encounters the Activity id. Upon loading JSON which is a valid Activity Definition from an IRL used as an Activity id, an LRS SHOULD incorporate the loaded definition into its internal definition for that Activity, while preserving names or definitions not included in the loaded definition."
The ADL LRS will now retrieve activity metadata, void statements, and check LRS hooks via celery. This speeds up the processing of incoming statements, especially those that are in large batches, and gives the client a faster response.
In case batch statements don't come in the correct order when voiding statements, the LRS will wait until all statements from that batch are stored, then will start worker processes to attempt to void the statements that were specified instead of returning a 404 error when trying to void statements sequentially.
The LRS will be using Celery as an asynchronous task queue and RabbitMQ as its message broker to retrieve the activity metadata. This will be running completely independent of Django. The libraries required for this should already be installed in your environment (amqp and celery) from the requirements.txt document.
For details and more in-depth documentation, visit the celery docs.
admin:~$ sudo apt-get install rabbitmq-server
admin:~$ sudo rabbitmqctl add_user <username_for_rabbitmq> <password_for_rabbitmq>
admin:~$ sudo rabbitmqctl add_vhost <vhost_name>
admin:~$ sudo rabbitmqctl set_permissions -p <vhost_name> <username_for_rabbitmq> ".*" ".*" ".*"
For details and more in-depth documentation, visit the celery docs.
-
Configure /path/to/adl_lrs/settings.ini
[ampq] USERNAME: <username_for_rabbitmq> PASSWORD: <password_for_rabbitmq> HOST: localhost PORT: 5672 VHOST: <vhost_name>
-
Create supervisord.conf file in /path/to/ADL_LRS/
[unix_http_server] file=/tmp/supervisor.sock ; path to your socket file [supervisord] logfile=/path/to/logs/supervisord/supervisord.log ; CHANGE THIS PATH childlogdir=/path/to/logs/supervisord/ ; CHANGE THIS PATH logfile_maxbytes=50MB ; maximum size of logfile before rotation logfile_backups=10 ; number of backed up logfiles loglevel=info ; info, debug, warn, trace pidfile=/var/run/supervisord.pid ; pidfile location minfds=1024 ; number of startup file descriptors minprocs=200 ; number of process descriptors [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket [program:celery] ; Set full path to celery program if using virtualenv command=/path/to/env/bin/celery worker -A lrs --loglevel=INFO ; CHANGE THIS PATH directory=/path/to/ADL_LRS ; CHANGE THIS PATH numprocs=1 stdout_logfile=/path/to/logs/celery/celery-worker.log ; CHANGE THIS PATH stderr_logfile=/path/to/logs/celery/celery-worker-errors.log ; CHANGE THIS PATH autostart=true autorestart=true startsecs=10 ; Need to wait for currently executing tasks to finish at shutdown. ; Increase this if you have very long running tasks. stopwaitsecs = 600 ; When resorting to send SIGKILL to the program to terminate it ; send SIGKILL to its whole process group instead, ; taking care of its children as well. killasgroup=true ; if rabbitmq is supervised, set its priority higher ; so it starts first priority=998
-
Create upstart script in
/etc/init/
(our file will be called celerylrs.conf). You will need to use sudo for admin privileges.description "supervisor for lrs-celery" start on runlevel [2345] stop on runlevel [!2345] respawn setuid <Name of system user LRS is running under> chdir /path/to/ADL_LRS exec /path/to/env/bin/supervisord --nodaemon
You can stop start/stop the celery tasks by typing
sudo {start|stop|restart} celerylrs
NOTE: If you're using Ubuntu 15+ it uses systemd instead of upstart by default. Follow these instructions for a systemd script
Create systemd script in
/lib/systemd/system/
(our file will be called celerylrs.service). You will need to use sudo for admin privileges.[Unit] Description=Supervisor to run celery for the LRS [Service] User=<Name of system user LRS is running under> WorkingDirectory=/path/to/ADL_LRS ExecStart=/path/to/env/bin/supervisord --nodaemon Restart=always [Install] WantedBy=multi-user.target
Then run
sudo systemctl daemon-reload
andsudo systemctl enable celerylrs.service
.You can stop start/stop the celery tasks by typing
sudo systemctl {start|stop|restart} celerylrs