Command-line tool and GoLang API for working with EOCS and OLX courseware. This project also supports importing/updating course to EXLskills.com and e-books in PDF/MD formats!
- Linux or OS X recommended (PRs for Windows support are welcome)
- GoLang 1.10.^ (
GOPATHmust be set) - GoLang
deptool Install Guide - NodeJS v8.10+
- Yarn 1.7+
# Prior to running `go get`, make sure that you have setup Go with the $GOPATH environment variable, otherwise this will not work
go get -u github.com/exlskills/eocsutil
cd $GOPATH/src/github.com/exlskills/eocsutil
dep ensure -v
yarn install
go build # Optional, but this will validate that you have the correct golang deps
The conversion process uses git commits of the source file structure to determine the last modification date for each course object. In the Server mode, the content of the git repository is cloned as part of the process. When loading a course from a local directory, if the directory is not a git clone, the assignment of timestamps will be bypassed
- EOCS-formatted course files are placed on the file system, preferably, in a git cloned local repository
- Target MongoDB is running locally (for a production/remote configuration, refer to your sysadmin/internal guides to get the mongodb connection URI; for MongoDB Atlas, use the 3.4 connection URI)
export MGO_DB_NAME="<name of the MongoDB target database>"
# Note: if Elasticsearch URI is not provided - the indexing will be bypassed
export ELASTICSEARCH_URI="http://localhost:9200"
# The default value for Elasticsearch "base" index is "learn". The actual index name will be set to the base plus "_<course launguage>", e.g., "base_en"
# To override the base name, set the Environment variable as below
export ELASTICSEARCH_BASE_INDEX="learn"
# Note: `go run` will compile eocsutil on the fly with any code changes, to compile ahead of time, use `go build` and then execute the binary
# MongoDB URI *must* start with `mongodb:` - version 3.4 style
go run main.go convert --from-format eocs --from-uri <path to the course files folder> --to-format eocs --to-uri mongodb://localhost:27017
Some Elasticsearch security models, e.g., AWS VPC Elasticsearch Service, require HTTPS connectivity in either Production or Test modes. To bypass Certificate validation in testing, ensure to set
export MODE=debug
A full example of connecting to AWS VPC Elasticsearch Service for testing:
// This is run in a separate terminal session that creates a tunnel to the AWS VPC via any instance running in the VPC:
ssh -i ~/.ssh/myAwsInstanceKey [email protected] -N -L 19200:vpc-my-es-service.us-west-2.es.amazonaws.com:443
// This is run in the eocsutil testing terminal session:
export MODE=debug
export ELASTICSEARCH_URI="https://localhost:19200"
The server mode is design to automatically process course load from GitHub repositories into MongoDB upon receiving push notifications via GitHub Webhooks with application/json content type.
In addition to the Environment variables described above, these values should be configured:
- GH_SERVER_ADDR and GH_SERVER_PORT, 0.0.0.0 and 3344 by default - used launching the server
- GH_SERVER_MONGO_URI - course load destination, see the note on MongoDB URI above
- GH_USER_TOKEN and GH_WEBHOOK_SECRET - see GitHub Webhook documentation on configuring Token on the account and Secret in the Webhook. The Token is used to push changes to the repository that can be made when the loader runs, as well as to pull from private repositories
- GH_WEBHOOK_BRANCH - one value or a comma-delimited list with no spaces defining the case-sensitive filter on the branch(es) of the GitHub repository in scope to trigger the processing. Default "master"
- GH_AUTOGEN_COMMIT_MSG - a string used when the supplementary files in the course content are updated by the loader with the assigned record key values and re-committed back to the GitHub repository. The value is matched to bypass processing of the Webhook generated by the auto-commit. Default "auto#gen"
- SMTP_FROM_NAME, SMTP_FROM_ADDRESS, SMTP_HOST, SMTP_CONNECTION_STRING, SMTP_USER_NAME, SMTP_PASSWORD - used to send a confirmation email on the load results to the email associated with the GitHub commit. Set SMTP_HOST to "" to bypass this functionality
- SERVER_NICKNAME - used in the emails to identify the Server. Default - EOCS_GH
To run the server:
go run main.go serve-gh-hook
/v1/github/repo-push-event
- Install ngrok on the local box and launch it with
http 3344options - Put the ngrok-generated URL into the GitHub webhook (without the port), e.g.,
https://1234ab56.ngrok.io/v1/github/repo-push-event - Start the server locally and submit/resubmit the webhook for testing
Sometimes there's an issue with showdownjs nodejs service that eocsutil spawns for markdown<->html conversion not exiting after eocsutil exits. If this occurs, run ps -ax | grep "node --harmony showdownjs/server.js" and kill that process.
Internally, we use a REST API server on http://localhost:6222 to communicate with showdownjs. We're working to select a port on the fly, but until that feature is complete, you will have to either (a) modify eocsutil to use a different port or (b) temporarily stop whatever process is using port 6222 on your machine.