Closure()

Planning your next courses easily.

About

This project is part of the Open Source Software Workshop at the Hebrew University of Jerusalem.

Getting Started

1. Clone this repo.
2. Run python3 -m venv venv to install a new Python virtual environment.
3. Activate the virtual environment so your commands will be executed inside a virtual environment:
   • Windows: run venv\Scripts\activate.bat.
   • Mac OS/Linux: run source venv/bin/activate.
     
4. Run pip install -r requirements.txt to install the project's dependencies.
5. Run python Closure_Project/manage.py makemigrations rest_api
6. Run python Closure_Project/manage.py migrate
7. On PyCharm, right-click the outer Closure_Project directory, choose Mark Directory as and click Sources Root (its icon will be colored cyan afterwards).
8. Start the Django server with python Closure_Project/manage.py runserver

You now have a django instance with the database configured (yet blank).

The next step would be populating the database with Course information, so the whole ordeal can work.

See the Parser folder, or read the following subsection to learn more about the data structures used.

Populating the DB: HUJI Parser as example

The Hebrew University of Jerusalem serves course information as follows:

• Course details provide everything to be known regards courses: points given, teachers, course schedule etc.
• Track data shows which courses are required or available on each study-track
• Corner-Stone courses are listed on the program website, only available in Hebrew. (These are required courses in topics unrelated to one's study program or topic)

Note:

• It is important to parse courses before parsing tracks. When parsing a track, the parser validates the relevant courses already exist in the database.
• The example parser provided in the Parser folder is configured to fetch and parse information from the Hebrew versions of the said websites.

Downloading the data

To download the course files, run the Parser/MoonDownloader.py file (make sure venv is activated!). The folders tracks_html and course_details_html will be created.

Downloading will take time! (the moon website is...fragile)

Loading data offline

In case of network issues when downloading (or, to save yourself time), it's possible to load the data of all courses and tracks.

1. Download the 7z dump file (HUJI account required)
2. Extract it into the Parser folder
3. Run OfflineParser.py -> load_all_dumped() (Tip: if fetching corner-stone coursers gets stuck, comment out that line and continue with out that data, there's an open issue)

Parsing the data

Once you have both folders populated, run Parser/OfflineParser.py.

Parsing happens in the following order:

1. Parsing course details, parsed data is stored in parsed_courses.json
2. Parsing data relevant for the Track and CourseGroup objects. parsed data is stored in folders named parsed_tracks and parsed_groups as json files.
3. Loading the parsed course data as Models.Course objects in the Django database.
4. Fetching CornerStone course information (course id only), and marking relevant Course objects as is_corner_stone = True (None by default)
5. Loading the parsed Track data as Models.Track objects in the Django database.
6. Loading the parsed CourseGroup data as Models.CourseGroup objects in the Django database.

The aforementioned order is important, as some objects assume existence of others (e.g. CourseGroup and Course objects).

Generating auth

1. If you don't have admin superuser, creat one with python Closure_Project/manage.py createsuperuser
2. Use basic authentication with your username and password created, or generate token with python Closure_Project/manage.py drf_create_token and add {Autharization: Token <key>} to request headers.

Using the API

• To use the API, start the server with python Closure_Project/manage.py runserver.

• The base API url is https://<host>/api/v1.

• API documentation is available in https://<host>/redoc/

• Try to use API with: https://<host>/swagger/

• When making real requests, always remember to add the Authorization header to the request.

Contributions

Feel free to PR or open issues.