Skip to content

Latest commit

 

History

History
344 lines (231 loc) · 14.1 KB

README.md

File metadata and controls

344 lines (231 loc) · 14.1 KB
Raw

Developing the Web Almanac

The Web Almanac can be developed on macOS, Windows or Linux. It requires Node v12, Python v3.8 and pip to be installed. Alternatively, use Docker to avoid manually configuring the development environment.

Style guide

The Web Almanac uses a specific "style guide" for code, including 2 spaces for indentation (except Python which uses 4 spaces).

An .editorconfig file exists for those using EditorConfig to help enforce those styles. This may require installing the editorconfig node module via npm (we suggest installing globally - npm install -g editorconfig ) and then installing the extension for your IDE (for example there is a Visual Studio Code extension).

Run Locally

Make sure you run the following commands from within the src directory by executing cd src first.

Source

Make sure Python (3.8 or above), pip and NodeJS (v12) are installed on your machine.

  1. If you don't have virtualenv, install it using pip.
sudo pip install virtualenv

Or for Windows

py -m pip install --user virtualenv
  1. Create an isolated Python environment, and install dependencies:
virtualenv --python python3.8 .venv
source .venv/bin/activate

Or for those on Windows:

virtualenv --python python3 .venv
.venv\Scripts\activate.bat
  1. Install generate and run the website:
npm install
npm run start
  1. In your web browser, enter the following address: http://127.0.0.1:8080

To stop the server run the following:

npm run stop

Generating chapters

The chapter generation is dependent on nodejs, so you will need to have nodejs installed as well. All of the following commands must be run from within the src directory by executing cd src first.

Note chapters are automatically generated by the npm run start command above, and individual chapters will automatically be regenerated when the markdown is changed. However it is listed here separately in case you want to regenerate if making changes to chapters.

Install the dependencies:

npm install

Run the generate chapters script:

npm run generate

If you have the server up you can test all the pages are being served correctly:

npm run test

You can also run single chapters, so you don't have to wait for the full run time:

npm run generate en/2019/css

Or even patterns (note patterns must be in quotes to prevent OS attempting to match to files):

npm run generate ".*/2019/css"
npm run generate "en/.*/css"
npm run generate ".*/2020/.*"

There is also a file watcher, which monitors the content directory and automatically regenerates a chapter when it sees it being modified (this is run automatically as part of npm run start):

npm run watch

Linting files

When you push changes to GitHub they will be linted using the GitHub Super-Linter.

It is possible to run the Super-Linter locally if you have Docker installed.

First up pull the Super-Linter Docker image (this takes some time to download but only need to do this once or when you want to upgrade the version of the super-linter):

docker pull github/super-linter:latest

Then to run the linting do this:

npm run lint

This can take a while to run so you can lint just subsets of files or folders:

npm run lint tools/generate templates/base

Linting SQL files

SQL files will be linted by SQLFluff. This tools adds the ability to autofix some simple errors (e.g. spacing, capitalisation and commas) when run locally. To lint the 2020 resource-hints SQL files, for example, install the python environment as per above, and then issue the following command:

sqlfluff lint ../sql/2020/resource-hints

This will return errors like the following:

% sqlfluff lint ../sql/2020/resource-hints
== [../sql/2020/resource-hints/adoption_service_workers.sql] FAIL
L:  25 | P:  26 | L010 | Inconsistent capitalisation of keywords.
L:  26 | P:  37 | L010 | Inconsistent capitalisation of keywords.
L:  34 | P:  63 | L038 | Trailing comma in select statement forbidden
All Finished 📜 🎉!

This states that:

  • On line 25, in position 26 you are using lowercase for keywords (e.g. as instead of AS) so failed rule L010.
  • Similarly on line 26, position 37.
  • And finally on line 34, position 63 you have an unnecessary comma (e.g. SELECT a,b, FROM table) and so failed rule L038. Remove the extra comma.

The list of rules can be found in the SQLFLuff documentation though we have turned a few of them off and configured others for our style (see the our .sqlfluff file if curious).

If you see any "unparseable" or PRS errors, then this is either an error in your code, or perhaps you've discovered a bug. Reach out to Barry (@tunetheweb) for help if stuck.

To attempt to autofix the errors you can use the fix command, instead of lint:

sqlfluff fix ../sql/2020/resource-hints

Which will produce similar output but with an offer to fix the issues it thinks it can fix:

% sqlfluff fix ../sql/2020/resource-hints
==== finding fixable violations ====
== [../sql/2020/resource-hints/adoption_service_workers.sql] FAIL
L:  25 | P:  26 | L010 | Inconsistent capitalisation of keywords.
L:  26 | P:  37 | L010 | Inconsistent capitalisation of keywords.
L:  34 | P:  63 | L038 | Trailing comma in select statement forbidden
==== fixing violations ====
3 fixable linting violations found
Are you sure you wish to attempt to fix these? [Y/n]

If you lint again you should see most of the errors are fixed. Note that not all errors can be autofixed and some will require manual intervention but autofixing is useful for the simple errors. So while it's generally OK to run the fix command, do run the lint command when all clean to make sure the fix command didn't miss any issues.

Generating Ebooks

For generating PDFs of the ebook, you need to install Prince. Follow the instructions on the Prince Website and pdftk.

To actually generate the ebooks, start your local server, then run the following:

npm run ebooks

There is a GitHub Action which can be run manually from the Actions tab to generate the Ebooks and open a Pull Request for them. This is the easiest way to generate them.

Generating ebooks - including print-ready ebooks if you want a hardcopy

It is also possible to generate the ebook from the website (either production or 127.0.0.1), with some optional params (e.g. to print it with different settings).

prince "http://127.0.0.1:8080/en/2019/ebook?print" -o web_almanac_2019_en.pdf --pdf-profile='PDF/UA-1'

Note --pdf-profile='PDF/UA-1' may not be needed if just intend to print.

Query params accepted are:

  • print - this adds left, right pages, footnotes, and sets roman numerals for front matter page numbers and adds footnotes. It is used by default when running npm run ebooks but we could change that if prefer a less print-like ebook.
  • printer - this adds crop marks, bleeds and trims. Also adds two additional pages at front which will need to be deleted in Acrobat or similar to get clean starting page.
  • page-size - this allows you to override the default page size of A5.
  • inside-margin - this allows you to set an inside margin for binding (e.g. on right for left hand pages and vice versa)
  • bleed - add a bleed for printing (3mm by default)
  • prince-trim - add a bleed for printing (5mm by default)
  • base-font-size - set the base font-size (10px by default), which is useful if changing page size.
  • max-fig-height - defaults to 610px (for A5) and prevents large images from causing overflows on to other pages with heading and caption.
  • cover - this genarates a 4 page cover (front cover + spine + back cover, and same on inside which is blank). This ignores above options but has further params discussed below.

You can also download the HTML and override the inline styles there if you want to customise this for something we haven't exposed as a param, and then run prince against the file.

So for a printer-ready A5 version, that you can send to a print to bind, you can do the following:

prince "http://127.0.0.1:8080/en/2019/ebook?print&printer" -o static/pdfs/web_almanac_2019_en_print_A5.pdf

This is the same as below since it uses all the default settings:

prince "http://127.0.0.1:8080/en/2019/ebook?print&printer&page-size=A5&inside-margin=19.5mm&bleed=3mm&prince-trim=5mm&base-font-size=10px" -o static/pdfs/web_almanac_2019_en_print_A5.pdf

Note this will create two extra pages at the begining which will need to be removed with a PDF editor (e.g. Adobe Acrobat) to start with a clean page starting on right hand side for printing. Please remove these before checking in versions into git.

It is also possible to generate a cover using the &cover URL param. This consists of basically 2 pages - the first page is a double width-page with front and back cover as one page (with spine in between) and the second page is a blank inside page.

prince "http://127.0.0.1:8080/en/2019/ebook?cover" -o static/pdfs/web_almanac_2019_en_cover_A5.pdf

Extra params accepted for the cover are are (note spine and pageWidth are unit-less to allow for easy addition in the code):

  • spine - the width of the spine (defaults to 25 for 2019 and 34.5 for 2020)
  • spinePadding - padding of the spine (defaults to "(spine - 25) / 2") to allow spine to look similar across years even though 2020 was a lot thicker than 2019.
  • pageWidth - the front cover width (note is just the page width and not the full width of front cover and back cover and spine) - defaults to 148 (for A5).
  • pageHeight - defaults to 210 (for A5)
  • unit - which unit the above measurements are in (defaults to mm)
  • base-font-size - set the base font-size (10px by default), which may need to be increased if changing page size.

So default is the same as:

prince "http://127.0.0.1:8080/en/2019/ebook?cover&spine=25&pageWidth=148&pageHeight=210&unit=mm&base-font-size=10px" -o static/pdfs/web_almanac_2019_en_cover_A5.pdf

Note, similar to above, this will create one extra page at the begining which will need to be removed with a PDF editor to start with a clean page for printing. Please remove this before checking in versions into git.

With the print-ready eBook and Cover you can send them to a printer. I used https://www.digitalprintingireland.ie/ before and they were excellent and charge about €35 for a full-colour A5 ebook. Most of the settings above are for them, so tweak them based on your own printer's requirements.

Deploying changes

If you've been added to the "App Engine Deployers" role in the GCP project, you're able to push code changes to the production website.

Make sure you have generated the ebooks PDFs first in the main branch, by running the Generate Ebooks GitHub Action

  1. Install the gcloud Google Cloud SDK.

  2. Authenticate the email address associated with the project with the webalmanac GCP project:

gcloud init
  1. Deploy the site:
npm run deploy

The deploy script will do the following:

  • Ask you to confirm you've updated the eBooks via GitHub Actions
  • Switch to the production branch
  • Merge changes from main
  • Do a clean install
  • Run the tests
  • Ask you to complete any local tests and confirm good to deploy
  • Ask for a version number (suggesing the last verision tagged and incrementing the patch)
  • Tag the release (after asking you for the version number to use)
  • Generate a deploy.zip file of what has been deployed
  • Deploy to GCP
  • Push changes to production branch on GitHub
  • Ask you to update the release section of GitHub
  1. Browse the website in production to verify that the new changes have taken effect. Not we have 3 hour caching so add random query params to pages to ensure you get latest version.

Developing in Docker

Assuming that you have Docker installed and running, ensure that the working directory is src, where the Dockerfile is present, before running the following commands.

  1. Build a Docker image named webalmanac (if you choose a different name, adjust following commands accordingly):
docker image build -t webalmanac .
  1. Run the application server (which is the default command of the Docker image, so no need to explicitly supply it as an argument):
docker container run --rm -it -v "$PWD":/app -p 8080:8080 webalmanac

  1. Open http://localhost:8080 in your web browser to access the site. You can kill the server when it is no longer needed using Ctrl+C.

  2. Make changes in the code using any text editor and run tests (need to build the image again if any Python or Node dependencies are changed):

docker container run --rm -it -v "$PWD":/app webalmanac npm run pytest
  1. To avoid running commands in one-off mode run bash in a container (with necessary volumes mounted and ports mapped) then run successive commands:
docker container run --rm -it -v "$PWD":/app -v /app/node_modules -p 8080:8080 webalmanac bash
root@[CID]:/app# python main.py
^C
root@[CID]:/app# npm run generate
root@[CID]:/app# npm run pytest
root@[CID]:/app# exit
  1. To customize the image use PYVER, NODEVER, and SKIPGC build arguments to control which versions of Python and Node are used and whether Google Cloud SDK is installed.
docker image build --build-arg PYVER=3.8 --build-arg NODEVER=14.x --build-arg SKIPGC=false -t webalmanac:custom .
  1. If you want to run the GitHub Super-Linter without npm being installed you need to call the command directly as given in package.json.

This will depend on your operating system but for MacOS/Linux this would be:

docker container run -it --rm -v /app/node_modules -v "$PWD/..":/app -w /app/src --entrypoint=./tools/scripts/run_linter_locally.sh github/super-linter

And for Windows:

docker container run --rm -v /app/node_modules -v %cd%\\..:/app -w /app/src --entrypoint=./tools/scripts/run_linter_locally.sh github/super-linter