Development

This may need revision, ping me if you need clarification as I wrote it quickly.

Prereqs

• To work on Lute v3, you'll need at least Python 3.8 and pip. You'll probably want to use some kind of virtual environments; I use venv and so will write that out here.
• Note that GitHub CI tests Python versions 3.8 through 3.11, as we can't be sure what version of Python users have, so stay away from newer language features.
• Install MeCab according to your platform as described on the User Installation Page. This is required for passing unit tests irrespective of whether you are developing features relevant to the Japanese language.

Setup and verify your dev environment

1. Clone as usual, then

# Check out the current development branch
git checkout develop

# Pull in submodules
git submodule init
git submodule update

Submodules:

• currently, only the language definitions, https://github.com/LuteOrg/lute-language-defs

2. set up your virtual environment, install all dev dependencies from requirements-dev.txt, activate it:

python3.8 -m venv .venv
source .venv/bin/activate

# verify version
python --version

# Install dev requirements (which include prod reqs)
pip install -r requirements-dev.txt

# Install pre-commit hooks (optional, but recommended):
pre-commit install

deactivate

3. Copy lute/config/config.yml.example to lute/config/config.yml, making changes as you see fit.

If you're going to work on Lute, you're going to want to run unit tests. The unit tests are destructive, in that they wipe and reset the configured database.

To guard against mistakes, the DBNAME in your config.yml must start with test_, DATAPATH must be set, and the ENV must be dev. This ensures that you won't accidentally run the tests against your real Lute data. I work with this by having two completely separate environments: one for dev work, and one for real Lute usage. My prod data (actual data) stays in the latter.

4. Start lute up, ensure it's configured correctly

source .venv/bin/activate   # if necessary

python -m lute.main

# Open web browser to http://localhost:5000
# ... work work work ...
# When done, Ctl-C then
deactivate

5. Do initial run of all tests

Shut down your dev instance of Lute if it's running, and then run

inv full

to do a full pylint, test, acceptance, and playwright test run. This should complete without errors, as lute master and develop branch are always kept passing in CI.

Development

You may/may not find the overview docs of Lute's architecture useful ... let me know.

Commit hooks

Pre-commit hooks are installed with the pre-commit install step, and are run on every commit. I find this useful, as it stops me from having to go back and clean up, but YMMV. You can skip a step, e.g.: SKIP=pylint git commit -m "Some non-lint-compliant commit."

Testing

Testing is done with pytest and pytest-bdd. Run them as usual:

pytest
pytest path/to/file.py
pytest -k filter_string
pytest -m somemark
pytest -s
pytest -vv

inv or invoke for tasks

Lute3 uses Invoke to run tasks. Tasks are in tasks.py. See inv --list for commands.

Some useful tasks:

task	desc
inv start	start the app on a development Flask server in dev/debug mode
inv lint	lint
inv accept	start a running instance of the app server if needed, and run acceptance tests
inv playwright	start a running instance of the app server if needed, and run playwright tests

Database changes

Database migrations are managed automatically when Lute starts up. See the DB schema README for notes about how the schema changes are managed.

DB migrations are stored in /lute/db/schema/migrations. To create a script, run inv db.newscript <somesuffix>, and edit the file to create a Sqlite-compliant change script. See the existing scripts for examples.

When doing releases, the baseline database gets recreated as described in Releases -- if you add a migration, you don't need to recreate the baseline.

Dependencies: pip and flit

Dependencies are managed with pip freeze, and are in requirements.txt (prod) and requirements-dev.txt (dev, prod pulled in by reference).

Make sure any new dependencies are added to the correct file!

Basic pip doesn't have a good way to manage prod vs dev dependencies, as far as I know. If you add any new dependencies, first add them to requirements-all.txt:

pip install <pkg>; pip freeze > requirements-all.txt

and then add any new lines in that file to requirements.txt or requirements-dev.txt as appropriate. Then commit all requirements*.txt files.

IMPORTANT NOTES

• When you add something to requirements.txt, you must also add it to the pyproject.toml, as Flit (the build/release tool) doesn't use the requirements file.
• Make sure that your dependencies don't introduce other unexpected dependencies (e.g. that aren't under MIT licensing, that need compilers or build tools, etc).

TODOs

Todos are in the code as comments, e.g. # TODO [<group name>:] detail, <!-- TODO ... -->. inv todos collects all of these in a simple report.

The "group name" is arbitrary: it's used to group things together, which is handy when several locations need to be changed as a single unit of work.

Docker

Notes for building and running a Docker container are at ../docker/README.com.

Developing parser plugins

As of v3.4.0, Lute supports a simple "language parser plugin" capability. Set up your dev environment as explained on this page, and then head over to Developing language parser plugins.

Misc dev notes

Finding mecab.so for Docker

This is much tougher than it needs to be ...

To find the correct path, first build and run the container, then connect to it, and find libmecab.so.2 like this:

$ docker exec -it lute_v3-lute-1 bash
root@cid/# which mecab
/usr/bin/mecab
root@cid:/# ldd /usr/bin/mecab
   ...
   libmecab.so.2 => /lib/aarch64-linux-gnu/libmecab.so.2 (0x0000ffff9b540000)

Different platform architectures have this in different locations. :-/

datatables

Datatables is used for the term and book listings. See lute/static/vendor/datatables/README.md for notes about getting and updating it.

read-only db during tests

It appears that killing acceptance tests mid-run results in a zombie (?) python process that keeps a handle on the db, causing it to get locked in read-only mode.

I couldn't find a better way to kill this process than do a full machine restart. Sledgehammer approach that works.

Acceptance tests suddenly failing

Worning during run of tests with inv accept --exitfail:

WARNING  selenium.webdriver.common.selenium_manager:selenium_manager.py:139 The chromedriver version (118.0.5993.70) detected in PATH at /opt/homebrew/bin/chromedriver might not be compatible with the detected chrome version (119.0.6045.105); currently, chromedriver 119.0.6045.105 is recommended for chrome 119.*, so it is advised to delete the driver in PATH and retry

brew upgrade chromedriver

Then, on a Mac, have to "allow" it:

/opt/homebrew/bin/chromedriver --version

Will show message: "“chromedriver” can’t be opened because Apple cannot check it for malicious software." Click "Show in Finder", then in Finder, click "Open" and say "OK" when it can't be verified. Yes, this is a security risk.