couchbase-lite-tests

This is the Couchbase Lite System Test Harness. It runs the system tests (tests requiring more than one device, e.g., a Couchbase Lite instance and a Sync Gateway) needed to verify a Couchbase Lite release.

The system consists of 4 components:

• An environment, built with docker compose, that contains one Couchbase Server and one Sync Gateway.
• The "client", a test harness written in python, that can configure the environment, run tests, and reports their results
• The tests, also written in python. Test are run within the client. They use it to configure the environment and then instruct servers to run tests in that environment.
• Per-platform "servers". The tests use the client instruct the server to run a specific test

What's here

• README.md - this file
• client - A python framework that configures the environment, runs tests and reports their results
• dataset - A collection datasets used in tests. Each dataset appears in two formats, as a cblite2 db, and as raw JSON
• environment - The docker compose script that creates and runs the environment
• jenkins - The jenkins pipelines and the docker compose for running the Jenkins server.
• servers - A test server for each CBL platform. Servers run on their platform and accept instructions from the client to run tests
• spec - Documentation: The specification for this system.
• tests - The python codelets that are run in the client to configure an environment and then instruct connected servers to run a test in that environment.

Running tests

Requirements

1. Docker
2. Python 3.9+
3. OpenSSL 1.1 for CBS Python SDK
4. Git LFS

Steps

1. Clone the repository.

   $ git clone https://github.com/couchbaselabs/couchbase-lite-tests.git
   

   This repository uses Git LFS to store binary dataset files. Ensure that you have Git LFS installed, and run git lfs install once to setup the extension hook before cloning the repository.

2. Open a terminal window and start the environment by running the ./start_environment.py script in the environment folder. The script will start CBS and SG in the docker container in the background and wait until SG is successfully started before exiting.

   cd environment
   ./start_environment.py
   

   • To shutdown the environment, run docker compose down.

3. Build and run the test server of the platform that you want to test.

   • C
   • Android
   • .Net
   • iOS

4. From the tests directory, set up a python virtual environment:

   cd tests
   python3 -m venv venv
   . venv/bin/activate
   pip install -r requirements.txt
   

   • You may need to use python<version> command e.g. python3.10 if you already have python3 for the other version.
   • You only need to create the python venv once. To reactivate run . venv/bin/activate, and to deactivate run deactivate.
   • When you update the repo or the Python TDK code, run pip install ../client to update the TDK.

5. Edit the file config.example.json with the URL of your Test server started in the Step 2.

   "test-servers": ["http://192.168.100.104:8080"]
   

6. Run the pytest tests as examples below.

   # Run one test file:
   pytest --config config.example.json test_basic_replication.py
   
   # Run all tests:
   pytest --config config.example.json
   
   # Run all with detail and without deprecation warning:
   pytest -v --no-header -W ignore::DeprecationWarning --config config.example.json