EnclaveLost error occurred when running DB in multi-threading mode

[hiroki-chen]

How did you build it?

First, build the enclave under platform_sgx using make:

SGX_MODE=HW TASK=task_db NATIVE_ENCLAVE=0 make -j

Then clone this repo:

git clone https://github.com/hiroki-chen/ycsb-pocf.git

Run the enclave:

$ ./app 127.0.0.1 1234
[2023-05-15T14:18:52Z INFO  app] [+] Listening to 127.0.0.1:1234
[2023-05-15T14:18:52Z INFO  app] [+] Initializing the enclave. May take a while...

How did you run it?

Start the YCSB benchmark client using workload A and make sure workloada.toml contains the following fields:

operationcount = 1000000
recordcount = 1000000

and then spawn 8 concurrent clients:

$ cd ycsb-pocf
$ TEE_TYPE=SGX cargo run -r --features=sgx,sgx_no_verify -- load -t 8 -w workloads/workloada.toml
[2023-05-15T14:18:53Z INFO  ycsb_pocf] Arguments are
    Args {
        run_type: Load,
        address: "127.0.0.1",
        port: 1234,
        workload: "./workloads/workloada.toml",
        thread_number: 8,
    }

Then the error will occur:

[2023-05-15T14:19:58Z ERROR app] [-] ECALL Enclave Failed, reason: EnclaveLost!

This also makes consequent request fails with EnclaveCrashed due to a missing enclave. Interestingly, however, using fewer threads makes the error disappear. The TCS number in the configuration file is set to 16 and the thread pool's TCS contant is 10.

Other stuffs

You cannot run the quote verification enclave on the same machine if you want to run the YCSB client in multi-threading mode because the dcap verification library launches an enclave too, which causes a conflict with our PoBF enclave.

Any insights? @ya0guang

[hiroki-chen]

See also apache/incubator-teaclave-sgx-sdk#432, apache/incubator-teaclave#688, and apache/incubator-teaclave#175.

[hiroki-chen]

Supplementary facts: this error is likely to be triggered if the system is currently undergoing a heavy workload.