Skip to content
This repository has been archived by the owner on May 3, 2024. It is now read-only.

SIGKILL when too many MockProver errors #130

Open
Brechtpd opened this issue Jul 31, 2023 · 4 comments
Open

SIGKILL when too many MockProver errors #130

Brechtpd opened this issue Jul 31, 2023 · 4 comments

Comments

@Brechtpd
Copy link

In the blockhash circuit there's a SIGKILL when there are too many errors generated by the MockProver.

Investigate the reason and solve/mitigate the problem.

@mratsim
Copy link

mratsim commented Aug 2, 2023

Discussion with @ggkitsas, it's possible that the reason is logged in dmesg since it seems to be from the kernel.

For example if an error requires allocating memory for displaying the panics, and you accumulate memory dumps quickly and you get SIGKILLed.

@ggkitsas
Copy link

ggkitsas commented Aug 15, 2023

dmesg of a sigkill shows it is running out of memory. Since this is happening only when the circuit fails, I assume it is running out of memory while collecting the humongous amount of errors halo2 produces.
[255025.374651] Out of memory: Killed process 360160 (zkevm_circuits-) total-vm:9780340kB, anon-rss:8324392kB, file-rss:708kB, shmem-rss:0kB, UID:1000 pgtables:16576kB oom_score_adj:0

Two possible approaches:

  1. We could find a way to limit the amount of errors rustc outputs. This might hide useful errors for debugging but still better than no error messages. If there is a way to do this it seems like it will be a low effort
  2. Investigate how halo2 manages error messages. It is likely that it accumulates all error messages in one buffer before printing them out. If that's the case we could modify it so that the errors are streamed out as they occur.

@ggkitsas
Copy link

might be of interest for case 1:
https://github.com/alopatindev/cargo-limit#readme

@ggkitsas
Copy link

A workaround for now is to initialize all cells to F::ZERO

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Status: 📝 Todo
Development

No branches or pull requests

3 participants