feat: add opentelemetry tracing support #39
Conversation
|
Any idea on the overhead this introduces? It doesn't look like it would be a lot, but we may want to have this set up so it can be enabled when needed and disabled in general. |
I think it is minimal, any tfhe or network operation will be much slower than tracing. At least in the beginning of the project I believe it is critical we have a good tracing of which ciphertexts are slowly executed so it would be easier to troubleshoot issues. As for the implementation, all traces are tracked in memory and flushed in the background by batches, it shouldn't be noticeable or disruptive to our application in any way. In production enabling traces post factum when something happened is already too late, it is standard practice to have that information beforehand. |
david-zk
force-pushed
the
davidk/tracing
branch
from
dc6c95e to
fdb469c
Compare
| if let Err(err) = tracing::setup_tracing() { | ||
| panic!("Error while initializing tracing: {:?}", err); | ||
| } |
There was a problem hiding this comment.
Nit: to avoid if statements, we could make main() return a Result and use ?.
| @@ -390,7 +390,8 @@ impl SupportedFheCiphertexts { | |||
| SupportedFheCiphertexts::FheBytes128(_) => 10, | |||
| SupportedFheCiphertexts::FheBytes256(_) => 11, | |||
| SupportedFheCiphertexts::Scalar(_) => { | |||
| panic!("we should never need to serialize scalar") | |||
| // need this for tracing as we join types of computation for a trace | |||
| 200 | |||
There was a problem hiding this comment.
Maybe we can define a constant somewhere to say that 200 or some bigger number is "invalid"?
Adds opentelemetry tracing for fhe backend worker, we can see every work item processed with operation, input types, output type and output handle
@dartdart26 @antoniupop