Adding lock to control experiment creation during multiprocessing

[pebeto]

This PR adds tests required by #25, but also solves the issue presented in JuliaAI/MLFlowClient.jl#40.

Using julia --threads 5, each thread will continue its own processing until reaches the log_evaluation function. In that step, a lock (mutex) will turn the entire process into a sync one ensuring thread-safeness.

Feature diagram:

[ablaom]

Since this PR introduces locks, I am wondering what the impact might be on the parallelization of the machine learning tasks it is tracking. Is it possible that by stealing control, MLJFlow.jl essentially prevents otherwise parallel threads from actually executing simultaneously?

Be great to see a benchmark showing this is not likely (e.g., using this MLJTuning branch. Can provide guidance with that.

[pebeto]

In this branch, I implemented a Channel approach to ensure a queue usage. However, it is still buggy without a multiprocessing step being done (i.e. evaluating without acceleration).

[ablaom]

In this branch, I implemented a Channel approach to ensure a queue usage. However, it is still buggy without a multiprocessing step being done (i.e. evaluating without acceleration).

Great to see some progress there. Does this exploration mean the current PR indeed "locks out" the parallelization? I still wonder if it's more natural to implement the buffering at MLFlowClient.jl, as suggested in my POC. Sorry I haven't had time to flush that out to full blown PR.

[pebeto]

I still wonder if it's more natural to implement the buffering at MLFlowClient.jl

I don't agree with adding this thread-safe "tweaks" into MLFlowClient.jl. That library should mirror the implementations in other languages (which don't support concurrency completely). That's why I preferred to apply this in MLJFlow.jl instead, because this issue belongs to MLJ multi-threading/multi-processing operations.

---

Benchmarks

Channel-oriented approach:

# CPUThreads
 Range (min … max):  2.988 s …   3.105 s  ┊ GC (min … max): 0.13% … 0.19%
 Time  (median):     3.046 s              ┊ GC (median):    0.16%
 Time  (mean ± σ):   3.046 s ± 82.388 ms  ┊ GC (mean ± σ):  0.16% ± 0.04%

                                                             
  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁
  0.5 ns      Histogram: log(frequency) by time        8 ns <


# CPUProcesses
 Range (min … max):  3.038 s …   3.171 s  ┊ GC (min … max): 0.12% … 0.13%
 Time  (median):     3.105 s              ┊ GC (median):    0.13%
 Time  (mean ± σ):   3.105 s ± 94.311 ms  ┊ GC (mean ± σ):  0.13% ± 0.01%

                                                             
  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁
  0.5 ns      Histogram: log(frequency) by time        8 ns <

Mutex-oriented approach:

# CPUThreads
 Range (min … max):  3.013 s …    3.194 s  ┊ GC (min … max): 0.10% … 0.67%
 Time  (median):     3.103 s               ┊ GC (median):    0.40%
 Time  (mean ± σ):   3.103 s ± 127.769 ms  ┊ GC (mean ± σ):  0.40% ± 0.40%

                                                              
  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁
  0.5 ns       Histogram: log(frequency) by time        8 ns <


# CPUProcesses
 Range (min … max):  3.036 s …   3.155 s  ┊ GC (min … max): 0.11% … 0.15%
 Time  (median):     3.096 s              ┊ GC (median):    0.13%
 Time  (mean ± σ):   3.096 s ± 84.371 ms  ┊ GC (mean ± σ):  0.13% ± 0.03%

                                                             
  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁
  0.5 ns      Histogram: log(frequency) by time        8 ns <

@ablaom

[pebeto]

Closing in favor of #41