Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with re-running the java version of the client #2

Open
neomatrix369 opened this issue Jan 27, 2021 · 4 comments
Open

Issues with re-running the java version of the client #2

neomatrix369 opened this issue Jan 27, 2021 · 4 comments

Comments

@neomatrix369
Copy link

After cloning this project, and running the steps as mentioned in the README. I can the generated uber jar. It runs and sends metrics to my project the first time when I run:

$ java -jar target/client-ng-java-1.0-SNAPSHOT-jar-with-dependencies.jar

but then it fails to shutdown the grpc server -- hangs for a long time.

Rerunning the jar results in hanging/pausing.

Workaround:

$ kill -9 $(ps aux | grep grpc | awk '{print $2}')

followed by

$ wandb grpc-server

And then re-running the jar works fine.

@neomatrix369
Copy link
Author

neomatrix369 commented Jan 27, 2021

One instance of execution gave this exception:

$ java -jar target/client-ng-java-1.0-SNAPSHOT-jar-with-dependencies.jar
Hello from wandb in java using GRPC!
Creating Runner
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/anaconda3/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/anaconda3/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/anaconda3/lib/python3.7/site-packages/grpc/_server.py", line 875, in _serve
    if not _process_event_and_continue(state, event):
  File "/anaconda3/lib/python3.7/site-packages/grpc/_server.py", line 842, in _process_event_and_continue
    concurrency_exceeded)
  File "/anaconda3/lib/python3.7/site-packages/grpc/_server.py", line 750, in _handle_call
    thread_pool)
  File "/anaconda3/lib/python3.7/site-packages/grpc/_server.py", line 726, in _handle_with_method_handler
    method_handler, thread_pool)
  File "/anaconda3/lib/python3.7/site-packages/grpc/_server.py", line 632, in _handle_unary_unary
    method_handler.response_serializer)
  File "/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 153, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

@neomatrix369
Copy link
Author

Would this PR #1 be able to help with this issue?

@raubitsj
Copy link
Member

We only support a single wandb run with the java client right now.
If you make sure there is no grpc-server running and launch the jar file, it will launch the grpcserver and it will shut it down.

I am leaving this open until we fix the single grpc-server issue, there is no reason for it as we can run them at different ports.

Eventually we will be able to run multiple java clients in parallel using the same grpc-server but that is farther out.

@raubitsj
Copy link
Member

There are cases where if the java code crashes, the grpc-server might remain running. That is a bug and will be fixed when we allow multiple java clients to run in parallel (on different ports)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants