Unpickling error in "get_server_weights()" in HogwildSparkModel.py line 33

[jiseungshin]

Hello, thank you very much for this beautiful project. I try to test mnist example on our cluster but so far without any success :(
The first iteration of training runs without any problems, but in the weight updating step it crashes with the error:

weights = get_server_weights(master_url)
File "/grid/3/hadoop/yarn/local/usercache/user/appcache/application_blabla/container_blabla/virtualenv_application_blabla/lib/python3.6/site-packages/sparkflow/HogwildSparkModel.py", line 33, in get_server_weights
weights = pickle.loads(r.content)
_pickle.UnpicklingError: invalid load key, '<'.

I thought, that maybe the weights are not corretly pickled before sending to master, so I checked in the source code, but in my point of view everything seems to be correct..
So I am wondering, why this fails :(
I Would really appreciate for further insights.

Thank you in advance!! :)

[dmmiller612]

Interesting. I am in the process of moving, so I might be a little slow on getting to this. But I will take a look when I get a chance.

[jiseungshin]

Thank you for the answer :)
I have been experimenting a lot, and we fixed the incidence. I think the framework itself is cool, the problem was with our proxy.. We need to set the proxy, but it was confused with connection for the weights transfer between driver /executors, so a lot of environment parameters have been set, now I think it runs..
I will keep you updated!