slow_control database connection problems

[lucrlom]

file hax/hax/slow_control.py L176.

In the for loop often the variable entry is not a dict type variable. It's expected a kind of ntupla where the entry "timestampseconds" (datetime.utcfromtimestamp(entry['timestampseconds'])) is supposed to be present but very often this is not true.
I don't understand the reasons (some errors in the sc database entries?) I fix this problem by insert in the for loop a check on the entry variable type:
if not isinstance(entry,dict):
continue

This fix the problem, but it's not the only one.
Several time when I try to process the dataset with pax_v6.5.0, cax by means of this hax function try to connect to the slow_control database and only after several attempts (40-50 times) is able to connect and read the Voltage values of each PMT in the AddGains function.

After few tentatives the cax starts to process the run without error messages.

[pdeperio]

@mcecilio Is this a throttling issue?

[mcecilio]

After looking at the logs I concluded that the problem is not the throttling.
Can you send-me the the https request that is failing? (variable and time-stamp) @pdeperio @lucrlom
On the logs there is nothing interesting, only a few bad request from someone in the LNGS wifi (172.17.131.33)
PS: A new Web Service will be available soon:
https://xecluster.lngs.infn.it/dokuwiki/doku.php?id=xenon:xenon1t:slowcontrol:webservicenew&#getscdata

[lucrlom]

Ciao Miguel
in this log you can find some print of the entry variable and some other prints defined in the /home/tunnell/hax_test_fl/hax/slow_control.py file (now have been commented)
/project2/lgrandi/xenon1t/cax/7731_v6.5.0/7731_v6.5.0_24737531.log
the main error happen when
the output of r variable is <Response [502]> (https://goo.gl/FkhHBp) and then we receive a 502 error:
requests.exceptions.HTTPError: 502 Server Error: Proxy Error for url: https://xenon1t-daq.lngs.infn.it/slowcontrol/GetSCData?QueryType=lab&username=slowcontrolwebserver&name=XE1T.CTPC.BOARD06.CHAN005.VMON&EndDateUnix=1489058213&StartDateUnix=1489054609&api_key=ssn1sslkvdhittoywjk88w9cmbxf4fmprn68r8hm