pysm does not use local healpy weights.

[iparask]

Hello team,

I'm running the following example script on a compute node that does not have internet access:

import os
os.environ['PYSM_LOCAL_DATA']='/scratch/gpfs/ip8725/pysm-data/'

import numpy as np
import healpy as hp
import pysm3
import pysm3.units as u
nside = 64 
nu = 90
unit = u.uK_CMB
np.random.seed(1000)
cmps = ["c1"] # sky components, currently CMB only

sky = pysm3.Sky(nside=nside, preset_strings=cmps, output_unit="uK_CMB")

mapp = sky.get_emission(nu * u.GHz)

map = pysm3.apply_smoothing_and_coord_transform(mapp, rot=hp.Rotator(coord="GC"))

and I am getting an error that it cannot download the weights from the internet.

Looking at your code I found that you are not using the datapath parameter (here) that healpy map2alm is offering.

Can you add that as a parameter where ever is needed or do you prefer a PR for this issue?

Thank you,
Ioannis

cc: @susannaaz

[zonca]

Can you please paste the error message?
Can you reproduce the error with healpy only?

[iparask]

Can you please paste the error message?

---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:1348, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
   1347 try:
-> 1348     h.request(req.get_method(), req.selector, req.data, headers,
   1349               encode_chunked=req.has_header('Transfer-encoding'))
   1350 except OSError as err: # timeout error

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1283, in HTTPConnection.request(self, method, url, body, headers, encode_chunked)
   1282 """Send a complete request to the server."""
-> 1283 self._send_request(method, url, body, headers, encode_chunked)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1329, in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked)
   1328     body = _encode(body, 'body')
-> 1329 self.endheaders(body, encode_chunked=encode_chunked)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1278, in HTTPConnection.endheaders(self, message_body, encode_chunked)
   1277     raise CannotSendHeader()
-> 1278 self._send_output(message_body, encode_chunked=encode_chunked)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1038, in HTTPConnection._send_output(self, message_body, encode_chunked)
   1037 del self._buffer[:]
-> 1038 self.send(msg)
   1040 if message_body is not None:
   1041 
   1042     # create a consistent interface to message_body

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:976, in HTTPConnection.send(self, data)
    975 if self.auto_open:
--> 976     self.connect()
    977 else:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1448, in HTTPSConnection.connect(self)
   1446 "Connect to a host on a given (SSL) port."
-> 1448 super().connect()
   1450 if self._tunnel_host:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:942, in HTTPConnection.connect(self)
    941 sys.audit("http.client.connect", self, self.host, self.port)
--> 942 self.sock = self._create_connection(
    943     (self.host,self.port), self.timeout, self.source_address)
    944 # Might fail in OSs that don't implement TCP_NODELAY

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/socket.py:824, in create_connection(address, timeout, source_address)
    823 err = None
--> 824 for res in getaddrinfo(host, port, 0, SOCK_STREAM):
    825     af, socktype, proto, canonname, sa = res

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/socket.py:955, in getaddrinfo(host, port, family, type, proto, flags)
    954 addrlist = []
--> 955 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    956     af, socktype, proto, canonname, sa = res

gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1500, in download_file(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure)
   1499 try:
-> 1500     f_name = _download_file_from_source(
   1501         source_url,
   1502         timeout=timeout,
   1503         show_progress=show_progress,
   1504         cache=cache,
   1505         remote_url=remote_url,
   1506         pkgname=pkgname,
   1507         http_headers=http_headers,
   1508         ssl_context=ssl_context,
   1509         allow_insecure=allow_insecure,
   1510     )
   1511     # Success!

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1284, in _download_file_from_source(source_url, show_progress, timeout, remote_url, cache, pkgname, http_headers, ftp_tls, ssl_context, allow_insecure)
   1282             raise
-> 1284 with _try_url_open(
   1285     source_url,
   1286     timeout=timeout,
   1287     http_headers=http_headers,
   1288     ftp_tls=ftp_tls,
   1289     ssl_context=ssl_context,
   1290     allow_insecure=allow_insecure,
   1291 ) as remote:
   1292     info = remote.info()

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1202, in _try_url_open(source_url, timeout, http_headers, ftp_tls, ssl_context, allow_insecure)
   1201 try:
-> 1202     return urlopener.open(req, timeout=timeout)
   1203 except urllib.error.URLError as exc:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:519, in OpenerDirector.open(self, fullurl, data, timeout)
    518 sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 519 response = self._open(req, data)
    521 # post-process response

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:536, in OpenerDirector._open(self, req, data)
    535 protocol = req.type
--> 536 result = self._call_chain(self.handle_open, protocol, protocol +
    537                           '_open', req)
    538 if result:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
    495 func = getattr(handler, meth_name)
--> 496 result = func(*args)
    497 if result is not None:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:1391, in HTTPSHandler.https_open(self, req)
   1390 def https_open(self, req):
-> 1391     return self.do_open(http.client.HTTPSConnection, req,
   1392         context=self._context, check_hostname=self._check_hostname)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:1351, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
   1350 except OSError as err: # timeout error
-> 1351     raise URLError(err)
   1352 r = h.getresponse()

URLError: <urlopen error [Errno -2] Name or service not known>

The above exception was the direct cause of the following exception:

URLError                                  Traceback (most recent call last)
Cell In[1], line 18
     14 sky = pysm3.Sky(nside=nside, preset_strings=cmps, output_unit="uK_CMB")
     16 mapp = sky.get_emission(nu * u.GHz)
---> 18 map = pysm3.apply_smoothing_and_coord_transform(mapp, rot=hp.Rotator(coord="GC"))

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/pysm3/models/template.py:131, in apply_smoothing_and_coord_transform(input_map, fwhm, rot, lmax, map_dist)
    129 if map_dist is None:
    130     nside = hp.get_nside(input_map)
--> 131     alm = hp.map2alm(
    132         input_map,
    133         lmax=lmax,
    134         use_pixel_weights=True if nside > 16 else False,
    135     )
    136     if fwhm is not None:
    137         hp.smoothalm(
    138             alm, fwhm=fwhm.to_value(u.rad), inplace=True, pol=True
    139         )

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/decorators.py:604, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    601             msg += f"\n        Use {alternative} instead."
    602         warnings.warn(msg, warning_type, stacklevel=2)
--> 604 return function(*args, **kwargs)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/healpy/sphtfunc.py:269, in map2alm(maps, lmax, mmax, iter, pol, use_weights, datapath, gal_cut, use_pixel_weights, verbose)
    265     if pixel_weights_filename is None:
    266         with data.conf.set_temp("dataurl", DATAURL), data.conf.set_temp(
    267             "dataurl_mirror", DATAURL_MIRROR
    268         ), data.conf.set_temp("remote_timeout", 30):
--> 269             pixel_weights_filename = data.get_pkg_data_filename(
    270                 filename, package="healpy"
    271             )
    273 if pol or info in (0, 1):
    274     alms = _sphtools.map2alm(
    275         maps,
    276         niter=iter,
   (...)
    282         pixel_weights_filename=pixel_weights_filename,
    283     )

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:739, in get_pkg_data_filename(data_name, package, show_progress, remote_timeout)
    737     return datafn
    738 else:  # remote file
--> 739     return download_file(
    740         conf.dataurl + data_name,
    741         cache=True,
    742         show_progress=show_progress,
    743         timeout=remote_timeout,
    744         sources=[conf.dataurl + data_name, conf.dataurl_mirror + data_name],
    745     )

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1534, in download_file(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure)
   1532         raise errors[sources[0]]
   1533     else:
-> 1534         raise urllib.error.URLError(
   1535             f"Unable to open any source! Exceptions were {errors}"
   1536         ) from errors[sources[0]]
   1538 if cache:
   1539     try:

URLError: <urlopen error Unable to open any source! Exceptions were {'https://healpy.github.io/healpy-data/full_weights/healpix_full_weights_nside_0064.fits': URLError(gaierror(-2, 'Name or service not known')), 'https://github.com/healpy/healpy-data/releases/download/full_weights/healpix_full_weights_nside_0064.fits': URLError(gaierror(-2, 'Name or service not known'))}>

Can you reproduce the error with healpy only?

I am using Della HPC at Princeton University and the compute nodes do not have internet access. For example:

[ip8725@della8 ~]$ ssh della-h17n4
Last login: Tue Nov  7 16:27:52 2023
[ip8725@della-h17n4 ~]$ date
Tue Nov  7 16:32:36 EST 2023
[ip8725@della-h17n4 ~]$ ping google.com
ping: google.com: Name or service not known
[ip8725@della-h17n4 ~]$

I have not tried it with healpy specifically, but it does not really matter.

[zonca]

could you try this? Don't have time to test myself.

download the weights following the instructions in:

https://healpy.readthedocs.io/en/latest/generated/healpy.sphtfunc.map2alm.html

and copy them to the data subfolder of the installed healpy package, for example for me is:

/home/zonca/.virtualenvs/pysm/lib/python3.8/site-packages/healpy/data

[iparask]

Unfortunately, it does not work.

[zonca]

Ok then please go ahead and submit a Pull Request thanks