handling arrays with missing values (with possible solution)

[Alexander-Barth]

I would like to plot (with PyPlot) an array containing missing values:

using PyPlot, Missings
d = [missing 1; 2 3] 
pcolor(d)

Unfortunately, I get the following error with PyCall v1.18.5, PyPlot v2.6.3 and Julia v1.0.1:

PyError ($(Expr(:escape, :(ccall(#= /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError('unorderable types: PyCall.jlwrap() <= int()',)
  File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 2761, in pcolor
    **kwargs)
  File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/__init__.py", line 1810, in inner
    return func(ax, *args, **kwargs)
  File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/axes/_axes.py", line 5785, in pcolor
    collection.autoscale_None()
  File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/cm.py", line 375, in autoscale_None
    self.norm.autoscale_None(self._A)
  File "/home/abarth/.local/lib/python3.5/site-packages/matplotlib/colors.py", line 988, in autoscale_None
    self.vmin = A.min()
  File "/home/abarth/.local/lib/python3.5/site-packages/numpy/core/_methods.py", line 32, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial)


Stacktrace:
 [1] pyerr_check at /home/abarth/.julia/packages/PyCall/0jMpb/src/exception.jl:60 [inlined]
 [2] pyerr_check at /home/abarth/.julia/packages/PyCall/0jMpb/src/exception.jl:64 [inlined]
 [3] macro expansion at /home/abarth/.julia/packages/PyCall/0jMpb/src/exception.jl:84 [inlined]
 [4] __pycall!(::PyCall.PyObject, ::Ptr{PyCall.PyObject_struct}, ::PyCall.PyObject, ::Ptr{Nothing}) at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44
 [5] _pycall!(::PyCall.PyObject, ::PyCall.PyObject, ::Tuple{Array{Union{Missing, Int64},2}}, ::Int64, ::Ptr{Nothing}) at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:22
 [6] #pycall#88 at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:11 [inlined]
 [7] pycall at /home/abarth/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:86 [inlined]
 [8] #pcolor#81(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Array{Union{Missing, Int64},2}) at /home/abarth/.julia/packages/PyPlot/fZuOQ/src/PyPlot.jl:179
 [9] pcolor(::Array{Union{Missing, Int64},2}) at /home/abarth/.julia/packages/PyPlot/fZuOQ/src/PyPlot.jl:176
 [10] top-level scope at In[1]:3

However, if declare the following convertion rule in PyCall, the plotting works:

using PyCall
using PyCall: PyObject
function PyObject(a::Array{Union{T,Missing},N}) where {T,N}
  numpy_ma = pyimport("numpy")["ma"]
  pycall(numpy_ma["array"], Any, coalesce.(a,zero(T)), mask=ismissing.(a))
end
pcolor(d)

The function PyCall creates essentially a numpy masked array. All missing values are replaced by zero and a mask is given to indicate which values are missing.

Can this be added to PyCall?

[Alexander-Barth]

Would it be useful to make a pull request adding this function?

Fixing deprecated function calls, this is what I currently use:

using PyCall
using PyCall: PyObject

# allow for plotting with missing values
function PyObject(a::Array{Union{T,Missing},N}) where {T,N}
    numpy_ma = PyCall.pyimport("numpy").ma
    pycall(numpy_ma.array, Any, coalesce.(a,zero(T)), mask=ismissing.(a))
end

[deszoeke]

This is a good idea. In a simple first case, adding the PyObject method to PyPlot would be a PyPlot pull request.
Are there other places PyCall needs missing support?

This works when it is defined in the user's scope, but not inside the PyPlot module.

using PyCall
using PyCall: PyObject

# extend PyObject to maskedarray to allow for plotting with missing values
function PyObject(a::Array{Union{T,Missing},N}) where {T,N}
    numpy_ma = PyCall.pyimport("numpy").ma
    pycall(numpy_ma.array, Any, coalesce.(a,zero(T)), mask=ismissing.(a))
end

# test missing support
using PyPlot
x = [missing, 1, 2, 3, 4] 
y = [1, 2, missing, 2, 3]
plot(x, y, marker=".")