WIP: Experiment to use pytypes to add support for python type hints #69

mariusvniekerk · 2017-12-27T19:08:38Z

Most of the actual hard bits are delegated to pytypes for determining if types match expected instances.

mariusvniekerk · 2017-12-27T19:09:36Z

This is probably not going to work on py26 and pypy for now

mrocklin · 2017-12-27T19:14:36Z

This seems cool to me. It would be good to check how this will affect performance.

cc @llllllllll

mariusvniekerk · 2017-12-27T19:27:19Z

Do we have a more extensive suite we can benchmark for performance?

mrocklin · 2017-12-27T19:28:08Z

Nope. There is a very small script in bench/

mariusvniekerk · 2017-12-27T19:29:57Z

might be worth using something like pytest-benchmark if we want to be able to track performance over time

mrocklin · 2017-12-27T19:31:08Z

I agree that that would be valuable. I am unlikely to set this up personally, but would welcome such an addition.

…

On Wed, Dec 27, 2017 at 11:29 AM, Marius van Niekerk < ***@***.***> wrote: might be worth using something like pytest-benchmark if we want to be able to track performance over time — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszBQyq9HRHzPX8a755O4NNZ0i_3nBks5tEpq2gaJpZM4RNvNY> .

mariusvniekerk · 2017-12-27T20:01:50Z

One area that might cause some issues is

In [1]: import pytypes

In [2]: issubclass(int, float)
Out[2]: False

In [3]: pytypes.is_subtype(int, float)
Out[3]: True

mrocklin · 2017-12-27T20:06:04Z

To be clear, pytypes doesn't consider that a bug?

mariusvniekerk · 2017-12-27T20:56:03Z

Opened Stewori/pytypes#26

mariusvniekerk · 2017-12-27T21:09:01Z

Actually looks like it is

https://github.com/Stewori/pytypes/blob/0f454aec2cc2ba92efa7dd3aed2215a2cc986394/pytypes/__init__.py#L163-L168
https://www.python.org/dev/peps/pep-3141/

mariusvniekerk · 2017-12-27T21:48:46Z

Failures are from

py26, py32 lacks the typing module

Most of the actual hard bits are delegated to pytypes for determining if types match expected instances.

mariusvniekerk · 2018-02-02T16:36:42Z

@llllllllll @mrocklin So i think this is ready for review again.

Test failures on py37 due to some pytypes incompatibility

llllllllll · 2018-02-02T21:58:10Z

multipledispatch/utils.py

+        if pytypes.is_Union(L[0]):
+           rest = expand_tuples(L[1:])
+           return [(item,) + t for t in rest for item in pytypes.get_Union_params(L[0])]
+        elif not pytypes.is_of_type(L[0], tuple):


somehow this doesn't actually get hit when L[0] is not a tuple:

(Pdb) p L[0] <type 'numpy.dtype'> (Pdb) p pytypes.is_of_type(L[0], tuple) True

This breaks importing datashape.

The root cause of this is actually more concerning than this one bug. Why doesn't pytypes.is_of_type work correctly for type objects?

llllllllll · 2018-02-02T22:10:10Z

I was trying to profile a real world application using this branch; however, the
bug I commented on blocked me. I added a small patch to continue:

diff --git a/multipledispatch/utils.py b/multipledispatch/utils.py
index 816c7a4..8701756 100644
--- a/multipledispatch/utils.py
+++ b/multipledispatch/utils.py
@@ -32,6 +32,9 @@ def expand_tuples(L):
         elif not pytypes.is_of_type(L[0], tuple):
             rest = expand_tuples(L[1:])
             return [(L[0],) + t for t in rest]
+        elif not isinstance(L[0], tuple):
+            rest = expand_tuples(L[1:])
+            return [(L[0],) + t for t in rest]
         else:
             rest = expand_tuples(L[1:])
             return [(item,) + t for t in rest for item in L[0]]

With this patch, I ran out of call stack space. I noticed that there is
recursion cycle in trhe trace:

  File "/home/joe/projects/python/multipledispatch/multipledispatch/dispatcher.py", line 191, in __call__
    types = tuple([pytypes.deep_type(arg) for arg in args])
  File "/home/joe/.virtualenvs/qexec/lib/python2.7/site-packages/pytypes/type_util.py", line 281, in deep_type
    return _deep_type(obj, [], depth, max_sample)
  File "/home/joe/.virtualenvs/qexec/lib/python2.7/site-packages/pytypes/type_util.py", line 383, in _deep_type
    elif _issubclass_2(res, Container, None, None) and len(obj) == 0:
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/interactive.py", line 428, in table_length
    return int(expr.count())
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/reductions.py", line 52, in __new__
    axis = _normalize_axis(axis, _child)
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/reductions.py", line 18, in _normalize_axis
    axis = tuple(range(child.ndim))
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/expressions.py", line 241, in __getattr__
    result = func(self)
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/expressions.py", line 955, in ndim
    return len(shape(expr))
  File "/home/joe/projects/python/multipledispatch/multipledispatch/dispatcher.py", line 191, in __call__
    types = tuple([pytypes.deep_type(arg) for arg in args])
  File "/home/joe/.virtualenvs/qexec/lib/python2.7/site-packages/pytypes/type_util.py", line 281, in deep_type
    return _deep_type(obj, [], depth, max_sample)
  File "/home/joe/.virtualenvs/qexec/lib/python2.7/site-packages/pytypes/type_util.py", line 383, in _deep_type
    elif _issubclass_2(res, Container, None, None) and len(obj) == 0:
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/interactive.py", line 428, in table_length
    return int(expr.count())
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/reductions.py", line 52, in __new__
    axis = _normalize_axis(axis, _child)
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/reductions.py", line 18, in _normalize_axis
    axis = tuple(range(child.ndim))
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/expressions.py", line 241, in __getattr__
    result = func(self)
  File "/home/joe/.virtualenvs/qexec/src/blaze/blaze/expr/expressions.py", line 955, in ndim
    return len(shape(expr))
  File "/home/joe/projects/python/multipledispatch/multipledispatch/dispatcher.py", line 191, in __call__
    types = tuple([pytypes.deep_type(arg) for arg in args])
  ...

This cycle occurs because we go through a dispatch to compute len, and
now dispatch calls len in _deep_type. I don't really think that it is a
bug in blaze to use a multiply dispatched function in the __len__, so I am not
really sure that this change will work.

mariusvniekerk · 2018-02-03T00:51:33Z

So I haven't configured pytypes to do a subset for deep type. That might help. The recursion might still be a problem though.

mariusvniekerk · 2018-02-26T21:25:37Z

@llllllllll Can you point me to a test that i can run using this branch to see if i can correct the behavior?

llllllllll · 2018-02-26T22:46:33Z

I don't have time to manually distil a minimal failure case; but the blaze test suite should be a very comprehensive use of multipledispatch.

mariusvniekerk · 2018-02-27T15:00:17Z

I've been working a bit on a branch a bit further along that adds in diagonal dispatch etc, but that requires pretty aggressive use of the typing system.

Testing this with blaze (some constrained deps required to make blaze work).

conda create -n blz python=3.6 'dask<0.13' pandas=0.19
source activate blz
pytest --pyargs blaze

This passes all but 1 of the tests, which might just be a dep mismatch somewhere.

Looking at this again though, and after trying to dig through the blaze test suite, it might be better to have a different @dispatch function / Dispatcher that supports ONLY pep484 style type annotations and works only in py3.

Alternatively we could add a parameter to dispatcher for pep484 mode vs original. blaze / odo could continue to use original annotations easily without any of the complex changes.

cc @mrocklin. @llllllllll

mariusvniekerk · 2018-02-27T16:30:21Z

test failures are py37 (some weirdness) and py34 a strange flake

mariusvniekerk mentioned this pull request Jan 12, 2018

Switch testing to pytest add benchmark test #70

Merged

mariusvniekerk added 3 commits January 26, 2018 15:22

Experiment to use pytypes to add support for python type hints

1477cfe

Most of the actual hard bits are delegated to pytypes for determining if types match expected instances.

Change test to pass with pytypes for now

8b0fa6b

Change doctest to work in py2/3 due to difference in reprs

184df19

mariusvniekerk force-pushed the pytypes branch from f8cfd85 to 184df19 Compare January 26, 2018 20:22

mariusvniekerk added 2 commits January 26, 2018 15:25

Remove python versions that dont support

bbbb292

Work towards supporting diagonal dispatch

5a8e401

mariusvniekerk closed this Feb 2, 2018

mariusvniekerk reopened this Feb 2, 2018

llllllllll reviewed Feb 2, 2018

View reviewed changes

This was referenced Feb 25, 2018

Hooks for XArray operations pydata/xarray#1938

Open

Dispatching to list subtypes #72

Open

mariusvniekerk added 2 commits February 26, 2018 17:52

Merge remote-tracking branch 'origin/master' into diagonal

c0b4d42

Cleanup behavior for diagonal dispatch

0cd6265

mariusvniekerk added 3 commits February 26, 2018 20:43

Hacky: More type bits

e72819c

Legacy arg support

a9e5d5f

Cleanup

5a4ea58

mariusvniekerk added 3 commits February 27, 2018 11:01

Simplify need for making silly fake functions for annotations

2ce1c9e

py27

cbbf0be

py27

a558c0f

francesco-ballarin mentioned this pull request Mar 30, 2018

Upstream integration of some new features #78

Closed

12 tasks

mrocklin mentioned this pull request May 8, 2018

better support for metaclass #84

Open

eb8680 mentioned this pull request Aug 17, 2020

Make interpreters compatible with Python type-checking pyro-ppl/funsor#355

Open

eb8680 mentioned this pull request Jan 20, 2021

Use typing and pytypes for type checking and multiple dispatch pyro-ppl/funsor#434

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Experiment to use pytypes to add support for python type hints #69

WIP: Experiment to use pytypes to add support for python type hints #69

mariusvniekerk commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017 via email

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017 •

edited

Loading

mariusvniekerk commented Feb 2, 2018

llllllllll Feb 2, 2018

llllllllll Mar 5, 2018

llllllllll commented Feb 2, 2018

mariusvniekerk commented Feb 3, 2018

mariusvniekerk commented Feb 26, 2018

llllllllll commented Feb 26, 2018

mariusvniekerk commented Feb 27, 2018

mariusvniekerk commented Feb 27, 2018

WIP: Experiment to use pytypes to add support for python type hints #69

Are you sure you want to change the base?

WIP: Experiment to use pytypes to add support for python type hints #69

Conversation

mariusvniekerk commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017 via email

mariusvniekerk commented Dec 27, 2017

mrocklin commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017

mariusvniekerk commented Dec 27, 2017 • edited Loading

mariusvniekerk commented Feb 2, 2018

llllllllll Feb 2, 2018

Choose a reason for hiding this comment

llllllllll Mar 5, 2018

Choose a reason for hiding this comment

llllllllll commented Feb 2, 2018

mariusvniekerk commented Feb 3, 2018

mariusvniekerk commented Feb 26, 2018

llllllllll commented Feb 26, 2018

mariusvniekerk commented Feb 27, 2018

mariusvniekerk commented Feb 27, 2018

mariusvniekerk commented Dec 27, 2017 •

edited

Loading