Multistage programming #618

kehemo · 2024-04-17T05:33:55Z

No description provided.

kehemo · 2024-04-17T05:46:14Z

Python is the metalanguage and the Exo base language has not been changed in any breaking way

Python unquote blocks are evaluated using exec. Each unquote behaves like a new scope in python. Both unquote and quote blocks have parent scopes that look at the nearest enclosing block of the same type. For example:

@proc
def foo():
    a: i8
    with unquote:
        b = 1
        with quote:
            a = 1
            with unquote:
                print(b)

When this procedure is evaluated, it will print "1", and then result in a procedure that is equivalent to:

@proc
def foo():
    a: i8
    a = 1

codecov-commenter · 2024-04-17T05:47:39Z

Codecov Report

Attention: Patch coverage is 98.44560% with 6 lines in your changes missing coverage. Please review.

Project coverage is 88.23%. Comparing base (d48aa36) to head (e09d7eb).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/exo/frontend/pyparser.py	97.62%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #618      +/-   ##
==========================================
+ Coverage   88.07%   88.23%   +0.15%     
==========================================
  Files          92       93       +1     
  Lines       21303    21625     +322     
==========================================
+ Hits        18763    19080     +317     
- Misses       2540     2545       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kehemo · 2024-04-17T05:50:16Z

Some relatively minor things are still broken:

I make the assumption that free variables in Python cannot be mutated. This is not true in the full semantics of Python due to the nonlocal keyword.
I do not check the AST when mangling some identifiers that I splice into the code. I only check variables from enclosing scope in that case
I allow users to write return statements inside unquote blocks, which kinda doesn't make sense

kehemo · 2024-10-30T04:44:43Z

This branch is only a subset of the desired features for metaprogramming. In future PRs, consider:

adding an API for reflection on Exo statement objects and expressions
adding macros (functions outside of exo procs that can be used in unquoted blocks)
adding syntax that lets users create an Exo identifier and unquote it in any position an Exo identifier would appear

yamaguchi1024 · 2024-10-30T13:29:27Z

Thanks Kenneth! I will review this PR more carefully, but meanwhile can you write a documentation for metaprogramming features and limitations? (see #729 for sample documentation for other features).

yamaguchi1024 · 2024-10-30T18:05:52Z

tests/test_typecheck.py

        y: f32
        y = sin(x)


 def test_sin2():
    @proc
-    def sin(x: f32):
+    def sin_proc(x: f32):


Was this change necessary? Is it because the procedure name is conflicting with the extern name?
I think procedure definitions should be able to overwrite whatever other global and local variables out there

The change is necessary because the sin function call inside the proc is parsed as an implicit unquote in the metalanguage, which means it is looked up like a normal python variable named sin would be looked up at that point. But, if the proc is named sin, it is registered as a local variable in the scope which shadows the sin extern that is imported as a global variable.

Hmmmm I see what you mean but this seems quite cryptic.... It's way too opaque to users... How bad is it going to be if we don't support implicit unquotes?

I think the code becomes pretty cluttered if there are no implicit unquotes because you will have to put braces around every variable you reference from the metalanguage.

Also, I think it should be expected that if you name a proc "sin" it will shadow any other "sin" inside its scope. In most other languages, you can refer to the function you are currently inside by its name (e.g. recursion), so it doesn't seem very surprising.

As a side note, if you try compiling this exo:

@proc def sin(x: f32): x = sin(x)

the result is:

#include <math.h> // sin( // x : f32 @DRAM // ) void sin( void *ctxt, float* x ) { *x = sin((float)*x); }

which is actually invalid C because the sin inside the function will refer to the proc sin not the function from math.h.

I see okay, this makes sense. Can you add a test case of declaring a proc named sin that shadows the extern sin, and call the proc sin from another procedure foo, and see if that works as expected?

yamaguchi1024 · 2024-10-30T18:10:25Z

tests/test_metaprogramming.py

+            with exo:
+                a = {bar(~{a + 1})}
+
+    c_file, _ = compile_procs_to_strings([foo], "test.h")


Why are you compiling the proc into C for every test? I think you should also sanity-check the ExoIR printing.
Can modify the asserts to sth like:

assert str(foo) + c_file == golden

Also, you know that you can update all the golden tests by running pytest tests/test_metaprogramming.py --update-golden , right?

yamaguchi1024 · 2024-10-30T18:54:06Z

src/exo/frontend/pyparser.py

@@ -15,6 +15,9 @@
 from ..core.prelude import *
 from ..core.extern import Extern

+from typing import Any, Callable, Union, NoReturn, Optional
+import copy
+from dataclasses import dataclass


What are those imports? Are they necessary? If so, you should add them as dependencies to requirements.txt

The imports are parts of the python standard library just to give some type annotations and reduce some clutter

yamaguchi1024 · 2024-10-30T18:55:08Z

src/exo/frontend/pyparser.py

@@ -35,54 +38,106 @@ def str_to_mem(name):
    return getattr(sys.modules[__name__], name)


+@dataclass
+class SourceInfo:


Why did you need a wrapper for SrcInfo?

I decided to make a wrapper because before we just passed a getsrcinfo function into Parser, which I thought was confusing because then you don't know where the behavior of getsrcinfo is defined. I think it is neater if it is wrapped in an object with a defined get_src_info method that you can jump to.

yamaguchi1024 · 2024-10-30T18:56:33Z

src/exo/frontend/pyparser.py

+                        srcinfo = self.getsrcinfo(e)
+                    else:
+                        srcinfo = self.getsrcinfo(node)
+                if isinstance(e, pyast.Slice):


Can you add tests to cover the lines Codecov is reporting?

yamaguchi1024 · 2024-10-30T19:07:05Z

src/exo/frontend/pyparser.py

+
+
+@dataclass
+class QuoteReplacer(pyast.NodeTransformer):


You should definitely add docstrings to each Class and functions you defined

yamaguchi1024 · 2024-11-06T18:29:40Z

src/exo/frontend/pyparser.py

+                if isinstance(unquoted, str) and unquoted in _prim_types:
+                    return _prim_types[unquoted]
+                else:
+                    self.err(node, "Unquote computation did not yield valid type")


We need else: on if len(unquote_eval_result) == 1: and raise error

yamaguchi1024 · 2024-11-06T18:38:05Z

src/exo/frontend/pyparser.py

+            )
+        else:
+            return tuple()
+
    def eval_expr(self, expr):


Can you add tests for

mems=[DRAM, ...] x : f32 @ mems[0]

kehemo · 2025-01-15T19:45:46Z

admittedly the docs could use better motivated examples, but I found it difficult to include any without overly complicating what should also serve as a reference that is easily skimmable. I think the first limitation (lack of quote outside of exo procs) also holds metaprogramming back by a lot more than i originally anticipated, but we should try to merge the existing features first and then hopefully reduce the limitations later.

I considered documenting how metaprogramming works internally in pyparser, but besides my existing comments, it doesn't really seem like there is a space for a more in-depth explanation.

…ules

kehemo · 2025-02-03T11:37:41Z

Implicit quote/unquote lookup now progressively looks up enclosing scopes without discerning whether it is a Python or Exo scope. Thus, Python variables can shadow Exo variables if the enclosing Python scope is closer to the reference.

I found that this change is easier to generalize if I also allow implicit unquotes on the left hand side of assignments. For example, in this program:

@proc
def foo(a: i32):
  with python:
    with exo:
      a += 1

a is not defined in the with exo: scope, but rather an enclosing scope. So, with the scoping changes, it is more natural to lookup a by deferring to the enclosing with python: scope, which then finally looks up the name in the top scope. However, by passing through the with python: scope, this name lookup will return a Python object that has to be unquoted. So, I created a new class of object code called ExoSymbol, which can be unquoted as both an expression and a left hand side variable. Implicit quotes will always return an ExoSymbol so something like this is also now possible:

@proc
def foo(a: i32, b: i32):
    with python:
        syms = [a, b]
        for sym in syms:
            with exo:
                sym += 1

kehemo marked this pull request as draft April 17, 2024 05:34

kehemo force-pushed the metaprogramming branch from 571006e to cf2593a Compare October 29, 2024 15:33

kehemo marked this pull request as ready for review October 30, 2024 04:45

kehemo requested a review from yamaguchi1024 October 30, 2024 04:45

yamaguchi1024 reviewed Oct 30, 2024

View reviewed changes

kehemo force-pushed the metaprogramming branch from 3082687 to 00b0d51 Compare November 6, 2024 16:46

yamaguchi1024 reviewed Nov 6, 2024

View reviewed changes

kehemo added 12 commits January 14, 2025 15:39

Finish working prototype of multistage programming

ec862cd

Add some tests

2119ef5

Fix namespace overlap issue

3d07c52

Add better syntax for metaprogramming

9bef8fb

Add new metaprogramming tests

8319306

Fix tests

0c8b758

Fix scoping issues

52ebf87

Fix quote statement placement

26868b8

Increase code coverage

151d26e

Better local extern test

6c10524

Update tests to include ir

72cc9ce

Document helpers used by metalanguage

89a6281

kehemo added 2 commits January 15, 2025 11:39

write doc

44e59a5

add to readme

91c62b2

kehemo force-pushed the metaprogramming branch from 826bd6d to 91c62b2 Compare January 15, 2025 19:42

Clarify unquoting use cases + change implicit quote/unquote scoping r…

e09d7eb

…ules

yamaguchi1024 approved these changes Feb 4, 2025

View reviewed changes

kehemo merged commit 4bd40ca into main Feb 4, 2025
9 checks passed

kehemo deleted the metaprogramming branch February 4, 2025 16:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multistage programming #618

Multistage programming #618

kehemo commented Apr 17, 2024

kehemo commented Apr 17, 2024

codecov-commenter commented Apr 17, 2024 •

edited by codecov bot

Loading

kehemo commented Apr 17, 2024 •

edited

Loading

kehemo commented Oct 30, 2024

yamaguchi1024 commented Oct 30, 2024

yamaguchi1024 Oct 30, 2024

kehemo Oct 30, 2024

yamaguchi1024 Nov 1, 2024

kehemo Nov 1, 2024

kehemo Nov 1, 2024

yamaguchi1024 Nov 5, 2024

kehemo Nov 6, 2024

yamaguchi1024 Oct 30, 2024

yamaguchi1024 Oct 30, 2024

kehemo Nov 6, 2024

yamaguchi1024 Oct 30, 2024

kehemo Oct 30, 2024

yamaguchi1024 Oct 30, 2024

kehemo Oct 30, 2024

yamaguchi1024 Oct 30, 2024

yamaguchi1024 Oct 30, 2024

yamaguchi1024 Nov 6, 2024

yamaguchi1024 Nov 6, 2024

kehemo commented Jan 15, 2025

kehemo commented Feb 3, 2025

Multistage programming #618

Multistage programming #618

Conversation

kehemo commented Apr 17, 2024

kehemo commented Apr 17, 2024

codecov-commenter commented Apr 17, 2024 • edited by codecov bot Loading

Codecov Report

kehemo commented Apr 17, 2024 • edited Loading

kehemo commented Oct 30, 2024

yamaguchi1024 commented Oct 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kehemo commented Jan 15, 2025

kehemo commented Feb 3, 2025

codecov-commenter commented Apr 17, 2024 •

edited by codecov bot

Loading

kehemo commented Apr 17, 2024 •

edited

Loading