Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python type inference does not work for pybind11 wrapped C++ #6349

Open
chipkent opened this issue Nov 8, 2024 · 1 comment · May be fixed by #6368
Open

Python type inference does not work for pybind11 wrapped C++ #6349

chipkent opened this issue Nov 8, 2024 · 1 comment · May be fixed by #6368
Assignees
@jmao-denver
Labels
bug Something isn't working devrel-watch DevRel team is watching python python-server-side

Comments

@chipkent
Copy link
Member

chipkent commented Nov 8, 2024

I generated a set of examples that integrate C++ into Deephaven. pybind11 was used to integrate C++ directly into python.

The code can be found at https://github.com/chipkent/deephaven-example-use-cases/tree/main/examples/cpp/deephaven-example.

There seem to be problems in both of the mechanisms I tried.

  1. In the first case below, a cast is needed to get the right column type. If the cast is not present, the UI says the column is a String.
  2. In the second case below, there is an error. Value: no signature found for builtin <built-in method price of PyCapsule object at 0xfffec56ae760>
from deephaven import empty_table
import blackscholes

v = blackscholes.price(100, 95, 0.05, 1.2, 0.4, True, False) 
print(v)
print(type(v))

t = empty_table(10).update([
    "UnderlyingPrice = 100+i",
    "Strke = 95",
    "RiskFree = 0.05",
    "YearsToExpiry = 0.6",
    "Vol = 0.4",
    "IsCall = true",
    "IsStock = false",
    "BlackScholesPythonCpp = (double) blackscholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
    "BlackScholesJavaCpp = io.deephaven.BlackScholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
])


price = blackscholes.price

tx = empty_table(10).update([
    "UnderlyingPrice = 100+i",
    "Strke = 95",
    "RiskFree = 0.05",
    "YearsToExpiry = 0.6",
    "Vol = 0.4",
    "IsCall = true",
    "IsStock = false",
    "BlackScholesPythonCpp = price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
    "BlackScholesJavaCpp = io.deephaven.BlackScholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
])

Error:

r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update operation failed. : Error in Python interpreter
Traceback (most recent call last):
  File "/build/venv/lib/python3.12/site-packages/deephaven/table.py", line 994, in update
    return Table(j_table=self.j_table.update(*formulas))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)
	at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:201)
	at io.deephaven.engine.table.impl.select.SwitchColumn.initDef(SwitchColumn.java:64)
	at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.createContext(SelectAndViewAnalyzer.java:128)
	at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$34(QueryTable.java:1527)
	at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
	at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$35(QueryTable.java:1509)
	at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3646)
	at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1508)
	at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1487)
	at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:98)
	at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
	at org.jpy.PyLib.executeCode(Native Method)
	at org.jpy.PyObject.executeCode(PyObject.java:138)
	at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
	at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
	at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
	at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
	at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
	at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
	at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
	at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
	at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
	at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
	at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
	at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
	at java.base/java.lang.Thread.run(Thread.java:829)
caused by io.deephaven.engine.table.impl.lang.QueryLanguageParser$QueryLanguageParseException: 

Having trouble with the following expression:
Full expression           : price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)
Expression having trouble : 
Exception type            : java.lang.RuntimeException
Exception message         : Error in Python interpreter:
Type: <class 'ValueError'>
Value: no signature found for builtin <built-in method price of PyCapsule object at 0xfffec56ae760>
Line: 2361
Namespace: _signature_from_builtin
File: /usr/lib/python3.12/inspect.py
Traceback (most recent call last):
  File "/build/venv/lib/python3.12/site-packages/deephaven/_udf.py", line 529, in _udf_parser
  File "/build/venv/lib/python3.12/site-packages/deephaven/_udf.py", line 496, in _parse_signature
  File "/usr/lib/python3.12/inspect.py", line 3310, in signature
  File "/usr/lib/python3.12/inspect.py", line 3054, in from_callable
  File "/usr/lib/python3.12/inspect.py", line 2571, in _signature_from_callable
  File "/usr/lib/python3.12/inspect.py", line 2361, in _signature_from_builtin


	at org.jpy.PyLib.callAndReturnObject(Native Method)
	at org.jpy.PyObject.call(PyObject.java:449)
	at io.deephaven.engine.util.PyCallableWrapperJpyImpl.prepareSignature(PyCallableWrapperJpyImpl.java:195)
	at io.deephaven.engine.util.PyCallableWrapperJpyImpl.parseSignature(PyCallableWrapperJpyImpl.java:206)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.pyCallableReturnType(QueryLanguageParser.java:2580)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:2389)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:133)
	at com.github.javaparser.ast.expr.MethodCallExpr.accept(MethodCallExpr.java:116)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:319)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:212)
	at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:240)
	at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:122)
	at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:181)
	... 32 more


Line: 996
Namespace: update
File: /build/venv/lib/python3.12/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "<string>", line 24, in <module>
  File "/build/venv/lib/python3.12/site-packages/deephaven/table.py", line 996, in update

	at org.jpy.PyLib.executeCode(Native Method)
	at org.jpy.PyObject.executeCode(PyObject.java:138)
	at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
	at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
	at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
	at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
	at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
	at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
	at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
	at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
	at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
	at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
	at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
	at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
	at java.base/java.lang.Thread.run(Thread.java:829)
@chipkent chipkent added the devrel-watch DevRel team is watching label Nov 8, 2024
@jmao-denver
Copy link
Contributor

  1. I have done some investigation and it turns out that to add signatures to functions in C extension modules isn't as one might expect. See the following.

python/cpython#123473
https://stackoverflow.com/questions/1104823/python-c-extension-method-signatures-for-documentation
https://stackoverflow.com/questions/25847035/what-are-signature-and-text-signature-used-for-in-python-3-4/25847066#25847066

  1. I have also tried creating .pyi files but they don't help runtime inspection via the inspect module.
  2. besides the use of __text_signature__, another alternative way is to wrap C extension modules (making them private) in pure Python code.
  3. numba, numpy choose to store the proprietary encoded signature in the types attribute of the function object
  4. DH UDF signature parser needs to properly handle the case where signature isn't available

@jmao-denver jmao-denver linked a pull request Nov 12, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jmao-denver jmao-denver

Labels
bug Something isn't working devrel-watch DevRel team is watching python python-server-side
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Correctly handle signature-less functions for Py UDF calls jmao-denver/deephaven-core
2 participants
@jmao-denver @chipkent