Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: python server (runtime) #726

Merged
merged 34 commits into from
Dec 5, 2023
Merged

feat: python server (runtime) #726

merged 34 commits into from
Dec 5, 2023

Conversation

WinPlay02
Copy link
Contributor

@WinPlay02 WinPlay02 commented Nov 6, 2023 •
edited by lars-reimann
Loading

Closes #806

Summary of Changes

  • when activating the extension, a runner is spawned
  • communication is done via websocket
  • when deactivating the extension, the runner is killed
  • sendMessageToPythonServer can be used to send messages
  • addMessageCallback can be used to listen for messages
  • messages are objects containing a type, an id and any data that can be serialized as JSON
  • placeholders, if assigned inside pipelines, are now saved in the runner
  • try shutting down runner process gracefully first and kill the process tree if it does not shut down in a timely fashion
  • use relative module paths for generating pipeline entry points
  • use module paths on runner to select correct module

@WinPlay02 WinPlay02 requested a review from SmiteDeluxe November 6, 2023 14:30
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2023 •
edited
Loading

🦙 MegaLinter status: ✅ SUCCESS

Descriptor Linter Files Fixed Errors Elapsed time
✅ JSON eslint-plugin-jsonc 3 0 0 1.85s
✅ JSON jsonlint 3 0 0.23s
✅ JSON npm-package-json-lint yes no 0.74s
✅ JSON prettier 3 0 0 1.13s
✅ JSON v8r 3 0 3.9s
✅ REPOSITORY git_diff yes no 0.13s
✅ TYPESCRIPT eslint 6 0 0 6.32s
✅ TYPESCRIPT prettier 6 0 0 1.17s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security

@codecov Codecov
Copy link

codecov bot commented Nov 6, 2023 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (37c5330) 100.00% compared to head (cc37d6c) 100.00%.

Additional details and impacted files 
@@            Coverage Diff            @@
##              main      #726   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           86        86           
  Lines        13595     13633   +38     
  Branches      2878      2881    +3     
=========================================
+ Hits         13595     13633   +38

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

WinPlay02 and others added 17 commits November 16, 2023 11:03
@WinPlay02
feat: actual running of pipelines
4fb1f01 
feat: use custom output channel
@WinPlay02
Merge branch 'main' of https://github.com/Safe-DS/DSL into runtime-se…
5848430 
…rver

# Conflicts:
#	packages/safe-ds-vscode/package.json
@WinPlay02
feat: use common output channel for safe-ds
ec68d51 
feat: document message types

misc: adapt to changes made to runner
@WinPlay02
Merge branch 'main' of https://github.com/Safe-DS/DSL into runtime-se…
ef3c7b8 
…rver
@WinPlay02
feat: print safe-ds stacktrace on error
ae40a37 
fix: finding next free port was broken
@WinPlay02
Merge branch 'main' of https://github.com/Safe-DS/DSL into runtime-se…
206e932 
…rver
@WinPlay02
misc: copy mappings.wasm to dist (from source-maps)
a12e64a
@WinPlay02
Merge branch 'main' into runtime-server
471d0eb
@WinPlay02
feat: use actually random execution ids
2322a33 
feat: link to source-file in stacktrace + reverse stacktrace to have the error at the top

feat: do not store execution context in global variable (would be overwritten)

docs: more docs
@WinPlay02
Merge branch 'runtime-server' of https://github.com/Safe-DS/DSL into …
cf23b4a 
…runtime-server
@WinPlay02
Merge branch 'main' of https://github.com/Safe-DS/DSL into runtime-se…
e0965ef 
…rver

# Conflicts:
#	package-lock.json
#	packages/safe-ds-vscode/package.json
@megalinter-bot
style: apply automated linter fixes
4d04f51
@WinPlay02
Merge branch 'main' into runtime-server
fed02b5
@WinPlay02
feat: save placeholders in runner
3109e86
@WinPlay02
Merge branch 'runtime-server' of https://github.com/Safe-DS/DSL into …
44dd489 
…runtime-server
@WinPlay02
feat: handle module path for program execution requests
b2fd9a2 
feat: use typed callbacks for python server messages

feat: add ability to query placeholders
@megalinter-bot
style: apply automated linter fixes
0229aed
@WinPlay02 WinPlay02 marked this pull request as ready for review November 28, 2023 14:39
lars-reimann
lars-reimann requested changes Dec 1, 2023
View reviewed changes
Copy link
Member

@lars-reimann lars-reimann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really great work. It just needs a little polish and then it's ready.

packages/safe-ds-vscode/src/extension/pythonServer.ts Outdated Show resolved Hide resolved
packages/safe-ds-vscode/src/extension/pythonServer.ts Outdated Show resolved Hide resolved
packages/safe-ds-vscode/src/extension/pythonServer.ts Outdated Show resolved Hide resolved
packages/safe-ds-lang/src/language/generation/safe-ds-python-generator.ts Show resolved Hide resolved
packages/safe-ds-lang/src/language/generation/safe-ds-python-generator.ts Outdated Show resolved Hide resolved
packages/safe-ds-vscode/src/extension/pythonServer.ts Outdated Show resolved Hide resolved
packages/safe-ds-vscode/src/extension/pythonServer.ts Outdated Show resolved Hide resolved
packages/safe-ds-vscode/src/extension/pythonServer.ts Outdated Show resolved Hide resolved
@lars-reimann
Copy link
Member

lars-reimann commented Dec 1, 2023 •
edited
Loading

An addendum to this:

After running a pipeline and closing the extension host, there are still two python.exe processes left. These should be cleaned up.

d998debe-ed12-432d-a856-1b901a27a79e.mp4

WinPlay02 and others added 3 commits December 4, 2023 14:31
@WinPlay02
misc: address review suggestions
4ec4bd9 
feat: react to configuration changes, if the runner was not successfully started

feat: try to start the runner, if a pipeline is executed and the startup before was not successful

feat: use exponential backoff when trying to connect to the runner

feat: before trying to connect to the runner, first validate that the runner would even launch (by testing the version of the runner)

misc: use error messages when appropriate
@WinPlay02
Merge branch 'runtime-server' of https://github.com/Safe-DS/DSL into …
31a3317 
…runtime-server
@megalinter-bot
style: apply automated linter fixes
23be8a3
@WinPlay02
Copy link
Contributor Author

@SmiteDeluxe The function to remove previously registered callbacks now exists (removeMessageCallback)

@WinPlay02 WinPlay02 mentioned this pull request Dec 5, 2023
Closed
lars-reimann
lars-reimann approved these changes Dec 5, 2023
View reviewed changes
Copy link
Member

@lars-reimann lars-reimann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@lars-reimann lars-reimann merged commit 78d16f1 into main Dec 5, 2023
6 checks passed
@lars-reimann lars-reimann deleted the runtime-server branch December 5, 2023 12:59
lars-reimann pushed a commit that referenced this pull request Jan 15, 2024
@semantic-release-bot
chore(release): 0.6.0 [skip ci]
6395eb6 
## [0.6.0](v0.5.1...v0.6.0) (2024-01-15)

### Features

* CLI command to run formatting ([#824](#824)) ([a74b8e0](a74b8e0)), closes [#702](#702)
* CLI command to run validation ([#820](#820)) ([7c2526d](7c2526d)), closes [#703](#703)
* Partial execution of pipelines ([#821](#821)) ([1e0d03b](1e0d03b))
* python server (runtime) ([#726](#726)) ([78d16f1](78d16f1)), closes [#806](#806)

### Bug Fixes

* **deps-dev:** Bump vite from 4.4.9 to 5.0.5 ([#808](#808)) ([c9575e7](c9575e7)), closes [#15203](https://github.com/Safe-DS/DSL/issues/15203) [#15203](https://github.com/Safe-DS/DSL/issues/15203) [#15140](https://github.com/Safe-DS/DSL/issues/15140) [#15140](https://github.com/Safe-DS/DSL/issues/15140) [#15225](https://github.com/Safe-DS/DSL/issues/15225) [#15225](https://github.com/Safe-DS/DSL/issues/15225) [#15223](https://github.com/Safe-DS/DSL/issues/15223) [#15223](https://github.com/Safe-DS/DSL/issues/15223) [#15233](https://github.com/Safe-DS/DSL/issues/15233) [#15233](https://github.com/Safe-DS/DSL/issues/15233) [#15229](https://github.com/Safe-DS/DSL/issues/15229) [#15229](https://github.com/Safe-DS/DSL/issues/15229) [#15189](https://github.com/Safe-DS/DSL/issues/15189) [#15189](https://github.com/Safe-DS/DSL/issues/15189) [#15218](https://github.com/Safe-DS/DSL/issues/15218) [#15218](https://github.com/Safe-DS/DSL/issues/15218) [#15220](https://github.com/Safe-DS/DSL/issues/15220) [#15220](https://github.com/Safe-DS/DSL/issues/15220) [#15181](https://github.com/Safe-DS/DSL/issues/15181) [#15181](https://github.com/Safe-DS/DSL/issues/15181) [#15194](https://github.com/Safe-DS/DSL/issues/15194) [#15194](https://github.com/Safe-DS/DSL/issues/15194) [#15179](https://github.com/Safe-DS/DSL/issues/15179) [#15179](https://github.com/Safe-DS/DSL/issues/15179) [#15228](https://github.com/Safe-DS/DSL/issues/15228) [#15228](https://github.com/Safe-DS/DSL/issues/15228) [#15204](https://github.com/Safe-DS/DSL/issues/15204) [#15204](https://github.com/Safe-DS/DSL/issues/15204) [#15207](https://github.com/Safe-DS/DSL/issues/15207) [#15207](https://github.com/Safe-DS/DSL/issues/15207) [#15162](https://github.com/Safe-DS/DSL/issues/15162) [#15162](https://github.com/Safe-DS/DSL/issues/15162) [#15135](https://github.com/Safe-DS/DSL/issues/15135) [#15178](https://github.com/Safe-DS/DSL/issues/15178) [#15135](https://github.com/Safe-DS/DSL/issues/15135) [#15178](https://github.com/Safe-DS/DSL/issues/15178) [#15173](https://github.com/Safe-DS/DSL/issues/15173) [#15173](https://github.com/Safe-DS/DSL/issues/15173) [#15137](https://github.com/Safe-DS/DSL/issues/15137) [#15137](https://github.com/Safe-DS/DSL/issues/15137) [#15166](https://github.com/Safe-DS/DSL/issues/15166) [#15166](https://github.com/Safe-DS/DSL/issues/15166) [#15171](https://github.com/Safe-DS/DSL/issues/15171) [#15171](https://github.com/Safe-DS/DSL/issues/15171) [#15093](https://github.com/Safe-DS/DSL/issues/15093) [#15093](https://github.com/Safe-DS/DSL/issues/15093) [#15135](https://github.com/Safe-DS/DSL/issues/15135) [#15135](https://github.com/Safe-DS/DSL/issues/15135) [#15154](https://github.com/Safe-DS/DSL/issues/15154) [#15154](https://github.com/Safe-DS/DSL/issues/15154) [#15155](https://github.com/Safe-DS/DSL/issues/15155) [#15155](https://github.com/Safe-DS/DSL/issues/15155) [#14569](https://github.com/Safe-DS/DSL/issues/14569) [#14569](https://github.com/Safe-DS/DSL/issues/14569) [#15145](https://github.com/Safe-DS/DSL/issues/15145) [#15145](https://github.com/Safe-DS/DSL/issues/15145) [#15049](https://github.com/Safe-DS/DSL/issues/15049) [#15049](https://github.com/Safe-DS/DSL/issues/15049) [#15059](https://github.com/Safe-DS/DSL/issues/15059) [#15059](https://github.com/Safe-DS/DSL/issues/15059) [#15233](https://github.com/Safe-DS/DSL/issues/15233) [#15189](https://github.com/Safe-DS/DSL/issues/15189) [#15229](https://github.com/Safe-DS/DSL/issues/15229) [#15220](https://github.com/Safe-DS/DSL/issues/15220) [#15225](https://github.com/Safe-DS/DSL/issues/15225) [#15228](https://github.com/Safe-DS/DSL/issues/15228) [#15223](https://github.com/Safe-DS/DSL/issues/15223) [#15207](https://github.com/Safe-DS/DSL/issues/15207) [#15218](https://github.com/Safe-DS/DSL/issues/15218)
* renaming of declarations that are imported under an alias ([#825](#825)) ([9f7363d](9f7363d)), closes [#635](#635)
* use correct paths and module names to correctly execute Safe-DS files with spaces ([#811](#811)) ([191ef33](191ef33)), closes [#810](#810)
* Wait for tree-kill during deactivation if child process does not respond ([#807](#807)) ([baf4a3c](baf4a3c))
@lars-reimann
Copy link
Member

🎉 This PR is included in version 0.6.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@lars-reimann lars-reimann added the released Included in a release label Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lars-reimann lars-reimann lars-reimann approved these changes

@SmiteDeluxe SmiteDeluxe Awaiting requested review from SmiteDeluxe

Assignees
No one assigned
Labels
released Included in a release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Infrequent ModuleNotFoundError when running a pipeline
3 participants
@WinPlay02 @lars-reimann @megalinter-bot