Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release bug fix for delayed closing of HTTP SSE (v0.2.7) #47

Merged
merged 58 commits into from
Oct 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
2ed6936
Merge pull request #32 from VigneshVSV/main
VigneshVSV Aug 9, 2024
3252997
IDS cam example
VigneshVSV Aug 9, 2024
c85eb1d
add git branching strategy info
VigneshVSV Aug 10, 2024
3517ec2
minor optimisations
VigneshVSV Aug 10, 2024
c9bb1b5
changed null type to string in TD schema
VigneshVSV Aug 10, 2024
31e5240
property schema can be specified with pydantic - no validation yet
VigneshVSV Aug 11, 2024
8aca6c5
labthings-fastapi license
VigneshVSV Aug 11, 2024
a8a6caa
pydantic is still optional
VigneshVSV Aug 11, 2024
27247a4
minor improvements
VigneshVSV Aug 11, 2024
867dfa2
Merge pull request #34 from VigneshVSV/main
VigneshVSV Aug 15, 2024
3794f87
update examples
VigneshVSV Aug 15, 2024
7b51af9
update doc commit
Aug 22, 2024
5fd3bfc
add sdist to build process
VigneshVSV Sep 11, 2024
c254b9f
update version to upload to anaconda
VigneshVSV Sep 11, 2024
eaa2566
Merge pull request #39 from VigneshVSV/anaconda-distribution
VigneshVSV Sep 11, 2024
dd83362
publish source distribution to pypi
VigneshVSV Sep 11, 2024
3d78a7b
merge develop to main
VigneshVSV Sep 15, 2024
fd3135a
Merge pull request #40 from VigneshVSV/main
VigneshVSV Sep 15, 2024
b317010
bug serializer mismatch for events
VigneshVSV Sep 15, 2024
d651a50
Merge pull request #41 from VigneshVSV/develop
VigneshVSV Sep 22, 2024
8938cdb
update version and edit changelog
VigneshVSV Sep 22, 2024
5b60520
update anaconda badge due to release success
VigneshVSV Sep 24, 2024
b60c2e0
update funding.yml
VigneshVSV Sep 29, 2024
c5bc46c
bug fix subthing URL
VigneshVSV Sep 29, 2024
8d05d58
fix HTTP server authority for subthings
VigneshVSV Sep 29, 2024
6ab2a04
added note on state machine description
VigneshVSV Oct 5, 2024
36c3910
update funding include thanks.dev
VigneshVSV Oct 5, 2024
a4ac660
Merge branch 'develop' into main
VigneshVSV Oct 5, 2024
21f8210
Merge pull request #43 from VigneshVSV/main
VigneshVSV Oct 5, 2024
578cc47
bug fix instance resources
VigneshVSV Oct 5, 2024
de3278c
allow reassign event dispatcher object with a different one
VigneshVSV Oct 5, 2024
83abcc2
allow reassign event dispatcher object with a different one
VigneshVSV Oct 5, 2024
0b0e0b9
auto deregister old event dispatcher
VigneshVSV Oct 5, 2024
54b91eb
update README
VigneshVSV Oct 5, 2024
8a24253
update README
VigneshVSV Oct 5, 2024
cbe83df
update README
VigneshVSV Oct 5, 2024
a5f4bcb
update with download stats
VigneshVSV Oct 5, 2024
9667baf
client side event deserialize=True option
Oct 5, 2024
bfeac06
set deserialize to True by default
VigneshVSV Oct 5, 2024
0dd95e3
update changelog
VigneshVSV Oct 5, 2024
4d6ef3d
Minor Changes and Bug Fixes
VigneshVSV Oct 5, 2024
58ab809
add open collective
VigneshVSV Oct 5, 2024
b428083
remove sponsorship info
VigneshVSV Oct 5, 2024
e9e0aa0
fast forward examples
VigneshVSV Oct 5, 2024
bb73bd3
fast forward examples
VigneshVSV Oct 5, 2024
698fccc
update readme
VigneshVSV Oct 11, 2024
b137fa4
update pyzmq version
VigneshVSV Oct 11, 2024
205a586
update pyzmq version
VigneshVSV Oct 11, 2024
3438c42
update readme
VigneshVSV Oct 12, 2024
8066ef9
update readme
VigneshVSV Oct 12, 2024
b6841c8
improve proeprty doc
VigneshVSV Oct 19, 2024
c0c1dad
update things being worked in readme
VigneshVSV Oct 19, 2024
336a311
update readme
Oct 21, 2024
74bfaff
fix non closing HTTP SSE when client disconnected
Oct 22, 2024
e367b94
update version to 0.2.7
Oct 22, 2024
836b71f
conda install info
VigneshVSV Oct 23, 2024
adffbfd
bugfix state machine input as list
VigneshVSV Oct 23, 2024
4c64ef3
fix potential clash in namepsace of msgspec json and python's own json
VigneshVSV Oct 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# FUNDING.yml

github: VigneshVSV
github: VigneshVSV
buy_me_a_coffee: vigneshvsv
thanks_dev: gh/vigneshvsv
open_collective: hololinked-dev
2 changes: 1 addition & 1 deletion .github/workflows/python-publish-pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
python -m pip install --upgrade pip
pip install build
- name: Build package
run: python -m build --wheel
run: python -m build
- name: Publish package
uses: pypa/gh-action-pypi-publish@release/v1
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-publish-testpypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
python -m pip install --upgrade pip
pip install build
- name: Build package
run: python -m build --wheel
run: python -m build
- name: Publish package
uses: pypa/gh-action-pypi-publish@release/v1
with:
Expand Down
24 changes: 22 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

✓ means ready to try

New:
- cookie auth & its specification in TD (cookie auth branch)
- image event handlers (develop branch) for streaming live video as JPEG and PNG ✓
- pydantic support for property models (develop branch) ✓
- adding custom handlers for each property, action and event to override default behaviour
- pydantic support for property models

Bug Fixes:
- composed sub`Thing`s exposed with correct URL path ✓

## [v0.2.7] - 2024-10-22

- HTTP SSE would previously remain unclosed when client abruptly disconnected (like closing a browser tab), but now it would close correctly
- retrieve unserialized data from events with `ObjectProxy` (like JPEG images) by setting `deserialize=False` in `subscribe_event()`
-

## [v0.2.6] - 2024-09-09

- bug fix events when multiple serializers are used
- events support custom HTTP handlers (not polished yet, use as last resort, not auto-added to TD)
- image event handlers for streaming live video as JPEG and PNG (not polished yet, not auto-added to TD)

## [v0.2.5] - 2024-09-09

- released to anaconda, it can take a while to turn up. A badge will be added in README when successful.

## [v0.2.4] - 2024-09-09

Expand Down
17 changes: 12 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,15 @@ All types of contributions are encouraged and valued.
> - Mention the project at local meetups/conferences and tell your friends/colleagues
> - Donate to cover the costs of maintaining it


## I Have a Question

Do feel free to reach out to me at [email protected]. I will try my very best to respond.
Do feel free to reach out to me at [email protected] or in discord. I will try my very best to respond.

Nevertheless, one may also refer the available how-to section of the [Documentation](https://hololinked.readthedocs.io/en/latest/index.html).
If the documentation is insufficient for any reason including being poorly documented, one may open a new discussion in the [Q&A](https://github.com/VigneshVSV/hololinked/discussions/categories/q-a) section of GitHub discussions.

For questions related to workings of HTTP, JSON schema, basic concepts of python like descriptors, decorators etc., it is also advisable to search the internet for answers first.
For generic questions related to web of things standards or its ideas, I recommend to join web of things [discord](https://discord.com/invite/RJNYJsEgnb) group and [community](https://www.w3.org/community/wot/) group.
For generic questions related to web of things standards or its ideas, it is recommended to join web of things [discord](https://discord.com/invite/RJNYJsEgnb) group and [community](https://www.w3.org/community/wot/) group.

If you believe your question might also be a bug, you might want to search for existing [Issues](https://github.com/VigneshVSV/hololinked/issues) that might help you.
In case you have found a suitable issue and still need clarification, you can write your question in this issue. If an issue is not found:
Expand All @@ -44,13 +43,21 @@ Otherwise, I will then take care of the issue as soon as possible.
> ### Legal Notice <!-- omit in toc -->
> When contributing to this project, you must agree that you have authored 100% of the content or that you have the necessary rights to the content. For example, you copied code from projects with MIT/BSD License. Content from GPL-related licenses may be maintained in a separate repository as an add-on.

Developers are always welcome to contribute to the code base. If you want to tackle any issues, un-existing features, let me know (at my email), I can create some open issues and features which I was never able to solve or did not have the time. You can also suggest what else can be contributed functionally or conceptually or also simply code-refactoring. The lack of issues or features in the [Issues](https://github.com/VigneshVSV/hololinked/issues) section of github does not mean the project is considered feature complete or I dont have ideas what to do next. On the contrary, there is tons of work to do.
Developers are always welcome to contribute to the code base. If you want to tackle any issues, un-existing features, let me know (at my email), I can create some open issues and features which I was never able to solve or did not have the time. You can also suggest what else can be contributed functionally, conceptually or also simply code-refactoring.

There are also repositories which can use your skills:
There are also other repositories which can use your skills:
- An [admin client](https://github.com/VigneshVSV/thing-control-panel) in react
- [Documentation](https://github.com/VigneshVSV/hololinked-docs) in sphinx which needs significant improvement in How-To's, beginner level docs which may teach people concepts of data acquisition or IoT, Docstring or API documentation of this repository itself
- [Examples](https://github.com/VigneshVSV/hololinked-examples) in nodeJS, Dashboard/PyQt GUIs or server implementations using this package. Hardware implementations of unexisting examples are also welcome, I can open a directory where people can search for code based on hardware and just download your code.

## Git Branching

A simpler model is used roughly based on [this article](https://www.bitsnbites.eu/a-stable-mainline-branching-model-for-git/) -
- main branch is where all stable developments are merged, all your branches must merge here
- main branch is merged to release branch when it is decided to created a release.
- A specific release is tagged and not created as its own branch. Instead release branch simply follows the main branch at the release time. People should clone the main branch for latest (mostly-) stable code base and release branch for released code base.
- other branches are feature or bug fix branches. A develop branch may be used to make general improvements as the package is constantly evolving, but its not a specific philosophy to use a develop branch.
- Bug fixes on releases must proceed from the tag of that release. Perhaps, even a new release can be made after fixing the bug by merging a bug fix branch to main branch.

## Attribution
This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)!
53 changes: 36 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,23 @@
### Description

`hololinked` is a beginner-friendly server side pythonic tool suited for instrumentation control and data acquisition over network, especially with HTTP. If you have a requirement to control and capture data from your hardware/instrumentation, show the data in a browser/dashboard, provide a GUI or run automated scripts, `hololinked` can help. Even for isolated applications or a small lab setup without networking concepts, one can still separate the concerns of the tools that interact with the hardware & the hardware itself.

For those that understand, this package is a ZMQ/HTTP-RPC.

[![Documentation Status](https://readthedocs.org/projects/hololinked/badge/?version=latest)](https://hololinked.readthedocs.io/en/latest/?badge=latest) [![PyPI](https://img.shields.io/pypi/v/hololinked?label=pypi%20package)](https://pypi.org/project/hololinked/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/hololinked)](https://pypistats.org/packages/hololinked) [![codecov](https://codecov.io/gh/VigneshVSV/hololinked/graph/badge.svg?token=JF1928KTFE)](https://codecov.io/gh/VigneshVSV/hololinked)
[![Documentation Status](https://readthedocs.org/projects/hololinked/badge/?version=latest)](https://hololinked.readthedocs.io/en/latest/?badge=latest) [![PyPI](https://img.shields.io/pypi/v/hololinked?label=pypi%20package)](https://pypi.org/project/hololinked/) [![Anaconda](https://anaconda.org/conda-forge/hololinked/badges/version.svg)](https://anaconda.org/conda-forge/hololinked)
[![codecov](https://codecov.io/gh/VigneshVSV/hololinked/graph/badge.svg?token=JF1928KTFE)](https://codecov.io/gh/VigneshVSV/hololinked)
<br>
[![email](https://img.shields.io/badge/email%20me-brown)](mailto:[email protected]) [![ways to contact me](https://img.shields.io/badge/ways_to_contact_me-brown)](https://hololinked.dev/contact)
<br>
[![PyPI - Downloads](https://img.shields.io/pypi/dm/hololinked?label=pypi%20downloads)](https://pypistats.org/packages/hololinked)
[![Conda Downloads](https://img.shields.io/conda/d/conda-forge/hololinked)](https://anaconda.org/conda-forge/hololinked)

### To Install

From pip - ``pip install hololinked``
From pip - ``pip install hololinked``
From conda - `conda install -c conda-forge hololinked`

Or, clone the repository (develop branch for latest codebase) and install `pip install .` / `pip install -e .`. The conda env ``hololinked.yml`` can also help to setup all dependencies.
Or, clone the repository (main branch for latest codebase) and install `pip install .` / `pip install -e .`. The conda env ``hololinked.yml`` can also help to setup all dependencies.

### Usage/Quickstart

Expand Down Expand Up @@ -77,6 +84,7 @@ class OceanOpticsSpectrometer(Thing):
super().__init__(instance_name=instance_name, serial_number=serial_number, **kwargs)

```
> There is an ongoing work to remove HTTP API from the property API and completely move them to the HTTP server

In non-expert terms, properties look like class attributes however their data containers are instantiated at object instance level by default.
For example, the `integration_time` property defined above as `Number`, whenever set/written, will be validated as a float or int, cropped to bounds and assigned as an attribute to each instance of the `OceanOpticsSpectrometer` class with an internally generated name. It is not necessary to know this internally generated name as the property value can be accessed again in any python logic, say, `print(self.integration_time)`.
Expand All @@ -103,9 +111,9 @@ class OceanOpticsSpectrometer(Thing):

```

In this case, instead of generating a data container with an internal name, the setter method is called when `integration_time` property is set/written. One might add the hardware device driver (say, supplied by the manufacturer) logic here to apply the property onto the device. In the above example, there is not a way provided by lower level library to read the value from the device, so we store it in a variable after applying it and supply the variable back to the getter method. Normally, one would also want the getter to read from the device directly.
In this case, instead of generating a data container with an internal name, the setter method is called when `integration_time` property is set/written. One might add the hardware device driver logic here (say, supplied by the manufacturer) or a protocol that talks directly to the device to apply the property onto the device. In the above example, there is not a way provided by the device driver library to read the value from the device, so we store it in a variable after applying it and supply the variable back to the getter method. Normally, one would also want the getter to read from the device directly.

Those familiar with Web of Things (WoT) terminology may note that these properties generate the property affordance schema. An example of autogenerated property affordance for `integration_time` is as follows:
Those familiar with Web of Things (WoT) terminology may note that these properties generate the property affordance. An example for `integration_time` is as follows:

```JSON
"integration_time": {
Expand All @@ -128,8 +136,8 @@ Those familiar with Web of Things (WoT) terminology may note that these properti
},
```
If you are <span style="text-decoration: underline">not familiar</span> with Web of Things or the term "property affordance", consider the above JSON as a description of
what the property represents and how to interact with it from somewhere else. Such a JSON is both human-readable, yet consumable
by a client provider to create a client object to interact with the property. For example, the Eclipse ThingWeb [node-wot](https://github.com/eclipse-thingweb/node-wot) supports this feature to produce a HTTP(s) client that can issue `readProperty("integration_time")` and `writeProperty("integration_time", 1000)` to read and write this property.
what the property represents and how to interact with it from somewhere else. Such a JSON is both human-readable, yet consumable by any application that may use the property, say a client provider to create a client object to interact with the property or a GUI application to autogenerate a suitable input field for this property.
For example, the Eclipse ThingWeb [node-wot](https://github.com/eclipse-thingweb/node-wot) supports this feature to produce a HTTP(s) client that can issue `readProperty("integration_time")` and `writeProperty("integration_time", 1000)` to read and write this property.

The URL path segment `../spectrometer/..` in href field is taken from the `instance_name` which was specified in the `__init__`.
This is a mandatory key word argument to the parent class `Thing` to generate a unique name/id for the instance. One should use URI compatible strings.
Expand Down Expand Up @@ -271,7 +279,7 @@ what the event represents and how to subscribe to it) with subprotocol SSE (HTTP

Events follow a pub-sub model with '1 publisher to N subscribers' per `Event` object, both through ZMQ and HTTP SSE.

Although the code is the very familiar & age-old RPC server style, one can directly specify HTTP methods and URL path for each property, action and event. A configurable HTTP Server is already available (from `hololinked.server.HTTPServer`) which redirects HTTP requests to the object according to the specified HTTP API on the properties, actions and events. To plug in a HTTP server:
To start the Thing, a configurable HTTP Server is already available (from `hololinked.server.HTTPServer`) which redirects HTTP requests to the object:

```python
import ssl, os, logging
Expand All @@ -293,6 +301,7 @@ if __name__ == '__main__':
# or O.run(zmq_protocols=['IPC', 'TCP'], tcp_socket_address='tcp://*:9999')
# both interprocess communication & TCP, no HTTP
```
> There is an ongoing work to remove HTTP API from the API of all of properties, actions and events and completely move them to the HTTP server for a more accurate syntax. The functionality will not change though.

Here one can see the use of `instance_name` and why it turns up in the URL path. See the detailed example of the above code [here](https://gitlab.com/hololinked-examples/oceanoptics-spectrometer/-/blob/simple/oceanoptics_spectrometer/device.py?ref_type=heads).

Expand All @@ -305,7 +314,20 @@ See a list of currently supported possibilities while using this package [below]

> You may use a script deployment/automation tool to remote stop and start servers, in an attempt to remotely control your hardware scripts.

One may use the HTTP API according to one's beliefs (including letting the package auto-generate it), but it is mainly intended for web development and cross platform clients like the [node-wot](https://github.com/eclipse-thingweb/node-wot) HTTP(s) client. If your plan is to develop a truly networked system, it is recommended to learn more and use [Thing Descriptions](https://www.w3.org/TR/wot-thing-description11) to describe your hardware. A Thing Description will be automatically generated if absent as shown in JSON examples above or can be supplied manually.
### Looking for sponsorships

Kindly read my message [in my README](https://github.com/VigneshVSV#sponsor)

### A little more about Usage

One may use the HTTP API according to one's beliefs (including letting the package auto-generate it), but it is mainly intended for web development and cross platform clients
like the interoperable [node-wot](https://github.com/eclipse-thingweb/node-wot) HTTP(s) client. If your plan is to develop a truly networked system, it is recommended to learn more and
se [Thing Descriptions](https://www.w3.org/TR/wot-thing-description11) to describe your hardware. A Thing Description will be automatically generated if absent as shown in JSON examples above or can be supplied manually. The default end point to
fetch thing descriptions are: <br> `http(s)://<host name>/<instance name of the thing>/resources/wot-td`
If there are errors in generation of Thing Description
(mostly due to JSON non-complaint types), one could use: <br> `http(s)://<host name>/<instance name of the thing>/resources/wot-td?ignore_errors=true`

(client docs will be updated here next)

### Currently Supported

Expand All @@ -315,22 +337,19 @@ One may use the HTTP API according to one's beliefs (including letting the packa
- use serializer of your choice (except for HTTP) - MessagePack, JSON, pickle etc. & extend serialization to suit your requirement. HTTP Server will support only JSON serializer to maintain comptibility with Javascript (MessagePack may be added later). Default is JSON serializer based on msgspec.
- asyncio compatible - async RPC server event-loop and async HTTP Server - write methods in async
- choose from multiple ZeroMQ transport methods which offers some possibilities like the following without changing the code:
- run HTTP Server & python object in separate processes or the same process
- serve multiple objects with the same HTTP server
- run direct ZMQ-TCP server without HTTP details
- expose only a dashboard or web page on the network without exposing the hardware itself
- run direct ZMQ-TCP server without HTTP details
- serve multiple objects with the same HTTP server, run HTTP Server & python object in separate processes or the same process

Again, please check examples or the code for explanations. Documentation is being activety improved.

### Currently being worked

- improving accuracy of Thing Descriptions
- separation of HTTP protocol specification like URL path and HTTP verbs from the API of properties, actions and events and move their customization completely to the HTTP server
- unit tests coverage
- separation of HTTP protocol specification like URL path and HTTP verbs from the API of properties, actions and events and move their customization completely to the HTTP server
- serve multiple things with the same server (unfortunately due to a small oversight it is currently somewhat difficult for end user to serve multiple things with the same server, although its possible. This will be fixed.)
- improving accuracy of Thing Descriptions
- cookie credentials for authentication - as a workaround until credentials are supported, use `allowed_clients` argument on HTTP server which restricts access based on remote IP supplied with the HTTP headers. This wont still help you in public networks or modified/non-standard HTTP clients.

### Internals

This package is an implementation of a ZeroMQ-based Object Oriented RPC with customizable HTTP end-points. A dual transport in both ZMQ and HTTP is provided to maximize flexibility in data type, serialization and speed, although HTTP is preferred for networked applications. If one is looking for an object oriented approach towards creating components within a control or data acquisition system, or an IoT device, one may consider this package.


2 changes: 1 addition & 1 deletion examples
2 changes: 1 addition & 1 deletion hololinked/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.2.4"
__version__ = "0.2.7"
Loading
Loading