Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create example of XML/HTML/JSON/Yaml templates #50

Merged
merged 5 commits into from
Jul 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker/Dockerfile-vscode
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ WORKDIR /home/vscode

# Setup python deps and configure rust.
#
RUN pip3 --disable-pip-version-check --no-cache-dir install mypy wasmtime \
RUN pip3 --disable-pip-version-check --no-cache-dir install mypy wasmtime==0.37.0 \
&& rm -rf /tmp/pip-tmp \
&& bash /tmp/library-scripts/install-rust-tools.sh \
&& bash /tmp/library-scripts/install-wit-bindgen.sh
Expand Down
17 changes: 17 additions & 0 deletions examples/rust/templates/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[package]
name = "templates"
version = "0.1.0"
edition = "2021"

[dependencies]
wit-bindgen-rust = { git = "https://github.com/bytecodealliance/wit-bindgen.git", rev = "60e3c5b41e616fee239304d92128e117dd9be0a7" }
tera = "1.19"
serde_json = "1.0"
skyscraper = "0.4"
jsonpath_lib = "0.1"

[dependencies.yaml-rust]
git = "https://github.com/chyh1990/yaml-rust.git"

[lib]
crate-type = ["cdylib"]
16 changes: 16 additions & 0 deletions examples/rust/templates/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.PHONY: debug
debug: $(eval TGT:=debug)
debug: wasm

.PHONY: release
release: $(eval TGT:=release)
release: RELFLAGS = --release
release: wasm

.PHONY: wasm
wasm:
cargo wasi build --lib $(RELFLAGS)

.PHONY: clean
clean:
@cargo clean
202 changes: 202 additions & 0 deletions examples/rust/templates/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
# Templating Functions

The templating functions in this example allow you to apply an
XML, HTML, JSON, or Yaml document / object to a
[Tera](https://tera.netlify.app) template. In the case of XML
and HTML, the documents can be queried using
[XPath](https://www.w3.org/TR/xpath-31/) expressions. For JSON
and Yaml, [JSONPath](https://datatracker.ietf.org/wg/jsonpath/about/)
expressions can be used. The output data format can be anything
supported by Tera templates, which can generate pretty much any
text-based format.

What this means is that you can transform XML, HTML, JSON, or Yaml
strings into any other format including new forms of XML, HTML, JSON,
or Yaml. For example, if you had the following XML structure:
```
<book category="web">
<title lang="en">XQuery Kick Start</title>
<author>James McGovern</author>
<author>Per Bothner</author>
<author>Kurt Cagle</author>
<author>James Linn</author>
<author>Vaidyanathan Nagarajan</author>
<year>2003</year>
<price>49.99</price>
</book>
```

You could convert it to a JSON structure like this:
```
{
"book": {
"title": "XQuery Kick Start",
"authors": [
"James McGovern",
"Per Bothner",
"Kurt Cagle",
"James Linn",
"Vaidyanathan Nagarajan"
],
"published": {
"year": 2003
},
"listing": {
"price": 49.99
}
}
}
```

Using the `render_xml` function included in this package and the
following template:
```
{
"book": {
"title": {{ q(path="/book/title") | get(key="text") | json_encode | safe }},
"authors": [{% for item in q(path="/book/author") %}
{{ item | get(key="text") | json_encode | safe }}{% if not loop.last %}, {% endif %}
{% endfor %}],
"published": { "year": {{ q(path="/book/year") | get(key="text") | int }} },
"listing": { "price": {{ q(path="/book/price") | get(key="text") | float }} }
}
}
```

## The `q` function

In addition to being able to traverse the objects in the parsed document using
the Tera syntax, a `q` function has also been added. It has the following
signature:
```
q(path="...") -> string
```

The `path` parameter when using XML or HTML input is an XPath query. For
JSON and Yaml input, the path is a JSONPath query. This allows you to use
more powerful queries to extract pieces of your input document than what
Tera can do by default.

## XML / HTML object structure

While JSON and Yaml are fairly straight-forward data structures that both map to
JSON structures entirely using maps and arrays, XML and HTML are a bit more
complex. There is no standard map or array type in XML / HTML. In order to make
it possible to traverse XML / HTML objects in a Tera template, those documents
are converted to a JSON-like object using the following mappings:
```
<book category="web">
<title lang="en">XQuery Kick Start</title>
<author>James McGovern</author>
<author>Per Bothner</author>
<author>Kurt Cagle</author>
<author>James Linn</author>
<author>Vaidyanathan Nagarajan</author>
<year>2003</year>
<price>49.99</price>
</book>
```

The JSON object for the above XML looks like:
```
{
"_": {
"name": "book",
"attributes": {"category": "web"},
"children": [
{
"name": "title",
"attributes": {"lang": "en"},
"text": "XQuery Kick Start",
"children": []
},
{
"name": "author",
"attributes": {},
"text": "James McGovern",
"children": []
},
...
{
"name": "year",
"attributes": {},
"text": "2003",
"children": []
},
{
"name": "price",
"attributes": {},
"text": "49.99",
"children": []
},
]
}
}
```

As you can see traversing an XML document using Tera's syntax or
even JSONPath is rather complicated. Using the `q(path="...")` function
to get nodes is much simpler. Even in an XML document, the `q` function
returns a JSON object that can be traversed from that point. For example,
to get the price of a book using the `q` function could be done as follows:
```
q(path="/book/price")
```

This will return an object of the following form:
```
{
"name": "price",
"attributes": {},
"text": "49.99",
"children": []
}
```

You can then extract the `text` attribute using Tera's `get` function:
```
q(path="/book/price") | get(key="text")
```

The above expression will return "49.99".

## UDFs

The functions included in this package are as follows:

```
render_json(json-string, template-string) -> string
render_xml(xml-string, template-string) -> string
render_yaml(yaml-string, template-string) -> string
```

## Compiling

To compile the functions in the example, use the following command.
```
cargo wasi build --lib --release
```

The Makefile can also be used to build the Wasm file.
```
make release
```

## Load functions into the database

Once you have compiled the functions, they can be loaded into the database
using the `pushwasm` command.

```
pushwasm udf --conn mysql://user:@127.0.0.1:3306/dbname --wit templates.wit \
--wasm target/wasm32-wasi/release/templates.wasm --name render_json
pushwasm udf --conn mysql://user:@127.0.0.1:3306/dbname --wit templates.wit \
--wasm target/wasm32-wasi/release/templates.wasm --name render_xml
pushwasm udf --conn mysql://user:@127.0.0.1:3306/dbname --wit templates.wit \
--wasm target/wasm32-wasi/release/templates.wasm --name render_yaml
```

## Using the functions

The `test.py` file contains a Python program that demonstrates the use
of each of the functions.
Loading
Loading