Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev version 1 #10

Open
wants to merge 38 commits into
base: version-1
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
e8ab0d2
[TH2-4730] Corrected decoding
pavelgarin Feb 9, 2023
7177e2c
[th2-4347] vulnerabilities fix
Xanclry Feb 10, 2023
9f24d0f
added github workflows
Xanclry Feb 10, 2023
7c9ed74
removed owasp scanner
Xanclry Feb 10, 2023
79e5e95
Updated dependencies
Nikita-Smirnov-Exactpro Feb 13, 2023
6e3af80
[TH2-4730] Added encoder
pavelgarin Feb 13, 2023
fc72ffa
Merge branch 'TH2-4730' into th2-4347-vulnerabilities-fix
pavelgarin Feb 14, 2023
755bc6e
Merge pull request #3 from th2-net/th2-4347-vulnerabilities-fix
pavelgarin Feb 14, 2023
a11ec62
[TH2-4730] Added support standart logical types
pavelgarin Feb 15, 2023
f0e6a6d
[th2-4730] Downgrades to common:3.44.0
Nikita-Smirnov-Exactpro Feb 15, 2023
4e2b324
[TH2-4730] Added support for multiple avro schemes
pavelgarin Feb 15, 2023
c8edc2c
[TH2-4730] Added support union type
pavelgarin Feb 17, 2023
acb35ed
[TH2-4730] Updating dependensies versions
pavelgarin Feb 17, 2023
f0dfeeb
[th2-4730] corrected gradlew files
Nikita-Smirnov-Exactpro Feb 17, 2023
6c7aaaa
Merge remote-tracking branch 'origin/TH2-4730' into TH2-4730-without-…
Nikita-Smirnov-Exactpro Feb 17, 2023
2ae6f4d
[TH2-4730] Added option for fast parsing union
pavelgarin Feb 17, 2023
0a40ab3
[TH2-4730] Added README
pavelgarin Feb 17, 2023
914c492
[th2-4730] Added dev-release
Nikita-Smirnov-Exactpro Feb 21, 2023
65da630
[th2-4730] Corrected version
Nikita-Smirnov-Exactpro Feb 21, 2023
051d0b4
[th2-4730] Corrected settings
Nikita-Smirnov-Exactpro Feb 21, 2023
a5c3d49
Merge remote-tracking branch 'origin/TH2-4730' into TH2-4730-without-…
Nikita-Smirnov-Exactpro Feb 21, 2023
8ee1d55
[th2-4730] Corrected version
Nikita-Smirnov-Exactpro Feb 21, 2023
bc77a1b
Merge pull request #4 from th2-net/TH2-4730-without-book-page
pavelgarin Feb 21, 2023
5474417
Vulnerabilities (#5)
isengrims Feb 22, 2023
852ee83
[TH2-4730] Added mode decode/encode by session alias
pavelgarin Feb 24, 2023
3afd25d
[TH2-4730] Added support wildcard aliases
pavelgarin Mar 2, 2023
e99accd
[TH2-4730] Corrected README
pavelgarin Mar 2, 2023
776756e
[TH2-4730] Corrected README
pavelgarin Mar 2, 2023
15868ed
[TH2-4730] Corrected code
pavelgarin Mar 3, 2023
dc7e951
[TH2-4730] Thread safe fixes
pavelgarin Mar 6, 2023
195f393
[TH2-4730] Thread safe fixes
pavelgarin Mar 6, 2023
d60e539
[TH2-4730] Refactoring structure
pavelgarin Mar 8, 2023
b4b0e14
Updated owasp plugin
Nikita-Smirnov-Exactpro Mar 9, 2023
872d8e7
Merge pull request #8 from th2-net/TH2-4730-alias
pavelgarin Mar 9, 2023
cb7e3dc
[TH2-4730] Corrected code. Adding missing parent-event-id on encode
pavelgarin Mar 10, 2023
47491e7
[TH2-4730] Update to version 1.2.0
pavelgarin Mar 10, 2023
ec65abc
Merge pull request #13 from th2-net/TH2-4730-parent-event-id
pavelgarin Mar 10, 2023
6639373
Additional logging (#14)
isengrims Apr 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
version: 2
updates:
# Enable version updates for Gradle
- package-ecosystem: "gradle"
# Look for `build.gradle` file in the `root` directory
directory: "/"
# Check for updates every day (weekdays)
schedule:
interval: "daily"
allow:
# Allow updates for Exactpro packages
- dependency-name: "com.exactpro*"
# Allow up to 10 open pull requests
open-pull-requests-limit: 10
23 changes: 23 additions & 0 deletions .github/workflows/ci-unwelcome-words.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: CI

on:
pull_request:

jobs:
test:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
with:
ref: ${{ github.sha }}
- name: Checkout tool
uses: actions/checkout@v2
with:
repository: exactpro-th2/ci-github-action
ref: master
token: ${{ secrets.PAT_CI_ACTION }}
path: ci-github-action
- name: Run CI action
uses: ./ci-github-action
with:
ref: ${{ github.sha }}
20 changes: 20 additions & 0 deletions .github/workflows/dependabot-java-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Dependabot build check

on:
push:
branches:
- dependabot**

jobs:
build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
# Build package
- name: Set up JDK 11
uses: actions/setup-java@v1
with:
java-version: '11'
- name: Build with Gradle
run: ./gradlew --info clean build
20 changes: 20 additions & 0 deletions .github/workflows/dev-docker-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Dev build and publish Docker distributions to Github Container Registry ghcr.io

on:
push:
branches-ignore:
- master
- version-*
- dev-version-*
- dependabot**
paths-ignore:
- README.md

jobs:
build-job:
uses: th2-net/.github/.github/workflows/compound-java-dev.yml@main
with:
build-target: 'Docker'
docker-username: ${{ github.actor }}
secrets:
docker-password: ${{ secrets.GITHUB_TOKEN }}
18 changes: 18 additions & 0 deletions .github/workflows/dev-release-java-publish-sonatype.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Build and publish dev-release Java distributions to sonatype.

on:
push:
branches:
- dev-version-*
paths:
- gradle.properties

jobs:
build-job:
uses: th2-net/.github/.github/workflows/compound-java.yml@main
with:
build-target: 'Docker'
devRelease: true
docker-username: ${{ github.actor }}
secrets:
docker-password: ${{ secrets.GITHUB_TOKEN }}
18 changes: 18 additions & 0 deletions .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Build and publish Docker distributions to Github Container Registry ghcr.io

on:
push:
branches:
- master
- version-*
paths:
- gradle.properties

jobs:
build-job:
uses: th2-net/.github/.github/workflows/compound-java.yml@main
with:
build-target: 'Docker'
docker-username: ${{ github.actor }}
secrets:
docker-password: ${{ secrets.GITHUB_TOKEN }}
16 changes: 16 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
.gradle
build
out
.idea
logs


# Ignore Gradle GUI config
gradle-app.setting

# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)
!gradle-wrapper.jar

# Cache of project
.gradletasknamecache

# Compiled class file
*.class

Expand Down
9 changes: 9 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM gradle:7.5.1-jdk11 AS build
ARG release_version
COPY ./ .
RUN gradle --no-daemon clean build dockerPrepare -Prelease_version=${release_version}

FROM adoptopenjdk/openjdk11:alpine
WORKDIR /home
COPY --from=build /home/gradle/build/docker .
ENTRYPOINT ["/home/service/bin/service"]
98 changes: 97 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,97 @@
# th2-codec-avro
# AVRO codec (1.2.1)
## Description
Designed for decode AVRO raw messages to parsed messages and encode back.
It is based on [th2-codec](https://github.com/th2-net/th2-codec).
You can find additional information [here](https://github.com/th2-net/th2-codec/blob/master/README.md)

## Decoding

The codec decodes each raw message in the received batch.
The codec can work in two modes:
### Standard mode
Enabled when setting `avroMessageIdToDictionaryAlias` is filled.
Each raw message must contain AVRO header.
The first byte of the AVRO header is always 0(MAGIC BYTE).
It is followed by four bytes which identify the AVRO schema to be used for decoding.
```text
[0],[][][][], [][][][]][][]...[]
schema id data
```
### Schema resolution mode by session alias
Enabled when setting `sessionAliasToDictionaryAlias` is filled.
AVRO header of 5 bytes is missing.
Schema resolves by session alias in message metadata.

All AVRO data types are supported, support conversion to AVRO standard logical types
Complex types are converted to:
```text
Record -> Message
Map -> Message
Array -> List
Union -> Object of resolve by schema
Enum -> Value
```
Primitive types are converted to:
```text
Fixed -> Value as Hex string
Bytes -> Value as Hex string
Null -> skip
other -> Value
```

**NOTE: To the field name with a UNION value added is a prefix, because when converting to Value, the data type is lost**.

From the choice of the `enableIdPrefixEnumFields` setting, the prefix can be a `schema id`
```text
Id0-fieldName
```
or `Avro data type`
```text
Record-fieldName
```

After the prefix used a unique delimiter `-`
## Encoding
When encoding, all values are converted back, including logical type conversion. Before the data added `AVRO header`.

**NOTE: When encoding UNION values, a prefix is required for field names. This is described in the decoding section.**

## Settings
AVRO codec has the following parameters:

```yaml
enableIdPrefixEnumFields: false
avroMessageIdToDictionaryAlias:
'1': "${dictionary_link:avro-schema-1-dictionary}"
'2': "${dictionary_link:avro-schema-2-dictionary}"
'3': "${dictionary_link:avro-schema-3-dictionary}"
sessionAliasToDictionaryAlias:
'sessionAlias1': "${dictionary_link:avro-schema-1-dictionary}"
'sessionGroup1Alias*': "${dictionary_link:avro-schema-2-dictionary}"
'???????Group2Alias*': "${dictionary_link:avro-schema-3-dictionary}"
```
**enableIdPrefixEnumFields** - prefix setting for UNION fields. If `false`, use prefix as `AVRO data type`(for example `Record-`, `Map-`), if `true` then use `schema id` prefix(for example `Id0-`, `Id3-`). The default value is `false`

**avroMessageIdToDictionaryAlias** - matching `schema id` pairs with its `alias` available for loading in the pipelineCodecContext.

**sessionAliasToDictionaryAlias** - matching `session alias` pairs with its `alias` available for loading in the pipelineCodecContext. For `session alias` supported `wildcard`.

Only one of settings `sessionAliasToDictionaryAlias` or `avroMessageIdToDictionaryAlias` can be used

## Release notes

### 1.2.1
* Added detailed TRACE logging for parsing into class com.exactpro.th2.codec.MessageDatumReader

### 1.2.0
+ Added mode resolution schema by session alias
+ Added support wildcard for session alias setting

### 1.1.0
+ th2-bom upgrade to `4.2.0`
+ th2-common upgrade to `3.44.1`

### 1.0.0
+ Added data types described in [avro 1.10.2](https://avro.apache.org/docs/1.10.2/spec.html)

### 0.0.1
Loading