Skip to content

Commit f6b43db

Browse files
vsochvsoch
authored
finishing up (#1)
* finishing up Signed-off-by: vsoch <[email protected]>
1 parent d18cb97 commit f6b43db

File tree

10 files changed

+1039
-25
lines changed

10 files changed

+1039
-25
lines changed

Diff for: ‎.devcontainer/Dockerfile

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
FROM ghcr.io/converged-computing/metric-ior:latest
2+
3+
LABEL maintainer="Vanessasaurus <@vsoch>"
4+
5+
# Match the default user id for a single system so we aren't root
6+
ARG USERNAME=vscode
7+
ARG USER_UID=1000
8+
ARG USER_GID=1000
9+
ENV USERNAME=${USERNAME}
10+
ENV USER_UID=${USER_UID}
11+
ENV USER_GID=${USER_GID}
12+
USER root
13+
14+
# extra interactive utilities
15+
RUN apt-get update \
16+
&& apt-get -qq install -y --no-install-recommends \
17+
vim less sudo python3-pip
18+
19+
# Ensure regular python is visible
20+
RUN ln -s /usr/bin/python3 /usr/bin/python
21+
22+
# Add the group and user that match our ids
23+
RUN groupadd -g ${USER_GID} ${USERNAME} && \
24+
adduser --disabled-password --uid ${USER_UID} --gid ${USER_GID} --gecos "" ${USERNAME} && \
25+
echo "${USERNAME} ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers
26+
USER $USERNAME

Diff for: ‎.devcontainer/devcontainer.json

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
"name": "Compspec IOR Development Environment",
3+
"dockerFile": "Dockerfile",
4+
"context": "../",
5+
6+
"customizations": {
7+
"vscode": {
8+
"settings": {
9+
"terminal.integrated.defaultProfile.linux": "bash"
10+
},
11+
"extensions": [
12+
"ms-vscode.cmake-tools"
13+
]
14+
}
15+
},
16+
"postStartCommand": "git config --global --add safe.directory /workspaces/compspec-ior"
17+
}

Diff for: ‎.github/workflows/main.yaml

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request: []
8+
9+
jobs:
10+
test:
11+
name: Test IOR
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Checkout Repository
15+
uses: actions/checkout@v4
16+
- name: Install compspec
17+
run: pip install compspec
18+
- name: Install compspec-ior
19+
run: pip install .
20+
- name: Test with loading data
21+
run: compspec extract ior --ior-load ./examples/test/ior-data.json
22+
- name: Test Python
23+
run: python ./examples/singleton-run.py
24+
25+
validate-schema:
26+
name: Validate schema
27+
runs-on: ubuntu-latest
28+
steps:
29+
- name: Checkout Repository
30+
uses: actions/checkout@v4
31+
- name: Validate Schema
32+
uses: compspec/actions/validate-schema@main
33+
with:
34+
schema: ./compspec_ior/schema.json
35+
36+
formatting:
37+
runs-on: ubuntu-latest
38+
steps:
39+
- uses: actions/checkout@v4
40+
41+
- name: Setup black linter
42+
run: conda create --quiet --name black pyflakes
43+
44+
- name: Check Spelling
45+
uses: crate-ci/typos@7ad296c72fa8265059cc03d1eda562fbdfcd6df2 # v1.9.0
46+
with:
47+
files: ./README.md
48+
49+
- name: Lint and format Python code
50+
run: |
51+
export PATH="/usr/share/miniconda/bin:$PATH"
52+
source activate black
53+
pip install -r .github/dev-requirements.txt
54+
pre-commit run --all-files

Diff for: ‎README.md

+194-8
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
<img height="300" src="https://raw.githubusercontent.com/compspec/spec/main/img/compspec-circle.png">
55
</p>
66

7+
[![PyPI version](https://badge.fury.io/py/compspec-ior.svg)](https://badge.fury.io/py/compspec-ior)
8+
79
A compspec (Composition spec) is a specification and model for comparing things. Compspec IOR is
810
a plugin for extraction of [IOR](https://github.com/hpc/ior) metadata from applications, and packaging in compatibility specification
911
artifacts. This means that we also maintain the compatibility schema here. To learn more:
@@ -23,21 +25,205 @@ pip install compspec-ior
2325
```
2426

2527
Then run an extraction with IOR. You can use defaults, or add any parameters to IOR after the plugin name "ior"
28+
Here is how to print to the terminal:
29+
30+
```bash
31+
compspec extract ior
32+
```
33+
34+
<details>
35+
36+
<summary>IOR output</summary>
37+
38+
```console
39+
{
40+
"version": "0.0.0",
41+
"kind": "CompatibilitySpec",
42+
"metadata": {
43+
"name": "compat-experiment",
44+
"schemas": {
45+
"io.compspec.ior": "https://raw.githubusercontent.com/compspec/compspec-ior/main/compspec_ior/schema.json"
46+
}
47+
},
48+
"compatibilities": [
49+
{
50+
"name": "io.compspec.ior",
51+
"version": "0.0.0",
52+
"attributes": {
53+
"version": "4.0.0rc1",
54+
"began": "Thu Feb 22 00:36:12 2024",
55+
"machine": "Linux 2b3ee0c4c948",
56+
"finished": "Thu Feb 22 00:36:12 2024",
57+
"command_line": "ior -O summaryFormat=JSON",
58+
"summary.write.operation": "write",
59+
"summary.write.API": "POSIX",
60+
"summary.write.TestID": 0,
61+
"summary.write.ReferenceNumber": 0,
62+
"summary.write.segmentCount": 1,
63+
"summary.write.blockSize": 1048576,
64+
"summary.write.transferSize": 262144,
65+
"summary.write.numTasks": 1,
66+
"summary.write.tasksPerNode": 1,
67+
"summary.write.repetitions": 1,
68+
"summary.write.filePerProc": 0,
69+
"summary.write.reorderTasks": 0,
70+
"summary.write.taskPerNodeOffset": 1,
71+
"summary.write.reorderTasksRandom": 0,
72+
"summary.write.reorderTasksRandomSeed": 0,
73+
"summary.write.bwMaxMIB": 904.92,
74+
"summary.write.bwMinMIB": 904.92,
75+
"summary.write.bwMeanMIB": 904.92,
76+
"summary.write.bwStdMIB": 0.0,
77+
"summary.write.OPsMax": 3619.6798,
78+
"summary.write.OPsMin": 3619.6798,
79+
"summary.write.OPsMean": 3619.6798,
80+
"summary.write.OPsSD": 0.0,
81+
"summary.write.MeanTime": 0.0011,
82+
"summary.write.xsizeMiB": 1.0,
83+
"summary.read.operation": "read",
84+
"summary.read.API": "POSIX",
85+
"summary.read.TestID": 0,
86+
"summary.read.ReferenceNumber": 0,
87+
"summary.read.segmentCount": 1,
88+
"summary.read.blockSize": 1048576,
89+
"summary.read.transferSize": 262144,
90+
"summary.read.numTasks": 1,
91+
"summary.read.tasksPerNode": 1,
92+
"summary.read.repetitions": 1,
93+
"summary.read.filePerProc": 0,
94+
"summary.read.reorderTasks": 0,
95+
"summary.read.taskPerNodeOffset": 1,
96+
"summary.read.reorderTasksRandom": 0,
97+
"summary.read.reorderTasksRandomSeed": 0,
98+
"summary.read.bwMaxMIB": 6615.6215,
99+
"summary.read.bwMinMIB": 6615.6215,
100+
"summary.read.bwMeanMIB": 6615.6215,
101+
"summary.read.bwStdMIB": 0.0,
102+
"summary.read.OPsMax": 26462.4858,
103+
"summary.read.OPsMin": 26462.4858,
104+
"summary.read.OPsMean": 26462.4858,
105+
"summary.read.OPsSD": 0.0,
106+
"summary.read.MeanTime": 0.0002,
107+
"summary.read.xsizeMiB": 1.0,
108+
"test.0.starttime": "Thu Feb 22 00:36:12 2024",
109+
"test.0.capacity": "1.8 TiB",
110+
"test.0.used_capacity": "20.2%",
111+
"test.0.inodes": "116.4 Mi",
112+
"test.0.used_inodes": "5.3%",
113+
"test.0.parameters.testID": 0,
114+
"test.0.parameters.refnum": 0,
115+
"test.0.parameters.api": "POSIX",
116+
"test.0.parameters.platform": "2b3ee0c4c(Linux)",
117+
"test.0.parameters.testFileName": "testFile",
118+
"test.0.parameters.deadlineForStonewall": 0,
119+
"test.0.parameters.stoneWallingWearOut": 0,
120+
"test.0.parameters.maxTimeDuration": 0,
121+
"test.0.parameters.outlierThreshold": 0,
122+
"test.0.parameters.options": "(null)",
123+
"test.0.parameters.dryRun": 0,
124+
"test.0.parameters.nodes": 1,
125+
"test.0.parameters.memoryPerTask": 0,
126+
"test.0.parameters.memoryPerNode": 0,
127+
"test.0.parameters.tasksPerNode": 1,
128+
"test.0.parameters.repetitions": 1,
129+
"test.0.parameters.multiFile": 0,
130+
"test.0.parameters.interTestDelay": 0,
131+
"test.0.parameters.fsync": 0,
132+
"test.0.parameters.fsyncperwrite": 0,
133+
"test.0.parameters.useExistingTestFile": 0,
134+
"test.0.parameters.uniqueDir": 0,
135+
"test.0.parameters.singleXferAttempt": 0,
136+
"test.0.parameters.readFile": 1,
137+
"test.0.parameters.writeFile": 1,
138+
"test.0.parameters.filePerProc": 0,
139+
"test.0.parameters.reorderTasks": 0,
140+
"test.0.parameters.reorderTasksRandom": 0,
141+
"test.0.parameters.reorderTasksRandomSeed": 0,
142+
"test.0.parameters.randomOffset": 0,
143+
"test.0.parameters.checkWrite": 0,
144+
"test.0.parameters.checkRead": 0,
145+
"test.0.parameters.dataPacketType": 0,
146+
"test.0.parameters.keepFile": 0,
147+
"test.0.parameters.keepFileWithError": 0,
148+
"test.0.parameters.warningAsErrors": 0,
149+
"test.0.parameters.verbose": 0,
150+
"test.0.parameters.data packet type": "g",
151+
"test.0.parameters.setTimeStampSignature/incompressibleSeed": 0,
152+
"test.0.parameters.collective": 0,
153+
"test.0.parameters.segmentCount": 1,
154+
"test.0.parameters.transferSize": 262144,
155+
"test.0.parameters.blockSize": 1048576,
156+
"test.0.options.api": "POSIX",
157+
"test.0.options.apiVersion": "",
158+
"test.0.options.test filename": "testFile",
159+
"test.0.options.access": "single-shared-file",
160+
"test.0.options.type": "independent",
161+
"test.0.options.segments": 1,
162+
"test.0.options.ordering in a file": "sequential",
163+
"test.0.options.ordering inter file": "no tasks offsets",
164+
"test.0.options.nodes": 1,
165+
"test.0.options.tasks": 1,
166+
"test.0.options.clients per node": 1,
167+
"test.0.options.repetitions": 1,
168+
"test.0.options.xfersize": "262144 bytes",
169+
"test.0.options.blocksize": "1 MiB",
170+
"test.0.options.aggregate filesize": "1 MiB",
171+
"test.0.results.0.access": "write",
172+
"test.0.results.0.bwMiB": 904.92,
173+
"test.0.results.0.blockKiB": 1024.0,
174+
"test.0.results.0.xferKiB": 256.0,
175+
"test.0.results.0.iops": 3842.6972,
176+
"test.0.results.0.latency": 0.0003,
177+
"test.0.results.0.openTime": 0.0001,
178+
"test.0.results.0.wrRdTime": 0.001,
179+
"test.0.results.0.closeTime": 0.0,
180+
"test.0.results.0.totalTime": 0.0011,
181+
"test.0.results.1.access": "read",
182+
"test.0.results.1.bwMiB": 6615.6215,
183+
"test.0.results.1.blockKiB": 1024.0,
184+
"test.0.results.1.xferKiB": 256.0,
185+
"test.0.results.1.iops": 27962.0267,
186+
"test.0.results.1.latency": 0.0,
187+
"test.0.results.1.openTime": 0.0,
188+
"test.0.results.1.wrRdTime": 0.0001,
189+
"test.0.results.1.closeTime": 0.0,
190+
"test.0.results.1.totalTime": 0.0002
191+
}
192+
}
193+
]
194+
}
195+
```
196+
197+
</details>
198+
199+
And how to save to file
200+
201+
```bash
202+
compspec extract --outfile ior-test.json ior
203+
```
204+
205+
And run the example to see how to use compspec-ior directly in Python to generate the same
206+
artifact.
207+
208+
```bash
209+
python ./examples/singleton-run.py
210+
```
211+
212+
213+
### Development
214+
215+
If you open the [Development container](.devcontainer) in VSCode, you'll find ior on the path:
26216

27217
```bash
28-
compspec extract ior ...
218+
$ which ior
219+
/usr/bin/ior
29220
```
30221

31-
More coming soon!
222+
This allows us to easily develop and test the compatibility plugin. You can
32223

33224
## TODO
34225

35-
- Developer environment with IOR installed (for others and me too)
36-
- testing, etc with pre-commit and spell checking
37-
- implement run functionality
38-
- use reasonable defaults for when nothing provided
39-
- outputs should map to new schema.json attributes
40-
- main library compspec should have support for oras push, etc.
226+
- how to handle adding lists (with indices) to schema?
41227

42228
## License
43229

Diff for: ‎compspec_ior/__init__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
from .plugin import ExtractorPlugin
1+
from .plugin import Plugin
22
from .version import __version__

Diff for: ‎compspec_ior/defaults.py

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# These are required to be given to compspec
2+
spec_version = "0.0.0"
3+
schema_url = "https://raw.githubusercontent.com/compspec/compspec-ior/main/compspec_ior/schema.json"
4+
namespace = "io.compspec.ior"

0 commit comments

Comments
 (0)