Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: Add Oslat test run #46

Merged
merged 4 commits into from
Dec 19, 2023
Merged

executor: Add Oslat test run #46

merged 4 commits into from
Dec 19, 2023

Conversation

RamLavi
Copy link
Collaborator

@RamLavi RamLavi commented Dec 14, 2023

This PR introduces the Oslat test run to the executor library.

@RamLavi RamLavi force-pushed the osnat branch 2 times, most recently from 74ecb9c to c3f195d Compare December 14, 2023 12:52
@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 14, 2023

passed on CNV 4.15 cluster, using my private vm image quay.io/ramlavi/kubevirt-realtime-checkup:devel:

make e2e-test
mkdir -p /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache
podman run --rm \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup:/go/src/github.com/kiagnose/kubevirt-realtime-checkup:Z \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache:/root/.cache/go-build:Z \
	-v /home/ralavi/.kube/sno03-cnvqe2-rdu2:/root/.kube:Z,ro \
	--workdir /go/src/github.com/kiagnose/kubevirt-realtime-checkup \
	-e KUBECONFIG=/root/.kube/kubeconfig \
	-e TEST_NAMESPACE=realtime-checkup-1 \
	-e TEST_CHECKUP_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup:devel \
	-e VM_UNDER_TEST_CONTAINER_DISK_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup-vm:latest \
	docker.io/library/golang:1.20.12 \
	go test -v ./tests/... -test.v -test.timeout=1h -ginkgo.v -ginkgo.timeout=1h 
=== RUN   TestTests
Running Suite: Tests Suite - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests
========================================================================================
Random Seed: 1702558502

Will run 1 of 1 specs
------------------------------
[BeforeSuite] 
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/tests_suite_test.go:39
[BeforeSuite] PASSED [0.008 seconds]
------------------------------
Checkup execution should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80
• [745.874 seconds]
------------------------------

Ran 1 of 1 Specs in 745.883 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestTests (745.89s)
PASS
ok  	github.com/kiagnose/kubevirt-realtime-checkup/tests	745.918s

logs:

$ oc logs -f job/rt-checkup
2023/12/14 12:55:09 kubevirt-realtime-checkup starting...
2023/12/14 12:55:09 Using the following config:
2023/12/14 12:55:09 	"vmUnderTestTargetNodeName": ""
2023/12/14 12:55:09 	"vmUnderTestContainerDiskImage": "quay.io/ramlavi/kubevirt-realtime-checkup-vm:latest"
2023/12/14 12:55:09 	"oslatDuration": "10m0s"
2023/12/14 12:55:09 	"oslatLatencyThresholdMicroSeconds": "45µs"
2023/12/14 12:55:09 Waiting for VMI "realtime-checkup-1/rt-vmi-l4bbn" to boot...
2023/12/14 12:56:14 VMI "realtime-checkup-1/rt-vmi-l4bbn" had successfully booted
2023/12/14 12:56:14 Login to VMI under test...
2023/12/14 12:57:15 Running Oslat test on VMI under test for 10m0s...
2023/12/14 13:07:17 Oslat test completed:
taskset -c 1-2 oslat --cpu-list 1-2 --rtprio 1 --duration 10m0s --workload memmove --workload-mem 4K 
oslat V 2.60
Total runtime: 		600 seconds
Thread priority: 	SCHED_FIFO:1
CPU list: 		1-2
CPU for main thread: 	0
Workload: 		memmove
Workload mem: 		4 (KiB)
Preheat cores: 		2

Pre-heat for 1 seconds...
Test starts...
Test completed.

        Core:	 1 2
Counter Freq:	 2096 2096 (Mhz)
    001 (us):	 0 0
    002 (us):	 6148641856 6192935687
    003 (us):	 0 1
    004 (us):	 1305 1263
    005 (us):	 291 315
    006 (us):	 31 32
    007 (us):	 2 13
    008 (us):	 4 2
    009 (us):	 2 5
    010 (us):	 9 4
    011 (us):	 21 1
    012 (us):	 1 15
    013 (us):	 0 16
    014 (us):	 0 0
    015 (us):	 0 0
    016 (us):	 0 0
    017 (us):	 0 0
    018 (us):	 0 0
    019 (us):	 0 0
    020 (us):	 0 0
    021 (us):	 0 0
    022 (us):	 0 0
    023 (us):	 0 0
    024 (us):	 0 0
    025 (us):	 0 0
    026 (us):	 2 0
    027 (us):	 0 0
    028 (us):	 0 0
    029 (us):	 0 0
    030 (us):	 0 1
    031 (us):	 0 0
    032 (us):	 0 0 (including overflows)
     Minimum:	 1 1 (us)
     Average:	 2.000 2.000 (us)
     Maximum:	 25 29 (us)
     Max-Min:	 24 28 (us)
    Duration:	 599.732 599.732 (sec)

[root@rt-vmi-l4bbn cloud-user]# 
2023/12/14 13:07:17 Max Oslat Latency measured: 29µs
2023/12/14 13:07:17 Trying to delete VMI: "realtime-checkup-1/rt-vmi-l4bbn"
2023/12/14 13:07:17 Waiting for VMI "realtime-checkup-1/rt-vmi-l4bbn" to be deleted...
2023/12/14 13:07:22 VMI "realtime-checkup-1/rt-vmi-l4bbn" was deleted successfully

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 14, 2023

example of failed test (using my private vm image quay.io/ramlavi/kubevirt-realtime-checkup:devel and a 5 us threshold):

make e2e-test
mkdir -p /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache
podman run --rm \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup:/go/src/github.com/kiagnose/kubevirt-realtime-checkup:Z \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache:/root/.cache/go-build:Z \
	-v /home/ralavi/.kube/sno03-cnvqe2-rdu2:/root/.kube:Z,ro \
	--workdir /go/src/github.com/kiagnose/kubevirt-realtime-checkup \
	-e KUBECONFIG=/root/.kube/kubeconfig \
	-e TEST_NAMESPACE=realtime-checkup-1 \
	-e TEST_CHECKUP_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup:devel \
	-e VM_UNDER_TEST_CONTAINER_DISK_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup-vm:latest \
	docker.io/library/golang:1.20.12 \
	go test -v ./tests/... -test.v -test.timeout=1h -ginkgo.v -ginkgo.timeout=1h 
=== RUN   TestTests
Running Suite: Tests Suite - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests
========================================================================================
Random Seed: 1702559420

Will run 1 of 1 specs
------------------------------
[BeforeSuite] 
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/tests_suite_test.go:39
[BeforeSuite] PASSED [0.002 seconds]
------------------------------
Checkup execution should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80
  [FAILED] in [It] - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:90 @ 12/14/23 13:22:55.745
• [FAILED] [756.202 seconds]
Checkup execution [It] should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80

  [FAILED] checkup failed: {
  	"spec.param.oslatDuration": "10m",
  	"spec.param.oslatLatencyThresholdMicroSeconds": "5",
  	"spec.param.vmUnderTestContainerDiskImage": "quay.io/ramlavi/kubevirt-realtime-checkup-vm:latest",
  	"spec.param.vmUnderTestTargetNodeName": "",
  	"spec.timeout": "15m",
  	"status.completionTimestamp": "2023-12-14T13:22:49Z",
  	"status.failureReason": "oslat Max Latency measured 30µs exceeded the given threshold 5µs",
  	"status.result.oslatMaxLatencyMicroSeconds": "30",
  	"status.result.vmUnderTestActualNodeName": "",
  	"status.startTimestamp": "2023-12-14T13:10:24Z",
  	"status.succeeded": "false"
  }
  In [It] at: /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:90 @ 12/14/23 13:22:55.745
------------------------------

Summarizing 1 Failure:
  [FAIL] Checkup execution [It] should complete successfully
  /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:90

Ran 1 of 1 Specs in 756.205 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestTests (756.21s)
FAIL
FAIL	github.com/kiagnose/kubevirt-realtime-checkup/tests	756.217s
FAIL
make: *** [Makefile:70: e2e-test] Error 1

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 14, 2023

@orelmisan note that when I'm using the "official" vm image quay.io/ramlavi/kubevirt-realtime-checkup:devel then it's seems like oslat is not installed:

make e2e-test
mkdir -p /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache
podman run --rm \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup:/go/src/github.com/kiagnose/kubevirt-realtime-checkup:Z \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache:/root/.cache/go-build:Z \
	-v /home/ralavi/.kube/sno03-cnvqe2-rdu2:/root/.kube:Z,ro \
	--workdir /go/src/github.com/kiagnose/kubevirt-realtime-checkup \
	-e KUBECONFIG=/root/.kube/kubeconfig \
	-e TEST_NAMESPACE=realtime-checkup-1 \
	-e TEST_CHECKUP_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup:devel \
	-e VM_UNDER_TEST_CONTAINER_DISK_IMAGE=quay.io/kiagnose/kubevirt-realtime-checkup-vm:main \
	docker.io/library/golang:1.20.12 \
	go test -v ./tests/... -test.v -test.timeout=1h -ginkgo.v -ginkgo.timeout=1h 
=== RUN   TestTests
Running Suite: Tests Suite - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests
========================================================================================
Random Seed: 1702560317

Will run 1 of 1 specs
------------------------------
[BeforeSuite] 
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/tests_suite_test.go:39
[BeforeSuite] PASSED [0.002 seconds]
------------------------------
Checkup execution should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80
  [FAILED] in [It] - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:90 @ 12/14/23 13:27:02.294
• [FAILED] [106.191 seconds]
Checkup execution [It] should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80

  [FAILED] checkup failed: {
  	"spec.param.oslatDuration": "10m",
  	"spec.param.oslatLatencyThresholdMicroSeconds": "5",
  	"spec.param.vmUnderTestContainerDiskImage": "quay.io/kiagnose/kubevirt-realtime-checkup-vm:main",
  	"spec.param.vmUnderTestTargetNodeName": "",
  	"spec.timeout": "15m",
  	"status.completionTimestamp": "2023-12-14T13:26:54Z",
  	"status.failureReason": "failed to run Oslat on VMI \"realtime-checkup-1/rt-vmi-g47k4\": oslat test failed with exit code: 127. See logs for more information",
  	"status.startTimestamp": "2023-12-14T13:25:21Z",
  	"status.succeeded": "false"
  }
  In [It] at: /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:90 @ 12/14/23 13:27:02.294
------------------------------

Summarizing 1 Failure:
  [FAIL] Checkup execution [It] should complete successfully
  /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:90

Ran 1 of 1 Specs in 106.194 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestTests (106.19s)
FAIL
FAIL	github.com/kiagnose/kubevirt-realtime-checkup/tests	106.211s
FAIL
make: *** [Makefile:70: e2e-test] Error 1

logs:

$ oc logs -f job/rt-checkup
2023/12/14 13:25:21 kubevirt-realtime-checkup starting...
2023/12/14 13:25:21 Using the following config:
2023/12/14 13:25:21 	"vmUnderTestTargetNodeName": ""
2023/12/14 13:25:21 	"vmUnderTestContainerDiskImage": "quay.io/kiagnose/kubevirt-realtime-checkup-vm:main"
2023/12/14 13:25:21 	"oslatDuration": "10m0s"
2023/12/14 13:25:21 	"oslatLatencyThresholdMicroSeconds": "5µs"
2023/12/14 13:25:21 Waiting for VMI "realtime-checkup-1/rt-vmi-g47k4" to boot...
2023/12/14 13:26:26 VMI "realtime-checkup-1/rt-vmi-g47k4" had successfully booted
2023/12/14 13:26:26 Login to VMI under test...
2023/12/14 13:26:49 Running Oslat test on VMI under test for 10m0s...
2023/12/14 13:26:49 oslat test returned exit code: 127. stdout: taskset -c 1-2 oslat --cpu-list 1-2 --rtprio 1 --duration 10m0s --workload memmove --workload-mem 4K 
taskset: failed to execute oslat: No such file or directory
[root@rt-vmi-g47k4 cloud-user]# 
2023/12/14 13:26:49 Trying to delete VMI: "realtime-checkup-1/rt-vmi-g47k4"
2023/12/14 13:26:49 Waiting for VMI "realtime-checkup-1/rt-vmi-g47k4" to be deleted...
2023/12/14 13:26:54 VMI "realtime-checkup-1/rt-vmi-g47k4" was deleted successfully
2023/12/14 13:26:54 kubevirt-realtime-checkup failed: failed to run Oslat on VMI "realtime-checkup-1/rt-vmi-g47k4": oslat test failed with exit code: 127. See logs for more information

Do you have an idea why the image is not the same as mine? I didn't do anything special AFAIK.

@RamLavi RamLavi requested a review from orelmisan December 14, 2023 13:43
@orelmisan
Copy link
Member

@RamLavi Could you please try the quay.io/kiagnose/kubevirt-realtime-checkup-vm:main image again with with the change from PR #47?

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 17, 2023

@RamLavi Could you please try the quay.io/kiagnose/kubevirt-realtime-checkup-vm:main image again with with the change from PR #47?

@orelmisan it looks like it's working now. what has changed?

$ oc logs -f job/rt-checkup
2023/12/17 08:06:03 kubevirt-realtime-checkup starting...
2023/12/17 08:06:03 Using the following config:
2023/12/17 08:06:03 	"vmUnderTestTargetNodeName": ""
2023/12/17 08:06:03 	"vmUnderTestContainerDiskImage": "quay.io/kiagnose/kubevirt-realtime-checkup-vm:main"
2023/12/17 08:06:03 	"oslatDuration": "10m0s"
2023/12/17 08:06:03 	"oslatLatencyThresholdMicroSeconds": "45µs"
2023/12/17 08:06:03 Waiting for VMI "realtime-checkup-1/rt-vmi-6spqx" to boot...
2023/12/17 08:07:08 VMI "realtime-checkup-1/rt-vmi-6spqx" had successfully booted
2023/12/17 08:07:08 Login to VMI under test...
2023/12/17 08:08:15 Running Oslat test on VMI under test for 10m0s...
2023/12/17 08:18:17 Oslat test completed:
taskset -c 1-2 oslat --cpu-list 1-2 --rtprio 1 --duration 10m0s --workload memmove --workload-mem 4K 
oslat V 2.60
Total runtime: 		600 seconds
Thread priority: 	SCHED_FIFO:1
CPU list: 		1-2
CPU for main thread: 	0
Workload: 		memmove
Workload mem: 		4 (KiB)
Preheat cores: 		2

Pre-heat for 1 seconds...
Test starts...
Test completed.

        Core:	 1 2
Counter Freq:	 2096 2096 (Mhz)
    001 (us):	 0 0
    002 (us):	 6147050441 6193267362
    003 (us):	 1 1
    004 (us):	 105 94
    005 (us):	 43 44
    006 (us):	 7 14
    007 (us):	 2 3
    008 (us):	 1 0
    009 (us):	 3 2
    010 (us):	 15 3
    011 (us):	 12 7
    012 (us):	 1 17
    013 (us):	 0 4
    014 (us):	 0 0
    015 (us):	 0 0
    016 (us):	 0 0
    017 (us):	 0 0
    018 (us):	 0 0
    019 (us):	 0 0
    020 (us):	 0 0
    021 (us):	 0 0
    022 (us):	 0 0
    023 (us):	 0 0
    024 (us):	 0 0
    025 (us):	 0 1
    026 (us):	 0 0
    027 (us):	 0 1
    028 (us):	 0 0
    029 (us):	 0 0
    030 (us):	 0 0
    031 (us):	 0 0
    032 (us):	 0 0 (including overflows)
     Minimum:	 1 1 (us)
     Average:	 2.000 2.000 (us)
     Maximum:	 11 26 (us)
     Max-Min:	 10 25 (us)
    Duration:	 599.731 599.731 (sec)

[root@rt-vmi-6spqx cloud-user]# 
2023/12/17 08:18:17 Max Oslat Latency measured: 26µs
2023/12/17 08:18:17 Trying to delete VMI: "realtime-checkup-1/rt-vmi-6spqx"
2023/12/17 08:18:17 Waiting for VMI "realtime-checkup-1/rt-vmi-6spqx" to be deleted...
2023/12/17 08:18:27 VMI "realtime-checkup-1/rt-vmi-6spqx" was deleted successfully

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 17, 2023

checked after rebased #47 - works!

Thanks @orelmisan. The PR is ready for your review

Copy link
Member

@orelmisan orelmisan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR @RamLavi.
Please see the inline comments.

pkg/internal/checkup/executor/oslat/client.go Outdated Show resolved Hide resolved
pkg/internal/checkup/executor/oslat/client.go Outdated Show resolved Hide resolved
pkg/internal/checkup/executor/oslat/client_test.go Outdated Show resolved Hide resolved
pkg/internal/checkup/checkup.go Outdated Show resolved Hide resolved
@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 18, 2023

Change: @orelmisan 's review fixes

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 18, 2023

Change: Rebase

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 18, 2023

Change: pass the oslatThreshold via the checkup object, as @orelmisan suggested here

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 18, 2023

passed on CNV4.15:

make e2e-test
mkdir -p /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache
podman run --rm \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup:/go/src/github.com/kiagnose/kubevirt-realtime-checkup:Z \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache:/root/.cache/go-build:Z \
	-v /home/ralavi/.kube/sno03-cnvqe2-rdu2:/root/.kube:Z,ro \
	--workdir /go/src/github.com/kiagnose/kubevirt-realtime-checkup \
	-e KUBECONFIG=/root/.kube/kubeconfig \
	-e TEST_NAMESPACE=realtime-checkup-1 \
	-e TEST_CHECKUP_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup:devel \
	-e VM_UNDER_TEST_CONTAINER_DISK_IMAGE=quay.io/kiagnose/kubevirt-realtime-checkup-vm:main \
	docker.io/library/golang:1.20.12 \
	go test -v ./tests/... -test.v -test.timeout=1h -ginkgo.v -ginkgo.timeout=1h 
=== RUN   TestTests
Running Suite: Tests Suite - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests
========================================================================================
Random Seed: 1702890031

Will run 1 of 1 specs
------------------------------
[BeforeSuite] 
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/tests_suite_test.go:39
[BeforeSuite] PASSED [0.002 seconds]
------------------------------
Checkup execution should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80
• [821.167 seconds]
------------------------------

Ran 1 of 1 Specs in 821.169 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestTests (821.17s)
PASS
ok  	github.com/kiagnose/kubevirt-realtime-checkup/tests	821.178s

logs:

$ oc logs -f job/realtime-checkup
2023/12/18 09:00:39 kubevirt-realtime-checkup starting...
2023/12/18 09:00:39 Using the following config:
2023/12/18 09:00:39 	"vmUnderTestTargetNodeName": ""
2023/12/18 09:00:39 	"vmUnderTestContainerDiskImage": "quay.io/kiagnose/kubevirt-realtime-checkup-vm:main"
2023/12/18 09:00:39 	"oslatDuration": "10m0s"
2023/12/18 09:00:39 	"oslatLatencyThresholdMicroSeconds": "45µs"
2023/12/18 09:00:39 Waiting for VMI "realtime-checkup-1/rt-vmi-tvpnb" to boot...
2023/12/18 09:02:54 VMI "realtime-checkup-1/rt-vmi-tvpnb" had successfully booted
2023/12/18 09:02:54 Login to VMI under test...
2023/12/18 09:04:00 Running Oslat test on VMI under test for 10m0s...
2023/12/18 09:14:02 Oslat test completed:
taskset -c 1-2 oslat --cpu-list 1-2 --rtprio 1 --duration 10m0s --workload memmove --workload-mem 4K 
oslat V 2.60
Total runtime: 		600 seconds
Thread priority: 	SCHED_FIFO:1
CPU list: 		1-2
CPU for main thread: 	0
Workload: 		memmove
Workload mem: 		4 (KiB)
Preheat cores: 		2

Pre-heat for 1 seconds...
Test starts...
Test completed.

        Core:	 1 2
Counter Freq:	 2096 2096 (Mhz)
    001 (us):	 0 0
    002 (us):	 6149216625 6192544634
    003 (us):	 1 1
    004 (us):	 108 99
    005 (us):	 40 38
    006 (us):	 8 12
    007 (us):	 1 4
    008 (us):	 1 1
    009 (us):	 2 4
    010 (us):	 11 0
    011 (us):	 14 0
    012 (us):	 1 10
    013 (us):	 1 17
    014 (us):	 0 2
    015 (us):	 0 0
    016 (us):	 0 0
    017 (us):	 0 0
    018 (us):	 0 0
    019 (us):	 0 0
    020 (us):	 0 0
    021 (us):	 0 0
    022 (us):	 0 0
    023 (us):	 0 0
    024 (us):	 0 0
    025 (us):	 0 0
    026 (us):	 0 0
    027 (us):	 0 0
    028 (us):	 0 0
    029 (us):	 0 0
    030 (us):	 0 1
    031 (us):	 0 0
    032 (us):	 0 0 (including overflows)
     Minimum:	 1 1 (us)
     Average:	 2.000 2.000 (us)
     Maximum:	 12 29 (us)
     Max-Min:	 11 28 (us)
    Duration:	 599.704 599.704 (sec)

[root@rt-vmi-tvpnb cloud-user]# 
2023/12/18 09:14:02 Max Oslat Latency measured: 29µs
2023/12/18 09:14:02 Trying to delete VMI: "realtime-checkup-1/rt-vmi-tvpnb"
2023/12/18 09:14:02 Waiting for VMI "realtime-checkup-1/rt-vmi-tvpnb" to be deleted...
2023/12/18 09:14:07 VMI "realtime-checkup-1/rt-vmi-tvpnb" was deleted successfully

pkg/internal/checkup/executor/oslat/client.go Outdated Show resolved Hide resolved

func TestRunSuccess(t *testing.T) {
expecter := &expecterStub{
injectedActualMaxResults: "27 56 (us)",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please consider taking this value as a time.Duration.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep to the original string and not do another conversion.
It's just how the Oslat outputs the results, so it's better to just show it how it is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its meaning is not very clear IMO when reading the test.

pkg/internal/checkup/executor/executor.go Outdated Show resolved Hide resolved
tests/checkup_test.go Outdated Show resolved Hide resolved
pkg/internal/checkup/checkup.go Show resolved Hide resolved
@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 18, 2023

change: Second round review

Introducing a package that will handle the oslat test [0], and the run
method used to run the test.
The Run command runs the oslat test inside the VMI console.
In order to make sure that the test duration does not exceed the checkup
timeout, passing the checkup ctx and wrapping the run command.
The output is checked for successful return value and then parsed for
the
maximum latency value out of the cores that are tested is returned.

the oslat command is run under the taskset command that makes sure that
it runs only on the specified cores under test.
flags used on the oslat command [0]:
- cpu-list: vCPUs under test.
- rtprio: SCHED_FIFO priority (1-99).
- duration: oslat test duration.
- workload: kind of workload tested.
- workload-mem: Size of the memory to use for the workload.

[0] https://www.mankier.com/8/oslat

Signed-off-by: Ram Lavi <[email protected]>
The checkup will now fail if the oslat Maximum latency measured exceeded
the oslatLatencyThreshold, and issue the appropriate failure reason.

Signed-off-by: Ram Lavi <[email protected]>
@orelmisan
Copy link
Member

@RamLavi does the E2E test passes?

@RamLavi
Copy link
Collaborator Author

RamLavi commented Dec 19, 2023

@RamLavi does the E2E test passes?

yes

make e2e-test
mkdir -p /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache
podman run --rm \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup:/go/src/github.com/kiagnose/kubevirt-realtime-checkup:Z \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache:/root/.cache/go-build:Z \
	-v /home/ralavi/.kube/sno03-cnvqe2-rdu2:/root/.kube:Z,ro \
	--workdir /go/src/github.com/kiagnose/kubevirt-realtime-checkup \
	-e KUBECONFIG=/root/.kube/kubeconfig \
	-e TEST_NAMESPACE=realtime-checkup-1 \
	-e TEST_CHECKUP_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup:devel \
	-e VM_UNDER_TEST_CONTAINER_DISK_IMAGE=quay.io/kiagnose/kubevirt-realtime-checkup-vm:main \
	docker.io/library/golang:1.20.12 \
	go test -v ./tests/... -test.v -test.timeout=1h -ginkgo.v -ginkgo.timeout=1h 
=== RUN   TestTests
Running Suite: Tests Suite - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests
========================================================================================
Random Seed: 1702976949

Will run 1 of 1 specs
------------------------------
[BeforeSuite] 
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/tests_suite_test.go:39
[BeforeSuite] PASSED [0.002 seconds]
------------------------------
Checkup execution should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80
• [750.633 seconds]
------------------------------

Ran 1 of 1 Specs in 750.636 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestTests (750.64s)
PASS
ok  	github.com/kiagnose/kubevirt-realtime-checkup/tests	750.646s

Copy link
Member

@orelmisan orelmisan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

@orelmisan orelmisan merged commit 3a93d4e into kiagnose:main Dec 19, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants