-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
automated: linux: add video output comparison test #533
Open
mwasilew
wants to merge
1
commit into
Linaro:master
Choose a base branch
from
mwasilew:video_compare
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
FROM python:3 | ||
|
||
COPY requirements.txt /home/ | ||
RUN python3 -m pip install opencv-python scikit-image requests | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
# SPDX-License-Identifier: GPL-2.0-only | ||
# Copyright (C) 2024 Qualcomm Inc. | ||
|
||
import cv2 | ||
import numpy as np | ||
import requests | ||
import sys | ||
from argparse import ArgumentParser | ||
from skimage.metrics import structural_similarity | ||
|
||
parser = ArgumentParser() | ||
parser.add_argument("--reference", required=True, help="Reference image path") | ||
auth_group = parser.add_mutually_exclusive_group() | ||
auth_group.add_argument( | ||
"--reference-auth-user", help="Username required to download reference image" | ||
) | ||
parser.add_argument( | ||
"--reference-auth-password", help="Password required to download reference image" | ||
) | ||
auth_group.add_argument( | ||
"--reference-auth-token", help="Token required to download reference image" | ||
) | ||
parser.add_argument( | ||
"--threshold", | ||
required=True, | ||
type=float, | ||
help="Minimal threshold to pass the test. Value must be between 0 and 1", | ||
) | ||
parser.add_argument( | ||
"--lava", | ||
default=True, | ||
action="store_true", | ||
help="Print results in LAVA friendly format", | ||
) | ||
parser.add_argument("--no-lava", dest="lava", action="store_false") | ||
group = parser.add_mutually_exclusive_group(required=True) | ||
group.add_argument("--device", help="Video device to capture image from") | ||
group.add_argument("--image", help="Compared image path") | ||
|
||
args = parser.parse_args() | ||
|
||
if args.threshold < 0 or args.threshold > 1: | ||
print(f"Invalid threshold: {args.threshold}") | ||
sys.exit(1) | ||
|
||
first = None | ||
|
||
if args.reference.startswith("http"): | ||
# download reference image | ||
s = requests.Session() | ||
if args.reference_auth_user and args.reference_auth_password: | ||
s.auth = (args.reference_auth_user, args.reference_auth_password) | ||
if args.reference_auth_token: | ||
s.headers.update({"Authorization": f"Token {args.reference_auth_token}"}) | ||
s.stream = True | ||
print(f"Retrieving reference from: {args.reference}") | ||
first_resp = s.get(args.reference) | ||
data = first_resp.raw.read() | ||
first = cv2.imdecode(np.frombuffer(data, dtype=np.uint8), cv2.IMREAD_COLOR) | ||
|
||
else: | ||
first = cv2.imread(args.reference) | ||
|
||
second = None | ||
|
||
if args.device is not None: | ||
cam = cv2.VideoCapture(args.device) | ||
|
||
if not (cam.isOpened()): | ||
print(f"Could not open video device {args.device}") | ||
sys.exit(1) | ||
|
||
cam.set(cv2.CAP_PROP_FRAME_WIDTH, first.shape[1]) | ||
cam.set(cv2.CAP_PROP_FRAME_HEIGHT, first.shape[0]) | ||
ret, second = cam.read() | ||
cam.release() | ||
cv2.destroyAllWindows() | ||
|
||
if args.image: | ||
second = cv2.imread(args.image) | ||
|
||
first_gray = cv2.cvtColor(first, cv2.COLOR_BGR2GRAY) | ||
second_gray = cv2.cvtColor(second, cv2.COLOR_BGR2GRAY) | ||
|
||
score, diff = structural_similarity(first_gray, second_gray, full=True) | ||
print("Similarity Score: {:.3f}%".format(score * 100)) | ||
print("Required threshold: {:.3f}%".format(args.threshold * 100)) | ||
if score < args.threshold: | ||
print("Test fail") | ||
if args.lava: | ||
print("<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=video_output RESULT=fail>") | ||
else: | ||
print("Test pass") | ||
if args.lava: | ||
print("<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=video_output RESULT=pass>") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
opencv-python | ||
requests | ||
scikit-image |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
#!/bin/sh | ||
|
||
# SPDX-License-Identifier: GPL-2.0-only | ||
# Copyright (C) 2024 Qualcomm Inc. | ||
|
||
# shellcheck disable=SC1091 | ||
. ../../lib/sh-test-lib | ||
OUTPUT="$(pwd)/output" | ||
RESULT_FILE="${OUTPUT}/result.txt" | ||
export RESULT_FILE | ||
REFERENCE_IMAGE="" | ||
REFERENCE_IMAGE_USER="" | ||
REFERENCE_IMAGE_PASSWORD="" | ||
REFERENCE_IMAGE_TOKEN="" | ||
THRESHOLD=0.99 | ||
DEVICE="/dev/video0" | ||
IMAGE="" | ||
|
||
usage() { | ||
echo "\ | ||
Usage: $0 [-p <pkcs11-tool>] [-t <true|false>] [-s <true|false>] [-l <true|false>] | ||
|
||
-r <reference image> | ||
-t <threshold> | ||
-d <device> | ||
-u <reference download username> | ||
-p <reference download password> | ||
-T <reference download token> | ||
-i <image> | ||
" | ||
} | ||
|
||
while getopts "p:t:d:u:r:T:i:h" opts; do | ||
case "$opts" in | ||
p) REFERENCE_IMAGE_PASSWORD="${OPTARG}";; | ||
t) THRESHOLD="${OPTARG}";; | ||
d) DEVICE="${OPTARG}";; | ||
u) REFERENCE_IMAGE_USER="${OPTARG}";; | ||
r) REFERENCE_IMAGE="${OPTARG}";; | ||
T) REFERENCE_IMAGE_TOKEN="${OPTARG}";; | ||
i) IMAGE="${OPTARG}";; | ||
h|*) usage ; exit 1 ;; | ||
esac | ||
done | ||
|
||
! check_root && error_msg "You need to be root to run this script." | ||
|
||
# Test run. | ||
create_out_dir "${OUTPUT}" | ||
|
||
if [ -z "${REFERENCE_IMAGE}" ]; then | ||
error_msg "Reference image missing" | ||
fi | ||
|
||
ARGS="--lava --threshold ${THRESHOLD} --reference ${REFERENCE_IMAGE}" | ||
|
||
if [ -n "${IMAGE}" ]; then | ||
ARGS="${ARGS} --image ${IMAGE}" | ||
fi | ||
|
||
if [ -n "${DEVICE}" ]; then | ||
ARGS="${ARGS} --device ${DEVICE}" | ||
fi | ||
|
||
if [ -n "${REFERENCE_IMAGE_USER}" ] && [ -n "${REFERENCE_IMAGE_PASSWORD}" ]; then | ||
ARGS="${ARGS} --reference-auth-user ${REFERENCE_IMAGE_USER} --reference-auth-password ${REFERENCE_IMAGE_PASSWORD}" | ||
elif [ -n "${REFERENCE_IMAGE_TOKEN}" ]; then | ||
ARGS="${ARGS} --reference-auth-token ${REFERENCE_IMAGE_TOKEN}" | ||
fi | ||
|
||
# shellcheck disable=SC2086 | ||
python3 compare.py ${ARGS} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
metadata: | ||
format: Lava-Test Test Definition 1.0 | ||
name: video-compare | ||
description: | | ||
The test compares two images. The second image | ||
can be either captured from v4l2 device by the script | ||
or delivered to the script from outside. | ||
|
||
Main purpose of the test is to verify video output | ||
using a capture device connected to DUT | ||
|
||
The test script should run on host, where the capture device is connected. | ||
|
||
maintainer: | ||
- [email protected] | ||
os: | ||
- debian | ||
- ubuntu | ||
- centos | ||
- fedora | ||
- openembedded | ||
scope: | ||
- functional | ||
devices: | ||
- imx8mm | ||
- imx6ull | ||
|
||
params: | ||
# https://example.com/image.jpg | ||
REFERENCE_IMAGE: "" | ||
REFERENCE_IMAGE_USER: "" | ||
REFERENCE_IMAGE_PASSWORD: "" | ||
REFERENCE_IMAGE_TOKEN: "" | ||
THRESHOLD: 0.99 | ||
DEVICE: "/dev/video0" | ||
# /home/user/image.jpg | ||
IMAGE: "" | ||
|
||
run: | ||
steps: | ||
- cd ./automated/linux/video/ | ||
- ./video.sh -r "${REFERENCE_IMAGE}" -t "${THRESHOLD}" -d "${DEVICE}" -u "${REFERENCE_IMAGE_USER}" -p "${REFERENCE_IMAGE_PASSWORD}" -T "${REFERENCE_IMAGE_TOKEN}" -i "${IMAGE}" | ||
- ../../utils/send-to-lava.sh ./output/result.txt |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why don't you have all this in requirements.txt and install from there?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It takes 1.5 hours. Both Dockerfile and requirements.txt are not needed for the test. They're added if someone would like to a) build the container to run the test in LAVA, b) run the test locally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, and using requirments.txt in Dockerfile will be tricky. If the requirements.txt changes, the layer cache will not be invalidated.