Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Fedora Image Mode #3229

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/shellcheck.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
exclude-path: |
tests/**
examples/**/test.sh
tmt/steps/execute/scripts/*.sh.j2
tmt/templates/**
token: ${{ secrets.GITHUB_TOKEN }}

Expand Down
10 changes: 10 additions & 0 deletions docs/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,16 @@ TMT_REBOOT_TIMEOUT
How many seconds to wait for a connection to succeed after
guest reboot. By default, it is 10 minutes.


TMT_SCRIPTS_DEST_DIR
Destination directory for storing ``tmt`` scripts on the guest.
By default ``/var/tmp/tmt/bin`` is used. For more information
see the `tmt internal test executor`__ documentation.

__ https://tmt.readthedocs.io/en/stable/spec/plans.html#tmt

.. versionadded:: 1.38

TMT_SSH_*
Every environment variable in this format would be treated as an SSH
option, and passed to the ``-o`` option of ``ssh`` command. See
Expand Down
13 changes: 13 additions & 0 deletions spec/plans/execute.fmf
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,19 @@ description: |
For more information see the
:ref:`/stories/features/abort` feature documentation.

The scripts are hosted by default in the ``/var/tmp/tmt/bin``
directory. The directory can be changed using
the ``TMT_SCRIPTS_DEST_DIR`` environment variable. This directory
works across all supported operating systems, including those using
``rpm-ostree`` and ``bootc``. The path is added to executable paths
using the system-wide ``/etc/profile.d/tmt.sh`` profile script.

.. warning::

Please be aware that the provided scripts will only be available
in a shell that loads the profile scripts. This is the default
for ``bash``-like shells, but might not work for others.

example: |
execute:
how: tmt
Expand Down
135 changes: 110 additions & 25 deletions tmt/steps/execute/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import threading
from contextlib import suppress
from dataclasses import dataclass
from tempfile import NamedTemporaryFile
from typing import TYPE_CHECKING, Any, Optional, TypeVar, Union, cast

import click
Expand All @@ -27,13 +28,15 @@
from tmt.steps.discover import Discover, DiscoverPlugin, DiscoverStepData
from tmt.steps.provision import Guest
from tmt.utils import (
Command,
Path,
ShellScript,
Stopwatch,
field,
format_duration,
format_timestamp,
)
from tmt.utils.templates import render_template_file

if TYPE_CHECKING:
import tmt.cli
Expand All @@ -55,32 +58,101 @@
# Metadata file with details about the current test
TEST_METADATA_FILENAME = 'metadata.yaml'

# Scripts source directory
#: Scripts source directory
SCRIPTS_SRC_DIR = tmt.utils.resource_files('steps/execute/scripts')

#: The default scripts destination directory
SCRIPTS_DEST_DIR = Path("/var/tmp/tmt/bin") # noqa: S108 insecure usage of temporary dir


@dataclass
class Script:
""" Represents a script provided by the internal executor """
"""
Represents a script provided by the internal executor.

Must be used as a context manager. The context manager returns
the source file path.

The source file name matches the name of the file specified via
the ``path`` attribute and must be located in the directory specified
via :py:data:`SCRIPTS_SRC_DIR` variable.
"""

path: Path
aliases: list[Path]
related_variables: list[str]

def __enter__(self) -> Path:
return SCRIPTS_SRC_DIR / self.path.name
happz marked this conversation as resolved.
Show resolved Hide resolved

def __exit__(self, *args: object) -> None:
pass


@dataclass
class ScriptCreatingFile(Script):
""" Represents a script which creates a file """
"""
Represents a script which creates a file.

See :py:class:`Script` for more details.
"""

created_file: str


@dataclass
class ScriptTemplate(Script):
"""
Represents a Jinja2 templated script.

Must be used as a context manager. The context manager returns
the source file path.

The source file name is constructed from the name of the file specified
via the ``path`` attribute, with the ``.j2`` suffix appended.
The template file must be located in the directory specified
via :py:data:`SCRIPTS_SRC_DIR` variable.
"""

context: dict[str, str]

_rendered_script_path: Optional[Path] = None

def __enter__(self) -> Path:
with NamedTemporaryFile(mode='w', delete=False) as rendered_script:
rendered_script.write(render_template_file(
SCRIPTS_SRC_DIR / f"{self.path.name}.j2", None, **self.context))

self._rendered_script_path = Path(rendered_script.name)

return self._rendered_script_path

def __exit__(self, *args: object) -> None:
assert self._rendered_script_path
os.unlink(self._rendered_script_path)


def effective_scripts_dest_dir() -> Path:
"""
Find out what the actual scripts destination directory is.

If the ``TMT_SCRIPTS_DEST_DIR`` environment variable is set, it is used
as the scripts destination directory. Otherwise, the default
of :py:data:`SCRIPTS_DEST_DIR` is used.
"""

if 'TMT_SCRIPTS_DEST_DIR' in os.environ:
return Path(os.environ['TMT_SCRIPTS_DEST_DIR'])

return SCRIPTS_DEST_DIR


# Script handling reboots, in restraint compatible fashion
TMT_REBOOT_SCRIPT = ScriptCreatingFile(
path=Path("/usr/local/bin/tmt-reboot"),
path=effective_scripts_dest_dir() / 'tmt-reboot',
aliases=[
Path("/usr/local/bin/rstrnt-reboot"),
Path("/usr/local/bin/rhts-reboot")],
effective_scripts_dest_dir() / 'rstrnt-reboot',
effective_scripts_dest_dir() / 'rhts-reboot'],
related_variables=[
"TMT_REBOOT_COUNT",
"REBOOTCOUNT",
Expand All @@ -89,43 +161,54 @@ class ScriptCreatingFile(Script):
)

TMT_REBOOT_CORE_SCRIPT = Script(
path=Path("/usr/local/bin/tmt-reboot-core"),
path=effective_scripts_dest_dir() / 'tmt-reboot-core',
aliases=[],
related_variables=[])

# Script handling result reporting, in restraint compatible fashion
TMT_REPORT_RESULT_SCRIPT = ScriptCreatingFile(
path=Path("/usr/local/bin/tmt-report-result"),
path=effective_scripts_dest_dir() / 'tmt-report-result',
aliases=[
Path("/usr/local/bin/rstrnt-report-result"),
Path("/usr/local/bin/rhts-report-result")],
effective_scripts_dest_dir() / 'rstrnt-report-result',
effective_scripts_dest_dir() / 'rhts-report-result'],
related_variables=[],
created_file="tmt-report-results.yaml"
)

# Script for archiving a file, usable for BEAKERLIB_COMMAND_SUBMIT_LOG
TMT_FILE_SUBMIT_SCRIPT = Script(
path=Path("/usr/local/bin/tmt-file-submit"),
path=effective_scripts_dest_dir() / 'tmt-file-submit',
aliases=[
Path("/usr/local/bin/rstrnt-report-log"),
Path("/usr/local/bin/rhts-submit-log"),
Path("/usr/local/bin/rhts_submit_log")],
effective_scripts_dest_dir() / 'rstrnt-report-log',
effective_scripts_dest_dir() / 'rhts-submit-log',
effective_scripts_dest_dir() / 'rhts_submit_log'],
related_variables=[]
)

# Script handling text execution abortion, in restraint compatible fashion
TMT_ABORT_SCRIPT = ScriptCreatingFile(
path=Path("/usr/local/bin/tmt-abort"),
path=effective_scripts_dest_dir() / 'tmt-abort',
aliases=[
Path("/usr/local/bin/rstrnt-abort"),
Path("/usr/local/bin/rhts-abort")],
effective_scripts_dest_dir() / 'rstrnt-abort',
effective_scripts_dest_dir() / 'rhts-abort'],
related_variables=[],
created_file="abort"
)

# Profile script for adding SCRIPTS_DEST_DIR to executable paths system-wide
TMT_ETC_PROFILE_D = ScriptTemplate(
path=Path("/etc/profile.d/tmt.sh"),
aliases=[],
related_variables=[],
context={
'SCRIPTS_DEST_DIR': str(effective_scripts_dest_dir())
})


# List of all available scripts
SCRIPTS = (
TMT_ABORT_SCRIPT,
TMT_ETC_PROFILE_D,
TMT_FILE_SUBMIT_SCRIPT,
TMT_REBOOT_SCRIPT,
TMT_REBOOT_CORE_SCRIPT,
Expand Down Expand Up @@ -597,16 +680,18 @@ def prepare_tests(self, guest: Guest, logger: tmt.log.Logger) -> list[TestInvoca

def prepare_scripts(self, guest: "tmt.steps.provision.Guest") -> None:
""" Prepare additional scripts for testing """
# Create scripts directory
guest.execute(Command("mkdir", "-p", str(effective_scripts_dest_dir())))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using rsync with --mkpath ?
Actually, there is also --executability or --perms/--chmod to get rid of the other ssh command.
(again, can be a separate PR)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martinhoyer rsync --mkpath was added in rsync 3.2.3 (6 Aug 2020). I wonder if it will work on all supported OSes

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I did check that before writing the comment.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That option has to exist for rsync on the guest as well, right? If so we definitely cannot use that as rhel-7 has just rsync-3.1.2


# Install all scripts on guest
for script in self.scripts:
source = SCRIPTS_SRC_DIR / script.path.name

for dest in [script.path, *script.aliases]:
guest.push(
source=source,
destination=dest,
options=["-p", "--chmod=755"],
superuser=guest.facts.is_superuser is not True)
with script as source:
for dest in [script.path, *script.aliases]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thrix Not related to the problem this PR is addressing, but I thought it might be a good idea to bring it up since we're doing changes in this code:
I've noticed the copying of the scripts can take quite some time when copying to remote host and I suspect it could be vastly sped-up if we copy them in batch instead of one at a time.

If you say it's more suitable to handle in separate PR, I'll open an issue to not forget about it :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martinhoyer yes please, not definitely something for this patch :)

guest.push(
source=source,
destination=dest,
options=["-p", "--chmod=755"],
superuser=guest.facts.is_superuser is not True)

def _tmt_report_results_filepath(self, invocation: TestInvocation) -> Path:
""" Create path to test's ``tmt-report-result`` file """
Expand Down
4 changes: 4 additions & 0 deletions tmt/steps/execute/scripts/tmt.sh.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# shellcheck shell=bash

# tmt provides executable scripts under this path
export PATH="{{ SCRIPTS_DEST_DIR }}:$PATH"
18 changes: 1 addition & 17 deletions tmt/steps/provision/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1303,23 +1303,7 @@ def _check_rsync(self) -> CheckRsyncOutcome:
except tmt.utils.RunError:
pass

# Install under '/root/pkg' for read-only distros
# (for now the check is based on 'rpm-ostree' presence)
# FIXME: Find a better way how to detect read-only distros
# self.debug("Check for a read-only distro.")
if self.facts.package_manager == 'rpm-ostree':
self.package_manager.install(
Package('rsync'),
options=tmt.package_managers.Options(
install_root=Path('/root/pkg'),
release_version='/'
)
)

self.execute(Command('ln', '-sf', '/root/pkg/bin/rsync', '/usr/local/bin/rsync'))

else:
self.package_manager.install(Package('rsync'))
self.package_manager.install(Package('rsync'))

return CheckRsyncOutcome.INSTALLED

Expand Down
Loading