… into 'main'

Attach to the dns_dispatchmgr in the dns_view object

Closes #4228

See merge request isc-projects/bind9!8203

The following files were reported in CI by the legacy system test runner
and prevented job to pass. They should be removed.

    $ if git rev-parse > /dev/null 2>&1; then ( ! grep "^I:.*:file.*not removed$" *.log ); fi
    autosign.log:I:autosign:file autosign/ns3/kskonly.example.db.jbk not removed
    autosign.log:I:autosign:file autosign/ns3/optout.example.db.jbk not removed
    autosign.log:I:autosign:file autosign/ns3/reconf.example.db.jbk not removed
    masterformat.log:I:masterformat:file masterformat/ns1/signed.db.raw.jbk not removed
    masterformat.log:I:masterformat:file masterformat/ns1/signed.db.raw.signed not removed
    masterformat.log:I:masterformat:file masterformat/ns1/signed.db.raw.signed.jnl not removed

Don't print an error when the ns*/inactive directory is not
present:

    rmdir: ns*/inactive: No such file or directory

Remove nsupdate.out.test file instead of nsupdate.out, as the latter
does not exist.

Clean leftover files in autosign and masterformat

See merge request isc-projects/bind9!8167

If the flaky plugin for pytest is available, use its decorator to
support re-running unstable tests. In case the package is missing,
execute the test as usual without attempts to re-run it in case of
failure.

This is mostly intended to increase the test stability in CI. Using a
custom decorator enables us to keep the flaky package as an optional
dependency.

In some cases, BIND is not fast enough to fill the send buffer and
manages to answer all queries, contrary to what the test expects.
Repeat the check up to 3 times to limit this test instability.

Make the test_send_timeout check more stable

Closes #4139

See merge request isc-projects/bind9!8168

This reverts commit 4eac32f.

With the v9.19.16 release tag merged, the "cross-version-config-tests"
GitLab CI job will no longer fail due to the two relevant system tests
being absent from the development branch.  This makes the pytest
filtering expression added to work around that issue unnecessary, so
remove it.

Update the release checklist with links to recently implemented tools
for automating certain tasks.

Post-release tweaks (cross-version testing cleanup, release checklist tweaks)

See merge request isc-projects/bind9!8210

A negative or excessively large Content-Length could cause a crash
by making `INSIST(httpd->consume != 0)` fail.

Parse statschannel Content-Length: more carefully

Closes #4125

See merge request isc-projects/bind9!8018

A supported pytest version finally made it to OpenBSD repositories,
allowing us to run system tests with the pytest runner.

Out-of-tree builds are built in a directory that is different from
source directory. The build directory doesn't contain the non-compiled
test files from bin/tests/system which are the test cases required by
the pytest runner.

In order to run the system tests for out-of-tree build, copy over the
contents (tests) of bin/tests/system/ from the source directory into the
build directory. Then, it is possible to invoke the pytest runner inside
the build directory.

Use pytest runner for out-of-tree and OpenBSD system tests in CI

Closes #4246

See merge request isc-projects/bind9!8193

Add release note for [GL #4125]

Closes #4125

See merge request isc-projects/bind9!8216

While temporary directories are useful for test execution to keep
everything clean, they are difficult to work with manually. Create a
symlink for each test artifact directory with a stable and predictable
path. The symlink always either points to the latest artifacts, or is
missing in case the last run succeeded.

Ensure these symlinked directories aren't detected as test suites by the
pytest runner.

At the end of the test, display the symlink path to the artifact
directory in case it's preserved. Log the full tempdir name in debug
log.

While it'd be fairly easy to split the function up into smaller ones,
the readability wouldn't be improved in this case. Silence the
suggestions instead.

The command finds all directories in bin/tests/system which contain an
underscore. Underscore indicates either a temporary directory (_tmp_), a
symlink to test artifacts (TESTNAME_MODULENAME), or a python-related
cache. Using underscore for a system test name is invalid and a hyphen
must be used instead.

Create symlinks to test artifacts for pytest runner

Closes #4252

See merge request isc-projects/bind9!8194

Instead of growing and never shrinking the list of the inactive
handles (to be reused mostly on the UDP connections), limit the number
of maximum number of inactive handles kept to 64.  Instead of caching
the inactive handles for all listening sockets, enable the caching on on
UDP listening sockets.  For TCP, the handles were cached for each
accepted socket thus reusing the handles only for long-standing TCP
connections, but not reusing the handles across different TCP streams.

Set the number of maximum free items for the uvreq memory pool to 64.

…'main'

Limit the number of inactive handles and uvreqs kept for reuse

Closes #4265

See merge request isc-projects/bind9!8206

Just replace "bullseye" with "bookworm" and reintroduce Debian 11
"bullseye" later.

Debian 12 has Perl 5.36 and the option is no longer needed.

We are not concerned by the "input text line longer than 80 bytes"
stylistic warning.

The gcc:sid:amd64 job was used to test PKCS11 via the OpenSSL engine API
but is now designated for future PKCS11 testing via the provider API.

The ability to use PKCS11 via engine API in clang:bookworm:amd64 has
been restored using only deprecated APIs in OpenSSL 3.

Many problems of the Debian 11 gcovr version were fixed in the Debian 12
one. Replace workarounds we accumulated over the years with two new,
simple ones.

Move clang:bullseye:amd64 PKCS11 testing to the gcc:bullseye:amd64 job
to evenly represent compilers in PKCS11 testing.

Make Debian 12 "bookworm" the base image

See merge request isc-projects/bind9!8075

This adds support for User Statically Defined Tracing (USDT).  On
Linux, this uses the header from SystemTap and dtrace utility, but the
support is universal as long as dtrace is available.

Also add the required infrastructure to add probes to libisc, libdns and
libns libraries, where most of the probes will be.

Add tracing probes to ISC own isc_rwlock implementation to allow
fine-grained tracing.  The pthread rwlock already has probes inside
glibc, and it's difficult to add probes to headers included from the
other libraries.

Add tracing probes to isc_job unit:

 * libisc:job_cb_before - before the job callback is called
 * libisc:job_cb_after - after the job callback is called

Add tracing probes to incoming transfers, so we can accurately measure
the individual events when sending, receiving and parsing the incoming
transfers.

Add a trace point that would report when a query gets dropped or slipped
by rate limits. It reports the client IP, the zone, and the RRL result
code.

Co-authored-by: Paul Frieden <[email protected]>

We already print the formatted zone name via the xfrin_log() function,
generate the text once and store it in xfr->info static buffer.  Then we
can reuse the string to pass it to the LIBDNS_XFRIN probes.

Add support for userspace tracing via USDT

Closes #4041

See merge request isc-projects/bind9!7872

Using check_PROGRAMS would postpone compiling the binaries needed by
system tests until `make check` would be called. Since it's preferable
to invoke pytest directly to run the system test suite, compile these
binaries without installing them during `make all` instead by using
noinst_PROGRAMS.

This removes the need to use TESTS= make -e check hack invoked from
pytest to work around this issue.

Value stored to 'rdatap' is never read. Remove the line which has no
effect.

The dnsrps build assumes that dlopen is available from libc. This
assumption isn't met when attempting to build with address or thread
sanitizer on debian (bullseye, bookworm). Disable dnsrps build to avoid
the issue, since it's not needed anyway for our respdiff tests.

See commit b396f55 for more info about
the dnsrps build.

Compile system test binaries during make

Closes #4249

See merge request isc-projects/bind9!8189

when synthesizing a new CNAME, we now check whether the target
matches the query already being processed. if so, we do not
restart the query; this prevents a waste of resources.

prevent query_coveringnsec() from running twice

See merge request isc-projects/bind9!8214

The DTrace on macOS only needs header file and the dtrace command
doesn't have the -G option.  Skip generating the object file on macOS,
so the build doesn't fail.

The DTrace generated files were missing dependency on the Makefile, so
they didn't get regenerated when ./configure was re-run.  This would
create problem especially between ./configure --enable-tracing vs
./configure --disable-tracing invocations.

Don't generate the probes.lo on macOS

Closes #4275

See merge request isc-projects/bind9!8234

Default servers usually don't have DoT set up. Only do SOA queries if
the server is specified.

Don't do TCP SOA requests for default servers

Closes #1181

See merge request isc-projects/bind9!8192

The dtrace command fails with segmentation fault in the FIPS-enabled
builds, thus we need to disable the tracing in the CI for the FIPS
jobs.

Disable tracing in the FIPS enabled builds

Closes #4271

See merge request isc-projects/bind9!8235

checkds test requires the capture_output argument for subprocess.run()
which was added in Python 3.7.

The _last_test_run entry was accidentally added in !8194. It came from a
work-in-progress version of the MR and was left there during a rebase.

The commands in after_script run in a new shell, separate from
before_script and script commands. Since the pytest.log.txt is for out
of tree tests is present in the build directory, switch to it (if set)
before running any postprocessing commands.

It is better to disable the specific check that causes the test to fail
rather than mark the entire test as xfail, which can mask other issues
which the test is capable of detecting.

The qmin test is inherently unstable. It fails quite often with failure
modes described in GL #904. Allow the pytest runner to re-run the test
up to 3 times to only detect a more persistent and reproducible failures
rather than random noise caused by the nature of the test.

The reclimit system test has been unstable and producing false positive
results for years (GL #1587). Allow the test to be re-run (once) to
reduce the noise it causes.

The rrl system test has been unstable and producing false positive
results for years (GL #172). Allow the test to be re-run (once) to
reduce the noise it causes.

system tests stability tweaks and cleanup

Closes #3001

See merge request isc-projects/bind9!8239

There used to be an extra layer of indirection in the memory functions
for certain dynamic linking scenarios. This involved variant spellings
like isc__mem and isc___mem. The isc___mem variants were removed in
commit 7de8469 so the token pasting is no longer needed and
only serves to obfuscate.

CHECKED_MEM_GET and ZERO_PTR are built-in features of isc_mem.

Mention the element size explicitly, so that `matching_sigs()`,
`signed_loose()`, and `signed_strict()` use the same calculation.

Remove some unnecessary token pasting macrology

See merge request isc-projects/bind9!8236

Instead of duplicating the destroy_forwarders() code in the cleanup
sections, just call dns_forwarders_detach() every time - in case of
failure, the forwarders aren't going to be attached, and forwarders
object will be destroyed right away.

Refactor cleanup code in the qp-trie for the forwarders table

See merge request isc-projects/bind9!8212

Move i386 and other less common or ancient CPU architectures to
Community-Maintened category.  Move armhf and arm64 to the Best-Effort
category as we do test them as part of development work (new MacBooks
are all arm64), we don't really do full set of tests in the CI.

…ain'

Move i386 and armhf to the Best-Effort category

See merge request isc-projects/bind9!8223

The new test sends a 64 KiB message over TCP to named, and expects
it to handle it correctly.

The isc_dnsstream_assembler_incoming() inline function expects that
when 'buf_size' is zero, then 'buf' must be NULL. The expectation is
not correct, because those values come from the libuv read callback,
and its documentation notes[1] that 'nread' ('buf_size' here) might
be 0, which does not indicate an error or EOF, but is equivalent to
EAGAIN or EWOULDBLOCK under read(2).

Change the isc_dnsstream_assembler_incoming() inline function to
remove the invalid expectation.

[1] https://docs.libuv.org/en/v1.x/stream.html#c.uv_read_cb

Before calling isc_buffer_putmem(), there is a condition to check
that 'buf_size' is greater than 0. At this point 'buf_size' is
guaranteed to be greater than zero, so either the condition is
redundant, or 'unprocessed_size' should be checked instead, which
seems more logical, because calling isc_buffer_putmem() with
'unprocessed_size' being zero is not useful, although harmless.

Resolve "crash while receiving 64 kiB message over TCP"

Closes #4273

See merge request isc-projects/bind9!8231

Due to bug in openssl3, the pkcs11-engine is made the default
provider if enabled. This causes key generation and load to
return legacy objects.

Openssl3 has limited glue and does not support the full set
of new style parameter to be inqueried from legacy key objects

Rewrite required functions to use first the new API (if available),
but fallback to the old API (if available). For the methods that
have proper OpenSSL compatiblity glue, ship only one version.

- Simplify configuration management by deducing SoftHSM module path
  from openssl config

- Determine the engine flag (-E) value from openssl config

- Drop unused/unneeded environment variables

- Run pkcs11-provider tests on Debian "sid" ossl3 flavor

Enable keyfromlabel and enginepkcs11 systemtests for pkcs11-provider

See merge request isc-projects/bind9!8170

To resolve the version select and search issue on readthedocs.org,
sphinx_rtd_theme>=1.2.1 is required.

Related readthedocs/sphinx_rtd_theme#1452

Update sphinx_rtd_theme

See merge request isc-projects/bind9!8246

There was a bug in rr_exists that caused it to fail when the
name didn't exist in the zone.

rr_exists errored if the name did not exist in the zone.  This was
not an issue prior to the addition of krb5-subdomain-self-rhs and
ms-subdomain-self-rhs as the only name used was the zone name which
always existed.

…attempted-deletions-of-non-existent-ptr-srv' into 'main'

Resolve "[ISC-support #22588] BIND now returning SERVFAIL for attempted deletions of non-existent PTR/SRV records"

Closes #4280

See merge request isc-projects/bind9!8247

    *** CID 464884:  Null pointer dereferences  (REVERSE_INULL)
    /bin/tests/system/dyndb/driver/db.c: 644 in create_db()
    638
    639     	*dbp = (dns_db_t *)sampledb;
    640
    641     	return (ISC_R_SUCCESS);
    642
    643     cleanup:
       CID 464884:  Null pointer dereferences  (REVERSE_INULL)
       Null-checking "sampledb" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
    644     	if (sampledb != NULL) {
    645     		if (dns_name_dynamic(&sampledb->common.origin)) {
    646     			dns_name_free(&sampledb->common.origin, mctx);
    647     		}
    648
    649     		isc_mem_putanddetach(&sampledb->common.mctx, sampledb,

Silence CID 464884 (REVERSE_INULL)

See merge request isc-projects/bind9!8248

The resolution of the uv_hrtime() function is bigger than the
intervals used in the timers, which can result in an unexpected
difference between the start_time and stop_time variables.

Use isc_loop_now(), which is based on uv_now() and has the same
milliseconds resolution as the functions in the uv_timer_t API.

Also fix a couple wrong numbers in the comments.

Resolve "timer_test unit test intermittently fails in CI"

Closes #4285

See merge request isc-projects/bind9!8253

to the zone code so that it can be used to adjust the expire time.

This allow for the EDNS options EXPIRE and NSID to be sent when
when making requests.  The existing controls controlling whether
EDNS is used and whether EXPIRE or NSID are sent are honoured.

Adjust the expected byte counts in the xfer system test to reflect
the EDNS overhead.  Adjust the dig call to match named's behavior
(don't set +expire as we are talking to a secondary).

If we are talking to a non EDNS aware primary that returns FORMERR
to EDNS requests retry the request without using EDNS.

and use that adjusted time to set the modification times.