[windows] Prepare toolchain build to run tests for LLDB #76894

weliveindetail · 2024-10-07T19:08:37Z

This PR adds handling for the -Test lldb option, which allows to enable the LLDB test suite on Windows builds of the swift toolchain, fixing swiftlang/llvm-project#9141

While we still see a lot of test failures, it's time to give the CI a try. Once we get to a stable subset of tests that pass, we can iterate to enable the remaining ones and increase coverage.

The PR adds a local Windows-specific mechanism to mark bulks of tests as XFAIL or SKIPPED. These tests are listed in file utils/windows-llvm-lit-test-overrides.txt. The goal is to get it down to a low number and remove the mechanism and file eventually.

weliveindetail · 2024-10-07T19:09:41Z

This is a preview. We need to land swiftlang/llvm-project#9370 first.

utils/build.ps1

weliveindetail · 2024-10-16T16:07:25Z

@compnerd This should be ready for testing with the companion PR linked above. Do we need to land that or can we ask the CI to checkout llvm-project from that PR instead of the original stable/20230725?

compnerd · 2024-10-16T23:30:23Z

Please test with following PRs:
swiftlang/llvm-project#9370

@swift-ci please test windows platform

weliveindetail · 2024-10-17T10:23:20Z

Oh wow, that works indeed! And sorry, it was easy to confuse. We need the latest linked PR. The other one was merged in the meantime :) But not too bad, I anyway had to patch build-windows-toolchain.bat.

@compnerd I checked locally and I think that the following will work:

Please test with following PRs:
https://github.com/swiftlang/llvm-project/pull/9429

@swift-ci please test windows platform

hjyamauchi · 2024-10-23T15:45:38Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test windows platform

weliveindetail · 2024-10-24T08:23:46Z

Great, thanks. Good news is that build.ps1 now runs with testing enabled for LLDB. Learning is that the build machine doesn't have winget, so we have to find another way to install make.exe from [GnuWin32]:(https://gnuwin32.sourceforge.net/packages/make.htm):

swift\utils\build.ps1 : The term 'winget' is not recognized as 
the name of a cmdlet, function, script file, or operable program.

Edit: We can just download and install it like we do for Python, WiX, FlexBison, etc.

hjyamauchi · 2024-10-24T16:42:30Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test windows platform

weliveindetail · 2024-10-24T19:13:59Z

The build failed with the following error in build step cmake.exe --build T:\5 --target distribution:

[4797/6491][ 73%][1334.403s] Linking Swift static library lib\swiftASTGen.lib
...
swift\lib\ASTGen\Sources\ASTGen\SourceFile.swift:69:45: error: type 'Parser.ExperimentalFeatures' has no member 'sendingArgsAndResults'
    mapFeature(.SendingArgsAndResults, to: .sendingArgsAndResults)
                                           ~^~~~~~~~~~~~~~~~~~~~~

This doesn't appear to be related to my changes. Is it due to the merge conflict in utils/build-windows-toolchain.bat maybe? It's not related either, but does it cause the bot to silently skip the merge and build my (outdated) branch against newer dependencies? Let me resolve that.

hjyamauchi · 2024-10-24T19:17:44Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

weliveindetail · 2024-10-25T07:15:15Z

Great, tests are running!

Testing Time: 202.24s
  Excluded           :   18
  Unsupported        : 1187
  Passed             : 2058
  Expectedly Failed  :   45
  Unresolved         :    1
  Failed             :   56
  Unexpectedly Passed:    4

SwiftREPL failures (28) will be fixed with swiftlang/llvm-project#9416, the others I will investigate today.

weliveindetail · 2024-10-25T10:49:45Z

In these NativePDB tests, a line breakpoint is set and it gets resolved to a location that seems right, but at startup time we add another location and that fails the test. This might be an actual bug:

  lldb-shell :: SymbolFile/NativePDB/local-variables.cpp
  lldb-shell :: SymbolFile/NativePDB/stack_unwinding01.cpp

The tests in 25f5eca should be fixed and they pass reliably on my machine, but they failed in the CI test run. Let's see if they XPASS next time, because the companion PR in llvm-project now has the SDKROOT patch: swiftlang/llvm-project@62b9626 Otherwise we might be missing a cherry-pick.

And for the record: The following test failed but the detailed report is lacking in the log! 🤷‍♂️ This is the only unreasonable difference to the log from the companion PR.

  lldb-api :: source-manager/TestSourceManager.py

weliveindetail · 2024-10-25T10:58:22Z

The latest two commits integrate the changes from PRs #76974 and #77005 into this one. I would usually prefer small isolated changes, but the per-merge infrastructure makes it unbearable. I will abaondon the other PRs and only continue with this one.

hjyamauchi · 2024-10-25T16:30:28Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

weliveindetail · 2024-10-28T10:06:18Z

SwiftREPL tests keep failing:

$ "t:\5\bin\lldb.exe" "--no-lldbinit" "-S" "T:/5/tools/lldb\test\Shell\lit-lldb-init-quiet" "--repl"
error: failed to stop process at REPL breakpoint

This means that the dummy executable for the REPL didn't load. It usually happens if it lacks a runtime dependency like swiftCore: https://github.com/swiftlang/llvm-project/blob/stable/20230725/lldb/tools/repl/swift/main.c#L66

weliveindetail · 2024-10-28T10:08:42Z

Otherwise only windows-unaligned-x86_64 failed this time:

llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test:20:10: error: CHECK: expected string not found in input
# CHECK: stop reason = breakpoint 1.1
         ^
<stdin>:9:61: note: scanning from here
Breakpoint 1: where = windows-unaligned-x86_64.test.tmp`func at windows-unaligned-x86_64.cpp:3, address = 0x0000000140001000
                                                            ^
<stdin>:13:14: note: possible intended match here
* thread #1, stop reason = breakpoint 1.2
             ^

This looks very similar to the problem we saw in the NativePDB tests last time. windows-unaligned-x86_64 would have failed for the same reason already, but it showed up as UNSUPPORTED back then. This indicates a flake at build-time of the inferior. It would be a pity if we had to skip it due to these issues, because there is a good chance that the actual bug behind it was fixed with llvm/llvm-project#101710

hjyamauchi · 2024-10-28T15:11:06Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

hjyamauchi · 2024-11-04T17:19:16Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

weliveindetail · 2024-11-05T08:21:28Z

The builds failed with compile errors that indicate a LLVM version mismatch. I guess we have to build against stable/20240723 now. I will prepare a new companion PR for that.

weliveindetail · 2024-11-05T16:45:26Z

Can we run again with this PR please? This time we should reach the tests.
swiftlang/llvm-project#9513

hjyamauchi · 2024-11-05T17:13:33Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test

utils/build.ps1

weliveindetail · 2024-11-06T10:04:55Z

@compnerd Thanks for your feedback.
@hjyamauchi Unfortunately, the bots didn't run. Can we try again please? We only need Windows platform right now.

Please test with following PRs:
https://github.com/swiftlang/llvm-project/pull/9513

@swift-ci please test windows platform

weliveindetail · 2025-01-17T15:37:38Z

@swift-ci please test Windows platform

weliveindetail · 2025-01-18T15:42:12Z

Apparently, tests failed but Swift CI is not reachable.

weliveindetail · 2025-01-20T09:14:40Z

ci.swift.org is back online, but ci-external.swift.org still down

weliveindetail · 2025-01-20T17:58:02Z

One test failed in first cycle. I am afraid we have to skip all lldb-server tests:

lldb-api :: tools/lldb-server/register-reading/TestGdbRemoteGPacket.py

weliveindetail · 2025-01-20T18:03:05Z

@swift-ci please test Windows platform

weliveindetail · 2025-01-21T08:08:05Z

The build failed because one test failed during setup:

Unresolved Tests (1):
  lldb-api :: lang/swift/async/tasks/TestSwiftTaskBacktrace.py

In particular, it failed due to error: unknown argument: '-gdwarf' in the following command for building the test subject:

T:/5/bin/swiftc \
  -emit-object C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\llvm-project\lldb\test\API\lang\swift\async\tasks/main.swift     \
  -output-file-map T:\5\lldb-test-build.noindex\lang\swift\async\tasks\TestSwiftTaskBacktrace.test_dwarf/a-output-file-map.json      \
  -gdwarf -Onone -Xfrontend -serialize-debugging-options -parse-as-library -module-name a                                            \
  -module-cache-path T:/5/lldb-test-build.noindex/module-cache-clang\lldb-api -tools-directory "T:\5\bin"                            \
  -emit-module-path T:\5\lldb-test-build.noindex\lang\swift\async\tasks\TestSwiftTaskBacktrace.test_dwarf/a.swiftmodule              \
  -emit-module-interface-path T:\5\lldb-test-build.noindex\lang\swift\async\tasks\TestSwiftTaskBacktrace.test_dwarf/a.swiftinterface \
  -###

Edit: This is just a new test that will need triaging.

weliveindetail · 2025-01-21T09:22:42Z

@swift-ci please test Windows platform

weliveindetail · 2025-01-21T14:57:14Z

Tests passed. Let's see if this is reproducible.

@swift-ci please test Windows platform

weliveindetail · 2025-01-22T09:54:02Z

Tests passed. Let's try a few more times, before landing this.

@swift-ci please test Windows platform

…lakes more efficiently

weliveindetail · 2025-01-23T10:37:34Z

Polished, squashed and rebased changes. And I split off the CI batch file change that actually enables the tests. (The temporary [dev] commit is only here to keep exercising the CI.) I will prepare a second PR that enables the tests. In case we clog Swift PR testing, we can easily revert that one without loosing all the infrastructure.

weliveindetail · 2025-01-23T10:42:17Z

@swift-ci please test Windows

weliveindetail · 2025-01-24T13:21:24Z

Tests passed. Looks like this could land on Monday next week.

@swift-ci please test Windows

weliveindetail · 2025-01-27T07:35:53Z

Build failure was unrelated: swift-driver configuration failed due to duplicate definition of imported target

CMake Error at T:/5/lib/cmake/llvm/LLVMExports.cmake:42 (message):
  Some (but not all) targets in this export set were already defined.

  Targets Defined: LLVMDemangle

weliveindetail · 2025-01-27T07:36:12Z

@swift-ci please smoke test

weliveindetail · 2025-01-27T11:21:36Z

Tests passed on Linux and macOS. Windows failed due to a new test lang/swift/async/tasks/TestSwiftTaskSelect.py. Reported with swiftlang/llvm-project#9887, adding a test override for it.

weliveindetail · 2025-01-27T11:27:11Z

New goal: land this today and start testing Tuesday morning

@swift-ci please test Windows

…detect flakes more efficiently" This reverts commit 2ec68d3.

weliveindetail · 2025-01-27T15:12:56Z

Tests passed. I have to re-run the bots after reverting the temporary PR-testing changes. Let's get this PR landed and start testing tomorrow morning by merging #78844 🤞

@swift-ci please smoke test

weliveindetail · 2025-01-27T22:12:47Z

@swift-ci please smoke test macOS

weliveindetail · 2025-01-27T22:14:01Z

@swift-ci please smoke test Linux

weliveindetail commented Oct 10, 2024

View reviewed changes

utils/build.ps1 Outdated Show resolved Hide resolved

weliveindetail commented Oct 16, 2024

View reviewed changes

utils/build.ps1 Outdated Show resolved Hide resolved

weliveindetail mentioned this pull request Oct 16, 2024

🍒 Cherrypicks to allow Swift to enable LLDB tests in Windows builds swiftlang/llvm-project#9429

Closed

weliveindetail marked this pull request as ready for review October 16, 2024 16:04

weliveindetail requested a review from compnerd as a code owner October 16, 2024 16:04

weliveindetail requested review from artemcm and tshortli as code owners November 4, 2024 14:54

weliveindetail mentioned this pull request Nov 5, 2024

Allow Swift to enable LLDB tests in Windows CI swiftlang/llvm-project#9513

Merged

compnerd reviewed Nov 5, 2024

View reviewed changes

utils/build.ps1 Outdated Show resolved Hide resolved

utils/build.ps1 Outdated Show resolved Hide resolved

utils/build.ps1 Outdated Show resolved Hide resolved

utils/build.ps1 Outdated Show resolved Hide resolved

weliveindetail added 2 commits January 23, 2025 11:12

[windows] Prepare toolchain build to run tests for LLDB

65f198e

[dev] Enable tests for LLDB in CI and run them many times to detect f…

2ec68d3

…lakes more efficiently

weliveindetail force-pushed the kendal/lldb-tests branch from 45059cb to 2ec68d3 Compare January 23, 2025 10:15

weliveindetail changed the title ~~[build.ps1] Enable LLDB tests in Windows builds~~ [windows] Prepare toolchain build to run tests for LLDB Jan 23, 2025

weliveindetail mentioned this pull request Jan 23, 2025

[windows] Enable tests for LLDB in Swift CI #78844

Merged

Stop dumping unsupported tests. It's too verbose for production.

d59a8bf

Skip test lang/swift/async/tasks/TestSwiftTaskSelect.py

69d9c0c

Revert "[dev] Enable tests for LLDB in CI and run them many times to …

afa9373

…detect flakes more efficiently" This reverts commit 2ec68d3.

weliveindetail enabled auto-merge (squash) January 27, 2025 15:12

weliveindetail merged commit 459e592 into swiftlang:main Jan 28, 2025
3 checks passed

weliveindetail deleted the kendal/lldb-tests branch January 28, 2025 08:52

[windows] Prepare toolchain build to run tests for LLDB #76894

[windows] Prepare toolchain build to run tests for LLDB #76894

Conversation

weliveindetail commented Oct 7, 2024

weliveindetail commented Oct 7, 2024

weliveindetail commented Oct 16, 2024

compnerd commented Oct 16, 2024

weliveindetail commented Oct 17, 2024

hjyamauchi commented Oct 23, 2024

weliveindetail commented Oct 24, 2024 • edited Loading

hjyamauchi commented Oct 24, 2024

weliveindetail commented Oct 24, 2024

hjyamauchi commented Oct 24, 2024

weliveindetail commented Oct 25, 2024 • edited Loading

weliveindetail commented Oct 25, 2024

weliveindetail commented Oct 25, 2024

hjyamauchi commented Oct 25, 2024

weliveindetail commented Oct 28, 2024

weliveindetail commented Oct 28, 2024

hjyamauchi commented Oct 28, 2024

hjyamauchi commented Nov 4, 2024

weliveindetail commented Nov 5, 2024

weliveindetail commented Nov 5, 2024

hjyamauchi commented Nov 5, 2024

weliveindetail commented Nov 6, 2024

weliveindetail commented Jan 17, 2025

weliveindetail commented Jan 18, 2025

weliveindetail commented Jan 20, 2025

weliveindetail commented Jan 20, 2025

weliveindetail commented Jan 20, 2025

weliveindetail commented Jan 21, 2025 • edited Loading

weliveindetail commented Jan 21, 2025

weliveindetail commented Jan 21, 2025

weliveindetail commented Jan 22, 2025

weliveindetail commented Jan 23, 2025

weliveindetail commented Jan 23, 2025

weliveindetail commented Jan 24, 2025

weliveindetail commented Jan 27, 2025

weliveindetail commented Jan 27, 2025

weliveindetail commented Jan 27, 2025 • edited Loading

weliveindetail commented Jan 27, 2025 • edited Loading

weliveindetail commented Jan 27, 2025

weliveindetail commented Jan 27, 2025 • edited Loading

weliveindetail commented Jan 27, 2025

weliveindetail commented Oct 24, 2024 •

edited

Loading

weliveindetail commented Oct 25, 2024 •

edited

Loading

weliveindetail commented Jan 21, 2025 •

edited

Loading

weliveindetail commented Jan 27, 2025 •

edited

Loading

weliveindetail commented Jan 27, 2025 •

edited

Loading

weliveindetail commented Jan 27, 2025 •

edited

Loading