Switch to Swift Testing #55
Assignees
Labels
testing
Includes all kinds of automated tests, the way that we run them and the infrastructure around them.
Comments
lawrence-forooghian
added
the
testing
Closed
lawrence-forooghian
added a commit
that referenced
this issue
Sep 18, 2024
This means: - using it for development (which will allow us to use Swift Testing, which we’ll do in #55) - requiring users of the library to use it (so that we can use Swift 6 features, which we’ll do in #56) — we’ve decided we’re happy to do this since from April 2025 Apple will force users wishing to upload to the App Store to use it anyway Whilst implementing this, I decided to revisit my comment from 646c220: > I haven’t committed the `.swiftpm/configuration/Package.resolved` file > created by Xcode 16. We already have two Package.resolved files (see the > `comparePackageLockfiles` lint task) and so let’s wait for Xcode 16 to > be released to see whether this file is going to stick around (perhaps > it will become part of the .gitignore created by the SPM that ships with > Xcode 16). With the release version of Xcode 16, I am no longer able to get Xcode to generate this file, so either they changed something or it’s going to reappear at some point when we poke Xcode in some very specific way (at which point we can decide what to do with it). Resolves #21.
lawrence-forooghian
added a commit
that referenced
this issue
Sep 18, 2024
This means: - using it for development (which will allow us to use Swift Testing, which we’ll do in #55) - requiring users of the library to use it (so that we can use Swift 6 features, which we’ll do in #56) — we’ve decided we’re happy to do this since from April 2025 Apple will force users wishing to upload to the App Store to use it anyway Whilst implementing this, I decided to revisit my comment from 646c220: > I haven’t committed the `.swiftpm/configuration/Package.resolved` file > created by Xcode 16. We already have two Package.resolved files (see the > `comparePackageLockfiles` lint task) and so let’s wait for Xcode 16 to > be released to see whether this file is going to stick around (perhaps > it will become part of the .gitignore created by the SPM that ships with > Xcode 16). With the release version of Xcode 16, I am no longer able to get Xcode to generate this file, so either they changed something or it’s going to reappear at some point when we poke Xcode in some very specific way (at which point we can decide what to do with it). Resolves #21.
lawrence-forooghian
added a commit
that referenced
this issue
Sep 19, 2024
We’re doing this because it’s, I guess, the future, and offers
performance improvements, but more immediately because it reduces
verbosity when testing `async` functions and properties.
I’ve done the migration in a pretty straightforward fashion, just
stripping the `test` prefix from the method names (which my eyes are
still getting used to), and not trying to use any new features like test
descriptions, nested suites, or tags. We can figure out how we want to
use these as we get used to using Swift Testing.
Decided to keep testing for errors via a do { … } catch { … } instead of
using #expect(…, throws:) because I like being able to write tests in a
Given / When / Then fashion — i.e. do things, then make assertions.
As an aside, I noticed that some of Swift Testing’s useful test failure
messages stop being so useful when testing asynchronous code, which
takes a bit of the shine off it. For example, if you write
> #expect(status.current == .attached)
then you’ll get a failure message of
> Expectation failed: (current → .detached) == .attached
that is, it shows you information about `current` which helps you to
understand why the expectation failed.
But if, on the other hand, you write
> #expect(await status.current == .attached)
then you’ll get a failure message of
> Expectation failed: await status.current == .detached
which is less useful. If we decide that this is a dealbreaker, then
we’ll need to switch back to the current way of doing things; that is,
first do the `await` and then the #expect. (Hopefully it’s something
that might improve in Swift Testing with time; I’ve asked about it in
[1].)
Resolves #55.
[1] https://forums.swift.org/t/expectation-failure-messages-are-less-useful-with-await/74754
lawrence-forooghian
added a commit
that referenced
this issue
Sep 19, 2024
We’re doing this because it’s, I guess, the future, and offers
performance improvements, but more immediately because it reduces
verbosity when testing `async` functions and properties.
I’ve done the migration in a pretty straightforward fashion, just
stripping the `test` prefix from the method names (which my eyes are
still getting used to), and not trying to use any new features like test
descriptions, nested suites, or tags. We can figure out how we want to
use these as we get used to using Swift Testing.
Decided to keep testing for errors via a do { … } catch { … } instead of
using #expect(…, throws:) because I like being able to write tests in a
Given / When / Then fashion — i.e. do things, then make assertions.
As an aside, I noticed that some of Swift Testing’s useful test failure
messages stop being so useful when testing asynchronous code, which
takes a bit of the shine off it. For example, if you write
> #expect(status.current == .attached)
then you’ll get a failure message of
> Expectation failed: (current → .detached) == .attached
that is, it shows you information about `current` which helps you to
understand why the expectation failed.
But if, on the other hand, you write
> #expect(await status.current == .attached)
then you’ll get a failure message of
> Expectation failed: await status.current == .detached
which is less useful. If we decide that this is a dealbreaker, then
we’ll need to switch back to the current way of doing things; that is,
first do the `await` and then the #expect. (Hopefully it’s something
that might improve in Swift Testing with time; I’ve asked about it in
[1].)
Resolves #55.
[1] https://forums.swift.org/t/expectation-failure-messages-are-less-useful-with-await/74754
lawrence-forooghian
added a commit
that referenced
this issue
Sep 19, 2024
We’re doing this because it’s, I guess, the future, and offers
performance improvements, but more immediately because it reduces
verbosity when testing `async` functions and properties.
I’ve done the migration in a pretty straightforward fashion, just
stripping the `test` prefix from the method names (which my eyes are
still getting used to), and not trying to use any new features like test
descriptions, nested suites, or tags. We can figure out how we want to
use these as we get used to using Swift Testing.
Decided to keep testing for errors via a do { … } catch { … } instead of
using #expect(…, throws:) because I like being able to write tests in a
Given / When / Then fashion — i.e. do things, then make assertions.
Aside:
My first attempt led me to believe that Swift Testing’s handy
expectation failure messages stop being so useful when testing
asynchronous code, which took a bit of the shine off it. For example, if
you write
> #expect(status.current == .attached)
then you’ll get a failure message of
> Expectation failed: (current → .detached) == .attached
that is, it shows you information about `current` which helps you to
understand why the expectation failed.
But if, on the other hand, you write
> #expect(await status.current == .attached)
then you’ll get a failure message of
> Expectation failed: await status.current == .detached
which is less useful. But then, I asked about this in [1] and found out
that it gives you the correct failure messages if you put the `await`
outside the #expect, i.e.
> await #expect(status.current == .attached)
The same applies to `try` as well; put it outside the #expect.
Resolves #55.
[1] https://forums.swift.org/t/expectation-failure-messages-are-less-useful-with-await/74754
lawrence-forooghian
added a commit
that referenced
this issue
Sep 19, 2024
We’re doing this because it’s, I guess, the future, and offers
performance improvements, but more immediately because it reduces
verbosity when testing `async` functions and properties.
I’ve done the migration in a pretty straightforward fashion, just
stripping the `test` prefix from the method names (which my eyes are
still getting used to), and not trying to use any new features like test
descriptions, nested suites, or tags. We can figure out how we want to
use these as we get used to using Swift Testing.
Decided to keep testing for errors via a do { … } catch { … } instead of
using #expect(…, throws:) because I like being able to write tests in a
Given / When / Then fashion — i.e. do things, then make assertions.
Aside:
My first attempt led me to believe that Swift Testing’s handy
expectation failure messages stop being so useful when testing
asynchronous code, which took a bit of the shine off it. For example, if
you write
> #expect(status.current == .attached)
then you’ll get a failure message of
> Expectation failed: (current → .detached) == .attached
that is, it shows you information about `current` which helps you to
understand why the expectation failed.
But if, on the other hand, you write
> #expect(await status.current == .attached)
then you’ll get a failure message of
> Expectation failed: await status.current == .detached
which is less useful. But then, I asked about this in [1] and found out
that it gives you the correct failure messages if you put the `await`
outside the #expect, i.e.
> await #expect(status.current == .attached)
The same applies to `try` as well; put it outside the #expect. As far as
I can tell, there’s nothing wrong with doing this 🤷
Resolves #55.
[1] https://forums.swift.org/t/expectation-failure-messages-are-less-useful-with-await/74754
lawrence-forooghian
added a commit
that referenced
this issue
Sep 20, 2024
We’re doing this because it’s, I guess, the future, and offers
performance improvements, but more immediately because it reduces
verbosity when testing `async` functions and properties.
I’ve done the migration in a pretty straightforward fashion, just
stripping the `test` prefix from the method names (which my eyes are
still getting used to), and not trying to use any new features like test
descriptions, nested suites, or tags. We can figure out how we want to
use these as we get used to using Swift Testing.
Decided to keep testing for errors via a do { … } catch { … } instead of
using #expect(…, throws:) because I like being able to write tests in a
Given / When / Then fashion — i.e. do things, then make assertions.
Whilst working on this, I noticed that some of Swift Testing’s useful
test failure messages stop being so useful when testing asynchronous
code, which takes a bit of the shine off it. For example, if you write
> #expect(status.current == .attached)
then you’ll get a failure message of
> Expectation failed: (current → .detached) == .attached
that is, it shows you information about `current` which helps you to
understand why the expectation failed.
But if, on the other hand, you write
> #expect(await status.current == .attached)
then you’ll get a failure message of
> Expectation failed: await status.current == .detached
which is less useful. I asked about this in [1] and was told that it’s a
known issue and that Swift macro limitations mean it’s unlikely to be
fixed soon. ([2] is a similar question that I found after.) If we decide
that this is a dealbreaker and that we want the rich failure messages,
then we’ll need to switch back to the current way of doing things; that
is, first do the `await` and then the #expect.
Resolves #55.
[1] https://forums.swift.org/t/expectation-failure-messages-are-less-useful-with-await/74754
[2] https://forums.swift.org/t/try-expect-throwing-or-expect-try-throwing/73076/17
lawrence-forooghian
added a commit
that referenced
this issue
Sep 23, 2024
We’re doing this because it’s, I guess, the future, and offers
performance improvements, but more immediately because it reduces
verbosity when testing `async` functions and properties.
I’ve done the migration in a pretty straightforward fashion, just
stripping the `test` prefix from the method names (which my eyes are
still getting used to), and not trying to use any new features like test
descriptions, nested suites, or tags. We can figure out how we want to
use these as we get used to using Swift Testing.
Decided to keep testing for errors via a do { … } catch { … } instead of
using #expect(…, throws:) because I like being able to write tests in a
Given / When / Then fashion — i.e. do things, then make assertions.
Whilst working on this, I noticed that some of Swift Testing’s useful
test failure messages stop being so useful when testing asynchronous
code, which takes a bit of the shine off it. For example, if you write
> #expect(status.current == .attached)
then you’ll get a failure message of
> Expectation failed: (current → .detached) == .attached
that is, it shows you information about `current` which helps you to
understand why the expectation failed.
But if, on the other hand, you write
> #expect(await status.current == .attached)
then you’ll get a failure message of
> Expectation failed: await status.current == .detached
which is less useful. I asked about this in [1] and was told that it’s a
known issue and that Swift macro limitations mean it’s unlikely to be
fixed soon. ([2] is a similar question that I found after.) If we decide
that this is a dealbreaker and that we want the rich failure messages,
then we’ll need to switch back to the current way of doing things; that
is, first do the `await` and then the #expect.
Resolves #55.
[1] https://forums.swift.org/t/expectation-failure-messages-are-less-useful-with-await/74754
[2] https://forums.swift.org/t/try-expect-throwing-or-expect-try-throwing/73076/17
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Requires #21 first.
┆Issue is synchronized with this Jira Task by Unito
The text was updated successfully, but these errors were encountered: