Cadence Testing framework feedback

[austinkline]

I'm not sure which repo to put this in, so I figured I'd throw up a discussion! I'm really excited about the cadence testing framework, I've always been annoyed that I need to go through lots of other setup steps to get things like overflow and flow-js-testing to work. They aren't hard to do, but then I'm switching IDEs (I like jetbrains more than vs code for go and node). I also find it distracts and makes things harder to follow when you have to look through tests in one language, but the actual content I care about are the scripts. So all around it makes a lot of sense to me to focus on using and improving cadence's built-in testing

I recently started giving the cadence testing framework a try, and have a number of feedback items that I think would make it much more useful. Originally it was a twitter thread but here are the items:

Full VS Code support

import Test isn't valid, so all of its type definitions fail to resolve:

Code Coverage

I noticed that there is a flag for it, but it didn't work for me:

Log statements

It would be great to get all log statements to get printed with some flag when running these tests. As of now, I have to put in a line like this:

Debug breakpoints would be great for this kind of iteration as well 🙂

Access to emitted events after running a transaction

This one is super important imo, I was trying to start using these tests in a contract I'm working on, but I found it really hard to make any resource in a transaction, then find what its ID was.

Control of the backend system's clock for testing out long-term flows.

I can't use this testing framework to verify any of Flowty's loan/rental systems because we time things based on the current block's timestamp. We need a way to fast forward so we can test those kinds of things

Ability to reference contract types

It would be fantastic to be able to reference the contract types I've defined in my contracts so I can return them in scripts. I wasn't able to get that to work but maybe I'm just not doing the right thing. If this is indeed not supported, only being able to use cadence "primitive" types (not sure the right term here) is pretty limiting. For instance, I'd like to be able to resolve MetadataViews to test for compliance with the NFT Catalog. I could do that in a script, I suppose, maybe that's the suggested approach?

Run some function after every test, or a way to snapshot and restore state

Based on documentation, tearDown runs after all tests have been run, but I like to write my tests in such a way that each one is its own isolated sandbox. Right now, I don't have an easy way to basically "reset" the state of the chain before running every test. It's a little weird to me that I can't reset the state of the backend being used to ensure I'm operating on a clean slate

Individual test files with a shared setup

This could also just be me not knowing how to properly use this tool yet, but ideally I could define a setup.cdc and somehow import or have my tests automatically pick that up and use it. Then I can split my setup from my tests and start to define many test files to help inform a reader about what the intent of the test is that they're reading. If we can't split them up, it'll become a pretty heavy mental burden to follow 100+ cadence tests all in one file

[m-Peter]

Hey @austinkline 👋

Very nice write up and the feedback is indeed quite legitimate 👏

Regarding Code Coverage, we are working on it as part of a grant proposal. The issue which you described has been identified and resolved. We are just waiting for a new release of flow-cli (that includes this PR), and you will be able to get coverage report for integration tests. Currently it is only supported for unit tests. The release will come with a few more handy features. Check them out here.

Regarding the feedback about the Cadence Testing Framework in general, we are also working on it as part of another grant proposal. Our starting point was the addition of new matchers, and we will continue with the rest of the ideas described here.

Some of your ideas are already described in the original issue, and we'll try to cater to them to accommodate any use case 🙏
For the rest, we'll have to investigate if they can form part of a follow-up grant.

[turbolent]

Opened onflow/cadence-tools#102 and onflow/cadence-tools#103 to track the language server and test framework improvements.

[m-Peter]

Hey @austinkline , the latest flow-cli should fix the code coverage issue that you faced.

[austinkline]

Thanks for the update! I'll give it a try today

[austinkline]

Adding another suggestion or issue I've run into, I'm sure it's an easy fix. Is there a way to get access to accounts defined in my flow config? I'm writing a test that needs to start by adding flow tokens to an account's balance, but I'm not seeing a way to get access to the right account to do that.

Any ideas?

[m-Peter]

Have you checked the useConfiguration helper method? Could you work around this by creating an account in the tests file, add the desired flow tokens to it, and then deploy whatever contract you want to this newly-created address?

Correct me if I'm wrong, maybe I have misunderstood your use-case here. We can always clarify it offline in our chat group, in order not to spam the people here 😇

[austinkline]

So that helper method is unfortunately not going to cover accounts I don't control the deployment of (like the FlowToken account). How can I mint flow tokens to an account for testing an nft mint, for example?

As for useConfiguration, that method needs a lot of boilerplate which I'm doing now but it doesn't give me access to what I need sometimes. Ideally, there is some kind of method like:
cadence blockchain.getAccount("FlowToken") // returns a Test.Account object which has the FlowToken contract deployed to it

We can always clarify it offline in our chat group, in order not to spam the people here

Offline chats are great, I'm happy to do them! But we should make sure we always summarize findings so that people don't ask for duplicate things. It's hard to gauge demand for features in a non-public setting but I can start making individual posts for things as they come up in the future

[AmarildoGrembi]

hey everyone, if you are using code coverage please spare few seconds of your time to fill this survey to help us make it more feature rich! https://forms.gle/Cw9kqh5JAAur4h6PA

[franklywatson]

HI @AmarildoGrembi - please feel free to share in discord as well! Thanks

[AmarildoGrembi]

I have done so in the developers channel and other Flow related communities. My impression is that it has not reached a bigger audience so far. Any suggestion is much appreciated 🙏

[franklywatson]

My bad, I didn't realize. yeah it's tricky. Let me ask Luke to tweet it thanks

[m-Peter]

Update:

Regarding Full VS Code support:
A fix has been merged: onflow/cadence-tools#111. It will probably be included in the next version bump of cadence-tools/lint & flow-cli 🙏

Regarding Code Coverage:
This is already available and fully functional, for both unit / integration tests, with [email protected]. See example: onflow/cadence#2485

Regarding Log statements:
This is already merged (onflow/cadence-tools#109). It will probably be included in the next version bump of cadence-tools/test & flow-cli 🙏

[m-Peter]

Update:

Regarding Individual test files with a shared setup:
A fix has been merged onflow/flow-cli#1030

[m-Peter]

Update:

Regarding Run some function after every test, or a way to snapshot and restore state:
We have added the necessary beforeEach/afterEach hooks and the blockchain.reset() method: onflow/cadence-tools#128

Regarding Access to emitted events after running a transaction:
We have added this functionality as well: onflow/cadence-tools#113

Regarding Ability to reference contract types:
We have added the ground work for this, we just need to figure out how to allow the import of these types in the test files. Probably we will need to phase the setup of the blockchain in a standalone step, preceding the tests execution.