Milo Block Integration Test

[TsayAdobe]

In the current Milo test strategy, we primarily use unit tests and Nala tests to evaluate Milo blocks. While Nala tests have UI and rendering checks, they rely on loading pages from a web server, bringing in many unnecessary components. This approach often leads to slower and less stable tests.

To address these issues, I propose adopting a block test approach akin to integration testing. This method involves testing a block in isolation while allowing necessary network calls. It focuses on verifying UI functions and rendering across multiple browsers, ensuring more comprehensive coverage.

The DC team has piloted an integration test prototype with the following strategies:

• Using Web Test Runner: This tool is already familiar to Milo developers, making the adoption smoother.
• Ensuring Essential Services: The test setup includes loading a block’s CSS, permitting dependency service network calls, and resolving CORS issues.
• Leveraging Playwright Automation: This enables testing on multiple browsers - Chromium, Firefox, and Webkit, ensuring cross-browser compatibility.

Advantages of Block Integration Tests:

• Early Issue Detection (Shift Left): These tests help identify problems earlier in the development process, enhancing the speed and reliability by eliminating unnecessary components.
• Comprehensive Evaluation: They effectively detect issues in block functions, rendering, and integration with other services.
• Shared Resources: Utility functions and helpers are utilized across both unit and integration tests, fostering efficiency and consistency.
• Empowering Milo Block Developers: Developers are encouraged to write both unit and block UI tests. This approach ensures accurate rendering across multiple browsers and facilitates immediate visual feedback within VSCode. Additionally, code reviewers can easily observe rendering differences in GitHub PRs. e.g. https://github.com/adobecom/dc/pull/509/files

Check out the pilot tests in the DC repo under /test/integration

[npeltier]

Hey @TsayAdobe may be that's just me, or me being picky on wording but:
testing something in isolation from the rest sounds more like unit testing to me, and testing something integrated with the rest of the page, an ... integration test.
So far the solution we are converging on as a team seems to be :

• web runner tests for unit testing, very close to the code, and the developer,
• nala tests for integration,

I understand you propose a solution in between the pre commit unit testing & the post PR raise nala test. I still think automation is needed in the context you describe (that is block served by EDS), do we agree on this?

[TsayAdobe]

For DC, it's an integration. We integrate code or services from another project. I would like to call them, unit tests, integration tests, and e2e tests. Yes, e2e tests test pages served by EDS.

[honstar]

Thanks @TsayAdobe for starting this discussion!

I'd like to better understand what issues you faced you mentioned here:

While Nala tests have UI and rendering checks, they rely on loading pages from a web server, bringing in many unnecessary components. This approach often leads to slower and less stable tests.

I'm interested in the shift left you mentioned: during the dexter days, we tried to make it easier for developers to actually contribute IT tests by allowing a setup where code to be tested and tests co-exist in the same repository. Is that the case with your proposal?

Can you please make sure to work with the Nala team to avoid working on parallel and potentially competing solutions? Thx

[TsayAdobe]

DC has units, integration, and e2e tests in the test directory. Developers contribute to different test types. Nala has different design principles.

[skumar09]

The proposed description wording is a bit confusing and may lead to misunderstandings about the capabilities of the Nala framework.

• The proposed description echoes a feature already present in the Nala ( or Playwright tool) framework, i.e., Visual testing.
• In Nala, we do visual comparison tests for blocks across various platform and browser combinations, adjusting for different pixel thresholds.
  Here's an example of a sample configuration:

		export default defineConfig({
			expect: {
				toHaveScreenshot: { maxDiffPixels: 100 },		},
		});

• As previously discussed and updated in other forums, the end goal of Nala is to make it a consuming tool (npm package) and move all Nala tests to their respective repositories. ( consuming tool Jira ticket is WIP). In line with this objective, the Nala framework and folder structure are designed to facilitate the easy moving of tests to the corresponding repository.

I see that, the proposed approach and description is more of a unit testing (with stubs or mocks) than integration or E2E UI functional testing.

[TsayAdobe]

The current Milo block tests can be mostly achieved using Web Test Runner. The test cases can be in the Milo repo. This can encourage developers to write block UI and visual tests.