Retry Bitrise Pipeline

Summary

The purpose of this document is to show anyone who works within Bitrise to do the following common tasks:

• identify issues reported,
• (re)trigger builds
• unblock common issues. Bitrise grants us the ability to identify common causes of errors and permits reruns targeting failing workflows.

What does a workflow failure look like?

This section illustrates the common failures that could occur at different stages in the pipeline and how we could investigate those failures. Every PR must pass Bitrise before merging.

build_applications

This stage performs the following:

• Builds the app (Firefox or Focus)
• Identifies lint issues
• Finds warnings
• Runs unit tests

Here is an example pipeline that fails at build_applications and no workflows from run_tests are run.

Investigating the Failure

This stage could fail due to one of the steps or Bitrise related issues.

You can determine the cause of the failure by opening the logs:

1. Click the workflow’s name (firefox_configure_build or focus_configure_build).
2. Open the Tests tab for any unit test failures.
   
3. Click Build Log for information for build failures, Swiftlint failures and any Bitrise errors. The same steps could be applied for focus_configure_build.
   

The same steps could be applied for focus_configure_build.

Common workflow remediation

Build Errors

1. Ensure the supported version of Xcode is installed and invoked. See README of the repository for the currently supported version.

   $ xcodebuild -version
   

2. Build the app using the command line.

   $ cd firefox-ios/
   $ xcodebuild build-for-testing -project Client.xcodeproj -scheme Fennec -configuration Fennec -sdk iphonesimulator -destination 'platform=iOS Simulator,name=<device>,OS=<iOS version>'
   

Swiftlint/Warnings

1. Run swiftlint on the command line.

Unit Tests

1. Run the unit tests in question using the command line.

   $ cd firefox-ios/
   $ xcodebuild test -target Client -scheme Fennec -destination 'platform=iOS Simulator,name=<device>,OS=<version>' -only-testing:<test>
   

2. If the unit tests in question passes, run the same tests repeatedly.

   $ xcodebuild test -target Client -scheme Fennec -destination 'platform=iOS Simulator,name=<device>,OS=<version>' -test-iterations 100 -only-testing:<test>
   

If the unrelated unit tests fail intermittently, please retrigger the pipeline (see Retrigger the complete pipeline).

For any issues that are not covered above, please message @mobiletest on Slack.

run_tests workflow

run_tests consists of the XCUITest in the Smoke Tests suite running in parallel. The smoke tests consist of the key functionalities of the app.

How to remediate the failures?

1. Click the failed workflow’s name under run_tests from the pipeline’s page.
   
2. Identify the test’s name.
   
3. You could also click the View Test Reports button to open up the Test Reports. From there you can get more detailed information about the error.
4. Run the particular test using command line command locally.

   xcodebuild test -target Client -scheme Fennec -destination 'platform=iOS Simulator,name=<device>,OS=<version>' -only-testing:XCUITests/<file>/<test> -test-plan Smoketest<number>
   

   For this example, the file would be CreditCardsTests and the test would be testAutofillCreditCardsToggleOnOff.
5. For more information on the test failure reported by Bitrise, you may check the Build log. Scroll down to read the relevant section of the log.
   .
6. If the smoke test passes locally on your computer, please rebuild the unsuccessful workflow instead of rebuilding the whole pipeline.

In the event that the issue persists, please ping @mobiletest on Slack if you are unsure.

Rerun the pipeline and workflows

We can re-run the failing pipeline or workflow without committing a change to the branch.

Rebuild only unsuccessful workflows from Bitrise

If one of the workflows for smoke tests fails, you may run just the failed workflow. (See Test Retries for an explanation on how we handle intermittently failing tests.)

1. Open the pipeline’s page on Bitrise.
2. Open the Rebuild dropdown menu and select Rebuild unsuccessful Workflows.
   
3. Open Bitrise Firefox iOS page. Locate the Manual Approval Builds section.
4. Click Approve and run Build button for your PR.

⛔Important: Please do not re-run any part of the builds via Github. These links from Github do not invoke Bitrise properly.

Retrigger the complete pipeline

If the pipeline fails on build_applications or earlier, you may retrigger the whole pipeline. An example use would be when a unit test fails intermittently.

1. Open the pipeline’s page on Bitrise.
2. Open the Rebuild dropdown menu and select Rebuild entire Pipeline.
   
3. Go back to Bitrise Firefox iOS page. Locate the _Manual Approval Builds section.
4. Click Approve and run Build button for your PR.

⛔Important: Please do not re-run any part of the builds via Github. These links do not invoke Bitrise properly.

Tips and Tricks

Ensure bitrise.yml is Valid

If the Bitrise pipeline fails before the stages are run, there may be a syntax error in the bitrise.yml file. We can do a sanity check on this file by using Bitrise CLI.

1. Install bitrise using brew: brew update && brew install bitrise.
2. Run bitrise validate bitrise.yml.

Test Retries

UI tests could be flaky due to its complexity, the nature of the framework and the external assets outside our control.

In order to prevent flaky issues, we have set up a re-try mechanism.

If a test fails, it runs again up to 3 times. Once it passes, Bitrise general check will be green ✅ and the PR unblocked to be merged.

NOTE that the github check when there is a re-try is not updated and remains red. However, the test run in Bitrise is green as well as the general check.

If the UI test fails 3 out of 3 times, both the test run check is red and the Bitrise general check. This will block the PR to be merged.

At this point, it will be necessary to debug the test and understand why the changes in the PR are causing its failure.

Please feel free to contact @mobiletest whenever you are in this situation and you are unsure about what to do.

Enable Manual Approval Builds Button

When a PR is created, the PR may need approval to run Bitrise. From the Bitrise Firefox iOS page on Bitrise, you may click Approve and run Build button for the PR’s entry to start a build.

If the Manual approval builds button does not appear or does not work for some reason, you may close and reopen the PR or fetch the changes from main to the branch.

Close and Reopen PR

1. Go to the PR’s page on Github.
2. Write a brief comment and click Close pull request.
   
3. Write a brief comment and click Reopen pull request.
   
4. Go to Bitrise Firefox iOS page on Bitrise. Scroll down to the list of running and recent pipelines to ensure that the pipeline is running.

Fetch changes from main

1. Open a terminal.
2. cd to the Firefox iOS repository.
3. Merge the latest changes from main to your branch.
4. Go to the Bitrise Firefox iOS page on Bitrise.
5. Scroll down to the list of running and recent pipelines to ensure that the pipeline is running.