Do Not Say Non-Functional in QA

Merriam Dictionary's definition of non-functional:

having no function : serving or performing no useful purpose

Background

"Functional" is relative to the goal you're trying to achieve. If your software product is a standalone desktop application, then performance testing might not be "functional" for that purpose, but for a SaaS application running in the cloud serving thousands if not millions of clients simultaneously, then performance becomes critical, not only functional. In the same sense that Selenium tests are not functional for a CLI tool or a web-server etc.

Therefore, the categorization of Selenium tests being "functional" while performance testing is "non-functional" is somewhat arbitrary and creates a lot of confusion.

Rationale

QA professionals, especially those working in Performance Engineering and Testing negates the term Non-Functional. The rationale behind is: the term performance is functional for any application. Similarly for Accessibility Testing, Security Testing, and more.

AFAIK, there is no official term for Non-Functional Testing or Non-Functional Requirements. Industry leaders James Bach, Cem Kaner, Alexander Podelko and more agrees that the term misrepresents the testing objective.

Alternative Terms (in alphabetical order by Term)

Term	Definition	Author
Extra Functional, Quasi-Functional		James Bach
Functional Testing Extended		NaveenKumar Namachivayam
Para-Functional		Cem Kaner
Specialized Functional, Advance Functional, Functional Performance		Sameer Kanguri

Testing Categorization

Individualist vs Collectivist

An Individualist test would evaluate the effect of a single user on the software, e.g.:

• Unit testing
• API testing
• UI testing
• Security testing
• Accessibility testing
• Single user performance testing ( e.g. Google's Lighthouse)

A collectivist test would evaluate the effect of a large number of dispersed users on the software, and this is almost exclusive to the various types of performance tests:

• Stress testing
• Load testing
• Soak testing
• Endurance testing
• Volume testing
• Chaos testing

User Requirements vs Operational Requirements

User Requirements are a type of requirements intended to meet the user expectations e.g. if I try to login with a valid user, the dashboard page should come up with the correct user name.

Examples of such testing realms:

• Unit testing
• API testing
• UI testing

Operation Requirements are a type of requirements referring to industry standards, with the intention to keep the software viable.

Examples:

• Security testing
• Accessibility testing
• Single user performance testing
• Load testing
• Endurance testing
• Soak testing
• Volume testing
• Chaos testing

Assertion based vs Metrics based

Assertions typically validates binary conditions and they provide a quick go or no-go` feedback e.g.:

• Unit testing
• API testing
• UI testing
• Accessibility testing
• Single user performance testing

Metrics are numerical features (response time, CPU usage, memory usage, reset count, etc...) which should be analyzed as a whole after test execution has been completed.

• Load testing
• Endurance testing
• Soak testing
• Volume testing
• Chaos testing

Contributors

Eldad Uzman 📖

sambgm 🖋