Skip to content

Commit

Permalink
Refactor random testing framework (#3130)
Browse files Browse the repository at this point in the history
This PR offers a substantial rewrite of the current framework for random integration testing.
The most significant change is that now every "attack method" is derived from `RandomAttackMethod` base class, and implements basic methods such as `log_to_console()`, `apply_to_dtframe()` and `apply_to_pyframe()`. Notably, `MetaFrame` is no longer responsible for performing operations on itself, instead, this functionality has been merged into each attack method. 

These changes make adding new random tests easier: they no longer need to live in 2 separate files and make those files unbearably large. Now, each test method lives in a separate file, and conforms to a strict API, which makes the code more consistent.

Additional changes include:
- An attack method can be skipped using `self.skipped = True`;
- An attack method can declare that it produces an exception: `self.raises = ValueError`, and the framework will catch that exception and verify that it has the correct type/message;
- Evaluation context now has the ability to add deferred checks, which will be automatically run at the end of a testing session.
  • Loading branch information
st-pasha authored Aug 6, 2021
1 parent 052a66e commit bb7d50a
Show file tree
Hide file tree
Showing 22 changed files with 1,397 additions and 628 deletions.
295 changes: 0 additions & 295 deletions tests_random/attacker.py

This file was deleted.

6 changes: 4 additions & 2 deletions tests_random/continuous.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@
# IN THE SOFTWARE.
#-------------------------------------------------------------------------------
#
# For usage information, run:
# Run a battery of randomized datatable tests.
#
# For usage information, see
#
# python tests_random/continuous.py --help
#
Expand Down Expand Up @@ -74,7 +76,7 @@ def start_random_attack(n_attacks=None,
def try_seed(seed, skip_successful_seeds, log_dir, no_forks):
utf8_env = os.environ
utf8_env['PYTHONIOENCODING'] = 'utf-8'
script = os.path.join(os.path.dirname(__file__), "attacker.py")
script = os.path.join(os.path.dirname(__file__), "single.py")
proc = subprocess.Popen([sys.executable, script, str(seed)] +
["--noforks"] * no_forks,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
Expand Down
Loading

0 comments on commit bb7d50a

Please sign in to comment.