deleting "test-results" folder early on by default

[arieluchka-bse]

Hola :)
have noticed that the pytest-playwright uses "test-results" folder by default, and that there is a session scoped fixture that deletes it.

@pytest.fixture(scope="session", autouse=True)
def delete_output_dir(pytestconfig: Any) -> None:
    output_dir = pytestconfig.getoption("--output")
    if os.path.exists(output_dir):
        try:
            shutil.rmtree(output_dir)
        except (FileNotFoundError, PermissionError):
            # When running in parallel, another thread may have already deleted the files
            pass
        except OSError as error:
            if error.errno != 16:
                raise
            # We failed to remove folder, might be due to the whole folder being mounted inside a container:
            #   https://github.com/microsoft/playwright/issues/12106
            #   https://github.com/microsoft/playwright-python/issues/1781
            # Do a best-effort to remove all files inside of it instead.
            entries = os.listdir(output_dir)
            for entry in entries:
                shutil.rmtree(entry)

Fixtures in pytest are not executed at the start of the pytest "lifecycle", but only before the actual test-run.

This can results in an un-wanted deletion of the "test-results" folder.

1. why delete the folder at the first place, and not delete the plugin generated data only?
   Can we maybe (by default) create a "pytest-playwright" folder inside the "test-results", and delete only that? (to not affect other uses of the "test-results" folder)

2. Can we move the folder deletion part to an earlier stage, using pytest hooks for example?

here are some candidates:

root
└── pytest_cmdline_main
    ├── pytest_plugin_registered
    ├── pytest_configure    <--------------------- good candidate
    │   └── pytest_plugin_registered
    ├── pytest_sessionstart <--------------------- good candidate 
    │   ├── pytest_plugin_registered
    │   └── pytest_report_header
    ├── pytest_collection
    │   ├── pytest_collectstart
    │   ├── pytest_make_collect_report
    │   │   ├── pytest_collect_file
    │   │   │   └── pytest_pycollect_makemodule
    │   │   └── pytest_pycollect_makeitem
    │   │       └── pytest_generate_tests
    │   │           └── pytest_make_parametrize_id
    │   ├── pytest_collectreport
    │   ├── pytest_itemcollected
    │   ├── pytest_collection_modifyitems
    │   └── pytest_collection_finish
    │       └── pytest_report_collectionfinish
    ├── pytest_runtestloop
    │   └── pytest_runtest_protocol
    │       ├── pytest_runtest_logstart
    │       ├── pytest_runtest_setup
    │       │   └── pytest_fixture_setup <---------------------------- Folder deletion happens here
    │       ├── pytest_runtest_makereport
    │       ├── pytest_runtest_logreport
    │       │   └── pytest_report_teststatus
    │       ├── pytest_runtest_call
    │       │   └── pytest_pyfunc_call
    │       ├── pytest_runtest_teardown
    │       │   └── pytest_fixture_post_finalizer
    │       └── pytest_runtest_logfinish
    ├── pytest_sessionfinish
    │   └── pytest_terminal_summary
    └── pytest_unconfigure

more on pytest_sessionstart

Fixtures are meant to prepare the environment for the test execution, and the deletion of the folder is not really preparing the tests, and in a stage this late it can (in my case already did :'} ) have unintended effects.

let me know what you think, im open to implementing something and creating a PR :)

[mxschmitt]

Hi, thank you for filing this issue. If this creates some issues for you, happy to adjust the hook and move it somewhere else. Feel free to create a PR and add a regression test. If this is a theoretical issue, I'd try to hold off from now imo.