Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Archive failed tests to avoid filling up disc [private instance] #1565

Closed
rposbo opened this issue Oct 4, 2021 · 3 comments
Closed

Archive failed tests to avoid filling up disc [private instance] #1565

rposbo opened this issue Oct 4, 2021 · 3 comments
Labels
stale Type: Enhancement An improvement to WebPageTest

Comments

@rposbo
Copy link
Contributor

rposbo commented Oct 4, 2021

Feature Request Summary

Archive and delete blatantly old tests that are considered waiting or running; e.g., is the test older than $MAX_DAYS? Archive and Delete. $MAX_DAYS is defined in archive.php but never used (nor is it referenced anywhere else, globally).

https://github.com/WPO-Foundation/webpagetest/blob/master/www/cli/archive.php#L287

What's the motivation or use case for the change/feature?

I run several thousand tests a day (so 10k+ runs) and need to aggresively archive them to S3 to avoid disc space filling up. I have archive_days set to a low number (used to be 0 but that now breaks, due to a check in archive.php that equates that to false!) - 0.1 - however I am seeing disc usage continue to increase which will result in total failure.

image

(The reason for the tests halting isn't clear - I'll raise another issue when I figure that out!)

AB#140409

@rposbo rposbo added the Type: Enhancement An improvement to WebPageTest label Oct 4, 2021
@rposbo
Copy link
Contributor Author

rposbo commented Oct 4, 2021

I've implemented this suggestion on our setup; something like this -

Near the top of archive.php:

if (GetSetting('max_days_before_archive')) {
    $MAX_DAYS = GetSetting('max_days_before_archive');
}

$MIN_DAYS = max($MIN_DAYS,0.02);
$MAX_DAYS = min($MAX_DAYS,30);

inside CheckTest()

function CheckTest($testPath, $id, $elapsedDays, $forced_only) {
  global $archiveCount;
  global $deleted;
  global $kept;
  global $log;
  global $MIN_DAYS;
  global $MAX_DAYS; // add the global variable ref
  global $is_cli;
  $logLine = "$id ($elapsedDays/$MIN_DAYS/$MAX_DAYS): ";

  if ($is_cli)
    echo "\rArc:$archiveCount, Del:$deleted, Kept:$kept, Checking:" . str_pad($id,45);

  $delete = false;

  // aggressively delete old stuff
  if ($elapsedDays > $MAX_DAYS) {
    if (!is_file("$testPath/testinfo.ini") &&
       !is_file("$testPath/testinfo.json.gz") &&
       !is_file("$testPath/testinfo.json")) {
     $delete = true;
    } else {
     $needs_archive = true;
    }
  }
  elseif (is_file("$testPath/test.waiting")) {
    // Skip tests that are still queued
   //.... etc

(I'd submit a PR, but my version of archive.php is such a mess at the moment - littered with echo statements to understand what's happening..)

@tkadlec
Copy link
Contributor

tkadlec commented Oct 5, 2021

Thanks @rposbo! This is helpful even without the PR. :)

Copy link

We're in the process of cleaning up issues on this project in order to ensure we're able to stay on top of high priority bugs and feature requests. As a part of this process, we're automatically closing any issues that have had no activity within the last two years, including this one, since the codebase has changed dramatically in that time. If you feel this is still relevant, please file a new issue using the relevant issue template so we can get it prioritized. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale Type: Enhancement An improvement to WebPageTest
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants