-
Notifications
You must be signed in to change notification settings - Fork 714
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Archive failed tests to avoid filling up disc [private instance] #1565
Comments
I've implemented this suggestion on our setup; something like this - Near the top of if (GetSetting('max_days_before_archive')) {
$MAX_DAYS = GetSetting('max_days_before_archive');
}
$MIN_DAYS = max($MIN_DAYS,0.02);
$MAX_DAYS = min($MAX_DAYS,30); inside function CheckTest($testPath, $id, $elapsedDays, $forced_only) {
global $archiveCount;
global $deleted;
global $kept;
global $log;
global $MIN_DAYS;
global $MAX_DAYS; // add the global variable ref
global $is_cli;
$logLine = "$id ($elapsedDays/$MIN_DAYS/$MAX_DAYS): ";
if ($is_cli)
echo "\rArc:$archiveCount, Del:$deleted, Kept:$kept, Checking:" . str_pad($id,45);
$delete = false;
// aggressively delete old stuff
if ($elapsedDays > $MAX_DAYS) {
if (!is_file("$testPath/testinfo.ini") &&
!is_file("$testPath/testinfo.json.gz") &&
!is_file("$testPath/testinfo.json")) {
$delete = true;
} else {
$needs_archive = true;
}
}
elseif (is_file("$testPath/test.waiting")) {
// Skip tests that are still queued
//.... etc (I'd submit a PR, but my version of archive.php is such a mess at the moment - littered with |
Thanks @rposbo! This is helpful even without the PR. :) |
We're in the process of cleaning up issues on this project in order to ensure we're able to stay on top of high priority bugs and feature requests. As a part of this process, we're automatically closing any issues that have had no activity within the last two years, including this one, since the codebase has changed dramatically in that time. If you feel this is still relevant, please file a new issue using the relevant issue template so we can get it prioritized. Thanks! |
Feature Request Summary
Archive and delete blatantly old tests that are considered
waiting
orrunning
; e.g., is the test older than$MAX_DAYS
? Archive and Delete.$MAX_DAYS
is defined inarchive.php
but never used (nor is it referenced anywhere else, globally).https://github.com/WPO-Foundation/webpagetest/blob/master/www/cli/archive.php#L287
What's the motivation or use case for the change/feature?
I run several thousand tests a day (so 10k+ runs) and need to aggresively archive them to S3 to avoid disc space filling up. I have
archive_days
set to a low number (used to be0
but that now breaks, due to a check inarchive.php
that equates that tofalse
!) - 0.1 - however I am seeing disc usage continue to increase which will result in total failure.(The reason for the tests halting isn't clear - I'll raise another issue when I figure that out!)
AB#140409
The text was updated successfully, but these errors were encountered: