daily tests data drop over past 10 days

[yachang]

2 | 2020-01-26 | 8539584 |  
-- | -- | --
3 | 2020-01-25 | 8802175 |  
4 | 2020-01-24 | 8804368 |  
5 | 2020-01-23 | 8989835 |  
6 | 2020-01-22 | 9547450 |  
7 | 2020-01-21 | 9553054 |  
8 | 2020-01-20 | 9684823 |  
9 | 2020-01-19 | 9787798 |  
10 | 2020-01-18 | 10059376 |  
11 | 2020-01-17 | 9847430 |  
12 | 2020-01-16 | 10236158 |  
13 | 2020-01-15 | 10198442 |  
14 | 2020-01-14 | 10056294 |  
15 | 2020-01-13 | 10208497

[yachang]

Not sure it is caused by delay from Gardener, or other unknown reason

[yachang]

iad drop more than 25% from 01/14 to 01/28 (about 2 weeks). Escalate this issue to P0

[yachang]

gsutil ls -l gs://archive-mlab-oti/ndt/traceroute/2020/01/14/iad
totally 75M
gsutil ls -l gs://archive-mlab-oti/ndt/traceroute/2020/01/27/iad
totally 77M

The drop of tests in BQ table could be related to m-lab/etl-gardener#204

[yachang]

Number of tests for 2020-02-11 increased back to 10017424 (about 10%)

Not sure it is a one day thing, will keep updating here.

[stephen-soltesz]

@yachang please add original query to help others reproduce.

[yachang]

m-lab/etl#820

Is a issued triggered by this issue

[yachang]

here is the BQ used:

SELECT
ts,
COUNT(*) AS num

FROM (
SELECT
DATE(TestTime) as ts
FROM mlab-oti.base_tables.traceroute

WHERE
DATE(_PARTITIONTIME) BETWEEN DATE("2020-01-01") AND DATE("2020-02-20")
)

GROUP BY ts
ORDER BY ts DESC