Remove annual_database_service models and manual scrape process.

Replaced by external_ntd__annual_reporting and new models: - dim_annual_service_agencies - dim_annual_service_mode_time_periods - fct_annual_service_modes [#3497]
cal-itp · Oct 31, 2024 · 5f864d6 · 5f864d6
1 parent 5306630
commit 5f864d6
Show file tree

Hide file tree

Showing 5 changed files with 0 additions and 76 deletions.
diff --git a/airflow/dags/create_external_tables/ntd_data_products/annual_database_service.yml b/airflow/dags/create_external_tables/ntd_data_products/annual_database_service.yml
diff --git a/script/scrape_ntd.py b/script/scrape_ntd.py
@@ -13,9 +13,6 @@
 https://www.transit.dot.gov/ntd/data-product/2021-annual-database-agency-information
 Contains basic contact and agency information for each NTD reporter.
 
-https://www.transit.dot.gov/ntd/data-product/2022-annual-database-service
-Contains operating statistics reported by mode and type of service. Categorized by vehicles operated and vehicles available in maximum service by day and time period.
-
 https://www.transit.dot.gov/ntd/data-product/monthly-module-adjusted-data-release
 Monthly ridership
 
@@ -28,7 +25,6 @@
 then
 poetry run python scrape_ntd.py annual-database-agency-information 2021 https://www.transit.dot.gov/sites/fta.dot.gov/files/2022-10/2021%20Agency%20Information.xlsx
 poetry run python scrape_ntd.py monthly-ridership-with-adjustments 2024 https://www.transit.dot.gov/sites/fta.dot.gov/files/2024-04/February%202024%20Complete%20Monthly%20Ridership%20%28with%20adjustments%20and%20estimates%29_240402_0.xlsx
-poetry run python scrape_ntd.py annual-database-service 2022 https://www.transit.dot.gov/sites/fta.dot.gov/files/2024-04/2022%20Service.xlsx
 """
 
 import gzip

diff --git a/warehouse/models/staging/ntd/_src.yml b/warehouse/models/staging/ntd/_src.yml
@@ -7,7 +7,6 @@ sources:
     schema: external_ntd_data_products
     tables:
       - name: annual_database_agency_information
-      - name: annual_database_service
       - name: monthly_ridership_with_adjustments_upt
       - name: monthly_ridership_with_adjustments_vrm
       - name: monthly_ridership_with_adjustments_vrh

diff --git a/warehouse/models/staging/ntd/_stg_ntd.yml b/warehouse/models/staging/ntd/_stg_ntd.yml
@@ -12,7 +12,6 @@ models:
       - name: ntd_id
         tests:
           - not_null
-  - name: stg_ntd__annual_database_service
   - name: stg_ntd__monthly_ridership_with_adjustments_upt
   - name: stg_ntd__monthly_ridership_with_adjustments_vrm
   - name: stg_ntd__monthly_ridership_with_adjustments_vrh

diff --git a/warehouse/models/staging/ntd/stg_ntd__annual_database_service.sql b/warehouse/models/staging/ntd/stg_ntd__annual_database_service.sql