Develop a census data intake pipeline using the newly refactored flow

[MattTriano]

Try to match the task grouping from issue #107

[MattTriano]

Regarding implementation of the metadata collector, I think I can achieve this by recursively scraping the "file system" descended from the root node (https://www2.census.gov/). In the data structure on my end, I think I'll create a tree where the id of each node instance is the URL and the node class has methods to:

• scrape the page
• identify child_directories
• identify child_files
• compare the last_modified time of a directory or file to a cache of checks in the data warehouse
  • this should significantly accelerate subsequent runs if implemented to short circuit update checks when a high-level directory's cached last_modified matches an update check.

The census_metadata table will need columns:

• metadata_url
• last_modified
• size
• description
• is_dir
• is_file
• check_time
• updated_data_available
  And likely some more fields as needed.

I don't want to automatically pull down all of the census data as that's a gigantic volume of data, and there are a lot of different file_types (most of which won't be handled until a need for the data or content arises).

[MattTriano]

My scraper for www2.census.gov metadata has been running since yesterday (I put in a 0.75 * random(0,1) + 0.5 second sleep between requests to avoid just hammering their server) and I've collected over 1.15M URL endpoints, and it looks like I'm still a long way from scraping everything. Still, it shouldn't take anywhere near this long in the future.

I might want to just use this side to short-circuit unnecessary data updates and use the census APIs (here's a json dict of endpoints) to act as the data source (rather than the endpoints from the metadata-side URL. I'll have to see how the data is formatted and organized, but using the APIs might allow me to sidestep a lot of explicit combining of files from the metadata-side endpoints.

[MattTriano]

Oh hey, the API json data menu (https://api.census.gov/data.json, .html and .xml also work) includes a modified field that contains datelike values. That would probably be much easier to work with than the stuff I built yesterday (although that work still has a lot of value as it will produce a full list of the pdfs documenting these data sets).

[MattTriano]

The scraper ran through the week and I cut it off this morning. It still has the following URLs to scrape (see list below; it has been scraping in a depth-first search pattern)

[
'https://www2.census.gov/2020Census',
'https://www2.census.gov/EEO_2006_2010',
'https://www2.census.gov/EEO_2014_2018',
'https://www2.census.gov/EEO_Disability_2008-2010',
'https://www2.census.gov/Econ2001_And_Earlier',
'https://www2.census.gov/about',
'https://www2.census.gov/acs',
'https://www2.census.gov/acs2002',
'https://www2.census.gov/acs2003',
'https://www2.census.gov/acs2004',
'https://www2.census.gov/acs2005',
'https://www2.census.gov/acs2005_2007_3yr',
'https://www2.census.gov/acs2005_2009_5yr',
'https://www2.census.gov/acs2006',
'https://www2.census.gov/acs2006_2008_3yr',
'https://www2.census.gov/acs2007_1yr',
'https://www2.census.gov/acs2007_2009_3yr',
'https://www2.census.gov/acs2007_3yr',
'https://www2.census.gov/acs2008_1yr',
'https://www2.census.gov/acs2008_3yr',
'https://www2.census.gov/acs2009_1yr',
'https://www2.census.gov/acs2009_3yr',
'https://www2.census.gov/acs2009_5yr',
'https://www2.census.gov/acs2010_1yr',
'https://www2.census.gov/acs2010_3yr',
'https://www2.census.gov/acs2010_5yr',
'https://www2.census.gov/acs2010_SPT_AIAN',
'https://www2.census.gov/acs2011_1yr',
'https://www2.census.gov/acs2011_3yr',
'https://www2.census.gov/acs2011_5yr',
'https://www2.census.gov/acs2012_1yr',
'https://www2.census.gov/acs2012_3yr',
'https://www2.census.gov/acs2012_5yr',
'https://www2.census.gov/acs2013_1yr',
'https://www2.census.gov/acs2013_3yr',
'https://www2.census.gov/acs2013_5yr',
'https://www2.census.gov/acs_latest_data',
'https://www2.census.gov/acs_special_tabs',
'https://www2.census.gov/adrm',
'https://www2.census.gov/cac',
'https://www2.census.gov/census_1940',
'https://www2.census.gov/census_1980',
'https://www2.census.gov/census_1990',
'https://www2.census.gov/census_2000',
'https://www2.census.gov/census_2010',
'https://www2.census.gov/ces',
'https://www2.census.gov/data',
'https://www2.census.gov/decennial',
'https://www2.census.gov/desen002',
'https://www2.census.gov/dssd',
'https://www2.census.gov/econ',
'https://www2.census.gov/econ1977',
'https://www2.census.gov/econ1982',
'https://www2.census.gov/econ1987',
'https://www2.census.gov/econ1992',
'https://www2.census.gov/econ1997',
'https://www2.census.gov/econ2002',
'https://www2.census.gov/econ2003',
'https://www2.census.gov/econ2004',
'https://www2.census.gov/econ2005',
'https://www2.census.gov/econ2006',
'https://www2.census.gov/econ2007',
'https://www2.census.gov/econ2008',
'https://www2.census.gov/econ2009',
'https://www2.census.gov/econ2010',
'https://www2.census.gov/econ2011',
'https://www2.census.gov/econ2012',
'https://www2.census.gov/econ2013',
'https://www2.census.gov/econ2014',
'https://www2.census.gov/econ2015',
'https://www2.census.gov/econ2016',
'https://www2.census.gov/econ2017',
'https://www2.census.gov/foia',
'https://www2.census.gov/geo/maps/DC2010',
'https://www2.census.gov/geo/maps/DC2020/ACO20',
'https://www2.census.gov/geo/maps/DC2020/AIANWall2020',
'https://www2.census.gov/geo/maps/DC2020/DC20BLK',
'https://www2.census.gov/geo/maps/DC2020/IFAC',
'https://www2.census.gov/geo/maps/DC2020/MCS',
'https://www2.census.gov/geo/maps/DC2020/PL20',
'https://www2.census.gov/geo/maps/DC2020/PL20Proto',
'https://www2.census.gov/geo/maps/DC2020/PSAPV',
'https://www2.census.gov/geo/maps/DC2020/PUMA',
'https://www2.census.gov/geo/maps/DC2020/PopCenter',
'https://www2.census.gov/geo/maps/DC2020/PopDist_Nighttime',
'https://www2.census.gov/geo/maps/DC2020/SLD_RefMap',
'https://www2.census.gov/geo/maps/DC2020/SR20',
'https://www2.census.gov/geo/maps/DC2020/TEA'
]