add Lab 10

Author: afeld

README.md

@@ -101,8 +101,8 @@ You can find the rubric under the [Assignment](https://courseworks2.columbia.edu
 | 7    | 3/6         | [Databases](lectures/lecture_07.md)                               | [Readings](readings/week_07.md)                                           | [Databases](labs/lab_07.md)                                    | [Lab 6](labs/lab_06.md)                            |
 | 8    | 3/13        | [Guest speaker; data warehousing](lectures/lecture_08.md)         | [Project Part 4](docs/project.md#part-4)                                  | [Data loading](labs/lab_08.md)                                 | [Lab 7](labs/lab_07.md)                            |
 | 9    | 3/20        | none ([Spring Recess][recess])                                    | none                                                                      | none ([Spring Recess][recess])                                 | none                                               |
-| 10   | 3/27        | [Data engineering (pipelines, ETL)](lectures/lecture_10.md)       | [Project Part 5](docs/project.md#part-5)                                  | TBD                                                            | [Lab 8](labs/lab_08.md)                            |
-| 11   | 4/3         | Infrastructure / cloud computing                                  | [Readings](readings/week_11.md)                                           | TBD                                                            | TBD                                                |
+| 10   | 3/27        | [Data engineering (pipelines, ETL)](lectures/lecture_10.md)       | [Project Part 5](docs/project.md#part-5)                                  | [Data loading, continued](labs/lab_10.md)                      | [Lab 8](labs/lab_08.md)                            |
+| 11   | 4/3         | Infrastructure / cloud computing                                  | [Readings](readings/week_11.md)                                           | TBD                                                            | [Lab 10](labs/lab_10.md)                           |
 | 12   | 4/10        | Big data; algorithms                                              | [Readings](readings/week_12.md)                                           | TBD                                                            | TBD                                                |
 | 13   | 4/17        | Privacy                                                           | [Readings](readings/week_13.md)                                           | TBD                                                            | TBD                                                |
 | 14   | 4/24        | buffer                                                            | TBD                                                                       | TBD                                                            | TBD                                                |

docs/project.md

@@ -192,7 +192,7 @@ At this point, your project should be looking more like one of the [examples](#i
 
 ### Steps
 
-Do the following for your regularly-updated data source. Repeating the middle steps for the additional dataset(s) is optional.
+Do the following for your regularly-updated data source. Only do one for now — we'll do the rest in [Lab 10](../labs/lab_10.md).
 
 1. Install [pandas-gbq](https://pandas-gbq.readthedocs.io/).
 1. Load data.

labs/lab_06.md

@@ -52,4 +52,4 @@ This is one of those times where you'll follow instructions without necessarily
 
 ### Submit
 
-[Submit the links to the pull request(s) via CourseWorks.](https://courseworks2.columbia.edu/courses/210480/assignments)
+[Submit links to the pull request(s) via CourseWorks.](https://courseworks2.columbia.edu/courses/210480/assignments)

labs/lab_08.md

@@ -28,7 +28,7 @@ The trick is avoiding duplicates. Your script might then need to say something l
 ## Lab work
 
 - You'll write methods to load continuously updated data into a database.
-    - You'll set up scripts to perform each of the [methods of data loading](#data-loading) into DuckDB.
+  - You'll set up scripts to perform each of the [methods of data loading](#data-loading) into DuckDB.
 - You'll [pair](../docs/pairing.md) in your Lab group.
 - Work on branches and submit pull requests for the chunks of work — you decide what the "chunks" are.
 
@@ -40,13 +40,13 @@ The trick is avoiding duplicates. Your script might then need to say something l
 - We have monthly observations (rows) and monthly vintages (columns)
 
 | DATE    | PCPI04M1 | PCPI04M2 | PCPI04M3 |
-|---------|---------:|---------:|---------:|
-| 2003:09 | 185.0    | 185.1    | 185.1    |
-| 2003:10 | 185.0    | 184.9    | 184.9    |
-| 2003:11 | 184.6    | 184.6    | 184.6    |
-| 2003:12 | 185.0    | 184.9    | 184.9    |
-| 2004:01 | #N/A     | 185.8    | 185.8    |
-| 2004:02 | #N/A     | #N/A     | 186.3    |
+| ------- | -------: | -------: | -------: |
+| 2003:09 |    185.0 |    185.1 |    185.1 |
+| 2003:10 |    185.0 |    184.9 |    184.9 |
+| 2003:11 |    184.6 |    184.6 |    184.6 |
+| 2003:12 |    185.0 |    184.9 |    184.9 |
+| 2004:01 |     #N/A |    185.8 |    185.8 |
+| 2004:02 |     #N/A |     #N/A |    186.3 |
 
 - A revision of past data is released in February of each year.
 - A revision released in year `t` can update the values in years `t-5` to `t-1`.
@@ -58,9 +58,9 @@ The trick is avoiding duplicates. Your script might then need to say something l
 Suppose your organization wants to maintain a database of CPI data
 
 - Write a `get_latest_data` function that accepts a `pull_date` and returns the latest data available up to that date
-   - For example, if the `pull_date` is 2004-01-15, the function should return the data from vintage `PCPI04M1`
+  - For example, if the `pull_date` is 2004-01-15, the function should return the data from vintage `PCPI04M1`
 - Write code that pulls the latest data at a given `pull_date` and loads it into a DuckDB database
-    - You will implement each of the methods `append`, `trunc`, and `incremental`
+  - You will implement each of the methods `append`, `trunc`, and `incremental`
 - Loop over a range of `pull_dates` to simulate running the scripts on a daily basis
 - Compare the performance of each method (consistency and speed)
 
@@ -79,12 +79,12 @@ Suppose your organization wants to maintain a database of CPI data
      - `_append`
      - `_trunc`
      - `_inc`
-    - Your code should accept a `pull_date` parameter and load the data up to that date
-    - The script should be able to run multiple times without duplicating data
-    - For incremental: a Python script may be easier than a SQL one
+   - Your code should accept a `pull_date` parameter and load the data up to that date
+   - The script should be able to run multiple times without duplicating data
+   - For incremental: a Python script may be easier than a SQL one
 4. On a notebook: simulate your organization running the scripts on a daily basis.
    - Start from empty tables
    - Loop over a range of `pull_dates` (e.g. 2000-01-01 to 2025-02-28) to simulate running the scripts on a daily basis.
    - If the loop takes way too long, use a shorter range
-   - Compare the performance of each method (data consistency and speed) 
-5. [Submit the links to the pull request(s) via CourseWorks.](https://courseworks2.columbia.edu/courses/210480/assignments)
+   - Compare the performance of each method (data consistency and speed)
+5. [Submit links to the pull request(s) via CourseWorks.](https://courseworks2.columbia.edu/courses/210480/assignments)

labs/lab_10.md

@@ -0,0 +1,14 @@
+# Lab 10
+
+**Objective:** Think through data loading for different data sources
+
+---
+
+Work in your [Project team](../docs/project_teams.csv) to load your other datasets to BigQuery. For each data source:
+
+1. What type of [data loading](lab_08.md#data-loading) will you use? Why? Explain as Markdown in your repository.
+1. Create a data loading script, repeating those steps from [Part 5](../docs/project.md#part-5).
+
+---
+
+[Submit links to the pull request(s) via CourseWorks.](https://courseworks2.columbia.edu/courses/210480/assignments)

readings/week_11.md

@@ -4,6 +4,7 @@
 
 - [Append Load vs Incremental Load vs Truncate and Load](https://medium.com/@santosh_beora/the-3-most-commonly-used-etl-processes-explained-through-everyday-analogies-a7aa9f7a3754)
 - Pipeline assignment
+  - Airflow
 - [Cracking the Cloud_Open](https://www.redhat.com/en/command-line-heroes/season-1/crack-the-cloud-open)
 - [Overview of Cloud Computing](https://dc.arcabc.ca/islandora/object/dc%3A54375?solr_nav%5Bid%5D=c0f46853d72e7e533f04&solr_nav%5Bpage%5D=0&solr_nav%5Boffset%5D=0), Chapters 1-2