python_cheatsheet.qmd

---
title: Getting started with Python + Jupyter Notebook
author: Simon Schulte, Johan Velez
jupyter: python3
format: 
  html: 
    self-contained: true
pandoc: 
  toc: true
  toc-depth: 3
  number-sections: true
date: last-modified
theme: litera

---


# Installing Python Packages from your Jupyter Notebook

## Using Conda

If you have installed Python via the package manager Anaconda, to
install a Python package you can use the following command in a Jupyter
notebook:

``` python
conda install <package-name>
```

For example, to install the `numpy` package, you would run the following
command from a Jupyter Notebook code cell:

``` python
conda install numpy
```

For some older versions of Jupyter Notebook running this command causes
the following error: `SyntaxError: invalid syntax`.

Then you can try adding a `%`-sign before `conda`:

``` python
%conda install <package-name>
```

## Using Pip

If you have just installed Python (without Anaconda), you can use the
following command in a Jupyter notebook to install a Python package:

``` python
pip install <package-name>
```

For example, to install the `matplotlib` package, you would run:

``` python
pip install matplotlib
```

For some older versions of Jupyter Notebook running this command causes
the following error: `SyntaxError: invalid syntax`.

Then you can try adding a `%`-sign before `pip`:

``` python
%pip install <package-name>
```

# Jupyter Notebook Keyboard Shortcuts

Here are some of the most important keyboard shortcuts for working with
Jupyter notebooks:

-   `Enter`: Enter edit mode
-   `Shift + Enter`: Run cell and move to the next one
-   `Ctrl + Enter`: Run cell and stay on the current one
-   `Esc`: Exit edit mode
-   `A`: Insert cell above
-   `B`: Insert cell below
-   `D, D`: Delete selected cell
-   `Z`: Undo last cell deletion
-   `M`: Change cell to markdown
-   `Y`: Change cell to code
-   `Ctrl + S`: Save notebook
-   `Ctrl + Shift + P`: Show command palette
-   `Shift + Tab`: Show docstring (in edit or command mode)

For a full list of keyboard shortcuts, you can click on the
`Keyboard Shortcuts` option in the `Help` menu.

# Python packages we use a lot

There are two Python packages that we use a lot in our course: Pandas
and NumPy. While some of their features overlap (e.g. matrix
multiplication), others are unique to one of the packages (e.g. matrix
inversion).

**We recommend to use Pandas whenever possible**, because you keep the
column-names and indices (=row-names) of your matrix/vector when you
manipulate matrices in Pandas `DataFrame` format instead of NumPy's
`arrays` or `matrix` format.

## Pandas

Pandas is a powerful library for data analysis in Python. It provides
data structures for efficiently storing and manipulating large,
heterogeneous datasets, as well as tools for data cleaning, merging,
reshaping, and analysis.

Here you find a cheatsheet explaining the main functionalities of
Pandas: <https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf>

To use Pandas in a Jupyter notebook, you can install it using either
Conda or Pip (see the "Installing Python Packages" section above), and
then import it using the following command:

```{python}
import pandas as pd
```

## NumPy

NumPy is a fundamental package for scientific computing in Python. It
provides powerful tools for working with arrays and matrices, as well as
a large library of mathematical functions for linear algebra, Fourier
analysis, and more.

Here you find a cheatsheet explaining the main functionalities of
Pandas:
<https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Numpy_Python_Cheat_Sheet.pdf>

To use NumPy in a Jupyter notebook, you can install it using either
Conda or Pip (see the "Installing Python Packages" section above), and
then import it using the following command:

```{python}
import numpy as np
```

# Converting between NumPy Arrays/Matrices and Pandas DataFrames

## From NumPy to Pandas

```{python}
import numpy as np

# First we create a NumPy array
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr
```

Note, a numpy array has not column- or row-names (=index)!

Converting to Pandas DataFrame:

```{python}
import pandas as pd

# create a Pandas DataFrame from a NumPy array
df = pd.DataFrame(arr, index=['row1', 'row2', 'row3'], columns=['col1', 'col2', 'col3'])
df
```

... but: Pandas DataFrame has column- and row-names!

## From Pandas to NumPy

```{python}
import pandas as pd

# create a Pandas DataFrame
df = pd.DataFrame({'col1': [1, 2, 3], 'col2': [4, 5, 6], 'col3': [7, 8, 9]}, index=['row1', 'row2', 'row3'])

```

Convert to NumPy Array:

```{python}
# convert a Pandas DataFrame to a NumPy array
arr = df.to_numpy()
arr
```

# Indexing Data in Python

## Pandas DataFrame

### Using .iloc and .loc

-   `.iloc` is used for index-based selection of rows and columns
-   `.loc` is used for label-based selection of rows and columns

```{python}
import pandas as pd

# create a sample dataframe
df = pd.DataFrame({'col1': [1, 2, 3, 4], 'col2': [5, 6, 7, 8]}, index=['row1', 'row2', 'row3', 'row4'])
print(df)
```

```{python}
# select a single row using .iloc
row = df.iloc[0]
print(row)
```

```{python}
# select a single column using .loc
col = df.loc[:, 'col1']
print(col)
```

```{python}
# select a subset of rows and columns using .iloc
subset = df.iloc[0:2, 0:2]
print(subset)
```

```{python}
# select a subset of rows and columns using .loc
subset = df.loc[['row1', 'row2'], ['col1', 'col2']]
print(subset)
```

## NumPy Array

-   Indexing in NumPy arrays can be done using integer arrays, boolean
    arrays, or slices
-   Slicing is used to select a range of elements in the array

```{python}
import numpy as np

# create a sample numpy array
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
```

```{python}
# select the first row using integer indexing
row = arr[0]
print(row)
```

```{python}
# select the first column using integer indexing
col = arr[:, 0]
print(col)
```

```{python}
# select a subset of the array using slices
subset = arr[0:2, 0:2]
print(subset)
```

```{python}
# select elements based on a boolean condition
bool_arr = arr > 5
subset = arr[bool_arr]
print(subset)
```


# MultiIndex in Pandas

In pandas, a MultiIndex allows you to have multiple levels of indexing in your DataFrame. This is useful when you want to work with higher-dimensional data or when you need to group and analyze data across multiple dimensions. Especially when working with Multi-Regional Input-Output (MRIO) Tables which have two dimensions (country and industry) this is feature is super helpful. 

Here's an example of creating and working with a DataFrame that has a MultiIndex representing a simple MRIO with two countries and two regions.

## Example: Creating a DataFrame with MultiIndex

Let's create a 4x4 DataFrame with random integers, where both the columns and rows have a 2-dimensional index representing country and industry. 

```{python}
import pandas as pd
import numpy as np

# Define the index levels
countries = ['country1', 'country2']
industries = ['industry1', 'industry2']

# Create the MultiIndex
multi_index = pd.MultiIndex.from_product([countries, industries], names=['Country', 'Industry'])

# Create a DataFrame with random integers
data = np.random.randint(0, 10, size=(4, 4))
df = pd.DataFrame(data, index=multi_index, columns=multi_index)

# Print the DataFrame
print(df)
```



In this example, we first define the index levels, `countries` and `industries`. Then, we create the `multi_index` by using the `pd.MultiIndex.from_product()` function and passing the index level arrays along with the `names` parameter to assign names to the levels.

Next, we create a random integer array using `np.random.randint()`, and finally, we create the DataFrame `df` with the specified index and columns.

## Working with MultiIndex

Once you have a DataFrame with a MultiIndex, you can access and manipulate the data using various indexing and slicing techniques. Here are a few examples:

```{python}
# Accessing specific rows using the MultiIndex
print(df.loc[('country1', 'industry1'),:])
```

```{python}
# Accessing specific columns using the MultiIndex
print(df.loc[:, ('country1', 'industry1')])
```


You can also aggregate data using the MultiIndex. Therefore, you first have to group the data using `groupby()`, and then append the mathematical operation, e.g. `.sum()`. The attribute `axis` determines whether the DataFrame is grouped along rows (0) or columns (1). The attribute `level` determines by which Index the data is grouped. 



```{python}
 # axis=0: aggregate rows
print(df.groupby(axis=0,level=('Country')).sum())
```

```{python}
# axis=1: aggregate columns
print(df.groupby(axis=1,level=('Country')).sum()) 
```


These are just a few examples of working with MultiIndex in pandas. You can explore more advanced techniques and functions available in the pandas documentation.


# Specifying Paths in Python with the `os` Library

In Python, the `os` module provides functions for working with operating system functionalities, including file and directory operations. One common use case is working with file paths. 

## Joining Path Components

To join multiple path components into a single path, you can use the `os.path.join()` function. This is useful when you need to construct a path dynamically, especially when dealing with different operating systems:

```{python}
import os

# Joining path components
path = os.path.join('folder', 'subfolder', 'file.txt')
print(path)
```



## Getting the Current Working Directory

To get the current working directory, you can use the `os.getcwd()` function:

```{python}
import os

# Getting the current working directory
current_dir = os.getcwd()
print(current_dir)
```


## Checking if a Path Exists

To check if a path exists, you can use the `os.path.exists()` function:

```{python}
import os

# Checking if a path exists
path = '/path/to/file.txt'
exists = os.path.exists(path)
print(exists)
```




## Specifying an Absolute Path in Windows with the `os` Library

In Windows, absolute paths are specified differently compared to other operating systems due to the use of drive letters (e.g., `C:`). The `os` module in Python provides functions to work with these paths.

### Absolute Path in Windows

To specify an absolute path in Windows, you need to include the drive letter followed by a colon (`:`), and use double backslashes (`\\`) as the path separator. Here's an example:

```{python}
import os

# Specify an absolute path in Windows
abs_path = "C:\\path\\to\\file.txt"
print(abs_path)
```


Make sure to escape the backslashes by using a double backslash (`\\`).

### Using Raw String

Alternatively, you can use a raw string by adding an `r` prefix before the path string. This allows you to specify the path using single backslashes (`\`) as the path separator. Here's an example:

```{python}
import os

# Specify an absolute path in Windows using a raw string
abs_path = r"C:\path\to\file.txt"
print(abs_path)
```

Using a raw string is convenient as you don't need to escape the backslashes.

### Converting to Windows Path

If you're working with a path string that follows the forward slash (`/`) path separator convention used in other operating systems (e.g., Linux, macOS), you can convert it to a Windows path using the `os.path.normpath()` function:

```{python}
import os

# Convert a path to Windows format
path = "/path/to/file.txt"
windows_path = os.path.normpath(path)
print(windows_path)
```


The `os.path.normpath()` function automatically converts the path separator to double backslashes (`\\`).

Keep in mind that when working with paths in Python, it's recommended to use the `os.path` functions to ensure cross-platform compatibility.





# Plotting in Python

Python provides several popular libraries for data visualization and plotting. Let's explore some of these libraries and create a bar plot using the example data.

## Example Data

Let's consider an example DataFrame with two columns: Country and Emissions. This data represents the emissions of each country.

```{python}
import pandas as pd

data = {
    'Country': ['USA', 'China', 'Russia', 'Germany', 'India'],
    'Emissions': [5500, 8500, 4300, 3700, 3000]
}

df = pd.DataFrame(data)

```

## Matplotlib

Matplotlib is a widely used plotting library in Python. It provides a flexible and comprehensive set of functions for creating various types of plots.

```{python}
import matplotlib.pyplot as plt

plt.bar(df['Country'], df['Emissions'])
plt.xlabel('Country')
plt.ylabel('Emissions')
plt.title('Emissions by Country (Matplotlib)')
plt.show()

```

<!-- ## Seaborn -->

<!-- Seaborn is a high-level library built on top of Matplotlib. It provides a simplified interface for creating visually appealing statistical graphics. -->

<!-- ```{python} -->
<!-- import seaborn as sns -->

<!-- sns.barplot(data=df, x='Country', y='Emissions') -->
<!-- plt.xlabel('Country') -->
<!-- plt.ylabel('Emissions') -->
<!-- plt.title('Emissions by Country (Seaborn)') -->
<!-- plt.show() -->

<!-- ``` -->

## Plotly

Plotly is an interactive plotting library that allows you to create interactive, web-based visualizations. It provides a range of graph types and customization options.




```{python}
import plotly.express as px

fig = px.bar(df, x='Country', y='Emissions', title='Emissions by Country (Plotly)')
fig.show()

```

## Plotnine (ggplot)

Plotnine is a Python implementation of the popular R library ggplot2. It follows the grammar of graphics approach and provides a powerful and flexible system for creating visually appealing plots.

```{python}
from plotnine import ggplot, aes, geom_bar, labs

(
  ggplot(df, aes(x='Country', y='Emissions')) + 
      geom_bar(stat='identity') + 
      labs(title='Emissions by Country (Plotnine)')
)
```

In the example above, we use the example data to create a bar plot for emissions using each library. Matplotlib, Seaborn, Plotly, and Plotnine (ggplot) all generate the same bar plot, but they offer different levels of customization and interactivity.

These are just a few examples of the many plotting libraries available in Python. Depending on your specific needs and preferences, you can explore other libraries such as Bokeh, Altair, or Plotly Express to create various types of visualizations.

Remember to install the necessary libraries using `pip` or `conda` before using them in your code.



# For-Loop in Python

In Python, a for-loop is used to iterate over a sequence of elements or to perform a set of instructions repeatedly for a specific number of times. The basic syntax of a for-loop in Python is as follows:

```python
for variable in sequence:
    # Code to be executed for each element
```

Let's break down the components of a for-loop:

- `variable`: This is a variable that takes on the value of each element in the sequence, one at a time, during each iteration of the loop.

- `sequence`: This can be any iterable object such as a list, tuple, string, or range. The loop iterates over each element in the sequence.

- `Code to be executed for each element`: This is the block of code that is executed for each element in the sequence. It can include one or more statements.

Here's an example that demonstrates the usage of a for-loop:

```{python}
fruits = ['apple', 'banana', 'cherry']

for fruit in fruits:
    print(fruit)
```


In this example, we have a list of fruits. The for-loop iterates over each fruit in the list and prints it.

You can also use the `range()` function to generate a sequence of numbers and iterate over them using a for-loop. Here's an example:

```{python}
for num in range(1, 6):
    print(num)
```


In this case, the for-loop iterates over the numbers from 1 to 5 (inclusive) and prints each number.

For-loops are powerful constructs in Python that allow you to perform repetitive tasks efficiently. They are commonly used when you have a known number of iterations or when you need to iterate over a sequence of elements.

Remember to properly indent the code block within the for-loop to indicate the scope of the loop. The indentation is typically four spaces or a tab.

You can also use the `break` and `continue` statements within a for-loop to control the flow of execution.

Experiment with for-loops to iterate over different sequences and perform various operations within the loop. It's a fundamental construct in Python programming that you'll frequently encounter and utilize.


# Functions in Python

Functions are reusable blocks of code that perform a specific task. They help organize code, promote code reuse, and make it easier to understand and maintain. In Python, you can define and use functions. Let's explore how to work with functions in Python.

## Defining Functions

To define a function in Python, you use the `def` keyword followed by the function name and parentheses. You can also specify parameters inside the parentheses if the function requires input values. The function body is indented below the function definition line.

```{python}
def greet():
    print("Hello, world!")

def add_numbers(a, b):
    return a + b
```

In this example, we define two functions. The `greet()` function doesn't take any parameters and simply prints a greeting. The `add_numbers()` function takes two parameters `a` and `b` and returns their sum.

## Using Functions

Once a function is defined, you can use it by calling its name followed by parentheses. If the function expects parameters, you can pass the necessary values inside the parentheses.

```{python}
greet()  # Output: Hello, world!

result = add_numbers(5, 3)
print(result)  # Output: 8
```

In this example, we call the `greet()` function, which prints the greeting message. Then, we call the `add_numbers()` function with arguments `5` and `3`, and the returned result is stored in the `result` variable. Finally, we print the value of `result`, which is `8`.

## Lambda Functions

In Python, a lambda function is a small anonymous function that doesn't have a name. It can take any number of arguments but can only have one expression. Lambda functions are typically used when you need a simple, one-line function.

```{python}
add = lambda x, y: x + y
result = add(2, 3)
print(result)  # Output: 5
```

In this example, we define a lambda function `add` that takes two arguments `x` and `y` and returns their sum. We then call the lambda function and store the result in the `result` variable. Finally, we print the value of `result`, which is `5`.

Lambda functions are useful when you need a function for a short, simple operation and don't want to define a separate named function.





<!-- # Object-Oriented Programming in Python -->

<!-- Python is an object-oriented programming language, which means that it supports the concepts of objects, classes, methods, functions, and attributes. This paradigm organizes code around the idea of objects, which are instances of classes. -->

<!-- ## Objects -->

<!-- In Python, an object is a runtime entity that encapsulates data (attributes) and behavior (methods). It represents a particular instance of a class. Objects are created from classes and can interact with each other. -->

<!-- ```{python} -->
<!-- # Creating an object -->
<!-- class MyClass: -->
<!--     pass -->

<!-- obj = MyClass()  # Creating an object of MyClass -->
<!-- ``` -->

<!-- ## Methods -->

<!-- Methods are functions defined within a class and are associated with objects. They define the behavior of the objects and can access and modify the object's attributes. Methods are called using dot notation on an object. -->

<!-- ```{python} -->

<!-- # Defining a method -->
<!-- class MyClass: -->
<!--     def my_method(self): -->
<!--         print("Hello, World!") -->

<!-- obj = MyClass()  # Creating an object of MyClass -->
<!-- obj.my_method()  # Calling the method -->
<!-- ``` -->

<!-- ## Attributes -->

<!-- Attributes are data associated with an object. They represent the state of an object and can be variables or properties. Attributes can be accessed and modified using dot notation on an object. -->


<!-- ```{python} -->
<!-- # Adding attributes to an object -->
<!-- class MyClass: -->
<!--     pass -->

<!-- obj = MyClass()  # Creating an object of MyClass -->
<!-- obj.attribute = 42  # Adding an attribute -->
<!-- print(obj.attribute)  # Accessing the attribute -->

<!-- ``` -->

<!-- ## Class -->

<!-- A class is a blueprint or template for creating objects. It defines the structure and behavior that the objects will have. A class encapsulates data (attributes) and functions (methods) that operate on that data. Objects are created from classes using the class name followed by parentheses. -->

<!-- ``` {python} -->

<!-- # Creating a class and objects -->
<!-- class MyClass: -->
<!--     def __init__(self, attribute): -->
<!--         self.attribute = attribute -->

<!--     def my_method(self): -->
<!--         print("Hello, World!") -->

<!-- obj1 = MyClass(42)  # Creating an object of MyClass -->
<!-- obj2 = MyClass(100)  # Creating another object of MyClass -->

<!-- print(obj1.attribute)  # Accessing the attribute of obj1 -->
<!-- obj1.my_method()  # Calling the method of obj1 -->

<!-- print(obj2.attribute)  # Accessing the attribute of obj2 -->
<!-- obj2.my_method()  # Calling the method of obj2 -->
<!-- ``` -->

<!-- In the example above, we define a class MyClass with an attribute attribute, a method my_method, and a constructor __init__ that initializes the attribute. We create two objects obj1 and obj2 from the class, and we can access the attributes and call the methods using dot notation. -->

<!-- Object-oriented programming allows for code organization, reusability, and modularity. It promotes the concept of objects as self-contained entities with their own behavior and state. -->


<!-- # Plotting in Python -->

<!-- Python provides a variety of powerful libraries for creating -->
<!-- high-quality visualizations. Here are some commonly used libraries for -->
<!-- plotting in Python: -->

<!-- ## Matplotlib -->

<!-- Matplotlib is a comprehensive library for creating static, animated, and -->
<!-- interactive visualizations in Python. It provides a wide range of plot -->
<!-- types, including line plots, scatter plots, bar plots, histograms, and -->
<!-- more. Matplotlib is highly customizable, with fine-grained control over -->
<!-- plot elements such as colors, markers, labels, and annotations. -->

<!-- ### Example -->

<!-- ```{python} -->
<!-- import matplotlib.pyplot as plt -->
<!-- import numpy as np -->
<!-- import pandas as pd -->

<!-- # Generate data and create a DataFrame -->
<!-- df = pd.DataFrame({'x': np.linspace(0, 10, 100), 'y': np.sin(np.linspace(0, 10, 100))}) -->

<!-- # Create a line plot -->
<!-- plt.plot(df['x'], df['y']) -->

<!-- # Add labels and a title -->
<!-- plt.xlabel('x') -->
<!-- plt.ylabel('y') -->
<!-- plt.title('A Simple Plot') -->

<!-- # Display the plot -->
<!-- plt.show() -->
<!-- ``` -->

<!-- ## Seaborn -->

<!-- Seaborn is a library for creating statistical visualizations in Python. -->
<!-- It provides a variety of plot types, including scatter plots, line -->
<!-- plots, bar plots, histograms, kernel density plots, and more. Seaborn is -->
<!-- built on top of Matplotlib and provides a simpler interface for creating -->
<!-- common statistical plots. -->

<!-- ### Example -->

<!-- ```{python} -->
<!-- import seaborn as sns -->
<!-- import numpy as np -->
<!-- import pandas as pd -->
<!-- import matplotlib.pyplot as plt -->

<!-- # Generate data and create a DataFrame -->
<!-- df = pd.DataFrame({'x': np.random.normal(size=100), 'y': 2 * np.random.normal(size=100) + np.random.normal(size=100)}) -->

<!-- # Create a scatter plot with a regression line -->
<!-- sns.regplot(data=df, x='x', y='y') -->

<!-- # Display the plot -->
<!-- plt.show() -->
<!-- ``` -->

<!-- ## Plotly -->

<!-- Plotly is a library for creating interactive visualizations in Python. -->
<!-- It provides a wide range of plot types, including scatter plots, line -->
<!-- plots, bar plots, histograms, and more. Plotly is highly customizable, -->
<!-- with fine-grained control over plot elements such as colors, markers, -->
<!-- labels, and annotations. It also provides advanced features such as -->
<!-- animations and interactivity. -->

<!-- ### Example -->

<!-- ```{python} -->
<!-- import plotly.express as px -->
<!-- import numpy as np -->
<!-- import pandas as pd -->

<!-- # Generate data and create a DataFrame -->
<!-- df = pd.DataFrame({'x': np.random.normal(size=100), 'y': 2 * np.random.normal(size=100) + np.random.normal(size=100)}) -->

<!-- # Create a scatter plot with a regression line -->
<!-- fig = px.scatter(data_frame=df, x='x', y='y', trendline='ols') -->

<!-- # Display the plot -->
<!-- fig.show() -->
<!-- ``` -->

<!-- ## ggplot and plotnine -->

<!-- ggplot and plotnine are libraries for creating graphics based on the -->
<!-- grammar of graphics, a system for describing and constructing -->
<!-- statistical graphics. They provide a powerful set of tools for creating -->
<!-- complex, layered visualizations with fine-grained control over plot -->
<!-- elements. ggplot is a library for R, while plotnine is a Python -->
<!-- implementation of the same system. -->

<!-- ### Example -->

<!-- ```{python} -->
<!-- from plotnine import * -->
<!-- import numpy as np -->
<!-- import pandas as pd -->

<!-- # Generate data and create a DataFrame -->
<!-- df = pd.DataFrame({'x': np.random.normal(size=100), 'y': 2 * np.random.normal(size=100) + np.random.normal(size=100)}) -->

<!-- # Create a line plot with color and labels -->
<!-- ( -->
<!--   ggplot(data=df, mapping=aes(x='x', y='y')) +  -->
<!--   geom_point(color='blue') -->
<!--   ) -->
<!-- ``` -->