forked from paulcbauer/apis_for_social_scientists_a_review
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathChapter_Mediacloud_api.Rmd
173 lines (118 loc) · 8.74 KB
/
Chapter_Mediacloud_api.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
# Media Cloud API
<chauthors>Chung-hong Chan</chauthors>
<br><br>
```{r mediacloud-1, include=FALSE}
knitr::opts_chunk$set(warning = FALSE, message = FALSE, cache=TRUE)
```
You will need to install the following packages for this chapter (run the code):
```{r mediacloud-2, echo=FALSE, comment=NA}
.gen_pacman_chunk("Mediacloud_api")
```
## Provided services/data
* *What data/service is provided by the API?*
According to [the official FAQ](https://mediacloud.org/support/), Media Cloud is "an open source and open data platform for storing, retrieving, visualizing, and analyzing online news." It is a consortium project across multiple institutions, including the University of Massachusetts Amherst, Northeastern University, and the Berkman Klein Center for Internet & Society at Harvard University. The full technical information about the project and the data provided are available in @roberts2021media. In short, the system continuously crawls RSS and similar feeds from a large collection of media sources (as of writing: > 25,000 media sources). Based on this large corpus of media contents, the system provides three services: Topic Mapper, Media Explorer, and Source Manager.
The services are accessible through the web interface and csv export is also supported from there. For programmatic access, Media Cloud also provides several [APIs](https://github.com/mediacloud/backend/tree/master/doc/api_2_0_spec). I will focus on the [main v2.0 API](https://github.com/mediacloud/backend/blob/master/doc/api_2_0_spec/api_2_0_spec.md), because it is currently the only public API.
The main API provides functions to retrieve stories, tags, and sentences. Probably due to copyright reasons, the API does not provide full-text stories. But it is possible to pull document-term matrices from the API.
## Prerequisites
* What are the prerequisites to access the API (authentication)? *
An API Key is required. One needs to register for an account at the [official website of Media Cloud](https://tools.mediacloud.org/#/user/signup). After having the access, click on [your profile](https://explorer.mediacloud.org/#/user/profile) to obtain the API Key.
It is recommended to set the API key as the environment variable `MEDIACLOUD_API_KEY`. Please consult [Chapter 2](#best-practices) on how to do that in the section on Environment Variables.
## Simple API call
* *What does a simple API call look like?*
The API documentation is available [here](https://github.com/mediacloud/backend/blob/master/doc/api_2_0_spec/api_2_0_spec.md). Please note the [request limit](https://github.com/mediacloud/backend/blob/master/doc/api_2_0_spec/api_2_0_spec.md#request-limits).
The most important end points are:
1. GET api/v2/media/list/
2. GET api/v2/stories_public/list
3. GET api/v2/stories_public/count
4. GET api/v2/stories_public/word_matrix
It is also important to learn about how to [write a solr query](https://mediacloud.org/getting-started-guide). It is used in either `q` ("query") and `fq` ("filter query") of many end point requests. For example, to search for stories with both "mannheim" and "university" in the New York Times (media_id = 1), the solr query should be: `text:mannheim+AND+text:university+AND+media_id:1`.
In this example, we are going to search for 20 stories in the New York Times (Media ID: 1) mentioning "mannheim AND university".
```{r}
library(httr)
library(stringr)
url <- parse_url("https://api.mediacloud.org/api/v2/stories_public/list")
params <- list(q = "text:mannheim+AND+text:university+AND+media_id:1",
key = Sys.getenv("MEDIACLOUD_API_KEY"))
url$query <- params
final_url <- str_replace_all(build_url(url), c("%3A" = ":", "%2B" = "+"))
res <- GET(final_url)
httr::content(res)
```
## API access in R
* *How can we access the API from R (httr + other packages)?*
As of writing, there are (at least) four R packages for accessing the Media Cloud API. Although the `mediacloudr` package by Dix Jan is available on CRAN, I recommend using the `mediacloud` package by Julian Unkel (LMU). It is available on Dr Unkel's [Github](https://github.com/joon-e/mediacloud). By default, the package always returns "tidy" objects. The package can be installed by:
```{r, eval=F, comment=NA}
devtools::install_github("joon-e/mediacloud")
```
The above "mannheim" example can be replaced by (the package looks for the environment variable `MEDIACLOUD_API_KEY` automatically):
<!-- cached as `mediacloud_mc_mannheim.RDS` -->
```{r, eval=F, comment=NA}
library(mediacloud)
mc_mannheim <- search_stories(text = "mannheim AND university", media_id = 1, n = 20)
mc_mannheim
```
```{r mediacloud-3, echo = FALSE, purl=F}
mc_mannheim <- readRDS("data/mediacloud_mc_mannheim.RDS")
mc_mannheim
```
### Media keywords of "Universität Mannheim"
In the following slightly more sophisticated example, we are going to first search for a list of all national German media outlets, search for a bunch of (German) articles mentioning "Universität Mannheim", and then extract keywords using term frequency-inverse document frequency (TF-IDF). There are three steps.
#### Search for all national German media outlets
All major German media outlets are tagged with `Germany___National`. The function `search_media()` is used to retrieve information about all national German media outlets.
<!-- cached as `mediacloud_de_media.RDS` -->
```{r, eval=F, comment=NA}
de_media <- search_media(tag = "Germany___National", n = 100)
```
```{r mediacloud-4, echo = FALSE, purl=F}
de_media <- readRDS("data/mediacloud_de_media.RDS")
de_media
```
#### Pull a list of articles
<!-- cached as `mediacloud_unima_articles.RDS` -->
The following query gets a list of 100 articles mentioning "universität mannheim" published in a specific date range from all national German media outlets. Unlike the AND operator, this search for the exact term. Also, a query is case insensitive. The function `search_stories()` can be used for this.
```{r, eval=F, comment=NA}
unima_articles <- search_stories(text = "\"universität mannheim\"",
media_id = de_media$media_id,
n = 100,
after_date = "2021-01-01",
before_date = "2021-12-01")
unima_articles
```
```{r mediacloud-5, echo = FALSE, purl=F}
unima_articles <- readRDS("data/mediacloud_unima_articles.RDS")
unima_articles
```
#### Pull word matrices
With the list of `stories_id`, we can then use the function `get_word_matrices()` to obtain word matrices. [^WORDMATRICES]
<!-- cached as `mediacloud_unima_mat.RDS` -->
```{r, eval=F, comment=NA}
unima_mat <- get_word_matrices(stories_id = unima_articles$stories_id, n = 100)
```
```{r mediacloud-6, echo = FALSE, purl=F}
unima_mat <- readRDS("data/mediacloud_unima_mat.RDS")
unima_mat
```
The data frame `unima_mat` is in the so-called "tidytext" format [@silge2016tidytext]. It can be used directly for analysis if one is fond of tidytext. For users of quanteda [@benoit2018quanteda], it is also possible to cast the data frame into a Document-Feature Matrix (DFM) [^DEDUP].
<!-- cached as `mediacloud_unima_dfm.RDS` -->
```{r, eval=F, comment=NA}
library(tidytext)
library(quanteda)
unima_dfm <- cast_dfm(unima_mat, stories_id, word_stem, word_counts)
unima_dfm
```
```{r mediacloud-7, echo = FALSE, message = FALSE, purl=F}
library(quanteda)
unima_dfm <- readRDS("data/mediacloud_unima_dfm.RDS")
unima_dfm
```
And then standard operations can be done.
```{r mediacloud-8}
unima_dfm %>% dfm_tfidf() %>% topfeatures(n = 20)
```
The faculties of *BWL* (Business Administration) and *Jura* (Law) would be happy with this finding.
## Social science examples
* *Are there social science research examples using the API?*
According to the paper by the official Media Cloud Team [@roberts2021media], there are over 100 papers mentioning Media Could. Many papers use the counting endpoint to generate a time series of media attention to specific keywords [e.g. @benkler:2015:SMN;@huckins:2020:MHB]. This function is widely used also in many [data journalism pieces](https://mediacloud.org/publications). The URLs collected from Media Cloud can also be used to do further crawling [e.g. @huckins:2020:MHB].
It is perhaps worth mentioning that the openly available useNews dataset [@puschmann:2021] provides a large collection of content from Media Cloud together with meta data other data sources.
[^WORDMATRICES]: For the sake of education, I split step 2 and 3 into two steps. Actually, it is possible to merge step 2 and step 3 by simply: `get_word_matrices(text = "\"universität mannheim\"")`
[^DEDUP]: It is quite obvious that there are (many) duplicates in the retrieved data. For example, the first few documents are almost the same in the feature space. Packages such as [textsdc](https://github.com/chainsawriot/textsdc) might be useful for deduplication.