library(httr)\nlibrary(jsonlite)\nlibrary(purrr)\nlibrary(rvest)\nknitr::opts_chunk$set(eval = FALSE)\nAPI_key <- Sys.getenv("rateBeer_API_key")In my last post I showed how using the package httr, You can access the RateBeer API to get information about beers made by a brewery. When I left off, I showed a problem - the API only shows 10 beers at a time.
Today, I’m going to show how we can get more beers at once. After that, I’m going to show how we can use the API, rvest, and purrr to get beers from all the brewers around me.
first: argumentIn the last post, I didn’t mention one of the arguments that can be used when making beersByBrewer query. Besides the argument for the brewerID, we can also use the first argument to specify how many beers we want to see from a brewer. As you will see below, one of the changes I make to the call to the API includes seting the value for the argument first to 999.
Ninety-nine beers on the wall? Let’s make it Nine hundred and ninety-nine.
\nAs you will see later on, it will be handy to have the call to the API as a function. Below, I declare that function:
\nget_beers_from_brewer <- function(brewer_id, api_key) {\n URL <- "https://api.ratebeer.com/v1/api/graphql"\n POST(URL,\n body = list(\n query = paste0(\n"query{\n beersByBrewer(brewerId: ", brewer_id, ", first: 999) {\n totalCount\n items{\n name\n abv\n averageRating\n ratingCount\n isRetired\n style{\n name\n }\n brewer {\n id\n name\n streetAddress\n city\n state {\n name\n }\n zip\n }\n }\n }\n}"),\n variables = "{}",\n operationName = NULL),\n encode = "json", # tells httr to encode the body of the request as json\n add_headers("content-type" = "application/json",\n "Accept" = "application/json",\n "x-api-key" = api_key))\n}Okay, so now, I have an easy way to get information about the beers that breweries make. All I need to do now is point the API to the brewer ID of each brewery I want information on.
\nRemember, the brewer id can be found by looking at the url for that brewery and is in the form:
\nhttps://www.ratebeer.com/brewers/<BREWERY_NAME_HERE>/<BREWER_ID_HERE>
The thing is, I like data and want A LOT of it. There’s no way I’m going to hand sort through urls to try to find breweries. Can’t this be automated?
\nYES!
\nRateBeer maintains lists of breweries by state. For example, the breweries in Pennsylvania can be found here. Using the package rvest, we can pull information about all of the breweries in the state, and then use purrr to iterate over that list, using the function, get_beers_from_brewer().
rvest is a package to make harvesting information from the web easy. Below, you see how, in three steps, I have a list of urls that point to all of the breweries in the state.
library(rvest)\n\nbrewery_list_url <- "https://www.ratebeer.com/breweries/pennsylvania/38/213/"\n\nbrewery_ids <- read_html(brewery_list_url) %>%\n html_nodes("#brewerTable a:nth-child(1)") %>%\n html_attr('href')To explain what happened in the previous code chunk:
\nread_html() loads the url for the list of breweris in PA. This is the same thing that happens if you click this link.html_nodes() searches for every place in the html file we navigated to that has a link to a brewery. The text in the argument for the function points to the css selector for where breweries can be found on the page. I found this selector using selectorgadget. This function gives me a list of the each time that selector shows up.html_attr() searches that list for an attribute of the specified type. In this case I specified a hyperlink.yards_url <- grep("yards", brewery_ids, value = TRUE)As I said, this outputs a list of urls. As an example, the url for Yards Brewing looks like this:
\n\n\n\n
/brewers/yards-brewing-company/166/
Now, I can take this list of urls, and use the function purrr::map_chr() to get a list of brewer IDs. As a note, I use map_chr because it flattens the list of IDs into a single character vector. I highly suggest that you check out the rest of the map_* functions.
Below, I take each item in the list of urls, split each url at any "/" and then use map_chr() to select the fourth element, the brewer ID.
brewery_ids <- brewery_ids %>%\n strsplit("/") %>%\n map_chr(4)Now I have a list of brewery IDs that I can feed into get_beers_from_brewer()
To iterate through the list, we’ll use map() and some functions from jsonlite, a package that can parse JSON. Then we’ll use map() to work through the levels of the response, from it’s highest level (“data”) down to the actual data frame (held in the named object, “items”).
brewery_beer_df <- brewery_ids %>%\n map(function(brewer_id){\n Sys.sleep(1) # the API restricts to 1 request per second.\n get_beers_from_brewer(brewer_id, API_key)\n }) %>%\n # Here we use jsonlite functions to turn the response of the request into\n # json\n map(content, type = "text") %>%\n # and then turn that into an r object which has dfs in it from each\n # brewery.\n map(fromJSON, flatten = TRUE) %>%\n # working down through the response levels.\n map("data") %>% map("beersByBrewer") %>% map_dfr("items")The object brewery_beer_df is the data frame with all the beers from breweries we requested.