Welcome to the first in a series of blog posts surrounding R’s {golem} package by Colin Fay. In case you missed it, we laid out what these posts will cover in last week’s post.

One main step of our development process is to think about how we want the app to look structurally and aesthetically. This is an important step since it directly influences the user’s experience. Before sketching out the app, we must have a better understanding of the data that is available to us. In our scenario with The Office, we have data regarding: advertising, revenue, ratings, and the script.

Let’s take a look at these datasets:

Script

Each row in the script dataset contains information surrounding the season, episode, script text, the character speaking, and whether the line was deleted or not. This dataset is great for a text analysis. We can likely disregard the “deleted” column and work on breaking down our script analysis by seasons and episodes.

Ratings

The rows in this dataset provide us with a season number, title, air date, rating, number of votes, description, director(s), and writer(s) for each episode. However, we notice that the episode’s number is not included. We will add an episode number identifier later on. Similar to the script data, we can gain insight for ratings across episodes, writers, and directors.

Revenue and Advertising

For the purpose of this example, we simulated data to represent revenue and advertising. The simulated revenue and advertising values start on the first episode air date, 3/24/05, and end on the last episode air date. The advertising sectors included are: Google, Facebook, Instagram, and display.

Based on specifications decided by the client, we agreed to the following layout for the Shiny dashboard:

From this glimpse of the app’s structure, we see that the sales analysis contains an overview page and a breakdown by seasons. We’re able to analyze ratings further by looking into episodes, characters, writers, and directors. The dashboard also includes a script analysis. Additionally, we can begin to think about possible plots and charts to incorporate.. Perhaps a plot of ratings across episodes for each season, a table of average seasonal ratings, or a pie chart of advertising expenditures.

In thinking ahead, we’ll stay organized by creating modules for each of these tabs to help keep it organized.

In our next post we will initialize our project in R and discuss the files/tools that come with golem. Until next time..

“This is a dream that I’ve had…since lunch…and I’m not giving it up now.” – Michael Scott

SCRIPT_DATA <- readr::read_csv('The Office - Script Data.csv')head(SCRIPT_DATA)IMDB_DATA <- readr::read_csv('The Office - IMDB Data.csv')head(IMDB_DATA)REVENUE_ADV <-  readr::read_csv('revenue_and_adv.csv')head(REVENUE_ADV)

Data sources used:

https://data.world/abhinavr8/the-office-scripts-dataset

https://www.kaggle.com/kapastor/the-office-imdb-ratings-per-episode

The post R Shiny {golem} - Designing the UI - Part 1 - Development to Production first appeared on R-bloggers.

In statistics, Kolmogorov–Smirnov test is a popular procedure to test, from a sample \(\{x_1,\cdots,x_n\}\) is drawn from a distribution \(F\), or usually \(F_{\theta_0}\), where \(F_{\theta}\) is some parametric distribution. For instance, we can test \(H_0:X_i\sim\mathcal{N(0,1)}\) (where \(\theta_0=(\mu_0,\sigma_0^2)=(0,1)\)) using that test. More specifically, I wanted to discuss today \(p\)-values. Given \(n\) let us draw \(\mathcal{N}(0,1)\) samples of size \(n\), and compute the \(p\)-values of Kolmogorov–Smirnov tests

n=300p = rep(NA,1e5)for(s in 1:1e5){X = rnorm(n,0,1)p[s] = ks.test(X,"pnorm",0,1)$p.value}

We can visualise the distribution of the \(p\)-values below (I added some Beta distribution fit here)

library(fitdistrplus)fit.dist = fitdist(p,"beta")hist(p,probability = TRUE,main="",xlab="",ylab="")vu = seq(0,1,by=.01)vv = dbeta(vu,shape1 = fit.dist$estimate[1], shape2 = fit.dist$estimate[2])lines(vu,vv,col="dark red", lwd=2)

It looks like it is quite uniform (theoretically, the \(p\)-value is uniform). More specifically, the \(p\)-value was lower than 5% in 5% of the samples

[note: here I compute‘mean(p<=.05)’ but I have some trouble with the ‘<‘ and ‘>’ symbols, as always]

mean(p<=.05)[1] 0.0479

i.e. we wrongly reject \(H_0:X_i\sim\mathcal{N(0,1)}\) is 5% of the samples.

As discussed previously on the blog, in many cases, we do care about the distribution, and not really the parameters, so we wish to test something like \(H_0:X_i\sim\mathcal{N(\mu,\sigma^2)}\), for some \(\mu\) and \(\sigma^2\). Therefore, a natural idea can be to test \(H_0:X_i\sim\mathcal{N(\hat\mu,\hat\sigma^2)}\), for some estimates of \(\mu\) and \(\sigma^2\). That’s the idea of Lilliefors test. More specifically, Lilliefors test suggests to use , Kolmogorov–Smirnov statistics, but corrects the \(p\)-value. Indeed, if we draw many samples, and use Kolmogorov–Smirnov statistics and its classical \(p\)-value to test for \(H_0:X_i\sim\mathcal{N(\hat\mu,\hat\sigma^2)}\),

n=300p = rep(NA,1e5)for(s in 1:1e5){X = rnorm(n,0,1)p[s] = ks.test(X,"pnorm",mean(X),sd(X))$p.value}

we see clearly that the distribution of \(p\)-values is no longer uniform

fit.dist = fitdist(p,"beta")hist(p,probability = TRUE,main="",xlab="",ylab="")vu = seq(0,1,by=.01)vv = dbeta(vu,shape1 = fit.dist$estimate[1], shape2 = fit.dist$estimate[2])lines(vu,vv,col="dark red", lwd=2)

More specifically, if \(x_i\)‘s are actually drawn from some Gaussian distribution, there are no chance to reject \(H_0\), the \(p\)-value being almost never below 5%

mean(p<=.05)[1] 0.00012

Usually, to interpret that result, the heuristics is that \(\hat\mu\) and \(\hat\sigma^2\) are both based on the sample, while previously \(0\) and \(1\) where based on some prior knowledge. Somehow, it reminded me on the classical problem when mention when we introduce cross-validation, which is Goodhart’s law

When a measure becomes a target, it ceases to be a good measure

i.e. we cannot assess goodness of fit using the same data as the ones used to estimate parameters. So here, why not use some hold-out (or cross-validation) procedure : split the dataset in two parts, \(\{x_1,\cdots,x_k\}\) (with \(kKolmogorov–Smirnov statistics on it to test if [latex]x_i\)‘s are drawn from some Gaussian distribution. More precisely, will the \(p\)-value computed using the standard Kolmogorov–Smirnov procedure be ok here. Here, I tried two scenarios, \(k/n\) being either \(1/3\) or \(2/3\),

p = matrix(NA,1e5,4)for(s in 1:1e5){X = rnorm(n,0,1)p[s,1] = ks.test(X,"pnorm",0,1)$p.valuep[s,2] = ks.test(X,"pnorm",mean(X),sd(X))$p.valuep[s,3] = ks.test(X[1:200],"pnorm",mean(X[201:300]),sd(X[201:300]))$p.valuep[s,4] = ks.test(X[201:300],"pnorm",mean(X[1:200]),sd(X[1:200]))$p.value}

Again, we can visualize the distributions of \(p\)-values,  in the case where \(1/3\) of the data is used to estimate \(\mu\) and \(\sigma^2\), and \(2/3\) of the data is used to test

fit.dist = fitdist(p[,3],"beta")hist(p[,3],probability = TRUE,main="",xlab="",ylab="")vu=seq(0,1,by=.01)vv=dbeta(vu,shape1 = fit.dist$estimate[1], shape2 = fit.dist$estimate[2])lines(vu,vv,col="dark red", lwd=2)

and in the case where \(2/3\) of the data is used to estimate \(\mu\) and \(\sigma^2\), and \(1/3\) of the data is used to test

fit.dist = fitdist(p[,4],"beta")hist(p[,4],probability = TRUE,main="",xlab="",ylab="")vu=seq(0,1,by=.01)vv=dbeta(vu,shape1 = fit.dist$estimate[1], shape2 = fit.dist$estimate[2])lines(vu,vv,col="dark red", lwd=2)

Observe here that we (wrongly) reject too frequently \(H_0\), since the \(p\)-values are  below 5% in 25% of the scenarios, in the first case (less data used to estimate), and 9% of the scenarios, in the second case (less data used to test)

mean(p[,3]<=.05)[1] 0.24168mean(p[,4]<=.05)[1] 0.09334

We can actually compute that probability as a function of \(k/n\)

n=300p = matrix(NA,1e4,99)for(s in 1:1e4){  X = rnorm(n,0,1)  KS = function(p) ks.test(X[1:(p*n)],"pnorm",mean(X[(p*n+1):n]),sd(X[(p*n+1):n]))$p.value  p[s,] = Vectorize(KS)((1:99)/100)}

The evolution of the probability is the following

prob5pc = apply(p,2,function(x) mean(x<=.05))plot((1:99)/100,prob5pc)

so, it looks like we can use some sort of hold-out procedure to test for \(H_0:X_i\sim\mathcal{N(\mu,\sigma^2)}\), for some \(\mu\) and \(\sigma^2\), using Kolmogorov–Smirnov test with \(\mu=\hat\mu\) and \(\sigma^2=\hat\sigma^2\) but the proportion of data used to estimate those quantities should be (much) larger that the one used to compute the statistics. Otherwise, we clearly reject too frequently \(\H_0\).

The post Lilliefors, Kolmogorov-Smirnov and cross-validation first appeared on R-bloggers.

My daughter received the board game First Orchard as a Christmas present and she’s hooked on it so far. In playing the game with her, a few probability/statistics questions came to mind. This post outlines how I answered some of them using simulation in R. All code for this blog post can be found here.

(In my googling I found that Matt Lane has an excellent blog post on this game, answering some of the questions that I was interested in.)

Gameplay

Before we get to the questions, let me give a quick explanation of how the game works (see Reference 1 for a more colorful explanation as well as an applet to play the game online).

• It’s a cooperative game, with all players playing against the board.
• The game starts with 16 fruit: 4 of each color (red, green, blue, yellow), and a raven at one end of a path that is 5 steps long.
• On each player’s turn, the player rolls a 6-sided die.
  • If the die comes up red, green, blue or yellow, the player gets to “harvest” a fruit of that color if there are any left to harvest. If all 4 fruits of that color have been harvested, nothing happens.
  • If the die shows a fruit basket, the player gets to harvest a fruit of any color.
  • If the die shows a raven, the raven moves one step along the path.
• The game ends when either all the fruit are harvested (players win) or when the raven reaches the end of the path (raven wins).

As you can see this is a really simple game (hence it’s “suitable for 2+ years” rating). The only strategy is in choosing what fruit to take if a fruit basket shows up: see Reference 1 for some simulations for different strategies. Intuitively it seems like choosing the color with the most fruit remaining is the best strategy, and that’s what I bake into my code. (Is there a proof for this, and is this still true in more general circumstances described in Reference 1?)

Code for simulating the game

The state of the game can be captured in a numeric vector of length 5. The first 4 numbers refer to the number of fruit left for each color, and the 5th number keeps track of the number of steps the raven has taken so far. I created 3 functions to simulate one game of First Orchard (see full code here):

• SimulateTurn(state, verbose) takes one dice roll and updates the state of the game. For simplicity, if a 1-4 is rolled, a fruit is harvested from that corresponding tree. If 5 is rolled, the raven takes a step. A rolled 6 is taken to mean “fruit basket”, and I remove a fruit from the tree with the most remaining fruits.
• CheckGameState(state, max_raven_steps) checks if the game has ended or not, and if so, who won.
• SimulateGame(fruit_count, max_raven_steps, verbose) runs an entire game of First Orchard: while the game has not ended, run SimulateTurn. Once the game has ended, this function returns (i) who won, (ii) the number of turns taken, (iii) the number of steps the raven took, and (iv) the number of fruit left.

We allow for two game parameters to be defined by the user: the number of fruit of each type at the start of the game (fruit_count, default is 4) and the number of steps the raven must take in order for it to win (max_raven_steps, default is 5). The verbose option for these functions so that the user can see what happened in the game. The code below is an example of the output from SimulateGame:

set.seed(1)results <- SimulateGame(fruit_count = 2, max_raven_steps = 3,                         verbose = TRUE)# Roll: 1 , State: 1,2,2,2,0# Roll: 4 , State: 1,2,2,1,0# Roll: 1 , State: 0,2,2,1,0# Roll: 2 , State: 0,1,2,1,0# Roll: 5 , State: 0,1,2,1,1# Roll: 3 , State: 0,1,1,1,1# Roll: 6 , State: 0,0,1,1,1# Roll: 2 , State: 0,0,1,1,1# Roll: 3 , State: 0,0,0,1,1# Roll: 3 , State: 0,0,0,1,1# Roll: 1 , State: 0,0,0,1,1# Roll: 5 , State: 0,0,0,1,2# Roll: 5 , State: 0,0,0,1,3# Raven wins# # of turns: 13# # of steps raven took: 3# # fruit left: 1

Simulation time!

What is the probability of winning the game? How does that change as we vary (i) the number of fruit of each color, and (ii) the number of steps the raven must take in order for the players to lose?

We let the number of fruit of each color vary from 1 to 8, and the number of steps the raven must take from 1 to 8. For each parameter setting, we simulate the game 10,000 times and compute the player win probability. We plot the results as a heatmap.

As one might expect, win probability goes down as the number of fruit increases and as the number of steps the raven must take decreases. For the original game (4 fruit, 5 steps), the win probability is approximately 62%. Sounds reasonable: we would like a win probability >50% so that kids will not get discouraged by too much losing, but not so high that they think the game is trivial.

For the original game (4 fruit, 5 steps), what is the expected number of steps until the game ends? Does this change depending on whether the player or the raven wins?

We simulate the game 100,000 times and keep track of the number of steps taken in each game. The shortest game took 5 steps while the longest took 45 steps, with the modal number of steps being 21 (it also happens to be the mean and median). Here is the histogram for all 100,000 runs:

Here are the histograms split by the outcome:

Games where the raven wins tend to be shorter than those when players win. Maybe that’s not too surprising, since a game where the raven wins needs just 5 steps, while a game where the players win needs at least 16 steps. On average, the game takes 19 steps for raven wins and 22 steps for player wins.

For the original game (4 fruit, 5 steps), given that the raven loses, what is distribution of the number of steps the raven has taken?

Because we programmed SimulateGame to return the number of steps the raven has taken as well, we don’t have to rerun the simulations: we can just use the 100,000 simulations we ran previously and look at the ones that the raven lost. Here is the histogram of steps the raven took in losing games, with the vertical red line representing the mean:

For the original game (4 fruit, 5 steps), given that the raven wins, what is distribution of the number of unharvested fruit?

Again, we can just use the 100,000 simulations we ran previously and look at the ones that the raven won. Here is the histogram along with the mean and median marked out with vertical lines:

The modal number of fruit left in player-losing games is 1: ugh tantalizingly close!

If there was no raven, how many turns would it take to harvest all the fruit?

“No raven” is the same as saying that the raven needs to take an infinite number of steps in order to win. Hence, we can use our existing simulation code with max_raven_steps = Inf to simulate this setting.

The shortest game took 16 turns while the longest game took 63 turns, with 22 and 24 turns being the modal and mean number of turns respectively. (In theory, a game could go on forever.) Here is the histogram:

References:

1. Matt Lane. (2018). Harvesting Wins.

The post Exploring the game “First Orchard” with simulation in R first appeared on R-bloggers.

 Added to my base stock position, very small. +1.7 %

The post Trades Jan. 4, 2021 first appeared on R-bloggers.

By: Brad LindbladLinkedIn | Github | Blog | Twitter

I recently gave a lightning talk at the Financial Industry R Meetup on how you can use RMarkdown to create extremely professional reports using RMarkdown and a slew of other popular R tools.

You can re-watch the talk at this link, use password: 9np$u.=p

You can also read the directions and overview at the Github repo.

The post Professional Financial Reports with RMarkdown first appeared on R-bloggers.

Photo by Alan Biglow on Unsplash

Back in November, I took readers step by step through the somewhat long process of authenticating and downloading Google Analytics web site data into R. This post will be much simpler; I’m going to walk you through creating a dashboard showing blog post popularity using the flexdashboard package.

Before we go there, however, I want to re-emphasize a correction that we made to the original credentials post. Mark Edmonston, the author of the terrific googleAnalyticsR package, has created a new version of his package that eliminates the need for OAUTH credentials when running on a server. Once that update is available on CRAN, I’ll update this post to document the simpler process of only submitting service account credentials. In the meantime, though, we’ll continue using both OAUTH and service account credentials.

library(flexdashboard)library(readr)library(ggplot2)library(dplyr)library(reactable)gadata <- read_csv("./data/clickbait_GA_data.csv")show(gadata %>% head(7))## # A tibble: 7 x 5##   date       pageviews users pageTitle            landingPagePath               ##                                                       ## 1 2020-12-01         2     2 3 Ways That Turtles… www.clickbait.com/2011/02/28/…## 2 2020-12-01         2     1 3 Ways That Turtles… www.clickbait.com/2011/02/28/…## 3 2020-12-01         3     3 Shocking Finding: W… www.clickbait.com/2012/06/04/…## 4 2020-12-01         1     1 Unexpected Research… www.clickbait.com/2012/11/29/…## 5 2020-12-01        11    10 Unexpected Research… www.clickbait.com/2013/06/10/…## 6 2020-12-01         1     1 3 Ways That Europea… www.clickbait.com/2013/10/22/…## 7 2020-12-01         2     2 Why Monkeys Deal wi… www.clickbait.com/2014/01/17/…

theme_set(theme_minimal())gadata_by_day <- gadata %>%   group_by(date) %>%   summarize(pagesums = sum(pageviews))g <- ggplot(gadata_by_day, aes(x = date, y = pagesums)) +  geom_point(color = "blue") +  geom_line(color = "blue") +  scale_x_date() +  labs(x = "", y = "", title = "")show(g)

gadata_most_popular <- gadata %>%   count(pageTitle, wt = pageviews, sort=TRUE) %>%   head(10)## For those who aren't as comfortable with the options in count, the following## code would also work# gadata_most_popular <- gadata %>% #   group_by(pageTitle) %>% #   summarize(n = sum(pageviews)) %>% #   arrange(desc(n))reactable(gadata_most_popular,           columns = list(pageTitle     = colDef(name = "Title",                                            align = "left",                                             maxWidth = 250),                          n             = colDef(name = "Page Views",                                             maxWidth = 100)),            pagination = FALSE,            searchable = TRUE,            striped = TRUE)

The post Custom Google Analytics Dashboards with R: Building The Dashboard first appeared on R-bloggers.

The Question

I am a sucker for IMDb ratings so don’t judge me. They are my priors before watching almost anything on a screen (home screen that is). But between movies (feature films), TV movies and TV (mini) series IMDb ratings are highly inconsistent. For example, series The Boys has rating 8.7 and so does movie Goodfellas by Martin Scorsese. Does it make sense The Boys ranked as high as #16 rated movie title in the whole IMDb database (among those with at least 25,000 user votes)? Or, in other words, if and how much apples vs. oranges those ratings are?

Rating Distributions

To start I downloaded IMDb datasets (here). Let’s show distributions of title ratings depending on the types: movie (i.e. feature film), TV movie, TV mini series, and TV series between fiction and documentaries:

Title ratings drift towards higher values depending on their types (shown on the right): movie, TV movie, TV mini series, and TV series. So indeed ratings of movies and TV series come from different distributions representing different things like apples and oranges. But how much different they are? (we will focus on fiction titles only from this point on.)

Percentiles

If a title has all time best rating then no doubt it’s worth giving a try (let’s say among titles with at least 1000 votes – number of votes is rather important consideration but we let it slide here and may come back to votes later). Why? Because 100% of other titles are rated below or at best the same and that indicates exceptional qualities. In statistics such rating has a name: 100th percentile. Following the same logic 99th percentile represents rating above 99% of all titles in the database (again, don’t forget about minimum threshold for number of votes to be considered).

Based on above we can assign IMDb titles to groups based on the highest percentile they belong to: 99% percentile suggests that the title is very best, 95% – excellent, 90% – very good, 75% – good, 50% – average, and 25% – bad. Feel free to assign and name percentiles differently in your analysis but we stick with this convention for this post. Last piece of the puzzle is taking percentiles not across whole IMDb set but rather for each title type separately and compare them:

Going back to our example, 8.7 in TV Series places The Boys firmly in “Excellent” (95th percentile), while Goodfellas at 8.7 sits at the top of “Very Best” (99th percentile) in movies – noticeable difference between the two.

The difference becomes even more meaningful when looking at the lower tiers “Very Good” (90th percentile) and below: while rating of 7.6 suffices for a movie (e.g. Love Actually) to place in “Very Good”, a TV series must achieve rating of 8.4 to qualify for the same 90th percentile. In fact, a TV Series with 7.6 rating (like Grey’s Anatomy) places just above “Average” 50th percentile. Furthermore, the rating of 8 would place a movie firmly in top 5% while the same 8 for a TV series barely cracks top 25%.

Percentiles Extra

Comparing and analyzing ratings between title types can be helped by organizing and visualizing the same percentile data in a few different ways:

• Overlapping bar charts by title types:

• Line chart by title types:

• Line chart by percentiles:

What About Documentaries?

The title percentiles above excluded documentaries. To be able to compare ratings between fiction and documentary titles the following visual computes and dissects rating percentiles between fiction and documentaries by title types:

Historical Perspective Mixed with Film Trivia

The oldest film on IMDb is Passage de Venus made in 1874, is ranked 6.9 with 1282 votes (as of January 2020), and is filed under title type short and genre Documentary. In chronological order it is followed by 2 titles in 1878 (short animation Le singe musicien and short documentary Sallie Gardner at a Gallop), 1 in 1881 (short documentary Athlete Swinging a Pick), 1 in 1883 (short documentary Buffalo Running), and 1 in 1885 (short animation L’homme machine). Starting with 1887 that cranked up 45 titles total there are no more gap years, but such production feast will be surpassed only 1894 with 97 titles. First movie title (and only that year) Reproduction of the Corbett and Fitzsimmons Fight was filmed in 1897 under Documentary, News, and Sport genres. Lastly, first year when total number of titles exceeded its year numerical value is 1952 with 2059 shorts, movies, etc. under the belt. Did I just say last? One more factoid if you excuse me: movie production in 2020 (35,109 titles total) dropped us exactly 10 years back when 35,062 titles were produced in 2010, while the absolute record belongs to 2017 with 51231 films total.

What about visualizing film production over time?

Final Thoughts

The post IMDb datasets: 3 centuries of movie rankings visualized first appeared on R-bloggers.

Three months ago we finished Why R? 2020 conference. One of the memorable moments of the conference were invited highlighted talks! Today we would like to remind you about the talk by Emil Hvitfeldt (from University of Southern California). The video from the recording is at the end of the post.

There are many palettes available in various R packages. Having a way to explore all of these palettes are already found within the https://github.com/EmilHvitfeldt/r-color-palettes repository and the paletteer package.

This talk shows what happens when we take one step further into explorability. Using handcrafted color features, dimensionality reduction, and interactive tools will we create and explore a color palette embedding. In this embedded space will we interactively be able to cluster palettes, find neighboring palettes, and even generate new palettes in a whole new way.

The post Emil Hvitfeldt - palette2vec - A new way to explore color paletttes first appeared on R-bloggers.

The world seems to have moved to a new phase of paying attention to COVID-19. We have gone from pondering daily plots of case counts, to puzzling through models and forecasts, and are now moving on to the vaccines and the science behind them. For data scientists, however, the focus needs to remain on the data and the myriad issues and challenges that efforts to collect and curate COVID data have uncovered. My intuition is that not only will COVID-19 data continue to be important for quite some time in the future, but that efforts to improve the quality of this data will be crucial for successfully dealing with the next pandemic.

An incredible amount of work has been done by epidemiologists, universities, government agencies and data journalists to collect, organize, and reconcile data from thousands of sources. Nevertheless, the experts caution that there is much yet to be done.

Roni Rosenfeld, head of the Machine Learning Department of the School of Computer Science at Carnegie Mellon University and project lead for the Delphi Group put it this way in a recent COPSS-NISS webinar:

Data is a big problem in this pandemic. Availability of high quality, comprehensive, geographically detailed data is very far from where it should be.

There are over 6,000 hospitals in the United States, and over 160,000 hospitals worldwide. Many of these are collecting COVID-19 data yet there few standards for recording cases, dealing with missing data, updating case count data, and coping with the time lag between recording and reporting cases. Nowcasting epidemiological and heath care data has become a vital field of statistical research.

The following slide from the COPSS-NISS webinar shows a hierarchy of relevant COVID data organized on the Severity Pyramid that epidemiologists use to study disease progression.

The Delphi Group is making fundamental contributions to the long term improvement of COVID data by archiving the data shown in such a way that versions can be retrieved by date, and also by collecting massive data sets of leading indicators.

The webinar is well worth watching, and I highly recommend listening through the Q&A session at the end. The speakers explain the importance of nowcasting and Professor Rosenfeld presents a vision of making epidemic forecasting comparable to weather forecasting. It seems to me that this would be a worthwhile project to help advance.

Note that the Delphi’s COVID-19 indicators, probably the nation’s largest public repository of diverse, geographically-detailed, real-time indicators of COVID activity in the US, are freely available through the public API which is easily accessible to R and Python users.

Also note that R users can contribute to R Consortium sponsored COVID related projects that include the COVID-19 Data Hub an organized archive of global COVID-19 case count data, and the RECON COVID-19 Challenge, an open source project to improve epidemiological tools.

The post COVID-19 Data: The Long Run first appeared on R-bloggers.

The Covid19 pandemic had, and unfortunately still have, a significant impact on most of the major industries. While for some sectors, the impact was positive (such as online retails, internet and steaming providers, etc.), it was negative for others (such as transportation, tourism, entertainment, etc.). In both cases, we can leverage time series modeling to quantify the effect of the Covid19 on the sector.

One simplistic approach for quantifying the impact of the pandemic (whether it is positive or negative) would include the following steps:

• Split the series into pre-covid and post-covid
• Train a time series model with the pre-covid data. This would enable us to simulate the value of the series if there was no pandemic
• Using the trained model to forecast the horizon of the post-covid series
• Use the difference between the forecast and actual (post-covid series) to quantify the Covid19 impact on the series

To demonstrate this approach, I will use the sfo_passengers dataset from the sfo package. The sfo_passengers dataset provides monthly statistics about the San Francisco International Airport (SFO) air traffic between July 2005 and September 2020. More details about the dataset available in the following post and vignette.

For this analysis, we will use the following packages:

• sfo– the passenger air traffic data
• dplyr– data prep
• plotly– data visualization
• TSstudio– time series analysis and forecasting

library(sfo)
library(dplyr)
library(plotly)
library(TSstudio)

data("sfo_passengers")

str(sfo_passengers)
## 'data.frame':    22576 obs. of  12 variables:
##  $ activity_period            : int  202009 202009 202009 202009 202009 202009 202009 202009 202009 202009 ...
##  $ operating_airline          : chr  "United Airlines" "United Airlines" "United Airlines" "United Airlines" ...
##  $ operating_airline_iata_code: chr  "UA" "UA" "UA" "UA" ...
##  $ published_airline          : chr  "United Airlines" "United Airlines" "United Airlines" "United Airlines" ...
##  $ published_airline_iata_code: chr  "UA" "UA" "UA" "UA" ...
##  $ geo_summary                : chr  "International" "International" "International" "International" ...
##  $ geo_region                 : chr  "Mexico" "Mexico" "Mexico" "Mexico" ...
##  $ activity_type_code         : chr  "Enplaned" "Enplaned" "Enplaned" "Deplaned" ...
##  $ price_category_code        : chr  "Other" "Other" "Other" "Other" ...
##  $ terminal                   : chr  "Terminal 3" "Terminal 3" "International" "International" ...
##  $ boarding_area              : chr  "F" "E" "G" "G" ...
##  $ passenger_count            : int  6712 396 376 6817 3851 3700 71 83 65 45 ...

df <- sfo_passengers %>%
  mutate(date = as.Date(paste(substr(sfo_passengers$activity_period, 1,4), 
                              substr(sfo_passengers$activity_period, 5,6), 
                              "01", sep ="/"))) 

df <- df %>%
  group_by(date) %>%
  summarise(y = sum(passenger_count), .groups = "drop")

head(df)  
## # A tibble: 6 x 2
##   date             y
##          
## 1 2005-07-01 3225769
## 2 2005-08-01 3195866
## 3 2005-09-01 2740553
## 4 2005-10-01 2770715
## 5 2005-11-01 2617333
## 6 2005-12-01 2671797

plot_ly(data = df,
        x = ~ date,
        y = ~ y,
        type = "scatter", 
        mode = "line",
        name = "Total Passengers") %>%
  add_segments(x = as.Date("2020-02-01"), 
               xend = as.Date("2020-02-01"),
               y = min(df$y),
               yend = max(df$y) * 1.05,
               line = list(color = "black", dash = "dash"),
               showlegend = FALSE) %>%
  add_annotations(text = "Pre-Covid19",
                  x = as.Date("2018-09-01"),
                  y = max(df$y) * 1.05, 
                  showarrow = FALSE) %>%
  add_annotations(text = "Post-Covid19",
                  x = as.Date("2021-08-01"),
                  y = max(df$y) * 1.05, 
                  showarrow = FALSE) %>%
  layout(title = "Total Number of Air Passengers - SFO Airport",
         yaxis = list(title = "Number of Passengers"),
         xaxis = list(title = "Source: San Francisco Open Data Portal"))

pre_covid <- df %>% 
  dplyr::filter(date < as.Date("2020-03-01")) %>%
  dplyr::arrange(date)

post_covid <- df %>% 
  dplyr::filter(date >= as.Date("2020-03-01")) %>%
  dplyr::arrange(date)

ts.obj <- ts(pre_covid$y, start = c(2005, 7), frequency = 12)

ts_plot(ts.obj,
        title = "Total Number of Air Passengers - SFO Airport",
        Ytitle = "Number of Passengers",
        slider = TRUE)

ts_seasonal(ts.obj = ts.obj, type = "all")

ts_cor(ts.obj = ts.obj, lag.max = 36)

methods <- list(ets1 = list(method = "ets",
                            method_arg = list(opt.crit = "lik"),
                            notes = "ETS model opt.crit=lik"),
                ets2 = list(method = "ets",
                            method_arg = list(opt.crit = "amse"),
                            notes = "ETS model opt.crit=amse"),
                ets3 = list(method = "ets",
                            method_arg = list(opt.crit = "mse"),
                            notes = "ETS model opt.crit=mse"),
                auto_arima = list(method = "auto.arima",
                              notes = "Auto ARIMA"),
                hw1 = list(method = "HoltWinters",
                          method_arg = NULL,
                          notes = "HoltWinters Model"),
                 hw2 = list(method = "HoltWinters",
                          method_arg = list(seasonal = "multiplicative"),
                          notes = "HW with multip. seasonality"),
                tslm = list(method = "tslm",
                            method_arg = list(formula = input ~ trend + season),
                            notes = "tslm with trend and seasonal"))


train_method = list(partitions = 6,
                    sample.out = 12,
                    space = 3)

md <- train_model(input = ts.obj,
                  methods = methods,
                  train_method = train_method,
                  horizon = nrow(post_covid),
                  error = "MAPE")
## # A tibble: 7 x 7
##   model_id   model       notes                        avg_mape avg_rmse `avg_coverage_80%` `avg_coverage_95%`
##                                                                           
## 1 hw1        HoltWinters HoltWinters Model              0.0277  149046.              0.792              0.958
## 2 ets2       ets         ETS model opt.crit=amse        0.0284  161754.              0.792              0.972
## 3 hw2        HoltWinters HW with multip. seasonality    0.0300  171702.              0.528              0.847
## 4 ets1       ets         ETS model opt.crit=lik         0.0307  173337.              0.833              0.958
## 5 ets3       ets         ETS model opt.crit=mse         0.0311  174238.              0.861              0.972
## 6 auto_arima auto.arima  Auto ARIMA                     0.0334  184381.              0.597              0.889
## 7 tslm       tslm        tslm with trend and seasonal   0.0370  223194.              0.569              0.75

plot_error(md)

plot_model(md)

post_covid$yhat <- as.numeric(md$forecast$hw1$forecast$mean)
post_covid$upper95 <- as.numeric(md$forecast$hw1$forecast$upper[,2])
post_covid$lower95 <- as.numeric(md$forecast$hw1$forecast$lower[,2])

post_covid$passengers_loss <- post_covid$y - post_covid$yhat

post_covid
## # A tibble: 7 x 6
##   date             y     yhat  upper95  lower95 passengers_loss
##                                 
## 1 2020-03-01 1885466 4675574. 4860610. 4490537.       -2790108.
## 2 2020-04-01  138817 4756550. 4963034. 4550066.       -4617733.
## 3 2020-05-01  286570 5062504. 5288613. 4836395.       -4775934.
## 4 2020-06-01  555119 5476241. 5720592. 5231889.       -4921122.
## 5 2020-07-01  765274 5641839. 5903342. 5380336.       -4876565.
## 6 2020-08-01  852578 5650253. 5928018. 5372488.       -4797675.
## 7 2020-09-01  905992 4541560. 4834848. 4248272.       -3635568.

sum(post_covid$passengers_loss)
## [1] -30414705

plot_ly() %>%
  add_ribbons(x = post_covid$date,
              ymin = post_covid$y,
              ymax = post_covid$yhat,
              line = list(color = 'rgba(255, 0, 0, 0.05)'),
              fillcolor = 'rgba(255, 0, 0, 0.6)',
              name = "Estimated Loss") %>%
  add_segments(x = as.Date("2020-02-01"), 
               xend = as.Date("2020-02-01"),
               y = min(df$y),
               yend = max(df$y) * 1.05,
               line = list(color = "black", dash = "dash"),
               showlegend = FALSE) %>%
  add_annotations(text = "Pre-Covid19",
                  x = as.Date("2017-09-01"),
                  y = max(df$y) * 1.05, 
                  showarrow = FALSE) %>%
  add_annotations(text = "Post-Covid19",
                  x = as.Date("2020-09-01"),
                  y = max(df$y) * 1.05, 
                  showarrow = FALSE) %>%
  add_annotations(text = paste("Estimated decrease in", " ",
                               "passengers volume: ~30M",
                               sep = ""),
                  x = as.Date("2020-05-01"),
                  y = 2 * 10 ^ 6, 
                  arrowhead = 1,
                  ax = -130,
                  ay = -40,
                  showarrow = TRUE) %>%
  add_lines(x = df$date,
            y = df$y,
            line = list(color = "#1f77b4"),
            name = "Actual") %>%
  layout(title = "Covid19 Impact on SFO Air Passenger Traffic",
         yaxis = list(title = "Number of Passengers"),
         xaxis = list(title = "Time Series Model - Holt-Winters",
                      range = c(as.Date("2015-01-01"), as.Date("2021-01-01"))),
         legend = list(x = 0, y = 0.95))

The post Quantify the Covid19 Impact on the SFO Airport Passenger Air Traffic first appeared on R-bloggers.

Yes. Playing with numbers is another call for useless R function. This time, we will be using a function that will find the simplest mathematical expression for a whole number from 0 to 9 (or even higher), using only: – common mathematical operations (and symbols) – only four digits of four (hence name Four Fours) – no concatenations of the numbers (yet).

For the sake of brevity, I have only used couple of numbers, but list can go on and on. And also only four mathematical operations are used: – addition – subtraction – multiplication – division.

I next version we can also add: – exponentiation – factorial – root extraction.

To make calculations a little bit faster to compute, forcing order of operations by adding parentheses is also a helpful way to do it. Next version should have multiple-parentheses available.

Useless function:

# Four Fours functionfour_fours <- function(maxnum) {  for (i in 0:maxnum) {      oper <- c("+","*", "-", "/")      para <- c("(",")")      step_counter <- 0      res <- i + 1        while (i != res) {        oper3 <- sample(oper,4,replace=TRUE)        for44 <- paste0("4",oper3[1],"4",oper3[2],"4",oper3[3],"4")                #adding paranthesis        stopit <- FALSE        while (!stopit){          pos_par <<- sort(sample(1:7,2))           nn <- pos_par[1]          mm <- pos_par[2]          rr <<- abs(nn-mm)                    if (rr == 4 | rr == 5 ){            stopit <- TRUE                      }        }                for44 <- paste0(substr(for44, 1, nn-1), "(", substr(for44, nn, nchar(for44)), sep = "")        for44 <- paste0(substr(for44, 1, mm-1+1), ")", substr(for44, mm+1, nchar(for44)), sep = "")               # if (for44 ) like "(/" or "(-" or "(*" or "(+" -> switch to -> "/(" or "-("        for44 <- gsub("\\(-", "-\\(", for44)        for44 <- gsub("\\(/", "/\\(", for44)        for44 <- gsub("(*", "*(", for44, fixed=TRUE)        for44 <- gsub("(+", "+(", for44, fixed=TRUE)        for44 <- gsub("\\+)", "\\)+", for44)        for44 <- gsub("\\-)", "\\)-", for44)        for44 <- gsub("\\*)", "\\)*", for44)        for44 <- gsub("\\/)", "\\)/", for44)                ### Adding SQRT         if (i >= 10){          lii <- lapply(strsplit(as.character(for44), ""), function(x) which(x == "4"))          start_pos <- sample(lii[[1]],1)          for44 <- paste0(substr(for44, 1, start_pos-1), "sqrt(", substr(for44, start_pos, start_pos), ")", substr(for44, start_pos+1, nchar(for44)),sep = "")        }                ###  Adding Factorial        if (i >= 11){          li <- lapply(strsplit(as.character(for44), ""), function(x) which(x == "4"))          start_pos_2 <- sample(li[[1]],1)          for44 <- paste0(substr(for44, 1, start_pos_2-1), "factorial(", substr(for44, start_pos_2, start_pos_2), ")", substr(for44, start_pos_2+1, nchar(for44)),sep = "")        }                res <- eval(parse(text=for44))        step_counter <- step_counter + 1        if (res==i){          print(paste0("Value: ", res, " was found formula: ", for44, " with result: ", res, " and steps: ", step_counter, collapse=NULL))        }        }      i <- i + 1    }}

And to run the function, simple as:

#Run functionfour_fours(6)

After the finish run, results are displayed with number of tries and the formula.

There are some solutions that do not need factorial or root extraction and some, that do. Therefore from step 10 till 14, there are some need for either and from 15 till 18, simple operations are enough, and so on. At some integer, also concatenation of two fours would be required.

The above solution should easily generate solutions from 1 to 25, based on my tests. There are also many optimisations possible; one useless would be using cloud scalable architecture – most useless (and expensive, though) optimisation – which I certainly do not approve of.

As always, code is available at Github.

Happy R-coding!

The post Little useless-useful R functions – Mathematical puzzle of Four fours first appeared on R-bloggers.

Introduction

Here is a quick data-scientist / data-analyst question: what is the overall trend or shape in the following noisy data? For our specific example: How do we relate value as a noisy function (or relation) of m? This example arose in producing our tutorial “The Nature of Overfitting”.

One would think this would be safe and easy to asses in R using ggplot2::geom_smooth(), but now we are not so sure.

Our Example

Let’s first load our data and characterize it a bit

d <- read.csv(  'sus_shape.csv',   strip.white = TRUE,   stringsAsFactors = FALSE)head(d)

##   m      value## 1 3 -12.968296## 2 3  -5.522812## 3 3  -6.893872## 4 3  -5.522812## 5 3 -11.338718## 6 3 -10.208145

summary(d)

##        m              value        ##  Min.   :   3.0   Min.   :-18.773  ##  1st Qu.:  86.0   1st Qu.: -1.304  ##  Median : 195.0   Median : -1.276  ##  Mean   : 288.8   Mean   : -1.508  ##  3rd Qu.: 436.0   3rd Qu.: -1.266  ##  Max.   :1000.0   Max.   : -1.260

nrow(d)

## [1] 15545

Now let’s try and look at this data. First we try a scatter plot with a low alpha, which gives us something similar to a density presentation.

library(ggplot2)ggplot(  data = d,  mapping = aes(x = m, y = value)) +   geom_point(    alpha = 0.005,     color = 'Blue') +   ggtitle("point plot of data")

Each m value has many different value measurements (representing repetitions of a noisy experiment). Frankly the above is not that legible, so we need tools to try and summarize it in the region we are interested in (value near -1.25).

Trying Default Smoothing

Let’s run a default smoothing line through this data to try to get the overall relation.

ggplot(  data = d,  mapping = aes(x = m, y = value)) +   geom_smooth() +  ggtitle("suspect shape in smoothing (default)")

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

This graph appears to imply some sort of oscillation or structure in the relation between mean value and m. We are pretty sure there is no such structure, and this is an artifact of the smoothing method. This defect is why we did not use ggplot2::geom_smooth() in our note on training set size.

We did see a warning, but we believe this is just telling us which default values were used, and not indicating the above pathology was detected.

At this point we are in a pickle. We had theoretical reasons to believe the data is a monotone increasing in m trend, with mean-zero noise that decreases with larger m. The graph doesn’t look like that. So our understanding or theory could be wrong, or the graph didn’t faithfully represent the data. The graph had been intended as a very small step in larger work. Re-examining the intricacies of what is the default behavior of this graphing software was not our intended task. We had been doing some actual research on the data.

Now have a second problem: is this unexpected structure in our data, or a graphing artifact? The point is: when something appears to work one can, with some risk, move on quickly; when something appears to not work in a surprising way, you end up with a lot of additional required investigation. This investigation is the content of this note, like it or not. Also in some loud R circles, one has no choice but to try “the default ggplot2::geom_smooth() graph”, otherwise one is pilloried for “not knowing it.”

We can try switching the smoothing method to see what another smoothing method says. Let’s try loess.

ggplot(  data = d,  mapping = aes(x = m, y = value)) +   geom_smooth(method = 'loess') +  ggtitle("suspect shape in smoothing (loess)")

## `geom_smooth()` using formula 'y ~ x'

Now we have a different shape. At most one of these (and in fact neither) is representative of the data. There is, again, a warning. It appears, again, to be a coding style guide- and not detection of the issue at hand.

Looking Again

Let’s try a simple grouped box plot. We will group m into ranges to get more aggregation.

d$m_grouped <- formatC(  round(d$m/50)*50,   width = 4,   format = "d",   flag = "0")ggplot(  data = d,  mapping = aes(x = m_grouped, y = value)) +   geom_boxplot() +  theme(axis.text.x = element_text(angle = 90,                                    vjust = 0.5,                                    hjust=1)) +  ggtitle("m-grouped bar chart, no obvious plotting artifacts.")

For legibility, we repeat these graphs zooming in to the area under disagreement. We are using coord_cartesian() to zoom in, so as to try and not change the underlying graphing calculation.

zoom <- coord_cartesian(xlim = c(0, 500), ylim = c(-1.5, -1)) 

ggplot(  data = d,  mapping = aes(x = m, y = value)) +   geom_smooth() +  zoom +  ggtitle("suspect shape in smoothing (default, zoomed)") +   geom_hline(    yintercept = max(d$value),     color = 'red',     linetype = 2)

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

This crossing above -1.0 is very suspicious, as we have max(d$value) = -1.2600449. We have annotated this with the horizontal red dashed line.

And the entirety of the loess hump is also a plotting artifact, also completely out of the observed data range.

ggplot(  data = d,  mapping = aes(x = m, y = value)) +   geom_smooth(method = 'loess') +  zoom +  ggtitle("suspect shape in smoothing (loess, zoomed)") +   geom_hline(    yintercept = max(d$value),     color = 'red',     linetype = 2)

## `geom_smooth()` using formula 'y ~ x'

The zoomed-in version of the box plot shows the noisy monotone asymptotic shape we expected for the original experiment that produced this data.

ggplot(  data = d[d$m <= 500, ],  mapping = aes(x = m_grouped, y = value)) +   geom_boxplot() +  coord_cartesian(ylim = c(-1.5, -1.2)) +  theme(    axis.text.x = element_text(      angle = 90,       vjust = 0.5,       hjust=1)) +  ggtitle("m-grouped bar chart, no obvious plotting artifacts, zoomed")

The point plot, when zoomed, qualitatively agrees with the boxplot.

ggplot(  data = d,  mapping = aes(x = m, y = value)) +   geom_point(alpha = 0.05, color = 'Blue') +   coord_cartesian(    xlim = c(0, 500),     ylim = c(-1.5, -1.25))  +  ggtitle("point plot of data, zoomed")

Directly calling loess/lowess

ggplot2 is documented as using loess, which in turn is documented as a newer adapter for lowess“with different defaults” then loess. However, the documented exposed controls on these two methods seem fairly disjoint.

That being said loess (without a ‘w’, as in “Uruguay”) called directly with default arguments shows the same chimeric artifact.

zoom2 <- coord_cartesian(ylim = c(-1.5, -1)) 

d$loess <- loess(value ~ m, data = d)$fittedggplot(  data = d,  mapping = aes(x = m)) +   geom_line(aes(y = loess)) +   geom_point(    aes(y = value),     alpha = 0.01,     color = 'Blue') +   zoom2 +   geom_hline(    yintercept = max(d$value),     color = 'red',     linetype = 2) +  ggtitle('direct loess (no w) call')

Playing with arguments can suppress the artifact, but we still saw weird (but smaller) effects even with the suggested degree = 1 alternate setting.

Directly calling lowess (with a ‘w’, as in “answer”) gives a more reasonable result out of the box.

d$lowess <- lowess(d$m, d$value)$yggplot(  data = d,  mapping = aes(x = m)) +   geom_line(aes(y = lowess)) +   geom_point(    aes(y = value),     alpha = 0.01,     color = 'Blue') +   geom_hline(    yintercept = max(d$value),     color = 'red',     linetype = 2) +  coord_cartesian(    ylim = c(-1.5, -1.25)) +   ggtitle('direct lowess (with w) call')

Simple Windowing

Simple methods from fields such as signal processing work well. For example, a simple square-window moving average appears to correctly tell the story. These are the methods I use, at the risk of being told I should have used geom_smooth().

# requires development version 1.3.2# remotes::install_github('WinVector/WVPlots')library(WVPlots)  

## Loading required package: wrapr

ConditionalSmoothedScatterPlot(  d,  xvar = 'm',   yvar = 'value',   point_color = "Blue",  point_alpha = 0.01,  k = 51,  groupvar = NULL,   title = 'Width 51 square window on data (zoomed)') +  coord_cartesian(ylim = c(-1.5, -1.25)) +   geom_hline(    yintercept = max(d$value),     color = 'red',     linetype = 2)

The fact that the hard window yields a jagged curve gives an indication of the amount of noise in each region of the graph.

Conclusion

Large data sets are inherently illegible. So we rely on summaries and aggregations to examine them. When these fail we may not always be in a position to notice the distortion, and this can lead to problems.

Many of the above default summary presentations were deeply flawed and showed chimerical artifacts not in the data being summarized. Starting a research project to understand the nature of the above humps and oscillations would be fruitless, as they are not in the data, but instead artifacts of the plotting and analysis software.

As a consultant this is disturbing: I end up spending time on debugging the tools, and not on the client’s task.

The above were not flaws in ggplot2 itself, but in the use of the gam and loess smoothers, which are likely introducing the artifacts by trying to enforce certain curvature conditions not in the data. We are essentially looking at something akin to Gibbs’ phenomenon or ringing. This could trip up the data scientist or the data analyst without a background in signal analysis.

This sort of problem reveals the lie in the typical “data scientist >> statistician >> data analyst” or “statistics are always correct in R, and never correct in Python” snobberies. In fact a data analyst would get the summary shapes right, as presentation of this sort is one of their specialties.

The post Smoothing isn’t Always Safe first appeared on R-bloggers.

On December 11, RStudio launched our third annual R Community Survey (formerly known as the Learning R Survey) to better understand how and why people learn and use the R language and associated tools. That survey closes TOMORROW, January 8, 2021. We encourage anyone who is interested in R to respond. The survey should only require 5 to 10 minutes to complete, depending on how little or how much information you choose to share with us. You can find the survey here:

• English version: https://rstd.io/r-survey-en
• Spanish version: https://rstd.io/r-survey-es

If you don’t know R yet or use Python or Julia more than R, that’s fine too! The survey has specific questions for you, and your responses will help us better understand how we can be more encouraging to you and others like you.

Data and analysis of the 2018 and 2019 community survey data can be found on github at https://github.com/rstudio/r-community-survey in the 2018/ and 2019/ folders. Results from the 2020 survey will also be posted as free and open source data that github repo in February 2021.

Please ask your students, Twitter followers, Ultimate Frisbee team, and anyone else you think may be interested to complete the survey. Your efforts will help RStudio, educators, and users understand and grow our data science community.

You will find a full disclosure of what information will be collected and how it will be used on the first page of the survey. The survey does not collect personally identifiable information nor email addresses, but it does have optional demographic questions.

Thank you in advance for your consideration and time. We look forward to sharing the results with you next month!

The post Last Call for the 2020 R Community Survey first appeared on R-bloggers.

Introduction

In this article, we’ll use the EPA Arcgis map that contains weather data going back to 1990. This will be a quick code-based blog post showing how to load, explore and visualize the data a local EPA monitor.

# Librariespackages <-   c("data.table",    "ggplot2",    "stringr",    "skimr",    "janitor",    "glue"    )if (length(setdiff(packages,rownames(installed.packages()))) > 0) {  install.packages(setdiff(packages, rownames(installed.packages())))  }invisible(lapply(packages, library, character.only = TRUE))knitr::opts_chunk$set(  comment = NA,  fig.width = 12,  fig.height = 8,  out.width = '100%',  cache = TRUE)

Weather Monitor Data Extraction

Below, we show how to use glue::glue() to hit the api for our local site (“09-001-0017”) for the annual daily data sets from 1990-2020 with datatable::fread(), which took only a few minutes. See how we create an integer vector of desired years, and glue the string into each iteration of the request to create a list. We could easily change the code above and get the same for the next closest monitor in Westport, CT, or from any group of monitors in Connecticut or beyond if needed. For the actual blog post, the data we extracted below and saved to disc will be used.

# Years to retrieveyear <- c(1990:2020)ac <-   lapply(    glue::glue(      'https://www3.epa.gov/cgi-bin/broker?_service=data&_program=dataprog.Daily.sas&check=void&polname=Ozone&debug=0&year={year}&site=09-001-0017'    ),    fread  )

Fortunately, the annual data sets were consistent over the period with all the same variables, so it took just a few minutes to get a clean data.table stretching back to 1990. In our experience with public data sets, this is almost always not the case. Things like variables or formatting almost always change. It seems surprising that the collected data would be exactly the same for such a long period of time, so we assume that the EPA is making an effort to clean it up and keep it consistent, which is very much appreciated. In a future post, we might look at a more complicated exploration of the EPA API, which has data going back much further for some monitors and some variables, and seems to be one of the better organized and documented government API’s we have come across.

# Bind lists to data.tableac_dt <- rbindlist(ac)# Clean namesac_dt <- janitor::clean_names(ac_dt)

Exploration and Preparation

We will start off with the full 34 columns, but throw out the identifier rows where there is only one unique value. We can see that there are only 10,416 unique dates though there are 55,224 rows, so the many fields are layered in the data set. Four of the logical columns are all missing, so they will have to go. There are 14 unique values in the parameter_name field, so we will have to explore those. A majority of the pollutant standard rows are missing. We can also see “aqi”(Air Quality Index) which we consider to be a parameter is included in a separate column as a character. The two main measurements for all the other parameters are the “arithmetic_mean” and “first_maximum_value”. There are a couple of time-related variables including year, day_in_year and date_local. There are a lot of fields with only one unique value to identify the monitor, so these can all be dropped. Its pretty messy, so he best thing we can think of doing is to tidy up the data set so it is easier to work with.

      # Summarize data.table    skimzr::skim(ac_dt)  

Variable type: character

Variable type: Date

Variable type: logical

Variable type: numeric

First we will drop all of the identifier rows with only one unique value (before column 9 and after column 27), and also the “tribe” and “nonreg” columns using data.table::patterns(). We then convert the air quality index (“aqi”) column to numeric for the cases where it is not missing. We are not clear why the “aqi” is not included in the “parameter_name” variable with the other measures, but seems to be associated with rows which have “ozone” and “sulfur dioxide” (two of five variables which compose the “aqi” itself). Air Quality is also stored in by the 1-hour average and separately a single 8-hour measurements for each day, and these numbers can be significantly different.

# Drop unneeded colsac_dt <- ac_dt[, c(9:27)][, .SD, .SDcols = !patterns("tribe|nonreg")]# Convert aqi to integerac_dt[, aqi := as.integer(str_extract(aqi, "\\d*"))]

We add the three measurement columns to value and 12 identifier columns to variable. We decided to separate the “aqi” index column from the rest of the data which is identified in the “parameter_name” column before tidying, and then bind them back together with three variables (“aqi”, “arithmetic_mean” and “first_maximum_value”).

# Separate out aqiaqi <- ac_dt[!is.na(aqi)]# Tidy key measures for parameters other than aqimeasures <- c("first_maximum_value", "arithmetic_mean")ids <- setdiff(names(ac_dt), measures)ids <- ids[!str_detect(ids, "aqi")]ac_dt_tidy <-  ac_dt[,         melt(.SD,             idcols = ids,             measure.vars = measures),        .SDcols = !"aqi"]# Tidy up aqiaqi <-   aqi[,       melt(.SD,           idcols = ids,           measure.vars = "aqi"),      .SDcols = !measures]# Put two tidied data sets back togetherac_dt_tidy <- rbind(ac_dt_tidy, aqi)# Show sample rowsac_dt_tidy             parameter_name    duration_description pollutant_standard     1: Outdoor Temperature                  1 HOUR                        2:      Sulfur dioxide                  1 HOUR    SO2 1-hour 2010     3:      Sulfur dioxide            3-HR BLK AVG    SO2 3-hour 1971     4:      Sulfur dioxide            3-HR BLK AVG    SO2 3-hour 1971     5: Outdoor Temperature                  1 HOUR                       ---                                                               120581:               Ozone 8-HR RUN AVG BEGIN HOUR  Ozone 8-hour 2015120582:               Ozone 8-HR RUN AVG BEGIN HOUR  Ozone 8-hour 2015120583:               Ozone 8-HR RUN AVG BEGIN HOUR  Ozone 8-hour 2015120584:               Ozone 8-HR RUN AVG BEGIN HOUR  Ozone 8-hour 2015120585:               Ozone 8-HR RUN AVG BEGIN HOUR  Ozone 8-hour 2015        date_local year day_in_year_local   units_of_measure     1: 1990-01-01 1990                 1 Degrees Fahrenheit     2: 1990-01-01 1990                 1  Parts per billion     3: 1990-01-01 1990                 1  Parts per billion     4: 1990-01-02 1990                 2  Parts per billion     5: 1990-01-02 1990                 2 Degrees Fahrenheit    ---                                                     120581: 2020-03-27 2020                87  Parts per million120582: 2020-03-28 2020                88  Parts per million120583: 2020-03-29 2020                89  Parts per million120584: 2020-03-30 2020                90  Parts per million120585: 2020-03-31 2020                91  Parts per million        exceptional_data_type observation_count observation_percent     1:                  None                24                 100     2:                  None                23                  96     3:                  None                 7                  88     4:                  None                 7                  88     5:                  None                24                 100    ---                                                            120581:                  None                17                 100120582:                  None                17                 100120583:                  None                17                 100120584:                  None                17                 100120585:                  None                12                  71        first_maximum_hour daily_criteria_indicator            variable value     1:                  0                        Y first_maximum_value  43.0     2:                  8                        Y first_maximum_value  11.0     3:                  8                        Y first_maximum_value   9.3     4:                 23                        Y first_maximum_value  17.6     5:                 14                        Y first_maximum_value  42.0    ---                                                                      120581:                 10                        Y                 aqi  48.0120582:                 22                        Y                 aqi  46.0120583:                  8                        Y                 aqi  46.0120584:                  7                        Y                 aqi  37.0120585:                 16                        N                 aqi  42.0

When we graph with “parameter_name” facets including separate colors for the mean and maximum values, we can see a few things. There are a few gaps in collection including a big one in sulfur dioxide from about 1997-2005. The Air Quality Index first created in the Clean Air Act has the following components: ground-level ozone, particulate matter, carbon monoxide, sulfur dioxide, and nitrogen dioxide. We are unsure how they calculate the AQI in our data set for the full period, because of the period where sulfur dioxide is missing. When we read up on AQI, we learned that there may be several ways of calculating the AQI. We will leave the details for later research.

# Look for missing periodsac_dt_tidy[  variable %in% c("arithmetic_mean", "first_maximum_value"),   ggplot(.SD,         aes(date_local,             y = value,             color = variable)) +    geom_line() +    facet_wrap( ~ parameter_name, scale = 'free') +    theme_bw() +    labs(caption = "Source: EPA Monitor 09-001-0017"    )]

Wind is Lowest during the Summer

We had hoped to look at the wind speeds during Hurricane Sandy, which hit us hard, but apparently, the monitor was knocked out, so there are no measurements for that date or for several months subsequent, so it looks like we may not do a lot with the wind data. It is hard to find much in the charts above, so we averaged up the values by month. We might have guessed, but hadn’t thought that wind was as seasonal as it seems to be below.

ac_dt_tidy[  str_detect(parameter_name, "Wind") &    variable %in% c("first_maximum_value", "arithmetic_mean"),  .(avg_speed = mean(value)), by = .(month(date_local), parameter_name, variable)][,  ggplot(.SD, aes(month, avg_speed, color = variable)) +    geom_line() +    facet_wrap( ~ parameter_name, scales = "free_y") +    theme_bw() +     labs(      x = "Month",      y = "Wind Speed",      caption = "Source: EPA Monitor 09-001-0017"    ) ]

Air Quality Has Improved

The data set actually records 4 measurements for ozone, the average and maximum values by hour, and separately, for 8-hour periods. The EPA sets a threshold for the level of Ozone to be avoided at 0.064, and days above this are shown in red. It looks like the “first_maximum_value” very often registers undesirable levels, although the hourly reading does much less so. We can see that there are clearly fewer unhealthy days over time, and only two unhealthy days based on the hourly arithmetic average since 2003. We can also see that the low end of the readings has been moving up over time, even though well in the healthy zone.

ac_dt_tidy[  parameter_name == "Ozone" &    variable %in% c("arithmetic_mean", "first_maximum_value")][  ][,    ggplot(.SD,           aes(date_local,               y = value)) +      geom_point(aes(color = cut(        value,        breaks = c(0, 0.064, 0.3),        labels = c("Good", "Unhealthy")      )),      size = 1) +      scale_color_manual(        name = "Ozone",        values = c("Good" = "green1",                   "Unhealthy" = "red1")) +      theme_bw() +      labs(x = "Year",           y = "Ozone",           caption = "Source: EPA Monitor 09-001-0017") +      facet_wrap(~ variable + duration_description, scales = "free_y")]

We can see in the chart looking at Air Quality based on the “Ozone 8-hour 2015” parameter below, that if the EPA calculates it, it doesn’t report AQI during the winter months, which probably makes sense because people are not out and air quality appears to be worst in the summer. Sometimes we get the iPhone messages about Air Quality and naturally worry, but when we look at the AQI daily over the last 30 years, we can see that the number of “Unhealthy” days has been declining similar two what we saw above with Ozone, and the last “Very Unhealthy” day was in 2006. The same trend with the low end of the AQI rising a little over time is apparent.

ac_dt_tidy[  variable == "aqi" &    pollutant_standard == "Ozone 8-hour 2015"][  ][,     ggplot(.SD,            aes(date_local,                y = value)) +       geom_point(aes(color = cut(         value,         breaks = c(0, 50, 100, 150, 200, 300),         labels = c(           "Good",           "Moderate",           "Unhealthy - Sensitive",           "Unhealthy",           "Very Unhealthy"         )       )),       size = 1) +       scale_color_manual(         name = "AQI",         values = c(           "Good" = "green1",           "Moderate" = "yellow",           "Unhealthy - Sensitive" = "orange",           "Unhealthy" = "red",           "Very Unhealthy" = "violetred4"         )       ) +       theme_bw() +       labs(x = "Year",            y = "Air Quality Indicator (AQI)",            caption = "Source: EPA Monitor 09-001-0017"            )]

A heatmap is another way to look at the Ozone which better shows the time dimension. The y-axis shows the day of the year, so the most unhealthy air quality is between days 175-225, or the end of June through the first half of August. We can also see that “Unhealthy” days might even have outnumbered healthy days back in the early 1990s, but we rarely see above “moderate” now.

breaks <- c(0, 50, 100, 150,200, 300, 1000)labels <- c("Good", "Moderate", "Unhealty - Sensitive Groups", "Unhealthy", "Very Unhealthy", "Hazardous")ac_dt[parameter_name == "Ozone" &        exceptional_data_type == "None", .(          year,          day_in_year_local,          observation_count,          duration_description,          date_local,          aqi= as.integer(str_extract(aqi, "\\d*")),          parameter_name,          `Air Quality` = cut(            as.integer(str_extract(aqi, "\\d*")),            breaks = breaks,            labels = labels          )        )][!is.na(`Air Quality`) &              day_in_year_local %in% c(90:260),           ggplot(.SD, aes(year, day_in_year_local, fill = `Air Quality`)) +              geom_tile() +             theme_bw() +             labs(               x = "Year",               y = "Day of Year",                caption = "Source: EPA Monitor 09-001-0017"              )]

Temperature

We can see above the dot plot of the Outside Temperature over the period. Hot days are defined as above 85, and very hot above 95, while cold are below 32. There isn’t much of a trend visible in the middle of the graphs. As might be expected the daily first maximum highs and lows tend to be significantly above the daily average levels. All in all, if there is change, it is less definitive than the air quality data looking at it this way.

ac_dt_tidy[  parameter_name == "Outdoor Temperature" &    variable %in% c("arithmetic_mean", "first_maximum_value")][  ][,  ggplot(.SD,         aes(date_local,             y = value)) +    geom_point(aes(      color = cut(value,                   breaks = c(-20, 32, 50, 65, 85, 95, 120),                  labels = c("Very Cold", "Cold", "Cool", "Moderate", "Hot", "Very Hot"))),             size = 1) +    scale_color_manual(        name = "Outside Temperature",        values = c(          "Very Cold" = "blue",          "Cold" = "yellow",          "Cool" = "green1",          "Moderate" = "green4",          "Hot" = "orange",          "Very Hot" = "red"          )      ) +    theme_bw() +     labs(      x = "Year",      y = "Outside Temperature",      caption = "Source: EPA Monitor 09-001-0017"    ) +    facet_wrap(~ variable)]

We also tried to look at the change in temperature over the period versus the first five years (1990-1995). By doing this, we probably learned more about heat maps than about the temperature. It does look like the bigger changes in temperature have probably happened more at the beginning and the end of the year. Movements in the maximum temperatures seem more pronounced than the averages, but again it makes sense that this would be the case.

temperature <-  ac_dt[parameter_name == "Outdoor Temperature",        c("year",          "day_in_year_local",          "arithmetic_mean",          "first_maximum_value")]baseline <-   temperature[year < 1995,              .(base_mean = mean(arithmetic_mean),                base_max = mean(first_maximum_value)), day_in_year_local]temperature <-   baseline[temperature[year > 1994], on = c("day_in_year_local")][,     `:=`(change_avg = arithmetic_mean - base_mean,         change_max = first_maximum_value - base_max)]temperature <-  temperature[, melt(    .SD,    id.vars = c("day_in_year_local", "year"),    measure.vars = c("change_max", "change_avg")  )]temperature[  year %in% c(1995:2019) &    !is.na(value),   ggplot(.SD,         aes(year,             day_in_year_local,             fill = cut(               value,               breaks = c(-100, -15, -5, 5, 15, 100),               labels = c("Much Colder", "Colder", "Similar", "Warmer", "Much Warmer")             ))) +    geom_tile() +    scale_fill_manual(name = "Temp. Change",                      values = c("skyblue4", "skyblue", "green", "red", "red4")) +    theme_bw() +    labs(      title = "Days Compared to 1990-1994 Average Temp. on That Day",      subtitle = "Hotter Days Shown Redder",      x = "Year",      y = "Day of Year",      caption = "Source: EPA"    ) +    facet_wrap(~ variable)]

Thoughts on Heatmaps

The interesting thing we learned about heat maps is how much we could control the perception of the chart based on our decisions about the size of the groupings and the color choices. Dark colors on the days with the biggest temperature increases could flood the chart with red. If we chose equal equal sized groups for the cut-offs, there would be a lot more days for the average which were colder (as shown below), but a lot more for the max which were hotter. It made us more wary of heat maps.

# Uneven hand selected cutoffs to find more balanced countslapply(list(cut(temperature[variable == "change_avg" &                    !is.na(value)]$value, c(-100,-15,-5, 5, 15, 100)), cut(temperature[variable == "change_max" &                    !is.na(value)]$value, c(-100,-15,-5, 5, 15, 100))), summary)[[1]](-100,-15]   (-15,-5]     (-5,5]     (5,15]   (15,100]        169       1489       4929       1786         65 [[2]](-100,-15]   (-15,-5]     (-5,5]     (5,15]   (15,100]        286       2016       4155       1821        160 # Even range limits with less even countslapply(list(cut(temperature[variable == "change_avg" &                    !is.na(value)]$value, 5),cut(temperature[variable == "change_max" &                    !is.na(value)]$value, 5)), summary)[[1]]    (-38,-25]   (-25,-12.1] (-12.1,0.827]  (0.827,13.8]   (13.8,26.8]             5           305          4253          3767           108 [[2]](-28.8,-15.8] (-15.8,-2.95]  (-2.95,9.95]   (9.95,22.9]   (22.9,35.8]           215          2858          4659           686            20 

Conclusion

That wraps up this quick exploration of an EPA monitor in Connecticut. We still have many questions about air quality, temperature and wind speed. We wonder why the EPA chose to put the monitor down by the edge of the water away from the heavy traffic of I-95 and Route 1 and the bulk of the population. We didn’t have a lot of time to spend, and acknowledge that we may have misread or misinterpreted some of the data, but now we at least know what to look for. The purpose of this blog is to explore, learn and get better, faster and more accurate in data analysis. If you are interested in learning R and would like to see what it takes to learn R, we invite you to download our free data science cheat sheet and join our free R webinar on January 20, 2021.

Author: David Lucy, Founder of Redwall Analytics David spent 25 years working with institutional global equity research with several top investment banking firms.

The post 30 Year Weather Data Analysis first appeared on R-bloggers.

library(tidyverse)library(nflfastR)library(scales)#Get Season 2020 Schedule and resultsnfl_games <- fast_scraper_schedules(2020) %>%   #Weeks Beyond Week 17 Are the Playoffs  filter(week <= 17)knitr::kable(head(nfl_games, 3))

Method 1: Pythagorean expectation

Pythagorean expectation was developed by Bill James for Baseball and estimates the % of games that a team “should win” based on runs scored and runs allowed.

It was adapted for Pro Football by Football Outsiders to use the following formula:

Football Outside Almanac in 2011 stated that “From 1988 through 2004, 11 of 16 Super Bowls were won by the team that led the NFL in Pythagorean wins, while only seven were won by the team with the most actual victories”

There needs to be a little data manipulation to get the NFL schedule data into a format to calculate the pythagorean expectation. Most notably splitting each game into two rows of data to capture information on both the home team and away teams.

p_wins <- nfl_games %>%   pivot_longer(    cols = c(contains('team')),    names_to = "category",    values_to = 'team'  ) %>%   mutate(points_for = (category=='home_team')*home_score+           (category=='away_team')*away_score,         points_against = (category=='away_team')*home_score+           (category=='home_team')*away_score  ) %>%   group_by(team) %>%  summarize(pf = sum(points_for, na.rm = T),            pa = sum(points_against, na.rm = T),            actual_wins = sum(points_for > points_against, na.rm = T),            .groups = 'drop'  ) %>%   mutate(p_expectation = pf^2.37/(pf^2.37+pa^2.37)*16)

By pythagorean expectation the top 3 teams in the NFL are:

team	points_for	points_against	actual_wins	expected_wins
BAL	468	303	11	11.8
NO	482	337	12	11.2
TB	492	355	11	10.9

According to Pythagorean Expectation, the Baltimore Ravens are the best team in the NFL while the formula would say that the Kansas City Chiefs, the team with the most actual wins, “should” have only had 10.5 wins vs. the 14 actual wins they had.

An aside: Who “outkicked their coverage”?

The concept of “Expected Wins” allows us to see who outperformed their expectation vs. under-performed. The following plot shows actual wins on the x-axis and expected wins on the y-axis.

library(ggrepel)p_wins %>%   mutate(diff_from_exp = actual_wins - p_expectation) %>%   ggplot(aes(x = actual_wins, y = p_expectation, fill = diff_from_exp)) +     geom_label_repel(aes(label = team)) +     geom_abline(lty = 2) +     annotate("label", x = 1, y = 10, hjust = 'left', label = "Underachievers") +    annotate("label", x = 10, y = 5, hjust = 'left', label = "Overachievers") +    labs(x = "Actual Wins", y = "Expected Wins",          title = "What NFL Teams Over/Under Performed?",          caption = "Expected Wins Based on Pythagorian Expectation") +     scale_fill_gradient2(guide = F) +     cowplot::theme_cowplot()

The largest over-achievers appear to be Kansas city, and Cleveland while the largest under-achievers were Atlanta and Jacksonville.

BTm(cbind(win1, win2), team1, team2, ~ team, id = "team", data = sports.data)

#Get List of All Teamsall_teams <- sort(unique(nfl_games$home_team))nfl_shaped <- nfl_games %>%  mutate(    home_team = factor(home_team, levels = all_teams),    away_team = factor(away_team, levels = all_teams),    home_wins = if_else(home_score > away_score, 1, 0),    away_wins = if_else(home_score < away_score, 1, 0)   ) %>%   group_by(home_team, away_team) %>%   summarize(home_wins = sum(home_wins),            away_wins = sum(away_wins),            .groups= 'drop') knitr::kable(head(nfl_shaped, 3), align = 'c')

library(BradleyTerry2)base_model <- BTm(cbind(home_wins, away_wins), home_team, away_team,                  data = nfl_shaped, id = "team")

base_abilities <- qvcalc(BTabilities(base_model)) %>%   .[["qvframe"]] %>%   as_tibble(rownames = 'team') %>%   janitor::clean_names()knitr::kable(base_abilities %>%                mutate(across(where(is.numeric), round, 2)) %>%                head(3),             align = 'c')

playoff_teams = c('BAL', 'BUF', 'CHI', 'CLE', 'GB', 'IND', 'KC', 'LA', 'NO',                  'PIT', 'SEA', 'TB', 'TEN', 'WAS')comparisons <- base_abilities %>%   filter(team %in% playoff_teams)#Generate All Potential Combination of Playoff Teamscomparisons <- comparisons %>%   rename_with(~paste0("t1_", .x)) %>%   crossing(comparisons %>% rename_with(~paste0("t2_", .x)))  %>%   filter(t1_team != t2_team)#Run 1000 Simulations per comparisonset.seed(20210107)#Draw from Ability Distributionsimulations <- comparisons %>%   crossing(simulation = 1:1000) %>%   mutate(    t1_val = rnorm(n(), t1_estimate, t1_quasi_se),    t2_val = rnorm(n(), t2_estimate, t2_quasi_se),    t1_win = t1_val > t2_val,    t2_win = t2_val > t1_val  )#Roll up the 1000 Resultssim_summary <- simulations %>%   group_by(t1_team, t2_team, t1_estimate, t2_estimate) %>%   summarize(t1_wins_pct = mean(t1_win), #Long-Term Average Winning % for Team 1            t2_wins_pct = mean(t2_win), #Long-Term Average Winning % for Team 2            .groups = 'drop') %>%   mutate(    #Create a label for the winner    winner = if_else(t1_wins_pct > t2_wins_pct, t1_team, t2_team)  )

winners <- function(t1, t2){  dt = sim_summary %>% filter(t1_team == t1 & t2_team == t2) %>%     inner_join(      nflfastR::teams_colors_logos %>%         filter(team_abbr == t1) %>%         select(t1_team = team_abbr, t1_name = team_name),      by = "t1_team"    ) %>%     inner_join(      nflfastR::teams_colors_logos %>%         filter(team_abbr == t2) %>%         select(t2_team = team_abbr, t2_name = team_name),      by = "t2_team"    )    return(     list(       team1 = dt$t1_name,       team1_prob = dt$t1_wins_pct,       team2 = dt$t2_name,       team2_prob = dt$t2_wins_pct,       winner = if_else(dt$winner == dt$t1_team, dt$t1_name, dt$t2_name)     )  )}

The post Predicting the Winner of Super Bowl LV first appeared on R-bloggers.

Three months ago we finished Why R? 2020 conference. One of the memorable moments of the conference were invited highlighted talks! Today we would like to remind you about the talk by Daniel Aleman (from Global BPM at Archroma). The video from the recording is at the end of the post.

Usually Data Scientist focus on: Which language? Package? Type of problem to solve? Technique to apply? What metrics to validate the model etc.. but my own experience and studies have shown that people(the final users) are distrustful of automated model for forecasting. In this presentation, I will not only share my experience using different techniques to have the best forecast analysis, but will disclose the journey to gain the trust of business over the project.

The post Daniel Aleman - The Key Metric for your Forecast is... TRUST first appeared on R-bloggers.

library(sf)library(dplyr)library(ggplot2) 

library(sf)footway <- st_sfc(st_linestring(rbind(c(-50,0),c(150,0))))st_crs(footway) = 3035 viewpoints <- st_line_sample(footway, density = 1/50)viewpoints <- st_cast(viewpoints,"POINT")buildings <- rbind(c(1,7,1),c(1,31,1),c(23,31,1),c(23,7,1),c(1,7,1),                   c(2,-24,2),c(2,-10,2),c(14,-10,2),c(14,-24,2),c(2,-24,2),                   c(21,-18,3),c(21,-10,3),c(29,-10,3),c(29,-18,3),c(21,-18,3),                   c(27,7,4),c(27,17,4),c(36,17,4),c(36,7,4),c(27,7,4),                   c(18,44,5), c(18,60,5),c(35,60,5),c(35,44,5),c(18,44,5),                   c(49,-32,6),c(49,-20,6),c(62,-20,6),c(62,-32,6),c(49,-32,6),                   c(34,-32,7),c(34,-10,7),c(46,-10,7),c(46,-32,7),c(34,-32,7),                   c(63,9,8),c(63,40,8),c(91,40,8),c(91,9,8),c(63,9,8),                   c(133,-71,9),c(133,-45,9),c(156,-45,9),c(156,-71,9),c(133,-71,9),                   c(152,10,10),c(152,22,10),c(164,22,10),c(164,10,10),c(152,10,10),                   c(44,8,11),c(44,24,11),c(59,24,11),c(59,8,11),c(44,8,11),                   c(3,-56,12),c(3,-35,12),c(27,-35,12),c(27,-56,12),c(3,-56,12),                   c(117,11,13),c(117,35,13),c(123,35,13),c(123,11,13),c(117,11,13),                   c(66,50,14),c(66,55,14),c(86,55,14),c(86,50,14),c(66,50,14),                   c(67,-27,15),c(67,-11,15),c(91,-11,15),c(91,-27,15),c(67,-27,15))buildings <- lapply( split( buildings[,1:2], buildings[,3] ), matrix, ncol=2)buildings   <- lapply(X = 1:length(buildings), FUN = function(x) {  st_polygon(buildings[x])})buildings <- st_sfc(buildings)st_crs(buildings) = 3035 # plot raw dataggplot() +  geom_sf(data = buildings,colour = "transparent",aes(fill = 'Building')) +  geom_sf(data = footway, aes(color = 'Footway')) +  geom_sf(data = viewpoints, aes(color = 'Viewpoint')) +  scale_fill_manual(values = c("Building" = "grey50"),                     guide = guide_legend(override.aes = list(linetype = c("blank"),                                         nshape = c(NA)))) +    scale_color_manual(values = c("Footway" = "black",                                 "Viewpoint" = "red",                                "Visible area" = "red"),                     labels = c("Footway", "Viewpoint","Visible area"))+  guides(color = guide_legend(    order = 1,    override.aes = list(      color = c("black","red"),      fill  = c("transparent","transparent"),      linetype = c("solid","blank"),      shape = c(NA,16))))+  theme_minimal()+  coord_sf(datum = NA)+  theme(legend.title=element_blank())

buildings <- st_cast(buildings,"POLYGON")

st_isovist <- function(  buildings,  viewpoint,    # Defaults  rayno = 20,  raydist = 100) {    # Warning messages  if(!class(buildings)[1]=="sfc_POLYGON")     stop('Buildings must be sfc_POLYGON')  if(!class(viewpoint)[1]=="sfc_POINT") stop('Viewpoint must be sf object')    rayends     <- st_buffer(viewpoint,dist = raydist,nQuadSegs = (rayno-1)/4)  rayvertices <- st_cast(rayends,"POINT")    # Buildings in raydist  buildintersections <- st_intersects(buildings,rayends,sparse = FALSE)    # If no buildings block max view, return view  if (!TRUE %in% buildintersections){    isovist <- rayends  }    # Calculate isovist if buildings block view from viewpoint  if (TRUE %in% buildintersections){        rays <- lapply(X = 1:length(rayvertices), FUN = function(x) {      pair      <- st_combine(c(rayvertices[x],viewpoint))      line      <- st_cast(pair, "LINESTRING")      return(line)    })        rays <- do.call(c,rays)    rays <- st_sf(geometry = rays,                  id = 1:length(rays))        buildsinmaxview <- buildings[buildintersections]    buildsinmaxview <- st_union(buildsinmaxview)    raysioutsidebuilding <- st_difference(rays,buildsinmaxview)        # Getting each ray segement closest to viewpoint    multilines  <- dplyr::filter(raysioutsidebuilding, st_is(geometry, c("MULTILINESTRING")))    singlelines <- dplyr::filter(raysioutsidebuilding, st_is(geometry, c("LINESTRING")))    multilines  <- st_cast(multilines,"MULTIPOINT")    multilines  <- st_cast(multilines,"POINT")    singlelines <- st_cast(singlelines,"POINT")        # Getting furthest vertex of ray segement closest to view point    singlelines <- singlelines %>%       group_by(id) %>%      dplyr::slice_tail(n = 2) %>%      dplyr::slice_head(n = 1) %>%      summarise(do_union = FALSE,.groups = 'drop') %>%      st_cast("POINT")        multilines  <- multilines %>%       group_by(id) %>%      dplyr::slice_tail(n = 2) %>%      dplyr::slice_head(n = 1) %>%      summarise(do_union = FALSE,.groups = 'drop') %>%      st_cast("POINT")        # Combining vertices, ordering clockwise by ray angle and casting to polygon    alllines <- rbind(singlelines,multilines)    alllines <- alllines[order(alllines$id),]     isovist  <- st_cast(st_combine(alllines),"POLYGON")  }  isovist}

isovists   <- lapply(X = 1:length(viewpoints), FUN = function(x) {  viewpoint   <- viewpoints[x]  st_isovist(buildings = buildings,             viewpoint = viewpoint,             rayno = 41,             raydist = 100)})

isovists <- do.call(c,isovists)visareapoly <- st_union(isovists) ggplot() +  geom_sf(data = buildings,colour = "transparent",aes(fill = 'Building')) +  geom_sf(data = footway, aes(color = 'Footway')) +  geom_sf(data = viewpoints, aes(color = 'Viewpoint')) +  geom_sf(data = visareapoly,fill="transparent",aes(color = 'Visible area')) +  scale_fill_manual(values = c("Building" = "grey50"),                     guide = guide_legend(override.aes = list(linetype = c("blank"),                                          shape = c(NA)))) +  scale_color_manual(values = c("Footway" = "black",                                 "Viewpoint" = "red",                                "Visible area" = "red"),                     labels = c("Footway", "Viewpoint","Visible area"))+  guides( color = guide_legend(    order = 1,    override.aes = list(      color = c("black","red","red"),      fill  = c("transparent","transparent","white"),      linetype = c("solid","blank", "solid"),      shape = c(NA,16,NA))))+  theme_minimal()+  coord_sf(datum = NA)+  theme(legend.title=element_blank())

The post Isovists using uniform ray casting in R first appeared on R-bloggers.

Introduction

Around four years ago I was given a copy of Time Magazine’s specialty issue on Coffee together with a French press as a gift. At the time, I was satisfied with a regular instant cup of joe and did not know much about the vastness and culture of the industry. However, it was thanks to these gifts that I was able to learn a lot about coffee, such as the two major species of beans (Arabica and Robusta),the tasting process done by connoisseurs to rank various coffees(called “cupping”), about the altitude, climate and countries various coffees grow around the world. If you read this specialty issue by Time, you probably not only got a more expensive interest piqued (if you haven’t already), but also probably learned enough to hold your own with the the best of the coffee snobs out there.

(PSA- this blog is not sponsored by Time Magazine, but I won’t say no if I got an offer!)

In this blog post we’re going to examine the coffee_ratings dataset released back in the beginning of July 2020 in the Tidy Tuesday Project by R4DS. I initially started analyzing this dataset seeking to answer a lot of questions. But, because there is so much to discover and analyze from this relatively small dataset, I thought it is best to try to focus my question on a very simple one:

Where in the world can I find the best coffee beans?

While this question seems simple enough. There is a lot to uncover to answer this question.

Our Data (Some Exploratory Data Analysis)

Loading our data

I am loading the data with the tidytuesdayR package, if you want you can load the raw data with the readr package’s read_csv() function as well.

A Quick Glimpse

library(tidyverse)coffee_ratings<-tuesdata$coffee_ratingsglimpse(coffee_ratings)## Rows: 1,339## Columns: 43## $ total_cup_points       90.58, 89.92, 89.75, 89.00, 88.83, 88.83, 88.75, 88.67, 88.42, 88.25, 88.08, 87.92, 87.92, 87.92, 87.8...## $ species                "Arabica", "Arabica", "Arabica", "Arabica", "Arabica", "Arabica", "Arabica", "Arabica", "Arabica", "Ar...## $ owner                  "metad plc", "metad plc", "grounds for health admin", "yidnekachew dabessa", "metad plc", "ji-ae ahn",...## $ country_of_origin      "Ethiopia", "Ethiopia", "Guatemala", "Ethiopia", "Ethiopia", "Brazil", "Peru", "Ethiopia", "Ethiopia",...## $ farm_name              "metad plc", "metad plc", "san marcos barrancas \"san cristobal cuch", "yidnekachew dabessa coffee pla...## $ lot_number             NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "YNC-06114", NA, NA, NA, NA, N...## $ mill                   "metad plc", "metad plc", NA, "wolensu", "metad plc", NA, "hvc", "c.p.w.e", "c.p.w.e", "tulla coffee f...## $ ico_number             "2014/2015", "2014/2015", NA, NA, "2014/2015", NA, NA, "010/0338", "010/0338", "2014/15", NA, "unknown...## $ company                "metad agricultural developmet plc", "metad agricultural developmet plc", NA, "yidnekachew debessa cof...## $ altitude               "1950-2200", "1950-2200", "1600 - 1800 m", "1800-2200", "1950-2200", NA, NA, "1570-1700", "1570-1700",...## $ region                 "guji-hambela", "guji-hambela", NA, "oromia", "guji-hambela", NA, NA, "oromia", "oromiya", "snnp/kaffa...## $ producer               "METAD PLC", "METAD PLC", NA, "Yidnekachew Dabessa Coffee Plantation", "METAD PLC", NA, "HVC", "Bazen ...## $ number_of_bags         300, 300, 5, 320, 300, 100, 100, 300, 300, 50, 300, 10, 10, 1, 300, 10, 1, 150, 3, 250, 10, 250, 14, 1...## $ bag_weight             "60 kg", "60 kg", "1", "60 kg", "60 kg", "30 kg", "69 kg", "60 kg", "60 kg", "60 kg", "60 kg", "1 kg",...## $ in_country_partner     "METAD Agricultural Development plc", "METAD Agricultural Development plc", "Specialty Coffee Associat...## $ harvest_year           "2014", "2014", NA, "2014", "2014", "2013", "2012", "March 2010", "March 2010", "2014", "2014", "2014"...## $ grading_date           "April 4th, 2015", "April 4th, 2015", "May 31st, 2010", "March 26th, 2015", "April 4th, 2015", "Septem...## $ owner_1                "metad plc", "metad plc", "Grounds for Health Admin", "Yidnekachew Dabessa", "metad plc", "Ji-Ae Ahn",...## $ variety                NA, "Other", "Bourbon", NA, "Other", NA, "Other", NA, NA, "Other", NA, "Other", "Other", NA, NA, "Othe...## $ processing_method      "Washed / Wet", "Washed / Wet", NA, "Natural / Dry", "Washed / Wet", "Natural / Dry", "Washed / Wet", ...## $ aroma                  8.67, 8.75, 8.42, 8.17, 8.25, 8.58, 8.42, 8.25, 8.67, 8.08, 8.17, 8.25, 8.08, 8.33, 8.25, 8.00, 8.33, ...## $ flavor                 8.83, 8.67, 8.50, 8.58, 8.50, 8.42, 8.50, 8.33, 8.67, 8.58, 8.67, 8.42, 8.67, 8.42, 8.33, 8.50, 8.25, ...## $ aftertaste             8.67, 8.50, 8.42, 8.42, 8.25, 8.42, 8.33, 8.50, 8.58, 8.50, 8.25, 8.17, 8.33, 8.08, 8.50, 8.58, 7.83, ...## $ acidity                8.75, 8.58, 8.42, 8.42, 8.50, 8.50, 8.50, 8.42, 8.42, 8.50, 8.50, 8.33, 8.42, 8.25, 8.25, 8.17, 7.75, ...## $ body                   8.50, 8.42, 8.33, 8.50, 8.42, 8.25, 8.25, 8.33, 8.33, 7.67, 7.75, 8.08, 8.00, 8.25, 8.58, 8.17, 8.50, ...## $ balance                8.42, 8.42, 8.42, 8.25, 8.33, 8.33, 8.25, 8.50, 8.42, 8.42, 8.17, 8.17, 8.08, 8.00, 8.75, 8.00, 8.42, ...## $ uniformity             10.00, 10.00, 10.00, 10.00, 10.00, 10.00, 10.00, 10.00, 9.33, 10.00, 10.00, 10.00, 10.00, 10.00, 9.33,...## $ clean_cup              10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10...## $ sweetness              10.00, 10.00, 10.00, 10.00, 10.00, 10.00, 10.00, 9.33, 9.33, 10.00, 10.00, 10.00, 10.00, 10.00, 9.33, ...## $ cupper_points          8.75, 8.58, 9.25, 8.67, 8.58, 8.33, 8.50, 9.00, 8.67, 8.50, 8.58, 8.50, 8.33, 8.58, 8.50, 8.17, 8.33, ...## $ moisture               0.12, 0.12, 0.00, 0.11, 0.12, 0.11, 0.11, 0.03, 0.03, 0.10, 0.10, 0.00, 0.00, 0.00, 0.05, 0.00, 0.03, ...## $ category_one_defects   0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...## $ quakers                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...## $ color                  "Green", "Green", NA, "Green", "Green", "Bluish-Green", "Bluish-Green", NA, NA, "Green", NA, NA, NA, N...## $ category_two_defects   0, 1, 0, 2, 2, 1, 0, 0, 0, 4, 1, 0, 0, 2, 2, 0, 0, 2, 0, 8, 0, 2, 0, 0, 1, 2, 2, 1, 3, 0, 2, 1, 2, 0, ...## $ expiration             "April 3rd, 2016", "April 3rd, 2016", "May 31st, 2011", "March 25th, 2016", "April 3rd, 2016", "Septem...## $ certification_body     "METAD Agricultural Development plc", "METAD Agricultural Development plc", "Specialty Coffee Associat...## $ certification_address  "309fcf77415a3661ae83e027f7e5f05dad786e44", "309fcf77415a3661ae83e027f7e5f05dad786e44", "36d0d00a37243...## $ certification_contact  "19fef5a731de2db57d16da10287413f5f99bc2dd", "19fef5a731de2db57d16da10287413f5f99bc2dd", "0878a7d4b9d35...## $ unit_of_measurement    "m", "m", "m", "m", "m", "m", "m", "m", "m", "m", "m", "m", "m", "ft", "m", "m", "m", "m", "m", "m", "...## $ altitude_low_meters    1950.0, 1950.0, 1600.0, 1800.0, 1950.0, NA, NA, 1570.0, 1570.0, 1795.0, 1855.0, 1872.0, 1943.0, 609.6,...## $ altitude_high_meters   2200.0, 2200.0, 1800.0, 2200.0, 2200.0, NA, NA, 1700.0, 1700.0, 1850.0, 1955.0, 1872.0, 1943.0, 609.6,...## $ altitude_mean_meters   2075.0, 2075.0, 1700.0, 2000.0, 2075.0, NA, NA, 1635.0, 1635.0, 1822.5, 1905.0, 1872.0, 1943.0, 609.6,...

A quick glimpse of our data (no pun intended) is enough to indicate that our dataset is far from clean. It also looks like there is missing data everywhere. Lets see how much.

Missing Data

library(naniar)vis_miss(coffee_ratings)

Thankfully, it’s not as bad as I thought it was going to be. For the nature of my question I am only going to using the total_cupper_points, country_of_origin, grading_date and species variables which all have little to no missing data (I thought this would be more of an issue, but looking back at it I’m thankful it isn’t for this case.)

Quantites of Coffee per Country

As stated in the description of our dataset (see the readme.md)

“These data were collected from the Coffee Quality Institute’s review pages in January 2018.”

(I am not sure how grammatical that phrase is but ok.)

To better understand our data, lets look at the frequencies of our data in terms of countries listed in our data set. Because there is only one instance of missing data, we will remove it from our plots for aesthetic reasons.

library(ggthemes)# Need to make a new transformed dataset for this visualization(  country_table<-coffee_ratings %>%    count(country_of_origin = factor(country_of_origin)) %>%     mutate(pct = prop.table(n)) %>%    arrange(-pct) %>%     tibble())## # A tibble: 37 x 3##    country_of_origin                n    pct##                              ##  1 Mexico                         236 0.176 ##  2 Colombia                       183 0.137 ##  3 Guatemala                      181 0.135 ##  4 Brazil                         132 0.0986##  5 Taiwan                          75 0.0560##  6 United States (Hawaii)          73 0.0545##  7 Honduras                        53 0.0396##  8 Costa Rica                      51 0.0381##  9 Ethiopia                        44 0.0329## 10 Tanzania, United Republic Of    40 0.0299## # ... with 27 more rows# Together with my knowledge of ggplot and google, these visualizations became possibleggplot(  country_table %>% filter(country_of_origin != "NA"),  mapping = aes(    x = reorder(country_of_origin, n),    y = pct,    group = 1,    label = scales::percent(pct)  )) +  theme_fivethirtyeight() +  geom_bar(stat = "identity",           fill = "#634832") +  geom_text(position = position_dodge(width = 0.9),            # move to center of bars            hjust = -0.05,            #Have Text just above bars            size = 2.5) +  labs(x = "Country of Origin",       y = "Proportion of Dataset") +  theme(axis.text.x = element_text(    angle = 90,    vjust = 0.5,    hjust = 1  )) +  ggtitle("Country of Origin Listed in Coffee Ratings Dataset " ) +   # This Emoji messes up this line in R markdown but hey, it  scale_y_continuous(labels = scales::percent) +                        # looks good.  coord_flip()

From a brief look at our table and bar chart we see that over 54% of our dataset consists of coffees from Mexico, Columbia, Guatemala and Brazil. But this only tells us part of the story, what species of coffees do we have in our dataset from each country?

Before looking at that lets look at the overall Arabica/Robusta proportion in our dataset:

# Need to make a new transformed dataset for this visualizationspecies_table<-coffee_ratings %>%     count(species = factor(species)) %>%     mutate(pct = prop.table(n)) %>% tibble()ggplot(species_table,mapping=aes(x=species,y=pct,group=1,label=scales::percent(pct)))+   theme_fivethirtyeight()+  geom_bar(stat="identity",           fill=c("#634832","#3b2f2f"))+    geom_text(position = position_dodge(width=0.9),    # move to center of bars              vjust=-0.5, #Have Text just above bars              size = 3)+  scale_y_continuous(labels = scales::percent)+  ggtitle("Arabica vs Robusta Proportion in Dataset ")

Wow! only 2% of Coffee in our dataset is from Robusta beans! But if you think about this in context, this shouldn’t be too much of a suprise. Robusta coffee is primarily used in instant coffee,espresso and filler for coffee blends. The reason why Robusta coffee beans are not graded proportionately as Arabica beans are is due to the fact that the quality of these bitter, earthy beans are usually not as desirable to coffee drinkers as their smoother, richer Arabica counterparts.

With that in mind, lets see how the breakdown proportionally per country:

# Need to make a new transformed datasets for this visualization(  arabica_countries<-coffee_ratings %>%   filter(species =="Arabica") %>%     count(species=factor(species),          country=country_of_origin) %>%     mutate(pct = prop.table(n)) %>%     arrange(-n) %>%   tibble())## # A tibble: 37 x 4##    species country                          n    pct##                                 ##  1 Arabica Mexico                         236 0.180 ##  2 Arabica Colombia                       183 0.140 ##  3 Arabica Guatemala                      181 0.138 ##  4 Arabica Brazil                         132 0.101 ##  5 Arabica Taiwan                          75 0.0572##  6 Arabica United States (Hawaii)          73 0.0557##  7 Arabica Honduras                        53 0.0404##  8 Arabica Costa Rica                      51 0.0389##  9 Arabica Ethiopia                        44 0.0336## 10 Arabica Tanzania, United Republic Of    40 0.0305## # ... with 27 more rowsggplot(arabica_countries %>% filter(country!="NA"),       mapping=aes(x=reorder(country,n),y=pct,group=1,label=scales::percent(pct))) +   theme_fivethirtyeight()+  geom_bar(stat="identity",           fill="#634832")+  geom_text(position = position_dodge(width = 0.9),            # move to center of bars            hjust = -0.05,            #Have Text just above bars            size = 2.5) +  ggtitle("Arabica Coffee Countries (for our dataset) ") +   scale_y_continuous(labels = scales::percent) +                       coord_flip()

(  robusta_countries<-coffee_ratings %>%     filter(species =="Robusta") %>%     count(species = factor(species),          country=country_of_origin) %>%     mutate(pct = prop.table(n)) %>%    arrange(-n) %>%   tibble())## # A tibble: 5 x 4##   species country           n    pct##                 ## 1 Robusta India            13 0.464 ## 2 Robusta Uganda           10 0.357 ## 3 Robusta Ecuador           2 0.0714## 4 Robusta United States     2 0.0714## 5 Robusta Vietnam           1 0.0357ggplot(robusta_countries %>% filter(country!="NA"),       mapping=aes(x=reorder(country,n),y=pct,group=1,label=scales::percent(pct))) +   theme_fivethirtyeight()+  geom_bar(stat="identity",           fill="#3b2f2f")+  geom_text(position = position_dodge(width = 0.9),            # move to center of bars            hjust = -0.05,            #Have Text just above bars            size = 2.5) +  ggtitle("Robusta Coffee Countries (for our dataset) ") +   scale_y_continuous(labels = scales::percent) +                       coord_flip()

The Robusta coffees that we have in this dataset are mostly from India and Uganda, with a few coffees from the Ecuador, the United States and Vietnam. With that being known, Lets look at the Arabica/Robusta ratio for countries that we have Robusta Data on.

coffee_ratings %>%   filter(country_of_origin %in% c("India","Uganda","Ecuador","United States","Vietnam")) %>%   count(country_of_origin,species) %>%   group_by(country_of_origin)## # A tibble: 10 x 3## # Groups:   country_of_origin [5]##    country_of_origin species     n##                    ##  1 Ecuador           Arabica     1##  2 Ecuador           Robusta     2##  3 India             Arabica     1##  4 India             Robusta    13##  5 Uganda            Arabica    26##  6 Uganda            Robusta    10##  7 United States     Arabica     8##  8 United States     Robusta     2##  9 Vietnam           Arabica     7## 10 Vietnam           Robusta     1ggplot(coffee_ratings %>% filter(country_of_origin %in% c("India","Uganda","Ecuador","United States","Vietnam")),       mapping=aes(x=country_of_origin,fill=species))+  theme_fivethirtyeight()+  geom_bar(position="fill")+  scale_fill_manual(values=c("#BE9B7B", "#3b2f2f"))+  theme(legend.title = element_blank())+  ggtitle("Arabica/Robusta Ratio from countries with Robusta data ")

Now that we have better understanding of where our coffees come from, we can get into trying to answer the question of where the best coffee beans are in the world.

Well, it depends.

What type? What year?

It would be nice to just pick out the highest rated coffee and be done with it, but that wouldn’t tell us anything (or really motivate a blog post). We need to consider is when was a given coffee graded. That can tell us the performance of a given country’s over time. Additionally, we need to consider the species of bean- where is the best ranked Arabica coffee from? Where is the best Robusta coffee from?

Before we can answer this question, we need to clean the grading_date and convert them into the date data from. Thankfully, the lubridate package will help us with doing this relatively easy. After that we will formulate our data set with the dplyr package to get the data in the form we need for our visualization. |

library(lubridate)# Getting the year data coffee_ratings$new_dates<-coffee_ratings$grading_date %>% mdy()coffee_ratings$score_year<- coffee_ratings$new_dates %>% year()# Dataset for visualizations(  top_annual_score<- coffee_ratings %>%  group_by(species,           score_year,           country_of_origin) %>%   summarise(max_points = max(total_cup_points)) %>%   filter(max_points == max(max_points)) %>%   arrange(-max_points))## # A tibble: 15 x 4## # Groups:   species, score_year [15]##    species score_year country_of_origin max_points##                               ##  1 Arabica       2015 Ethiopia                90.6##  2 Arabica       2010 Guatemala               89.8##  3 Arabica       2013 Brazil                  88.8##  4 Arabica       2012 Peru                    88.8##  5 Arabica       2016 China                   87.2##  6 Arabica       2014 Costa Rica              87.2##  7 Arabica       2011 Brazil                  86.9##  8 Arabica       2017 Honduras                86.7##  9 Arabica       2018 Kenya                   84.6## 10 Robusta       2014 Uganda                  83.8## 11 Robusta       2017 India                   83.5## 12 Robusta       2015 India                   83.2## 13 Robusta       2012 India                   82.8## 14 Robusta       2016 India                   82.5## 15 Robusta       2013 India                   81.2ggplot(top_annual_score,       mapping=aes(x=score_year,                   y=max_points,                   label=paste0(score_year,"\n",country_of_origin,"\n", max_points),                   color=country_of_origin))+  theme_fivethirtyeight()+  geom_text(position = position_dodge(width = 0.9),            # move to center of bars            hjust =-0.2,            #Have Text just above bars            size =3.5) +  geom_point(size=4,             alpha=0.8)+  theme(legend.position = "none")+  facet_wrap(~species)+  ggtitle(" Top Scoring Coffees by Year - Faceted on Species ")

From our visualization and table we see for Arabica beans, the top coffee varied from country to country for a given year. However for Robusta, India seemed to have dominated with consistent wins from 2012 – 2017 with an exception of Uganda beating them in 2014.

Overall, for our given timespan in our dataset, for Arabica beans (as well as our entire dataset) Ethiopia scored the highest with a score of 90.58 and for Robusta Beans Uganda had the highest score of 83.75.

The overall summary for of scores for Arabica and Robusta beans accross the years is plotted in the below visualization with boxplots.

(arabica_robusta_average_score<-   coffee_ratings %>%   group_by(species) %>%   summarise(average_score = mean(total_cup_points),            lower_ci = mean(total_cup_points) - 1.96*sqrt(var(total_cup_points)/length(total_cup_points)),            upper_ci = mean(total_cup_points) + 1.96*sqrt(var(total_cup_points)/length(total_cup_points)))  )## # A tibble: 2 x 4##   species average_score lower_ci upper_ci##                      ## 1 Arabica          82.1     81.9     82.3## 2 Robusta          80.9     80.0     81.8ggplot(coffee_ratings,mapping=aes(x=score_year,y=total_cup_points,group=score_year))+  theme_fivethirtyeight()+  geom_boxplot(color="#3b2f2f")+  coord_flip()+  facet_wrap(~species)+  geom_hline(data=arabica_robusta_average_score,             mapping=aes(yintercept=average_score),             size= 0.5)+  geom_hline(data=arabica_robusta_average_score,             mapping=aes(yintercept=lower_ci),             linetype="dashed",             size= 0.5)+   geom_hline(data=arabica_robusta_average_score,             mapping=aes(yintercept=upper_ci),             linetype="dashed",             size= 0.5)+  ggtitle("Boxplots of Arabica and Robusta Beans from 2010-2018 \n           with confidence intervals plotted")

Besides for some outliers on the lower end of the scoring range, most of these coffees in this dataset are on average score around 80 or above. What can be implied from here is that the coffees that come in to be graded by the Coffee Quality Institute are usually those which have are assumed to be high in quality. This shouldn’t be a surprise because it appears that beans graded by the CQI are usually those which are submitted as it says it on the site’s banner

Welcome to the Coffee Quality Institute (CQI) database, which allows users to submit a sample for Q Grading…

Conclusion

Its not surprising for our data set that Robusta beans scored poorer than their Arabica counterparts. That is something that anyone with some background in coffee will tell you- Arabica is generally more desirable by coffee drinkers and Robusta is usually used for instant coffee, Espresso and filler for coffee blends.

What is more telling about this dataset is that the best coffee is not something which is country specific for Arabica beans. However for Robusta beans, India and Uganda being top annual scorers in this domain seems to be indicative of higher quality Robusta in these countries.

Additionally, based on the fact that the average score for both Arabica and Robusta Beans is in the 80 range is telling that the beans which are being graded are those with a assumed to possess higher quality are submitted to be graded. Does this mean we have a overarching view of the worldwide coffee quality range? Certainly not.

But it could be telling on what quality of coffee you’re having the next time you visit a coffee shop which has coffee which is graded by the CQI.

(Let me know how it tastes!)

Thanks for reading!

The post RObservations #7 – #TidyTuesday – Analysing Coffee Ratings Data first appeared on R-bloggers.

The rOpenSci community is supported by our Code of Conduct with a clear description of unacceptable behaviors, instructions on how to make a report, and information on how reports are handled. We, the Code of Conduct Committee, are responsible for receiving, investigating, deciding, enforcing and reporting on all reports of potential violations of our Code. We are committed to transparency with our community while upholding the privacy of victims and people who report incidents.

In 2020, we did not receive any Code of Conduct incident reports.

We acknowledge that a lack of incident reports does not mean we’re perfect. If in doubt, please contact us at conduct at ropensci.org. We welcome feedback.

The post rOpenSci 2020 Code of Conduct Transparency Report first appeared on R-bloggers.

Our community is our best asset. It’s so important to us, it’s in our mission statement. We recognize that communities are not inclusive by default; they require deliberate attention, including an enforceable Code of Conduct.

rOpenSci is committed to providing a safe, inclusive, welcoming, and harassment-free experience for everyone. We welcome people of all backgrounds and identities, including but not limited to gender, gender identity and expression, age, sexual orientation, disability, physical appearance, body size, race, ethnicity, religion (or lack thereof), or technology choices. We are anti-racist. We welcome anyone, no matter their technical expertise, career stage, or work sector.

We are all supported by rOpenSci’s Code of Conduct that applies to all people participating in the rOpenSci community, including rOpenSci staff and leadership. It applies to all modes of interaction online including GitHub project repositories, the rOpenSci discussion forum, Slack, Community Calls, and in person at rOpenSci-hosted events or events officially endorsed by rOpenSci, including social gatherings affiliated with the event. It is developed and enforced by a committee including rOpenSci staff and an independent community member.

Here we report on our annual review of rOpenSci’s Code of Conduct, reporting process, and internal guidelines for handling reports and enforcement.

Updates

1. A change to the committee. After serving on the Code of Conduct Committee for two years, Scott Chamberlain is stepping down and Mark Padgham has joined. Committee members for 2021 are Stefanie Butland (rOpenSci Community Manager), Mark Padgham (rOpenSci Software Research Scientist) and Kara Woo (independent community member). We are responsible for receiving, investigating, deciding, enforcing and reporting on all reports of potential violations of our Code.
2. No changes have been made to the text of the Code
3. This code of conduct now also applies to packages in the rOpenSci GitHub organization, as described in our package development guide.

With this change of committee members, this will be version 2.2, dated January 5, 2021.

We welcome your feedback by email to conduct at ropensci.org, and we thank you for continuing to work with us to make rOpenSci a safe, enjoyable, friendly and enriching experience for everyone who participates.

The post rOpenSci Code of Conduct Annual Review first appeared on R-bloggers.