R & Python Rosetta Stone: EDA with dplyr vs pandas

November 5, 2020, 9:00 am

≫ Next: It’s time to retire the “data scientist” label

≪ Previous: RvsPython #5.1: Making the Game even with Python’s Best Practices

[This article was first published on R on head spin - the Heads or Tails blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

This is the first post in a new series featuring translations between R and Python code for common data science and machine learning tasks. A Rosetta Stone, if you will. I’m writing this mainly as a documented cheat sheet for myself, as I’m frequently switching between the two languages. Personally, I have learned Python and R around the same time, several years ago, but tidy R is much more intuitive to me than any other language. Hopefully, these posts can be useful to others in a similar situation. My point of reference is primarily R – with the aim to provide equivalent Python code – but occasionally I will look at translations of Python features into R.

The first post will focus on the tidyverse backbone dplyr and compare it to Python’s data science workhorse pandas.

As with all my blog posts, the code will run in Rmarkdown through the fantastic Rstudio IDE. All the output will be reproducible. Rstudio provides Python support via the great reticulate package. If you haven’t heard of it yet, check out my intro post on reticulate to get started.

Note: you need at least RStudio version 1.2 to be able to pass objects between R and Python. In addition, as always, here are the required packages. We are also setting the python path (see the intro post for more details on the path) and importing pandas:

# R
libs <- c('dplyr',                # wrangling
          'palmerpenguins',       # data
          'knitr','kableExtra',   # table styling
          'reticulate')           # python support
invisible(lapply(libs, library, character.only = TRUE))

use_python("/usr/bin/python")

df <- penguins %>%
  mutate_if(is.integer, as.double)
# Python
import pandas as pd
pd.set_option('display.max_columns', None)

We will be using the Palmer Penguins dataset, as provided by the brilliant Allison Horst through the eponymous R package– complete with her trademark adorable artwork. I was looking for an excuse to work with this dataset. Therefore, this will be a genuine example for an exploratory analysis, as I’m encountering the data for the first time.

General overview

It’s best to start your EDA by looking at the big picture: How large is the dataset? How many features are there? What are the data types? Here we have tabular data, but similar steps can be taken for text or images as well.

In R, I typically go with a combination of head, glimpse, and summary to get a broad idea of the data properties. (Beyond dplyr, there’s also the skimr package for more sophisticated data overviews.)

With head, we see the first (by default) 6 rows of the dataset:

# R
head(df)
## # A tibble: 6 x 8
##   species island bill_length_mm bill_depth_mm flipper_length_… body_mass_g
##                                             
## 1 Adelie  Torge…           39.1          18.7              181        3750
## 2 Adelie  Torge…           39.5          17.4              186        3800
## 3 Adelie  Torge…           40.3          18                195        3250
## 4 Adelie  Torge…           NA            NA                 NA          NA
## 5 Adelie  Torge…           36.7          19.3              193        3450
## 6 Adelie  Torge…           39.3          20.6              190        3650
## # … with 2 more variables: sex , year

We find that our data is a mix of numerical and categorical columns. There are 8 columns in total. We get an idea of the order of magnitude of the numeric features, see that the categorical ones have text, and already spot a couple of missing values. (Note, that I converted all integer columns to double for easier back-and-forth with Python).

The pandashead command is essentially the same. One general difference here is that in pandas (and Python in general) everything is an object. Methods (and attributes) associated with the object, which is a pandasDataFrame here, are accessed via the dot “.” operator. For passing an R object to Python we preface it with r. like such:

# Python
r.df.head()
##   species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
## 0  Adelie  Torgersen            39.1           18.7              181.0
## 1  Adelie  Torgersen            39.5           17.4              186.0
## 2  Adelie  Torgersen            40.3           18.0              195.0
## 3  Adelie  Torgersen             NaN            NaN                NaN
## 4  Adelie  Torgersen            36.7           19.3              193.0
##
##    body_mass_g     sex    year
## 0       3750.0    male  2007.0
## 1       3800.0  female  2007.0
## 2       3250.0  female  2007.0
## 3          NaN     NaN  2007.0
## 4       3450.0  female  2007.0

The output is less pretty than for Rmarkdown, but the result is pretty much the same. We don’t see column types, only their values. Another property of pandas data frames is that they come with a row index. By default this is a sequential number of rows, but anything can become an index.

With dplyr’s glimpse we can see a more compact, transposed display of column types and their values. Especially for datasets with many columns this can be a vital complement to head. We also see the number of rows and columns:

# R
glimpse(df)
## Observations: 344
## Variables: 8
## $ species            Adelie, Adelie, Adelie, Adelie, Adelie, Adelie…
## $ island             Torgersen, Torgersen, Torgersen, Torgersen, To…
## $ bill_length_mm     39.1, 39.5, 40.3, NA, 36.7, 39.3, 38.9, 39.2, …
## $ bill_depth_mm      18.7, 17.4, 18.0, NA, 19.3, 20.6, 17.8, 19.6, …
## $ flipper_length_mm  181, 186, 195, NA, 193, 190, 181, 195, 193, 19…
## $ body_mass_g        3750, 3800, 3250, NA, 3450, 3650, 3625, 4675, …
## $ sex                male, female, female, NA, female, male, female…
## $ year               2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007…

Alongside the 8 columns, our dataset has 344 rows. That’s pretty small.

In pandas, there’s no identical equivalent to glimpse. Instead, we can use the info method to give us the feature types in vertical form. We also see the number of non-null features (the “sex” column has the fewest), together with the number of rows and columns.

# Python
r.df.info()
## 
## RangeIndex: 344 entries, 0 to 343
## Data columns (total 8 columns):
## species              344 non-null category
## island               344 non-null category
## bill_length_mm       342 non-null float64
## bill_depth_mm        342 non-null float64
## flipper_length_mm    342 non-null float64
## body_mass_g          342 non-null float64
## sex                  333 non-null category
## year                 344 non-null float64
## dtypes: category(3), float64(5)
## memory usage: 14.8 KB

While several columns have missing values, the overall number of those null entries is small.

Additionally, pandas gives us a summary of data types (3 categorical, 5 float) and tells us how much memory our data takes up. In R, you can use the utils tool object.size for the latter:

# R
object.size(df)
## 21336 bytes

Next up is summary, which provides basic overview stats for each feature. Here in particular, we learn that there are 3 penguin species, 3 islands, and 11 missing values in the sex column. “Chinstrap” penguins are rarer than the other two species; and Torgersen is the least frequent island. We also see the fundamental characteristics of the numeric features (min, max, quartiles, median). Mean and median are pretty close for most of those, which suggests little skewness:

# R
summary(df)
##       species          island    bill_length_mm  bill_depth_mm
##  Adelie   :152   Biscoe   :168   Min.   :32.10   Min.   :13.10
##  Chinstrap: 68   Dream    :124   1st Qu.:39.23   1st Qu.:15.60
##  Gentoo   :124   Torgersen: 52   Median :44.45   Median :17.30
##                                  Mean   :43.92   Mean   :17.15
##                                  3rd Qu.:48.50   3rd Qu.:18.70
##                                  Max.   :59.60   Max.   :21.50
##                                  NA's   :2       NA's   :2
##  flipper_length_mm  body_mass_g       sex           year
##  Min.   :172.0     Min.   :2700   female:165   Min.   :2007
##  1st Qu.:190.0     1st Qu.:3550   male  :168   1st Qu.:2007
##  Median :197.0     Median :4050   NA's  : 11   Median :2008
##  Mean   :200.9     Mean   :4202                Mean   :2008
##  3rd Qu.:213.0     3rd Qu.:4750                3rd Qu.:2009
##  Max.   :231.0     Max.   :6300                Max.   :2009
##  NA's   :2         NA's   :2

The closest pandas equivalent to summary is describe. By default this only includes the numeric columns, but you can get around that by passing a list of features types that you want to include:

# Python
r.df.describe(include = ['float', 'category'])
##        species  island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
## count      344     344      342.000000     342.000000         342.000000
## unique       3       3             NaN            NaN                NaN
## top     Adelie  Biscoe             NaN            NaN                NaN
## freq       152     168             NaN            NaN                NaN
## mean       NaN     NaN       43.921930      17.151170         200.915205
## std        NaN     NaN        5.459584       1.974793          14.061714
## min        NaN     NaN       32.100000      13.100000         172.000000
## 25%        NaN     NaN       39.225000      15.600000         190.000000
## 50%        NaN     NaN       44.450000      17.300000         197.000000
## 75%        NaN     NaN       48.500000      18.700000         213.000000
## max        NaN     NaN       59.600000      21.500000         231.000000
##
##         body_mass_g   sex         year
## count    342.000000   333   344.000000
## unique          NaN     2          NaN
## top             NaN  male          NaN
## freq            NaN   168          NaN
## mean    4201.754386   NaN  2008.029070
## std      801.954536   NaN     0.818356
## min     2700.000000   NaN  2007.000000
## 25%     3550.000000   NaN  2007.000000
## 50%     4050.000000   NaN  2008.000000
## 75%     4750.000000   NaN  2009.000000
## max     6300.000000   NaN  2009.000000

You see that the formatting is less clever, since categorical indicators like “unique” or “top” are shown (with NAs) for the numeric columns and vice versa. Also, we only get told that there are 3 species and the most frequent one is “Adelie”; not the full counts per species.

We have already learned a lot about our data from those basic overview tools. Typically, at this point in the EDA I would now start plotting feature distributions and interactions. Since we’re only focussing on dplyr here, this part will have to wait for a future “ggplot2 vs seaborn” episode. For now, let’s look at the most simple overview before moving on to dplyr verbs: number of rows and columns.

In R, there is dim while pandas has shape:

# R
dim(df)
## [1] 344   8
# Python
r.df.shape
## (344, 8)

Subsetting rows and columns

For extracting subsets of rows and columns, dplyr has the verbs filter and select, respectively. For instance, let’s look at the species and sex of the penguins with the shortest bills:

# R
df %>%
  filter(bill_length_mm < 34) %>%
  select(species, sex, bill_length_mm)
## # A tibble: 3 x 3
##   species sex    bill_length_mm
##                 
## 1 Adelie  female           33.5
## 2 Adelie  female           33.1
## 3 Adelie  female           32.1

All of those are female Adelie penguins. This is good news for a potential species classifier.

In pandas, there are several ways to achieve the same result. All of them are a little more complicated than our two dplyr verbs. The most pythonic way is probably to use the loc operator like such:

# Python
r.df.loc[r.df.bill_length_mm < 34, ['species', 'sex', 'bill_length_mm']]
##     species     sex  bill_length_mm
## 70   Adelie  female            33.5
## 98   Adelie  female            33.1
## 142  Adelie  female            32.1

This indexing via the “[]” brackets is essentially the same as in base R:

# R
na.omit(df[df$bill_length_mm < 34, c('species', 'sex', 'bill_length_mm')])
## # A tibble: 3 x 3
##   species sex    bill_length_mm
##                 
## 1 Adelie  female           33.5
## 2 Adelie  female           33.1
## 3 Adelie  female           32.1

Another way is to use the pandas method query as an equivalent for filter. This allows us to essentially use dplyr-style evaluation syntax. I found that in practice, query is often notably slower than the other indexing tools. This can be important if your’re dealing with large datasets:

# Python
r.df.query("bill_length_mm < 34").loc[:, ['species', 'sex', 'bill_length_mm']]
##     species     sex  bill_length_mm
## 70   Adelie  female            33.5
## 98   Adelie  female            33.1
## 142  Adelie  female            32.1

In tidy R, the dplyr verb select can extract by value as well as by position. And for extracting rows by position there is slice. This is just an arbitrary subset:

# R
df %>%
  slice(c(1,2,3)) %>%
  select(c(4,5,6))
## # A tibble: 3 x 3
##   bill_depth_mm flipper_length_mm body_mass_g
##                               
## 1          18.7               181        3750
## 2          17.4               186        3800
## 3          18                 195        3250

In pandas, we would use bracket indexing again, but instead of loc we now need iloc (i.e. locating by index). This might be a good point to remind ourselves that Python starts counting from 0 and R starts from 1:

# Python
r.df.iloc[[0, 1, 2], [3, 4, 5]]
##    bill_depth_mm  flipper_length_mm  body_mass_g
## 0           18.7              181.0       3750.0
## 1           17.4              186.0       3800.0
## 2           18.0              195.0       3250.0

To emphasise again: pandas uses loc for conditional indexing and iloc for positional indexing. This takes some getting uses to, but this simple rule covers most of pandas’ subsetting operations.

Retaining unique rows and removing duplicates can be thought of as another way of subsetting a data frame. In dplyr, there’s the distinct function which takes as arguments the columns that are considered for identifying duplicated rows:

# R
df %>%
  slice(c(2, 4, 186)) %>%
  distinct(species, island)
## # A tibble: 2 x 2
##   species island
##      
## 1 Adelie  Torgersen
## 2 Gentoo  Biscoe

In pandas we can achieve the same result via the drop_duplicates method:

r.df.iloc[[2, 4, 186], :].drop_duplicates(['species', 'island'])
##     species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
## 2    Adelie  Torgersen            40.3           18.0              195.0
## 186  Gentoo     Biscoe            49.1           14.8              220.0
##
##      body_mass_g     sex    year
## 2         3250.0  female  2007.0
## 186       5150.0  female  2008.0

In order to retain only the two affected rows, like in the dplyr example above, you would have to select them using .loc.

Arrange and Sample

Let’s deal with the arranging and sampling of rows in one fell swoop. In dplyr, we use sample_n (or sample_frac) to choose a random subset of n rows (or a fraction frac of rows). Ordering a row by its values uses the verb arrange, optionally with the desc tool to specific descending order:

# R
set.seed(4321)
df %>%
  select(species, bill_length_mm) %>%
  sample_n(4) %>%
  arrange(desc(bill_length_mm))
## # A tibble: 4 x 2
##   species   bill_length_mm
##                 
## 1 Chinstrap           47.5
## 2 Adelie              42.7
## 3 Adelie              40.2
## 4 Adelie              34.6

In pandas, random sampling is done through the sample function, which can take either a fraction or a number of rows. Here, we can also pass directly a random seed (called random_state), instead of defining it outside the function via R’s set.seed. Arranging is called sort_values and we have to specify an ascending flag because pandas wants to be different:

# Python
r.df.loc[:, ['species', 'bill_length_mm']].\
  sample(n = 4, random_state = 4321).\
  sort_values('bill_length_mm', ascending = False)
##        species  bill_length_mm
## 305  Chinstrap            52.8
## 301  Chinstrap            52.0
## 291  Chinstrap            50.5
## 165     Gentoo            48.4

As you see, the pandas dot methods can be chained in a way reminiscent of the dplyr pipe (%>%). The scope for this style is much narrower than the pipe, though. It only works for methods and attributes associated with the specific pandas data frame and its transformation results. Nevertheless, it has a certain power and elegance for pipe aficionados. When written accross multiple lines it requires the Python line continuation operator \.

Group By and Summarise

Here is where dplyr really shines. Grouping and summarising transformations fit seemlessly into any wrangling workflow by preserving the tidy tibble data format. The verbs are group_by and summarise. Let’s compare average bill lengths among species to find that “Adelie” penguins have significantly shorter bills:

# R
df %>%
  group_by(species) %>%
  summarise(mean_bill_length = mean(bill_length_mm, na.rm = TRUE),
            sd_bill_length = sd(bill_length_mm, na.rm = TRUE))
## # A tibble: 3 x 3
##   species   mean_bill_length sd_bill_length
##                             
## 1 Adelie                38.8           2.66
## 2 Chinstrap             48.8           3.34
## 3 Gentoo                47.5           3.08

This is consistent to what we had seen before for the 3 rows with shortes bills. The results also indicate that it would be much harder to try to separate “Chinstrap” vs “Gentoo” by bill_length.

In terms of usage, pandas is similar: we have groupby (without the “_”) to define the grouping, and agg to aggregate (i.e. summarise) according to specific measures for certain features:

# Python
r.df.groupby('species').agg({'bill_length_mm': ['mean', 'std']})
##           bill_length_mm
##                     mean       std
## species
## Adelie         38.791391  2.663405
## Chinstrap      48.833824  3.339256
## Gentoo         47.504878  3.081857

This is pretty much the default way in pandas. It is worth noting that pandas, and Python in general, gets a lot of milage out of 2 data structures: lists like [1, 2, 3] which are the equivalent of c(1, 2, 3), and dictionaries{'a': 1, 'b': 2} which are sets of key-value pairs. Values in a dictionary can be pretty much anything, including lists or dictionaries. (Most uses cases for dictionaries are probably served by dplyr tibbles.) Here we use a dictionary to define which column we want to summarise and how. Note again the chaining via dots.

The outcome of this operation, however, is slightly different from dplyr in that we get a hierarchical data frame with a categorical index (this is no longer a “normal” column") and 2 data columns that both have the designation bill_length_mm:

# Python
r.df.groupby('species').agg({'bill_length_mm': ['mean', 'std']}).info()
## 
## CategoricalIndex: 3 entries, Adelie to Gentoo
## Data columns (total 2 columns):
## (bill_length_mm, mean)    3 non-null float64
## (bill_length_mm, std)     3 non-null float64
## dtypes: float64(2)
## memory usage: 155.0 bytes

You will see the limitations of that format if you try to chain another method. To get something instead that’s more closely resembling our dplyr output, here is a different way: we forego the dictionary in favour of a simple list, then add a suffix later, and finally reset the index to a normal column:

# Python
r.df.groupby('species').bill_length_mm.agg(['mean', 'std']).add_suffix("_bill_length").reset_index()
##      species  mean_bill_length  std_bill_length
## 0     Adelie         38.791391         2.663405
## 1  Chinstrap         48.833824         3.339256
## 2     Gentoo         47.504878         3.081857

Now we can treat the result like a typical data frame. Note, that here we use another form of column indexing to select bill_length_mm after the groupby. This shorthand, which treats a feature as an object attribute, only works for single columns. Told you that pandas indexing can be a little confusing.

Joining and binding

There’s another source of dplyr vs pandas confusion when it comes to SQL-style joins and to binding rows and columns. To demonstrate, we’ll create an additional data frame which holds the mean bill length by species. And we pretend that it’s a separate table. In terms of analysis steps, this crosses over from EDA into feature engineering territory, but that line can get blurry if you’re exploring a dataset’s hidden depths.

# R
df2 <- df %>%
  group_by(species) %>%
  summarise(mean_bill = mean(bill_length_mm, na.rm = TRUE))

df2
## # A tibble: 3 x 2
##   species   mean_bill
##            
## 1 Adelie         38.8
## 2 Chinstrap      48.8
## 3 Gentoo         47.5

The dplyr verbs for SQL-like joins are very similar to the various SQL flavours. We have left_join, right_join, inner_join, outer_join; as well as the very useful filtering joins semi_join and anti_join (keep and discard what matches, respectively):

# R
set.seed(4321)
df %>%
  left_join(df2, by = "species") %>%
  select(species, bill_length_mm, mean_bill) %>%
  sample_n(5)
## # A tibble: 5 x 3
##   species   bill_length_mm mean_bill
##                      
## 1 Adelie              42.7      38.8
## 2 Chinstrap           47.5      48.8
## 3 Adelie              40.2      38.8
## 4 Adelie              34.6      38.8
## 5 Gentoo              53.4      47.5

Now we can for instance subtract the mean bill length from the individual values to get the residuals or to standardise our features.

In pandas, joining uses the merge operator. You have to specify the type of join via the how parameter and the join key via on, like such:

# Python
r.df.\
  sample(n = 5, random_state = 24).\
  merge(r.df2, how = "left", on = "species").\
  loc[:, ['species', 'bill_length_mm', 'mean_bill']]
##      species  bill_length_mm  mean_bill
## 0     Gentoo            43.4  47.504878
## 1  Chinstrap            51.7  48.833824
## 2     Gentoo            46.2  47.504878
## 3     Adelie            43.1  38.791391
## 4     Adelie            41.3  38.791391

Now, to bring in the aforementioned source of confusion: pandas also has an operator called join which joins by matching indeces, instead of columns, between two tables. This is a pretty pandas-specific convenience shortcut, since it relies on the data frame index. In practice I recommend using merge instead. The little conveniece provided by join is not worth the additional confusion.

Binding rows and columns in dplyr uses bind_rows and bind_cols. For the rows, there’s really not much of a practical requirement other than that some of the column names match and that those ideally have the same type (which is not strictly necessary):

# R
head(df, 2) %>%
  bind_rows(tail(df, 2) %>% select(year, everything(), -sex))
## # A tibble: 4 x 8
##   species island bill_length_mm bill_depth_mm flipper_length_… body_mass_g
##                                             
## 1 Adelie  Torge…           39.1          18.7              181        3750
## 2 Adelie  Torge…           39.5          17.4              186        3800
## 3 Chinst… Dream            50.8          19                210        4100
## 4 Chinst… Dream            50.2          18.7              198        3775
## # … with 2 more variables: sex , year

For columns, it is necessary that the features we’re binding to a tibble have the same number of rows. The big assumption here is that the row orders match (but that can be set beforehand):

# R
select(df, species) %>%
  bind_cols(df %>% select(year)) %>%
  head(5)
## # A tibble: 5 x 2
##   species  year
##      
## 1 Adelie   2007
## 2 Adelie   2007
## 3 Adelie   2007
## 4 Adelie   2007
## 5 Adelie   2007

The pandas equivalent to bind_rows is append. This nomenclature is borrowed from Python’s overall reliance on list operations.

# Python
r.df.head(2).append(r.df.tail(2).drop("sex", axis = "columns"), sort = False)
##        species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
## 0       Adelie  Torgersen            39.1           18.7              181.0
## 1       Adelie  Torgersen            39.5           17.4              186.0
## 342  Chinstrap      Dream            50.8           19.0              210.0
## 343  Chinstrap      Dream            50.2           18.7              198.0
##
##      body_mass_g     sex    year
## 0         3750.0    male  2007.0
## 1         3800.0  female  2007.0
## 342       4100.0     NaN  2009.0
## 343       3775.0     NaN  2009.0

Here we also demonstrate how to drop a column from a data frame (i.e. the equivalent to dplyrselect with a minus).

Binding pandas columns uses concat, which takes a Python list as its parameter:

# Python
pd.concat([r.df.loc[:, 'species'], r.df.loc[:, 'year']], axis = "columns").head(5)
##   species    year
## 0  Adelie  2007.0
## 1  Adelie  2007.0
## 2  Adelie  2007.0
## 3  Adelie  2007.0
## 4  Adelie  2007.0

And that’s it for the basics!

Of course, there are a number of intricacies and special cases here, but by and large this should serve as a useful list of examples to get you started in pandas coming from dplyr, and perhaps also vice versa. Future installments in this series will go beyond dplyr, but likely also touch back on some aspect of it that are easy to get lost in translation.

More info:

I have been planning to start this series for several months now; since I’m an R user in a Python-dominated team and often use both languages om any given day. What finally pushed me to overcome my laziness was an initiative by the ever prolific Benjamin Wolfe to gather resources for R users interested in Python. If you’re interested to contribute too, just drop by in the slack and say Hi.
Next up: most likely tidyr and/or stringr vs pandas. Watch this space.

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true; // s.defer = true; // s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: R on head spin - the Heads or Tails blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post R & Python Rosetta Stone: EDA with dplyr vs pandas first appeared on R-bloggers.

↧

It’s time to retire the “data scientist” label

November 4, 2020, 10:00 am

≫ Next: Single-source publishing for R users

≪ Previous: R & Python Rosetta Stone: EDA with dplyr vs pandas

[This article was first published on Econometrics and Free Software, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The “Data Scientist” label served its purpose; it allowed us to signal a transition happening in our profession from using only applied mathematical statistical methods to something else, which now also involves the use of a subset of software engineering practices. This transition was mentioned back in 2010 by Deborah Nolan (https://www.stat.berkeley.edu/~statcur/Preprints/ComputingCurric3.pdf), and this transition might now be complete. Version control systems, document generation from annotated source code (or even full reports generation à la rmarkdown), containers and build automation tools have now entered the toolbox of the run-of-the-mill statistician. Maybe not all of these tools, of course, it largely depends on what it is exactly you do, but certainly some of these. Same goes for software engineering practices. I have had the opportunity to work with some old-school statisticians (and still do), and the difference is clear; just like old school users of WYSIWYG editors like Word don’t use its “newest” features such as “Track changes” (and thus keep writing their text in different colors to signal which paragraphs are new), or the concept of versions of a document synced on Sharepoint (and thus keep multiple versions of the same document with different names) old school statisticians have not included the tools I mentioned before in their toolbox.

Now don’t get me wrong here; that is absolutely ok. We need and respect old school statisticians because they’ve been in the business of getting insights from data for longer than I’ve been alive. This blog post is not a jab at them because they don’t know how to use git (if you interpret it like that, that’s on you). Old school statisticians now have very senior positions and for many of them, their job does not involve getting their hands dirty on data anymore; most of them are now more like managers or mentors, and share their deep knowledge with their more junior team members. (Obviously there’s exceptions, when I say all old school statisticians do this or that, I don’t mean all of them, but most of them. Of course, I don’t have any evidence to back that up).

What this blog post is about is the label “Data Scientist” that gets used by these more junior team members and by companies that want to hire talented and motivated young people. This label, and the purported difference between a “Data Scientist” and “statistician” does not make any sense in 2020 anymore. (I know I’m beating a dead horse here, but this is my blog. I’ll blog about dead horses as much as I want thank you very much.)

Firstly, this label has always been confusing. “Data Scientist”… what does it even mean? The fact it took so long to find a definition, and that almost everyone working in the profession has a different one speaks volumes. Also, don’t all scientists use data? Data from experiments, from observational studies, from surveys, from the literature?

Secondly, I don’t believe that you can get a degree in statistics today without any exposition whatsoever to at least some of the tools I mentioned before. I really doubt that there’s people out there getting Master’s degrees in statistics without having ever touched these tools, or the unix command line. The degrees they’re going for might not focus a lot on these tools, true, but they certainly touch upon them. And of course, once they join a team at their first job, they’ll get more exposed to these tools and incorporate them in their day to day work. So, they’re not statisticians anymore? Their degree magically transformed into a data science degree?

But what about data science degrees? Are the students graduating with these degrees statisticians? I’d argue that yes, they are indeed statisticians; it’s just that they took a statistics degree that might have focused more than usual on these “new” practices/tools, and changed its name to “Data Science degree” for marketing purposes.

Anyways, the label “Data Scientist” is now completely defunct; as I mentioned in the very beginning, it served us well to signal that a transition was happening in the profession. I believe that this transition is now complete, or should be nearing its final stages. Also, this transition was not only about the tools used, but also about the deliverables. Statisticians now don’t only deliver tables, graphs and studies but more and more of them deliver products. This product can be a package implementing a bleeding edge statistical method for the profession as a whole, or it can be part of a piece of software that needs it to run (like your smartphone keyboard using a statistical model for word predictions). See this paper for an interesting exposition about how curricula and deliverables have evolved in the past two decades.

Currently, this label gets used by people that try to get insights from data. But we already have a word for them; statisticians. It’s just that the tools of the statistician have evolved over the past decade or so. Actually, I would perhaps even make another distinction; we should reserve the label of “statistician” to people that do statistics without ever touching any data. The other statisticians, the ones that get dirty wrestling in the mud with the data (they’re the pigs that like it from that famous quote) should be called “data janitors”. I’m not even joking; not only does that term already exist and gets used, I think it suits what we do perfectly. What do janitors do? They clean stuff and put things in order. We clean data and put it in order; meaning creating summary tables, visualizations, interactive applications, and models. Oh, and we do so (preferably) in a reproducible way.

Hope you enjoyed! If you found this blog post useful, you might want to follow me on twitter for blog post updates and buy me an espresso or paypal.me, or buy my ebook on Leanpub.

.bmc-button img{width: 27px !important;margin-bottom: 1px !important;box-shadow: none !important;border: none !important;vertical-align: middle !important;}.bmc-button{line-height: 36px !important;height:37px !important;text-decoration: none !important;display:inline-flex !important;color:#ffffff !important;background-color:#272b30 !important;border-radius: 3px !important;border: 1px solid transparent !important;padding: 1px 9px !important;font-size: 22px !important;letter-spacing:0.6px !important;box-shadow: 0px 1px 2px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 1px 2px 2px rgba(190, 190, 190, 0.5) !important;margin: 0 auto !important;font-family:'Cookie', cursive !important;-webkit-box-sizing: border-box !important;box-sizing: border-box !important;-o-transition: 0.3s all linear !important;-webkit-transition: 0.3s all linear !important;-moz-transition: 0.3s all linear !important;-ms-transition: 0.3s all linear !important;transition: 0.3s all linear !important;}.bmc-button:hover, .bmc-button:active, .bmc-button:focus {-webkit-box-shadow: 0px 1px 2px 2px rgba(190, 190, 190, 0.5) !important;text-decoration: none !important;box-shadow: 0px 1px 2px 2px rgba(190, 190, 190, 0.5) !important;opacity: 0.85 !important;color:#82518c !important;}

Buy me an Espresso

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: Econometrics and Free Software.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post It's time to retire the "data scientist" label first appeared on R-bloggers.

↧

Single-source publishing for R users

November 5, 2020, 10:00 am

≫ Next: Top 5 Best Articles on R for Business [October 2020]

≪ Previous: It’s time to retire the “data scientist” label

[This article was first published on Maëlle's R blog on Maëlle Salmon's personal website, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

A big part of my work includes putting content about R online, in blog posts and online books. I’m therefore very interested in the technical infrastructure that allows us R users to produce beautiful products out of R Markdown or Markdown source files. In this post I shall summarize my recent experiments around making HTML and PDF versions of books. Thanks to Julie Blanc’s inspiring post in French, I have learnt this is called single-source publishing. So let’s dive into the incredible machines allowing single-source publishing for R users.

What I want for writing books

I want technical tools that allow me to procrastinate whilst learning a ton. Just kidding, or not. My official criteria are:

For a book(let), I want a beautiful, usable website as well as a beautiful, usable PDF since some folks might prefer that for printing or offline usage.
I want to be able to write content in R Markdown or Markdown files, with some helpers for elements that Markdown doesn’t allow, for instance I might use a custom block to show an alert.
I want continuous deployment for my books, using git and a continuous integration service, so that every change in the source is reflected in the online versions of my book without my needing to think about it.
I would prefer to knit R Markdown files only once.
I don’t want to have to learn two ways to style things. ¹

Now, let’s dive into three workflows I’ve used or experimented with.

The standard way: bookdown gitbook and pdf_book

We R users are very lucky to be able to use bookdown for making books. You write content in Rmd files in a single folder, you order their filenames in a config file and choose an output format when knitting. There are many books out there built using bookdown, and often they’re knit twice so that there’s an HTML and PDF version. See for instance rOpenSci dev guide, that is deployed with GitHub Actions.

Advantages of bookdown include its widespread usage and stability, as well as its docs, both by its authors and users. For instance:

There is a bookdown book by bookdown maintainer Yihui Xie.
Emil Hvitfeldt wrote a post about deploying your bookdown project to Netlify with Github Actions.

A downside of bookdown for the procrastinators out there is, in my opinion, that if you want a PDF and HTML versions of a book, you have to knit it twice. Furthermore, if you introduce some fancy things such as a custom block, you will have to define it both with CSS and with LaTeX environments. Now, I have written a whole book with LaTeX so I used to partially know how to use it, but these days I’d prefer learning about CSS only as there’s already a lot to digest in there.

So let’s dive into two DIY ways of publishing a book in HTML and PDF.

Hugo and pagedjs-cli

In this proof-of-concept, content is stored in Markdown files, that could have been generated with hugodown. Actors are Hugo, a powerful static website generator that you get in the form of a binary, and pagedjs-cli, a Node package that prints HTML to PDF.

You can find a repo corresponding to this experiment on GitHub, and the resulting book on Netlify.

This proof-of-concept is based on the great hugo-book theme. My “work” was just to add a bit of setup magic (Netlify config, config/ dir).

The idea is to run two Hugo builds with different configurations. In the words of Hadley Wickham, “hugo provides a bewildering array of possible config locations”. In this proof-of-concept/incredible machine, this array of possibilities is key to success. The two configurations use different layout directories, layout directories being the templates telling Hugo where to put your words, e.g. every blog post on its own page, and a page showing summaries of all of them.

One Hugo build, hugo -d public --environment 'website', creates the HTML website with a link to “book.pdf” somewhere, in the public/ folder.
The second one, hugo -d public2 --environment 'pdf', creates a website with a single HTML page containing all the book content.
Then that single page is printed to PDF using pagedjs-cli and copied as “book.pdf” to public/. pagedjs-cli public2/all/index.html -o public/book.pdf"
The live website uses the content of the public/ folder so it has the website and the PDF.

Netlify can run all these commands

[build]publish = "/public"command = "hugo -d public --environment 'website' && hugo -d public2 --environment 'pdf' && pagedjs-cli public2/all/index.html -o public/book.pdf"

The dependency on pagedjs-cli is indicated with package.json (the DESCRIPTION of Node projects).

Now if the content lived in R Markdown files, we would need to use a service like GitHub Actions to knit things.

Advantages

The Markdown files that are the source of both versions of the book would be generated only once if you started from R Markdown.
Styling both versions rely on CSS. I haven’t added any print styles but if you used this workflow for real, then it’d be better to, of course.
Compared to creating a gitbook with bookdown, one could customize the HTML website more easily, and use any Hugo feature.

Further work

If I decided to take this further, that is! All the work would be aimed at making the PDF version better

Improve the Hugo layout to add various tables of contents (for the whole book, for chapters).
Actually work on the CSS print sheet for the PDF version.
Make Katex, Mermaid etc. work for the PDF version.

Related work

The work presented by Julie Blanc is a similar pipeline, where content lives in Markdown files rendered to two HTML versions by Jekyll, another static website generator. One HTML version is a website, the other one is a printable version thanks to Paged.js (the PDF is not pre-generated but you can get it from any modern browser). The big differences with my experiment are:

Julie Blanc’s work is not an experiment, it’s deployed to production!
Both versions of the book are absolutely stunning, there was actual CSS work involved.

bookdown::bs4_book(), xml2 and pagedjs-cli

In another experiment, I used bookdown’s brand-new HTML book format, bs4_book() by Hadley Wickham (only available in bookdown’s dev version). I used this one because I find it looks better than gitbook, and because it uses Bootswatch themes, you get to use divs such as a alerts. See bookdown’s docs about custom blocks.

The source is open. See the HTML version and PDF version. This time I took time to learn a bit about print CSS.

Here the steps are knit using the bs4_book() template for HTML and then for getting a PDF

tweaks of the HTML e.g. moving the chapter table of contents from the sidebar to the main content, and merging of all chapters using xml2 and XPath (so yeah you’re ditching LaTeX in favor of a workflow involving XPath, so modern ) in build.R. I could have made my life easier by using selectr for translating ” CSS3 Selectors and translates them to XPath 1.0 expressions. “.
some print CSS, see stylesheet. It turns out that following Paged.js docs about print CSS is not that hard. See my stylesheet, I changed the display of some things in print, and I used properties to add the page number and chapter title to the footer.
pagedjs-cli.

Locally I sourced build.R to see whether it works, but then I also wrote a GitHub Actions workflow that installs dependencies, runs the script, and deploys the docs/ folder to a gh-pages branch.

Note that Bootstrap provides classes you can add to elements for changing their display in print. So to not show the burger menu in print, you can either do as I did with a CSS property

@media print {  .btn.btn-outline-primary.d-lg-none.ml-2.mt-1 {    display: none;}}

or you’d add the .d-print-none to that element in your HTML template. If you can control said HTML template, that is.

Further work needed in this experiment is listed in GitHub issues. For instance footnotes aren’t tackled yet.

Conclusion

So, as an R user writing content in R Markdown, you have different possibilities for respecting the principles of single-source publishing. I would still recommend using bookdown or other official rmarkdown formats, knitting twice² if you need two versions, as it is the only stable and widely used solution, but I found it fun to explore other workflows. Both the DIY ways I presented use pagedjs-cli to generate a PDF out of HTML. I strongly recommend following the work done by the lovely Paged.js folks in the world of paged media, and to learn more about CSS for print. If you want to produce only paged content out of your R Markdown file, check out the pagedown package and its fantastic output formats.

If you too want to experiment with homemade pipelines, have fun customizing things. Now, of course, the possibility of customization might actually be a curse when you are trying to write something. But this was not a blog post about productivity.

At this point, let me thank Romain Lesur (one of pagedown authors) for his work promoting Paged.js in the R community.

Yes, I should probably re-style this very website. ︎
Potentially, knitr cache might make knitting a thing a second time very fast? ︎

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: Maëlle's R blog on Maëlle Salmon's personal website.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Single-source publishing for R users first appeared on R-bloggers.

↧

Little useless-useful R functions – Wacky Password generator

November 5, 2020, 5:10 pm

≫ Next: 4 R projects to form a core data analyst portfolio

≪ Previous: Top 5 Best Articles on R for Business [October 2020]

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Generating password is a hustle and I don’t want to go into the philosophy of passwords. Strong password, lengthy password, automatically generated password, etc. For me, they are all hard to remember, even harder to type (due to typos, different language setups, OS, software, etc.). In return, let’s have another useless R function, that will generate useless, – hard-to-use and impossible to remember – password, that will comply to all the standard rules (8 characters or more, one upper case, a kidney, a number, a special character, section from war & Peace, etc).

Typical “strong” password that human brain struggles to comprehend.

To make this password generation rather wacky, let me play with following characters:

There is less diversity in characters but nonetheless, the correct length and small caps, number, special characters and kidney – all comply with the regular quality assurance.

Wacky R function generates a set of ill generated password that would be even harder to type, and stupid to remember.

# Running on Linux/MacOSWackyPassword <- function(WP_length){  #charblock1 = c(176:178, 185: 188, 200:206)  charblock1 <- c("\u2591","\u2592","\u2593")  charblock2 = c(73,105,108,124,49,33)  numberblock3 <- sample(0:9, length(5),replace = TRUE)    pass = ""  Encoding(pass) <- "UTF-8"  ran2 <- floor(sample(1:WP_length/2))  ran1 <- floor(sample(1:WP_length/2))      while (nchar(pass) <= WP_length) {        res2 <- sample(charblock2, 100,replace = TRUE)        res2 <- rawToChar(as.raw(res2))        Encoding(res2) <- "UTF-8"        start2 <- sample(1:90,1)        pass <- paste0(pass,substr(res2,start2,start2+ran2),collapse="", sep= "")                        res1 <- sample(charblock1, 100,replace = TRUE)        Encoding(res1) <- "UTF-8"        start1 <- sample(20:70,1)        res <- paste0(res1, sep = "", collapse = "")        pass <- paste0(pass,substr(res,start1,start1+ran1), sep="", collapse = "")          }       cat(eval(substr(pass,1,WP_length)))}

Running the function is as simple as:

WackyPassword(18)

but the results can be as useless and wacky as you can imagine

!il1|I▓▒▓▓▓░!I|lIi

Good luck typing this and I buy a beer to every community member brave enough to use one

As always, useless code is available at Github.

Happy R-coding! And stay healthy my friends!

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Little useless-useful R functions – Wacky Password generator first appeared on R-bloggers.

↧

4 R projects to form a core data analyst portfolio

November 5, 2020, 10:00 pm

≫ Next: How isolate() can bite you in the butt

≪ Previous: Little useless-useful R functions – Wacky Password generator

[This article was first published on Articles - The Analyst Code, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Introduction

The job market for data analysts is large and highly competitive. Many companies, including companies not traditionally classified as “tech” or “coding” companies are looking to hire people with analytical coding experience. Yet, the numbers of applicants seems to be rising even faster. It’s a competitive market and you want your application to stand out.

Many jobs descriptions include lines like “Executes and advises on reporting needs and works cross-functionally to analyze data and make actionable recommendations at all levels” or “Utilizes advanced analytical and/or statistical ability to evaluate data and make judgments and recommendations“, “Experience in at least one computer programming language or analytical programming language (R, Python, SAS, etc.)” (emphasis added).

Notice that these job postings include two common themes (1) experience analyzing data (2) and experience providing recommendations. Your goal as an aspiring analyst is to be able to demonstrate experience in both of these domains. But how can you do this when you are applying for your first job in the field? Easy: spend some time building a core portfolio that shows the types of skills that recruiters want.

This article covers four projects that can form the core of your application portfolio. We recommend completing these prior to applying for jobs so that you can have demonstrable experience to include on your resume and discuss in your interviews. Be sure to create a GitHub repo for each project and link prominently to your GitHub profile on your resume.

I expect that building this portfolio will take at least one month of focused work.

Week 1: Exploratory data analysis
Week 2: Interactive Shiny dashboard
Week 3: Natural Language Processing
Week 4: Machine Learning

As you work through the projects, keep in mind that your goal is not just to gain experience analyzing data but also providing insightful recommendation. Even though this is practice, structure your output to include conclusions like “If I wanted to improve my exercise habits, the data show that…”. Prove to future recruiters that you have the skills they want to see.

Core portfolio projects

Exploratory data analysis

Project goal: Load a messy dataset into R, clean the data, create 4-5 charts or tables that have summary stats about your data, and create 4-5 charts or tables that provide analytic insight. Your output should be an html rmarkdown document.
Questions to answer: How big is my dataset? What do the top 10 items of data look like? What is the distribution of my variables of interest (mean/median, skew, outliers)? How does data change over time? What is the relationship between Variable A and Variable B? Why does Variable A behave in such and such a manner?
Resume skills practiced: R, data cleaning, data visualization
Recommended packages: dplyr, tidyr, ggplot2, kableExtra, others as needed
Examples:R for Data Science, Data Science Heroes
Data ideas to get you started:spotifyr, NYC Airbnb, college football games, Fitbit data

Interactive Shiny dashboard

Project goal: Similar to exploratory data analysis, load, clean, and analyze a dataset. Focus, however, on visualizing data in a Shiny dashboard. Your output should be either a Shiny dashboard with “server.R” and “ui.R” files, a flexdashboard with “runtime: shiny” or an html rmarkdown with “runtime: shiny.” Host your dashboard on shinyapps.io.
Questions to answer: Ask the same basic questions as the Exploratory Data Analysis project, but build a dashboard that allows users to generate the answers themselves. For example, instead of showing a histogram of Variable A, create a histogram chart that allows the user to select which variable to plot.
Resume skills practiced: R, Shiny, data visualization
Recommended packages: shiny, DT, others as needed
Examples: Covid-19 in the US (html document, source), hospital info (Shiny app, source)
Data ideas to get you started:Stock prices, World Bank data

Natural Language Processing (NLP) with R

Project goal: Replicate the analysis completed in this comprehensive example in order to draw analytical insight from a large body of unstructured text.
Questions to answer: What are the ten most frequent words in my corpus (excluding “stop words”)? What is the sentiment score of my corpus? How does sentiment change in each section in my corpus (e.g., chapter in my book)? What are the most common positive / negative words? What can tf-idf tell us about words unique to a part of my corpus (e.g., What words are most distinct to books A, B, and C)? Can I implement Latent Dirichlet allocation to separate my corpus into various topics?
Resume skills practiced: R, NLP, Text Mining, Machine Learning
Recommended packages: tidytext, topicmodels
Examples:Usenet text, Twitter data
Data ideas to get you started: Project Gutenberg library of books (website, gutenbergr package), Twitter API (rwteet package)

Machine learning with R

Project goal: Load a dataset, train a machine learning algorithm on part of the dataset, and use the rest of the dataset to test it. Create summary stats to evaluate the performance of your model.
Questions to answer: The types of questions you ask will vary tremendously based on the type of machine learning algorithm you want to implement. Try implementing one unsupervised and a handful of supervised algorithms. If you implement k-means (unsupervised), run your model with different seed values to find the lowest total within cluster sum of squares (SS), and plot total with SS against various numbers of clusters (k) to determine the optimal cluster count. Try to train several different supervised models (random forest, kNN, etc.) on the same dataset, and compare their results to pick the best one. Example here.
Resume skills practiced: R, Machine Learning
Recommended packages: caret
Examples:Machine Learning Mastery, MIT
Data ideas to get you started:UCI Machine Learning Repository, Hotel booking demand

To leave a comment for the author, please follow the link and comment on their blog: Articles - The Analyst Code.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post 4 R projects to form a core data analyst portfolio first appeared on R-bloggers.

↧

How isolate() can bite you in the butt

November 6, 2020, 2:00 am

≫ Next: time-lining the Trump presidency

≪ Previous: 4 R projects to form a core data analyst portfolio

[This article was first published on Dean Attali's R Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Debugging with Dean episode 2: Find out what mistake to avoid when using isolate() –

I’ll keep it short: The first episode of Debugging with Dean was a success, so I decided to keep going. In this video, I set out to see if an issue submitted to {shiny} was indeed a bug in shiny, and instead I found out a way that isolate() can break your app.

Watch on YouTube

As always, I would warmly welcome any comments, including constructive criticism so I can improve. If you like my videos, make sure to subscribe so that I’ll know to keep making more!

To leave a comment for the author, please follow the link and comment on their blog: Dean Attali's R Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post How isolate() can bite you in the butt first appeared on R-bloggers.

↧

time-lining the Trump presidency

November 5, 2020, 10:00 am

≫ Next: Warpspeed confidence what is credible? — Part 2

≪ Previous: How isolate() can bite you in the butt

[This article was first published on Jason Timm, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Introduction

A simple function from the uspols package – dubbed uspols_wiki_timeline()– that scrapes Wikipedia’s Timeline of the Donald Trump presidency. Facilitating, among other things, the ability to hover over data points in a time series to see what the President was up to on any given day over the course of the last four years.

library(tidyverse)
library(devtools)
devtools::install_github("jaytimm/uspols")
library(uspols)

A simple call to the function returns a table of the president’s daily happenings via Wikipedia – from 20 January 2017 to the most recent update. This is clearly not an official accounting, and Wikipedia is not gospel. But a super quick/easy way to get one’s bearings (in a maelstrom) and contextualize data points.

longs <- uspols::uspols_wiki_timeline()

Table structure is detailed some below. The folks at Wikipedia have delineated events (chronologically?) via bullet points per each day of Trump’s presidency, which have been enumerated here in the bullets column as independent rows. I have mostly done this to make text easier to read/access/etc. I am not sure how meaningful event distinctions (for a given day) actually are.

longs %>%
  select(-quarter:-daypres, -dow) %>%
  head() %>%
  DT::datatable(rownames = FALSE, 
                options = list(dom = 't',
                               pageLength = 6,
                               scrollX = TRUE))

{"x":{"filter":"none","data":[["2017-01-20","2017-01-20","2017-01-20","2017-01-20","2017-01-20","2017-01-20"],[1,2,3,4,5,6],["45th President Donald Trump and 48th Vice President Mike Pence take the Oath of Office.","President Trump proclaims a National Day of Patriotic Devotion.","According to contested reports in December 2017, while seated at Trump's inauguration speech, forthcoming National Security Advisor Michael Flynn texts a former business partner that Russian sanctions blocking a private Russian-backed plan to build nuclear plants in the Middle East will now be 'ripped up'.","State officials in Florida, Delaware and New York confirm that they have not received paperwork that the President has relinquished control over his companies despite earlier promises to do so.","President Trump issues Executive Order 13765 to scale back parts of the Affordable Care Act.","The Trump administration suspends an Obama administration cut to Federal Housing Authority mortgage insurance premiums."]],"container":"

\n \n \n

date<\/th>\n	bullet<\/th>\n	Events<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"dom":"t","pageLength":6,"scrollX":true,"columnDefs":[{"className":"dt-right","targets":1}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[6,10,25,50,100]}},"evals":[],"jsHooks":[]}

A simple use-case

summary <- longs %>%
  filter(!is.na(Events)) %>%
  mutate(tokens = tokenizers::count_words(Events)) %>%
  group_by(date) %>%
  mutate(daily_count = sum(tokens)) %>%
  slice(1) %>%
  ungroup()

For demonstration, we calculate total word counts of event descriptions per day from the time-line table. The plot below, then, summarizes these daily counts since inauguration. The plot itself is fairly meaningless, but the hover-action should be useful. For clarity purposes, only the first event for each day is included in the pop-up,.

dp <- summary %>%
  mutate(text = stringr::str_wrap(string = Events,
                                  width = 20,
                                  indent = 1,
                                  exdent = 1)) %>%
  
  plotly::plot_ly(x = ~date, 
                  y = ~daily_count,
                  text = ~text,
                  type = 'scatter',
                  mode = 'lines') %>%
  
  plotly::layout(#atitle = "Top 10 Drug Types", 
                 tooltip = c('Events'),
                 yaxis = list (title = "Daily event word count per Wikipedia"))

Example plot with Trump daily event on hover

To leave a comment for the author, please follow the link and comment on their blog: Jason Timm.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post time-lining the Trump presidency first appeared on R-bloggers.

↧

Warpspeed confidence what is credible? — Part 2

November 5, 2020, 10:00 am

≫ Next: Explainable Statistical/Machine Learning explainability using Kernel Ridge Regression surrogates

≪ Previous: time-lining the Trump presidency

[This article was first published on Posts on R Lover ! a programmer, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

In my last post I started exploring a basic statistical analysis of how confident we should feel about the effectiveness of a vaccine for COVID-19 (Operation Warpspeed). A reminder I’m not an epidemiologist, I don’t have any data for safety, and we’re still quite early in the Phase III trials – just approaching the point at which they can take a first “peek” at the data (which is doubleblind). You’ll be best served if you’ve already read the previous post.

The three key conditions that have to be (there are certainly other conditions as well):

53 confirmed covid-19 cases among the 30,000 participants
At least five severe cases of covid-19 among people who receive the placebo
That there be at least two months of follow-up on half the study participants.

Regulators will require that the vaccine be at least 50 percent effective before the manufacturer can seek emergency use authorization.

Last time

Last time we used a variety of frequentist tools and tests to model our potential results, including a handy table that maps effectiveness to the number of the 53 cases in the placebo and vaccine conditions. We want our vaccine to be at least 50% effective. We can operationalize that most simply with \[\frac{placebo~cases – vaccinated~cases}{placebo~cases}\] So for example if 19 of the 53 cases had received the vaccine our effectiveness is \[\frac{placebo~cases – vaccinated~cases}{placebo~cases} = (34-19)/34 = 0.4411765\] which is very close to what we need. So in the best possible scenario (remember we require that at least 5 people who got the vaccine contract COVID before we can run the numbers) our effectiveness is ~90%. We need the number of people who received the vaccine and still contracted COVID to be 17 or less.

After our classical frequentist efforts we began a journey of exploring bayesian tools and techniques. That’s where we’ll pick up in this post. First let’s load some essential libraries.

library(dplyr)library(ggplot2)library(kableExtra)theme_set(theme_bw())

Last post we ventured as far as building a simple model with a flat prior and treating the outcomes as simple binomial equations (a coin flip if you) will. We built the model in the JAGS language and fed it to run.jags for 40,000 MCMC iterations. We used tidybayes::tidy_draws to extract the chains/draws. That gave us estimates for the rate of infection for both those who received the placebo and those who received the actual vaccine. \[\frac{placebo~infection~rate – vaccinated~infection~rate}{placebo~infection~rate} * 100\] gives us what we really want to investigate which is the percentage difference in infection rates. We’re looking for (since we multiplied by 100) a number that is around 50. Which in essence is a 50% effectiveness rating. We kept it simple assuming both subject pools (the number who received placebo versus the real vaccine was an even split 15,000 each). A simple call to bayestestR::hdi gives us a graphical portrayal of what we want to know. Even with a simple 19/34 split we have mounting evidence that the vaccine is working. We’re not at all totally convinced it’s 50% or more. But we can take heart with the fact that 95% of our draws wind up in the region 7.7% to 71.1% effectiveness. N.B. yes we’d really like to see a split like 17/36 or even lower to boost our credibility, I’m using lower numbers simply to make it more “exciting”, we all have to wait on the real data.

## Calling 4 simulations using the parallel method...## All chains have finished## Simulation complete.  Reading coda files...## Coda files loaded successfully## Finished running the simulation

Okay now that you’re caught up let’s push our bayesian skills even farther.

More credibility for our results

The analysis last time was based upon a very nice functionbayes.prop.testcontained in a package called bayesian_first_aid. It was the first thing I stumbled upon that was a near equivalent to the frequentist tests we were using. Let’s investigate some other tools. Fair warning in advance they will give us similar results, but you’re reading this to learn right? Personally I find it satisfying to know that no matter which R tool I choose I get the same answer.

The next tool we’ll use is bayesAB. This package is on CRAN and recent. For those of you steeped in other disciplines AB testing generally seeks to ask questions about whether option A or option B is “better”. It is often used in marketing for example presenting users with two different versions of a website or ad and seeking to determine which option yields the most purchases. Hopefully the analog to our vaccine trial is readily apparent. You can read more about the package here.

bayesAB expects its input as a pair of vectors one for option A and one for option B. Instead of MCMC methods it uses random sampling from a limited set of distributions. That means it is faster to run but not suitable for very complex models like runjags, or brms or STAN. But for our relatively simple problem it is quite useful.

Let’s make two vectors labeled vaccinated and placebo with the right quantities (again assuming an equal 15,000 in each category). The authors have decided to calculate and plot (A – B) / B with no way to easily customize. Since we require placebo in the denominator that simply means by default we get the right value but with a negative sign. We’ll “correct” that in a bit. By default it samples 100,000 times. It has a custom summary and plot method we’ll display. As with our earlier example we’ll use a flat uninformative pretending we have no information about the effectiveness of the vaccine.

vaccinated <- c(rep(0, 15000 - 19), rep(1, 19))placebo <- c(rep(0, 15000 - 34), rep(1, 34))AB1 <-    bayesAB::bayesTest(B_data = placebo,                       A_data = vaccinated,                      priors = c('alpha' = 1, 'beta' = 1),                      distribution = 'bernoulli')summary(AB1)## Quantiles of posteriors for A and B:## ## $Probability## $Probability$A##           0%          25%          50%          75%         100% ## 0.0004575392 0.0011201359 0.0013089507 0.0015199810 0.0030369367 ## ## $Probability$B##           0%          25%          50%          75%         100% ## 0.0009776941 0.0020570086 0.0023106757 0.0025868629 0.0045641318 ## ## ## --------------------------------------------## ## P(A > B) by (0)%: ## ## $Probability## [1] 0.0199## ## --------------------------------------------## ## Credible Interval on (A - B) / B for interval length(s) (0.9) : ## ## $Probability##         5%        95% ## -0.6473618 -0.1075419 ## ## --------------------------------------------## ## Posterior Expected Loss for choosing B over A:## ## $Probability## [1] 0.8460082plot(AB1)

Again I’ll remind you the sign is reversed, but you can clearly see that except for the sign the results are comparable to our earlier results. The plots (3) are informative, displaying our prior, the A & B superimposed and our coveted % difference in infection rate.

The third plot explicitly calls out the fact that ~98% of the time our data shows the vaccine to be effective. With the peak (high density) occurring near 50% effective.

The plots are good, but I’m fussy. It used to be slightly complex to get the draws or chains from a bayesian model. It’s getting easier and easier. In this case a quick look at our AB1 object tells us they are in AB1$posteriors$Probability. This is a list with two vectors A & B. Let’s convert to a tibble and rename them back to what they actually are vaccinated = A, placebo = B. Now if we tidyr::pivot_longer(everything(), names_to = "group") we have a nice tibble we can easily send to ggplot and format our (my) way.

ggdist has a variety of nice functions for summarizing and displaying draws and chains. stat_halfeye not only plots it as a density curve but allows us to add per group credible intervals. If we leave the data “wide” we can compute our diff_rate and plot it. If we prefer a text table bayestestR::describe_posterior(ci = c(0.89, 0.95))

as_tibble(AB1$posteriors$Probability) %>%   rename(vaccinated = A, placebo = B) %>%   tidyr::pivot_longer(everything(), names_to = "group") %>%   ggplot(aes(x = value, fill = group)) +   ggdist::stat_halfeye(.width = c(0.89, 0.95),                        alpha = .8,                        slab_colour = "black",                        slab_size = .5) +   ggtitle("Infection rates for vaccine and placebo groups (N = 30,000)") +   xlab("Infection rate") +   theme(axis.title.y = element_blank(),        axis.text.y = element_blank(),        axis.ticks.y = element_blank(),        panel.grid.major.y = element_blank(),        panel.grid.minor.y = element_blank())

as_tibble(AB1$posteriors$Probability) %>%   rename(vaccinated = A, placebo = B) %>%   mutate(diff_rate = (placebo - vaccinated) / placebo * 100) %>%   ggplot(aes(x = diff_rate)) +   ggdist::stat_halfeye(.width = c(0.89, 0.95),                        alpha = .9,                        slab_fill = "dark green",                        slab_colour = "black",                        slab_size = .5) +   ggtitle("% Difference in Infection rates for vaccine and placebo groups (N = 30,000)") +   xlab("% Difference in Infection Rates") +   theme(axis.title.y = element_blank(),        axis.text.y = element_blank(),        axis.ticks.y = element_blank(),        panel.grid.major.y = element_blank(),        panel.grid.minor.y = element_blank())

as_tibble(AB1$posteriors$Probability) %>%   rename(vaccinated = A, placebo = B) %>%   mutate(diff_rate = (placebo - vaccinated) / placebo * 100) %>%   bayestestR::describe_posterior(ci = c(0.89, 0.95))## # Description of Posterior Distributions## ## Parameter  | Median |                      CI |      pd |        89% ROPE | % in ROPE## -------------------------------------------------------------------------------------## vaccinated |  0.001 | 89% CI [ 0.001,  0.002] | 100.00% | [-0.100, 0.100] |       100## vaccinated |  0.001 | 95% CI [ 0.001,  0.002] | 100.00% | [-0.100, 0.100] |       100## placebo    |  0.002 | 89% CI [ 0.002,  0.003] | 100.00% | [-0.100, 0.100] |       100## placebo    |  0.002 | 95% CI [ 0.002,  0.003] | 100.00% | [-0.100, 0.100] |       100## diff_rate  | 43.305 | 89% CI [16.664, 67.485] |  98.01% | [-0.100, 0.100] |         0## diff_rate  | 43.305 | 95% CI [ 8.100, 71.230] |  98.01% | [-0.100, 0.100] |         0

Actually let’s go ahead and save the results in a tibble named diff_rate_results and showcase the functions in bayestestR. Rather than explain each function and what they mean. I’m going to refer you to their website which does a great job of explaining it..

The bottom line is that the results are comparable to our two earlier explorations.

diff_rate_results <-   as_tibble(AB1$posteriors$Probability) %>%   rename(vaccinated = A, placebo = B) %>%   mutate(diff_rate = (placebo - vaccinated) / placebo * 100)bayestestR::describe_posterior(diff_rate_results$diff_rate)## # Description of Posterior Distributions## ## Parameter | Median |           89% CI |     pd |        89% ROPE | % in ROPE## ----------------------------------------------------------------------------## Posterior | 43.305 | [16.664, 67.485] | 98.01% | [-0.100, 0.100] |         0bayestestR::ci(diff_rate_results$diff_rate, method = "HDI", ci = c(0.005,0.025,0.5,0.975,0.995)) %>% plot

bayestestR::p_direction(diff_rate_results$diff_rate) %>% plot

bayestestR::p_significance(diff_rate_results$diff_rate) %>% plot

bayestestR::rope(diff_rate_results$diff_rate, ci = c(0.9, 0.95))  %>% plot

One final bayesian tool (this post)

Okay one more set of analyses using brms.

The brms package provides an interface to fit Bayesian generalized (non-)linear multivariate multilevel models using Stan, which is a C++ package for performing full Bayesian inference (see https://mc-stan.org/).

Quite frankly it is more than we need for our simple little analysis. A bit like using a sledge hammer as a fly-swatter. Besides the simple joy of learning about it however, there is one practical reason to use it. As our data grows and we inevitably want to conduct more sophisticated analysis factoring things in like gender, age and race. It is likely to become the tool of choice. I’m not going to dwell on it but it’s worthy to note that STAN uses different method for its markov chains.

First we need to put the data in the preferred format which is actually a simple one row tibble. Then we put things in a format brms can understand. A bit of an oddity here, because we are moving to a GLM model we’re not operating on the binomial distribution directly rather transforming back and forth through logit and inverse_logit`.

We’ll run ~50,000 across 5 cores and get our summary and diagnostic plots.

library(brms)d2 <-   tibble(placebo = 34,          vaccine = 19,          n1 = 15000,          n2 = 15000)model_1 <- bf(placebo | trials(n1) ~ 1)model_2 <- bf(vaccine | trials(n2) ~ 1)priors <- get_prior(data = d2,                    family = binomial(),                    formula = model_1 + model_2)priors$prior[2] <- "normal(0, 1)"priors$prior[3] <- "normal(0, 1)"priors##         prior     class coef group    resp dpar nlpar bound  source##        (flat) Intercept                                     default##  normal(0, 1) Intercept            placebo                  default##  normal(0, 1) Intercept            vaccine                  defaultfit2 <-   brm(data = d2, family = binomial(),       model_1 + model_2,       prior = priors,       iter = 10000,        warmup = 1000,        cores = 6,        chains = 6,       seed = 7)fit2##  Family: MV(binomial, binomial) ##   Links: mu = logit##          mu = logit ## Formula: placebo | trials(n1) ~ 1 ##          vaccine | trials(n2) ~ 1 ##    Data: d2 (Number of observations: 1) ## Samples: 6 chains, each with iter = 10000; warmup = 1000; thin = 1;##          total post-warmup samples = 54000## ## Population-Level Effects: ##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS## placebo_Intercept    -5.94      0.16    -6.26    -5.64 1.00    41344    33647## vaccine_Intercept    -6.40      0.20    -6.80    -6.03 1.00    48654    36148## ## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS## and Tail_ESS are effective sample size measures, and Rhat is the potential## scale reduction factor on split chains (at convergence, Rhat = 1).plot(fit2)

Everything looks good except for the funny scale we’re operating on. Rather than use tidybayes::tidy_draws to extract the results of our chains we’ll use posterior_samples(fit2, add_chain = FALSE). We’ll use transmute to convert back to our original scale and get names we prefer.

As we did earlier we’ll use tidyr::pivot_longer to convert to long format and pipe it through to ggplot to compare the two distributions. Then we’ll plot the effectiveness (diff_rate) estimate much as we have done before.

Finally we pipe this cleaned up data to bayestestR::describe_posterior() we get back a table that is a little easier to read. Focus on just the line for diff_rate. We can of course also pipe it into other plotting functions if you have favorites.

library(tidybayes)theme_set(theme_bw())post2 <- posterior_samples(fit2, add_chain = FALSE)post2 %>%   transmute(placebo = inv_logit_scaled(b_placebo_Intercept),             vaccine = inv_logit_scaled(b_vaccine_Intercept)) %>%   tidyr::pivot_longer(cols = everything()) %>%   ggplot(aes(x = value, y = name)) +   tidybayes::stat_halfeye(point_interval = mode_hdi,                            .width = c(.5, .95)) +   scale_y_discrete(NULL) +   scale_x_continuous(labels = scales::label_percent()) +   xlab("Percent Infected") +   theme(panel.grid = element_blank())

post2 %>%   mutate(placebo = inv_logit_scaled(b_placebo_Intercept),          vaccine = inv_logit_scaled(b_vaccine_Intercept)) %>%   mutate(diff_rate = (placebo - vaccine) / placebo) %>%   ggplot(aes(x = diff_rate)) +   geom_histogram(color = "grey92", fill = "grey67",                  size = .2, bins = 40) +   tidybayes::stat_pointinterval(aes(y = 0),                       point_interval = mode_hdi, .width = .95) +   scale_y_continuous(NULL, breaks = NULL) +   scale_x_continuous(labels = scales::label_percent()) +   xlab("Percent Difference in rate of infection") +   theme(panel.grid = element_blank())

post2 %>%   mutate(placebo = inv_logit_scaled(b_placebo_Intercept),          vaccine = inv_logit_scaled(b_vaccine_Intercept)) %>%   mutate(diff_rate = (placebo - vaccine) / placebo) %>%   bayestestR::describe_posterior()## # Description of Posterior Distributions## ## Parameter         |  Median |             89% CI |      pd |        89% ROPE | % in ROPE## ----------------------------------------------------------------------------------------## placebo_Intercept |  -5.934 | [ -6.191,  -5.686] | 100.00% | [-0.100, 0.100] |         0## vaccine_Intercept |  -6.393 | [ -6.710,  -6.086] | 100.00% | [-0.100, 0.100] |         0## lp__              | -46.880 | [-48.398, -46.174] | 100.00% | [-0.100, 0.100] |         0## placebo           |   0.003 | [  0.002,   0.003] | 100.00% | [-0.100, 0.100] |       100## vaccine           |   0.002 | [  0.001,   0.002] | 100.00% | [-0.100, 0.100] |       100## diff_rate         |   0.368 | [  0.101,   0.604] |  96.73% | [-0.100, 0.100] |         0preplot <-    post2 %>%   mutate(placebo = inv_logit_scaled(b_placebo_Intercept),          vaccine = inv_logit_scaled(b_vaccine_Intercept)) %>%   mutate(diff_rate = (placebo - vaccine) / placebo) %>%   select(diff_rate)    plot(bayestestR::hdi(preplot, ci = c(.89, .95, .99)))

Done

I’ve decided to make this a three parter. Please hang in there more bayesian “magic” follows soon. Probably next week.

Hope you enjoyed the post. Comments always welcomed. Especially please let me know if you actually use the tools and find them useful.

Extra credit for me for not expressing a political view at any point. Let the data speak and the counting continue.

Chuck

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: Posts on R Lover ! a programmer.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Warpspeed confidence what is credible? -- Part 2 first appeared on R-bloggers.

↧

Explainable Statistical/Machine Learning explainability using Kernel Ridge Regression surrogates

November 5, 2020, 10:00 am

≫ Next: Upcoming workshop: Sparkling Shiny App

≪ Previous: Warpspeed confidence what is credible? — Part 2

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

As announced last week, this week’s topic is Statistical/Machine Learning (ML) explainability using Kernel Ridge Regression (KRR) surrogates. The core idea underlying this type of ML explainability methods is to apply a second learning model to the predictions of the first so-called black-box model.

How am I envisaging it? Not by utilizing KRR as a learning model, but as a flexible (continuous and derivable) function of model covariates and predictions. For more details, you can read this pdf document on ResearchGate:

Statistical/Machine learning model explainability using Kernel RidgeRegression surrogates

Comments, suggestions are welcome as usual.

pres-image

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Explainable Statistical/Machine Learning explainability using Kernel Ridge Regression surrogates first appeared on R-bloggers.

↧

Upcoming workshop: Sparkling Shiny App

November 6, 2020, 2:25 am

≫ Next: Undersampling by Groups in R

≪ Previous: Explainable Statistical/Machine Learning explainability using Kernel Ridge Regression surrogates

[This article was first published on Mirai Solutions, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Mirai’s workshop series continues. Next in the pipeline is Sparkling Shiny App on 24th November.

In recent years, shiny has become increasingly popular in the R community. No wonder! Shiny is a master at empowering users to easily visualize and communicate their data science work. However, there is a huge difference between quickly drafting a user interface on top of your R functions, and properly building up an application that is going to serve from prototyping all the way to production. In fact, with increasing complexity and addition of features, a shiny app can easily become a little wild monster, and reactivity can quickly get out of wing.

How to make sure your app remains manageable, maintainable and understandable, while retaining its simple-to-develop attitude? The key lays in structuring your app and your development correctly from the beginning, as well as using the layout that best suits your app needs. In this workshop, Mirai’s Shiny experts will guide you through the state of the art approaches to shiny development — including best practices, modern development framework, and advanced layout & app structure. Join us, and learn how to make your shiny code shine.

For an introduction to professional shiny development, check out our article and have a look at our gallery for some practical examples of complex shiny apps.

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: Mirai Solutions.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Upcoming workshop: Sparkling Shiny App first appeared on R-bloggers.

↧

Undersampling by Groups in R

November 6, 2020, 4:34 am

≫ Next: future 1.20.1 – The Future Just Got a Bit Brighter

≪ Previous: Upcoming workshop: Sparkling Shiny App

[This article was first published on R – Predictive Hacks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

When we are dealing with unbalanced classes in Machine Learning projects there are many approaches that you can follow. Just to main some of them:

Undersampling: We try to reduce the observations from the majority class so that the final dataset to be balanced
Oversampling: We try to generate more observations from the minority class usually by replicating the samples from the minority class so that the final dataset to be balanced.
Synthetic Data Generation (SMOTE): We generate artificial data using bootstrapping and k-Nearest Neighbors algorithms.

Generate the Unbalanced Data

The scenario is that we are dealing with 3 email campaigns that have different CTRs and we want to apply undersampling to normalize the CTR by the campaign so that to avoid any skewness and biased when we will build the Machine Learning model. The hypothetical dataset is the following:

Campaign A: 5000 Observations with 10% CTR (approx)
Campaign B: 10000 Observations with 20% CTR (approx)
Campaign C: 1000 Observations with 30% CTR (approx)

Let’s try to generate this random sample in R.

library(tidyverse)

set.seed(5)
df = rbind(data.frame(Campaign = "A", Click = rbinom(n=5000, size=1, prob=0.1)),
           data.frame(Campaign = "B", Click = rbinom(n=10000, size=1, prob=0.2)),
           data.frame(Campaign = "C", Click = rbinom(n=1000, size=1, prob=0.3)))

head(df)

Output:

  Campaign Click
1        A     0
2        A     0
3        A     1
4        A     0
5        A     0
6        A     0

Let’s get the CTR by Campaign

df%>%group_by(Campaign)%>%
    summarise(CTR=mean(Click))

Output:

# A tibble: 3 x 2
  Campaign   CTR
      
1 A        0.106
2 B        0.198
3 C        0.302

As we can see the A campaign has 10.6% CTR, the B 19.8% and the C 30.2%. Let’s add also a random column called attribute which takes the values “X”, “Y”, “Z” since we will deal with datasets with more than two columns.

df$Attribute<-sample(c("X","Y", "Z"), size = dim(df)[1], replace = TRUE, prob = c(0.2, 0.6, 0.2))
head(df)

  Campaign Click Attribute
1        A     0         Y
2        A     0         Y
3        A     1         Z
4        A     0         Y
5        A     0         Z
6        A     0         Z

Now, our goal is to apply undersampling so that each campaign will have around 50% CTR

Undersampling by Group

We will use the map2 function from the purrr package which belongs to the tidyverse family:

campaign_summary <- df %>% group_by(Campaign)%>% summarize(rr=sum(Click)/n(), pos= sum(Click))


df_neg_sample<- df %>% filter(Click==0) %>%
  group_by(Campaign) %>% 
  nest() %>%             #group all data by campaign name
  ungroup() %>% 
  inner_join(campaign_summary, by="Campaign") 



sampled_df_neg<-df_neg_sample %>%
  mutate(samp = map2(data, pos, sample_n, replace = FALSE))  %>%# sample based on the campaing summary
  select(-data) %>%  #remove original nested data
  unnest(samp) %>% select(c(-"rr",-"pos"))


df_pos <- df %>% filter(Click==1) #positive samples
new_df <- rbind(df_pos,sampled_df_neg) #balanced set positive negative within each campaign

head(new_df)

  Campaign Click Attribute
1        A     1         Z
2        A     1         Y
3        A     1         Y
4        A     1         Y
5        A     1         Y
6        A     1         Y

tail(new_df)

     Campaign Click Attribute
5613        C     0         Y
5614        C     0         Y
5615        C     0         Z
5616        C     0         Y
5617        C     0         Y
5618        C     0         X

Let’s check if the new_df is balanced by campaign. We will group by campaign and we will show the CTR and the number of observations:

new_df%>%group_by(Campaign)%>%summarise(CTR=mean(Click), Observations=n())

  Campaign   CTR Observations
              
1 A          0.5         1064
2 B          0.5         3950
3 C          0.5          604

As we can see, we sacrificed a sample but we have a balanced number of classes for every campaign (50-50).

Undersampling by Group using the ROSE Package

We can use also the ROSE package. Below we will apply a for loop by campaign so that to get a balanced sample using the undersampling technique.

library(ROSE)


balanced_sample = NULL


for (c in unique(df$Campaign)) {
  tmp_df = df%>%filter(Campaign==c)
  tmp<-ovun.sample(Click ~ ., data = tmp_df, method = "under", p = 0.5, seed = 5)$data
  balanced_sample<-rbind(balanced_sample, tmp)
 }

Let’s check if the balanced_sample is actually balanced.

balanced_sample%>%group_by(Campaign)%>%summarise(CTR=mean(Click), Observations=n())

Output:

  Campaign   CTR Observations
              
1 A        0.504         1056
2 B        0.496         3978
3 C        0.510          592

Awesome. We showed two different approaches of how you can apply undersampling by group.

To leave a comment for the author, please follow the link and comment on their blog: R – Predictive Hacks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Undersampling by Groups in R first appeared on R-bloggers.

↧

future 1.20.1 – The Future Just Got a Bit Brighter

November 6, 2020, 7:00 am

≫ Next: About confidence intervals for the Biontech/Pfizer Covid-19 vaccine candidate

≪ Previous: Undersampling by Groups in R

[This article was first published on JottR on R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

future 1.20.1 is on CRAN. It adds some new features, deprecates old and unwanted behaviors, adds a couple of vignettes, and fixes a few bugs.

Interactive debugging

First out among the new features, and a long-running feature request, is the addition of argument split to plan(), which allows us to split, or “tee”, any output produced by futures.

The default is split = FALSE for which standard output and conditions are captured by the future and only relayed after the future has been resolved, i.e. the captured output is displayed and re-signaled on the main R session when value of the future is queried. This emulates what we experience in R when not using futures, e.g. we can add temporary print() and message() statements to our code for quick troubleshooting. You can read more about this in blog post ‘future 1.9.0 – Output from The Future’.

However, if we want to use debug() or browser() for interactive debugging, we quickly realize they’re not very useful because no output is visible, which is because also their output is captured by the future. This is where the new “split” feature comes to rescue. By using split = TRUE, the standard output and all non-error conditions are split (“tee:d”) on the worker’s end, while still being captured by the future to be relayed back to the main R session at a later time. This means that we can debug ‘sequential’ future interactively. Here is an illustration of using browser() for debugging a future:

> library(future)> plan(sequential, split = TRUE)> mysqrt <- function(x) { browser(); y <- sqrt(x); y }> f <- future(mysqrt(1:3))Called from: mysqrt(1:3)Browse[1]> str(x) int [1:3] 1 2 3Browse[1]> debug at #1: y <- sqrt(x)Browse[2]> debug at #1: yBrowse[2]> str(y) num [1:3] 1 1.41 1.73Browse[2]> y[1] <- 0Browse[2]> cont> v <- value(f)Called from: mysqrt(1:3) int [1:3] 1 2 3debug at #1: y <- sqrt(x)debug at #1: y num [1:3] 1 1.41 1.73> v[1] 0.000000 1.414214 1.732051

Comment: Note how the output produced while debugging is relayed also when value() is called. This is a somewhat unfortunate side effect from futures capturing all output produced while they are active.

Preserved logging on workers (e.g. future.batchtools)

The added support for split = TRUE also means that we can now preserve all output in any log files that might be produced on parallel workers. For example, if you use future.batchtools on a Slurm scheduler, you can use plan(future.batchtools::batchtools_slurm, split = TRUE) to make sure standard output, messages, warnings, etc. are ending up in the batchtools log files while still being relayed to the main R session at the end. This way we can inspect cluster jobs while they still run, among other things. Here is a proof-of-concept example using a ‘batchtools_local’ future:

> library(future.batchtools)> plan(batchtools_local, split = TRUE)> f <- future({ message("Hello world"); y <- 42; print(y); sqrt(y) })> v <- value(f)[1] 42Hello world> v[1] 6.480741> loggedOutput(f) [1] "### [bt]: This is batchtools v0.9.14"                                  [2] "### [bt]: Starting calculation of 1 jobs"                              [3] "### [bt]: Setting working directory to '/home/alice/repositories/future'" [4] "### [bt]: Memory measurement disabled"                                 [5] "### [bt]: Starting job [batchtools job.id=1]"                          [6] "### [bt]: Setting seed to 15794 ..."                                   [7] "Hello world"                                                           [8] "[1] 42"                                                                [9] ""                                                                     [10] "### [bt]: Job terminated successfully [batchtools job.id=1]"          [11] "### [bt]: Calculation finished!"

Without split = TRUE, we would not get lines 7 and 8 in the batchtools logs.

Near-live progress updates also from ‘multicore’ futures

Second out among the new features is ‘multicore’ futures, which now join ‘sequential’, ‘multisession’, and (local and remote) ‘cluster’ futures in the ability of relaying progress updates of progressr in a near-live fashion. This means that all of our most common parallelization backends support near-live progress updates. If this is the first time you hear of progressr, here’s an example of how it can be used in parallel processing:

library(future.apply)plan(multicore)library(progressr)handlers("progress")xs <- 1:5with_progress({  p <- progressor(along = xs)  y <- future_lapply(xs, function(x, ...) {    Sys.sleep(6.0-x)    p(sprintf("x=%g", x))    sqrt(x)  })})# [=================>------------------------------]  40% x=2

Note that the progress updates signaled by p(), updates the progress bar almost instantly, even if the parallel workers run on a remote machine.

Multisession futures agile to changes in R’s library path

Third out is ‘multisession’ futures. It now automatically inherits the package library path from the main R session. For instance, if you use .libPaths() to adjust your library path and then call plan(multisession), the multisession workers will see the same packages as the parent session. This change is based on a feature request related to RStudio Connect. With this update, it no longer matters which type of local futures you use – ‘sequential’, ‘multisession’, or ‘multicore’ – your future code has access to the same set of installed packages.

As a proof of concept, assume that we add tempdir() as a new folder to R’s library path;

> .libPaths(c(tempdir(), .libPaths()))> .libPaths()[1] "/tmp/alice/RtmpwLKdrG"[2] "/home/alice/R/x86_64-pc-linux-gnu-library/4.0-custom"   [3] "/home/alice/software/R-devel/tags/R-4-0-3/lib/R/library"

If we then launch a ‘multisession’ future, we find that it uses the same library path;

> library(future)> plan(multisession)> f <- future(.libPaths())> value(f)[1] "/tmp/alice/RtmpwLKdrG"[2] "/home/alice/R/x86_64-pc-linux-gnu-library/4.0-custom"   [3] "/home/alice/software/R-devel/tags/R-4-0-3/lib/R/library"

Best practices for package developers

I’ve added a vignette ‘Best Practices for Package Developers’, which hopefully provides some useful guidelines on how to write and validate future code so it will work on as many parallel backends as possible.

Saying goodbye to ‘multiprocess’ – but don’t worry …

Ok, lets discuss what is being removed. Using plan(multiprocess), which was just an alias for “plan(multicore) on Linux and macOS and plan(multisession) on MS Windows”, is now deprecated. If used, you will get a one-time warning:

> plan(multiprocess)Warning message:Strategy 'multiprocess' is deprecated in future (>= 1.20.0). Instead, explicitlyspecify either 'multisession' or 'multicore'. In the current R session,'multiprocess' equals 'multicore'.

I recommend that you use plan(multisession) as a replacement for plan(multiprocess). If you are on Linux or macOS, and are 100% sure that your code and all its dependencies is fork-safe, then you can also use plan(multicore).

Although ‘multiprocess’ was neat to use in documentation and examples, it was at the same time ambiguous, and it risked introducing a platform-dependent behavior to those examples. For instance, it could be that the parallel code worked only for users on Linux and macOS because some non-exportable globals were used. If a user or MS Windows tried the same code, they might have gotten run-time errors. Vice versa, it could also be that code works on MS Windows but not on Linux or macOS. Moreover, in future 1.13.0 (2019-05-08), support for ‘multicore’ futures was disabled when running R via RStudio. This was done because forked parallel processing was deemed unstable in RStudio. This meant that a user on macOS who used plan(multiprocess) would end up getting ‘multicore’ futures when running in the terminal while getting ‘multisession’ futures when running in RStudio. These types of platform-specific, environment-specific user experiences were confusing and complicates troubleshooting and communications, which is why it was decided to move away from ‘multiprocess’ in favor of explicitly specifying ‘multisession’ or ‘multicore’.

Saying goodbye to ‘local = FALSE’ – a good thing

In an effort of refining the Future API, the use of future(..., local = FALSE) is now deprecated. The only place where it is still supported, for backward compatible reason, is when using ‘cluster’ futures that are persistent, i.e. plan(cluster, ..., persistent = TRUE). If you use the latter, I recommended that you start thinking about moving away from using local = FALSE also in those cases. Although persistent = TRUE is rarely used, I am aware that some of you got use cases that require objects to remain on the parallel workers also after a future has been resolved. If you have such needs, please see future Issue #433, particularly the parts on “sticky globals”. Feel free to add your comments and suggestions for how we best could move forward on this. The long-term goals is to get rid of both local and persistent in order to harmonize the Future API across all future backends.

For recent bug fixes, please see the package NEWS.

What’s on the horizon?

There are still lots of things on the roadmap. In no specific order, here are the few things in the works:

Sticky globals for caching globals on workers. This will decrease the number of globals that need to be exported when launching futures. It addresses several related feature requests, e.g. future Issues #273, #339, #346, #431, and #437.
Ability to terminate futures (for backends supporting it), which opens up for the possibility of restarting failed futures and more. This is a frequently requested feature, e.g. Issues #93, #188, #205, #213, and #236.
Optional, zero-cost generic hook function. Having them in place opens up for adding a framework for doing time-and-memory profiling/benchmarking futures and their backends. Being able profile futures and their backends will help identify bottlenecks and improve the performance on some of our parallel backends, e.g. Issues #59, #142, #239, and #437.
Add support for global calling handlers in progressr. This is not specific to the future framework but since its closely related, I figured I mention this here too. A global calling handler for progress updates would remove the need for having to use with_progress() when monitoring progress. This would also help resolve the common problem where package developers want to provide progress updates without having to ask the user to use with_progress(), e.g. progressr Issues #78, #83, and #85.

That’s all for now – Happy futuring!

About confidence intervals for the Biontech/Pfizer Covid-19 vaccine candidate

November 10, 2020, 1:00 am

≫ Next: See Appsilon Presentations on Computer Vision and Scaling Shiny at Why R? 2020

≪ Previous: future 1.20.1 – The Future Just Got a Bit Brighter

[This article was first published on Economics and R - R posts, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Probably, many of you have read the positive news from the Biontech/Pfizer press release from November 9th:

“Vaccine candidate was found to be more than 90% effective in preventing COVID-19 in participants without evidence of prior SARS-CoV-2 infection in the first interim efficacy analysis”

Not being a biostatistician, I was curious how vaccine efficacy is exactly measured. Also, how does the confidence interval look like? Helpfully, Biontech and Pfizer also published a detailed study plan here.

The sample vaccine efficacy can be defined as

\[VE = 1-IRR = 1 – \frac{c_v/n_v}{c_p/n_p}\]

where $n_v$ and $n_p$ are the number of subjects that got a Covid-19 vaccine and a placebo, respectively, while $c_v$ and $c_p$ are the respective number of subjects that fell ill to the Covid-19 disease. IRR stands for incidence rate ratio and measures the ratio of the share of vaccinated subjects that got Covid-19 to the share in the placebo group.

The press release stated that so far 38955 subjects got the two doses of the vaccine or placebo of which 94 subjects fell ill with Covid-19. Furthermore, the study plan stated that the same number of subjects was assigned to the treatment and control group and let’s assume that also in the 38955 subjects analysed so far the ratio is almost equal. An efficacy of at least 90% then implies that from the 94 subjects with Covid-19 at most 8 could have been vaccinated.

The following code computes the IRR and vaccine efficacy assuming that indeed 8 vaccinated subjects got Covid-19.

n = 38955 # number of subjects
nv = round(n/2) # vaccinated
np = n-nv # got placebo

# number of covid cases
cv = 8
cp = 94-cv

# percentage of subjects in control group
# who got Covid-19
round(100*cp/np,2)

## [1] 0.44

# percentage of vaccinated subjects
# who got Covid-19
round(100*cv/nv,2)

## [1] 0.04

# incidence rate ratio in % in sample
IRR = (cv/nv)/(cp/np)
round(IRR*100,1)

## [1] 9.3

# vaccine efficacy in % in sample
VE = 1-IRR
round(VE*100,1)

## [1] 90.7

Assume for the moment this data came from a finished experiment. We could then compute an approximative 95% confidence interval for the vaccine efficacy e.g. using the following formula described in Hightower et. al. 1988

arv = cv/nv
arp = cp/np

# CI for IRR
ci.lower = exp(log(IRR) - 1.96 * sqrt((1-arv)/cv + (1-arp)/cp)) 
ci.upper = exp(log(IRR) + 1.96 * sqrt((1-arv)/cv + (1-arp)/cp)) 

IRR.ci = c(ci.lower, ci.upper)
round(100*IRR.ci,1)

## [1]  4.5 19.2

VE.ci = rev(1-IRR.ci)
round(100*VE.ci,1)

## [1] 80.8 95.5

This means we would be 95% confident that the vaccine reduces the risk of getting Covid-19 between 80.8% and 95.5%. As far as I understood, e.g. the function ciBinomial in the package gsDesign allows a more precise computation of the confidence interval:

library(gsDesign)
IRR.ci = ciBinomial(cv,cp,nv,np,scale = "RR")
VE.ci = rev(1-IRR.ci)
round(100*VE.ci,1)

##   upper lower
## 1  81.1  95.4

Given that it is only stated that the vaccine is more than 90% effective, the number of Covoid-19 cases may also have been lower than 8 subjects.

The next clean threshold for a press statement would probably be at least 95% effectiveness, which would be exceeded if only 4 vaccinated subjects had Covid-19. So it also seems well reasonable that only 5 vaccinated subjects had Covid-19. This would yield the following vaccine efficacy and confidence interval:

cv = 5; cp = 94-cv
VE = 1-(cv/nv)/(cp/np)
round(VE*100,1)

## [1] 94.4

IRR.ci = ciBinomial(cv,cp,nv,np,scale = "RR")
VE.ci = rev(1-IRR.ci)
round(100*VE.ci,1)

##   upper lower
## 1  86.6  97.7

Looks even better.

However, those confidence intervals assume a finished, non-adaptive experiment. Yet, the interim evaluations are triggered when the number of Covid-19 cases among the subjects exceeds certain thresholds. The press release states:

“After discussion with the FDA, the companies recently elected to drop the 32-case interim analysis and conduct the first interim analysis at a minimum of 62 cases. Upon the conclusion of those discussions, the evaluable case count reached 94 and the DMC performed its first analysis on all cases.”

“The trial is continuing to enroll and is expected to continue through the final analysis when a total of 164 confirmed COVID-19 cases have accrued.”

I am no expert, but possible the calculation of the confidence interval is not valid for such adaptive rules where the evaluation is triggered by the number of disease cases.

Indeed, Biontech and Pfizer state that they will the assess the precision of the estimated vaccine efficacy using a Bayesian framework with a particular prior distribution described on p. 102-103 of their study plan. Alas, I know very little of Bayesian analysis so I abstain from computing the posterior distribution given the data at hand.

But even absent the full-fledged Bayesian analysis, the numbers really look like very good news.

To leave a comment for the author, please follow the link and comment on their blog: Economics and R - R posts.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post About confidence intervals for the Biontech/Pfizer Covid-19 vaccine candidate first appeared on R-bloggers.

↧

See Appsilon Presentations on Computer Vision and Scaling Shiny at Why R? 2020

November 10, 2020, 1:40 am

≫ Next: Preview: satRday Columbus

≪ Previous: About confidence intervals for the Biontech/Pfizer Covid-19 vaccine candidate

[This article was first published on r – Appsilon | End to End Data Science Solutions, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Mark your calendars! Join Appsilon at the Why R? Conference on Thursday, November 12th @ 8PM UTC+1 / 2PM EST for two talks by Appsilonians. We will discuss wildlife preservation with computer vision and scaling Shiny apps on a budget. You can find the meetup link here and a direct link to the presentations on YouTube here.

Preserving Wildlife with Computer Vision

Appsilon Senior Data Scientist Jędrzej Świeżewski, PhD will discuss how Appsilon has managed to assist wildlife conservationists in their efforts in central Africa using computer vision. He will touch on the modeling, real-life implementation, and will also share a treat for R lovers. If you’re interested in the intersection of R and computer vision, see a previous talk by Jędrzej about how to make a computer vision model within an R environment here.

Scaling Shiny Dashboards on a Budget

Appsilon Infrastructure Engineer Damian Budelewski will show you how to prepare your Shiny dashboard for a high traffic load. He will also help you understand the costs and requirements of hosting Shiny apps in the cloud. He will illustrate by showing you an example of a Shiny app deployed on Shiny Server Open Source and hosted on AWS. To learn more about alternative approaches to scaling Shiny, go here.

Vote for the People’s Choice Award in Appsilon’s 2020 shiny.semantic PoContest! Vote here until Friday, November 10th.

Appsilon is Hiring

Appsilon is currently hiring for multiple open roles, including Senior External R Shiny Developers, a Frontend Engineer, a Senior Infrastructure Engineer, a Project Manager, and a Community Manager. Appsilon is a fully remote company with team members in Europe, the UK, Africa, and South America. We are global leaders in R Shiny seeking top talent to work with us in a highly collaborative and creative environment.

Article See Appsilon Presentations on Computer Vision and Scaling Shiny at Why R? 2020 comes from Appsilon | End to End Data Science Solutions.

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: r – Appsilon | End to End Data Science Solutions.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post See Appsilon Presentations on Computer Vision and Scaling Shiny at Why R? 2020 first appeared on R-bloggers.

↧

Preview: satRday Columbus

November 10, 2020, 9:15 am

≫ Next: Benford’s Law and Census Data

≪ Previous: See Appsilon Presentations on Computer Vision and Scaling Shiny at Why R? 2020

[This article was first published on George J. Mount, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Online conferences are getting to be a tough sell: the supply has exceeded the demand. It’s just so hard to make (and keep) the time for a conference when you’re sitting at home, and hard to know what’s quality. At the same time, the communities I belong to are some of the best things going for what I do. So, whither the gatherings?

satRday Columbus, at least for me — and I hope you too.

If you’re not familiar, satRdays are locally-organized events for the R programming community — similar to my beloved WordCamps for the WordPress community. Nobody’s working an angle here… this is a truly free and open-to-all conference, organized by volunteers and presented by volunteers, just like the R community at large. And like that product we all rally around, it will be better quality than what you might pay for.

Like every other conference — SatRdays have had to go online this year. But that didn’t stop organizers from taking on an inaugural SatRday Columbus, Ohio, Saturday November 14, 2020. This will be a free conference and I’ll be giving a 20-minute presentation at 2:50pm Eastern.

Register now for satRday Columbus

With this conference, there’s only one track and each session is 20-minutes. I like this idea of a shared experience for all members, and 20 minutes is a good length for everyone to get an “appetizer” for different topics, regardless of their exact interest or ability level.

I wanted to get the word out about the event in this post and also provide a sneak preview of my presentation.

R-Powered Excel

The sessions toward the end of the day are on the theme of “Communication,” and that’s where my “R-Powered Excel” presentation comes in. In this talk I’ll share some tips for automating the production of Excel workbooks in R. This approach pairs R’s knack for reproducible statistical analysis with the end-user interaction and accessibity that makes Excel so prevalent.

Attendees will learn the basic workflow of the openxlsx package, and they may also pick up some R riffs on building visualizations, iteration and more. I was particularly interested in showing how to map the “mental model” of Excel’s hierarchy (workbooks, worksheets, cells) into building and manipulating corresponding R objects. I also lean highly on the tidyverse to visualize and clean up regression outputs — then shoot over the results to Excel, graph and all.

Check out my preliminary repo here.

R + Juypter + Binder … just because

And now, let’s geek out on presentation media — Just to be different and see if I can, I decided to build the presentation in a Jupyter Notebook. You can view the presentation at the repo.

If you’ve not run R in Jupyter… it’s OK. It’s probably best-suited for data visualizations… lots of screen real estate. Working in RStudio for R is hard to beat, but this is a solid alternative. You can set this up by downloading the IRkernel package from CRAN and following these install steps. Jupyter comes installed with the Anaconda distribution of Python.

Another thing I did just to see if I could was connect the repo to Binder. This lets anyone run the repo interactively in Jupyter right from the web — i.e., no downloads required. I’d used Binder with Python but never with R. Turns out you can do it by leveraging Microsoft’s R Application Network (MRAN): just include a runtime.txt file with the MRAN snapshot you want, include any packages to install with an install.R file, and you’re off. Full instructions are here.

Running R on a Jupyter notebook with no installation thanks to Binder? Yeah, you can do that.

One thing I wanted to try which couldn’t quite pull off was presenting this notebook as a RISE presentation. With RISE, it’s required to change a setting in the package configurations to enable scrolling on longer slides, and I wasn’t sure how to do that while leveraging Binder’s image. It was possible, however, to use Python in the R notebook with the help of the reticulate package. If anybody’s been able to run RISE with R on Binder, you know where to find me! (That’d be the comments… or here.)

See you SatRday

I’m sure there will be just as many surprises and delights in each of the other 20-minute presentations – and they’re on everything from data journalism to supercomputing. Even if it’s not speaking your language… hey, it’s 20 minutes! Take a coffee break in your kitchen and come back to your office — the joy (and temptation) of online conferences.

So register now, bookmark the repo, and I’ll see you digitally on the 14th.

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: George J. Mount.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Preview: satRday Columbus first appeared on R-bloggers.

↧

Benford’s Law and Census Data

November 10, 2020, 10:00 am

≫ Next: lmDiallel: a new R package to fit diallel models. Introduction

≪ Previous: Preview: satRday Columbus

[This article was first published on R & Census, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Benford’s law attracted lots of attention after the 2020 election, as some people attempted to use this law to question the integrity of the election. Benford’s law states that the frequency of the first digit of a large set of numerical data follows this rule: 30.1% are 1s, 17.6% are 2s, 12.5% are 3s, …, 4.6% are 9s. Therefore it can be used to detect data manipulation that leads to the violaton of the law.

I am not going to repudiate those “findings” in the election data but rather provide some examples using census data to show when this law holds and when you should not expect it to work. It is well known that this law works best when the data spread several orders of magnitude and not so well when the numbers are in a narrow range. We will verify this using the population data from the 2010 decennial census, which contains the voting district data.

Counties: The population of the 3221 counties varies from a few hundreds to several millions so we expect the distribution of the first digits is a good match to the Benford’s law. It is true as shown in the figure below (red points and line show the values for Benford’s law).

Cities: We have the population of 29494 cities (and places) and the distribution of the first digit (almost) perfectly follows the Benford’s law.

Census tracts: We have seen two good examples. Here we will see examples that do not follow the Benford’s law. The first one is the population in census tracts. Most census tracts have a few thousands people. For the 2756 census tract in Michigan, the most frequent leading digit is 3 instead of 1.

Voting districts population does not obey the Benford’s law either. Here we show the data for two cities, Chicago IL and Milwaukee WI. What can you tell from the distribution of the first digit? Do not expect the Benford’s law to bring you any excitement.

Code:

library(totalcensus)library(ggplot2)library(scales)library(grid)library(gridExtra)library(data.table)library(magrittr)library(stringr)df_benford <- data.table(    first_digit = 1:9,    Freq = log10(1 + 1 / (1:9)))plot_benford <- function(x, title = ''){    first_digit <- x %>%        abs() %>%        as.character() %>%        str_sub(1, 1) %>%        as.integer()    count <- (table(first_digit) / length(first_digit)) %>%        as.data.frame()    count$first_digit <- as.integer(as.character(count$first_digit))    count %>% ggplot(aes(first_digit, 100 * Freq)) +        geom_col(fill = 'grey70') +        geom_point(data = df_benford, color = 'red') +        geom_line(data = df_benford, color = 'red') +        scale_x_continuous(breaks = 1:9, minor_breaks = NULL) +        labs(title = title,             x = 'First Digit',             y = 'Frequency (%)') +        theme_bw()}plot_hist <- function(x, title = ''){    ggplot() +        geom_histogram(aes(x), alpha = 0.3, bins = 100) +        scale_x_continuous(trans = 'log10', labels = comma, minor_breaks = NULL) +        labs(title = title,             x = 'Population',             y = 'Count') +        theme_bw()}plot_together <- function(x, title = ''){    g1 <- plot_hist(x)    g2 <- plot_benford(x)    grid.arrange(g1, g2, nrow = 1,                 top = textGrob(title,                                gp = gpar(fontsize = 16)))}    # counties                county <- read_decennial(2010, "US", summary_level = 'county') %>%    .[population != 0, population]plot_together(county, 'All counties in US. N = 3221.')# cities and placesplace <- read_decennial(2010, "US", summary_level = 'place') %>%    .[population != 0, population]plot_together(place, 'All cities (places) in US. N = 29494.')# centus tractstract <- read_decennial(2010, "MI", summary_level = 'tract') %>%    .[population != 0, population]plot_together(tract, 'All census tracts in Michigan. N = 2756.')# voting districtschicago_vtd <- read_decennial(2010, 'IL',                        geo_headers = c('PLACE', 'VTD'),                        summary_level = 'block') %>%    .[PLACE == '14000'] %>% # select Chicago    .[population != 0] %>%    .[, .(popul = sum(population)), VTD] %>%    .[, popul]plot_together(chicago_vtd, 'All voting districts in Chicago, IL. N = 2572.')milwaukee_vtd <- read_decennial(2010, 'WI',                              geo_headers = c('PLACE', 'VTD'),                              summary_level = 'block') %>%    .[PLACE == '53000'] %>% # select Milwaukee    .[population != 0] %>%    .[, .(popul = sum(population)), VTD] %>%    .[, popul]plot_together(milwaukee_vtd, 'All voting districts in Milwaukee, WI. N = 315.')

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: R & Census.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post Benford's Law and Census Data first appeared on R-bloggers.

↧

lmDiallel: a new R package to fit diallel models. Introduction

November 10, 2020, 10:00 am

≫ Next: YAPOEH! (Yet another post on error handling)

≪ Previous: Benford’s Law and Census Data

[This article was first published on R on The broken bridge between biologists and statisticians, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Together with some colleagues from the plant breeding group, we have just published a new paper, where we presented a bunch of R functions to analyse the data from diallel experiments. The paper is titled ‘Linear models for diallel crosses: a review with R functions’ and it is published in the ‘Theoretical and Applied Genetics’ Journal. If you are interested, you can take a look here at this link.

Diallel experiments are based on a set of possible crosses between some homozygous (inbred) lines. For example, if we have the male lines A, B and C and the female lines A, B and C (same lines used, alternatively, as male and female), we would have the following selfed parents: AA, BB and CC and the following crosses: AB, AC, BC. In some instances, we might also have the reciprocals BA, CA and CB. Selfed parents and crosses are compared on a Randomised Complete Block Design, usually replicated across seasons and/or locations.

For these diallel experiments, six main diallel models are available in literature, to quantify genetic effects, such as general combining ability (GCA), specific combining ability (SCA), reciprocal (maternal) effects and heterosis. If you are an expert in plant breeding, you do not need any other explanation; if you are not an expert, well… you are like me: we only need to know that these effects are determined as linear combinations of means for crosses, means for selfed parents and reciprocals. However, as I recently discovered, fitting diallel models to experimental data from diallel experiments is a relevant task for plant breeders.

When I started dealing with diallel models, I was very surprised by the fact that they are often presented as separate entities, to be fitted by using specialised software; indeed, to the eyes of a biostatistician, it would appear that all diallel models are only different parameterisations of the same general linear model (Mohring et al., 2011). Therefore, it seemed to me very strange that we could not fit diallel models by simply using the lm() function in R and related platform.

A deeper diving in this subject showed me that the main implementation problem was that certain effects, such as the GCA effect, require the definition of unconventional design matrices, which were not yet available in R. Indeed, the packages ‘asreml-R’ and ‘sommer’ permit, e.g., the overlay of design matrices (function and() in ‘asreml’ and overlay() in ‘sommer’), which is useful to code GCA effects, but none of the two packages played well with the lm() function in R. Therefore, together with Niccolò and Luigi, we decided to enhance the model.matrix() function in R, building a handful of new R functions, aimed at producing the correct design matrices for all types of diallel models. All these functions are available within the ‘lmDiallel’ package, which is available on gitHub; it can be installed by using the ‘install_github()’ function, as available in the ‘devtools’ package. Therefore, if necessary, install this package first. The code is as follows:

install.packages("devtools") # Only at first instance
library(devtools)
install_github("OnofriAndreaPG/lmDiallel")

The core functions for ‘lmDiallel’ are named after the corresponding genetic effects, i.e.: GCA() (general combining ability), tSCA() (total Specific Combining Ability), RGCA() (reciprocal general combining ability), RSCA() (reciprocal specific combining ability), REC() (RECiprocal effects = RGCA + RSCA), DD() (Dominance Deviation), MDD() (Mean Dominance Deviation), H.BAR() (Average Heterosis), Hi() (Average hetorosis for one parent), VEi() (Variety Effect), SP() (effect of Selfed Parents) and GCAC() (GCA for parents in their crosses). The usage of these functions is very simple. For example, let’s assume that we have the two variables ‘Par1’ and ‘Par2’ in a dataset, to represent the two parental lines (father and mother); the GCA effect is coded as:

GCA(Par1, Par2)

while the SCA effect is coded as:

SCA(Par1, Par2)

By using these R functions as building blocks, we can fit all diallel models inside the lm() and lme() functions. For example, the following line of code fits a diallel model containing the GCA and SCA effects, to the data contained in the ‘df’ dataframe:

lm(yield ~ GCA(Par1, Par2) + SCA(Par1, Par2), data = df)

Similarly, the effect of reciprocals and random blocks can be introduced by the following code:

lme(yield ~ GCA(Par1, Par2) + SCA(Par1, Par2) +
            REC(Par1, Par2),
            random = ~1|Block, data = df)

The model building process outlined above is clearly rooted in the frame of general linear models, although we recognise that plant breeders usually refer to certain relevant parameterisations of diallel models by using the name of the authors. In this respect, it is very common to use the terms “HAYMAN1”, “GRIFFING1”, “GRIFFING2”, “HAYMAN2”, “GE2” and “GE3” to refer to the main six diallel models available in literature (see Hayman, 1954; Griffing, 1956; Gardner and Eberhart, 1966). Although these models can be built and fit by using the above method, we thought it might be useful to simplify the whole process. For this reason, we also built a wrapper function named lm.diallel(), which can be used in the very same fashion as lm(). The syntax is:

lm.diallel(formula, Block, Env, data, fct)

where ‘formula’ uses the regular R syntax to specify the response variable and the two variables for parentals (e.g., Yield ~ Par1 + Par2). The two arguments ‘Block’ and ‘Env’ are used to specify optional variables, coding for blocks and environments, respectively. The argument ‘data’ is a ‘dataframe’ where to look for explanatory variables. Finally, ‘fct’ is a string variable coding for the selected model, i.e. “HAYMAN1”, “GRIFFING1”, “GRIFFING2”, “HAYMAN2”, “GE2”, “GE3”, according to the existing literature.

We have also built the summary(), vcov(),anova() and predict() methods for ‘lm.diallel’ objects, in order to obey to some peculiar aspects of diallel models.

In our paper (‘Linear models for diallel crosses: a review with R functions’) we have reviewed diallel models and gave examples on how they can be fitted with our new package ‘lmDiallel’. We have also shown how the facilities we provide can be used to fit random effects diallel models with ‘jags’. We intend to provide a more lengthy documentation for our package in a coming series of posts; thus, if you are interested, please, stay tuned.

I believe that increasing the usability of existing packages that have gained a wide popularity may be an advantageous programming strategy, compared to the usual strategy of building brand new platforms. From the point of view of the developer, it is efficient, as it requires a minor programming effort. From the point of view of the users (professionals, technicians and students), it is handy to be put in the conditions of making statistical analyses, without the need of learning new softwares and/or languages and/or syntaxes. Due to its open-source nature, the R environment is often overwhelming for users, that are confused by the extremely wide availability of alternative methods to perform the same task. In this regard, a programming strategy aimed at supporting some existing reference platforms might help build a more comfortable environment for statistical analyses.

Thanks for reading and, please, stay tuned! If you have comments, please, drop me a line at the email address below. Best wishes,

Andrea Onofri Department of Agricultural, Food and Environmental Sciences University of Perugia (Italy) Borgo XX Giugno 74 I-06121 – PERUGIA andrea.onofri@unipg.it

References

Covarrubias-Pazaran G (2016) Genome-assisted prediction of quantitative traits using the R package sommer. PLoS ONE 11:e0156744.
Gardner CO, Eberhart SA (1966) Analysis and interpretation of the variety cross diallel and related populations. Biometrics 22:439–452.
Gilmoure A, Gogel BJ, Cullis BR, Whelam SJ, Thompson R (2015) ASReml user guide release 4.1 structural specification. VSN International Ltd, Hemel Hempstead, HP1 1ES, UK
Griffing B (1956) Concept of general and specific combining ability in relation to diallel crossing systems. Aust J Biol Sci 9:463–493
Möhring J, Melchinger AE, Piepho HP (2011) REML-based diallel analysis. Crop Sci 51:470–478.

To leave a comment for the author, please follow the link and comment on their blog: R on The broken bridge between biologists and statisticians.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post lmDiallel: a new R package to fit diallel models. Introduction first appeared on R-bloggers.

↧

YAPOEH! (Yet another post on error handling)

November 10, 2020, 10:00 am

≫ Next: The Pfizer-Biontech Vaccine May Be A Lot More Effective Than You Think

≪ Previous: lmDiallel: a new R package to fit diallel models. Introduction

[This article was first published on R | Adi Sarid - Personal Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Error catching can be hard to catch at times (no pun intended). If you’re not used to error handling, this short post might help you do it elegantly.

There are many posts about error handling in R (and in fact the examples in the purrr package documentation are not bad either). In this sense, this post is not original.

However, I do demonstrate two approaches: both the base-R approach (tryCatch) and the purrr approach (safely and possibly). The post contains a concise summary of the two methods, with a very simple example.

In this post I’m assuming you’re familiar with the basic concepts of functional programming.

A simple example for a function with errors

Let’s analyze a very simple function which divides number by 2. The function should return an error if its input is not an actual number, otherwise it will return number/2. This function looks like this:

divide2 <- function(number){    # make sure the input is numeric (otherwise yield an error)  if (!is.numeric(number)) {    stop(paste(number, "is not a number!"))  }    # everything is fine, return the result  number/2}divide2(10)## [1] 5divide2(pi)## [1] 1.570796

But trying a string will yield an error:

divide2("foobar")Error in divide2("foobar") : foobar is not a number!

Where is the problem?

What happens if we have a dataframe (or a list, or any other object for that matter) and we want to try to divide numbers within that object? Invalid values might crash our attempt completely.

For example, this loops through two values (10 and $\pi$) and yields their division by 2.

library(purrr)map(list(10, pi), divide2)## [[1]]## [1] 5## ## [[2]]## [1] 1.570796

But the next version fails completely, because it tries to loop through “foobar” which cannot be divided.

map(list("foobar", 10, pi), divide2)Error in .f(.x[[i]], ...) : foobar is not a number!

No matter where we place the invalid value “foobar”, it will fail our code completely, and we get nothing.

The solution: a `safely`/`possibly` wrapper

Fortunately, there are very simple wrappers which can help us handle the errors elegantly. These are the two functions from the purrr package: safely and possibly.

We’ll first demonstrate the simpler version, possibly. It allows us to replace errors with a chosen value. Since we expect a number, let’s replace errors with NA_real_ (which is like saying an unknown value which is a real number).

possibly_divide2 <- possibly(divide2, otherwise = NA_real_, quiet = TRUE)

The quiet = TRUE argument is just so errors will not be printed during the loop. Now we are ready to try our error safe variation.

possibly_out <- map(list("foobar", 10, pi), possibly_divide2)possibly_out## [[1]]## [1] NA## ## [[2]]## [1] 5## ## [[3]]## [1] 1.570796

We can also use unlist to turn the output into a simple vector, i.e. unlist(possibly_out) will yield the vector NA, 5, 1.5707963.

The safely variation has some more strength into it, since it also provides the error messages. The output is slightly more complex though.

safely_divide2 <- safely(.f = divide2, otherwise = NA_real_, quiet = T)safely_out <- map(list("foobar", 10, pi), safely_divide2)safely_out## [[1]]## [[1]]$result## [1] NA## ## [[1]]$error## ## ## ## [[2]]## [[2]]$result## [1] 5## ## [[2]]$error## NULL## ## ## [[3]]## [[3]]$result## [1] 1.570796## ## [[3]]$error## NULL

Following the approach suggested in the documentation of safely (in the examples section), we can use transpose() and simplify_all() to arrange the output.

simply_safe <- safely_out %>%   transpose() %>%   simplify_all()simply_safe$result  # for the output values## [1]       NA 5.000000 1.570796simply_safe$error  # for the error messages## [[1]]## ## ## [[2]]## NULL## ## [[3]]## NULL

For comparison’s sake, how would you `tryCatch` it?

The function tryCatch is a base-R approach for error handling. The concept is similar but the syntax is different. Here is an example of how to build a safe divide2 function with tryCatch:

try_divide2 <- function(number){  tryCatch(expr = {    divide2(number)  },   error = function(e){    NA_real_  }  )}map(list("foobar", pi, 10), try_divide2)## [[1]]## [1] NA## ## [[2]]## [1] 1.570796## ## [[3]]## [1] 5

As said - same output, different syntax. Choose your approach. As an appetizer, the same works with base-R functional programming as well. For example, with lapply it will look like this:

lapply(list("foobar", pi, 10), try_divide2)## [[1]]## [1] NA## ## [[2]]## [1] 1.570796## ## [[3]]## [1] 5

Bonus: a benchmark

Before going off on your merry, error handled way, I also provide a short comparison between safely, possibly, and tryCatch.

Let’s assume a data set with 20% errors. For simplicity, we’ll use a modified log instead of our divide2 (similar to the example provided in purrr’s documentation). It makes sense to fail log in an all numeric vector (yield an error if there is a negative value). Since a negative log is NaN (not an error but rather a warning) I’m creating an error_prone_log function.

library(tibble)library(dplyr)library(microbenchmark)library(ggplot2)error_prone_log <- function(x){  if (x < 0){    stop("Negative x")  }    log(x)}safe_log <- safely(error_prone_log, otherwise = NaN, quiet = T)possible_log <- possibly(error_prone_log, otherwise = NaN, quiet = T)tryCatch_log <- function(x){  tryCatch(expr = error_prone_log(x),           error = function(e){             NaN             })}set.seed(42)test_values <- sample(  c(runif(n = 80, min = 0, max = 100),    runif(n = 20, min = -100, max = 0)),  size = 100, replace = FALSE)bench_results <- microbenchmark(map(test_values, safe_log),                                map(test_values, possible_log),                                map(test_values, tryCatch_log),                                 times = 1000)gt::gt(summary(bench_results)) %>%   gt::fmt_number(columns = 2:7, decimals = 2)

html { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;}#zbltwciejw .gt_table { display: table; border-collapse: collapse; margin-left: auto; margin-right: auto; color: #333333; font-size: 16px; font-weight: normal; font-style: normal; background-color: #FFFFFF; width: auto; border-top-style: solid; border-top-width: 2px; border-top-color: #A8A8A8; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #A8A8A8; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3;}#zbltwciejw .gt_heading { background-color: #FFFFFF; text-align: center; border-bottom-color: #FFFFFF; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3;}#zbltwciejw .gt_title { color: #333333; font-size: 125%; font-weight: initial; padding-top: 4px; padding-bottom: 4px; border-bottom-color: #FFFFFF; border-bottom-width: 0;}#zbltwciejw .gt_subtitle { color: #333333; font-size: 85%; font-weight: initial; padding-top: 0; padding-bottom: 4px; border-top-color: #FFFFFF; border-top-width: 0;}#zbltwciejw .gt_bottom_border { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3;}#zbltwciejw .gt_col_headings { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3;}#zbltwciejw .gt_col_heading { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 6px; padding-left: 5px; padding-right: 5px; overflow-x: hidden;}#zbltwciejw .gt_column_spanner_outer { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; padding-top: 0; padding-bottom: 0; padding-left: 4px; padding-right: 4px;}#zbltwciejw .gt_column_spanner_outer:first-child { padding-left: 0;}#zbltwciejw .gt_column_spanner_outer:last-child { padding-right: 0;}#zbltwciejw .gt_column_spanner { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 6px; overflow-x: hidden; display: inline-block; width: 100%;}#zbltwciejw .gt_group_heading { padding: 8px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle;}#zbltwciejw .gt_empty_group_heading { padding: 0.5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: middle;}#zbltwciejw .gt_from_md > :first-child { margin-top: 0;}#zbltwciejw .gt_from_md > :last-child { margin-bottom: 0;}#zbltwciejw .gt_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; margin: 10px; border-top-style: solid; border-top-width: 1px; border-top-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; overflow-x: hidden;}#zbltwciejw .gt_stub { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 12px;}#zbltwciejw .gt_summary_row { color: #333333; background-color: #FFFFFF; text-transform: inherit; padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px;}#zbltwciejw .gt_first_summary_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3;}#zbltwciejw .gt_grand_summary_row { color: #333333; background-color: #FFFFFF; text-transform: inherit; padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px;}#zbltwciejw .gt_first_grand_summary_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; border-top-style: double; border-top-width: 6px; border-top-color: #D3D3D3;}#zbltwciejw .gt_striped { background-color: rgba(128, 128, 128, 0.05);}#zbltwciejw .gt_table_body { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3;}#zbltwciejw .gt_footnotes { color: #333333; background-color: #FFFFFF; border-bottom-style: none; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3;}#zbltwciejw .gt_footnote { margin: 0px; font-size: 90%; padding: 4px;}#zbltwciejw .gt_sourcenotes { color: #333333; background-color: #FFFFFF; border-bottom-style: none; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3;}#zbltwciejw .gt_sourcenote { font-size: 90%; padding: 4px;}#zbltwciejw .gt_left { text-align: left;}#zbltwciejw .gt_center { text-align: center;}#zbltwciejw .gt_right { text-align: right; font-variant-numeric: tabular-nums;}#zbltwciejw .gt_font_normal { font-weight: normal;}#zbltwciejw .gt_font_bold { font-weight: bold;}#zbltwciejw .gt_font_italic { font-style: italic;}#zbltwciejw .gt_super { font-size: 65%;}#zbltwciejw .gt_footnote_marks { font-style: italic; font-size: 65%;}

expr	min	lq	mean	median	uq	max	neval
map(test_values, safe_log)	2.89	3.28	3.50	3.34	3.42	11.94	1000
map(test_values, possible_log)	2.62	3.14	3.35	3.21	3.28	7.62	1000
map(test_values, tryCatch_log)	1.96	2.29	2.52	2.33	2.38	77.12	1000

autoplot(bench_results) +   ggtitle("Comparison of safely, possibly, and tryCatch") +   coord_flip(ylim = c(1.5, 15))

As can be seen from the benchmark’s results, there is no doubt about it: tryCatch is the fastest. The purrr functions safely and possibly take longer to run. In my opinion though they are slightly more convenient in terms of syntax.

In addition, the safely variation allows us to retrieve the error messages conveniently for later examination. Again, this capability comes with a price when compared to tryCatch, but it is roughly the same run time when compared to possibly.

Conclusion

When you have a long looping process which is prone to errors, for example a pricey web API call or a really large data set, you should aim to catch and handle errors gracefully instead of hoping for the best.

If you really need to be efficient, it’s probably worth while to tryCatch, and if your looking more for ease of use and code readability you should use possibly. In case you need to examine the error messages more thoroughly, use safely.

Good luck hunting down the errors!

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: R | Adi Sarid - Personal Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post YAPOEH! (Yet another post on error handling) first appeared on R-bloggers.

↧

The Pfizer-Biontech Vaccine May Be A Lot More Effective Than You Think

November 10, 2020, 11:51 am

≫ Next: A look at Biontech/Pfizer’s Bayesian analysis of their Covid-19 vaccine trial

≪ Previous: YAPOEH! (Yet another post on error handling)

[This article was first published on Fells Stats, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

tl;dr; The point estimate for vaccine effectiveness may be 97%, which is a lot higher than 90%

Yesterday an announcement went out that the SARS-CoV-2 vaccine candidate developed by Pfizer and Biontech was determined to be effective during an interim analysis. This is fantastic news. Perhaps the best news of the year. It is however another example of science via press release. There is very limited information contained in the press release and one can only wonder why they couldn’t take the time to write up a two page report for the scientific community.

That said, we can draw some inferences from the release that may help put this in context. From the press release we know that a total of 94 COVID-19 cases were recorded.

“Upon the conclusion of those discussions, the evaluable case count reached 94 and the DMC performed its first analysis on all cases. “

However, we don’t know how many of these come from the control group, and how many come from the treatment group. We also don’t know how many total subjects are in the treatment and control arms. We do get two important quotes regarding efficacy.

“Vaccine candidate was found to be more than 90% effective in preventing COVID-19 in participants without evidence of prior SARS-CoV-2 infection in the first interim efficacy analysis
…
The case split between vaccinated individuals and those who received the placebo indicates a vaccine efficacy rate above 90%, at 7 days after the second dose.”

How should we interpret these? Was the observed rate of infection 90% lower in the treatment group, or are we to infer that the true (population parameter) efficacy is at least 90%? I would argue that the wording supports the later. If they were just providing a point estimate why express it as a bound? Why would they phrase it as “indicates a vaccine efficacy rate above 90%” if there was a reasonable probability that the actual vaccine efficacy rate is below 90%?

We can get some additional insight by looking at the study design. It specifies how the interim analysis is to be done. Specifically on pages 102-103, it calls for a Bayesian analysis using a beta binomial model with a weakly-informative prior.

To me, the most compatible statistical translation of their press release is that we are sure with 95% probability that the vaccine’s efficacy is greater than 90%. Why “95% probability?” Well, 95% probability intervals are standard for the medical literature if you are doing Bayesian analysis (deal with it), and 95% intervals with 2.5% probabilities on each tail are littered through the design document. They are going to the FDA with these claims, so they will likely stick to the standard evidentiary rules.

Assuming my interpretation is correct, let’s back out how many cases were in the treatment group. Conditional on the total number of infections, the number of infections in the treatment group is distributed binomially. We apply the beta prior to this posterior and then transform our inferences from the binomial proportion to vaccine effectiveness. Vaccine effectiveness is one minus the infection rate ratio between the two groups, and the rate ratio is related to the binomial proportion as the odds.

^?View Code R

> # reference: https://pfe-pfizercom-d8-prod.s3.amazonaws.com/2020-09/C4591001_Clinical_Protocol.pdf> > # prior interval (matches prior interval on page 103)> qbeta(c(.025,.975),.700102,1)[1] 0.005148448 0.964483043> > > # posterior> cases_treatment <- 3> cases_control <- 94 - cases_treatment> theta_ci <- qbeta(c(.025,.975),cases_treatment+.700102,cases_control+1)> rate_ratio_ci <- theta_ci / (1-theta_ci)> > # effectiveness> 100 * (1 - rate_ratio_ci)[1] 98.98688 90.68447> library(ggplot2)> xx <- (0:60)/500> yy <- sapply(xx, function(x) dbeta(x,cases_treatment+.700102,cases_control+1))> xx <- 100 * (1 - xx / (1 - xx))> ggplot() + +   geom_area(aes(x=xx,y=yy)) + +   theme_bw() + +   xlab("Vaccine Effectiveness") + +   ylab("Posterior Density")

The largest number of treatment cases that would have a lower bound greater than 90% is 3, corresponding to 91 cases in the control group. The estimated effectiveness of the vaccine is then 97% with a probability interval from 90.7% to 99.0%. So sure, the effectiveness could be 90% or so, but odds are that it is a lot higher as the posterior plot below shows.

To put this in perspective, consider the rates at which a 97% effective vaccine fails to provide protection, leading to an infection. A 90% effective vaccine has a 3.3 times higher failure rate, so if you vaccinated a population with a 90% effective vaccine and everyone was exposed you’d expect to see 3.3 times more infections compared to if you had used a 97% effective vaccine.

I do note that the analysis plan calls for sequential stopping rules that preserve type I error; however, I don’t believe that any reported statistics would be adjusted for that. Unlike frequentist intervals, Bayesian intervals are unchanged no matter how many interim analyses you do.

There is a lot we don’t know, and hopefully we will get more scientific clarity in the coming weeks. As it stands now, it seems like this vaccine has efficacy way above my baseline expectations, perhaps even in the 97% range or higher.

I could be wrong in my interpretation of the press release, and they are in fact talking about the sample effectiveness rather than the true effectiveness. In that case, 8 of the 94 cases would have been in the treatment group, and the interval for the true effectiveness would be between 81.6% and 95.6%. The posterior distribution would look pretty darn good, but not quite as nice as the previous one.

It is important to have realistic expectations though. Efficacy is not the only metric that is important in determining how useful the vaccine is. Due to the fact that the study population has only been followed for months, we do not know how long the vaccine provides protection for. There is significant evidence of COVID-19 reinfection, so the expectation is that a vaccine will not provide permanent immunity. If the length of immunity is very short (e.g. 3 months), then it won’t be the silver bullet we are looking for. I’d be happy to see a year of immunity and ecstatic if it lasts two.

Additionally, there are the side effects. We’ll have to see what the results are from this trial, but in the phase II trial, something like 8% or 17% of subjects (I’m unsure of the dosage for the phase III) experienced a fever after their booster. It is likely that you’ll want to take the day after you get the second shot off work in case you don’t feel well. The rate of side effects may harm vaccine uptake.

var vglnk = {key: '949efb41171ac6ec1bf7f206d57e90b8'}; (function(d, t) { var s = d.createElement(t); s.type = 'text/javascript'; s.async = true;// s.defer = true;// s.src = '//cdn.viglink.com/api/vglnk.js'; s.src = 'https://www.r-bloggers.com/wp-content/uploads/2020/08/vglnk.js'; var r = d.getElementsByTagName(t)[0]; r.parentNode.insertBefore(s, r); }(document, 'script'));

To leave a comment for the author, please follow the link and comment on their blog: Fells Stats.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The post The Pfizer-Biontech Vaccine May Be A Lot More Effective Than You Think first appeared on R-bloggers.

↧

General overview

Subsetting rows and columns

Arrange and Sample

Group By and Summarise

Joining and binding

What I want for writing books

The standard way: bookdown gitbook and pdf_book

Hugo and pagedjs-cli

Advantages

Further work

Related work

bookdown::bs4_book(), xml2 and pagedjs-cli

Conclusion

No. 1: How to Visualize Time Series Data Tidy Forecasting in R

No. 2: Introducing Modeltime Ensemble Time Series Forecast Stacking

No. 3: How I Started My Data Science Business Throwback

No. 4: How to Automate Excel with R Video Tutorial

No. 5: How to Automate Exploratory Analysis Plots Code Tutorial

Finding it Difficult to Learn R for Business?

Introduction

Core portfolio projects

Exploratory data analysis

Interactive Shiny dashboard

Natural Language Processing (NLP) with R

Machine learning with R

Introduction

A simple use-case

Example plot with Trump daily event on hover

Last time

More credibility for our results

One final bayesian tool (this post)

Done

Generate the Unbalanced Data

Undersampling by Group

Undersampling by Group using the ROSE Package

Interactive debugging

Preserved logging on workers (e.g. future.batchtools)

Near-live progress updates also from ‘multicore’ futures

Multisession futures agile to changes in R’s library path

Best practices for package developers

Saying goodbye to ‘multiprocess’ – but don’t worry …

Saying goodbye to ‘local = FALSE’ – a good thing

What’s on the horizon?

Links

See also

Preserving Wildlife with Computer Vision

Scaling Shiny Dashboards on a Budget

Appsilon is Hiring

R-Powered Excel

R + Juypter + Binder … just because

See you SatRday

References

A simple example for a function with errors

Where is the problem?

The solution: a safely/possibly wrapper

For comparison’s sake, how would you tryCatch it?

Bonus: a benchmark

Conclusion

The solution: a `safely`/`possibly` wrapper

For comparison’s sake, how would you `tryCatch` it?