datawrangling_tidyverse.Rmd

---
title: "Data wrangling with `tidyverse`"
subtitle: "Maria Novosolov"
date: "`r format(Sys.Date(),'%d-%m-%Y')`"
output:
  xaringan::moon_reader:
    lib_dir: libs
    css: ["default", "custom-fonts.css"]
    chakra: "libs/remark-latest.min.js"
    nature:
      highlightStyle: github
      highlightLines: true
      ratio: '16:9'
---
```{r setup, include=FALSE}
options(htmltools.dir.version = FALSE)
library(tidyverse)
library(ggplot2)
library(icon)
```

```{r use-logo, echo=FALSE}
xaringanExtra::use_logo("img/tidyverse.png")
```

# Tidyverse is a collection of packages
.center[
<img src="img/tidyverse_core.png" width=50%>
]

---

# The advantages

.large[
- shared syntax & conventions


- tibble/data.frame in, tibble out

- neat code

]
---

# Tidy data

>If I had one thing to tell biologists learning bioinformatics, it would be “write code for humans, write data for computers”.
>
>— Vince Buffalo (@vsbuffalo) July 20, 2013

---

|Film                       |Gender |Race   | Words|
|:--------------------------|:------|:------|-----:|
|The Fellowship Of The Ring |Female |Elf    |  1229|
|The Fellowship Of The Ring |Male   |Elf    |   971|
|The Fellowship Of The Ring |Female |Hobbit |    14|
|The Fellowship Of The Ring |Male   |Hobbit |  3644|
|The Fellowship Of The Ring |Female |Man    |     0|
|The Fellowship Of The Ring |Male   |Man    |  1995|
|The Two Towers             |Female |Elf    |   331|
|The Two Towers             |Male   |Elf    |   513|
|The Two Towers             |Female |Hobbit |     0|
|The Two Towers             |Male   |Hobbit |  2463|

---

# Does your code resemble this?

```{r}
starwars_human_subset <- subset(starwars,species == "Human")
starwars_human_subset$bmi <- starwars_human_subset$mass / 
  (0.01 * starwars_human_subset$height)^2
fattest_human_from_each_planet <- aggregate(bmi ~ homeworld,data = 
      starwars_human_subset, FUN = "max")
fattest_human_from_each_planet <- merge(
  x=fattest_human_from_each_planet,
  y=starwars_human_subset,by = c("homeworld","bmi"))
fattest_human_from_each_planet <- fattest_human_from_each_planet [,1:5]
```

![](https://jamesskemp.github.io/gits-matrix/images/green6.jpg)

---
# Code should be pleasant to read

![](https://media.giphy.com/media/OWyYSmZT43pxm/giphy.gif)

---
# Tibbles

>Tibbles are data.frames that are lazy and surly: **they do less** (i.e. they don't change variable names or types, and don't do partial matching) and **complain more** (e.g. when a variable does not exist). This forces you to confront problems earlier, typically leading to cleaner, more expressive code.

.center[
<img src="img/tibble.png" width=20%>
]

https://tibble.tidyverse.org/

---
# `data.frame`

```{r}
iris
```

---
# Tibbles print nicely!

```{r warning=FALSE,message=FALSE}
library(tidyverse)
as_tibble(iris)
```

---

# Pipe ("then")

.pull-left[
![](img/pipe.png)
]

.pull-right[
Data in, data out


```r
do_another_thing(do_something(data))

# versus

data %>% 
    do_something() %>% 
    do_another_thing() 
```
]

.footnote[
* keyboard shortcut: ctrl/cmd + shift + m
]
--- 
---

class: inverse, center, middle

# Let's go over the various packages
---

class: center, middle

# `readr` package

<img src="img/readr.png" width=30%>

---

# read_xxx function

* Neater import than `read.table` and `read.csv`

* Does data check and prints a report of the data imported

* Character columns are not converted to factors

* Most useful are `read_csv`, `read_table`, and `read_delim`

* Competible with pipe workflow

---
# Example

```{r}
mydata<- read_csv("data/sub_PanTHERIA.csv")
```

---

```{r}
mydata
```


---
class: center, middle

# `janitor` package

<img src="img/janitor.png" width=40%>
---
# `clean_names()` function

* cleans the column names to something more computer friendly

* For example, brings all the column names to lowercase and adds underscores between words
---

# Regular column names
```{r}
mydata
```

---
# Clean column names

```{r}
mydata %>% 
  janitor::clean_names() #<<
```
---

# Can also be:
```{r warning=FALSE,message=FALSE}
mydata<- read_csv("data/sub_PanTHERIA.csv") %>% 
  janitor::clean_names()
mydata
```

---
class: exercise, center, middle

# Lets practice!

## Load the data and change all the column names to caps lock

.footnote[
**Hint:** Check the help file for the `clean_names()` function
]
---
class: center, middle
# `dplyr` function


<img src="img/dplyr.png" width=30%>

---
# Load the Star Wars data

```{r}
library(tidyverse)
data(starwars)
starwars
```

---

# `select()`

![](img/xls-select.PNG)

---
# Select the name, height, mass, and species columns only

```{r}
starwars %>% 
    select(name, height, mass, species) #<<
```

---

# `mutate()`

![](img/xls-mutate.PNG)
---
# Add a bmi column


```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2) #<<
```

---

class: exercise, center, middle

# Your turn!

## select the height, gender, and species and add a new column with height in meters

---

# `filter()`

![](img/xls-filter.PNG)

---
# Filter the data to have only Droids (found in species column)


```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2) %>% 
    filter(species == "Droid") #<<
```

---
# Filter the same data to have only Droids shorter than 100 cm


```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2) %>% 
    filter(species == "Droid", height < 100) #<<
```

---

# `arrange()`

![](img/xls-arrange.PNG)
---

# Sort the data based on the bmi

```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2) %>% 
    arrange(desc(bmi)) #<<
```

---
# `group_by(), summarize()`

![](img/xls-summary.PNG)


---

## Create a summary data in which you have the avarage mass and maximum hight for each species, 
## and sort it by the avarage mass


```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    group_by(species) %>% #<<
    summarize(avg_mass = mean(mass, na.rm = TRUE), #<<
              max_height = max(height, na.rm = TRUE)) %>% 
    arrange(desc(avg_mass)) #<<
```

---

class: exercise, middle

# Your turn!

What is the most common eye color?

Who is the youngest human?

Which homeworld has the most characters?

---
# Rename columns with `rename()`
```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    rename(char_name = name) #<<
```
---

# `rename_all()`

Change all the column names to upper case
```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    rename_all(toupper) #<<
```

---
# `left_join()`

```{r}
starwars %>% 
  mutate(height_m = height*0.01) %>% 
  select(name,height_m) %>% 
  left_join(starwars,by = "name") #<<
```


---

class: exercise, center, middle

# Practice time!

Create a new data with species, mass, and height, calculate the bmi, and join it with the starwars data

---
# What else can you do?

- conditional functions: `*_at`, `*_if`, `*_all`
- `lead` & `lag` for time series
- `inner_join`,`semi_join`
- `bind_cols`, `bind_rows`

---
class: center, middle

# `tidyr` functions

<img src="img/tidyr.png" width=30%>

---

### spread == pivot_wider

### gather == pivot_longer

![](https://raw.githubusercontent.com/gadenbuie/tidyexplain/master/images/tidyr-spread-gather.gif)

---

# Community matrix!

```{r}
sw <- starwars %>% 
  select(name, films) %>% 
  unnest(films) %>% 
  mutate(present = 1) %>% 
  pivot_wider(names_from = name,values_from = present,values_fill = list(present = 0)) %>% #<<
  janitor::clean_names() %>% 
  print()
```

---

# Gather back the community matrix to a long format

```{r}
sw %>% 
  pivot_longer(cols = -films,names_to = "name",values_to = "present") #<<
```
---

class: exercise, center, middle

#Lets practice!

Subset the data to have species, mass, height, and birth year columns. 
Then create a long format with species as untouched column, one column with the categories (mass,height, birth year), and one column with the values.

---

.center[
# Other function that are compatible with ` %>% `
]


---

## `ggplot2` 

* can be piped into the sequence
```{r fig.height=4, fig.width=4, message=FALSE, warning=FALSE}
starwars %>% 
  select(height,mass,species) %>% 
  ggplot(.,aes(log10(mass),log10(height),color = species))+ #<<
  geom_point()+
  theme(legend.position = "none")
```

---
### omit all rows that contain `NA` somewhere in the data

```{r}
starwars %>% 
  na.omit() #<<
```

---
# Run models on subset of the data

```{r}
starwars %>% 
  filter(species =="Droid") %>% 
  lm(height~mass,data = .) %>% #<<
  summary()
```
---
class: center, middle

# `broom` package

<img src="img/broom.png" width=30%>


---
# `tidy()` function

Allows you to print and save model results in a tabular view

---

# Make it into a nice tibble

```{r}
starwars %>% 
  filter(species =="Droid") %>% 
  lm(height~mass,data = .) %>% 
  summary()%>% 
  broom::tidy() #<<
```

---

class: inverse, center, middle

# Homework Time

---

# Some instructions

- Uninstall the previous lesson with

  `uninstall_course("exercise_REcoStat2020")`
  
- Re-install it with `install_course_github("marianovosolov","exercise_REcoStat2020")`

- Start with `swirl()`

- Give it a new name

- Choose the tidyverse exercise within the exercise_REcoStat2020

- Good luck!!

---
class: inverse, center, middle

# Additional useful (though more tricky to understand) functions

---
# `nest` datasets

Useful when you want to run analyses group based but run it only ones

```{r}
nested_sw<- starwars %>% 
  select(name,height,films,mass, species,gender) %>% 
  group_by(gender) %>% 
  nest() %>% #<<
  print()

```

---
# To see the first sub data

```{r}
nested_sw$data[1]
```

---
class: center, middle

# `purrr` package

<img src="img/purrr.png" width=30%>

---

# `map` function

* transform the input by applying a function to each element. (similar to `apply` function)

```{r}
starwars %>% 
  split(.$gender) %>% 
  map(~ summary(lm(height~mass,data = .))) #<<
```

---
# Combine `nest()` with `map()`

```{r}
lm_nest<- starwars %>% 
  group_by(gender) %>% 
  nest() %>% 
  mutate(lm_results = map(data,~ summary(lm(height~mass,data =.)))) %>% #<<
  print()

```

---
```{r}
lm_nest$lm_results[1]
```


---

class: inverse, center, middle

# Now it's really the end `r icon::fa("smile")`