-
Notifications
You must be signed in to change notification settings - Fork 27
/
README.Rmd
204 lines (123 loc) · 10.1 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%",
dpi = 300,
message = F,
warning = F
)
devtools::load_all()
library(tidyverse)
```
# correlationfunnel <img src="man/figures/logo-correlationfunnel.png" width="147" height="170" align="right" />
_by [Business Science](https://www.business-science.io/)_
[](https://www.tidyverse.org/lifecycle/#maturing)
[](https://travis-ci.org/business-science/correlationfunnel)
[](https://codecov.io/github/business-science/correlationfunnel?branch=master)
[](https://cran.r-project.org/package=correlationfunnel)
> Speed Up Exploratory Data Analysis (EDA)
The goal of `correlationfunnel` is to speed up Exploratory Data Analysis (EDA). Here's how to use it.
## Installation
You can install the latest stable (CRAN) version of `correlationfunnel` with:
``` r
install.packages("correlationfunnel")
```
You can install the development version of `correlationfunnel` from [GitHub](https://github.com/business-science/) with:
``` r
devtools::install_github("business-science/correlationfunnel")
```
## Correlation Funnel in 2-Minutes
__Problem__:
Exploratory data analysis (EDA) involves looking at feature-target relationships independently. This process is very time consuming even for small data sets. ___Rather than search for relationships, what if we could let the relationships come to us?___
<img src="man/figures/README-corr_funnel.png" width="35%" align="right" style="border-style: solid; border-width: 2px; border-color: #2c3e50; margin-left: 10px; "/>
__Solution:__
Enter `correlationfunnel`. The package provides a __succinct workflow__ and __interactive visualization tools__ for understanding which features have relationships to target (response).
__Main Benefits__:
1. __Speeds Up Exploratory Data Analysis__
2. __Improves Feature Selection__
3. __Gets You To Business Insights Faster__
## Example - Bank Marketing Campaign
The following example showcases the power of __fast exploratory correlation analysis__. The goal of the analysis is to determine which features relate to the bank's marketing campaign goal of having customers opt into a TERM DEPOSIT (financial product).
We will see that using __3 functions__, we can quickly:
1. Transform the data into a binary format with `binarize()`
2. Perform correlation analysis using `correlate()`
3. Visualize the highest correlation features using `plot_correlation_funnel()`
__Result__: Rather than spend hours looking at individual plots of capaign features and comparing them to which customers opted in to the TERM DEPOSIT product, in seconds we can discover which groups of customers have enrolled, drastically speeding up EDA.
### Getting Started
First, load the libraries.
```{r example}
library(correlationfunnel)
library(dplyr)
```
Next, collect data to analyze. We'll use Marketing Campaign Data for a Bank that was popularized by the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Bank+Marketing). We can load the data with `data("marketing_campaign_tbl")`.
```{r}
# Use ?marketing_campagin_tbl to get a description of the marketing campaign features
data("marketing_campaign_tbl")
marketing_campaign_tbl %>% glimpse()
```
### Response & Predictor Relationships
Modeling and Machine Learning problems often involve a response (Enrolled in `TERM_DEPOSIT`, yes/no) and many predictors (AGE, JOB, MARITAL, etc). Our job is to determine which predictors are related to the response. We can do this through __Binary Correlation Analysis__.
### Binary Correlation Analysis
Binary Correlation Analysis is the process of converting continuous (numeric) and categorical (character/factor) data to binary features. We can then perform a correlation analysis to see if there is predictive value between the features and the response (target).
#### Step 1: Convert to Binary Format
The first step is converting the continuous and categorical data into binary (0/1) format. We de-select any non-predictive features. The `binarize()` function then converts the features into binary features.
- __Numeric Features:__ Are binned into ranges or if few unique levels are binned by their value, and then converted to binary features via one-hot encoding
- __Categorical Features__: Are binned by one-hot encoding
The result is a data frame that has only binary data with columns representing the bins that the observations fall into. Note that the output is shown in the `glimpse()` format. THere are now 80 columns that are binary (0/1).
```{r}
marketing_campaign_binarized_tbl <- marketing_campaign_tbl %>%
select(-ID) %>%
binarize(n_bins = 4, thresh_infreq = 0.01)
marketing_campaign_binarized_tbl %>% glimpse()
```
#### Step 2: Perform Correlation Analysis
The second step is to perform a correlation analysis between the response (target = TERM_DEPOSIT_yes) and the rest of the features. This returns a specially formatted tibble with the feature, the bin, and the bin's correlation to the target. The format is exactly what we need for the next step - Producing the __Correlation Funnel__
```{r}
marketing_campaign_correlated_tbl <- marketing_campaign_binarized_tbl %>%
correlate(target = TERM_DEPOSIT__yes)
marketing_campaign_correlated_tbl
```
#### Step 3: Visualize the Correlation Funnel
A __Correlation Funnel__ is an tornado plot that lists the highest correlation features (based on absolute magnitude) at the top of the and the lowest correlation features at the bottom. The resulting visualization looks like a Funnel.
To produce the __Correlation Funnel__, use `plot_correlation_funnel()`. Try setting `interactive = TRUE` to get an interactive plot that can be zoomed in on.
```{r, fig.height=8}
marketing_campaign_correlated_tbl %>%
plot_correlation_funnel(interactive = FALSE)
```
### Examining the Results
The most important features are towards the top. We can investigate these.
```{r, fig.height=3}
marketing_campaign_correlated_tbl %>%
filter(feature %in% c("DURATION", "POUTCOME", "PDAYS",
"PREVIOUS", "CONTACT", "HOUSING")) %>%
plot_correlation_funnel(interactive = FALSE, limits = c(-0.4, 0.4))
```
We can see that the following prospect groups have a much greater correlation with enrollment in the TERM DEPOSIT product:
- When the DURATION, the amount of time a prospect is engaged in marketing campaign material, is 319 seconds or longer.
- When POUTCOME, whether or not a prospect has previously enrolled in a product, is "success".
- When CONTACT, the medium used to contact the person, is "cellular"
- When HOUSING, whether or not the contact has a HOME LOAN is "no"
## Other Great EDA Packages in R
The main addition of `correlationfunnel` is to quickly expose feature relationships to semi-processed data meaning missing (`NA`) values have been treated, date or date-time features have been feature engineered, and data is in a "clean" format (numeric data and categorical data are ready to be correlated to a Yes/No response).
Here are several great EDA packages that can help you understand data issues (cleanliness) and get data preprared for Correlation Analysis!
- [Data Explorer](https://boxuancui.github.io/DataExplorer/) - Automates Exploration and Data Treatment. Amazing for investigating features quickly and efficiently including by data type, missing data, feature engineering, and identifying relationships.
- [naniar](http://naniar.njtierney.com/) - For understanding missing data.
- [UpSetR](https://github.com/hms-dbmi/UpSetR) - For generating upset plots
- [GGally](https://ggobi.github.io/ggally/) - The `ggpairs()` function is one of my all-time favorites for visualizing many features quickly.
## Using Correlation Funnel? You Might Be Interested in Applied Business Education
[___Business Science___](https://www.business-science.io/) teaches students how to apply data science for business. The entire curriculum is crafted around business consulting with data science. _Correlation Analysis_ is one of the many techniques that we teach in our curriculum.
__Learn from our data science application experience with real-world business projects.__
### Learn from Real-World Business Projects
Students learn by solving real world projects using our repeatable project-management framework along with cutting-edge tools like the Correlation Analysis, Automated Machine Learning, and Feature Explanation as part of our ROI-Driven Data Science Curriculum.
<a href="https://university.business-science.io/p/machine-learning-web-apps-level-1-bundle-r-track-courses-101-102-201"><img src="man/figures/README-3-course-system.jpg" width="100%" style="border-style: solid; border-width: 2px; border-color: #2c3e50"/></a>
- [__Learn Data Science Foundations (DS4B 101-R)__](https://university.business-science.io/p/ds4b-101-r-business-analysis-r): Learn the entire `tidyverse` (`dplyr`, `ggplot2`, `rmarkdown`, & more) and `parsnip` - Solve 2 Projects - Customer Segmentation and Price Optimization projects
- [__Learn Advanced Machine Learning & Business Consulting (DS4B 201-R)__](https://university.business-science.io/p/hr201-using-machine-learning-h2o-lime-to-predict-employee-turnover/): Churn Project solved with Correlation Analysis, `H2O` AutoML, `LIME` Feature Explanation, and ROI-driven Analysis / Recommendation Systems
- [__Learn Predictive Web Application Development (DS4B 102-R)__](https://university.business-science.io/p/ds4b-102-r-shiny-web-application-business-level-1/): Build 2 Predictive `Shiny` Web Apps - Sales Dashboard with Demand Forecasting & Price Prediction App