Feature_engineering_basics (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b9b44ddd-f71d-4cab-92d8-fa6b0878bd07",
   "metadata": {},
   "source": [
    "Q1: What are missing values in a dataset? Why is it essential to handle missing values? Name some\n",
    "algorithms that are not affected by missing values.\n",
    "\n",
    "Ans.\n",
    "\n",
    "Missing values occur in a datset when some of the information is not stored for a variable.\n",
    "\n",
    "It is essential to handle misssing values as\n",
    "\n",
    "1. Biased Results: Missing values can lead to biased estimates and incorrect conclusions if not appropriately handled during data analysis.\n",
    "\n",
    "2. Reduced Model Performance: Machine learning algorithms might struggle to perform well when confronted with missing data, leading to reduced predictive power and generalization capabilities.\n",
    "\n",
    "3. Loss of Information: Ignoring missing values can result in the loss of valuable information and patterns present in the data.\n",
    "\n",
    "Algorithms that are not affected by missing values.algorithms that are not affected by missing values are - \n",
    "\n",
    "Decision Trees\n",
    "\n",
    "Random Forest:\n",
    "\n",
    "k-Nearest Neighbors (k-NN)\n",
    "\n",
    "Gaussian Mixture Models (GMM)\n",
    "\n",
    "Gradient Boosting Machines (GBM)\n",
    "\n",
    "Principal Component Analysis (PCA)\n",
    "\n",
    "Singular Value Decomposition (SVD)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9f0f8c3-7543-4cef-96a3-86c00179749f",
   "metadata": {},
   "outputs": [],
   "source": [
    "Q2: List down techniques used to handle missing data. Give an example of each with python code.\n",
    "\n",
    "Ans.\n",
    "\n",
    "some technique are - \n",
    "\n",
    "1. row wise and column wise deletion\n",
    "\n",
    "eg -\n",
    "import seaborn as sns\n",
    "df = sns.load_dataset('titanic')\n",
    "df.head() #gives the first five details\n",
    "\n",
    "df.isnull().sum()#check missing values\n",
    "\n",
    "df.dropna()#it will delete the misssing values row wise\n",
    "\n",
    "df.dropna(axis = 1)#it will delete the missing values column wise\n",
    "\n",
    "2. Imputational missing values - \n",
    "\n",
    " 1. mean value imputation - replacing null values with the mean values,it works well when we have normally distributed data.\n",
    "    eg- \n",
    "    df['age_mean'] = df['age'].fillna(df['age'].mean())\n",
    "    df[['age_mean','age']]\n",
    "    \n",
    " 2. median value imputation - If we have outliers we need to use this.\n",
    "     eg- \n",
    "    df['age_median'] = df['age'].fillna(df['age'].median())\n",
    "    df[['age_median','age']]\n",
    "    \n",
    " 3. mode imputation technique -  It is used with categorical values\n",
    "     eg - \n",
    "    df['embarked_mode'] = df['embarked'].fillna(df['embarked'].mode())\n",
    "    df[['embarked_mode','embarked']]\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9c0e322-bec7-42f7-8d58-8a86acb93499",
   "metadata": {},
   "source": [
    "Q3: Explain the imbalanced data. What will happen if imbalanced data is not handled?\n",
    "\n",
    "Ans.\n",
    "Imbalanced data refers to those types of datasets where the target class has an uneven distribution of observations, i.e one class label has a very high number of observations and the other has a very low number of observations.\n",
    "\n",
    "Suppose we have two 1000 data points in our dataset, and 900 are yes and 100 are no.\n",
    "so the ratio will be 900:100 = 9:1,this is an imbalanced datatset, our model will get biased towards maximum no. of datapoints, so we need to fix and make them equal.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe71066-daa7-4597-9b4a-9d8fe1a5d689",
   "metadata": {},
   "source": [
    "Q4: What are Up-sampling and Down-sampling? Explain with an example when up-sampling and down-\n",
    "sampling are required.\n",
    "\n",
    "Ans.\n",
    "\n",
    "Up-sampling (Over-sampling):\n",
    "Up-sampling involves increasing the number of instances in the minority class by creating duplicates or generating synthetic samples. The goal is to balance the class distribution by ensuring that the minority class has a similar number of samples as the majority class.\n",
    "\n",
    "Example of Up-sampling:\n",
    "Let's consider a binary classification problem where we want to predict whether customers will leave a subscription service. In the dataset, we have 90% of customers who don't leave (majority class) and only 10% of customers who do leave (minority class). To up-sample the minority class, we duplicate or generate synthetic samples for the churned customers, making the number of churned and non-churned customers roughly equal.\n",
    "\n",
    "Down-sampling (Under-sampling):\n",
    "Down-sampling involves reducing the number of instances in the majority class by randomly removing some samples. The aim is to create a balanced class distribution with a similar number of samples in both classes.\n",
    "\n",
    "Example of Down-sampling:\n",
    "Continuing with the customer churn example, if we have an imbalanced dataset with 90% non-churned customers (majority class) and 10% churned customers (minority class), down-sampling would involve randomly removing some non-churned customers' data to make the class distribution more balanced.\n",
    "\n",
    "**When to Use Up-sampling and Down-sampling:\n",
    "\n",
    "Up-sampling:\n",
    "\n",
    "When the number of instances in the minority class is significantly smaller, leading to imbalanced data.\n",
    "When you have a limited amount of data, and collecting more data is challenging.\n",
    "When the dataset contains valuable information in the minority class that you don't want to lose.\n",
    "\n",
    "Down-sampling:\n",
    "\n",
    "When the dataset is too large, and processing time or memory resources are limited.\n",
    "When you have confidence that the majority class has enough representative samples and downsampling won't result in significant loss of information.\n",
    "When the majority class has noisy or redundant data, and reducing the size could help the model focus on the most informative instances.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebbdd01c-7d0e-40b3-b830-82a8bb91cf95",
   "metadata": {},
   "source": [
    "Q5: What is data Augmentation? Explain SMOTE.\n",
    "\n",
    "Ans.\n",
    "Data augmentation is a technique commonly used in machine learning and computer vision to increase the diversity and size of a dataset by applying various transformations to the existing data. \n",
    "\n",
    "SMOTE(Synthetic Minority  Oversampling technique) - It is a technique used in machine learning to address imbalanced dataset where the minority class has significantally fewer instances than the majority class.\n",
    "It involves generating synthetic instatnces of the minority by interpolating between existing instances.\n",
    "SMOTE joins the two nearest datapoints and add more datapoints on that joined line.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfd9feda-e3b1-40ba-8348-98b8604b4222",
   "metadata": {},
   "source": [
    "Q6: What are outliers in a dataset? Why is it essential to handle outliers?\n",
    "Ans.\n",
    "\n",
    "Outliers are data points that deviate significantly from the rest of the data in a dataset. They are extreme values that lie far away from the central tendency of the data distribution. Outliers can occur due to various reasons, such as measurement errors, data entry mistakes, natural variations, or even rare events.\n",
    "\n",
    "\n",
    "It is essential to handle outliers for several reasons:\n",
    "\n",
    "1. Impact on Descriptive Statistics: Outliers can significantly affect the mean, median, and other summary statistics, leading to inaccurate interpretations of the data.\n",
    "\n",
    "2. Influence on Model Performance: In predictive modeling and machine learning, outliers can disproportionately influence model training. They may lead to overfitting, where the model learns to perform well only on the outlier-affected data but poorly on new, unseen data.\n",
    "\n",
    "3. Skewed Analysis: Outliers can distort the distribution and spread of the data, leading to skewed analysis results and incorrect conclusions.\n",
    "\n",
    "4. Data Normalization: Some machine learning algorithms rely on the assumption that the data is normally distributed. Outliers can violate this assumption and negatively impact model performance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "117e292a-29d0-4f27-ac7f-9b052bfd0459",
   "metadata": {},
   "source": [
    "Q7: You are working on a project that requires analyzing customer data. However, you notice that some of\n",
    "the data is missing. What are some techniques you can use to handle the missing data in your analysis?\n",
    "\n",
    "Ans.\n",
    "\n",
    "Some techinques we can use are -\n",
    "1. Deletion -  Remove entire rows with missing values. This approach is simple but may result in data loss, especially if missingness is prevalent in several attributes.\n",
    "2. Mean/Median/Mode Imputation -  replace missing values with the mean, median, or mode of the available data for numerical or categorical variables, respectively.\n",
    "3. Data Augmentation-  data augmentation techniques can be applied to generate additional data points, thereby filling in missing values.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc39d404-76c5-4a61-81ce-0eca2c3694d8",
   "metadata": {},
   "source": [
    "Q8: You are working with a large dataset and find that a small percentage of the data is missing. What are\n",
    "some strategies you can use to determine if the missing data is missing at random or if there is a pattern\n",
    "to the missing data?\n",
    "\n",
    "Ans.\n",
    "Some strategies are - \n",
    "\n",
    "1. Data Visualization:\n",
    "Create visualizations (e.g., bar plots, heatmaps) to visualize the distribution of missing values across variables. \n",
    "This can help identify patterns and correlations between missing values.\n",
    "\n",
    "2. Calculate the percentage of missing values for each variable in the dataset. If the missingness is random, the percentage of missing values should be approximately the same across all variables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ffb07c5-26e7-4e1f-b359-66b6d476c2b9",
   "metadata": {},
   "source": [
    "Q9: Suppose you are working on a medical diagnosis project and find that the majority of patients in the\n",
    "dataset do not have the condition of interest, while a small percentage do. What are some strategies you\n",
    "can use to evaluate the performance of your machine learning model on this imbalanced dataset?\n",
    "\n",
    "Ans.\n",
    "Choose Proper Evaluation Metric.\n",
    "Resampling (Oversampling and Undersampling)\n",
    "SMOTE."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "284b6566-131e-4888-ad53-b6474618f183",
   "metadata": {},
   "source": [
    "Q10: When attempting to estimate customer satisfaction for a project, you discover that the dataset is\n",
    "unbalanced, with the bulk of customers reporting being satisfied. What methods can you employ to\n",
    "balance the dataset and down-sample the majority class?\n",
    "\n",
    "Ans.\n",
    "To balance an unbalanced dataset with the majority class (satisfied customers) when estimating customer satisfaction, you can employ down-sampling techniques. \n",
    "Down-sampling involves reducing the number of instances in the majority class to create a more balanced dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "475498c4-c581-42d7-9221-b7b23e15b374",
   "metadata": {},
   "source": [
    "Q11: You discover that the dataset is unbalanced with a low percentage of occurrences while working on a\n",
    "project that requires you to estimate the occurrence of a rare event. What methods can you employ to\n",
    "balance the dataset and up-sample the minority class?\n",
    "\n",
    "Ans.\n",
    "When dealing with an unbalanced dataset with a low percentage of occurrences of a rare event, you can employ up-sampling techniques to balance the dataset and increase the number of instances in the minority class. \n",
    "Up-sampling involves creating duplicates or generating synthetic samples of the minority class to match the size of the majority class. \n",
    "\n",
    "We can use random oversampling and SMOTE."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}