Data Integration • Combines data from multiple sources into one unified dataset. • Creating a Unified Dataset • Enriching Data with More Features • Addressing Data Quality Issues • Preparing Data for Feature Engineering • Challenges: Varying formats, structures, semantics. •Techniques: ❑ Record Linkage: Match records that refer to the same entity, even if represented differently. ❑ Data Fusion: Merge incomplete/inconsistent data to create a complete, accurate dataset. ❑ ETL (Extract, Transform, and Load): • This is a traditional and widely used technique. • Extract: Data is extracted from various source systems. • Transform: The extracted data is cleaned, standardized, filtered, and transformed to fit the target schema. • involve data type conversions, aggregations, and joining datasets. • Load: The transformed data is loaded into a target data warehouse or data mart. ❑ Data Warehousing: • Involves creating a central repository (data warehouse) to store integrated data from multiple sources. Data is typically processed using ETL before being loaded into the warehouse. ❑ Data Federation (Virtualization): • Instead of physically moving and storing data in a central location, data federation creates a virtual layer that provides a unified view of data residing in its source systems. • Queries are translated and sent to the respective sources, and the results are integrated virtually. ❑ API Integration: • Modern applications often use APIs (Application Programming Interfaces) to exchange data in real-time. • Integrating data through APIs allows for seamless data flow between systems. ❑ Master Data Management (MDM): • Focuses on creating and maintaining a single, consistent, and accurate "master" record for key business entities like customers, products, or suppliers. • Data from different systems is reconciled and linked to this master record. ❑Change Data Capture (CDC): • Identifies and tracks changes made to data in source systems and replicates these changes in the target integrated system in near real-time. Common Issues in Data Integration • Data integration or merging causes: a) Entity identification problem. b) Data duplication and redundancy c) Data conflict & inconsistencies Entity identification problem • It occurs when objects/entities do not have the same identifiers across different data sources. • This makes it hard to merge, analyze, or reconcile records referring to the same real-world entity. • Example: The same student is represented by different identifiers, such as student ID in the school database, a registration number in the exam system etc. solutions • Schema Integration: It is the process of mapping and aligning fields from different datasets by comparing attribute names, data types, formats, and ranges to ensure compatibility before merging. • Object Matching: It is a technique used to determine whether two records from different data sources refer to the same real-world entity, often using rules, similarity scores, or metadata. • Metadata Comparison: It is the practice of checking additional attributes to confirm whether two records represent the same entity. Data Duplication and Redundancy • Data Duplication Object Duplication occurs when two or more instances (objects, records) in a dataset have identical feature values across all attributes, but may still represent different real-world entities. • Example: Two students with the same grades in all courses ➢Here, the best practices are: • Always include or check for unique identifiers (like Student ID, Register No., etc.). • Before dropping duplicates, verify if they truly refer to the same entity. • Use domain knowledge to distinguish real duplicates from naturally similar entries. • Redundancy Redundancy occurs when one feature (attribute) can be derived from one or more other features in the dataset. • These redundant features don't add new information and can be safely removed to reduce dimensionality and improve model efficiency. • Example: If a dataset includes both "total marks" and "marks in each subject", storing "average marks" is redundant, as it can be calculated from the total marks divided by the number of subjects. • When multiple sources are combined, redundancy often arises due to overlapping or duplicated information, which needs to be managed carefully. • Redundant data can often be detected by correlational analysis Data Conflict & Inconsistencies • Data value conflicts occur when the same attribute has different values across multiple data sources. These conflicts can lead to inconsistencies and inaccuracies in the data. • Example: A student's name might be listed as "Aiswarya" in the school database but as "Aishwarya" in the national ID system. • To resolve such conflicts, expert knowledge or domain-specific rules are often required to determine the correct value, ensuring that the data is consistent and accurate. Data Transformation • Modifies data to enhance suitability for analysis or modeling. • Improved Data Quality • Enhanced Model Performance • Easier Analysis and Visualization • Integration of Data Sources • Easier scalability • Feature Engineering Techniques: ❑ Scaling & Normalization / Standardization • Scale numerical features for better performance. • Normalization is often applied when the data has varying scales. • involves adjusting the data values to a common scale without distorting differences in the ranges of values. • ensures that the data is within a specific range, often, making it suitable for models that are sensitive to the magnitude of input data. • common methods • Min-Max Scaling: This rescales the data to a specific range, often. The formula is: • Where: • X is the original value of datapoint • Xmin is minimum value in the feature • Xmax is the maximum value in the feature • Xnormalised is the rescaled value of X after normalization • Z-Score Normalization (Standardization) • Involves centering the data by subtracting the mean and scaling by the standard deviation, resulting in a distribution with a mean of 0 and a standard deviation of 1. • It is useful for data with Gaussian-like distribution. • Robust Scaling • It is a variation that uses the median and interquartile range (IQR) for scaling, which makes it more robust to outliers. • It is useful when the data has heavy outliers and you're trying to avoid their influence in scaling. ❑ Encoding Categorical Variables • Convert categories into numbers (e.g., one-hot, label encoding). • When dealing with categorical variables, many machine learning algorithms need numeric representations to process them. • One-Hot Encoding: Creates binary variables for each possible category (e.g., for a feature "Color" with values "Red", "Blue", and "Green", it creates three binary columns: Color_Red, Color_Blue, and Color_Green where a 'l' indicates the presence of that color). • Label Encoding: Assigns a unique numerical label to each category. For example, "Red" → 0, "Blue" → 1, "Green" → 2. This method works well when the categories have an ordinal relationship, i.e., a natural order (e.g., "Low", "Medium", "High"). • Binary Encoding: Converts each category into its binary representation. • Target Encoding: Replaces each category with the mean of the target variable for that category. This can be useful for predictive modeling but can lead to overfitting if not done carefully. ❑ Log Transformation • When data is heavily skewed (i.e., it has a long tail), a log transformation helps compress the scale of larger values and reduce the impact of outliers. • Log transformation is useful for reducing the skewness of data with large, exponential values. • Its formula is: ❑Power Transformation • Power transformations stabilize variance and make the data more Gaussian (normal distribution). • The two common power transformations are: • Box-Cox: • Applies a transformation to continuous data that ensures it is positive, and it can be considered as a family of transformations. It is particularly useful for non-normal data. • Yeo-Johnson: • An extension of Box-Cox that can handle negative values, making it applicable to datasets where some features can be zero or negative. ❑ Binning (Discretization) •Binning involves converting continuous data into discrete categories (or "bins"). •This can make the model less sensitive to small fluctuations in the data. •Equal-Width Binning: Divides the range of the data into equal-sized intervals. However, I t may lead to uneven distribution of data within bins if the data is not uniformly distributed •Equal-Frequency Binning: Divides the data into bins with approximately the same number of data points in each. This helps maintain a balance in the binning process. ❑ Handling Missing Data ❑ Feature Engineering • Create new features from existing ones to improve insights and model accuracy. • such as combining features, extracting date-time features, or aggregating data (e.g., averages, sums) ❑ Outlier Detection • Outliers are values that are significantly different from the majority of the data. • Outliers can significantly distort the results of machine learning algorithms. So identifying and treating them is crucial. ❑ Feature Selection/Reduction • Reducing the number of input features is essential when dealing with high-dimensional data. • reduces the number of features by selecting the most relevant ones or combining features to reduce dimensionality. • It improves model performance, reduces overfitting, and speeds up computation. • This can be achieved through dimensionality reduction techniques Data Reduction •Reduces dataset size for better efficiency without losing important information. • Techniques ❑ Dimensionality Reduction • reduces the number of dimensions that data may be seen from. • As the number of features rises, the sparsity of the qualities of features in the data collection increases. • These algorithms rely heavily on the sparsity of the data. Data may be more easily seen and manipulated when it has fewer dimensions. • Dimensionality reduction is crucial when your dataset boasts a large number of features. • This high dimensionality can lead to several problems. ➢ Feature Selection • aims to identify and retain only the most relevant features, discarding the redundant or irrelevant ones. • a) Filter Methods: • Select features independently of any learning algorithm using some selection criterion. • They are lightweight, computationally efficient, and provide a generic, unbiased feature selection. • By analyzing relationships among features, they can detect data abnormalities without depending on specific models, making them suitable for diverse datasets. • The important methods that come under filter methods are given below. • Correlation: Measures the linear relationship between two features. Highly correlated features might be redundant, and one can be removed. • For a target variable, features with high correlation (positive or negative) are often more relevant. • It is computationally efficient and doesn't consider the interaction between features or the relationship with the chosen model. • Chi-Squared Test: • Used for categorical features to determine if there's a statistically significant association between them and the target variable (for classification tasks). • Features with a high chi-squared statistic are considered more relevant. This method is effective for categorical data. • Information Gain: • Measures the reduction in entropy (uncertainty) of the target variable when a particular feature is known. • Features with high information gain are considered more informative. Often used in decision tree algorithms. • Variance Thresholding: • Removes features whose variance falls below a certain threshold. • Features with low variance provide little information as they don't change much across the data points. It is a simple and fast method. • b) Wrapper Methods: • evaluate subsets of features by training and testing a specific machine learning model. • They search through the possible feature subsets and select the one that yields the best model performance. • Forward Selection: • Starts with an empty set of features and iteratively adds the most significant feature until a stopping criterion is met (e.g., no further improvement in model performance). • It can be more effective than filter methods as it considers the model. • But it is computationally expensive, especially with a large number of features. • Backward Elimination: • Starts with all features and iteratively removes the least significant feature until a stopping criterion is met. • It can sometimes identify better feature subsets than forward selection. It is also computationally expensive. • Recursive Feature Elimination (RFE): • Repeatedly fits a model (e.g., SVM, linear regression) and removes the weakest feature(s) until the desired number of features is reached. • This method is effective in finding optimal feature subsets and computationally intensive. • c) Embedded Methods: • These methods perform feature selection as an integral part of the model training process. • L1 Regularization (Lasso): Adds a penalty term to the loss function of linear models proportional to the absolute value of the coefficients. • This encourages some feature coefficients to become exactly zero, effectively performing feature selection. • It is efficient and performs feature selection during model training. • Tree-based Feature Importance: Algorithms like Random Forests and Gradient Boosting inherently provide a measure of feature importance based on how much each feature contributes to reducing impurity or error in the model. • Less important features can be discarded. ➢Feature Extraction • Feature extraction aims to transform the original features into a new set of features with lower dimensionality while preserving the essential information. • a) Principal Component Analysis (PCA): A linear dimensionality reduction technique that transforms the original features into new axes or principal components. • It finds the directions (principal components) of maximum variance in the data and projects the data onto a lower-dimensional subspace formed by the top principal components. • It reduces dimensionality while retaining most of the variance, often reveals underlying structure in the data, removes correlation between features. • b) Linear Discriminant Analysis (LDA): A supervised dimensionality reduction technique primarily used for classification. • It aims to find a linear combination of features that best separates different classes in the data. • c) Singular Value Decomposition (SVD): It is a matrix factorization technique, breaking down a matrix into three smaller matrices that reveal important patterns, structures, and relationships in the data. • It is powerful and widely applicable, but can be computationally expensive for very large matrices; • the resulting components might not be easily interpretable. • It can be applied in recommender systems, text analysis and so on. • d) Non-linear Dimensionality Reduction: These techniques are designed to preserve the local structure of high-dimensional data when projecting it to a lower dimension, making them particularly useful for visualization. • t-distributed Stochastic Neighbour Embedding (t-SNE): • Models the probability of pair-wise similarity between data points in both the high-dimensional and low-dimensional spaces and tries to minimize the difference between these probabilities. • It is well-suited for embedding high-dimensional data for visualization in a low-dimensional space of two or three dimensions. • Uniform Manifold Approximation and Projection (UMAP): • A technique used to reduce high-dimensional data into 2D or 3D for easier visualization, while keeping the important patterns and relationships between data points as close as possible. • It constructs a high-dimensional graph representing the neighbourhood relationships in the data and then tries to find a low-dimensional embedding that preserves this graph structure. ❑ Numerosity Reduction • It aims to decrease the number of data samples while retaining the representativeness of the original dataset. A few common methods are given here. a) Sampling: Process of selecting a subset of the data. • The key is to choose a sample that accurately reflects the properties of the entire dataset. • Simple Random Sampling (SRS): • Each data point has an equal probability of being included in the sample. • Stratified Sampling: • Divides the dataset into homogeneous subgroups (strata) based on one or more relevant attributes. • Then, a random sample is taken from each stratum, ensuring proportional representation of each subgroup in the final sample. • Cluster Sampling: • Divides the dataset into clusters (groups), and then a random sample of clusters is selected. • All data points within the selected clusters are included in the sample. This is often used when data is naturally grouped. • Systematic Sampling: • Selects data points at regular intervals (e.g., every 10th data point). • The starting point is usually chosen randomly. • b) Clustering: Groups similar data points into clusters. • Instead of keeping all data points, you can represent each cluster by its centroid (mean of the data points in the cluster) or by selecting one representative data point from each cluster. • c) Aggregation: Combines multiple data points into a single summary data point. • This is often done based on time intervals (e.g., daily to monthly), geographical regions, or other relevant categories. • d) Histogramming: Divides the range of values for a continuous attribute into bins and stores the frequency of data points falling into each bin. This provides a summary of the data distribution. Data Discretization • process of converting continuous data or numerical values into discrete categories or bins. • known as binning. • It is used to simplify complex data and make it easier to analyse and work with. • Instead of dealing with exact values, discretization groups the data into ranges and helps algorithms perform better especially in classification tasks. • Several regression and classification models, like decision trees and Naive Bayes, perform better with discrete values. • The discretization process assigns a discrete value to each interval of a continuous attribute 'A' to create a new discrete attribute A’. • The discretization algorithm determines the interval boundaries that should preserve as much useful information as possible from the original attribute. • If the data set is used for the creation of a classification model, data set discretization should preserve the relationship between the class and the discretized attributes.  • we convert continuous variables into discrete features. • To do this, we compute the limits of the contiguous intervals that span the entire variable value range. • Next, we sort the original values into those intervals. These intervals, which are now discrete values, are then handled as categorical data. • Techniques 1. Unsupervised Discretization Methods 2. Supervised Discretization Methods Unsupervised Discretization Methods • These methods do not consider the class label while dividing the data. • a) Equal Width Binning (Equal Interval Binning) • uses the equal width criterion for interval bound setting. • The range of the attribute is divided into k intervals of equal size. • Its interval width can be calculated using the given equation. • Interval width = (max-min)/k. • Example: • For values from 0 to 100 and k=5, bins would be:[0-20), [20-40), [40-60), [6080), [80-100]. Here we have data values ranging from 0 to 100, and we want to divide them into 5 equal-width bins. Then, • Range = (max-min) = (100 - 0) = 100 • Number of bins (k) = 5 • Interval width = 100 / 5 = 20 • The bins are: • [0-20) → includes values from 0 up to (not including) 20 • [20-40) → includes values from 20 up to (not including) 40 • [40-60) → includes values from 40 up to (not including) 60 • [60-80) → includes values from 60 up to (not including) 80 • [80-100] → includes values from 80 up to and including 100 • Equal Width Binning is a simple and fast discretization method. • It is computationally efficient, suitable for large datasets, and easy to implement. • However, it may result in poor bin quality if the data is not uniformly distributed, and it is sensitive to outliers. • Despite these limitations, it provides sufficiently good discretization for many practical applications. • b) Equal Frequency Binning • It is unsupervised discretization algorithm that uses the equal frequency (data count) criterion for interval bound setting. • Each bin contains approximately the same number of data points. • It is also known as Equal Height Binning, Equal Depth Binning or Equal Quantile Binning. • Example: • If we have 100 data points and k=4, each bin will contain 25 sorted data points. • Equal Frequency Binning is less complex and therefore computationally more efficient and can handle large numbers of attributes. • It divides data so that each bin contains approximately the same number of data points, making it more robust to outliers compared to equal width binning. However, the bin widths vary, which can be confusing, and it may separate similar values into different bins, affecting data interpretation. • c) Clustering-based Discretization • It uses clustering algorithms like k-means to group data into clusters and each cluster is treated as a bin. • This method captures the natural structure of the data. • However, it requires predefining the number of clusters and is sensitive to initialization and outliers, which can affect the quality of discretization. • The quality of discretization depends on the effectiveness of the clustering algorithm and the choice of the number of clusters. • Example: • Let's have the following data showing the ages of customers: 18, 19, 20, 23, 25, 45, 48, 60, 61, 66. Now we are applying k-means clustering. Suppose we want to discretize these ages into 3 bins (clusters) using k-means clustering. • The algorithm might group the data into: • Cluster 1: → Young • Cluster 2: → Middle-aged • Cluster 3: → Senior • Each cluster is now treated as a discrete bin: • Bin 1: Age group 18–25 • Bin 2: Age group 45–48 • Bin 3: Age group 60–66 • d) Histogram Analysis for Discretization • It is an unsupervised method that relies on the distribution of data values to determine bin edges. • It builds a histogram of a continuous attribute and divides data into bins based on either using a fixed bin width (as in equal-width binning) or using data density (variable-width bins: narrower in dense areas, wider in sparse ones). • This method captures the underlying structure of the data without relying on class labels, making it ideal when label information is not available. • Example: Let's say we have the following values representing the ages of 22 people: We create the following bins with equal width (width=10). The various histogram representation of this example is shown in Figures 2.7 and 2.8. • 0–9 → includes: 2, 2, 2, 5, 8 → 5 items • 10–19 → includes: 12, 14, 14, 14, 15, 18 → 6 items • 20–29 → includes: 21, 21, 21, 21, 23, 23 → 6 items • 30–39 → includes: 30, 35, 38, 38 → 4 items • 40–49 → includes: 42 → 1 item Supervised Discretization Methods • These methods use the class label during Discretization • a) Entropy-Based Binning • It uses class label information to guide the splitting of continuous data into intervals. • It is commonly used in decision tree algorithms like ID3 (Iterative Dichotomiser 3) and C4.5. • It uses information gain to find optimal split points for continuous attributes. • It evaluates all possible splits, calculates the entropy for each, and selects the split with the highest information gain, which indicates the best separation between classes. • This process is applied recursively to create multiple bins. • Information Gain (IG) measures the reduction in entropy (class uncertainty) achieved by splitting a dataset based on a particular attribute. It tells us how well an attribute separates the classes. • A higher information gain means the split does a better job of differentiating between classes. • This method is class-aware, leading to more meaningful splits and improved classification accuracy. • b) ChiMerge (Chi-Square Merge) • initially treats each distinct value of a continuous attribute as a separate interval. • Chi-square tests are performed for every pair of adjacent intervals. • Adjacent intervals with the least Chi-2 values are merged because low Chi-2 values for a pair indicate similar class distributions. • This merging process continues until a predefined stopping criterion is reached. • It is statistically sound since it uses the chi-square test, a robust statistical method, to evaluate the relationship between the intervals and class labels. • It produces meaningful bins ensures that the intervals created have high inter-class distance, improving the ability to differentiate between classes. • A significance level or threshold must be chosen for the chi-square test, which can impact the resulting bins. Student Study Hours Result A 1 Fail B 2 Fail C 3 Pass D 4 Pass E 5 Pass F 6 Fail • Initially, each distinct study hour is treated as a separate interval. • Step 1: Calculate Chi-Square Statistic • Merge adjacent intervals based on the chi-square statistic, which measures the difference between observed and expected frequencies. • For example, consider merging intervals and if their combined chisquare statistic indicates they don't significantly differ in terms of the class label distribution. • Step 2: Merging Intervals • If the chi-square statistic is below a chosen threshold (e.g., 3.84 for a 5% significance level), we merge the intervals. For instance, we may merge intervals and to create [1-2]. • This process continues until no further merges are needed, creating intervals that best separate the classes (Pass and Fail). • c) Decision Tree-Based Discretization • It is a supervised method that uses decision trees (like ID3, C4.5, or Classification and Regression Trees (CART)) to find optimal split points for continuous attributes based on class labels. • The tree's decision boundaries (i.e., split thresholds) are used as bin edges for discretization. • The method works by training a decision tree using the continuous attribute along with its corresponding class labels. • During this process, the decision tree identifies optimal split points (thresholds) that best separate the classes. • These split points, also known as decision boundaries, are then used as bin edges to convert the continuous values into discrete intervals. • This approach ensures that the discretization is class-aware and aligned with the structure learned by the tree, often improving classification performance. Age Class 22 No 25 No 28 Yes 35 Yes 45 Yes • In the given dataset with the attribute "Age" and class labels like "Yes" or "No", a decision tree might split the data at Age < 30 as one group and Age \(\ge \) 30 as another based on class separation. • effectively converting the continuous "Age" into categorical bins. Then the discretized bins become: • Bin 1: Age < 30 • Bin 2: Age ≥ 30 Top-Down Discretization and Bottom-Up Discretization • Discretization methods can also be categorized by their approach: • Top-Down (Splitting): • These methods start with a single interval containing all the data and recursively split it into smaller intervals based on some criterion (e.g., entropy, equal width). • Ex: Decision tree-based discretization. • Bottom-Up (Merging): • These methods start with each continuous value (or a small initial interval) as a separate interval and then iteratively merge adjacent intervals based on a certain criterion (e.g., chi-squared test). • Ex: ChiMerge Data Compression • Data compression aims to reduce the storage size of the data. Its classifications are: • Lossless Compression: • Allows perfect reconstruction of the original data from the compressed form. • These methods exploit redundancies in the data. • The primary methods that fall under this category are Run-Length Encoding (RLE), Huffman Coding, and Lempel-Ziv-Welch (LZW). • Lossy Compression: • Sacrifices some information to achieve a higher compression ratio. • The goal is to remove less important or redundant information in a way that the loss is not perceptually significant. • Several key techniques fall under this category .JPEG,MP3,MPEG etc Data Validation • Final step to verify data quality and readiness for modeling. • Involves: • Checking Data Types: Ensure correct data types (int, float, string, etc.). • Verifying Value Ranges: Ensure values fall within acceptable limits. • Ensuring Feature Completeness: Confirm all required features are present and properly formatted. • Detecting remaining issues: identifying any leftover missing values or inconsistencies Tools and Techniques for Data Preprocessing • a) Python Libraries • Pandas: Essential for data manipulation, offering powerful structures like DataFrames and Series and functions for for reading, transforming, and analyzing data. • NumPy: Supports high-performance numerical operations on large arrays and matrices. • Scikit-learn: Offers preprocessing utilities such as scaling, encoding, and imputing for machine learning pipelines. • b) SQL • c) Apache Spark Ideal for bigdata preprocessing, Spark distributes data processing across clusters, enabling efficient handling of large-scale datasets. • d) AutoML Tools Automated machine learning tools streamline preprocessing tasks like feature selection and encoding. Popular examples: • H2O AutoML: Provides automatic feature engineering and model selection. • AutoSklearn: An automated machine learning toolkit based on scikitlearn. • e) Custom Preprocessing Functions For domain-specific tasks, custom functions are often needed, such as text preprocessing for natural language processing, image preprocessing for computer vision, time series preprocessing for forecasting etc.