Why is it better to provide raw data to Khiops instead of preprocessed or encoded data?

[lucaurelien]

This discussion is inspired by questions we often receive about data preparation for Khiops:
“Should I preprocess my data (e.g., encode categorical variables or normalize numerical ones) before using Khiops? Are there best practices for data cleaning when working with Khiops?”

[lucaurelien]

Khiops is specifically designed to work directly with raw data, minimizing the need for manual preprocessing. In fact, it is strongly recommended to avoid preparing your data to prevent loss of information, whether it concerns variable encoding, missing values, or data flattening (propositionalization).

Khiops employs the MODL formalism, based on the MDL (Minimum Description Length) principle, to encode variables optimally:

• Categorical variables: Khiops automatically groups values into clusters based on their correlation with the target variable.
• Numerical variables: Khiops uses discretization to partition values into intervals, ensuring that each interval represents a meaningful range of data relative to the target variable.

This optimal encoding is inherently tied to the modeling process and adapts dynamically to the data, making manual encoding methods unnecessary (and less effective !).

Example: Why Encoding Isn’t Necessary

Imagine a dataset with a categorical variable "city" containing hundreds of unique values. In a traditional workflow:

• One-hot encoding "city" would create hundreds of binary columns, resulting in a sparse and computationally expensive matrix.
• Frequency encoding might lose context, treating "city" as independent from the target variable.

With Khiops:

• "City" is automatically grouped into meaningful clusters (e.g., based on correlations with the target variable’s distribution), preserving interpretability and maximizing predictive power.

Similarly, for a numerical variable "age", Khiops would:

• Dynamically determine intervals (e.g., [0-18], [19-35], [36-50], [50-97]) based on the correlation between "age" and the target variable.
• Avoid creating unnecessary intervals, ensuring a balance between simplicity and accuracy.

Handling Missing Values

Khiops also treats missing values as part of the data, recognizing that they can carry meaningful information. Missing values are not discarded or imputed arbitrarily. Instead, Khiops assigns them to their own group or interval if they are informative for the target variable. This approach ensures that missing data is leveraged effectively, rather than being treated as noise or ignored outright.

Scientific Basis

Khiops’ preprocessing capabilities are grounded in rigorous statistical principles. The MODL formalism:

• Automatically balances model complexity and predictive information to prevent overfitting.
• Dynamically adapts to the structure of the data, selecting groupings or intervals only when they provide significant predictive value.
• Avoids arbitrary preprocessing decisions by tying encoding directly to the learning objective.

For more details and examples, see this notebook tutorial or read about Optimal Encoding.

This approach ensures that manual preprocessing is unnecessary (in fact, it is often counterproductive).