Feature Engineering

Raw features — House size, Bedrooms, Year built

Possible engineered features

• Age of house (current year − year built)
• Price per square foot
• Bedroom density (bedrooms / house size)
• Distance to city center

Useful — Study hours, Previous exam scores, Attendance rate

Less useful — Student ID, Favorite color, Random noise

• Relevant — Related to the prediction target
• Available — Present at the time of prediction
• Reliable — Consistent and not missing for most examples
• Distinct — Not duplicating another feature
• Interpretable — Understandable by humans

Raw feature — Year of birth: 1998

Engineered feature — Age: 2026 − 1998 = 28

Age is often more useful than birth year because the relationship between age and outcomes is usually more direct.

• Logarithm of a skewed variable
• Square root for count data
• Polynomial features (x², x × y)
• Ratio features (price per square foot)
• Interaction terms (study hours × attendance)

Original — House Size, Income, Age

• Normalization (scale to 0–1 range)
• Standardization (center at mean 0)
• Logarithm (for skewed data)
• Binning (group into ranges like Low/Medium/High)

Original — Color, Country, Product Category

• Label Encoding — Assign integer to each category
• One-Hot Encoding — Binary vector for each category
• Target Encoding — Replace with mean of label
• Frequency Encoding — Replace with how often it appears

Raw text — "This product is amazing!"

Engineered features

• Word count: 5
• Sentiment score: +0.9 (positive)
• TF-IDF vector representation
• Word embeddings (dense numerical vectors)

• Pixel intensity histograms
• Edge detection outputs
• Color distributions
• Object counts from detection models

Medical imaging — May use texture analysis or edge detection to find patterns not visible from raw pixels alone

• Year: 2026
• Month: 6 (June)
• Day of week: Tuesday
• Hour: 15
• Is weekend: No
• Season: Summer

Time features help models learn patterns related to cycles and seasonality.

Raw — "123 Main Street, Toronto"

• Latitude and longitude
• Distance to city center
• Distance to nearest school
• Neighborhood name
• Region or state

Study Hours × Attendance Rate → Combined Learning Effort

This can be more informative than either feature alone because both variables together describe effort better than either individually.

Price per square foot = price / house size

Clicks per visit = clicks / total visits

Revenue per customer = revenue / customers

House size: 1500 sq ft | Price: $300,000

Ratio: $300,000 / 1500 = $200 per sq ft

Age — 0–17: Child, 18–64: Adult, 65+: Senior

Income — 0–30k: Low, 30k–80k: Middle, 80k+: High

• Total purchases in last 30 days
• Average order value
• Number of product categories purchased

Aggregated features provide context that individual records alone cannot.

Healthcare — BMI = weight / height², Blood pressure ratio, Symptom duration in days

Finance — Debt-to-income ratio, Loan-to-value ratio, Credit utilization rate

Education — Grade point average, Completion rate, Attempts per assignment

Feature 1 — Age, range 18–80

Feature 2 — Income, range 20,000–200,000

Without scaling the model may give more weight to income simply because its values are larger.

Fix — Apply normalization or standardization

Customer income missing

• Fill with mean (if no major outliers)
• Fill with median (if outliers present)
• Fill with mode for categorical
• Create missing indicator: Age_missing = 1

A missing indicator feature tells the model that the fact that a value is missing may itself carry information.

Predicting — Whether the student will pass the final exam

Leaked feature — "Final exam score" (this IS the thing you are predicting — it cannot be an input feature)

Including information that would not be available at prediction time is data leakage.

Training accuracy — Very high (e.g., 99%)

Test accuracy — Much lower (e.g., 72%)

Adding too many features can cause the model to memorize training data rather than learn patterns that generalize.

1. Location — Most important

2. House size — Very important

3. Number of bedrooms — Moderately important

4. Age of house — Less important

Raw — Square footage, year built, zip code

Engineered — Age of house, price-per-sqft, distance to city center

Raw — Grades, attendance

Engineered — Average score trend, improvement rate, subject difficulty score

Raw — User click history, product categories

Engineered — Click frequency, recency score, category preference ratio

• Creating features that leak future information
• Creating too many redundant features
• Not scaling features when required
• Engineering features without domain knowledge
• Including features not available at prediction time

• Feature engineering creates better inputs for machine learning models
• Good features are relevant, available, reliable, and distinct
• New features can be created through transformations, ratios, and aggregations
• Domain knowledge helps identify the most useful features
• Always check for data leakage in engineered features
• Feature importance scores show which features helped the model most