Predictive Analytics Prompt Templates

**Forecasting Approach Recommendation**

**Recommended: Prophet or XGBoost with time features**

Given your requirements, here's my analysis:

**Why not simple methods:**
- Moving average: Won't capture seasonality or external factors
- ARIMA: Can handle seasonality but struggles with multiple external regressors
- Exponential smoothing: Good for seasonality but limited regressor support

**Why Prophet (Facebook's library):**
- Built for business time series with strong seasonality
- Handles holidays natively (perfect for your Q4 spike)
- Accepts additional regressors (marketing spend, promotions)
- Solid to missing data and outliers
- Interpretable components (trend, seasonality, holiday effects)
- 8-week horizon is well within its sweet spot

**Prophet implementation:**
```python
from prophet import Prophet
import pandas as pd

# Prepare data (Prophet requires 'ds' and 'y' columns)
df = pd.DataFrame({
 'ds': your_dates,
 'y': your_sales,
 'marketing_spend': your_marketing_data
})

# Initialize with seasonality
model = Prophet(
 yearly_seasonality=True,
 weekly_seasonality=True,
 seasonality_mode='multiplicative' # Better for sales data
)

# Add holidays
model.add_country_holidays(country_name='US')

# Add marketing as regressor
model.add_regressor('marketing_spend')

# Fit
model.fit(df)

# Forecast
future = model.make_future_dataframe(periods=8, freq='W')
future['marketing_spend'] = planned_marketing_values # You'll need to provide this
forecast = model.predict(future)
```

**Alternative: XGBoost with engineered features**

If Prophet doesn't hit your 15% accuracy target, try gradient boosting:

```python
import xgboost as xgb
from sklearn.model_selection import TimeSeriesSplit

# Feature engineering
df['week_of_year'] = df['date'].dt.isocalendar().week
df['month'] = df['date'].dt.month
df['is_holiday_week'] = df['date'].isin(holiday_dates)
df['lag_1'] = df['sales'].shift(1)
df['lag_4'] = df['sales'].shift(4)
df['rolling_4_mean'] = df['sales'].rolling(4).mean()

# Train with time-series cross-validation
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, val_idx in tscv.split(df):
 model = xgb.XGBRegressor()
 model.fit(X_train, y_train)
 # Evaluate on validation set
```

XGBoost advantages: Often more accurate, can capture complex interactions. Disadvantages: Less interpretable, requires more feature engineering.

**Evaluation plan:**

1. Split data: Train on first 2.5 years, validate on last 6 months
2. Forecast 8 weeks, measure MAPE (Mean Absolute Percentage Error)
3. Target MAPE < 15%
4. Check performance by category (some may be harder to predict)

**Watch out for:**
- Don't include future information in features (data leakage)
- Evaluate on recent data, not random samples (time series requires temporal splits)
- Check if marketing spend is known 8 weeks ahead (if not, you'll need to forecast that too or use planned values)

**Classification Model: Churn Prediction**

**Problem framing:**
- Imbalanced dataset (15% positive class)
- Asymmetric costs: FN costs $500, FP costs $20 (ratio 25:1)
- Goal: Catch churners for intervention, accept some false alarms

**Recommended approach: Gradient Boosting (XGBoost or LightGBM)**

Why: Handles imbalance well with class weights, provides feature importance for interpretability, strong performance on tabular data.

**Implementation:**

```python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
import xgboost as xgb
from sklearn.metrics import classification_report, precision_recall_curve

# Define feature types
numeric_features = ['tenure_months', 'monthly_charges', 'total_charges',
 'num_support_tickets', 'num_service_outages', 
 'last_nps_score', 'product_usage_score']
categorical_features = ['contract_type', 'payment_method']

# Preprocessing pipeline
preprocessor = ColumnTransformer([
 ('num', StandardScaler(), numeric_features),
 ('cat', OneHotEncoder(drop='first', sparse=False), categorical_features)
])

# Calculate class weight based on cost ratio
# FN (missing churner) is 25x worse than FP
# scale_pos_weight = negative_cases / positive_cases * cost_ratio_adjustment
scale_pos_weight = (1 - 0.15) / 0.15 # ~5.67, adjust based on cost if needed

# Model
model = Pipeline([
 ('preprocess', preprocessor),
 ('classifier', xgb.XGBClassifier(
 scale_pos_weight=scale_pos_weight,
 max_depth=5,
 learning_rate=0.1,
 n_estimators=100,
 eval_metric='aucpr' # Better than AUC for imbalanced data
 ))
])

# Train/test split (stratified to preserve class ratio)
X_train, X_test, y_train, y_test = train_test_split(
 X, y, test_size=0.2, stratify=y, random_state=42
)

# Fit
model.fit(X_train, y_train)

# Get probabilities instead of hard predictions
y_prob = model.predict_proba(X_test)[:, 1]
```

**Choosing the right threshold:**

Don't use default 0.5 threshold. Improve for your cost structure:

```python
from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, y_prob)

# Calculate cost at each threshold
costs = []
for thresh in thresholds:
 y_pred = (y_prob >= thresh).astype(int)
 fn = ((y_test == 1) & (y_pred == 0)).sum() # Missed churners
 fp = ((y_test == 0) & (y_pred == 1)).sum() # False alarms
 cost = fn * 500 + fp * 20
 costs.append(cost)

optimal_threshold = thresholds[np.argmin(costs)]
print(f"Optimal threshold: {optimal_threshold:.3f}")
```

**Evaluation metrics (in priority order):**

1. **Recall (Sensitivity)**: What % of actual churners do we catch? (Most important given high FN cost)
2. **Precision**: Of those we flag, what % actually churn? (Affects team workload)
3. **PR-AUC**: Better than ROC-AUC for imbalanced classes
4. **Cost-weighted accuracy**: Custom metric using your $500/$20 costs

**Feature importance analysis:**
```python
importances = model.named_steps['classifier'].feature_importances_
# Map back to feature names and visualize
```

Expected top predictors (based on typical churn models):
- tenure_months (short tenure = higher risk)
- num_support_tickets (more issues = higher risk)
- contract_type (month-to-month = higher risk)
- product_usage_score (low usage = higher risk)

**Deployment considerations:**

1. **Score customers weekly** and flag those above threshold
2. **Provide risk score, not just yes/no**: Let retention team prioritize
3. **Monitor model drift**: Churn patterns change; retrain quarterly
4. **A/B test the intervention**: Does outreach actually reduce churn?

**Feature Engineering: Trial-to-Paid Conversion**

**Time-based features (relative to signup):**

```python
# Days since signup
df['days_since_signup'] = (reference_date - df['signup_date']).dt.days

# Day of week signed up (some days convert better)
df['signup_dow'] = df['signup_date'].dt.dayofweek

# Time to first action (engagement speed)
df['hours_to_first_login'] = (df['first_login'] - df['signup_date']).dt.total_seconds() / 3600
```

**Engagement intensity features:**

```python
# Login patterns
df['total_logins'] = df.groupby('user_id')['login_event'].transform('count')
df['logins_week_1'] =... # Count in first 7 days
df['logins_week_2'] =... # Count in days 8-14
df['login_trend'] = df['logins_week_2'] - df['logins_week_1'] # Increasing or decreasing?

# Session depth
df['avg_session_duration'] = df.groupby('user_id')['duration'].transform('mean')
df['total_time_in_product'] = df.groupby('user_id')['duration'].transform('sum')

# Active days (not just count, but consistency)
df['unique_active_days'] = df.groupby('user_id')['login_date'].transform('nunique')
df['pct_days_active'] = df['unique_active_days'] / df['days_since_signup']
```

**Feature adoption features (often most predictive):**

```python
# Count of distinct features used
df['features_used_count'] = df.groupby('user_id')['feature_name'].transform('nunique')

# Key feature flags (identify your 'sticky' features)
df['used_feature_A'] = df['feature_name'].apply(lambda x: 'feature_A' in x).astype(int)
df['used_feature_B'] =... # Repeat for key features

# Feature velocity (how fast they explore)
df['features_used_day_1'] =...
df['feature_adoption_speed'] = df['features_used_count'] / df['days_since_signup']
```

**Support interaction features:**

```python
# Volume and sentiment indicators
df['support_tickets_count'] = df.groupby('user_id')['ticket_id'].transform('count')
df['had_support_ticket'] = (df['support_tickets_count'] > 0).astype(int)

# Issue severity (if categorized)
df['critical_tickets'] = df[df['ticket_category'] == 'critical'].groupby('user_id').size()

# Resolution experience
df['avg_resolution_time'] = df.groupby('user_id')['resolution_time'].transform('mean')
df['had_long_resolution'] = (df['resolution_time'] > threshold).astype(int)
```

**Email engagement features:**

```python
# Email responsiveness
df['emails_opened'] = df.groupby('user_id')['email_open'].transform('sum')
df['emails_clicked'] = df.groupby('user_id')['email_click'].transform('sum')
df['email_open_rate'] = df['emails_opened'] / df['emails_sent']
df['email_ctr'] = df['emails_clicked'] / df['emails_opened']
```

**Acquisition source features:**

```python
# One-hot encode referral source
df = pd.get_dummies(df, columns=['referral_source'], prefix='source')

# Or group into categories
df['is_organic'] = df['referral_source'].isin(['google_organic', 'direct']).astype(int)
df['is_paid'] = df['referral_source'].str.contains('paid|ad').astype(int)
```

**Derived/interaction features:**

```python
# Engagement per session (depth vs. breadth)
df['features_per_session'] = df['features_used_count'] / df['total_logins']

# Support burden (high support need = risk)
df['tickets_per_login'] = df['support_tickets_count'] / df['total_logins']

# Engagement quality score (composite)
df['engagement_score'] = (
 df['pct_days_active'] * 0.3 + 
 df['features_used_count'].clip(upper=10) / 10 * 0.4 +
 df['email_open_rate'] * 0.3
)
```

**Features to avoid (leakage risk):**

- Any data from after the prediction point (day 30)
- Direct indicators of conversion (payment attempts, pricing page views just before conversion)
- Features that are effects, not causes of conversion

**Feature selection approach:**

1. Start with all features
2. Check correlation matrix for redundant features (keep one if correlation > 0.8)
3. Use model feature importance to identify top 15-20
4. Test simpler model with fewer features against full model

**Expected top predictors (based on typical SaaS):**
1. Feature adoption (what they used, not just how much)
2. Engagement trend (increasing = good sign)
3. Time to first value (fast activation predicts conversion)
4. Acquisition source (some sources convert better)

**Validation Strategy: Loan Default Model**

**Key challenge**: This is a time-dependent problem where:
1. Future economic conditions may differ from training period
2. Model must generalize to truly unseen future data
3. Overfitting to historical patterns is a real risk

**Validation approach: Time-based holdout + out-of-time testing**

```
|----Training-----|--Validation--|----Test----|
2019 2020 2021 2022 2023 2024
 (3 years) (tune) (final eval) (monitor)
```

**Step 1: Temporal train/validation/test split**

```python
# Never use random split for time-dependent data!

# Training: 2019-2021 (3 years)
train = df[df['loan_date'] < '2022-01-01']

# Validation: 2022 (for hyperparameter tuning)
validation = df[(df['loan_date'] >= '2022-01-01') & (df['loan_date'] < '2023-01-01')]

# Test: 2023 (final, untouched evaluation)
test = df[df['loan_date'] >= '2023-01-01']

print(f"Train: {len(train)}, Val: {len(validation)}, Test: {len(test)}")
```

**Step 2: Time-series cross-validation for tuning**

```python
from sklearn.model_selection import TimeSeriesSplit

# Expanding window CV within training data
tscv = TimeSeriesSplit(n_splits=5)

for fold, (train_idx, val_idx) in enumerate(tscv.split(train)):
 X_fold_train = train.iloc[train_idx]
 X_fold_val = train.iloc[val_idx]
 # Train and evaluate
 # This simulates repeatedly training on past, predicting future
```

**Step 3: Stability analysis across time periods**

```python
# Check if model performance is consistent across time
for year in [2022, 2023]:
 year_data = df[df['loan_date'].dt.year == year]
 year_auc = roc_auc_score(year_data['default'], model.predict_proba(year_data)[:, 1])
 print(f"{year} AUC: {year_auc:.3f}")

# Big drops suggest model isn't stable across economic conditions
```

**Step 4: Segment validation**

```python
# Check performance doesn't degrade for subgroups
for segment in ['low_income', 'high_income', 'new_customer', 'existing_customer']:
 segment_data = test[test['segment'] == segment]
 segment_auc = roc_auc_score(segment_data['default'], 
 model.predict_proba(segment_data)[:, 1])
 print(f"{segment}: AUC={segment_auc:.3f}, n={len(segment_data)}")

# Watch for fair lending concerns if protected classes perform differently
```

**Step 5: Calibration check**

```python
from sklearn.calibration import calibration_curve

# Are predicted probabilities accurate?
fraction_of_positives, mean_predicted_value = calibration_curve(
 test['default'], model.predict_proba(test)[:, 1], n_bins=10
)

# Plot: Should be close to diagonal if well-calibrated
# If predicted 20% default, about 20% should actually default
```

**Step 6: Stress testing**

```python
# What if default rates increase (recession scenario)?
# Simulate by up-weighting defaults in test set

# What if key features become unavailable?
# Re-run model without each top feature, check performance drop

# What if feature distributions shift?
# Compare feature distributions between train and test
for feature in important_features:
 train_mean = train[feature].mean()
 test_mean = test[feature].mean()
 shift = (test_mean - train_mean) / train[feature].std()
 if abs(shift) > 0.5:
 print(f"Warning: {feature} shifted {shift:.2f} std devs")
```

**Ongoing monitoring plan (post-deployment):**

```python
# Track these weekly/monthly:
# 1. Prediction distribution (are scores shifting?)
# 2. Actual vs. predicted default rate
# 3. Feature distributions vs. training data
# 4. Model performance on recent cohorts (once outcomes known)

# Set alert thresholds:
# - AUC drops more than 0.03 from baseline
# - Predicted default rate diverges more than 20% from actual
# - Any feature drifts more than 1 std dev from training
```

**Red flags that suggest overfitting:**

1. Training AUC much higher than validation (0.95 vs 0.75)
2. Performance drops significantly on 2022 vs 2021 data
3. Model relies heavily on high-cardinality features (ZIP code, employer)
4. Small changes to hyperparameters cause big performance swings