Statistical Analysis Prompt Templates

Use a Chi-square test for independence (or equivalently, a two-proportion z-test).

Why this test:
- Categorical outcome (activated yes/no) with two independent groups
- Large sample size (n > 30 per group), so normal approximation works
- You're comparing proportions between groups

Alternative: Fisher's exact test if any cell count drops below 5, but with 2,500 per group that's unlikely.

Python code:
```python
from scipy.stats import chi2_contingency
import numpy as np

# Create contingency table
# [[control_activated, control_not], [test_activated, test_not]]
contingency = np.array([[320, 2180], [385, 2115]])

chi2, p_value, dof, expected = chi2_contingency(contingency)
print(f"Chi-square: {chi2:.2f}, p-value: {p_value:.4f}")
```

How to interpret: If p < 0.05, the difference in activation rates is statistically significant. But also calculate the actual difference (test rate - control rate) and confidence interval to understand practical significance.

```python
import pandas as pd
import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Prepare data: encode categorical variables
df_model = pd.get_dummies(df, columns=['plan_tier', 'referral_source'], drop_first=True)

# Define X and y
X = df_model[['months_active', 'support_tickets', 'plan_tier_Pro', 'plan_tier_Enterprise', 
 'referral_source_organic', 'referral_source_paid']]
X = sm.add_constant(X) # Add intercept
y = df_model['ltv']

# Fit model
model = sm.OLS(y, X).fit()
print(model.summary())

# Check multicollinearity
vif_data = pd.DataFrame()
vif_data['feature'] = X.columns
vif_data['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print(vif_data)
```

Interpretation guide:
- R-squared: What % of LTV variance is explained by these factors
- Coefficients: How much LTV changes per unit change in each variable
 - If months_active coef = 45, each additional month adds $45 to LTV
 - For plan_tier_Enterprise, the coefficient shows the difference vs. Basic (the reference)
- p-values: If < 0.05, that variable has a statistically significant relationship with LTV
- VIF: If > 5, you have multicollinearity concerns. Consider dropping one of the correlated variables

Your concern about plan_tier and months_active: check the VIF. If both are important, you might keep both but note the correlation in your interpretation.

```python
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.032 # 3.2%
new_rate = 0.0352 # 3.52% (10% relative lift)

# Calculate effect size (Cohen's h for proportions)
effect_size = proportion_effectsize(new_rate, baseline)

# Calculate required sample size per group
analysis = NormalIndPower()
n_per_group = analysis.solve_power(
 effect_size=effect_size,
 alpha=0.05,
 power=0.80,
 alternative='two-sided'
)

print(f"Required per group: {n_per_group:.0f}")
print(f"Total required: {n_per_group * 2:.0f}")
```

Result: You need approximately 23,500 per group, or 47,000 total visitors.

With 10,000 visitors per week split 50/50 between control and test, that's:
- 5,000 per group per week
- About 5 weeks to reach required sample size

Reality check: A 10% relative lift on 3.2% is a small absolute change (0.32 percentage points). Detecting small changes requires large samples. If you can only run for 2-3 weeks, consider either:
1. Testing a bigger change (maybe 20% relative lift)
2. Accepting lower power (higher risk of missing a real effect)
3. Using a different metric with more variance

Good instinct to question causality. Here's the analysis framework:

Possible explanations:
1. Tickets cause churn: Frustrating support experience drives people away
2. Underlying problem causes both: Product issues lead to tickets AND churn
3. Reverse causation: Customers planning to churn file tickets (maybe to get refunds or complain)

Analyses to run:

1. **Timeline analysis**: Do tickets come before or after signals of disengagement?
```python
# Check if tickets precede reduced usage
df['days_ticket_before_usage_drop'] = df['first_usage_drop_date'] - df['first_ticket_date']
```
If tickets come after usage drops, tickets are a symptom, not a cause.

2. **Control for confounders**: Run a regression with product usage metrics
```python
# Does ticket count still predict churn after controlling for usage?
model = logistic_regression(churn ~ tickets + monthly_logins + feature_adoption + plan_tier)
```
If the ticket coefficient drops significantly, usage is the real driver.

3. **Segment analysis**: Check if correlation holds across different groups
- Does it hold for all plan tiers?
- Does it hold for new vs. established customers?
- Does it hold for different ticket types (bugs vs. how-to questions)?

4. **Natural experiment**: Look for situations where ticket volume changed for external reasons
- Did a support policy change affect ticket volume without changing the product?
- How did churn respond?

My hypothesis: Tickets are likely a symptom of problems (explanation 2). The actionable insight isn't 'reduce tickets' but 'fix the issues that generate tickets.'