MinneMUDAC 2026 · this year's challenge NEW

This playbook was built on the 2025 case (BBBS Twin Cities). The 2026 competition is a brand-new problem with a new client — but the method you learn here is exactly what you'll run on the 2026 data. This page is your orientation and your personal to-do list on the road to Saturday, October 17, 2026.

This year's problem

Same three divisions as always — Novice Undergraduate Graduate — with teams of up to 5 students + 1 faculty/staff advisor. Any team with a graduate student is automatically in the Graduate division. Awards: Analytic Acumen, Serendipitous Discovery, and Overall Prediction (all divisions except Novice for the prediction award).

Key dates

What I need to do before Oct 17

Work top to bottom. The first two blocks are the only hard, time-boxed items — the rest is the month of actual project work, and this whole site is the how-to for it.

The MinneMUDAC 2025 Analytics Playbook

A step-by-step teaching case that takes you from a folder of raw files to a judge-ready data story — using the real 2025 MinneMUDAC datasets for Big Brothers Big Sisters Twin Cities (BBBS). Written for students who have not done a full analytics project before.

Three groups, one dataset, three depths

The competition splits students into three divisions by experience. This playbook is organized the same way, as three workshops that build on each other. Do them in order — each one assumes you finished the one before.

How to use this site

1. Read the orientation pages first

   Understand who the client is, what the files contain, and how you're graded. Skipping this is the #1 mistake.

2. Set up your tools

   Follow Set up Python & R once. Every code block has a Python / R toggle and a copy button.

3. Learn the mental model

   How an analyst thinks is the reusable EDA → ETL → model → communicate loop you'll apply everywhere.

4. Work your division's workshop

   Follow the numbered steps. Each ends with a rubric-mapped deliverable checklist so you know when you're done.

The client & the question

You can't analyze what you don't understand. Before a single line of code, learn the client's world and restate the challenge in plain language.

Who is the client?

The business problem, in one sentence

The two things they measure ("response variables")

What does "a successful match" even mean?

The challenge gives you four valid definitions of success. A strong team picks one (or reports several), states it explicitly, and analyzes against it. Don't leave this vague — judges reward a clear, defended definition.

1. Still active

   The match has not closed (Stage = "Active").

2. Lasts a long time

   A high Match Length (e.g. above the median of ~16.8 months, or above a policy target like 12 months).

3. Closed because the Little grew up

   Closure Reason relates to the Little aging out / graduating — a natural, positive ending.

4. Officially a "Successful match closure"

   That exact value in Closure Reason.

Restate the challenge as questions you can test

Translate the prompt into concrete, answerable questions. This list is your project plan.

Understand the data

The single most important — and most skipped — step. Know the files, their grain (what one row means), and their quirks before you touch a model.

The files in the folder

The most important idea: grain (long vs wide)

The response variables at a glance (real numbers)

Field families (68 fields, grouped so they're not scary)

The truncation trick (why the Test set is "unfair" on purpose)

Load the data and prove you understand it

Your very first script should print the shape, the grain, the response distribution, and missingness. Here's that "hello data" script in both languages.

# hello_data.py — first look at Novice.xlsx
import pandas as pd

df = pd.read_excel("Data/Novice.xlsx")

print("shape:", df.shape)                 # (3275, 66)
print("1 row = 1 match? unique IDs =", df["Match ID 18Char"].nunique())

# response variable summary
print(df["Match Length"].describe())
print(df["Stage"].value_counts(dropna=False))

# which columns are the most empty? (top 12)
missing = (df.isna().mean() * 100).sort_values(ascending=False)
print(missing.head(12).round(1))

# hello_data.R — first look at Novice.xlsx
library(readxl); library(dplyr)

df <- read_excel("Data/Novice.xlsx")

dim(df)                                  # 3275   66
n_distinct(df$`Match ID 18Char`)          # 3275 -> 1 row per match

summary(df$`Match Length`)
count(df, Stage)

# missingness by column (top 12)
sort(colMeans(is.na(df)) * 100, decreasing = TRUE)[1:12]

The rubric = your grade

Judges score you on a fixed rubric with weighted categories. The weights tell you exactly how much time and how many slides each topic deserves. Reverse-engineer it.

What each category actually asks for

Logistics that quietly cost points

• Presentation is 5–7 minutes, judged 4–6 times; expect interruptions with questions.
• No school branding (logos, lanyards, clothing) — impartiality rule.
• UG & Grad must submit exactly 300 predictions in Testset_Predictions_Submit.csv. A wrong/short file = prediction score of 1.
• Judging panel = a business pro + a technical pro + an academic. Speak to all three: impact, method, and rigor.

Set up Python & R

Do this once. You can use either language for the whole project — this playbook shows both so you can pick what your team knows.

Option 1 — Python (pandas ecosystem)

# In a terminal. Python 3.10+ recommended.
pip install pandas numpy openpyxl matplotlib seaborn scikit-learn
# Text mining (UG/Grad):
pip install nltk scikit-learn wordcloud vaderSentiment
# Optional stronger models:
pip install xgboost lightgbm

Option 2 — R (tidyverse ecosystem)

# In the R console / RStudio.
install.packages(c("tidyverse", "readxl", "janitor", "lubridate",
                   "tidytext", "textdata", "tidymodels", "ranger", "vip"))

A project folder that keeps you sane

The universal pipeline (all stages in one script)

Prefer to see the whole process wired together? starter_code/pipeline.py and starter_code/pipeline.R run every stage end-to-end on the real files:

load→ ETL / feature engineering → EDA→ text mining → model (RMSE)→ submission file

• Config-driven — set DATA_DIR and TEAM_ID at the top, then run.
• Fail-soft — each stage is wrapped; a missing file or library skips that stage with a message instead of crashing.
• Reusable — the same feature functions work on the Novice file (one row per match) and the long Training/Test files (one row per note) after an automatic collapse.
• Safe — read-only on your data; everything it writes lands in starter_code/outputs/ (EDA report + figure, and the 300-row Testset_Predictions_Submit.csv).

pip install pandas numpy openpyxl matplotlib scikit-learn
# optional: pip install seaborn vaderSentiment
python starter_code/pipeline.py

# install.packages(c("tidyverse","readxl","janitor","lubridate","tidytext","textdata","ranger"))
Rscript starter_code/pipeline.R

How an analyst thinks

Every workshop below is the same reusable loop. Learn it once and you can attack any dataset, not just this one.

1. Frame — turn the ask into questions

   Write the business question, the response variable, and a testable list of sub-questions (you did this on the client page). No framing → aimless charts.

2. ETL — get to tidy data

   Extract (load the files), Transform (fix types, parse dates, handle missing values, engineer features, set the right grain), Load (save one clean table). Rule of tidy data: one row per unit, one column per variable.

3. EDA — look before you model

   Univariate (each variable alone), then bivariate (variable vs the response), then multivariate. Distributions, group comparisons, trends over time, correlations. EDA is where the insights the rubric rewards come from.

4. Model (UG/Grad) — only after EDA

   Split train/validation, pick a model matched to the question, tune it, and measure honestly (RMSE for Match Length). Never judge a model on the data it trained on.

5. Interpret — from output to action

   Translate coefficients / feature importance into plain-language recommendations BBBS could act on. "So what?" is the whole point.

6. Communicate — the data story

   5–7 slides: problem → data → what we found → what to do → how confident we are. Weighted by the rubric.

ETL vs EDA — two different jobs (people confuse these)

Beginners often blur these together. They're distinct crafts that happen in sequence and then loop. ETL builds the table; EDA interrogates it; ML learns from it.

You don't pick one — they interleave

A real project bounces between building and exploring several times before any model is trained:

1. Micro-ETL

   Load raw files, fix column types, format timestamps, resolve delimiters.

2. First EDA

   Look at missing values, class balance, and raw distribution skew.

3. Deep ETL (feature engineering)

   Aggregate the longitudinal call logs to match level, join external tables, compute rolling/cadence metrics.

4. Focused EDA

   Inspect relationships between your engineered features and the target.

5. Machine learning

   Train models on the features that survived EDA — then loop back if they underperform.

How that maps to this BBBS challenge

Handling missing data — a reusable playbook

Real datasets are full of holes (this one has columns 70–99% empty). Don't guess — match the amount and reason for missingness to a strategy. Decide per column, not globally.

Picking a method from the question's shape

The Which method, when? reference expands this with code pointers.

Step 0 · Inspect any dataset ★

Before you clean, engineer, or model anything, you look. Initial inspection is a read-only triage that tells you what the data is — and what you find here decides every method you'll use next. This toolkit works on any dataset, not just this one.

The 5-command inspection toolkit

Run this on the raw file first, every time. Read the comment after each line — it names the question that line answers.

import pandas as pd
df = pd.read_excel("Test-Truncated-Restated.xlsx")

# 1) SCHEMA + MEMORY + DTYPES — are dates stored as text ('object')?
df.info()

# 2) MISSING-DATA AUDIT — % missing per column, worst first
missing = df.isnull().mean() * 100
print(missing[missing > 0].sort_values(ascending=False))

# 3) UNIT OF ANALYSIS — is 1 row a person? a match? a repeated call note?
print("Total rows:", len(df))
print("Unique matches:", df["Match ID 18Char"].nunique())

# 4) DISTRIBUTIONS — numeric spread vs categorical top values
print(df.describe(include="number"))   # mean, std, min, max
print(df.describe(include="object"))   # unique counts, top category

# 5) EYEBALL IT — spot formatting quirks the stats hide
print(df.head(5))     # first rows
print(df.sample(5))   # random rows

library(readxl); library(dplyr)
df <- read_excel("Test-Truncated-Restated.xlsx")

# 1) SCHEMA + DTYPES — glimpse shows type + first values per column
glimpse(df)                    # or skimr::skim(df) for a rich report

# 2) MISSING-DATA AUDIT — % missing per column, worst first
miss <- sort(colMeans(is.na(df)) * 100, decreasing = TRUE)
print(miss[miss > 0])

# 3) UNIT OF ANALYSIS
cat("Total rows:", nrow(df), "\n")
cat("Unique matches:", n_distinct(df$`Match ID 18Char`), "\n")

# 4) DISTRIBUTIONS
summary(select(df, where(is.numeric)))              # numeric spread
sapply(select(df, where(is.character)), n_distinct)  # categorical cardinality

# 5) EYEBALL IT
head(df, 5)
slice_sample(df, n = 5)

The decision matrix — inspection → method

This is the part people miss: what you observe in Step 0 dictates what you do in Steps 1–6. Read each observation as an "if → then." This table is dataset-agnostic — keep it next to you.

The target variable decides the model family

The single most important inspection question: what type is the thing you're predicting?

Your goal also picks your features: predictive vs descriptive

From inspection to action: the two findings that trigger Step 1

Inspection isn't busywork — specific findings force specific next steps. On this challenge, two diagnostics from Step 0 decide the entire shape of your ETL pipeline. (Both are verified against the actual files.)

The decision-logic flow

What Step 0 reveals about this BBBS data

Workshop A Novice — Explore & explain

Goal: clean Novice.xlsx and produce a clear, quantified story about what makes a BBBS match succeed. No heavy modeling — this is a masterclass in EDA done right.

Step 1 · ETL — load & clean

Load the file, parse the date columns, and build a success flag and an activation_year for the time analysis.

import pandas as pd, numpy as np

df = pd.read_excel("Data/Novice.xlsx")

# parse the dates we need
for c in ["Match Activation Date", "Match Closure Meeting Date"]:
    df[c] = pd.to_datetime(df[c], errors="coerce")

df["activation_year"] = df["Match Activation Date"].dt.year

# --- define "success" (definition #2: lasts beyond a policy target of 12 months) ---
# plus flag the official successful-closure category
df["success_long"]   = (df["Match Length"] >= 12).astype(int)
df["success_closure"] = (df["Closure Reason"] == "Successful match closure").astype(int)
df["is_active"]       = (df["Stage"] == "Active").astype(int)

print(df[["Match Length","activation_year","success_long"]].head())

library(tidyverse); library(readxl); library(lubridate); library(janitor)

df <- read_excel("Data/Novice.xlsx") |> clean_names()   # names -> snake_case

df <- df |>
  mutate(
    match_activation_date = as_date(match_activation_date),
    activation_year       = year(match_activation_date),
    success_long   = as.integer(match_length >= 12),
    success_closure= as.integer(closure_reason == "Successful match closure"),
    is_active      = as.integer(stage == "Active")
  )

glimpse(select(df, match_length, activation_year, success_long))

Step 2 · Distribution of the response variables

Start univariate. A right-skewed Match Length means you should report the median, not just the mean, and consider a log scale.

import matplotlib.pyplot as plt, seaborn as sns

fig, ax = plt.subplots(1, 2, figsize=(11,4))
sns.histplot(df["Match Length"], bins=30, ax=ax[0])
ax[0].set_title("Match Length (months) — right-skewed")

# top closure reasons (drop blanks = active matches)
top = df["Closure Reason"].value_counts().head(10)
top.sort_values().plot.barh(ax=ax[1])
ax[1].set_title("Top 10 closure reasons")
plt.tight_layout(); plt.savefig("outputs/response_dist.png", dpi=150)

library(scales)
# Match Length histogram
ggplot(df, aes(match_length)) +
  geom_histogram(bins = 30, fill = "#2a9d8f") +
  labs(title = "Match Length (months) — right-skewed", x = "months")

# top 10 closure reasons
df |> filter(!is.na(closure_reason)) |>
  count(closure_reason, sort = TRUE) |> slice_head(n = 10) |>
  ggplot(aes(n, reorder(closure_reason, n))) +
  geom_col(fill = "#1f5fa8") + labs(y = NULL, title = "Top 10 closure reasons")

Step 3 · Does it vary by Program Type?

This is a "number by category" question → grouped boxplots + a table of group medians.

# group medians (report median for skewed data)
print(df.groupby("Program Type")["Match Length"]
        .agg(["count","median","mean"]).round(1))

sns.boxplot(data=df, x="Program Type", y="Match Length")
plt.xticks(rotation=20); plt.title("Match Length by Program Type")

# formal test: do groups differ? (Kruskal-Wallis handles skew)
from scipy.stats import kruskal
groups = [g["Match Length"].values for _, g in df.groupby("Program Type")]
print("Kruskal-Wallis:", kruskal(*groups))

df |> group_by(program_type) |>
  summarise(n = n(), median = median(match_length),
            mean = mean(match_length)) |> arrange(desc(median))

ggplot(df, aes(program_type, match_length)) +
  geom_boxplot(fill = "#2a9d8f") +
  labs(title = "Match Length by Program Type")

# formal test (non-parametric, robust to skew)
kruskal.test(match_length ~ factor(program_type), data = df)

Step 4 · Has it changed over time?

yearly = (df.dropna(subset=["activation_year"])
            .groupby("activation_year")["Match Length"].median())
yearly.plot(marker="o", title="Median Match Length by activation year")
plt.ylabel("median months")

df |> filter(!is.na(activation_year)) |>
  group_by(activation_year) |>
  summarise(med = median(match_length)) |>
  ggplot(aes(activation_year, med)) +
  geom_line() + geom_point() +
  labs(title = "Median Match Length by activation year", y = "median months")

Step 5 · Which demographics matter?

Loop the response against several Big/Little demographic variables. Bin Big Age into ranges so it's comparable to the categorical ones.

df["big_age_band"] = pd.cut(df["Big Age"],
        bins=[18,25,35,45,100], labels=["19-25","26-35","36-45","46+"])

for col in ["big_age_band", "Big Gender", "Big Level of Education", "Big Car Access"]:
    print(f"\n=== {col} ===")
    print(df.groupby(col)["Match Length"].median().round(1).sort_values(ascending=False))

df <- df |> mutate(big_age_band = cut(big_age,
        c(18,25,35,45,100), labels = c("19-25","26-35","36-45","46+")))

for (col in c("big_age_band","big_gender","big_level_of_education","big_car_access")) {
  cat("\n===", col, "===\n")
  print(df |> group_by(.data[[col]]) |>
          summarise(med = median(match_length)) |> arrange(desc(med)))
}

Step 6 · The creative one — alignment features

The prompt explicitly asks whether alignment (same gender, similar hobbies) relates to success. This is where you earn "creativity" points. Engineer alignment flags, then compare.

# 1) same gender flag
df["same_gender"] = (df["Big Gender"] == df["Little Gender"]).astype(int)

# 2) shared-hobbies flag: the interest fields are ';'-separated lists
def to_set(x):
    return set(str(x).lower().split(";")) if pd.notna(x) else set()

big_h    = df["Big Contact: Interest Finder - Hobbies"].map(to_set)
little_h = df["Little Contact: Interest Finder - Hobbies"].map(to_set)
df["shared_hobbies"] = [len(a & b) for a,b in zip(big_h, little_h)]
df["any_shared_hobby"] = (df["shared_hobbies"] > 0).astype(int)

# compare
print(df.groupby("same_gender")["Match Length"].median())
print(df.groupby("any_shared_hobby")["Match Length"].median())

library(stringr)
overlap <- function(a, b) {
  A <- str_split(str_to_lower(coalesce(a, "")), ";")
  B <- str_split(str_to_lower(coalesce(b, "")), ";")
  mapply(function(x, y) length(intersect(trimws(x), trimws(y))), A, B)
}
df <- df |> mutate(
  same_gender     = as.integer(big_gender == little_gender),
  shared_hobbies  = overlap(big_contact_interest_finder_hobbies,
                            little_contact_interest_finder_hobbies),
  any_shared_hobby= as.integer(shared_hobbies > 0))

df |> group_by(same_gender)      |> summarise(median(match_length))
df |> group_by(any_shared_hobby) |> summarise(median(match_length))

Level-up: Jaccard similarity & one-hot interests

A raw count of shared interests is biased — a Big who lists 20 hobbies overlaps with everyone. Jaccard similarity normalizes by the size of the combined interest sets, so it measures genuine compatibility:

# Jaccard similarity for one interest category
def jaccard(a, b):
    a, b = to_set(a), to_set(b)            # to_set() from Step 6
    if not (a | b): return np.nan     # both empty -> undefined
    return len(a & b) / len(a | b)

df["hobby_jaccard"] = [jaccard(x, y) for x, y in
    zip(df["Big Contact: Interest Finder - Hobbies"],
        df["Little Contact: Interest Finder - Hobbies"])]

# bonus: explode a ';'-list into one-hot columns for modeling
sports = df["Big Contact: Interest Finder - Sports"].str.get_dummies(sep=";")
print(sports.shape, "one-hot sport columns")

jaccard <- function(a, b) {
  A <- to_set(a); B <- to_set(b)      # split ';' -> character vector
  u <- union(A, B); if (length(u) == 0) return(NA_real_)
  length(intersect(A, B)) / length(u)
}
df$hobby_jaccard <- mapply(jaccard,
  df$big_contact_interest_finder_hobbies,
  df$little_contact_interest_finder_hobbies)

Step 7 · Rubric-mapped deliverable checklist

Workshop B Undergraduate — Text + Predict

Everything in Workshop A, on the bigger Training-Restated.xlsx, plus three new skills: reshaping note-level data, mining the check-in notes, and building a Match-Length model scored by RMSE.

Step 1 · The reshape you must get right

Training is long (one row per note). To do the Novice-style analysis and to model at the match level, collapse it to one row per match while turning the notes into features.

import pandas as pd, numpy as np
notes = pd.read_excel("Data/Training-Restated.xlsx")
notes["Completion Date"] = pd.to_datetime(notes["Completion Date"], errors="coerce")
notes = notes.sort_values(["Match ID 18Char", "Completion Date"])

# --- note-level features, aggregated per match ---
g = notes.groupby("Match ID 18Char")

feat = pd.DataFrame({
    "n_notes":       g.size(),
    "first_note":    g["Completion Date"].min(),
    "last_note":     g["Completion Date"].max(),
    "note_chars":    g["Match Support Contact Notes"].apply(
                        lambda s: s.astype(str).str.len().mean()),
})
# call cadence = average days between consecutive notes
def cadence(s):
    d = s.dropna().sort_values().diff().dt.days
    return d.mean()
feat["avg_days_between_calls"] = g["Completion Date"].apply(cadence)

# bring the match-level (constant) columns over — take the first row per match
const_cols = ["Program Type","Big Age","Big Gender","Little Gender",
              "Match Activation Date","Match Length"]
match = g[const_cols].first().join(feat)
print(match.shape)   # ~3275 rows, one per match

library(tidyverse); library(readxl); library(janitor); library(lubridate)
notes <- read_excel("Data/Training-Restated.xlsx") |> clean_names() |>
  mutate(completion_date = as_date(completion_date)) |>
  arrange(match_id_18char, completion_date)

match <- notes |>
  group_by(match_id_18char) |>
  summarise(
    n_notes    = n(),
    first_note = min(completion_date, na.rm = TRUE),
    last_note  = max(completion_date, na.rm = TRUE),
    note_chars = mean(nchar(coalesce(match_support_contact_notes, ""))),
    avg_days_between_calls = mean(diff(sort(completion_date)), na.rm = TRUE) |> as.numeric(),
    # match-level constants: take the first value
    program_type = first(program_type), big_age = first(big_age),
    big_gender = first(big_gender), little_gender = first(little_gender),
    match_length = first(match_length), .groups = "drop")
dim(match)   # ~3275 x ...

Step 2 · Does call cadence affect Match Length?

import seaborn as sns, matplotlib.pyplot as plt
m = match.dropna(subset=["avg_days_between_calls","Match Length"])

sns.regplot(data=m, x="avg_days_between_calls", y="Match Length",
            scatter_kws={"alpha":.2}, lowess=True)
plt.title("Do longer gaps between calls shorten matches?")

print(m[["avg_days_between_calls","Match Length"]].corr(method="spearman"))

m <- filter(match, !is.na(avg_days_between_calls))

ggplot(m, aes(avg_days_between_calls, match_length)) +
  geom_point(alpha = .2) + geom_smooth() +
  labs(title = "Do longer gaps between calls shorten matches?")

cor(m$avg_days_between_calls, m$match_length, method = "spearman")

Step 3 · Mine the check-in notes (text)

The notes look like "Question: Activities: Answer: BB and LB have been…". Clean the prompt scaffolding, tokenize, and find words associated with long vs short matches.

import re
from sklearn.feature_extraction.text import TfidfVectorizer

# join all notes per match into one document
docs = (notes.groupby("Match ID 18Char")["Match Support Contact Notes"]
             .apply(lambda s: " ".join(s.dropna().astype(str))))

# redaction placeholders carry NO signal (see Client Q&A) -> remove them
REDACT = r"b_firstname|l_lastname|phone_number|email"
# role/process acronyms: drop the pure-noise ones, keep meaningful stems
NOISE  = r"\b(bb|lb|bp|msc|mec|mc|fc|pg|kj)\b"

def clean(t):
    t = t.lower()
    t = re.sub(r"question:|answer:", " ", t)   # drop scaffolding
    t = re.sub(REDACT, " ", t)               # drop redacted names/contacts
    t = re.sub(NOISE,  " ", t)               # drop anonymized role tags
    return re.sub(r"[^a-z ]", " ", t)

docs = docs.map(clean)

tfidf = TfidfVectorizer(stop_words="english", max_features=400,
                        ngram_range=(1,2), min_df=20)
X = tfidf.fit_transform(docs)

# correlate each term with Match Length
import numpy as np, pandas as pd
y = match.loc[docs.index, "Match Length"].values
corr = np.asarray([(np.corrcoef(X[:,j].toarray().ravel(), y)[0,1])
                   for j in range(X.shape[1])])
terms = pd.Series(corr, index=tfidf.get_feature_names_out()).dropna()
print("words linked to LONG matches:\n", terms.sort_values(ascending=False).head(15))
print("words linked to SHORT matches:\n", terms.sort_values().head(15))

library(tidytext); library(stringr)

clean_txt <- function(t) t |>
  str_to_lower() |>
  str_replace_all("question:|answer:", " ") |>                       # scaffolding
  str_replace_all("b_firstname|l_lastname|phone_number|email", " ") |> # redactions -> no signal
  str_replace_all("\\b(bb|lb|bp|msc|mec|mc|fc|pg|kj)\\b|[^a-z ]", " ")   # role tags + punct

tok <- notes |> transmute(match_id_18char,
          txt = clean_txt(coalesce(match_support_contact_notes, ""))) |>
       unnest_tokens(word, txt) |> anti_join(stop_words, by = "word")

# term frequency per match, join to length, correlate
term_len <- tok |> count(match_id_18char, word) |>
  left_join(select(match, match_id_18char, match_length), by = "match_id_18char") |>
  group_by(word) |> filter(n() > 20) |>
  summarise(r = cor(n, match_length)) |> arrange(desc(r))
head(term_len, 15); tail(term_len, 15)

Early vs late notes

The prompt asks whether language differs between early and late stages. Tag each note by its position in the match's sequence, then compare word frequencies.

notes["rank"]  = notes.groupby("Match ID 18Char").cumcount() + 1
notes["total"] = notes.groupby("Match ID 18Char")["rank"].transform("max")
notes["phase"] = np.where(notes["rank"]/notes["total"] <= 0.34, "early", "late")
# then run the same tokenize+count, grouped by phase, and compare top terms

notes <- notes |> group_by(match_id_18char) |>
  mutate(rank = row_number(), total = n(),
         phase = if_else(rank/total <= 0.34, "early", "late")) |> ungroup()
# tokenize, count by phase, and compare with a log-odds or simple ratio

Turn findings into the coordinator "drop-down" questions

Deliverable (required for UG): a small set of check-in questions with limited drop-down answers that flag at-risk matches. Each question maps to a data-driven risk signal, and specific answers act as risk triggers. A judge-ready version looks like a coordinator triage tool:

Keep it to 4–5 questions with closed picklists so answers are consistent and scoreable — a coordinator can complete it during a monthly check-in call.

Step 4 · Predict Match Length (the scored part)

Build a regression at the match grain. Validate honestly with a hold-out split and RMSE. Start simple (linear), then try a random forest / gradient boosting.

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline

# choose TEST-SAFE features (knowable early, not truncation-dependent)
num = ["Big Age","avg_days_between_calls","note_chars"]
cat = ["Program Type","Big Gender","Little Gender"]
data = match.dropna(subset=["Match Length"]).copy()
X, y = data[num+cat], data["Match Length"]

pre = ColumnTransformer([("cat", OneHotEncoder(handle_unknown="ignore"), cat)],
                        remainder="passthrough")
model = Pipeline([("pre",pre),
    ("rf", RandomForestRegressor(n_estimators=400, random_state=42))])

Xtr,Xva,ytr,yva = train_test_split(X, y, test_size=0.2, random_state=42)
model.fit(Xtr, ytr)
pred = model.predict(Xva)
rmse = mean_squared_error(yva, pred, squared=False)
print("validation RMSE:", round(rmse, 2))  # compare vs baseline below

# baseline = predict the mean for everyone; you MUST beat this
baseline = mean_squared_error(yva, [ytr.mean()]*len(yva), squared=False)
print("baseline (mean) RMSE:", round(baseline, 2))

library(tidymodels); set.seed(42)

data <- match |> filter(!is.na(match_length)) |>
  select(match_length, big_age, avg_days_between_calls, note_chars,
         program_type, big_gender, little_gender)

split <- initial_split(data, prop = 0.8)
tr <- training(split); va <- testing(split)

rec <- recipe(match_length ~ ., data = tr) |>
  step_impute_median(all_numeric_predictors()) |>
  step_novel(all_nominal_predictors()) |>
  step_dummy(all_nominal_predictors())

rf <- rand_forest(trees = 400) |> set_engine("ranger") |> set_mode("regression")
fit <- workflow(rec, rf) |> fit(tr)

augment(fit, va) |> rmse(match_length, .pred)      # your RMSE
# baseline you must beat:
sqrt(mean((va$match_length - mean(tr$match_length))^2))

Leakage-safe validation: GroupKFold + synthetic truncation

This is what separates strong models from ones that look great locally and collapse on the real test set. Two rules: (1) never let one match land in both train and validation folds — group on Match ID 18Char; (2) the test notes were truncated, so truncate your training folds the same way before computing features, or your model learns from "future" notes it won't have.

from sklearn.model_selection import GroupKFold
import numpy as np

def truncate_notes(notes_df, frac_range=(0.2, 0.6), seed=42):
    """Drop the last 20-60% of each match's notes to mimic the test set."""
    rng = np.random.default_rng(seed)
    keep = []
    for mid, g in notes_df.groupby("Match ID 18Char"):
        g = g.sort_values("Completion Date")
        frac = rng.uniform(*frac_range)              # how much to chop
        n_keep = max(1, int(len(g) * (1 - frac)))
        keep.append(g.iloc[:n_keep])
    return pd.concat(keep)

# cross-validate: truncate INSIDE each training fold, then build features
gkf = GroupKFold(n_splits=5)
groups = match_ids                                  # Match ID per row of your match-level X
scores = []
for tr_idx, va_idx in gkf.split(X, y, groups):
    # 1) truncate the raw notes for the training matches, 2) rebuild features,
    # 3) fit on the truncated-derived features, 4) score on the held-out matches
    model.fit(X.iloc[tr_idx], y.iloc[tr_idx])
    p = model.predict(X.iloc[va_idx])
    scores.append(np.sqrt(mean_squared_error(y.iloc[va_idx], p)))
print("CV RMSE:", np.mean(scores).round(2), "±", np.std(scores).round(2))

Step 5 · Produce the submission file (exactly 300 rows)

Retrain on all training data, predict the test matches, and write the CSV in the exact required format. A malformed file scores a 1 automatically.

# 1) build the SAME features for the test file (one row per match)
test = pd.read_excel("Data/Test-Truncated-Restated.xlsx")
# ... repeat the Step-1 aggregation on `test` to get test_match (300 rows) ...

# 2) refit on ALL training rows, predict test
model.fit(X, y)
test_pred = model.predict(test_match[num+cat])

# 3) write the required 4-column file
sub = pd.DataFrame({
    "RowID": range(1, len(test_match)+1),
    "MatchIDChar": test_match.index,
    "PredictedMatchLength": np.round(test_pred, 2),
    "YourTeamID": "U50",          # <-- your assigned Team ID
})
assert len(sub) == 300, "must be exactly 300 predictions!"
sub.to_csv("outputs/Testset_Predictions_Submit.csv", index=False)

# build test_match the same way as `match`, then:
final <- workflow(rec, rf) |> fit(data)      # refit on ALL training
preds <- predict(final, test_match)$.pred

sub <- tibble(
  RowID = seq_len(nrow(test_match)),
  MatchIDChar = test_match$match_id_18char,
  PredictedMatchLength = round(preds, 2),
  YourTeamID = "U50")
stopifnot(nrow(sub) == 300)
write_csv(sub, "outputs/Testset_Predictions_Submit.csv")

Step 6 · Rubric-mapped deliverable checklist

Workshop C Graduate — Deep NLP + external data

Everything in A & B, plus: sentiment/emotion in the notes, modeling Closure Reason, mining Rationale for Match & Closure Details, joining U.S. Census data by Block Group, and an early-intervention strategy.

Step 1 · Sentiment & emotional tone of notes

Move beyond keywords to sentiment (positive/negative) and emotion. Track how sentiment trends across a match — a downward slope is an early-warning signal.

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()

notes["sent"] = notes["Match Support Contact Notes"].fillna("") \
                    .map(lambda t: sia.polarity_scores(t)["compound"])

# per-match sentiment features (test-safe: use EARLY notes only)
early = notes[notes["rank"] <= 3]
sent_feat = early.groupby("Match ID 18Char")["sent"].agg(["mean","min"])
sent_feat.columns = ["early_sent_mean","early_sent_min"]

# sentiment SLOPE across the match (early -> late)
import numpy as np
def slope(g):
    if len(g) < 3: return np.nan
    return np.polyfit(g["rank"], g["sent"], 1)[0]
sent_slope = notes.groupby("Match ID 18Char").apply(slope)

library(tidytext)
afinn <- get_sentiments("afinn")          # word -> score

note_sent <- notes |> transmute(match_id_18char, rank,
      txt = coalesce(match_support_contact_notes, "")) |>
  unnest_tokens(word, txt) |> inner_join(afinn, by = "word") |>
  group_by(match_id_18char, rank) |> summarise(sent = mean(value), .groups="drop")

# early-note sentiment + slope per match
sent_feat <- note_sent |> group_by(match_id_18char) |>
  summarise(early_sent = mean(sent[rank <= 3]),
            slope = if(n() >= 3) coef(lm(sent ~ rank))[2] else NA)

Step 2 · Model Closure Reason (classification)

Closure Reason has ~30 values. Group them into a few actionable buckets (Volunteer-side, Child/Family-side, Agency, Successful, COVID) and classify.

def bucket(r):
    if pd.isna(r): return "Active"
    r = str(r)
    if r.startswith("Volunteer"): return "Volunteer"
    if r.startswith("Child"):     return "Child/Family"
    if r.startswith("Agency"):    return "Agency"
    if "Successful" in r:       return "Successful"
    return "Other"

df["closure_bucket"] = df["Closure Reason"].map(bucket)
print(df["closure_bucket"].value_counts())

# then a RandomForestClassifier on demographic + alignment + early-note features,
# evaluated with a confusion matrix and macro-F1 (classes are imbalanced).

bucket <- function(r) case_when(
  is.na(r) ~ "Active",
  str_starts(r, "Volunteer") ~ "Volunteer",
  str_starts(r, "Child")     ~ "Child/Family",
  str_starts(r, "Agency")    ~ "Agency",
  str_detect(r, "Successful") ~ "Successful",
  TRUE ~ "Other")

df <- df |> mutate(closure_bucket = bucket(closure_reason))
count(df, closure_bucket, sort = TRUE)
# classify with a ranger workflow; evaluate with conf_mat() + f_meas()

Step 3 · Mine Rationale-for-Match & Closure Details

Two extra text fields. Rationale for Match (why the match was made) may predict longevity; Closure Details explains why matches actually ended — gold for designing the coordinator questions. Apply the same tokenize/TF-IDF/sentiment toolkit from Workshop B to each, grouped by closure_bucket.

Step 4 · Join U.S. Census data (external data = rubric points)

The data ships Census Block Group IDs for the Little's home, the Big's home, and the Big's employer (Community program). Join public Census tables (income, education, etc.) to test whether neighborhood context relates to match success. Data must be free & public.

1. Decode the Block Group ID

   A 12-digit FIPS code = State(2) + County(3) + Tract(6) + Block Group(1). Split it into parts.

2. Get a free Census table

   American Community Survey (ACS) 5-year, via the free Census API (api.census.gov) or the tidycensus R package (needs a free key) / census Python package.

3. Join on the Block Group key

   Attach median household income, % bachelor's+, etc. to each match.

4. Test the relationship

   Does Match Length differ by neighborhood income band? Add the strongest Census feature to your model and report the RMSE change.

5. Compute Big↔Little distance

   Convert each block group to a lat/long centroid, then take the Haversine distance between the Big's and Little's block groups. For Community matches (where the volunteer drives), commute distance is a strong, intuitive predictor of whether meetings keep happening.

# split the 12-digit block group FIPS into components
bg = df["Little Mailing Address Census Block Group"].astype("string").str.zfill(12)
df["state"]  = bg.str[:2]
df["county"] = bg.str[2:5]
df["tract"]  = bg.str[5:11]
df["bgrp"]   = bg.str[11:]

# then pull ACS median income (B19013_001E) for MN block groups from the free API,
# and merge on [state, county, tract, bgrp].

library(tidycensus)    # census_api_key("YOUR_FREE_KEY") once
mn_income <- get_acs(geography = "block group", state = "MN",
                     variables = c(med_income = "B19013_001"), year = 2022)
# GEOID is the 12-digit block group key -> join to your data
df <- df |> mutate(GEOID = str_pad(little_mailing_address_census_block_group, 12, pad="0")) |>
  left_join(select(mn_income, GEOID, med_income = estimate), by = "GEOID")

Step 5 · The early-intervention strategy (the payoff)

Grad's headline deliverable: an early-intervention model + a plan BBBS can run. Combine everything into a risk score computed from information available early in a match.

Make the intervention framework concrete

Judges reward specificity. Map your top predictive drivers to named BBBS programs and a trigger, so the recommendation is something staff could operationalize on Monday:

Advanced method: survival analysis for censored matches

• Kaplan–Meier curves — visualize "retention over time" by Program Type or risk group (a great slide).
• Cox Proportional Hazards — which factors raise/lower the hazard of closing, with interpretable hazard ratios.
• Random Survival Forests — non-linear survival prediction on your full feature matrix.

Tools: lifelines (Python) or survival/survminer (R). Even if your submitted prediction is a plain RMSE regression, showing a survival view demonstrates real methodological depth to the academic judge.

Step 6 · Rubric-mapped deliverable checklist

From the client Q&A ★

The written handouts don't tell you everything. In the live Q&A, BBBS staff (Daniel Glass, Pat, Dan Atkins, John Malone) shared operational realities, decoded the note jargon, and — crucially — described how teams actually win. Treat this page as insider context that most teams won't have.

1 · Operational cadence — the real call schedule

2 · Decode the notes — acronyms & redaction placeholders

The check-in notes are full of shorthand. Knowing it lets you (a) clean the text correctly and (b) engineer smarter features. Redaction placeholders carry zero predictive power — strip them or they'll pollute your TF-IDF vocabulary.

3 · Participant lifecycle & rematches

4 · Program model — Site vs Community changes which features matter

5 · How teams actually win (judging dynamics)

6 · Using AI tools — allowed, with transparency

The organizers addressed AI directly in the Q&A: you may use it, but you must own your work and disclose how you used it. Judges are explicitly instructed to ask teams about their AI use, so prepare for the question rather than dodging it.

How to handle it well

• Keep a short "AI usage" note (a backup slide or appendix): which tools, for what tasks (e.g. "used an LLM to draft the text-cleaning regex; we reviewed and tested it").
• Make sure every team member can explain any part a judge points to — features, metric, a specific code block.
• Frame AI as leverage on your own thinking: "we hypothesized X, used AI to prototype it faster, then validated it on the data."
• Prefer free, reproducible tools; don't lean on expensive managed services that a judge would read as buying an unfair advantage.

Which method, when?

A quick reference so you always reach for the right tool — and can defend it to the "methods" judge.

Validation discipline (the difference between UG/Grad winners and the rest)

• Always keep a hold-out (or cross-validation). Never report training-set performance.
• Always beat a baseline (predict the mean). If you don't, your features aren't working.
• Set a seed so numbers are reproducible.
• For the truncated test set, ask of every feature: "knowable early?" If not, drop it.
• Report RMSE in months so it's interpretable ("we're off by ~9 months on average").

Pitfalls & judge tips

Data traps specific to this competition

Presentation tips (from the guidelines)

• Time yourself to 5–7 minutes. Judges cut you off; front-load your best insight.
• Round 1 is a laptop table pitch to 3–4 judges, 4–6 times — no projector until finals. Use big fonts, one idea per slide, and few dense tables (readable at arm's length on a laptop).
• Match slide count to rubric weights — ~25% of slides on completeness/outcomes.
• Speak to all three judges: business impact, method correctness, rigor.
• Every member speaks (team synergy is graded).
• Lead with the "so what" for BBBS, not with your tech stack.
• Have a backup: WiFi may be slow — don't rely on live/online demos.
• Be ready to explain your AI use — judges are instructed to ask. Keep a short "AI usage" note and make sure everyone can defend their own work. See From the client Q&A.
• No school branding anywhere (slides, clothing, lanyards).
• Expect interruptions — rehearse handling a mid-slide question and returning to flow.

Do's & Don'ts for MinneMUDAC success

A 5–7 minute story skeleton that fits the rubric

1. The ask (15s)

   "BBBS wants longer, healthier matches and early warning of closures."

2. The data (30s)

   Files, grain, response variables — prove you understand it.

3. What we found (2–3 min)

   Top drivers of success, quantified; your best 2–3 charts.

4. The model (UG/Grad, 1 min)

   Approach, validation, RMSE vs baseline, key features.

5. What BBBS should do (1–2 min)

   Concrete actions + the coordinator drop-down questions / intervention plan.

6. Confidence & limits (20s)

   What you're sure of, and caveats (censoring, truncation).

Team GitHub workflow

How the team writes code together without stepping on each other. The goal: no one's work is ever lost, everyone gets full commit credit, and main stays clean and presentation-ready.

Three rules that never bend

• Never commit directly to main. Code enters only via a reviewed PR.
• Work in your own feature branch. All development, EDA, and testing happen there.
• Commit history = your credit. Commit often with clear messages under your own identity.

Step-by-step

1. Clone the repo & set your identity (once)

   Clone, then make sure your commits carry your name and GitHub email.

   bash

   Copy

   git clone https://github.com/<your-org-or-team-lead>/minnemudac-2026.git
   cd minnemudac-2026
   
   # stamp every commit with your identity (use your GitHub email)
   git config user.name  "Your Name"
   git config user.email "your-github-email@example.com"

   Tip: drop the --global flag (as above) so this identity applies to this repo only — handy if you share a machine.

2. Create a personal feature branch

   Always branch off an up-to-date main. Name it feature/<your-name>-<task>.

   bash

   Copy

   # 1) get the latest main
   git checkout main
   git pull origin main
   
   # 2) branch off it
   git checkout -b feature/alex-census-etl

3. Develop & commit safely

   Commit frequently with descriptive messages, then push your branch to GitHub.

   bash

   Copy

   git status                              # see what changed
   git add src/etl_census.py               # stage specific files (not "git add .")
   git commit -m "feat: census block group demographic parser"
   git push -u origin feature/alex-census-etl

   Message convention Start messages with a type: feat: (new), fix: (bug), docs:, refactor:, eda:. It makes the history skimmable and looks professional to judges.

4. Open a Pull Request for review

   On GitHub, click the Compare & pull request banner. In the description say what you changed and how you tested it, then assign a teammate as Reviewer.

   [your feature branch] ──(Pull Request + review)──▶ [protected main]

5. Review, merge, clean up

   A teammate reviews the Files changed tab, comments, and Approves. Merge with Squash and merge (tidy history) or Create a merge commit — both keep author attribution. Then delete the merged branch.

   bash

   Copy

   git checkout main
   git pull origin main                    # bring the merged code down
   git branch -d feature/alex-census-etl   # delete your local branch

Repository protection (set once by the team lead)

In GitHub → Settings → Branches → Add branch ruleset for main:

Jupyter notebook rules (avoid corruption)

• Personal notebooks for exploration: notebooks/eda_alex.ipynb, notebooks/eda_taylor.ipynb — one per person.
• Move reusable logic into src/: feature functions, the ETL pipeline, and NLP cleaners live as .py modules and get imported into notebooks.
• Optional but great: add nbstripout so notebook outputs aren't committed — smaller diffs, fewer conflicts.

Adding your presentation to the repo

The slide deck is your final deliverable, so it belongs in the repo like everything else — but a .pptx/.key file is binary. Git can store it, but it can't merge it, so two people editing the deck at once corrupts it exactly like a notebook. Treat the deck as a single-owner file.

Where it lives

presentation/
├── MinneMUDAC2026_TeamName.pptx   # the editable source (single owner)
├── MinneMUDAC2026_TeamName.pdf    # exported PDF = the reviewable "source of truth"
├── figures/                       # charts pulled in from outputs/ (PNG/SVG)
│   ├── match_length_by_program.png
│   └── sentiment_slope.png
└── notes/
    └── talk_track.md              # speaker notes / who-says-what, plain text = diffable

Choose how you author it

Commit & version it like code

# the deck owner works on a branch, same as any change
git checkout main && git pull origin main
git checkout -b feature/alex-slides-v1

# add the deck AND its exported pdf
git add presentation/MinneMUDAC2026_TeamName.pptx presentation/MinneMUDAC2026_TeamName.pdf
git commit -m "docs: draft slides — findings + model + recommendations"
git push -u origin feature/alex-slides-v1
# then open a PR so the team reviews the PDF before it hits main

• Don't rename to death. Keep one filename (..._TeamName.pptx) and let git hold the history — no final_v2_REALLYfinal.pptx. Git already tracks every version.
• Tag the submitted deck. When you submit, mark it: git tag -a submission-v1 -m "Round 1 deck" (or cut a GitHub Release and attach the PDF). That freezes the exact judged version.
• Big media? If you embed video or many large images, enable Git LFS for *.pptx/*.mp4 so the repo stays fast: git lfs track "*.pptx".
• Pull figures from outputs/. Copy the final charts your pipeline produced into presentation/figures/ and reference those — so the deck and the analysis never drift apart.

Suggested repo layout

minnemudac-2026/
├── src/                    # shared, importable logic (one owner per file is easiest)
│   ├── etl.py              #   loading, cleaning, feature engineering
│   ├── text.py             #   note cleaners, TF-IDF, sentiment
│   └── model.py            #   train, validate, predict, write submission
├── notebooks/              # personal EDA — one notebook per person
│   ├── eda_alex.ipynb
│   └── eda_taylor.ipynb
├── presentation/           # the deck (.pptx) + exported .pdf + figures + talk track
│   ├── MinneMUDAC2026_TeamName.pptx
│   ├── MinneMUDAC2026_TeamName.pdf
│   └── figures/
├── outputs/                # figures + Testset_Predictions_Submit.csv (gitignored)
├── Data/                   # raw xlsx — NEVER committed (gitignored)
├── .gitignore
└── README.md

Starter .gitignore

Keep raw data, secrets, and environments out of the repo. A ready-to-use copy ships in this project as .gitignore.sample — rename it to .gitignore in your repo.

# --- raw + generated data (never commit competition data) ---
Data/
*.xlsx
*.csv
outputs/

# --- secrets / API keys (Census etc.) — keep these private ---
.env
*.key
secrets.*

# --- python ---
.venv/
__pycache__/
*.pyc
.ipynb_checkpoints/

# --- R ---
.Rhistory
.RData
.Rproj.user/

# --- presentation autosave (keep the .pptx + .pdf, drop the cruft) ---
~$*.pptx
~$*.docx

# --- OS / editor cruft ---
.DS_Store
.vscode/