IHM Academy
Performance Metrics Masterclass - Lesson 272: Feature Selection Without Overfitting
Coach Answer
Feature Selection Without Overfitting evaluates whether a hockey model is consistent, calibrated, reproducible and resistant to measurement error. It asks whether the number remains trustworthy when data quality, season environment or feature selection changes.
Extended Core Definition
Feature Selection Without Overfitting evaluates whether a hockey model is consistent, calibrated, reproducible and resistant to measurement error. It asks whether the number remains trustworthy when data quality, season environment or feature selection changes. The final layer of performance work is not collecting more numbers; it is deciding which numbers deserve trust. Hockey contains small samples, changing roles, uneven opponents, shifting score states and incomplete tracking. Any metric that ignores those conditions can look precise while giving the staff a weak decision.
This lesson treats methodology as part of coaching intelligence. Staff should know where the data came from, what denominator was used, how stable the estimate is, whether context changed, whether video agrees and what decision the metric is meant to support. A transparent process is more valuable than a complicated score nobody can explain.
Why This Metric Matters
Feature Selection Without Overfitting matters because the final stage of performance work is deciding how much confidence to place in a number. A metric that ignores sample size, context or validation can be precise in appearance and weak in meaning. The staff needs a method that distinguishes a real process change from random movement before the number becomes a coaching decision.
What the Metric Actually Measures
Start with hockey-relevant features, remove redundant variables, validate out-of-sample and prefer simpler models when added complexity does not improve stability.
The correct measurement form depends on the question. Raw totals describe workload, rates describe frequency, shares describe control, adjusted measures describe difficulty and uncertainty ranges describe confidence. One number should not be forced to answer all of those questions at once.
What It Does NOT Measure
This metric does not eliminate uncertainty or make coaching decisions automatic. Adjustments, models and dashboards are tools for organising evidence. They cannot remove judgement, role context or the possibility that underlying data is incomplete. No single result should be treated as total team or player value.
Inputs and Events Required
Useful inputs include raw event counts, denominators, minutes, possession opportunities, role, opponent, venue, score state, rest, event definitions, data completeness, model outputs and video tags. Record metadata about how the number was produced, not only the final value.
Measurement Model
Start with hockey-relevant features, remove redundant variables, validate out-of-sample and prefer simpler models when added complexity does not improve stability.
Every transformation should remain auditable. If a raw value becomes a rate, adjusted value, composite or model output, keep the steps available. Version event definitions and model assumptions so a change in the number can be separated from a change in methodology.
Step-by-Step Calculation or Tagging Method
- Define the hockey question before selecting the metric.
- Write the event definition and denominator.
- Check data completeness and tagging consistency.
- Calculate the raw result before any adjustment.
- Add only the contextual adjustment relevant to the question.
- Show event count and a confidence or uncertainty indicator.
- Compare multiple rolling windows or out-of-sample periods.
- Validate with systematically selected video.
- Translate evidence into change, keep, monitor or investigate.
- Log the decision and re-measure without changing the definition.
How to Read High, Average and Low Results
A strong result is more trustworthy when the sample is adequate, the definition is stable, independent metrics agree and video confirms the hockey process. A weak or uncertain result should be labelled as such. Do not interpret a leaderboard position without checking opportunity, role and environment.
A smaller but trustworthy signal is more useful than a dramatic unstable one. If the estimate is changing rapidly with every new event, the staff should describe it as provisional rather than use confident language that the data cannot support.
Team-Level Interpretation
At team level, separate executive metrics from diagnostic metrics. The head coach should see the few indicators that affect the next decision, while analysts keep deeper layers available for explanation. This prevents every post-game fluctuation from becoming an agenda item.
Player and Line-Level Interpretation
At player and line level, preserve role context and opportunity. Rates, shares and adjusted numbers should explain different parts of the profile rather than compete to become one universal ranking. Compare players within similar responsibility before making broader comparisons.
Context and Environment
Always check score state, venue, opponent, rest and role before comparing samples. The same raw value can mean something different if incentives or difficulty have changed. Context should refine the conclusion, not be used to explain away every unfavourable result.
Sample Size and Noise
Report event count and uncertainty beside the estimate. Short windows are good for detecting movement; medium and long windows test persistence. Rare-event metrics require more caution than high-frequency possession events, and percentages without denominators should never drive major decisions.
Common False Signals and False Positives
- A cleaner-looking number can still be wrong if the event definition is unstable.
- Large decimals can imply more certainty than the sample supports.
- Opportunity changes can move a metric without any change in efficiency.
- Context adjustments can overcorrect when the reference model is weak.
- Video review can confirm bias if clips are selected only to support the preferred story.
- Complexity can improve in-sample fit while reducing reliability on new games.
Video Validation: What Must Be Visible on Tape
Video should be sampled systematically, including ordinary and contradictory examples. The review should identify the hockey mechanism implied by the metric: space created, pressure escaped, support timing, defensive reaction, workload or role execution. If the mechanism cannot be found, investigate the model before coaching to the number.
Validation should include clips from the middle of the distribution, not only dramatic examples. Ordinary events test whether the metric describes repeatable hockey instead of highlight-reel exceptions.
Real-Game Scenario
Two models give different values to the same shot. One includes lateral pre-shot movement and one does not. The disagreement exposes different feature sets rather than automatically proving one model wrong.
The lesson is that measurement quality changes decision quality. Good staff work makes uncertainty visible early, before a noisy result becomes a confident story.
Coaching Application
Translate the framework into one of four actions: change, keep, monitor or investigate. Not every metric movement deserves intervention. Sometimes the correct coaching decision is to preserve the current role and wait for more evidence.
How This Changes a Staff Decision
Pause when the model is unstable, poorly calibrated or fed by inconsistent data. Fix the measurement system before changing the hockey system.
Repeatable Tracking Workflow
Use the same sequence every review cycle: define the question, collect and clean the data, calculate the raw result, add relevant context, expose uncertainty, validate with video, choose an action, log the action and re-measure. Keeping the order stable makes the process auditable.
Use a decision log so the staff can later see whether the expected process changed. Without a record, hindsight tends to rewrite why a decision was made and whether it actually worked.
Practice or Observation Drill
Staff exercise: take one conclusion and rebuild it from raw total, normalised rate, context-adjusted value and representative video. Ask each coach to state the conclusion before and after every layer. If the conclusion changes, document exactly which evidence changed it.
Red Flags and Corrective Actions
Red flags include metrics without denominators, adjusted values with no raw baseline, dashboards overloaded with correlated numbers, model changes without version control, conclusions from tiny samples and video chosen only to confirm a preferred story. Correct the measurement process before changing the hockey process.
Coach Mark Lehtonen Insight
A number is not intelligent because it has three decimals. It becomes useful when the staff knows what it measures, how stable it is, what hockey behaviour produced it and what decision should follow. The best performance system makes us less likely to overreact and more likely to recognise a real change early.
Quick Reference: Bench Card
Bench-card questions: Is the sample large enough? What is the denominator? Has context changed? Does video confirm the process? Which companion metric agrees or disagrees? What is the uncertainty? Does this require a change now, monitoring, or more investigation?
Glossary
- Sample size: The number of relevant observations supporting an estimate.
- Denominator: The opportunity base used to turn raw events into a rate or share.
- Calibration: How closely predicted probabilities match observed outcomes over large samples.
- Context adjustment: Accounting for differences such as opponent, role, venue or rest.
- Uncertainty: The plausible range around an estimate caused by limited information and variation.
- Validation: Testing whether a metric behaves as intended using independent data, video or future samples.
- Model drift: A change in data relationships that reduces the reliability of an older model.
- Decision log: A record linking evidence, staff action and later outcome.
End-of-Lesson Checklist
- Write the hockey question before choosing the metric.
- Show numerator, denominator and sample size.
- Keep raw and adjusted values visible together.
- Record the model or tagging version.
- Check opponent, score, venue, rest and role context.
- Make uncertainty or confidence visible.
- Validate with systematically selected video.
- Choose change, keep, monitor or investigate.
- Log the decision and expected process change.
- Re-measure with the same definition.
Questions & Answers | IHM Performance Metrics
What does Feature Selection Without Overfitting mean in hockey analytics?
Feature Selection Without Overfitting evaluates whether a hockey model is consistent, calibrated, reproducible and resistant to measurement error. It asks whether the number remains trustworthy when data quality, season environment or feature selection changes.
Why is uncertainty important in hockey metrics?
Because hockey events are noisy and many useful situations occur infrequently. A point estimate without sample information can make a temporary swing look permanent.
Should adjusted metrics replace raw results?
No. Keep both. Raw results show what happened; adjustments help explain difficulty and opportunity.
What should happen when video and data disagree?
Audit the event definition, tagging quality, sample, context and clip selection. Disagreement is a reason to investigate, not to automatically trust one source.
How can coaches avoid overfitting a dashboard?
Use a small hierarchy of metrics tied to real decisions, keep diagnostic layers underneath and remove numbers that duplicate the same process.
What makes a model useful to coaches?
Transparency, stable definitions, visible uncertainty, hockey-relevant inputs and a clear path from the number to an observable decision.
How often should the framework be reviewed?
The workflow can run weekly, while definitions and hierarchy should change less often unless role, data quality or competition environment changes.
What is the final purpose of performance metrics?
To improve hockey decisions: what to change, what to keep, what to monitor and what not to overreact to.
Key Takeaways
- Feature Selection Without Overfitting evaluates whether a hockey model is consistent, calibrated, reproducible and resistant to measurement error. It asks whether the number remains trustworthy when data quality, season environment or feature selection changes.
- Start with hockey-relevant features, remove redundant variables, validate out-of-sample and prefer simpler models when added complexity does not improve stability.
- A metric is only as trustworthy as its definition, denominator and data quality.
- Context and uncertainty should stay visible instead of disappearing inside one score.
- Video validation and out-of-sample review protect the staff from false confidence.
- The complete framework ends with a documented decision and re-measurement.