Seven data analysis mistakes that lead to bad business decisions
Most analysis mistakes aren't about math. They come from asking the wrong question of the data, or reading more into a result than it can support.
A dashboard can be accurate and still point a company the wrong way. The errors below show up again and again in business analysis, and each has a simple check.
1. Treating "statistically significant" as "important"
In March 2016 the American Statistical Association published a statement on p-values, one of the few times it has taken a formal position on a statistical practice. Its six principles include that a p-value does not measure the size of an effect or the importance of a result, and that business and policy decisions should not rest only on whether a p-value passes a threshold.
In practice: an A/B test on a large website can show a "significant" lift of 0.1%. That may not be worth the engineering cost. Report the size of the effect and a range, not only whether it passed.
2. Survivorship bias
Analyzing only the customers, projects or products that are still around skips the ones that failed. A study of "what our best customers have in common" says little unless you compare them with customers who churned.
Check: ask what's missing from the dataset, and why.
3. Simpson's paradox
A trend in combined data can reverse when you split it into groups. Conversion can rise in every region while falling overall, because the mix of traffic shifted toward a low-converting region.
Check: break important results down by the main segments before acting on the total.
4. Correlation read as cause
Sales and marketing spend may rise together because both follow the season. Without a controlled test or a clear mechanism, a correlation is a lead to investigate, not a finding.
5. A bad baseline
A 40% rise in sign-ups means little without knowing what normally happens that month. Compare against the same period last year, or against a control group, not only against last week.
6. Testing many things and reporting the winner
Run 20 comparisons and one will likely look significant by chance. The ASA statement makes the related point that sound inference needs complete reporting: say how many analyses were run, not only the one that worked.
7. Trusting the data without checking it
Duplicate records, time zone mismatches, changed definitions and tracking that broke for a week all produce confident wrong answers. Data quality checks belong at the start of an analysis, not after a surprising result.
What IT teams should do
- Require analysis reports to show effect size and a range, not only significance.
- Split headline results by main segments before they reach decision-makers.
- Add automated data quality checks for duplicates, gaps and definition changes in core datasets.
- Keep a log of tests run, so the losing experiments are counted too.