GitHub Says Copilot Makes You 55% Faster.
I Read The Study. It’s One JavaScript Task. With 35 Completers.
Subscribe to BSKiller for more uncomfortable tech truths
TL;DR: GitHub’s 55% figure comes from one standardized JS task with 35 completers. In the wild, data shows more clones, less refactoring, and modest acceptance rates. Use Copilot for boilerplate/tests but not for architecture/security.
Everyone’s boss just sent them that “developers are 55% faster with AI!” article. Nobody read past the headline. I did. I need therapy now.
The ENTIRE “55% faster” study:
• Task: Build an HTTP server (one standardized JS task via GitHub Classroom with tests)
• Recruited: 95 developers
• Timing analysis on: 35 completers
• Results: 71.17 min with Copilot vs 160.89 min without
• What they measured: Speed on ONE standardized task
• What they DIDN’T measure: Code quality
• The authors literally wrote: “This study does not examine code quality... productivity benefits may vary across tasks/languages”
But sure, revolutionize your enterprise based on 35 people doing a standardized JS task.
Meanwhile, in production (2025):
GitClear analyzed 211 MILLION lines of real code:
• Cloned code up sharply (5+-line dupes reported up to 8×)
• Refactoring down: ‘Moved’ lines fell from ~25% to <10% of changes
• Copy/paste now exceeds ‘moved’ code in GitClear’s dataset
Translation: More duplication, less cleanup. The technical debt compound interest calculator just exploded.
GitClear’s Full Analysis DevClass Coverage
Microsoft’s own guide notes: “Can take ~11 weeks for teams to fully realize satisfaction/productivity gains”
That’s 77 days of wondering if you’re using it wrong before maybe seeing benefits.
Accenture’s enterprise study found:
• Pull requests +8.69%
• Merge rate +15%
• Successful builds +84%
• Long-term code survival/defects not measured in that study
ZoomInfo’s ACTUAL production data (Jan 2025):
• 6,500 suggestions per day across entire org
• Acceptance rate: 33% of suggestions, 20% of lines
• Per-language: 14-32% acceptance (Go highest on small volume)
• Most accepted: The simplest possible completions
Copilot’s Greatest Hits:
✅ Writing boilerplate at light speed
✅ Generating tests that test nothing
✅ Creating 47 ways to do the same thing
✅ Comments that explain what but never why
✅ Variable names from a parallel universe
❌ Understanding your architecture
❌ Knowing your business rules
❌ Following your team’s patterns
❌ Security (what’s that?)
❌ Performance (lmao)
The timeline nobody talks about:
Week 1: “This is amazing!”
Week 4: “Why does it keep suggesting React in my Python file?”
Week 8: “I spend more time reviewing than coding”
Week 11: Microsoft says you might start seeing real benefits now
Week 12: Your tech debt has achieved sentience
Real conversation from last week:
CTO: “We’re going AI-first!”
Senior Dev: “Based on what data?”
CTO: “GitHub’s study shows 55% improvement!” Senior Dev: “In what?”
CTO: “...JavaScript HTTP servers” Senior Dev: “We don’t build those”
CTO: “It generalizes!” Senior Dev: Shows them the paper saying it doesn’t
CTO: “...We’re still doing it”
Fun fact GitHub buried: The “success rate” was only 7 percentage points higher with Copilot. Not statistically significant (95% CI [−0.11, 0.25]). Page 5 of the PDF. You’re welcome.
The math that matters:
Savings: 55% less time writing initial code (on one JS task)
Cost: Refactoring (’moved’ lines) fell from ~25% to <10% of changes
Reality: Up to 8× more copy-paste disasters
ROI: Ask me in 5 years when we’re still finding duplicates
Test this yourself:
Ask Copilot to implement your auth system
Count how many Stack Overflow answers from 2019 it combines
Show your security team
Watch them update their resumes
Use it for boilerplate. Love it for writing tests. Never trust it with anything important.
But definitely don’t restructure your entire engineering org based on 35 developers building an HTTP server in JavaScript.
📋 The Receipts Box:
✓ GitHub Study (n=35): arxiv.org/pdf/2302.06590 - Pages 3-5 for methodology, p.5 for non-significant success rate
✓ GitClear 211M lines: gitclear.com/ai_assistant_code_quality_2025_research
✓ Microsoft’s “11 weeks”: GitHub Copilot Adoption Guide, “Measuring Impact” section
✓ Accenture metrics: github.blog, “Research: Quantifying GitHub Copilot’s impact”
✓ ZoomInfo production: arxiv.org/pdf/2501.13282 - Table 2 for acceptance rates
All data January 2025. Errors mine. Opinions definitely mine. GitHub’s? In their own paper.
Want more uncomfortable truths about tech? Subscribe to BSKiller where I actually read past abstracts.
#AIReality #CopilotGate #ActualData #BSKiller