What 'technical enough' means now: evaluating an AI-assisted developer's real skill
A new evaluation problem has landed in co-founder searches: the portfolio arms race is over, because everyone's portfolio is suddenly impressive. AI-assisted builders can produce polished demos in a weekend — which is genuinely good — but it means shipped-product screenshots no longer separate a strong technical partner from someone who prompts well and understands little. Non-technical founders are the most exposed to this. Here's the evaluation that still works.
What no longer discriminates
- Working demos and slick UIs (table stakes now)
- Repo counts and green squares
- Framework name-dropping
- Speed on greenfield builds — AI makes everyone fast at the start
What still discriminates
1. Behavior at the failure boundary. Give them a broken thing, not a blank page: a bug that requires forming hypotheses across systems. AI-assisted or not — doesn't matter; let them use every tool they'd really use. What you're watching: do they interrogate the system, or re-prompt in circles and call the wall a dead end?
2. The "explain your own code" test. Pick a non-obvious file from their project: why is this structured this way? What breaks if we change this? Where's the ugliest part and why is it still there? Builders who understand their systems answer with tradeoffs and mild embarrassment. Builders who don't, answer with tour-guide narration — describing what's there rather than defending why.
3. Judgment about what not to build. Describe a feature request with a hidden simpler path. Strong engineers spend their scarcity — they find the 20-line answer, or push back on the premise. Weak ones enthusiastically generate the 2,000-line answer, because generation is free now and judgment still isn't.
4. Maintenance instincts. Ask what happens to this codebase at 10x users, or when a core dependency deprecates. The AI-era failure profile is systems that work until they suddenly don't; you're hiring for the person who sees around that corner.
Running this without technical skill yourself
Borrow the judgment: have your retainer engineer (or a paid senior friend-of-friend, one hour) sit in on test 1 and 2. And weigh the plain-language signal heavily — the candidate who can make you genuinely understand a technical tradeoff is demonstrating the exact skill a partnership with you requires.
None of this is anti-AI-tooling — the best candidates will use AI fluently in front of you. Fluency plus comprehension is the hire. Fluency alone is the trap.
What's your version of the failure-boundary test — and if you've been burned by the polished-demo problem, what was the tell in hindsight?
Replies (4)
Follow-up from maker intake: "What's a concrete broken-thing exercise I can reuse? I don't have a codebase of bugs lying around."
Use your own product's history: reintroduce (in a branch or copy) the worst bug you've actually had — you know its shape, its red herrings, and what fixing it required, which makes you a calibrated judge. No product yet? Public option: take a small open-source AI app, break one integration point deliberately (a schema mismatch between two services is ideal — it forces cross-system reasoning), and ask them to diagnose live in an hour, tools allowed. You're grading the questions they ask the system, not time-to-fix.
The 'mild embarrassment' tell in test 2 deserves its own comment, because it's the most reliable small signal I know. Every real system has parts its author dislikes; someone who built with comprehension knows exactly where theirs are and winces on approach. Someone who generated without comprehension presents every file with equal, cheerful confidence — the codebase has no shameful corners because they've never really been in its corners. Uniform confidence about a large system is, counterintuitively, the red flag.
Follow-up from maker intake: "Do credentials still mean anything — degrees, big-company experience?"
They mean what they've always meant, which was less than assumed: base-rate evidence of some rigor, no evidence of the specific skills a two-person AI startup needs (full-system ownership, judgment under resource constraints, shipping without infrastructure teams). Big-company engineers can be outstanding co-founders and can also be lost without the scaffolding they're used to — the four tests above separate those cases where the résumé can't. Treat credentials as a tiebreaker between candidates who both passed the tests, never as a substitute for them.
Follow-up from maker intake: "The candidate did great on all four tests but has never operated anything in production. Disqualifying?"
Not disqualifying — priceable. Production operation (monitoring, incidents, migrations, the 2am muscle) is learnable and the tests you ran measure the aptitude that predicts learning it. Mitigations for the gap: their first-quarter goals include instrumenting and running production properly; your retainer/audit checkpoints stay in place during the transition; and their onboarding includes shadowing one real incident response if you can arrange it. What you can't do is let 'great at building' silently stand in for 'ready to operate' — name the gap in the partnership plan and it stops being a risk.
Threads are permanent — locked, not deleted, once resolved. New posts go through a submission form and are published by moderators, usually within 1 business day.