Running an Onboarding Friction Survey

The annual engineering survey asks "How satisfied were you with onboarding? (1–5)", the answer averages 3.4 every year, and nothing changes because nobody can tell what a 3.4 means or which step to fix. Meanwhile the specific frustrations — the certificate step that failed twice, the undocumented VPN requirement, the hour spent finding who owns the staging credentials — are forgotten by the time anyone asks. A friction survey designed around the actual setup steps, sent at the right moment and joined with timing data, produces a ranked list of fixes instead. This page builds one, as part of time-to-first-PR metrics.

Surveys complement telemetry: timing data shows where time went; the survey explains why, and catches friction that no script can measure, like unclear ownership or missing context.

Diagnostic

Look at what the current feedback process can actually answer:

#!/usr/bin/env bash
set -euo pipefail
jq -r '.questions[] | "\(.type)\t\(.text)"' surveys/onboarding-2025.json 2>/dev/null || echo "no structured survey on file"
jq '[.responses[] | .days_since_start] | {n: length, median: (sort | .[length/2|floor])}' surveys/onboarding-2025-responses.json 2>/dev/null || true
grep -c 'onboarding' docs/decisions/*.md 2>/dev/null | awk -F: '{s+=$2} END {print "decisions citing onboarding feedback:", s+0}'

Expected output that shows the gap:

scale	How satisfied were you with onboarding?
text	Any other comments?
{ "n": 11, "median": 212 }
decisions citing onboarding feedback: 0

Two generic questions, answered a median of seven months after starting, and no decision has ever referenced the results.

Root cause

Generic satisfaction questions measure mood, not friction. They cannot be traced to a step, a tool or an owner, so they cannot become work. Timing makes it worse: memory of specific problems fades within days, and by the time an annual survey arrives, "the certificate thing" has become "setup was a bit rough". Low response rates follow when surveys are long and nothing visibly changes after answering. Finally, survey results live in a separate tool from engineering work, so even a clear signal ("four of five new hires got stuck on VPN access") does not reach the backlog. A useful friction survey is short, timed to the moment, structured around the setup steps people actually went through, and connected to the same identifiers — bootstrap step names, runbook IDs — that the rest of the onboarding tooling uses.

Resolution

1. Ask step-level questions using the same step names as the bootstrap script and runbook, plus two open questions:

{
  "id": "onboarding-day3",
  "questions": [
    { "id": "steps_blocked", "type": "multi", "text": "Which setup steps blocked you for more than 15 minutes?",
      "options": ["check-tools", "env", "certs", "deps", "images", "seed", "vpn-access", "credentials", "none"] },
    { "id": "hours_to_green", "type": "number", "text": "Roughly how many hours until tests passed locally?" },
    { "id": "who_helped", "type": "multi", "text": "Who or what unblocked you?", "options": ["runbook", "doctor output", "buddy", "team channel", "figured it out alone"] },
    { "id": "worst_moment", "type": "text", "text": "What was the single most frustrating moment?" },
    { "id": "one_fix", "type": "text", "text": "If we could fix one thing before the next hire, what should it be?" }
  ]
}

Five questions take under three minutes, which keeps response rates high.

2. Send it at day 3 and day 30 automatically, triggered by the start date in the HR or directory system, with a shorter day-30 version focused on first-PR experience.

3. Join answers with bootstrap telemetry by step name, so "blocked on images" can be compared with measured image-pull times on the same platform:

SELECT s.step,
       count(*) FILTER (WHERE s.step = ANY(r.steps_blocked)) AS reported_blocked,
       percentile_cont(0.9) WITHIN GROUP (ORDER BY t.s) AS p90_seconds
FROM unnest(ARRAY['check-tools','env','certs','deps','images','seed']) AS s(step)
LEFT JOIN survey_day3 r ON true
LEFT JOIN bootstrap_steps t ON t.step = s.step AND t.received_at > now() - interval '90 days'
GROUP BY s.step
ORDER BY reported_blocked DESC;

4. Turn the top items into tickets and publish what changed, in the onboarding channel and in the next new hire's welcome message. Visible follow-through is what keeps response rates up.

Expected output

A quarterly summary that names steps and owners rather than scores:

Onboarding friction, Q3 (n=9 day-3 responses, 78% response rate)
step          blocked  p90 time   action
vpn-access    6 of 9   n/a        IT ticket auto-filed on hire date (done)
images        5 of 9   24 min     regional registry mirror (in progress)
certs         3 of 9   6 min      RB-004 runbook entry + doctor check (done)
seed          2 of 9   5 min      smaller default seed dataset (planned)
median hours to green tests: 6.5 (was 11 in Q2)

The survey identifies a problem telemetry could never see (VPN access requests), confirms one telemetry already showed (image pulls), and the median time to green tests shows whether fixes worked.

That combination is the point. Telemetry would have ranked image pulls first and missed VPN access entirely, because waiting for an access request produces no timing data in a bootstrap script. Survey answers alone would have been vague about how long image pulls really take. Together they give a prioritised list with evidence, which is far easier to get resourced than a general request to "improve onboarding".

Prevention

1. Keep the survey short. Resist adding questions; each one lowers the response rate. Rotate one experimental question per quarter if needed.

2. Keep option lists in sync with the bootstrap steps. When a step is renamed or added, update the survey options in the same pull request, so answers stay joinable with telemetry.

3. Review results at a fixed cadence — monthly or quarterly — with an owner who turns the top items into tickets; a survey without a review meeting becomes the annual score again.

Platform caveats

Distributed teams: schedule sends in the new hire's local time zone and working days; a survey arriving at 3 a.m. on a Sunday is ignored.

Small teams: with one or two hires a quarter, report individual answers only with consent and aggregate over longer periods to keep responses anonymous.

Contractors and transfers: include internal transfers and contractors, whose onboarding often differs and is rarely measured.

Rollback

Surveys are process, not code. To stop, disable the automated sends and archive the results; keep the step names in tooling, which remain useful for telemetry:

#!/usr/bin/env bash
set -euo pipefail
gh workflow disable onboarding-survey.yml
mkdir -p surveys/archive && git mv surveys/onboarding-day3.json surveys/archive/

Frequently Asked Questions

Related

• Instrument the bootstrap script with timing
• Measure onboarding time in distributed teams
• Reduce setup friction for junior engineers