Skip to content
Meir Dick
anon@meirdick
More in Lab
~/lab
Where next
~/lab/next
Lens, coaching in the moment
~/lab/lens

Lens, coaching in the moment

A coach on smart glasses for children with autism: it names the social moment, checks the child's plan and shows one short instruction.

learning plan

Goals: initiation, reciprocity, regulation. Full instructions. Coach greetings, answering and asking back.

Replaylunch at school · no microphone, free

    conversation

      the glasses

      ·

      decisions

        Try a lineClef-flash classifies it live

        Liveyour microphone · Nova-3 → Clef-flash → plan → cue

        Speak as a classmate might: "Hi, I'm Sam", "Do you want to play?"

        heard

          the glasses

          ·
          clip
          —
          speech
          —
          decide
          —
          cost
          $0.000000

          Audio goes to Workers AI. Not stored. Stops after 2 minutes.

          About this demo
          ~/lab/lens/notes

          What

          Lens is a social coach for children with autism, on smart glasses. It listens to a conversation, names the kind of moment, checks the child's own learning plan, and shows one short instruction the child can act on now.

          How

          model
          Nova-3 + Clef-flash
          id
          @cf/deepgram/nova-3@cf/cloudflare/clef-flash
          runs on
          Cloudflare Workers AI, through this site's Worker
          costs
          about $0.0006 per 3 s of audio; replay is free
          1. Classify. Nova-3 turns speech into words. Clef-flash names the moment with one of six categories from social-skills programs: initiation, reciprocity, reading the room, navigation, regulation and repair.
          2. Personalise. The child's plan decides if this moment is a goal, and how much help to give. A mastered skill gets no cue.
          3. Respond. The glasses show what to do, in at most three short lines. The cue gets shorter as the child improves.

          Why

          Earlier coaching glasses taught children to recognise emotions. The gap the research leaves is acting in context: what to do, right now, in this conversation. Lens puts the research methods into the moment, for one child's plan. The goal is a coach the child stops needing.

          Try

          • Play the replay on the cloud path, then on the edge path. Watch the delay.
          • Switch the plan to "Nearly independent" and replay. Most cues go quiet.
          • Type a line a classmate might say: "Can I sit with you?"

          Limits

          • The learners are made up. Real plans stay private.
          • Replay times are the design budgets, not live measurements. In a recorded test on the edge path, cues came 1.4 s after the moment, on average over 17 cues.
          • The live mic sends 3-second clips through a copy of the pipeline on this site. The production Lens is separate.
          • Lens never coaches a child to hide who they are.

          The pipelinedesign budget per step

          stepwhat happenswherebudget
          hearthe glasses send audio every 500 ms; Nova-3 turns it into words, with who said whatedge200–500 ms
          1 classifyClef-flash names the moment with a category from social-skills researchedgeabout 600 ms
          2 personalisethe child's learning plan decides: is this a goal, and how much help to giveDurable Objectunder 1 ms
          3 responda short, literal instruction appears on the glasses, then fades as the child improvesedge → glasses10–30 ms

          Moving the loop from the cloud to the edge cut the delay from 1.6–5.8 s to 1.2–3.0 s. The hosted version is SocialLens on Cloudflare. Next, the research moves out of the prompt and into the model's weights: thousands of example moments train a small model that runs on the glasses in under 100 ms. That is the pattern behind Callozum.

          Cues that fadefull → short → one word → a dot

          • initiation: "Say 'hi' back and their name." → "Hi + name" → "Greet" → "•"
          • reciprocity: "Answer, then ask one back." → "Answer, ask back" → "Ask back" → "•"
          • reading the room: "They look busy. Wait a moment." → "Wait a moment" → "Wait" → "•"
          • navigation: "Ask to join: 'Can I play too?'" → "Ask to join" → "Join" → "•"
          • regulation: "Breathe in slowly. Count to three." → "Slow breath" → "Breathe" → "•"
          • repair: "Say: 'Sorry, I meant…'" → "Fix it: 'I meant…'" → "Repair" → "•"

          The research7 sources

          PEERS Laugeson & Frankel, UCLA. Social-skills training with concrete steps for joining, conversation and repair.
          Social Stories Carol Gray. Short, literal descriptions of a situation and what to do in it.
          Naturalistic developmental behavioral interventions Schreibman et al., 2015. Teach inside real interactions, with prompts that fade.
          CBT for social anxiety in autism White et al., 2013; Wood et al., 2009. Regulation strategies for stressful moments.
          Theory of mind Baron-Cohen; Ozonoff. Why reading another person's state is hard, and how to scaffold it.
          Wearable coaching trials Voss et al., JAMA Pediatrics 2019; Sahin 2018; Keshav 2017. Glasses helped children recognise emotions; the gap left is acting in context.
          Camouflaging Hull et al., 2017. Why Lens never coaches a child to mask who they are.
          Contact
          ~/contact

          Write to me about the problem you want to fix.

          Write to me
          ~/mail/new
          Ctrl+Enter sends.
          1 – 5 switch workspace n b where next: the next step, or back Enter go to the menu ↑ ↓ Enter in the menu: move, open space / Ctrl K launcher: essays, builds, eras, themes, commands h l j k move focus: h l between columns, j k within one H L resize: move the split left or right J K resize: grow or shrink the focused side window f fullscreen the focused window w or q close the window under the mouse (reopen it from the launcher) t pick a theme m or c contact: write to me Esc leave a text box, the menu, fullscreen or a dialog Ctrl Enter send the message in the mail window ? this list