- JavaScript 99.4%
- Python 0.4%
The shared instructions and skills in /cosmoshot/skills changed, so the managed section of this repository's agent instructions and the skill folders are brought level with them. Changed: AGENTS.md. Text outside the managed markers is untouched. |
||
|---|---|---|
| .agents/skills | ||
| .claude/skills | ||
| .github/workflows | ||
| demo | ||
| docs | ||
| faces | ||
| src | ||
| tests | ||
| tools | ||
| .gitignore | ||
| .releaserc.cjs | ||
| AGENTS.md | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| contract-request.md | ||
| LICENSE | ||
| package.json | ||
| program.json | ||
| PROVENANCE.md | ||
| README.md | ||
| THIRD-PARTY-NOTICES.md | ||
lipsync
Approaches to lip sync: matching a character's mouth to speech. Exploratory work that began before the Cosmoshot programme and is now developed under it. It may never be used by an application; it is kept as a reference and a place to try things.
Part of ai of our own (https://aiofourown.com). Licence: AGPL-3.0 (LICENSE). When this goes public it is mirrored to
the programme forge, https://forge.aiofourown.com, and this README and PROVENANCE.md stay with it.
Where the history begins and who wrote what is in PROVENANCE.md.
Scope
The library is pure functions: no DOM, no Web Audio, no network, no files written. It supports en-US only, has no
dependencies, and is licensed AGPL-3.0-only. The plan is docs/CORE-PLAN.md.
Coarticulation: the next sound starts before this one ends
weightsAt(segments, t, options) (and createMouth) use model: 'dominance' by default: src/coarticulation.js, coarticulate(segments, t, options). It follows the idea of dominance functions (Cohen and Massaro, "Modeling Coarticulation in Synthetic Visual Speech"; the technique
only, no code is copied, see PROVENANCE.md). Every sound has, for each group of articulators, a dominance over time: its alpha while it
holds (the central plateau fraction of its segment), falling off exponentially before it (anticipation: the mouth is already forming
the sound) and after it (carryover: it still looks a little like the sound that ended). A channel is the dominance-weighted average of the
targets of the sounds around t. The mouth never snaps, lip rounding starts during the consonant before an "oo" and lingers after it, and
bilabials (P B M) and labiodentals (F V) have a high and narrow dominance on their closure channels, so they still seal and tuck exactly.
The result is { articulation (13 channels), openness, viseme, visemeWeights }; visemeWeights is the dominance share per Oculus label,
at most three labels summing to 1, so a morph-target face cross-fades. Silence: the answer is null more than 0.6 s before the first
sound (COARTICULATION_LEAD_SECONDS) and more than 0.5 s after the last (COARTICULATION_TAIL_SECONDS); in between the gaps, the lead and
the tail are virtual pauses that relax the mouth to rest (below 0.05 at the start and 0.03 at the end of that range). en-US only, like the rest.
Styles (STYLES, option style, default 'soft'; or give intensity, 0 to 2): Soft (intensity 0.5, windows x1.15, the default),
Natural (0.7, x1.0) and Crisp (1.0, x0.8). The intensity scales the jaw, the lips' shape and the tongue by sqrt(intensity / 0.6)
(the calibrated table is the posture at 0.6) and never the closure channels, so P, B and M seal at 1 and F and V tuck at 1 in every style. On
the three demo sentences at 85 ms a sound the jaw's 95th percentile and maximum are 0.26 and 0.36 in Soft, 0.32 and 0.46 in Natural and
0.42 and 0.61 in Crisp (the legacy blend reaches 0.65 on "ah"; tests/coarticulation.test.mjs holds the bounds). { model: 'legacy' } keeps
the first blend exactly (a 50 ms ramp between neighbours and the speachtest table), so tools/check-parity.mjs can still compare it with
Sonalon's driver and tests/fixtures/legacy-weights.json keeps it honest.
The parameters, DOMINANCE in src/coarticulation.js (alpha, lead ms, lag ms; each row there has a comment with the phonetic reason):
| Class | Sounds | Plateau | Jaw (alpha, lead, lag ms) | Shape | Closure | Tongue | Viseme (alpha, reach ms) |
|---|---|---|---|---|---|---|---|
| vowelOpen | AA AE AH AW AY | 0.7 | 1 / 65 / 80 | 0.7 / 90 / 90 | 0.15 / 45 / 45 | 0.35 / 125 / 125 | 1 / 110 / 120 |
| vowelMid | EH ER EY OW AO OY | 0.7 | 1 / 65 / 80 | 0.9 / 120 / 100 | 0.15 / 45 / 45 | 0.35 / 125 / 125 | 1 / 110 / 120 |
| vowelClose | IY IH UH UW | 0.7 | 0.9 / 65 / 80 | 1.2 / 140 / 100 | 0.15 / 45 / 45 | 0.35 / 125 / 125 | 1 / 110 / 120 |
| glide | W Y R | 0.5 | 0.35 / 80 / 100 | 1.5 / 140 / 110 | 0.15 / 45 / 45 | 0.5 / 125 / 125 | 0.9 / 110 / 120 |
| lateral | L | 0.4 | 0.3 / 80 / 100 | 0.35 / 100 / 100 | 0.15 / 45 / 45 | 3 / 125 / 125 | 0.9 / 110 / 120 |
| bilabial | P B M | 0.4 | 5.5 / 50 / 60 | 0.35 / 100 / 100 | 6 / 11 / 14 | 0.3 / 125 / 125 | 1.1 / 100 / 110 |
| labiodental | F V | 0.45 | 1.2 / 60 / 70 | 0.5 / 100 / 100 | 5 / 14 / 18 | 0.3 / 125 / 125 | 1.1 / 100 / 110 |
| dental | TH DH | 0.5 | 0.4 / 70 / 80 | 0.4 / 100 / 100 | 0.15 / 45 / 45 | 3 / 125 / 125 | 1 / 110 / 120 |
| alveolar | T D N S Z | 0.4 | 0.3 / 70 / 80 | 0.35 / 100 / 100 | 0.15 / 45 / 45 | 3 / 125 / 125 | 0.9 / 110 / 120 |
| postAlveolar | SH CH JH | 0.5 | 0.35 / 80 / 90 | 1 / 130 / 100 | 0.15 / 45 / 45 | 0.4 / 125 / 125 | 1 / 110 / 120 |
| velar | K G NG | 0.4 | 0.25 / 80 / 90 | 0.25 / 100 / 100 | 0.15 / 45 / 45 | 3 / 125 / 125 | 0.8 / 110 / 120 |
| glottal | HH | 0.3 | 0.08 / 70 / 80 | 0.1 / 100 / 100 | 0.05 / 45 / 45 | 0.05 / 125 / 125 | 0.3 / 110 / 120 |
| pause | pause | 1 | 0.6 / 90 / 100 | 0.7 / 90 / 100 | 0.2 / 30 / 60 | 3 / 125 / 110 | 0.8 / 110 / 130 |
The sound shapes. ARTICULATION in src/data.js is the calibrated table: a jaw of 0.72 on "ah" at the widest (the legacy table: 0.92),
0.04 to 0.16 on consonants, spread only a little, rounding only where the lips round, closures 1 and a tuck of 1. The audit and the looks
that set the numbers are the comment above the table. The speachtest table remains as ARTICULATION_LEGACY (with legacyArticulationFor)
because the legacy model and the parity check against Sonalon need exactly those values; the parity tool calls the legacy model explicitly.
Kinematics: the mouth has mass
src/kinematics.js gives each articulator group a second-order spring-damper (jaw about 130 ms, lips about 70 ms, tongue about
45 ms to reach 90 percent of a step; LEGACY_GROUPS has the first, quicker values: 90, 50 and 35 ms), so the mouth glides
and has weight on top of the dominance blend. Use
bakeTrajectory(segments, { style }) once per utterance and trajectoryAt(baked, t) for exact random access (scrubbing either way is
deterministic), or createSmoother().step(channels, dt) for live streaming; sample the target lookaheadMs (26 ms) ahead of the
audio clock to cancel the spring's lag. Closures still seal: lips and the lower-lip bite use a contact model, and the jaw closes
faster while the lips close. smoothVisemeWeights does the same for morph-target weights, and { enabled: false } switches
smoothing off everywhere (the result then equals weightsAt exactly). en-US only, like the rest. The parameters and the measured
guarantees are in the header of src/kinematics.js and in tests/kinematics.test.mjs.
Movement scale and subtle head motion (Step 20)
The owner's request of 2026-11-18: all movement down to 30 percent, and a head that moves ever so slightly in six degrees of freedom. Both are in the library, so the 2D and the 3D face get them alike (en-US only, like the rest).
movementScale(0 to 1, default 0.3,src/movement-scale.js; option ofweightsAt,createMouthandbakeTrajectory) scales every articulation channel and the openness after the coarticulation and the springs. The contact channelslipCloseandlowerLipBitekeep the value that makes the lips touch (v * (scale + (1 - scale) * v^3)), so P, B, M, F and V still seal.{ movementScale: 1 }is the model's own numbers (the parity tool and the model tests use it).- Head motion (
src/head-motion.js,src/pitch.js):bakeHeadMotionorbakeHeadMotionFromAudioonce per clip,headPoseAt(baked, t)for the six numbers (pitch, yaw, roll in degrees; x, y, z in millimetres) that a face applies to its head's root only (face.setHead, seefaces/CONTRACT.md; the 2D rig does it with one svg transform). It is called SUBTLE, never "natural": the peaks at gain 1 are 0.4 degrees (pitch, yaw), 0.3 (roll), 0.3 mm (x, y), 0.4 mm (z), about 5 percent of what a real talker does, chosen for a tight close-up.gain0 to 5 (default 1), per-axis scale, a seed, and ZERO underprefers-reduced-motion. It listens to the loudness and the pitch (a deterministic YIN tracker,trackPitch) of the audio: slow drivers, at most one small nod per phrase, an occasional postural shift, a slow drift and breathing; never a sustained tilt, a left-right swing or a rhythm. Pure and deterministic per seed. The numbers, the budget (the mouth centre moves at most about 12 px at 1080 px wide at gain 1) and the checks are infaces/3d/README.mdanddocs/DEMO-PLAN.md.
Two faces, and a face you can customise
The repository holds two sibling faces behind one contract (faces/CONTRACT.md): the vector face faces/2d/ (an svg rig, no dependencies) and the three.js head
faces/3d/. Both take the library's frames and head pose the same way, and neither is a fallback for the other. The demo page has a labelled Face: 2D or 3D switch
(3D is the default where the browser can run it and Save-Data is off; the 3D code and head load only when wanted; a visitor's own choice is remembered in this browser)
and a Customise the face group: six skin tones, a face shape slider (Feminine, Neutral, Masculine quick picks), stubble, five lipsticks with a finish, the lipstick's thickness (Thin, Medium, Thick: a thick lipstick hides the irregularities of the lip beneath it) and its application (Even or Natural), blush, Reset and
a link that holds only these settings (faces/look.mjs, #tone=4&shape=0.5&lips=rose; the capture tool's face block reads and writes the same look). The 2D face draws every
setting; the 3D head draws the face shape with four morph targets (lips, cheeks, jaw angle, chin width: faces/3d/README.md, "The face shape"), so every control changes both faces (a face that reported capabilities.faceShape: false would get a note under the slider). faces/2d/README.md documents how each setting is drawn.
Acoustic mode: the mouth from sound only (Step 23a)
The owner's request of 2026-11-18: besides text-to-speech, the demo must also work from the person's OWN recorded or uploaded audio, which has no text.
src/acoustic.js is the library side (en-US only, like the rest; the page parts are a later step):
acousticAnalysis(audio, sampleRate)gives per-frame (10 ms) features: loudness, voicing and F0 (the YIN tracker ofsrc/pitch.js), zero crossings, a crude F1 and F2 and a low-versus-highbalance(linear prediction at 8 kHz), silence, plosive-like bursts and the closures before them.acousticTrajectory(audio | analysis, sampleRate, opts)bakes the clip with the same springs, movement scale (default 0.3, contacts keep contact) and fields asbakeTrajectory, sotrajectoryAt, the faces (2D and 3D), the head motion and the capture tool take it unchanged.acousticWeightsAt(analysis, t)is the unsmoothed mouth at t, shaped likeweightsAt.- What it does: the jaw follows the loudness with an attack (25 ms) and a release (90 ms), a little higher for a high F1; the lips round for a low F2 or a low-heavy balance and spread for a high one; the lips close over the 80 ms of silence before a burst and release at it; silence is the rest mouth.
- What it does NOT do, honestly: it recognises no phoneme and reads no lips. It cannot tell /p/ from /t/ (every burst closes the lips), /s/ from /f/, or a vowel from its neighbour when the formants are close; the formant estimate is crude (within about 100 Hz on clean synthetic vowels, worse on real, noisy or whispered speech, and the balance shifts with the voice pitch). Loudness is scaled to the clip's own 95th percentile, so a clip with only quiet speech still opens fully, and background noise louder than about 3 percent of that reads as sound. Music or several voices will move the mouth to the loudness, not to any speech.
- It is NOT expected to match the text path, which knows the phonemes and their timing; the two are different products and no parity is tested. The result
is labelled
alignment: 'acoustic-phone-alignment'(the label align.js reserved for acoustic work) withalignmentNote: 'from sound only ...': no phone is recognised, so do not hand this mouth to a consumer that needs phone timing. 60 s of audio is analysed in about 0.15 s (tests/acoustic.test.mjs prints it).
Word-anchored alignment (Step 26)
alignPhonemes spreads phonemes by where the audio is loud, which drifted by up to a word against a real recording. When the time of each word is known
(a transcript tool, typed by hand, or the TTS model), alignWithWords(words, audio, sampleRate, opts) (src/anchors.js, en-US only) uses it:
words is [{ text, start, end }] in seconds. Each word goes through the text lexicon (phonemizeToken; a word outside the 65-word starter lexicon uses the letter fallback
and is listed in unknownWords of alignWithWordsReport), its phonemes are spread inside [start, end] by the same energy quantiles restricted to that interval, and a gap between
words is one pause segment (none before the first or after the last word). The segments have the shape alignPhonemes returns, so everything downstream works unchanged, and
carry alignment: 'word-anchored': the word times are given, the sounds inside a word are still an estimate, not a forced alignment. Bad input is a plain Error (NaN or
non-numbers, negative or out-of-range times, an end not after its start, words out of order, empty text, overlaps over opts.maxOverlap, default 30 ms); smaller overlaps are
clipped and reported in clipped. On a synthetic continuous voice the plain aligner put a word boundary 471 ms off; the anchored one was exact (tests/anchors.test.mjs).
Grab the mouth: the lip drag controller (Step 28)
createManipulator(opts) (src/manipulate.js, en-US only) is the pure, DOM-free controller both faces call when a visitor grabs the mouth. A face's pointer
code sends grab({ part, x, y, t }) (part is upperLip, lowerLip, leftCorner, rightCorner or jaw; x and y are offsets in mouth-widths, y down), move({ x, y, t })
and release({ t }); each frame it asks channelsAt(t, baseChannels, out) and gets the 13 articulation channels with the drag added ON TOP of the speech (out is reused when given,
so nothing is allocated). The drag is the displacement since the grab. Lower lip or jaw down opens the jaw, up closes it and seals the lips; a corner out widens, in rounds and
puckers; the upper lip parts or covers the lips. The library has no smile, frown, upper-lip or teeth channel, so a corner lift maps to lipWide (smile) and lipPress with
mouthNarrow (frown), and left and right corners give the same pose. The pointer sets a target; a spring-damper (jaw 140 ms, lips 70 ms to 90 %, damping 0.7) follows it with weight,
and on release the target eases away (time constant 0.2 s) so the mouth settles back to the speech pose within 2 % after 1 s. Every channel is clamped to 0..1 and a sealed mouth never
gapes or shows the tongue. The movement scale is NOT applied to the drag (the visitor is the animator); baseChannels is the speech as the face already receives it. Events never throw
(unknown part, NaN, huge values, time going backwards, a release without a grab are ignored or clamped); only bad construction options throw. About 1 microsecond per call (tests/manipulate.test.mjs).
The mapping table is the header of src/manipulate.js and DRAG_MAPPING.
The faces and the page take the grab (Step 36)
Both faces implement hitTest(x, y) (faces/CONTRACT.md): the part of the mouth under a point, or null for the cheeks, nose and background, with generous regions
(2D: faces/2d/mouth-hit.mjs; 3D: the projected loop of the morphed mouth, faces/3d/src/mouth-hit.mjs). faces/pointer-grab.mjs turns Pointer Events (mouse, touch, pen) and the arrow
keys into grab/move/release calls: the pointer is captured, touch-action: none is set on the stage for the grab only, so a touch that starts off the mouth still scrolls the page; the drag is
scaled by the face's own mouth width and clamped; pointercancel, a lost capture, Escape and a blur let go; a tap gives a gentle pulse. On the demo page the face shows the controller's channels
on top of the speech (whether or not speech is playing, no movement scale on the drag) and eases back to the speech pose on release. See demo/ and docs/DEMO-PLAN.md (Step 36).
Only en-US is supported, as everywhere in the library.
If an application ever needs it
It is published as a versioned package on the forge and the application vendors a pinned copy, never a live link. The rules
are /cosmoshot/tools/docs/VENDORING.md. Until then nothing depends on this repository.
Viseme benchmark (Step 31)
docs/VISEME-BENCHMARK.md benchmarks whether the library's viseme shapes match published lip-posture data: it measures the actual geometry of both faces (the 3D head from its mesh, the 2D face from its SVG lip paths) for every ARPAbet phoneme and every Oculus viseme, compares each quantity against a qualitative expectation table (tests/fixtures/viseme-expectations.json), and reports PASS / FAIL / UNKNOWN with findings. The browser harness is tools/measure-visemes.mjs (not part of npm test; it needs headless Chromium and the 3D build).
Demo
An optional demo page shows the library at work: type any text, hear it spoken in your browser and watch a mouth move to it,
with a four-step walk-through of how it works. It is not part of the library and nothing in src/ depends on it.
cd faces/3d && npm ci --ignore-scripts && npm run build # the 3D face's bundle (build/face3d.js); without it the page stays on 2D
cd ../../demo && npm ci --ignore-scripts && npm run build && node serve.mjs
Then open the address it prints (127.0.0.1 only). The page starts getting the voice by itself when it opens: it downloads about 26 MB
(the Kitten TTS model, from the programme CDN), checks every file against a pinned size and SHA-256, and keeps it in the browser, so a
second visit is instant. Speak is enabled when the voice is ready. If the visitor's browser asks to save data (Save-Data), the page
waits for a click on "Get the voice (about 26 MB)" instead. A failed or mismatching download shows an error and a "Try again" button; the
files are never used unverified. Nothing you type leaves the browser. English (en-US) only. The browser end-to-end run is
node e2e.mjs in demo/ (it uses the real CDN and headless Chromium; it is not part of npm test).
The waveform of the last utterance is the scrubber: press or drag on it (mouse, touch or keys) to see the mouth at any moment, with no
sound, then "Play from here". The last utterance can be downloaded as WAV, a PNG or SVG picture of the waveform, and the timing as
JSON; all are made in the browser. "Download video" makes a video of the face (2D or 3D, with the current look, head motion and movement scale) speaking the current
audio (typed text, a recording or an upload), with its sound, in the browser: MP4 (H.264 and AAC) where the browser can encode both, otherwise WebM (VP9 and Opus);
sizes 720 x 900 (the default), 720 x 720, 1280 x 720 and 1080 x 1350 (slow); 30 frames a second; at most 90 seconds and 200 MB; a small "ai of our own" mark in a corner
(on by default; the setting is cornerMark in demo/video-model.mjs); Cancel leaves no file. The engine is faces/export/; the page parts are demo/video.mjs and demo/video-model.mjs.
Captions are not made yet, so the page has no such control. node demo/serve.mjs --csp "<policy>" (or the environment variable LIPSYNC_CSP) sends a
Content-Security-Policy header on every response, to try the page under the host's policy.
For hosting, cd demo && npm run dist writes demo/dist/ (not in git): a static folder with only what the page needs, every URL
relative, no inline script or style, and a BUILD.json naming the commit, the library version and the size and sha256 of every
file. It refuses to run without demo/build/ or on a dirty tree (--allow-dirty is for tests). Try it as the host will serve it with
node serve.mjs --root dist --prefix /lipsync --csp "<policy>" and E2E_TARGET=dist node e2e.mjs. See demo/DEPLOY.md; nothing is
deployed by this repository.
The page has an Articulation: Soft / Natural / Crisp control beside the Motion one (Soft by default, remembered in the browser only) that
sets the library's style for the blend and the baked trajectories, and its step 4 explains coarticulation in plain words. demo/motion-sheet.html
(no model, no network) draws filmstrips and channel traces of a phrase in each way, and node tools/shoot-motion.mjs [prefix] [--out DIR] [--plots]
photographs them with headless Chromium (default folder /var/lib/cosmoshot/lipsync-shots; it refuses system folders, /dev included).
The look follows the brand packets copied into demo/brand/ (the page's tokens; the 2D art is in faces/2d/brand/) (identity 1.1.0 files in aoo/, the mouth palette of the lipsync
packet 0.1.0-review in packet/; see PROVENANCE.md). The two advertising texts are in demo/claims.mjs, confirmed by the Private
Local Chat and Sonalon teams (the sources are recorded there).