A local-first AI companion that watches gameplay, speaks and chats on Discord with memory and a Live2D avatar, using local models by default, with an optional cloud fallback. Built mainly by keithku321; I contributed the avatar’s motion and Discord picture reading.
Poster drawn from the repository
By the numbers
646motion engine lines, all from my commits
single motion loop30 Hz
feelings held as mood points16
test functions for the motion and install work55
test functions for picture reading35
§ 01
OverviewOverview
A private Windows companion project, built mainly by keithku321, that watches a game on screen, talks about it, chats on Discord with memory and appears as an animated Live2D character, using local models by default, with an optional cloud fallback. My part is the motion layer that makes the character blink, look around and gesture as it talks, picture reading in Discord chats through a local vision model, and an install path for newer Python versions. The stack listed here is the whole project’s; I worked in Python against the VTube Studio API. The repository is private, so this page carries prose and counts only.
§ 02
In motionInmotion
Blink, gaze, sway, mood and breath each add a small offset; one mixer sums them and one 30 Hz loop sends the result out.§ 03
My contributionsMycontributions
A motion engine for VTube Studio, 646 of 646 lines mine: seeded blink, gaze and sway, and a face that eases towards any of 16 feelings, each held as a valence–arousal–dominance point, with a 0.45 s time constant, all sent in one 30 Hz loop.
Speech gestures planned from a clip’s loudness: small accents on loud onsets, at most one per 0.6 s, plus up to three larger gestures by clip length, tried with a made-up voice clip.
A text-driven mouth, so lines nobody speaks aloud still move the mouth and head, using a repeatable mouth built from the text; verified live in the terminal chat, while the Discord side is tested with fakes only.
Mouse glances: every 25 to 70 s the eyes go to the cursor for 3.5 to 7.5 s and give way to speech gestures, checked on a 14-second clip by reading the engine’s values back.
Picture reading in Discord chats through a local Ollama vision model, with captions treated as untrusted data so nothing from a picture reaches memory or the cloud. It adds 35 test functions, run against fakes; the real vision model on real pictures is still unchecked.
An install path for Python 3.13 and 3.14 that leaves the 3.11 and 3.12 setup unchanged; a full run on the newer versions is unverified.
55 test functions for the motion and install work remain in the project, counted from the source rather than from a test run.
§ 04
How it worksHowitworks
1
The companion’s reply
keithku321’s system decides what to say, which feeling to show and whether the line is spoken.
a spoken clip, or the reply text
2
Mouth envelope
A spoken clip uses keithku321’s voice-following mouth; a line nobody speaks uses my text-built mouth on the same path.
loudness over time
3
Speech planner
Places small accents on loud onsets and up to three larger gestures by clip length.
planned gestures
4
Motion layers
Seeded blink, gaze, sway and breath, the eased mood and the mouse glances each produce small offsets.
offsets from every layer
5
Mixer and loop
One mixer adds the layers together, and one loop is the only writer, so mouth and motion cannot overwrite each other.
one request per tick, 30 times a second
6
VTube Studio
Receives the values and moves the Live2D character on screen.
§ 05
Engineering notesEngineeringnotes
Dispatch № 1
A character that stood still between lines
The problem
Hotkeys could only switch faces and props on and off, and the avatar model has no blink, gaze or sway of its own. Between lines the character stood motionless.
The decision
I added independent seeded controllers whose outputs one mixer adds together, held each feeling as a point the face eases towards, and had one 30 Hz loop write everything, mouth included. The repository records these ideas as drawn from a public Apache-2.0 emotion SDK and re-implemented with no code copied.
The result
Read back from the live model, the head turned between -4.5 and 2.2 units and blinks closed the eye; an angry face took the mouth form to -0.54 and a happy one to +0.82. These are measured values, not a judgement of how the motion looks.
Dispatch № 2
Lines nobody speaks aloud
The problem
The mouth only followed real audio, so text replies left the character silent and still.
The decision
With no sound to measure, I built a mouth from the text: one opening per kana or Chinese character, about one per three letters of a Latin word, and rests at punctuation, seeded from the text so a line always gives the same mouth. It enters the same path as real audio, so head gestures are planned as normal, and the Discord side first checks whether a reply is really spoken.
The result
Verified live in the terminal chat: the mouth peaked at 0.82 for about 5.6 s with a 2-degree nod, then closed. The Discord side was written against fakes and has not been seen live.
Dispatch № 3
A glance that yields to a gesture
The problem
The first mouse glance fired only when the mouse had moved and the character was silent, so it never happened during speech or with a still mouse.
The decision
Fourteen minutes after the first version I dropped both conditions and ran it on a timer of 25 to 70 s. A glance moves the head, so only a planned speech gesture can now block one or fade it out, within 0.25 s.
The result
On a 14 s clip with a still mouse, read back from the engine, glances fell at 0 to 2.3 s, 5 to 6.8 s and 9.3 to 11.8 s, and gestures ran at 2.3 to 3.1, 7.0 to 8.4 and 11.7 to 13.0 s. No glance ran during a gesture, apart from a 0.1 s overlap as the last one faded out.
§ 06
Development timelineDevelopmenttimeline
No recorded video for this project.
19 Oct 2026
Discord picture reading merged
Milestone
17 Oct 2026
Mouse glances made to give way to speech gestures
Milestone
17 Oct 2026
Text-driven mouth for lines nobody speaks aloud
Milestone
17 Oct 2026
Motion engine added
Milestone
16 Oct 2026
My first commit: install path for Python 3.13 and 3.14
