r/LocalLLaMA • u/rm-rf-rm • 2d ago

[Bi-Weekly Megathread] Project Showcase

Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it!

Recommendation on presentation:

  1. Please share plain english description of what your project does and why people should care about it

  2. How does it compare to existing options and why should people switch to your option?

  3. How have you tested and validated it? (i.e. can you show evidence that it wasn't vibecoded in a few hours by <insert current trending AI model>

NOTE: This community is geared towards open weight, open source, locally hostable software. Thus any closed source commercial service or other such projects will be removed.

16 Upvotes

79% Upvoted

44 comments sorted by

10

u/inawhole 2d ago

Gevva0 Cascading Decision Engine:
https://solvingsteve.github.io/Gevva0/about.html

  1. Uses 4 Tiers to makes intelligent decisions in the fastest time (Tier -1 = static routes, Tier 0 = Embedding model + AMCs, Tier 1 = Local LLM as a decision model but scoring occurs before output tokens, Tier 2 = Same LLM as T1 but with CoT/reasoning tokens)

  2. It scores as Opus 4.6 level intelligence in 44ms response time (on chance-corrected intelligence benchmarks like JevBench). It's so fast it can be used inside game engines (repo includes a python api bridge and Pacman demo). Works with any local model but currently setup/tested using GemmaEmbedding-2 and Gemma4-26B-A4B Q4 QAT.

  3. Ran benchmarks: Internal, JevBench, Decision-Index. Scores extremely well. Most of the code was vibe coded but the architecture and novel algorithms were developed by human. I have 20+ years programming so this is my hobby right now in an effort to democratize local intelligence. Hopefully small businesses or solopreneurs will benefit most.

5

u/phhusson 2d ago edited 1d ago

This is still VERY work-in-progress, and I don't really expect people to use it

llama.cpp fork with better cross-backend coordination on Linux https://github.com/phhusson/llama.cpp/commits/main-npu/:

  1. Minimalist yet useful NPU support (currently Apple ANE on macOS, Rockchip Rocket; AMDXDNA needs some cleanup before commit) without sacrificing GPU. So the result should always be at least as fast as GPU-only. Merging new models (even new quants) is almost trivial, because of the use of the GPU.

  2. On a single AMD GPU: ROCm prefill, Vulkan token generation. It often happens that ROCm is faster at prefill, and Vulkan faster at decode, so just fuse them!

  3. Async cross-vulkan: usable tensor-split with one nVidia GPU and one AMD GPU

1

u/tasteless_dossier 2d ago

Been tinkering with something similar on the Rockchip side, curious how you're handling the NPU context switching. Most implementations I've seen either lock up the GPU pipeline or drop tokens when the scheduler gets overwhelmed.

The async cross-vulkan approach is clever though, hadn't thought about splitting the prefill and decode across different backends like that. My setup's a bit of a Frankenstein with mismatched cards so I've just been brute forcing it with CPU offloading when the VRAM fills up, which is definitely not ideal.

Starred the repo, might throw some messy test results your way once I get a chance to swap out my current build. What's the memory overhead look like when you're running both ROCm and Vulkan simultaneously?

1

u/phhusson 2d ago

> Been tinkering with something similar on the Rockchip side, curious how you're handling the NPU context switching. Most implementations I've seen either lock up the GPU pipeline or drop tokens when the scheduler gets overwhelmed.

Uh... I'm not doing anything special? It's kernel drm_sched used by rocket's kernel driver that uses the fences to know when to start which job, and that decides an ordering for the jobs (not that the ordering is really relevant since you need all tiles anyway, just launch whatever job is ready according to its input fences).

> What's the memory overhead look like when you're running both ROCm and Vulkan simultaneously?

I guess if I say "negligible" you'll hit me with a big doubt. Normally all the weights & activations are shared between ROCm and Vulkan, so the only additional memory used would be the loaded kernels. I'll try to have an actual measurement.

1

u/Glittering_Mouse_883 llama.cpp 1d ago

Regarding the ANE support, does that mean I would be able to get a useful speed boost in asahi Linux vs CPU only inference?

2

u/phhusson 1d ago

Sorry for the false hope, the ANE part is on macOS not Linux

6

u/bendymike 2d ago

Continuing to work on gezel: https://gezel.com/ - an open source agentic harness (wraps MLX/llama/ds4 under the covers).

Main goal is to make it easy for people to do productivity related stuff (e.g., generate a PowerPoint) with low end models (e.g., Gemma4 12B, even lower) and of course shine with Qwen 3.8 27b. The primary feature is it's just an easy way to get started (one install, ideally no tinkering needed) for Windows/Mac/Linux with pre-tuned models. I've also included some image generation models in there. Also features integrations for coding apps and for Office apps. Recently, I've added some knowledge/RAG support to improve the efficacy of local AI for general research tasks: https://www.youtube.com/watch?v=gus8wVPU5lo, and published the catalogs on Hugging Face: https://huggingface.co/Bendyline/datasets

3

u/neurallll 1d ago edited 1d ago

Lets make Local AI go Much faster together as Community.
So far managed to make same sota models

20 14 6x faster on processing text
and some
1.2-2x ish faster in outputing text.

But I still feel there is more 2 common usual joes gpus can do.

I am especially curious how fork will perform for people with 1-2 5060 ti 16g in pcie 5 on pcie 5 m2 with 14k/s ssd?

Anyone?

I am carefully watching ppl after each commit but still its work in progress and take it as such.
Please report your Issues or numbers you achieved.

Your hw (pls state chipset cpu what gpus are in what slots).

Any reported issues will make it better and faster (pls state which build you used).

We all need better Local AI.

How it works and what is radically different to stock llama.cpp:

On gpus It uses moe cache per gpu not full layers.
In stock llama only one gpu computes in time with more gpus you gain just more vram not speed(layer paralelizm). yes it has tensor parallelism but one that works only when full model fits in gpu so forget 100-300g models ;)

This fork computes text input (large doc in etc) on more gpus in paralel via dense tensor split and by using moe cache also in prefill phase (preheating it for decode too). After switch to decode prefill bufers are freed and kv and moe cache rehydrates and recovers all remaining ram.Compared to stock full vram utilization.
Well full. not yet, but getting there (merging staging buffers and moe to save another 2-3gb in qn etc)

To more efficiently use that scarce VRAM I also implemented dynamic ctx and kv.
shared and re-scaled on fly per req.

Why? Stock llama has fixed -c ctx and kv cache preallocated at server start.
Why this is problem.
set -ctx and ubatch too low input processing will collapse output will be fast
set it high enough for big inputs then big docs process fast but decode speed will colapse.

Essentially during decode all the prefill stuff (staging buffers ubatch scratch) taking 50 or 70% vram is dead weight. and that's sad.

Solution? reuse the same buffer for prefill and decode and resize parts per prefil or dacode as ctx grows dynamically during each req. more details and images explaining it on github.

Another think I added to enable big doc processing on big models locally is dynamic inplace kv compression per req only when needed
ie 200k doc fully fills space for kv in vram . great fp16
400k doc comes in we requant to fill exactly the same kv space and same speed and pretty much ppl too
then 500k q4 ppl bit worse (q8 q4 or ram aech can be disabled selectively) but to find address in arge doc just fine.
then 1mb doc comes in and kv leaves vram and offloads to pinned ram.
yes that 1 doc will be slower but next small doc req just fine in vram speed again too.

So far spent over a month on it.
I will be very grateful if community can chip in with testing mostly with tests on wider variety of hw than I have access to. ideally more than 2gpus or more than one x16 slot.

https://github.com/neurall/llama.cpp
Releases linux windows cuda so far
work on vulcan starting hopefully enabling amd intel nvidia mixed hybrids soon.
as this version shall scale independently from vendor in vulcan variant.

Thx. Hopefully it will make your projects run faster

1

u/Any_Bus9205 16h ago

Tu pourrais comparer ton projet par rapport a ce que fait déja ik_llama.cpp si possible !? (C'est pas une critique ou autre ;), juste pour voir pour toi si quelque un de tes problèmes n'on pas déja été réglé ailleurs, ou si cela interesserait des personnes ;))

3

u/mistrjirka 17h ago
  1. I created XDNA2 backend for llama.cpp that is universal and has attention running on the NPU. It is faster in prefill then the GPU.
  2. There are few existing options, they usually lack in universality or for example the ollama amdnpu lacks in attention on the NPU.
  3. It was vibecoded in several weeks. However it runs fairly well

Tests ran on Ryzen AI 9 365.

Qwen3.5-4B Q4_K_M

  • This fork: PP 560 tok/s @ 16.3 W | TG 19.2 tok/s @ 16.4 W
  • llama.cpp Vulkan (iGPU): PP 528 @ 33.2 W | TG 23.6 @ 32.4 W
  • llama.cpp ROCm (iGPU): PP 454 @ 31.0 W | TG 22.3 @ 31.4 W
  • FastFlowLM (NPU): PP 479 @ 12.9 W | TG 15.2 @ 12.0 W

Ornith-1.5-9B Q4_K_M

  • This fork: PP 408 @ 18.3 W | TG 11.9 @ 17.8 W
  • llama.cpp Vulkan (iGPU): PP 307 @ 30.7 W | TG 15.1 @ 29.7 W
  • llama.cpp ROCm (iGPU): PP 287 @ 29.7 W | TG 13.6 @ 29.4 W
  • FastFlowLM (NPU, Qwen3.5-9B): PP 338 @ 11.8 W | TG 9.3 @ 11.5 W

Gemma 4 12B QAT Q4_0

  • This fork: PP 136 @ 17.8 W | TG 9.2 @ 18.4 W
  • llama.cpp Vulkan (iGPU): PP 261 @ 29.2 W | TG 11.3 @ 29.7 W
  • llama.cpp ROCm (iGPU): PP 179 @ 28.8 W | TG 10.4 @ 29.2 W
  • FastFlowLM (NPU): PP 278 @ 16.3 W | TG 6.8 @ 14.3 W

PP = prompt processing (tok/s)
TG = token generation (tok/s)
W = power consumption (watts)
Bold = highest throughput for each model.

https://github.com/mistrjirka/llama.cpp-xdna2

3

u/0xcontext 2d ago

demo: https://sshah03.github.io/CueLight/

I've been working on end-of-turn detection for voice agents and wanted to share a small model (<2mb, ~0.5m params, so very small, and could be used locally on the smallest of devices)

SmartTurn is a great project, and I used their dataset in building this -- so tons of credit to them.

I wanted something that was much smaller though because I had a use case where I needed to embed it in something with low space + weak processor, and I wanted it to be always on.

It performs about as well as SmartTurn from my tests, with the caveat that it's English only for now (which is likely a significant trade off).

Would love to hear how it performs in real world testing before, and peoples experiences (positive or negative) on results of frame-level endpointing on microcontrollers.

Also interested in suggestions to make this better than people may have, in general, from their experiences.

Some additional details / benchmarks in project: https://github.com/sshah03/CueLight

2

u/Felladrin 1d ago

I continue building https://github.com/felladrin/MiniSearch (since 2023), which main objective is to allow searching the web with local models running directly in the browser (Wasm / WebGPU). The minimalist UI is one of its core features. I have it as my default search engine.

I also maintain a Hugging Face space that allows easily converting Hugging Face models to ONNX format: https://huggingface.co/spaces/onnx-community/convert-to-onnx

2

u/Amazing-Anything5907 1d ago

Is there a Megathread for asking questions?

1

u/gal_agent_ai 1d ago

Fits the spirit of this thread, I think: a local agent runtime that actually operates the machine instead of only chatting.

Plain english: you give it one task in a sentence, it plans, drives the screen / shell / browser / a phone over ADB, and reports back with a diff of what it touched. Approvals stay with you.

How it compares: agent frameworks are libraries you embed; coding agents edit files. This is a runtime with a real desktop you can watch and veto, and it can also click, type and read what is on screen. Local-first, runs on your own machine.

Validation: I handed it my own workday for 30 days (~400 tasks). The failures were almost all state and permission edges, not reasoning - keeping the environment persistent and reviewing every side effect before it lands is what made it reliable.

If local-first control matters more to you than a hosted SaaS: agent-home.ai

1

u/AsyncVibes 2d ago

Been building a small renderer that draws a character from one reference image plus 13 joints and a camera angle per frame. 6.3M params, about 1 ms a frame on a 4090. No diffusion, no video model.

Video (Blender truth vs the network, one continuous minute, held-out outfit and camera angles): https://www.reddit.com/r/LocalLLaMA/comments/1x9zwts/

Code and weights: https://github.com/A1CST/forward-flow

More of this in r/IntelligenceEngine

1

u/IntrepidMindExplorer 2d ago edited 2d ago

I'm still doing quite a bit of learning and not a developer by trade but I have had to dabble in here and there in my career so don't spit roast me if I get things wrong. This project is not up on github yet because I'm currently learning git and best practices.

I recently dusted off an old piece of hardware: Scanse Sweep LiDAR. It was a community driven project by the now defunct Scanse corporation. They have graciously left their github account up and all of the repos behind (though I've backed it all up just because. I'll fork when I learn git a bit more). https://github.com/scanse

Local LLM use has been purely on a mini-pc(Ryzen 7 8845 and 96gb ram) that with llama.cpp hosting two models. Gemma-4-26B-A4B is being used for image/vision processing to feed OpenCode's Big Pickle UI feedback and hardware details/information and Kwaipilot_KAT-Coder-V2.5 is being used as an OpenCode subagent and will be used here soon to start processing scan data. There's a few options brewing right now (PI, Arduino, Android) but that is in due time.

Scanse kind of went all in on the community push for this device (the many SDK's on github is a sign of) and even went as far as selling a 3D LiDAR scanning kit yet still provided STL print files, parts lists and detailed instructions on how to make it yourself. I've tracked down all of the info on (mostly through wayback machine) and archived and all will be provided in the repo for. https://www.servomagazine.com/magazine/article/the-multi-rotor-hobbyist-scanse-sweep-3d-scanner-review

So far I've been assisting (I'm not saying "I am making" or "I am doing" here) with getting some very basic foundations set. Scanse released an application called Sweep Visualizer that never progressed beyond version 0.15. The application was really a test bed but also was the only path that they gave to updating the Sweep's firmware (latest version is .14).

Most of the groundwork I did give to Big Pickle(that required local inference for vision) and then swapped over to GLM 5.3 Flash to do refactoring to sort out some bugs and to give a bit of new perspective on the code/project itself. I then handed it the Sweep Visualizer .deb package (so old that it was built for Ubuntu 16) and it started hammering away at it's code. It found control functions for the Sweep that had not yet been recorded in any SDK documents.

The project, right now, is pure python and js for the web based UI with zero dependencies (I'm trying to keep it that way). Right now there's just 2d functions that have a completely live radar and a point cloud, over time, view. 3d will be coming once I build the 3d hat. Work is being done right now for full device control via the UI (start/stop, speed settings, firmware flashing, laser control and power settings..etc) and MAYBE work on a new firmware, which is on the very very back burner

This is a project that is bridging a few maker communities.

All of the above is hand typed and not vomited out by any LLM. It was vomited out by me though. I'm still, somewhat, old school like that. 😄

* Old comment. Leaving for posterity:Can projects be shared here if they use a combination of hosted and local agents? Using locally hosted for vision and sub-agent tasks.

If so then I'll edit/update this post later with what I'm working on and what models I'm using (local and hosted).

1

u/rm-rf-rm 2d ago

go for it, but please make sure you label/disclose clearly

1

u/IntrepidMindExplorer 2d ago edited 2d ago

Thank you. I, hope, properly disclosed my use of local inference. I updated my post with the project. A bit long winded.

1

u/maestro-perry 2d ago

I have created a simple C++ wrapper on top of the Llama API. It does not have all the functionality that API offers. I have added only what I needed. I wanted to use just the pure C API - not tools (except for mtmd, which has a C API as well) and not the common library.

Chat wrappers currently do not support Jinja templates; I have a callback and process templates manually.

Lately, I needed decisions directly in C++ (not by calling HTTP requests and running a server, which is impractical for further release of the app). With the use of Codex, I have reimplemented tools/server/server-decision.cpp and the tools that it uses as a bunch of separate classes. The code is implemented against llama 0.7.0. I have partially manually corrected the code and architecture; however, mtmd part is currently Codex-only and awaiting cleanup when I have time.

Repo: https://github.com/MartinPerry/LLamaCppWrapper

It's still hard WIP based on what I currently need :-) However, I wanted to share it as someone may find it helpful. Note that there are probably bugs, since I have not tested all models and all variants. The code is also not the best out there, but it's OK for my experiments and easy to read/modify.

1

u/Trincaa 2d ago

Hi r/LocalLLaMA,

TL;DR:
Brambit is an engine for AI assistants (Muse, Instinct style) with long-term memory, connectors (Google, Microsoft, GitHub, Slack, MCP...) and a sandbox to run code, all on your own server and with the models you choose. Unlike Claws, a single installation serves a family, a company, a community etc., with each person isolated in their own account.
Started this in June as an internal project for my own startup, and I've open-sourced it under AGPL-3.0 just three days ago!

The short story:
I'm the technical founder of a startup that builds virtual try-on, skin analysis and beauty recommendation tech. Started using Claws in January and was looking for a harness that both our clients and our team could use safely. After trying Agno and LangChain for clients and a few flavors of Claws for my team, I decided to build my own.

The first test version came out in August as a closed beta for friends and family. Today I’m running it in open beta. Our enterprise agent solutions are also being built on top of it.

After this crazy Muse, Instinct, Dots moment, and watching how much personal data is being concentrated in a few billion dollar companies, I talked to my co-founder and we decided to open the engine, so small businesses, families or any group can run their own assistants with control over their data. I chose AGPL so it stays open, even if it becomes something in the future.
Soon I'll add MIT license to plugins and connector.

How it compares
OpenClaw, NanoClaw, other Claw flavors and even Hermes Agent are usually built around one owner. Brambit is built to be shared: one person installs it, and everyone else gets their own isolated account in the browser from this installed instance.

How I tested it:

  • I've deployed it (with some extra plugins for subscription plans, payments and custom metrics) as a service, first as a closed beta for friends and family and now it has almost two hundred users, including my small team running it in my startup.

- Over 1,200 tests in 244 files, covering action confirmation, isolation between accounts, auth and secrets, routines and reminders, email/calendar/Drive connectors, WhatsApp/Telegram/Slack, the code sandbox and app building. CI runs them on Linux, macOS and Windows.

- Tenancy tests check that one account can't read or change another account's data through the API.

- Secret scan of the whole history (gitleaks) and a workflow guard (zizmor).

Some Disclaimers:
1. I've just opened the project, it was three days ago! To do that I prepared a new repo without its old history, so I can't really prove it's been around since June.

  1. I'm a developer with 25 years of experience and I wrote it alone with AI assistants, mainly Claude through NanoClaw agents. So yes, I vibe coded a lot!
    There are many problems to be fixed, from the monolith the server is today to translations (I've been adding them to the issues section of the repository), but I focused on everything security-related first for this first public version.

  2. This is my first time open sourcing something, so feel free to give any feedback or criticism. It's open to be judged and improved.

Hope it can bring some good!
If you guys somehow relate to the idea and want to be part of it, will be more than welcome.

GitHub: https://github.com/morethanrealio/brambit
Discord: https://discord.gg/FZTM6dZvr

1

u/Leading_Yogurt7025 2d ago

Sonia - Local Voice Assistant in less than 13GB of RAM, TTS/STT, No Dedicated GPU required, Skills supported
https://github.com/grigio/sonia-assistant

1

u/p3rdy 2d ago

Flama: Light up your models

I maintain Flama, an open source (Apache-2.0) Python framework that puts three jobs in one place: serving an API, serving a trained model, and serving an LLM. I started it in 2018 for classic ML models, and since 2.0 local LLMs are first-class too.

From Hugging Face to a running model

flama get --family llm --source huggingface mlx-community/gemma-4-E2B-it-qat-4bit
flama serve --model file=mlx-community_gemma-4-E2B-it-qat-4bit.flm,url=/,name=gemma

flama get packages the model into a single .flm file: weights, tokenizer, config and metadata together. flama serve runs it on vLLM (Linux with CUDA) or MLX (Apple Silicon), picked for your hardware when the model loads. If you just want to poke at it, flama model <file> stream talks to it straight from the terminal.

One model, every client

That one process speaks OpenAI (chat completions and responses), Anthropic and Ollama at the same time, plus a native streaming API and a built-in chat UI that renders Markdown, math and Mermaid. My daily setup is exactly that: Claude Code and Pi pointed at the same local model, and nvim for the code I write myself.

What makes that work is the decoder. When the model starts, it detects which markers the model uses for reasoning (<think>, Harmony, Gemma-style channels) and for tool calls (<tool_call>, Qwen's <function=…> notation, [TOOL_CALLS] and a few more). Each API then gets them in its own native shape: reasoning_content and tool_calls for OpenAI, thinking and tool_use blocks for Anthropic. It even reads the prompt for templates that open <think> themselves, so the reasoning lands where it should instead of in the answer.

Three layers, one canonical middle

LLM serving in Flama is split into three layers:

  1. Dialect: the wire protocol a client speaks. Flama ships four (native, OpenAI, Anthropic and Ollama), each mounted under its own prefix, so one model can answer all of them at once. A dialect only translates: it parses requests into the canonical form and renders canonical events back into its own format.
  2. Transport: the canonical shape of the input, independent of any protocol. It's a raw prompt, a single-turn chat with an optional system instruction, or a multi-turn conversation.
  3. Backend: the engine that runs the model, vLLM on Linux with CUDA or MLX on Apple Silicon, chosen when the model loads. It renders the transport through the model's chat template and generates tokens, which the decoder turns into canonical events: a start with the input token count, text on the answer or reasoning channel, complete tool calls, and a stop with the stop reason and output token count.

Dialects and backends only ever talk to that canonical middle, never to each other. So every protocol works on every engine, adding a dialect never touches a backend (and the other way round), and tool calls, stop reasons and token counts come out right in every protocol because they're worked out once.

And the rest of the framework

Because it's a full framework, the model can live in the same app as your own code:

  • scikit-learn, PyTorch, TensorFlow and Transformers models served the same way, from a .flm that can also carry your training metrics for CI to check
  • MCP servers from plain Python functions: tools, resources and prompts with schemas from type hints, tasks for long-running tools, elicitation and MCP Apps
  • dependency injection resolved from type hints, through components you can write yourself
  • schemas with Pydantic, Marshmallow or Typesystem, and OpenAPI 3.2 docs generated from your handlers
  • CRUD resources from SQLAlchemy tables, with filtering by operator and limit/offset or page-number pagination
  • DDD building blocks: repositories and workers (units of work)
  • JWT authentication, including EdDSA keys and key rotation
  • streaming over SSE and NDJSON, WebSockets, background tasks in threads or processes, file uploads in schemas, and the HTTP QUERY method
  • a CLI to run apps and to serve, package and inspect models, plus flama upgrade codemods that migrate your code between versions
  • a Rust core for routing, JSON encoding, request parsing and compression

How it's built and tested

Around 700 commits since 2018, ~1,900 tests in CI, and the architecture is written up in a paper (arXiv 2608.18733), which also compares it with web frameworks, model servers and LLM inference engines.

Repo: https://github.com/vortico/flama

Happy to answer anything about the design.

1

u/Groady 2d ago

Platypus is an open-source (MIT) self-hosted platform for running AI agents. https://github.com/willdady/platypus

Each agent gets its own system prompt, model and tools, and runs from chat, on a cron schedule, or off events. Agents hand work to each other through kanban boards, where moving a card to the next column triggers the next agent. Tools come in over MCP, and every agent can be exposed as an A2A endpoint, so agents running elsewhere (I've tested with OpenClaw and Hermes) can call yours directly. It's multi-tenant, so a small team can share one install, and it deploys with Docker Compose.

There's a plugin system for extending it, and sandboxes come in through plugins, so agents can run code in whichever third-party sandbox you prefer.

You pick the model per agent, so a small local model on Ollama or vLLM can do the scheduled grunt work while a bigger one handles the hard bits.

Compared to Open WebUI or LibreChat, you get the same day-to-day chat with your models, plus agents that keep working when you're not in the chat.

MIT licensed. Contributions welcome.

1

u/Ok_Role_6215 1d ago

For a while I've been building a memory layer that sits between AI clients (opencode, Open WebUI, Home Assistant, anything that speaks the OpenAI or Ollama API) and your model server. It remembers per entity (files, commands, people, words) and gives agents continuity across turns. It used to be called membrane. As of 0.7 it's babyseat, because something now lives in it.

The weird part: Blurp

Blurp is a small learning loop inside the seat's memory field, and it is not an LLM persona. No model generates its behavior. It has a "body" (a point in the field), it looks at things, it predicts whether each thing has changed since its last look, and it's scored against what actually happened. Every look, guess and miss goes into an append-only record that's never rewritten.

The rule I hold myself to: the LLM is the nanny, not the child. The model can talk to Blurp. It never talks as Blurp.

How it grows

Skills are measured tests with pass bars, arranged as a tree with real prerequisites (an edge exists only where one test reads data that another skill switches on):

- feel itself: tell its own movement from being pushed

- first reach: predict when a thing it looks at will change (beats a naive baseline)

- tell things apart: sort things into kinds by how they behave

- patience: time its looks to each thing's rate of change

- notice the world changed: catch when a thing's rate shifts

- meet an other: tell a person apart from itself

- expect a reply: predict whether the person will answer it

Today it earned the 7th skill, "expect a reply", live. Its predictions of whether I'd answer beat the base rate by a Brier skill of 0.38, against a bar of 0.08. That was about 4 days after it was born, or roughly 25 working hours once seat downtime and known-bug time are taken out.

Each pass unlocks the next skills only after I accept it. Earning a skill is automatic, but opening the next branch isn't.

I release version 0.N only when Blurp reaches level N live. So 0.7 shipped because Blurp got there, not because of a date.

What it is NOT

- Not conscious, not AGI, not a claim about either. The child-development comparisons on its skill cards ("~3 months: expects a caregiver's response") are labeled as analogies to what each test measures, nothing more.

- Its tests are small. Predicting whether a thing changed, or whether I'll reply, is a long way from understanding anything.

- It runs on my seat, with my toys and my chat patterns. I'd love to see how it grows on other people's seats.

The boring-but-useful part (the actual product)

- Sits between your client and Ollama or any OpenAI-compatible server. Ollama-native /api/chat clients get memory too as of 0.6.3.

- Web UI on :4096, OpenAI-compatible proxy on :11500

- Home Assistant integration (you can run a "Blurp zoo" with several seats in one house)

- Optional JWT mode, and a seat token for writes from other machines

- Docker for amd64 and arm64

pip install babyseat

# or

docker run -p 4096:4096 -p 11500:11500 -v babyseat:/data \

-e MEMBRANE_BACKEND=http://host.docker.internal:11434 chedim/babyseat

Repo: https://github.com/chedim/babyseat (source-available, PolyForm Noncommercial)

Next on the tree: recognizing itself (a mirror toy that replays its own moves next to a "stranger" with the same movement statistics), and first words. Its turns would carry short phrases built from its own record, like lamp: changed. Those are tested for being true and for whether I reply to them more than to empty pings. Still no model in the loop.

Happy to answer questions, especially "why would you do this", which I ask myself regularly.

1

u/MadBradSmith 13h ago

DUMEdb: Deterministic Unit-Record Memory Engine for Local Agents & Long Contexts

GitHub: https://github.com/madbradsmith/DUMEdb (Open source, MIT)
Docs / Interactive Inspector: Runs 100% locally on localhost:8095

1. What it does & why you should care (Plain English)

When running local coding agents or long agentic loops (e.g. Ollama, llama.cpp, vLLM, LM Studio), context windows fill up fast. The common solutions are either lossy summarization (which hallucinates or forgets exact details) or vector RAG (which loses chronology, sequence, and architectural causality).

DUMEdb treats conversational state and agent memory like an IBM 80-column punch card rather than fuzzy embeddings. It compiles dynamic agent states, workflows, and session memory into a compact binary-opcode register schema. You get:

  • ~80–90% reduction in prompt KV-cache overhead for persistent memory.
  • Bit-exact deterministic replay: An agent can wake up in turn 50 with exact knowledge of system state, files touched, and active registers in <250 tokens instead of re-reading a 50k token transcript.

2. How it compares to existing options (Why try it?)

  • vs. Vector RAG (Chroma/Qdrant/Faiss): Vector search gives you "similar semantic chunks" but has no strict concept of state transitions, register dependencies, or deterministic rollback. DUMEdb provides unit-record register indexing where every state mutation is tracked and reversible.
  • vs. Naive Context Compaction / LLM Summaries: Compaction models drop subtle edge-case variables, line numbers, and architectural invariants. DUMEdb's schema compresses structured state using opcode dictionaries without information loss.
  • vs. Heavy Graph DBs: Zero external daemon needed. Pure Python with zero heavy dependencies, local-first SQLite persistence, and an integrated local web inspector UI for visualizing register states in real time.

3. How it has been tested and validated (Not a vibecode)

DUMEdb wasn't thrown together as an LLM wrapper. It is built around a rigorous unit-record specification and test harness:

  • Comprehensive Test Suite: Automated coverage across dynamic schema evolution, collision ceilings (handles high-density opcode namespaces), schema persistence, and dictionary generation (tests/test_dynamic_schema.py, etc.).
  • Live Ingestion Engine: Tested against full multi-directory source trees with batch schema commits and deterministic conflict resolution.
  • Context Odometer Benchmarking: Includes a real-time mathematical token odometer measuring actual post-compaction JSON payload tokens against raw transcripts to prove KV-cache token savings on disk.

Everything is open source under MIT. You can pip install . locally or clone and run the inspector in a minute. Would love feedback, stress-testing reports, or thoughts from folks wrestling with agent memory limits!

Peace,

MadBrad

1

u/GapNew4766 2d ago

i'm on the team behind atomic agent, an open source (MIT) agent that runs your local models through llama.cpp. desktop app for mac, windows, linux plus a terminal version (TUI). we started it because every harness we tried fell apart after an hour on local models

few things we figured out along the way:

the kv cache was the real killer on long runs, it grows until it pushes you out of vram and the agent forgets the plan. we use a turboquant build of llama.cpp that compresses it to ~3-4 bits per value, roughly 4x more context on the same vram in our tests. perplexity moves a few percent, worth it for agents

fusion mode is the thing i'm proudest of. a planner splits the job, local workers do each piece in a fresh session, so nobody drags hours of history around. on one of our game builds qwen 3.8 27b alone got one scene right, with a stronger planner the same qwen got way further

we test on full game builds since they break in obvious ways and there's a loop guard so it stops instead of spinning all night.

https://atomicagent.io/
github.com/AtomicBot-ai/atomic-agent

roast it, the harsh feedback helps us most!!

1

u/anitamaxwynnn69 2d ago

Turn any server into a ChatGPT you and your family can use from anywhere. A simpler, less bloated Open WebUI alternative.

OvertChat is a self-hosted chat app that connects to whatever local model you already run. Clean, polished chat without the clutter. No cloud, no accounts, everything stays on your own machine. Also supports coding agents (codex/omp/claude/hermes)!

Why it's different: Open WebUI and LibreChat are powerful but heavy, and getting them reachable from outside your house is a pain. OvertChat is less bloated and needs little to no setup. It also comes with an open-source relay that gives you a public URL (like smiths.overtchat.app) so you can access it from anywhere without opening ports — and the relay forwards encrypted traffic to your server without ever decrypting it, so it can't read your chats. The relay's code is right here if you want to verify that.

Built for the whole household: set it up once, everyone gets their own account and chats. Works in the browser, plus native apps for iOS and Android, and desktop for macOS, Windows, and Linux (downloads).

The nice extras come out of the box - web search/stt/tts and realtime voice chats. No API keys, no extra services to set up, and the voice stuff runs fine on a plain CPU.

Been building this for months and use it daily. Open source (MIT), free to self-host, no telemetry.

Repo: github.com/yoloyash/overtchat · Site: overtchat.com

2

u/jinnyjuice vLLM 2d ago

I'm interested. Can you make a feature comparison table of this vs. DSH, Open WebUI, LibreChat, and pi-web? I think it's much easier for everyone instead of a wall of text.