A miniature internal ML cloud. Schedule GPU training, track experiments, verify artifacts, version models, and serve them with real traffic splitting.
MAX_STEPSAshML is a control plane that turns ash job submit into a
Pod running on a real Kubernetes cluster, and then keeps an honest record of what
happened. Projects, datasets, experiments and jobs live in PostgreSQL behind an
append-only event log and a SKIP LOCKED queue. AshML's own scheduler —
not Kubernetes' — decides whether a job may run and where.
It is built to a hard rule from its own specification: never fake
GPU functionality, scheduling, distributed training, or performance numbers.
Simulated components are named sim, off by default, and carry
simulated: true through the API. Every number on this page was measured.
Dependencies flow one way: routes → services → repos → db,
with domain — the job state machine, placement and quota — importable from
anywhere and importing nothing.
Overfill the cluster and jobs queue rather than over-committing it.
ash job why <id> prints every node the scheduler considered and
what was wrong with it.
A GPU job on this host is queued with an explanation, never placed onto a GPU the cluster will not grant.
Checkpoints go from the training pod to object storage over a presigned upload, and AshML asks the bucket whether they arrived before marking one usable.
An upload that never landed is refused; one stored where AshML cannot check is
labelled NO rather than passing for verified.
At most one version of a model is PRODUCTION, with promotion
displacing the incumbent in the same transaction.
A version can only be registered from a READY artifact — otherwise
the failure just moves from "the upload failed" to "production cannot load the
model".
A deployment is a Service whose selector moves — onto one version's pods, or onto the router's the moment two versions take traffic. It keeps its ClusterIP and DNS name throughout.
The address only ever moves onto something with a ready pod, which is what makes a version change a blue/green rather than a gap.
Replica-count splitting needs a hundred pods for a 99/1 canary and makes resizing silently change the split. Weight and replicas are separate columns, and a router chooses per request.
Weights must sum to exactly 100 and are never normalised — accepting 100/10 and calling it 91/9 is how you get a split you did not choose.
SIGKILL a training pod and the retry resumes weights, optimizer moments, the
learning-rate schedule and the data order — each epoch's permutation is
derived from (seed, epoch), so resuming is slicing that order.
SIGKILL the control plane and the job comes back with the same attempt, Job and placement, and the event log gains nothing.
A registered version becomes something that answers requests — and every answer carries the version and artifact that produced it. A prediction nobody can attribute to a model version is how the wrong model serves for a week.
test-00001-ship.png: 32x32 truecolour
prediction: ship
confidence: 81.5%
served by: resnet18-cifar10 v1 → artifact 7228968a-… (resnet18-cifar)
latency: 269.6 ms in the pod, 284 ms round tripThose are CIFAR-10 test images with their true labels in the filename, so the answer can be checked rather than admired — and over the first eight this model gets six right, which is roughly what a 65.59% model should look like. A demo that scores 8/8 is predicting on its own training set.
The specification's §50 describes one user journey. make
journey runs all nine steps against the real cluster, in order, as one story —
driving the ash CLI rather than the API, because the question it answers is
whether a person can type them.
Step 3 QUEUED → STARTING → RUNNING, placed on k3d-ashml-server-0
Step 4 loss at step 10: 5.4429 (while the job is still RUNNING)
Step 5 4 checkpoint(s), model artifact verified, v1 is PRODUCTION
Step 7 3/8 correct on real CIFAR-10 test images
Step 8 41 series across 4 dashboards, all exported
Step 9 DEGRADED 0/1, then the same artifact back; a killed run resumed at step 15Three things it refuses to round up: step 3's "GPU node selected" prints the scheduler's real reason, because no GPU reaches a node here; step 7's score is printed and not asserted, because a passing threshold would be one tuned until it passed; and step 10 is not run, because it is post-v1.
The demo page runs the exact
artifact an AshML training run produced — the one its registry promoted to
PRODUCTION after verifying the bytes against object storage. Every
answer is attributed to that model version and artifact id, and it runs entirely in
your browser; nothing is uploaded anywhere.
The demo is on GitHub Pages, not the Hugging Face Space: Hugging Face serves that
Space under default-src 'none'; sandbox, which blocks its scripts
entirely. The Space still hosts the 45 MB model.onnx the page
fetches. And because a browser cannot run serve.py, the page runs the
same weights through ONNX — an export that
make space-onnx refuses to write unless it reproduces the recorded
accuracy over all 10 000 test images. It does: 0.6428 from both
runtimes, agreeing on 100.00% of predictions.
It serves its own artifact and its own number — resnet18-cifar10
v1 from artifact 519cecd1, one epoch, 64.28% top-1 — not the
65.59% run described above. Those are two separate one-epoch runs, and neither is
rounded toward the other. make space-verify re-evaluates the shipped
weights over all 10 000 test images inside the serving image and has to reproduce
0.6428 / 0.9933, which is what AshML recorded for that
artifact.
It is not the control plane. The scheduler, the executor and the REST API
need a Kubernetes cluster, PostgreSQL and an object store. The API authenticates
every request now, but it still has no rate limiting and no audit of refused
requests, so it is deliberately not something to put on a public URL. To run the
whole thing, clone the repository — make cluster && make journey
stands it up locally.
| Component | Where it runs |
|---|---|
| The trained model | Live in your browser (ONNX), and in-cluster behind serve.py |
| Control plane API + scheduler | Local only — needs Kubernetes; authenticated, but no rate limiting yet |
| Router / traffic splitting | Local only — needs two versions and real pods |
| Prometheus + Grafana | Local only — make observability, dashboards in git as JSON |
| Unit + integration tests | CI, against real Postgres and MinIO service containers |
| e2e, chaos, journey | Local only — they create and kill real Pods |
Two numbers sit next to each other on the cluster dashboard:
ashml_gpu_visible (2) and ashml_gpu_schedulable (0). Both are
true on this host, and either one alone is a lie about it.
nvidia.com/gpu: 0 and GPU jobs queue with an explanation.sim executor records simulated: true on its events, and its
"logs" say plainly that no container ran.git clone https://github.com/AuthRan/AshML && cd AshML
npm install
make db-up # PostgreSQL + MinIO
make migrate # apply schema
make cluster # create the local k3d cluster
make image # build the workload image, load it into the cluster
npm start # the control plane: API + executor
export ASHML_TOKEN=$(make -s token) # the API is default-deny
make journey # the §50 user journey, all nine steps, ~8 minutesRequires Node.js 22+, Docker, and k3d + kubectl. Full instructions, configuration and the honest limitations are in the README.