Safe Deploys and Fast Rollback for Jac Applications

Overview

Jac is an open-source, Python-like programming language from Jaseci Labs. The Jac stack, its tooling for running applications in the cloud, takes a Jac application from a laptop to Kubernetes: jac scale deploy builds the images, provisions Redis and MongoDB, and ships the application code as a content-addressed bundle into a constant container image. In 2026 I worked on three parts of that path: making a deploy run the exact Jac release the app pins, rolling back safely, and blue-green deploys in jacBuilder, the Jaseci Labs app builder that deploys apps through the same stack.

The changes follow one rule: a deploy should fail before it changes anything, and if it cannot complete, it should stop with an error instead of reporting partial success.

Version pinning

An app declares its toolchain in jac.toml with [project] jac-version. Before this work the pin was parsed but unused, so every deploy pulled the latest release.

#7713 made the pin govern deployment. jac create now writes an exact pin for the current release into new projects. The pin resolves to one release, and the pod binary, admin console and base image all come from that release. A pin that cannot be honoured, such as an unpublished version or a missing platform binary, stops the deploy with an error instead of falling back to latest. Version ranges are matched by a small comparator with no new dependency.

Several build steps still ran on the deploying machine’s own Jac version. The follow-up changes moved each of them onto the pinned release:

Artifact	Built by, before	Built by, after
Pod binary, base image, admin console	latest release	pinned release (#7713)
Sealed application bundle	deploying Jac	pinned release (#7811)
Client bundle and host dependencies	deploying Jac	pinned release (#8037)
Manifests and database provisioning	deploying Jac	pinned release (#8777)

#8011 moved the pin check to the start of the deploy. A pin naming an unpublished release used to fail minutes in, with a namespace, a MongoDB and a Redis already created. Reproduced on a local kind cluster with the same pin:

	First failure	Namespace afterwards
Before	manifest generation failed: … HTTP Error 404	exists
After	Pinned jac-version v0.34.99 is not a published release	not found

Two bugs turned up once pins were in real use:

• Range pins picked the wrong release (#8961, backported in #8962). The release list still contained tags from the retired Jaseci meta-package, which are numerically larger than every Jac tag, so jac-version = ">=0.37.3" resolved to 2.3.28 and the deploy failed. A tag now counts as a candidate only if it ships a Jac binary named after its own version. The jaclang.org site template was also switched to an exact pin (#8967), so new projects created from it are reproducible.
• A deploy could lose its database (#8777). An older deployer could provision a backend the newer pinned pod could not use. The pod then found no database URL and created its own on the pod filesystem, one per replica, which was lost on the next restart. A pinned deploy now runs on the release it pins, deploys across minor versions are stopped before contacting the cluster, and the runtime refuses to create its own database when it runs inside Kubernetes.

The same rule applies in jacBuilder, which had been replacing every app’s pin with its own Jac version. It now deploys each app on the version its author pinned.

Rollback across all services

A deploy already left what rollback needs in the cluster: the image is constant, the code is a bundle identified by its digest, and each deploy patches the existing objects instead of recreating them. #8304 added a way to list that history, keep it, and roll back to it.

The main design question was how to identify a release. Kubernetes revision numbers drift apart across services: a service whose pod template did not change gets no new revision, so “revision 2” can mean different releases for different services. A release is therefore identified by its bundle digest. rollback_to reverts every service to the revision with that digest. It stops with an error if any service no longer keeps that revision, and restores already-reverted services if a patch fails part-way, so the app never ends up running two versions. Rollback covers code only, and the PR says so. A related fix makes a configuration-only change roll the pods (#8303).

Blue-green deploys in jacBuilder

jacBuilder is a private repository, so this section describes the work without linking it.

With blue-green deploys on, a redeploy keeps the version it replaced running as a warm standby for a retention window. Rolling back inside that window only moves traffic: the Kubernetes Services switch to the standby’s pods, which are already running. A rollback picks the fastest path that can serve the requested release:

Path	What happens	Time on a local test cluster
Switch to standby	Services point at the warm standby	about 2 seconds
Revert revision	Deployments return to a retained revision	about 7 seconds
Rebuild	Check out the old commit and redeploy	about 4.5 minutes

The same paths are available from the IDE and the command line. Cancelling a deploy now reverts every service, including the gateway that serves the client bundle; before, a cancelled full-stack deploy could leave the new frontend in front of the old backend.

Validation

In the Jac repository, the deploy logic sits in pure functions that are tested directly: which releases exist, which revision each service reverts to, and in what order. #8777 was checked with fourteen deliberate mutations of the new logic, each caught by the tests.

For jacBuilder, an end-to-end test on a real cluster deploys three versions and rolls back through each path. It checks that no request fails while traffic switches to the standby, that a rollback is refused while another deploy is running, and that nothing is left behind after the app is destroyed.