Fuzzball

Fuzzball v4.1.0 is a minor release centered on workload scheduling: workflow priority is now driven by a cluster-configurable expression, higher-priority work can preempt lower-priority preemptible workloads, and a backfill pass -- enabled by default -- puts idle capacity to work while large jobs wait, all backed by new observability commands, events, and Prometheus metrics. The release also adds HuggingFace Hub volume ingress, storage segmentation for clusters with multiple shared filesystems, an OCI Managed Lustre storage driver, and Podman support for the Docker Compose deployment stack, alongside a broad set of CLI, deployment, and federation improvements and bug fixes.

The scheduler now ranks, preempts, and backfills work based on a fully configurable notion of priority. Together these features let busy clusters keep urgent work moving without leaving capacity idle.

Configurable priority

Allocation priority is computed by a cluster-configurable scheduler.priority expression instead of a fixed formula. The expression combines organization, account (group), user, and workflow inputs plus queue aging, and is recomputed on every scheduling pass.

• Cluster admins set organization priority with fuzzball organization update <ID> --priority; organization owners set account, group, and user priority with the corresponding update --priority commands.
• fuzzball workflow start --priority takes a signed value less than or equal to zero (default 0), so users can deprioritize their own workflows but never raise them above their entitlement.
• Priority is visible throughout: a Priority column in fuzzball group list, fuzzball org member list, and fuzzball group member list; priority sorting in fuzzball workflow list; and fuzzball workflow describe <WORKFLOW> <STAGE> shows the full priority breakdown for a stage.
• fuzzball workflow events renders priority events readably ("Priority set (workflow: 0, organization: 5, group: 2, user: 1)") and emits an update event whenever an entity priority change is applied. Priority changes now propagate to queued and running allocations instead of affecting only new submissions.

Preemption

Higher-priority allocations can evict running, lower-priority allocations that are marked preemptible. Preemption is off by default and is enabled through the scheduler configuration.

• Workloads opt in via a top-level preemptible: true field in the workflow definition, per-job and per-service preemptible options, or fuzzball workflow start --preemptible.
• Eviction is conservative: a victim must be running, preemptible, past a minimum runtime, and below the preemptor by a configurable priority gap -- and it is only evicted when the freed capacity actually seats the blocked allocation. A utilization threshold gate keeps preemption from firing while the resource pool still has free capacity.
• Opt-in multi-victim preemption (preemptionMultiVictimEnabled) evicts a set of victims in one pass when no single victim frees enough, bounded by maxEvictionsPerTick.
• Organization owners control which groups each group is allowed to preempt with fuzzball group update <GROUP> --preempt/--no-preempt. Preemption lists are empty by default, and deleted groups are removed from all lists automatically.
• Preempted jobs and services are re-queued and restart cleanly on their next placement. fuzzball workflow events shows both sides of an eviction ("Preempted by workflow X job Y" / "Preempted workflow X job Y"), and Prometheus metrics report preemption misses, time blocked before an eviction, near-miss priority gaps, and preempted-allocation outcomes.

Backfill

When a high-priority job is blocked waiting for capacity, the scheduler now runs shorter jobs in the capacity it does not need yet -- without delaying the blocked job's predicted start time. Backfill is enabled by default.

• A job qualifies as a backfill candidate when it has an execute timeout so its runtime is bounded; the workflow DSL default satisfies this. Image and data staging jobs are also backfilled whenever they fit.
• Backfill applies to static pools and to dynamic provisioner definitions running at their node cap.
• Backfill decisions are surfaced in fuzzball workflow events and in Prometheus metrics covering reservations held and predicted head-job wait.

Scheduler observability

A set of new commands and endpoints explains what the scheduler is doing and why.

• fuzzball queue show and fuzzball queue stats display the scheduling queue and aggregate statistics. Both work against federate clusters: --cluster <id> (repeatable or comma-separated) scopes the query, and omitting it queries all member clusters.
• fuzzball workflow why explains an allocation's current placement state -- what it is waiting on, or why it landed where it did. It works on operator-deployed clusters and through federate. Owners can always explain their own allocations; explaining others' requires a dedicated permission.
• Every service's monitoring HTTP server (and the single binary) now serves Prometheus metrics at /metrics, including new scheduler interval metrics (tick counts by trigger and outcome, inter-tick spacing, tick duration) for tuning scheduler responsiveness.

Workflow volumes can now pull content directly from the HuggingFace Hub. An hf://<org>/<repo>[/<path>] source URI ingests a model, dataset, or space repository -- or a single file from one -- with optional revision, repository type, and include/exclude file filters. A dedicated HuggingFace secret type stores a personal access token for authenticated and gated/private repositories, and the workflow editor offers HuggingFace as a first-class ingress source with structured fields for the repository, type, path, revision, filters, and secret.

Clusters whose nodes do not all share the same filesystem can now express that topology. Storage provisioners and node provisioner definitions accept an optional segment field that restricts volume access to nodes with a matching segment; workflows referencing volumes from non-existent or conflicting segments are rejected at submission with a clear error. Unsegmented provisioners remain globally accessible, so existing deployments are unaffected.

Image staging is segment-aware: when jobs in different segments reference the same container image, a separate image stage runs in each segment so the converted SIF is cached on the correct shared filesystem, and the distributed image pull lock is scoped per segment to allow concurrent pulls.

A new oci_lustre storage driver backs a provisioner with an Oracle Cloud Infrastructure Managed Lustre file system -- supplied as a bring-your-own filesystem OCID or auto-provisioned from a compartment, availability domain, and subnet -- with each volume a subdirectory mounted via the native Lustre client. Self-provisioned capacity and performance tier are configurable. Substrate nodes serving these volumes must have the Lustre client kernel modules installed.

Self-provisioning storage provisioners (AWS EFS, OCI FSS, OCI Managed Lustre) now create their backing cloud infrastructure asynchronously, reporting a PROVISIONING status until ready, and the web interface's Volume Provisioners page recognizes the new driver type and status.

The fuzzball cluster docker-compose deployment stack can now run on Podman. The CLI auto-detects the container runtime, preferring Docker and falling back to Podman; pass --runtime podman or set FUZZBALL_CONTAINER_RUNTIME=podman to force it. Podman support is intended for local testing; production deployments should continue to use Docker.

• A generate-substrate-config subcommand was added to fuzzball cluster aws, azure, gcp, and oci. Like the docker-compose subcommand of the same name, it extracts the configuration an external substrate node needs to join the cluster -- for compute nodes that cannot mount the cluster's shared filesystem, such as nodes attached over a VPN.
• fuzzball run now expresses the full v4 volume model: repeatable --volume accepts ephemeral volumes by path and size (/scratch,size=100GiB), named persistent volumes (data:/mnt/data), provisioner-scoped volumes (nfs/data:/mnt/data), and selector-based ephemeral volumes (also in v4.0.1).
• fuzzball object put and get infer the object or file name from the source basename when the destination omits it (also in v4.0.1), and object put no longer replaces an existing object unless --force is given.
• fuzzball object list walks every page automatically instead of stopping at the first response with a next-page-token hint (also in v4.0.1).
• fuzzball member list filters by role with --owner; the old --relationship flag is hidden and deprecated.
• fuzzball organization list lists all organizations in the cluster for cluster admins, making it easy to find an organization ID for priority and other updates.
• fuzzball cluster <cloud> deploy accepts a --version flag on all four cloud providers (AWS, GCP, Azure, OCI) to deploy a specific Fuzzball version.
• fuzzball cluster aws cleanup is scoped to a single deployment by default, selected interactively or with --stack-name; --all restores the old clean-everything behavior.
• fuzzball cluster oci logs now uses the same log streaming as the AWS, GCP, and Azure equivalents: the default namespace is the Fuzzball services namespace, multi-container pods prompt for a container, and new --component and --since-time flags filter the stream.
• fuzzball queue show uses the standard table formatting and shows full allocation IDs so they can be copied into other commands.

• Reworked the AWS Marketplace deployment's provisioning flow to shorten deployment time and avoid CloudFormation timeouts on slow regions.
• fuzzball cluster aws deploy accepts --vpc-cidr to specify a custom CIDR block for the VPC (for example 172.16.0.0/16) to avoid overlap with existing networks, and the deployment's task runner now runs inside the stack's VPC.
• The operator CRD surfaces additional trusted issuers, allowing endpoints to trust tokens generated from a separate Keycloak (also in v4.0.1).
• OCI (OKE) deployments install metrics-server so HorizontalPodAutoscalers can scale under load.
• The Docker Compose stack persists the object cache across restarts and self-heals a desynced cache by purging metadata for missing blob files so they repopulate instead of returning errors.

• Workflow events now propagate from member clusters to federate, so fuzzball workflow events gives the full picture on federated workflows.
• fuzzball queue show, queue stats, and fuzzball workflow why work against federate clusters (see Scheduler observability above).
• Federate materializes service-scoped secret references when forwarding score requests to downstream clusters (also in v4.0.1).
• Fixed a storage provisioner conflict during replication on federate.

• Job and service script fields no longer need to begin with a #! shebang line; when no interpreter is specified, #!/bin/sh is assumed.
• Application-template inputs of type secret and volume can be left unset. An empty reference is treated as "unset" instead of rejected, so inputs that are only needed for some run configurations no longer have to be required or downgraded to plain strings. The web UI pickers gain an explicit "(None)" option.
• Catalog template run forms group configurable values under their display_category headings instead of lumping every categorized value under "Advanced".

• A Download CLI page is accessible from the Links menu in the sidebar (also in v4.0.1).
• Workflow events and logs can be downloaded from the interface.
• The workflow details page gains a stage selector, and the editor gains a node provisioner field and the ability to open a template directly in the editor.
• Member lists are unified across organization and group views, and the defaults form includes more fields (also in v4.0.1).
• Persistent volumes support annotations, and the egress field accepts custom values.
• The object browser shows the effective TTL for objects that inherit expiry from a group, organization, or cluster default, instead of labeling them "No expiry", and the TTL status filter classifies them accordingly.
• The filter button uses a funnel icon to better indicate filtering.

• Scheduler annotation matching is strict by default and configured via an ignored-annotations list (Services.Scheduler.IgnoredAnnotations, which replaces RecognizedAnnotations). Deployments relying on the old lenient matching should review their scheduler configuration when upgrading.
• A provisioner definition's maxNodes is now enforced as a per-definition node-pool cap. Previously it only clamped a single allocation's node spread, so a configured maxNodes had no effect on total pool size. At the cap, an allocation that fits waits for a node to free instead of over-provisioning.
• The definition.nodes policy variable for static node provisioners counts usable nodes (including fully allocated ones) instead of only nodes with free capacity, so nodes-gated policies no longer reject a pool exactly when it is fully utilized. Policies that relied on the old behavior will see jobs queue instead.
• The GCP provisioner selects hyperdisk-balanced boot disks for machine families that reject pd-balanced (G4, A4, A4X, H4D, X4, Z4) and routes GPU families to the NVIDIA substrate image.
• Scheduling no longer aborts an allocation when a node is missing a requested device; the node is skipped instead.

• Application-template rendering is hardened: the template functions that could read the orchestrate process environment or resolve hostnames (env, expandenv, getHostByName) are removed, so caller-supplied templates can no longer probe the server environment.
• Fixed a workflow list filter that could return other tenants' workflows.

• The v3 scheduler API adds a PreemptAllocation admin RPC and a PREEMPTED allocation status; existing RPCs and statuses are unchanged.
• The legacy reference field was removed from volume info and list API responses. Its v3-shape volume:// URI was incompatible with the v4 provisioner-based volume addressing model and misled users into pasting an unusable reference into --volume.

Storage & Objects

• Fixed silent data loss when copying to S3-compatible object stores such as Oracle Cloud: failed egress uploads now surface as errors instead of reporting success, and non-AWS S3 endpoints no longer receive a default request checksum encoding they reject.
• Fixed fuzzball volume list silently stopping at the first 25 volumes; it now returns every volume on a provisioner, and excludes system-generated ephemeral volumes from the output.
• Fixed SIF image caching silently failing when a deployment uses a custom domain; the correct domain is now used for certificate and upload-host generation.
• Image object cache references are recorded with the image architecture, so mixed-architecture clusters cache correctly.
• Hardened the object cache background purge, GC, and reconcile jobs against restart races.

Workflow Engine

• Fixed workflow catalogs written with deprecated v1 mount syntax by parsing specifications server-side and upgrading them to the v4 format (also in v4.0.1).
• Fixed workflow summary filters not being applied correctly.
• Service autoscaler minimum replica counts are honored on startup.
• Fixed two rendering errors triggered by opening or rendering a workflow or application carrying a value the client could not represent.
• Fixed scheduler leader takeover hanging indefinitely after an orchestrator pod replacement on clusters where the central config was never set.

Command-Line Interface

• Fixed fuzzball node deprovision silently doing nothing when given the node ID shown in fuzzball node list; both the ID and the hostname now work (also in v4.0.1).
• Filtering fuzzball member list by --email, --name, and other fields now returns matching members.
• Fixed fuzzball cluster aws subcommands that connect to the EKS cluster: the cluster name is resolved from stack metadata instead of assumed, and the generated bearer token includes the expiry parameter the cluster authenticator requires.
• Fixed fuzzball cluster azure list ignoring --location.
• Fixed fuzzball cluster oci logs crashing from a flag shorthand collision, and OCI subcommands failing with "no kubernetes endpoint" on VCN-native OKE clusters.
• Fixed a crash when interactively rendering or starting a catalog entry whose prompts include an optional secret or volume input left unset.

Deployment & Infrastructure

• Fixed AWS deployment failing in accounts whose 12-digit account ID begins with a zero: the ID was parsed as an integer, producing invalid IAM principal ARNs that failed the deploy about 30 minutes in. Account IDs are now handled as strings end-to-end.
• Decoupled the NATS JetStream cluster name from the Fuzzball display cluster name to prevent rolling update failures during version upgrades (also in v4.0.2).
• TLS certificates are issued before ingress resources are created, fixing a race on Kong-based deployments (also in v4.0.1).
• Fixed Orchestrate and Federate clusters being deployed with unhelpful default names (also in v4.0.1).
• Fixed a missing S3 session token in AWS deployments using temporary credentials.

Organization & Administration

• Organization data-access audit events are recorded under organization-specific subject methods instead of a workflow subject method, so audit logs can be queried and filtered by operation.
• Annotation and backend-spec edits to existing provisioner definitions are no longer silently dropped when re-applying the central config.
• Removing or renaming a provisioner definition no longer fails with a false "currently used by N active instance(s)" error when no instances are running (also in v4.0.1).

Web UI

• Bare ephemeral volumes are preserved when re-opening a workflow in the editor or rerunning from a previous run (also in v4.0.1).
• Workflow list filters now intersect instead of unioning.
• Fixed a missing volume driver entry on the GC volume form.