Fuzzball

Fuzzball v4.3.1 is a patch release. It makes volume egress wait for the services that write the data, keeps compute nodes connected through JetStream and substrate-bridge outages, and makes provisioned cloud nodes honor the configured shared filesystem mount point and maxNodes cap. It also fixes TLS trust and DNS for nodes on Docker Compose deployments and keeps provisioning credentials out of node console logs and fuzzball debug dump bundles.

Important

Upgrade notes.

• A service that exits on its own with a nonzero code now ends FAILED, including exit codes 130 and 143 from SIGINT or SIGTERM. Previously it ended FINISHED. A service stopped with fuzzball workflow stop still ends CANCELED, and an autoscaled replica above replicas.min that exits returns to its pool without failing the workflow. Services meant to end successfully should exit 0.
• Provisioner shared filesystem settings are now validated during the operator reconcile. A provisioner's sharedFs uri, type and opts may contain only letters, digits and =,_./:@-, and type and opts must be set when uri is. GCP and OCI provisioners now require sharedFs.uri, since their nodes could not finish bootstrapping without one. An invalid value fails the reconcile with an error naming the field, so check enabled provisioners before upgrading.
• jetstream.replicas must be at least 1. Scaling an existing clustered JetStream deployment down to 1 is not supported.

• Workflows list sorts by submission time. The list now defaults to Submitted, newest first, instead of priority, so a running workflow submitted at a negative priority is no longer paged out of the default view.
• Pagination for storage lists. The storage provisioners and volumes lists now page through results 25, 50 or 100 at a time. Previously only the first page the server returned was reachable.
• Catalog search matches descriptions. Workflow catalog search now matches template descriptions as well as names and tags.
• Read-only templates. Opening the editor on a platform-provided template now offers Copy and Edit and View template instead of a notice with no actions.

• fuzzball-admin audit list paging and filtering. The command now lists the newest events first and accepts --page-size, --page-token, --filter, --order-by and --all. Previously it returned one page of 20 events with no way to request the next.

• Fixed an issue where provisioned nodes exposed the depot token and the substrate-bridge password in cloud-init commands and instance console logs. On AWS and Azure the credentials are now passed through root-only files that are removed after bootstrap, and on GCP and OCI the startup script no longer traces the commands that write them.
• Fixed an issue where fuzzball debug dump bundles contained live credentials from ConfigMaps, such as the SpiceDB preshared key, the Web UI session and cookie keys, and the substrate-bridge password. ConfigMaps, container environment variables, Fuzzball custom resource fields and object annotations are now scrubbed the same way Secrets already were.
• Fixed two cases where the instance metadata guard on compute nodes did not work as intended. If the metadata endpoint was briefly unreachable when the substrate started, the guard stayed off for the life of the process and workflow containers could reach instance metadata. Guard rules left behind by an unclean shutdown blocked metadata access for the host as well. Detection now retries for up to five minutes, and leftover rules are removed at startup.

• Fixed an issue where egress for a volume mounted only by services ran as soon as the volume was created, failing with "no such file or directory" before any service had written its output. Egress now waits for every service that mounts the volume to finish, and for autoscaled replica-pool services to go idle.
• Fixed an issue where multinode jobs and services using Open MPI could hang when ranks exchanged data across nodes, because Open MPI sent peer traffic over the node-local fb0 bridge, whose address is the same on every node. Multinode containers now set OMPI_MCA_btl_tcp_if_exclude=lo,fb0 unless the job's environment or command already selects TCP interfaces.
• Fixed an issue where a service that exited on its own with a nonzero code, including from SIGINT or SIGTERM, ended FINISHED. It now ends FAILED; see the upgrade notes.

• Fixed an issue where a burst of submissions, or a workflow's parallel jobs, launched more instances than a dynamic definition's maxNodes. Nodes that have been launched but not yet registered now count toward the cap.
• Fixed an issue where nodes started by the AWS, Azure, GCP and OCI provisioners mounted the shared filesystem at /mnt/fuzzball-sharedfs regardless of substrate.nfs.destination, which left their image cache on local disk and made multinode jobs fail when the image was pulled on another node. They now mount it at the configured destination, and a destination provisioned nodes cannot use fails the reconcile.
• Fixed an issue where the AWS, Azure, GCP and OCI provisioners wrote the shared filesystem's uri, type and opts into each node's fstab entry without validation, which could break node bootstrap. Invalid values now fail the reconcile, and an AWS or Azure provisioner without a shared filesystem no longer writes an fstab entry with empty fields.
• Fixed an issue where AWS-provisioned nodes never joined the cluster when the ingress certificate was signed by the cluster's internal CA. Nodes now trust the cluster CA before installing the orchestrate extension.
• Fixed an issue where compute nodes that substrate-bridge could not connect back to were deregistered even though they kept reporting health.
• Fixed an issue where the organization's compute policy view failed with cannot resolve backend_kind on Docker Compose deployments with static definitions, because Orchestrate could not read the central configuration there.

• Fixed an issue where storage provisioners and volumes could not be removed after a workflow ended without releasing its volumes. Such volumes are now torn down automatically, and in-use errors name the workflows still holding them.

• Fixed an issue where internal service calls failed with "token is expired" for up to an hour after the host or VM running Fuzzball resumed from suspend. Internal tokens are now renewed against the wall clock when they are used.
• Fixed an issue where an unreachable DNS upstream made lookups in the fuzzball. zone retry in a tight loop until the query timed out. Substrate nodes now cap connection attempts per query, so such lookups fail fast with SERVFAIL.

JetStream

• Fixed an issue where compute nodes permanently lost their JetStream connection after a substrate-bridge or JetStream outage of more than about two minutes. The substrate now keeps reconnecting, so nodes recover without restarting fuzzball-substrate.
• Fixed an issue where the workflow-state and scheduler key-value buckets were created with a single replica even on a JetStream cluster of three or more servers, so losing one JetStream server took the workflow service down. They now replicate across the cluster, existing single-replica buckets are raised after the upgrade, and a bucket's replica count is never lowered.
• Fixed an issue where new deployments with jetstream.replicas: 1 never became Ready. A single replica now runs as a standalone NATS server.

Operator

• Fixed two issues in the pre-upgrade database restore, which runs when an upgrade moves the database to PostgreSQL 16 or to the fuzzball-database namespace. A failed or partial restore was reported as successful, after which the dump was deleted. The restore could also fail on SpiceDB's transaction table, or bring back relationships deleted shortly before the upgrade. The operator now also declines to start the upgrade when SpiceDB transaction IDs are beyond 2147483647, which the repair step cannot handle.

Compose

• Fixed an issue where workflows on Docker Compose deployments failed TLS verification of the Fuzzball API, on both external substrate nodes and the built-in substrate-localnode. Compose now publishes the cluster root CA to both.
• Fixed an issue where external substrate nodes on Docker Compose deployments could not reach the Fuzzball DNS service, which broke workflow service names and short-name lookups in musl-based images. Nginx now proxies TCP port 5353 to Orchestrate.
• Fixed an nginx 502 error when signing in to the Web UI on Docker Compose deployments. Nginx's proxy buffers are now large enough for the login callback cookie.

• Fixed the dashboard showing 0 running workflows. The workflows summary now honors the requested sort order, so it returns the newest workflows.
• Fixed an issue where the active sort column on Web UI tables was hard to tell apart from other sortable columns, so a table could still look sorted by a column after switching to another.
• Fixed dates on the workflow detail page always showing as dd/MM/yy. Dates across the Web UI now follow the browser locale for English (US), Spanish, French, Portuguese, Japanese and Korean.
• Fixed the workflow graph missing the edge between an image stage and the jobs and services that use the image.
• Fixed filtering the workflow catalog by more than one category or provider showing featured templates in the Popular section.
• Fixed text inputs without a label when configuring a workflow template. Each field is now labeled with its name, with its display name shown as a hint.
• Restored vertical resizing of the job and service script editors in the workflow editor.
• Fixed the workflow detail page showing the workflow ID twice for unnamed workflows.

• Fixed fuzzball cluster config get -o json printing YAML.