Architecture¶

Labmill is one Go binary (cmd/labmill), plus the desktop app (cmd/labmill-gui). The agent loop runs on the user's laptop and calls Claude directly with the user's own ANTHROPIC_API_KEY, or through an optional broker. It supports Slurm clusters only. Every action the agent can take is a typed tool with its own approval rule. The design decisions behind this are recorded in CLAUDE.md.

Packages¶

Everything is implemented except the "Hey Labmill" voice feature and the headless Agent mode, which are planned and have no code. The packages Agent mode needs (agent, app, store, sshx, launch) don't depend on the TUI. The desktop app is built and self-tested (see GUI).

How a request flows¶

The layers, from the keyboard to the cluster:

TUI (Bridge) -> session -> agent -> app tools -> backends / transfer / store
                  |                                 |
                  +-- Approver = Bridge             +-- sshx.Connect -> helper -> cluster

1. TUI to session. The user presses Enter. tui calls Session.Send(ctx, text), implemented by launch.Session, which serializes turns and passes the history to agent.Agent.Send(ctx, history, text, origin). Origin is Typed or Voice; dictated text would get VoiceNote prepended (the voice front end doesn't exist yet).
2. Agent to model. The system prompt is Agent.System (fixed per runtime) plus Agent.SystemExtra(), read once per Send: launch.Build wires it to Workspace.Prompt, which names the user's chosen code and data folders (empty when none). Send runs the turn on the Go SDK's streaming tool runner (client.Beta.Messages.NewToolRunnerStreaming, internal/agent/runner.go) over the service and request options that agent.Model.BetaMessages returns (in production the direct model in internal/launch/model.go, or *broker.Client when a broker is configured). A request middleware sets X-Labmill-Task: broker.HintPlan for the first request of a Send, broker.HintRoutine after. The direct model maps the hint to the model (planning: claude-opus-5-5, else claude-sonnet-5-5; LABMILL_MODEL overrides both) and drops the header; a broker picks the model itself. Only the planning request asks for adaptive thinking (display: omitted, agent.PlanMaxTokens); a thinking block shows as the step "Thinking…", never as text. Prompt caching: a breakpoint on the fixed system block (SystemExtra is a second, uncached block), one on the last tool definition, and the top-level automatic breakpoint that follows the conversation (loaded skill text included). When a reply's prompt (input plus cache tokens) reaches Agent.CompactAt (default 150,000), runner.CompactBeforeNextTurn compacts the conversation with the compact-2026-09-04 beta before the next turn (after the last reply it compacts and stops); the UI shows the step "Summarizing earlier conversation…" and then the note "Summarized earlier conversation" (EventNote). The runner starts a reply's tools together; agent keeps them in Claude's order and one at a time. Text deltas reach the UI as EventText through the Bridge. With no broker URL and no ANTHROPIC_API_KEY, unavailableModel makes each turn fail with a plain explanation. Tests run the real runner against agent/agenttest, a fake broker that streams scripted messages.
3. Model access. Without a broker, newDirectModel builds an Anthropic client with the key from ANTHROPIC_API_KEY (read from the environment, kept in memory only) and WithoutEnvironmentDefaults, so no other ANTHROPIC_* variable changes where requests go. With a broker URL (broker_url in Site.yaml, or LABMILL_BROKER_URL), the broker client's middleware puts the bearer token from LABMILL_BROKER_TOKEN on every request. It never uses credentials or base URLs from the environment, and a stream that ends without message_stop is an error.
4. Tools. If Claude's reply asks for tools, the loop handles each tool_use block in order. A tool is a typed agent.Tool (fixed JSON schema, registered in an agent.Registry); there is no shell tool on the laptop (remote_command runs one argv command on a login node, see below). app.Tools provides the tools (see below) and agent.SkillTools the two skill tools.
5. Approval. A tool with NeedsApproval first builds a Proposal (Propose) and the loop asks the Approver, which is the TUI Bridge: it queues an ApprovalRequestMsg for the Bubble Tea program and waits for exactly one Decision. The wait also ends when the turn's context is canceled, so a cancel never hangs on an open dialog. If the user declines, Run is never called and Claude receives "The user declined: ..." as a normal (not is_error) tool result. If approved (EventApproved), the Decision (carrying the exact Proposal shown, and the user's edit only when the proposal was Editable) goes to Run. With no Approver, a gated tool fails with ErrNoApprover. A Propose that returns agent.ErrNoApproval (a read-only remote_command) skips the approval: the tool runs with the zero Decision and nobody is asked, also without an Approver.
6. Run. The tool does the work: it gets a backend or SSH connection from the app.Env function fields, and records state in internal/store (CreateJob, UpdateJob, AddEvent, SetManifest). Tool errors become error results and do not stop the loop; an approver error or a canceled context aborts Send. When that happens while the latest reply's tools run, their recorded results are kept, unknown tools keep "not found", and the rest get "aborted"/"canceled", so the history stays valid for the API; a cancel before any tool of this Send ran leaves the history (earlier turns included) as it was. Tools get Claude's input bytes as sent. In the TUI, writes to os.Stderr (e.g. SDK warnings) go to labmill.log (launch.Runtime.LogPath) while the screen is up.
7. Loop end. Tool results go back to Claude as one user message and the loop repeats until Claude stops asking for tools, MaxTurns (the runner's MaxIterations; 25 for a bare agent.Agent, 80 for the TUI/GUI sessions via launch.DefaultMaxTurns, override LABMILL_MAX_TURNS) is reached (not an error; that reply's tool calls get "not run" results and Send appends the assistant message agent.MaxTurnsMessage, "I stopped after N steps so you can check in. Say 'continue' and I'll go on.", so the history stays valid and the next user message carries on; ErrMaxTurns is no longer returned), or the reply is truncated or refused (ErrTruncated, ErrRefused). Send returns the updated history even on error.

The typed tools¶

Defined in internal/app/tools.go (the doc comment on app.Tools is the reference).

• Read-only: list_clusters, cluster_info, find_local_data, preview_file, stage_data, draft_job, job_status, list_jobs, cluster_jobs (the user's own jobs on the scheduler via the optional backend.Lister; includes jobs not started through Labmill), job_logs, diagnose_job (collects the facts of a failed job and keeps a local failure memory), update_todos (the to-do list shown above the input), ask_user (a question with 2 to 8 options shown as a selection dialog; see below).
• Approval-gated: upload_data, submit_job (editable script), cancel_job, fetch_results, cleanup_job (after a verified download, removes the ended Slurm job's own working files in its Labmill job folder in the global scratch: exact files only, each re-checked by SHA-256 on the cluster, then the folders that held them; never environments, never a tree-wide delete; see internal/app/cleanup.go), s3_credentials_to_cluster (copies one AWS profile to the login node's ~/.aws/credentials as its own section; always asks, the values never reach the model), set_cluster_account (changes the User of a cluster's login and files hosts in Labmill's own SSH file after a sign-in rejected the reused name; refuses entries of the user's own SSH config and entries another cluster shares; makes the SSH helper forget its logins for those aliases first, then clears the cached connections and user ID of the cluster), add_cluster (sets up a cluster of another institution this build has a profile for, only on the user's explicit request; writes clusters.yaml and the SSH settings, never runs anything on a cluster, and always asks, also under "Approve everything automatically"; the cluster is usable in the same session). Its scope also covers a cluster of the user's own institution that the install lacks (the profile gained it after setup; SiteInfo.HomeMissing, computed by launch.homeMissing from the home profile and clusters.yaml, is named in the prompt; site is then the home domain and username may be left empty for any institution: Env.KnownUsername(site) supplies the account already used there, from the login host of a configured cluster of that institution; the proposal names the cluster it came from, and an existing SSH entry's own User wins over it). It goes through the same plan/apply code (planAddCluster, addSiteCluster over the home profile plus the others), appends to clusters.yaml without site: (Cluster.yaml's order only orders the first-run list) and reuses an alias that already points at the host (AddClusterHost.ReusedAlias/ReusedUser, shown in the proposal and the preview; the entry's own User wins at sign-in, so AddClusterPlan.AccountConflict and a guard in addSiteCluster refuse a different account). "Missing" is site.Site.MissingClusters: not configured by name, nor by the same Slurm login HostName under another name; the settings page and the prompt share it. The settings path reads only the locked Config and the files (Runtime.CheckHomeCluster before the add starts), never Env.Site; a home add keeps the home key hosts non-foreign (connCache.noteAddedCluster). The GUI settings page lists these clusters (settings.View.MoreClusters) with an Add button: SettingsAPI.MoreClusterPlan previews, AddMoreCluster runs Runtime.AddHomeCluster as a turn of its own (the click is the approval); the TUI's /settings text lists them.
• Gated except read-only commands: remote_command (one command on a cluster's login node; see "Commands on the login node" below).

ask_user takes {question, options: [{label, description}], allow_other} (question at most 300 characters, 2 to 8 options with unique labels of at most 60 characters, descriptions at most 300, allow_other default true). It reaches the user through app.Env.Choose, which launch.Build wires to the front end's agent.Chooser (part of launch.UI): the TUI shows a modal list, the GUI a labmill:choice dialog. The result is "The user chose: ", "The user answered: " or "The user skipped the question.". A dismissed dialog or a turn that ends answers skipped. It changes nothing, so it is not gated, and the system prompt tells Claude never to use it to ask permission for an action that has its own approval. The system prompt also tells Claude to call it whenever it offers the user a choice, instead of a numbered list the user has to type an answer to.

Slurm placement rules. draft_job on a Slurm cluster reads the cluster's rules.SlurmHints (Cluster.yaml preemptible, local_scratch, global_scratch; SiteInfo.SlurmFor, stated by cluster_info). Inputs: scratch (local, global or job_folder, the default) and preemptible (leave out to use the preemptible partition when the profile names one, the partition was not chosen explicitly and the time limit fits; false; true to insist), plus spec.nodes. local needs a known node-local scratch and a single node: slurm.Render then wraps the command in a stage-in / run / stage-out of the whole job folder (the live %x-%j.out excluded) with a trap ... EXIT that removes the job's own local folder only after a good rsync back. global needs the work folder below the profile's global_scratch (a job without data must be given its own work_dir there; Labmill invents no folder shared with other users). Preemptible sets --partition, --qos, --requeue from the profile (a partition the model named becomes the fallback; with no fallback and no default partition the plan says the last step needs one); the local-scratch script also gets --signal=B:USR1@n and a USR1 trap that stops the program so the stage-out has time (JobSpec.Partition, QOS, Requeue). The draft records a plan event (placementPlan: storage, partition, fallback partition, resubmit limit of 2, the sbatch --test-only estimate) next to the spec event; automatic resubmission reads the fallback from it. submit_job's summary and the draft's answer show the plan in plain words, so the approval covers the resubmissions. An estimate that can't be made is not an error.

Commands on the login node (remote_command)¶

(Approvals can also be given automatically by agent.AutoApprover, the "Approve everything automatically" setting; proposals with Proposal.Secret, such as a secret-file read here, are excluded. See security.md.) remote_command {cluster, argv[]?, script?, timeout_seconds?} runs one command on the cluster's ssh_host over the cached SSH login (Env.LoginRunner, connCache.loginRunner), never on the laptop. The argv is joined with backend.Quote, so each element is one literal word for the login shell (no pipes, redirections, variables or globs); control and format characters are rejected (they could hide what an approval shows), and argv is capped at 64 words, 4 KB each, 16 KB in all. These are mechanics. Labmill puts no restrictions on what runs on the remote HPC system (owner rule): no path limits, no secret-file list, no interactive-program list, no check of resolved paths. The cluster's own permissions decide, and an interactive program (no terminal) just hangs until the timeout, which the tool description says. The only decision Labmill makes is whether a command waits for the user's OK, in remoteNeedsApproval (one function; the user setting "Run commands on the cluster without asking" lives in gui.json as runWithoutAsking, is read per command through Env.AutoRunRemote (settings.AutoRunRemote, so the GUI and the TUI share it and a change applies at once), and makes every non-secret command run at once). Scripts. Multi-line shell code goes in the optional script field (up to 64 KB, UTF-8, no NUL), never in an argv word (the control-character error says so). It is sent on stdin (Runner.Run's stdin) to the interpreter in argv, default bash -s (argv may name another, e.g. ["sh","-s"]), under timeout like any command; the script text never appears in the command line, so the login shell's quoting (tcsh included) and newlines don't matter. A script is always remoteGated (scriptPlan; there is no way to judge shell code read-only), the approval Details show the whole script, and Run re-plans and compares with the approved Details, so an edited script is refused. With the user's setting it runs at once, but the secret-file name check (secretArg over the script's words) still asks, and an auto-run's output is redacted. Steering. The prompt, the tool description and skillToolMap tell Claude: job and helper scripts come from draft_job files, submit only with submit_job, monitor with job_status/job_logs, fetch with fetch_results. A command (argv, a bash -c string, or a script line) that starts sbatch or srun is not refused; after the usual approval it runs, and its result starts with "(Use submit_job for submissions: it records the job, shows the review and monitors it.)" (submitsJob).

classifyRemote (internal/app/remote.go) decides:

• Read-only, runs at once (Propose returns agent.ErrNoApproval, so the loop runs the tool with the zero Decision and Run checks the class again): squeue/sinfo (not --iterate/-i), sacct (not -f/--file), sshare, sprio, sstat, seff, scontrol show job|node|partition|reservation|config [id], sacctmgr show|list (not -i), module avail|spider|list|show|whatis|keyword with plain module names (leading options only from a known no-value set such as -t, -r, --terse; no word starting with -, / or ~, none containing .., a leading . only as a pattern for avail/spider/keyword; every module and ml call, approved ones too, goes via bash -lc; run through bash -lc 'module "$@" 2>&1', since module is a shell function), command -v|-V <names> (names not starting with -; run through bash -c 'command "$@"' with the flag, -- and the names as parameters, since command is a builtin), and ls, cat, head, tail (not -f/-F/--follow/--pid), wc, du, df, findmnt (not --poll/--tab-file), lsblk, bare mount, stat, file (no options that read file names from a file or write; an exact long option wins over an abbreviation, and a long option's value may be the next word only when the option is written in full) on any path the account can read (a leading ~/ becomes ./, since argv has no shell and the command runs in the home folder; every other path is passed on as given, hidden and secret-looking ones included).
• Everything else needs approval. The proposal shows the command, the host (alias (HostName)), the exact line that runs (with the timeout wrapper), the time limit, "This command can change files on the cluster. Approve it only if you want it to run." and why Labmill asks (it is not on the read-only list). This is consent, not a restriction. Run rebuilds the proposal and refuses any difference (sameAsApproved).
• Nothing is refused, but secrets are not piped into Claude's context unasked. Reading ~/.ssh/config, .bashrc, a hidden file or /etc/passwd runs at once like any other read, and an approved command may do anything the account may. The one exception is a read-only command that prints content (cat, head, tail, file -z; not ls, stat, du, wc) and names a sign-in secret (secretArg, name-based only: symlinks, hard links, /proc/*/environ and tokens inside ordinary files such as .bashrc aren't detected; a minimal use of privacy.go's hardSecretPath: SSH private keys, .netrc, cloud credentials, Hugging Face and other tokens, *.pem, *.key, shell histories; not *.pub, ~/.ssh/config, config.d/*, known_hosts, authorized_keys): remotePlan moves it to approval, and the approval says "This file's content will be sent to Claude." instead of the change warning. After approval it runs as given. This is about a credential leak into the model's context, not a limit on the user. For the same reason the output of those content commands goes through redactSecrets (localdata.go, as preview_file does): lines that assign a password, token or key show «redacted». The laptop side is different and unchanged: privacy.go's rules for local files (and hardSecrets) still protect the user's own computer.

Every command runs under coreutils timeout -k 5 <N> (default 60 s, max 600 s; Labmill itself gives up 20 s later), with stdin closed and no PTY. The result has the exit code (124 explained as the time limit, 127 as "not found") and the last 64 KB of stdout and 16 KB of stderr; when SSHRunner already cut a stream at 4 MB (backend.TruncatedMarker), its first part is kept instead and the heading says so. Output is cleaned with textsafe.Clean, then the whole result with cleanText like job_logs.

The chosen folders¶

The user can choose a data folder and a code folder (desktop: the + menu in the chat box, dialog or drop; terminal: /data and /code). With only a data folder, Workspace.Prompt tells Claude to offer writing the analysis code (shown before anything runs). app.Env.Workspace holds them; app.CheckFolder refuses drive roots, folders that contain home, system folders and hidden or secret-looking paths. They are extra allowed roots and the defaults of the local tools: find_local_data searches them when under is empty, preview_file resolves relative paths in them (code first), and stage_data stages from the data folder (else the code folder) when root is empty. The privacy rules and approvals don't change. workspace.json next to clusters.yaml keeps the choice and the last five folders per kind for both front ends. Details: GUI.

The expected order is find, stage, upload (approval), draft, submit (approval), status/logs, fetch (approval). draft_job uses the remote folder that upload_data records when there is local data.

Samples without run scripts. preview_file never guesses: a missing path is answered by toolset.missingFile (localdata.go) with " doesn't exist in ; list the folder (find_local_data) instead of guessing names" plus up to 20 real entries of that folder (privacy skipper: no hidden or secret-looking names, only inside the allowed roots); after 3 such answers in a row (reset by find_local_data or a successful preview) it says to stop guessing. README and build files are paged and the prompt and tool description tell Claude to read all pages. draft_job takes an optional files list of {name, content} (jobfiles.go): plain names (no slash, no leading dot, 64 characters), text up to 64 KiB each, at most 8, checked with backend.CheckText. They are saved as a files job event, appended to the submit review below the script after marker lines (reviewWithFiles; splitReviewFiles parses the possibly edited text back, so the user can edit both), and submit_job writes them through the login-node runner (Env.LoginRunner, sh -c program with the content on stdin, temp file, chmod 755, mv -n) into spec.WorkDir just before Backend.Submit, under the submit approval. A different existing file is never overwritten (exit 3 and a message to pick another name), a symlinked folder is refused. safeJobName suggests a valid job name when a folder name such as "OneDrive - Example University" is given; transfer tests cover manifests and quoted remote paths with spaces.

Background job monitor¶

launch.Build starts a job monitor (internal/launch/monitor.go) and Runtime.Close stops it first, before the connections and the store close. Every 45 s (the pause starts when a check ends, so checks never overlap) it lists the store's jobs in submitted, pending or running with a remote ID, looks up each cluster's backend once per check, and makes at most one Status call per job, each with a 20 s timeout. A changed state or reason is saved with UpdateJob, but only if the job is still in a monitored state with the same remote ID (a job_status call or another Labmill process may have saved a newer state meanwhile). When the save moves a job to completed, failed or canceled, the monitor calls UI.JobNotice with a sentence such as "Job #12 (train) on bigcluster finished. Ask me to fetch the results." or "... failed (TIMEOUT). Ask me to look at the logs.". The notice goes to the UI only, not into the conversation. The TUI shows it as a highlighted chat line, the GUI as a labmill:job system line, and both refresh their job list at once. With two Labmill processes open, only the one whose save made the change shows the notice.

The monitor never logs in. It gets backends from connCache.backendNoLogin, which returns nothing (the cluster is skipped until the next check) unless the cluster can be reached without a login or a question. Slurm needs an open SSH login. connCache.existingClient uses the session's cached client if it is still alive, or else sshx.ConnectCached, which joins a login the helper already holds and never starts a helper or logs in. If the cache is busy (a login dialog may be open), it gives up at once.

sshx.ConnectCached connects to the helper with the user name labmill-cached-only:<alias>. The helper accepts that only while it holds a live upstream for the alias, without waiting for a login another client is running, and treats the connection as one for <alias>. An older helper that doesn't know the prefix tries to resolve the whole name as a host, which fails without prompting.

SSH helper and ssh-exec¶

SSH does not use an OpenSSH binary. sshx.Connect finds the per-user helper socket ($XDG_RUNTIME_DIR/labmill/ssh-v1.sock, else ~/.local/state/labmill/ssh-v1.sock; the number is sshx.HelperVersion, also in the lock file name ssh-v1.lock, so a Labmill never talks to a helper of another version: after an upgrade that bumps it the old helper serves old Labmill processes on its own socket and idles out, and the new Labmill starts a matching helper, which costs one extra login) and starts labmill ssh-helper if nothing answers. The helper holds the authenticated upstream connection per alias (ProxyJump chains included) and exits after 10 idle minutes. It speaks SSH on the local socket, with the alias as the user name, and proxies every channel (exec, sftp subsystem, direct-tcpip) 1:1 upstream. When it must log in, it relays the prompts (password, Duo, host-key question) to the foreground process, so the dialog always appears in the UI. sshx.ConnectCached is the variant that only reuses an existing login (see "Background job monitor"). launch.Build caches one client per alias for the session (connCache in internal/launch/launch.go).

labmill ssh-exec <alias> [-s sftp | <command>...] bridges stdin/stdout to a channel on that connection. internal/transfer points rclone's sftp backend at it through rclone's ssh option, so uploads and downloads run through the same login without a second one. TransferCheck makes sure a login exists first, so a transfer doesn't fail late. The cluster's files_host (a dedicated transfer host, e.g. files.example.edu) is the host upload_data and fetch_results use when set (the transfer check and ssh-exec run against that alias); without it they fall back to ssh_host, whose shell host may have the sftp subsystem disabled (some login hosts do, so ssh-exec falls back to running sftp-server as a command, which a site may not accept). Upload destinations are home-relative or absolute group folders such as /data/mygroup/...; hidden and system folders are refused, and approvals show home-relative folders as cluster:~/dir. The approval text names the transfer host. The first-run setup writes an OpenSSH entry for it (reusing the user's own alias for that HostName) and sets files_host.

How rclone is used¶

rclone is linked in as a library (internal/transfer); no rclone binary is run. Its sftp backend is pointed, through the ssh option, at labmill ssh-exec <alias> -s sftp, so the SFTP subsystem runs on the helper's existing connection. The s3 backend uses the AWS SDK credential chain (env_auth, optional profile) and never an access key. Backends are built directly with NewFs and the option defaults, never from a config file or RCLONE_* environment (the config is effectively /dev/null). Transfers copy file by file (CopyFile per manifest entry or requested path): never sync and never a delete. Before each send the local file is re-hashed (sha256) against the manifest; downloads check the file names (no .., no names the local OS can't hold) and refuse to replace existing files. Remote.RcloneCommand and S3Remote.RcloneCommand render the rclone-compatible parameters as a line starting rclone-lib: copy ... (quoted for POSIX or PowerShell by GOOS; the prefix makes clear that no rclone process exists; swapping it for rclone gives a stand-alone command). upload_data and fetch_results do not put it in the approval (the user approves the file list and destination, not the copy command); it goes into the tool result and is announced at the start of the transfer. Live progress: Progress carries bytes, files and the current file; transfer.Meter smooths the rate, computes the ETA and throttles to at most one line per 500 ms (the poller reports at 1 Hz, so in practice one per second); Env.Transfer receives transfer.Update values (start, progress, end), which launch.UI.Transfer hands to the front end (GUI event labmill:progress drives a bar row, the TUI shows the rclone-style line in its status bar). The result ends with Transferred 4.0 GiB in 1m31s (45 MiB/s), 10 files. Display only; never any credentials.

Download verification. After an SFTP download, transfer (verify.go) runs sha256sum on the cluster through the same transfer connection, in batches (at most 200 files, about 16 KiB of quoted command and 16 GiB of file data per call, with a timeout that grows with the bytes), and compares it with the SHA-256 of each local file. A file that differs is renamed <name>.mismatch (then .mismatch2, ...; never deleted) and is not in the result list; fetch_results then reports a WARNING naming the files, doesn't set the job to fetched and leaves its LocalDir empty, so the results can be fetched again. Where sha256sum isn't available, can't read a file or times out, the affected files are checked by size only and the result says so (Verification.SizeOnly, Failed). An S3 download relies on rclone's own checks (size and MD5). The skill side is research-computing filesystems-and-storage.md, "Integrity: checksums".

Every fenced code block in the GUI chat also has a Copy button (md.js copyButton, aria-label "Copy", attached with addEventListener; the CSP still allows no inline handlers).

Job records for the Labmill account¶

launch.Build starts account.Uploader (internal/launch/upload.go) next to the job monitor; Runtime.Close stops it before the store closes. Every 60 s a pass reads account.json; with no file it does nothing (no request, no queue row). Otherwise it:

1. Queues. Each job in completed, fetched, failed or canceled with a remote ID, saved after LinkedAt, whose last final state event is not before LinkedAt, and with no uploads row for its (job_id, remote_id), gets one row: a random client_job_ref (lmc- + 32 hex) and the report (account.BuildReport) as JSON. A resubmitted job (new remote ID) is a new report.
2. Sends. Due rows (next_at passed, not sent) go to POST /api/jobs with hpc_system_id filled in: first GET /api/systems/resolve with the login host (the ssh HostName of the cluster's alias), the cluster name and the site domain; when that gives no single system (400, 404, 409, or an older server without the route), GET /api/systems?q=<cluster> and the one entry with the same name and the institution of the cluster's site profile. The answer is cached per process. Success marks the row sent. A failure sets next_at by account.Backoff (1 min doubling, at most 6 h); a 400 stops the row (store.Never); a 401/403 stops the pass, so the rows wait for a new token. The API stores a client_job_ref once, so two Labmill processes sending the same row make one job.

The report has only scheduler facts: Slurm state (the raw state of a failed job such as TIMEOUT or OUT_OF_MEMORY, else FAILED; CANCELLED; COMPLETED), the failure category for time-out and out of memory, exit_code (code:signal), submit/start/end times and queue wait (start and end from sacct when the job's saved facts are of its current remote ID, else from the state events), used (elapsed, cpu_time, max_rss, gpu: what sacct reported), req (cpu, mem, gpu, gpu_type, time, nodes) and the partition from the draft's spec event, and module names (ecosystem: module; a module given as a path is left out). It never holds the script, command, environment, job name, folders, venv/conda/container names, log lines or the failure memory, so no free text needs redaction; the tests assert this.

internal/store is one SQLite database (pure-Go driver, WAL mode, 5 second busy timeout) at $XDG_STATE_HOME/labmill/labmill.db, falling back to ~/.local/state/labmill/labmill.db. The TUI, GUI and headless agent all open it, so there is no daemon. A job has a JobState (draft, staging, uploading, submitted, pending, running, completed, fetched, failed, canceled), the approved script, a manifest of staged files (path, size, SHA-256) and an append-only event history. UpdateJob runs in one transaction and records state changes as events. Job.Facts (schema 5, column facts, JSON) holds the scheduler facts of the last ended run (remote ID, exit code and signal, start, end, elapsed and CPU seconds, largest MaxRSS, allocated GPUs); the job monitor and job_status save them from backend.Status when sacct reports an end. Schema changes are migrations tracked with PRAGMA user_version; a database from a newer Labmill is refused. The scheduler remains the source of truth for live status.

Skills¶

Skills are Anthropic/Claude Code-format directories: SKILL.md (YAML frontmatter with name and description, then a markdown body) plus optional tools/*.sh and tools/*.md (helper scripts and notes; read_reference serves them as text as tools/<file>, and Labmill never runs them), references/*.md and <sub>/references/*.md (a site skill keeps one subfolder per cluster; read_reference names those files <sub>/<file>.md). They are embedded in the binary with go:embed.

• One set of skills, two hosts. The same skills work on their own in Claude Code and inside Labmill; Labmill adapts to them and never forks them. Some users only ever use the skills in Claude Code (see sites/README.md).
• Generic skills plus a site skill. skills/ holds the generic slurm-user and research-computing (see skills/SOURCE.md); they contain no site facts and say "see your site skill" where a value differs per site. A sites/<domain>/ folder is a site skill named site-<domain with dashes> with the institution's hosts, accounts and 2FA, partitions and GPUs, storage, proxy and software notes, plus Site.yaml for Labmill (Sites). Only sites/generic/ is part of this repository. launch.Build loads the generic skills plus the chosen site's skill (Site.SkillsFS(skills.FS)); sites/generic supplies site-generic, which explains how to find site facts without a profile.

• Shell steps map to typed tools. The skills were written for a shell. Labmill's system prompt names the site and its skill and carries a fixed mapping (app.skillToolMap): sinfo → cluster_info, squeue/sacct → job_status and job_logs, sbatch → draft_job and submit_job, looks at the code → find_local_data and preview_file, AskUserQuestion → ask_user, any other login-node step (module avail, sacctmgr show, scontrol show node, ls of a job folder, ...) → remote_command, and so on; logins, installs and proxy fixes are Labmill's job. The skills stay unmodified.

• Embedded plus override directory. skill.LoadWithOverride loads the embedded skills, then the skills in an override directory such as ~/.config/labmill/skills. An override skill replaces an embedded one with the same name, and new names are added, so a skill can be updated without a new release. The override can never disable the embedded set: problems come back as warnings and the embedded skills still load. Override files are read into memory through an os.Root (symlinks cannot escape the directory) and capped at skill.MaxFileSize (256 KiB).

• Progressive disclosure. The system prompt (Set.SystemPrompt) lists only each skill's name and description. Claude pulls in a skill's body with the load_skill tool and individual reference files with read_reference; both come from agent.SkillTools(set). Both are read-only and need no approval. Set.Preamble is put on top of every loaded skill to say that shell instructions must be carried out with Labmill's typed tools, never by installing software.
• Skills are guidance only. They never override the approval rules.

Know-how from a broker¶

When a broker is configured, it can serve newer skill bundles (internal/launch/knowledge.go): once per session, in the background, the client fetches the manifest, downloads a newer bundle, verifies the version hash and every file's sha256, and stores it under os.UserCacheDir()/labmill/knowledge/<domain>/<version>/ with a current pointer. Bundles are swapped only at startup. Skill source priority: the override folder (~/.config/labmill/skills, always wins), the verified cached bundle, the embedded copy. Without a broker, or offline, the embedded copy is used.

Key-only SSH hosts¶

A host with auth: key in Site.yaml (schema in Sites) accepts SSH keys only. The pieces:

• Offering keys (sshx.Dial). Password+2FA hosts are offered the host's IdentityFile keys only, as before. On a key-only host Dial offers, in this order, opts.Signers, the IdentityFile keys (or, when there is none, the default id_ed25519, id_ecdsa and id_rsa in the ssh folder) and the keys of the SSH agent, de-duplicated by fingerprint and capped at maxOfferedKeys. The scope keeps a password host from seeing every key (and from counting them against MaxAuthTries). Key files must exist, parse and, off Windows, be private (no group/other bits); loose configured keys are noted once, loose default keys are skipped silently. The agent is SSH_AUTH_SOCK (a Unix socket) or, on Windows, the OpenSSH agent's named pipe through go-winio (agent_windows.go); listing its keys has a 3 s deadline and ends with the login's context, so a stuck agent cannot hang the sign-in. agentDial is the test seam, and the tests of sshx, launch, e2e and transfer switch the developer's own agent off. The agent connection is open only while the login runs.
• Passphrases. On a key-only host (LoginOptions.KeyOnlyHosts, set by RunHelper from the site profile; matched by HostName, case-insensitive) the agent's keys are loaded first; an encrypted key file whose .pub fingerprint is among them is skipped without a question, any other encrypted key makes Dial ask AskKeyPassphrase (up to three tries per key, empty skips the key, a canceled dialog aborts). The question travels through the helper relay like the password. On password hosts encrypted keys are still skipped silently.
• Errors. A key-only server that accepts none of the keys gives ErrKeyRejected (or ErrNoKey when no key was offered) instead of ErrAuthDenied; both cross the helper relay as codes key and nokey. No password is ever asked on such a host, and there is no retry without keys: a server that hangs up after too many offered keys also gives ErrKeyRejected, with a hint to name the right key (IdentityFile, IdentitiesOnly).
• Guided setup (sshx.KeyGuide, launch.connCache.connect). connCache knows the site's key-only host names and key_upload text; keyGuideFor maps the alias to its HostName through sshconfig.Lookup. KeyGuide.Run wraps the helper connect and always signs in first (the helper may already hold a login or see another agent): ErrNoKey goes to the setup, and ErrKeyRejected offers a new key once. The setup asks AskKeyCreate, AskNewPassphrase (twice when one is typed), runs GenerateKey, writes IdentityFile into Labmill's file with sshconfig.AddLabmillIdentityFile (an alias that only the user's own config defines is never edited; a notice then gives the line to add and the upload step still follows), and asks AskKeyUpload with the public key in Question.Data. If the sign-in is rejected afterwards it asks for the upload again, at most three times; declining ends with ErrKeySetupDeclined.
• Reusing a key before creating one (internal/sshx/keyquery.go, keyreuse.go). KeyGuide.useExistingKey runs before the create step. LocalKeys lists the key files in the ssh folder that have a private sibling, plus the agent's keys. ListRemoteKeys runs one read-only POSIX script over an already-open connection to the home login node (HomeKeys.Client never logs in or asks; without a live connection the step is skipped silently) and returns file name, type, fingerprint and comment from the .pub files, at most 16, skipping symlinks that point outside ~/.ssh and names with control characters. QueryKeys asks the key-only host with the public-key query of RFC 4252 section 7: several keys per connection (at most 5, fewer after a "too many authentication failures" disconnect; 7 s pause between connections; partial results are returned when a later connection fails), authenticating with signers whose Sign never signs, so the server's SSH_MSG_USERAUTH_PK_OK ends the attempt before any login and no private key is needed. Jump hosts are dialed like Dial does (dialChain, same LoginOptions, shared hostKeyCheck for unknown-host questions); when a jump host is not key-only the check is skipped with one notice, because it would need a password and 2FA just for this. A local accepted key gets an IdentityFile entry (AddLabmillIdentityFile). An accepted key that exists only on the home cluster goes to askCopy (AskKeyCopy: the prompt names file, source, destination and the risk; default No; never auto-approved, as it counts as a secret read), then copyKeyFile reads the private key in memory (64 KB cap, 30 s timeout), checks it matches the accepted public key, writes it to a new file named id_<type>_<home alias> (type ed25519, rsa, ecdsa or key; _2, _3 when the name is taken) with mode 0600 (never overwriting) and adds the IdentityFile; the contents never reach Claude, the chat, logs or the store, and the notice after the copy says whether the key has a passphrase (asked at sign-in, never stored). A host that closes the connection before the handshake (OpenSSH per-source penalties) gives ErrHostLimiting: and when no key has been accepted yet the guide notifies, waits a minute, asks only the keys not yet asked, once, and if the host still refuses ends with "... is still refusing connections; wait a few minutes and try again" instead of creating a key (details in SSH). When nothing is accepted, the user declines, or the check was skipped because of a password jump host, the create-and-upload path above follows, naming the right institution (siteLabel) and its key_upload text.
• Key files. GenerateKey makes an ed25519 key in OpenSSH format (private 0600 with O_EXCL, .pub 0644, the folder 0700), named id_ed25519 when neither it nor its .pub exists, else id_ed25519_labmill_<alias> (then _2, _3, …); it never overwrites. On Windows Labmill does not touch ACLs: the files inherit the profile folder's, which is what Win32-OpenSSH accepts.
• Front ends. The questions are four new sshx.QuestionKind values. The GUI bridge maps them to prompt kinds keycreate, newpassphrase, keypassphrase and keyupload (the last carries PromptEvent.Data with the key; the page shows it as text with a Copy button, and make gui-selftest drives the whole dialog). The TUI renders AskKeyCreate as the yes/no confirm, AskKeyUpload as the instructions plus key above a confirm (Ctrl+Y copies), and the passphrases as masked inputs.
• Tests. sshtest.Options.KeyOnly makes a server that accepts only keys (Server.Authorize simulates the registry upload); internal/sshx/keys_test.go covers default-key discovery, the in-process agent, passphrases, key generation and the whole guide; internal/launch/keysetup_test.go covers the wiring.

No typed tool maps to this flow (it happens inside the sign-in dialog), so toolSkills has no anchor for it; the skill text is the research-computing ssh reference's "Clusters that accept only keys", and skillToolMap tells Claude that Labmill does the setup itself.

SSH config files¶

Labmill keeps SSH settings in OpenSSH format, so a user who later opens a terminal can run ssh <alias> and get the same connection.

• Labmill owns exactly one file, ~/.ssh/config.d/labmill, written by sshconfig.WriteHosts (validated, atomic, mode 0600). clusters.yaml stores only an alias (ssh_host), never host names or users.
• sshconfig.EnsureInclude puts Include config.d/labmill at the top of ~/.ssh/config. It has to be first: an Include after a Host or Match block only applies inside that block. If the line is missing, the original file is backed up as config.labmill-backup-<timestamp> and the rest of it is kept unchanged.
• Labmill's own Go SSH client does not depend on the parser following Include. sshconfig.Lookup reads the Labmill file first and the user's file second, and the first value for each keyword wins, which is what OpenSSH does.
• sshconfig.AddLabmillIdentityFile adds one IdentityFile line to an alias's block in Labmill's file (the key setup of key-only hosts); nothing else in the file changes.
• Values are validated before writing (Host.Validate, ValidAlias), because a newline in a host name would inject directives into a file that every ssh reads. ControlMaster options are left out on Windows, where Win32-OpenSSH rejects them.

Dependency direction¶

cmd/labmill → (tui, app, site) → agent → (skill, broker, backend, transfer, store, knowledge, rules) → (sshx, config, sshconfig). cmd/labmill and cmd/labmill-gui wire the pieces together through launch; app takes everything through the app.Env function fields. rules has no internal dependencies on the heavy packages. Every backend, transfer target and the model client has an interface, so tests use in-process fakes and need no network.