Skip to content

SSH login and OpenSSH config (design decisions)

The full design of the one-time SSH login, Labmill's own connection sharing and the OpenSSH config file. CLAUDE.md keeps a short summary. The implementation is in Architecture ("SSH helper and ssh-exec", "SSH config files").

Audience: researchers who already run code successfully on their own Mac or Windows computer, often on a single local GPU, and now need to scale out. They don't know the cluster environment, often don't know containers well, and want a chat interface to run their jobs. Nothing may assume they know the terminal, SSH or ~/.ssh/config.

Go-native SSH (golang.org/x/crypto/ssh) on every platform. Don't shell out to ssh. Labmill doesn't reuse the user's terminal SSH sessions. It runs its own login and its own multiplexing.

  • First login is a GUI/TUI dialog that looks like a normal SSH login.
  • Labmill asks for the username, then the password (keyboard-interactive auth).
  • When the server shows the 2FA menu (Duo-style, e.g. Passcode or option (1-3):), Labmill picks the "Duo Push" option itself (usually 1; the number is read from the menu, since phone-only users may have a call or SMS there) and tells the user to approve it on their phone. Some servers push without any menu, so after the password any wait is treated as a 2FA wait, with the push timeout armed. 2b. The push notice always clears. Prompter.Notify("") means "clear the status line". sshx.login sends it on every way out once it showed a push notice (success, failure, timeout, cancel), and sshx.Connect does the same for notices relayed from the helper, so a front end never keeps "Approve the sign-in request ..." after the wait ended. A helper that already holds the login relays no notice at all. The GUI's labmill:notify event with empty text and the TUI's NotifyMsg clear the line.
  • If no push arrives (timeout, or the server says the user isn't enrolled), show instructions for setting up 2FA. Make the enrollment link and text configurable per site rather than hardcoded. 3b. Lockout safety. Some institutions lock an account (e-mail too) after a few failed password logins. Labmill never resubmits a password by itself, and a password that was sent and rejected without any 2FA step (ErrPasswordRejected, which wraps ErrAuthDenied and has its own helper relay code password) now ends with the site's two_factor.help (when there is one) and sshx.PasswordCareNote: "Check your password before trying again: many institutions lock the account after repeated failed sign-ins." When no password was ever sent (for example a key-only host not flagged as such), the failure stays a plain ErrAuthDenied with neutral text ("the server accepted none of the offered sign-in methods") and no advice about the password. 2FA failures (ErrTwoFactor) keep their own message. The system prompt tells Claude to stop after such a result, explain it and wait for the user. In code, connCache.client records a host whose login ended in ErrPasswordRejected, together with the other hosts of the same institution (the cluster's ssh_host and files_host, siblingHosts, computed from the live config so a cluster added later counts; they share the account; a sibling is refused with "a sign-in at this institution failed a moment ago; check your password, then ask me to try again"), and, until the next turn (Session.Send runs connCache.newTurn), refuses a new login to it with "sign-in to failed a moment ago; check your password, then ask me to try again" without dialing. 2FA and network failures do not arm the guard.
  • Let the user type a passcode instead if they prefer.
  • Key-only sites (no password login) get a guided key setup instead of the password dialog. The guide first reuses a key the host already accepts: QueryKeys asks the host about public keys only (the RFC 4252 public-key query, which stops before any signature), for local key files not offered at sign-in (the agent's keys were already offered by Dial) and, over a connection that is already open, the home cluster's keys (ListRemoteKeys). An accepted key that exists only on the home node is copied to a new 0600 file after an approval dialog that defaults to No and is never auto-approved. If nothing is accepted, it creates a new key and uploads it per the site's instructions. See Architecture ("Guided setup"). 5b. Key query batching and host limiting (keyquery.go, keyreuse.go). Every rejected key in a query counts toward the server's MaxAuthTries, and OpenSSH 9.8+ penalizes a source after failed logins by closing new connections before the handshake. So QueryKeys offers several keys on one connection (ssh.PublicKeys with the query signers), at most maxKeysPerConn (5) per connection, and fewer after a "too many authentication failures" disconnect. It pauses queryPause (7 s, longer than the 5 s penalty; tests shorten it) between connections and returns the results gathered so far instead of nothing when a later connection fails. A host that closes before the handshake gives ErrHostLimiting (wrapped with the host name). When no key has been accepted yet, KeyGuide.useExistingKey then notifies, waits limitWait (a minute), and asks once more only the keys not yet asked. If the host still limits, it returns an error ending "is still refusing connections; wait a few minutes and try again" and does not go on to create a new key (the user would register one they may not need). A key accepted before the failure still counts.
  • Several institutions in one session. LoginOptions.Hosts maps a lowercase host name to its own HostLogin (2FA help and name), so each host shows its institution's 2FA text; the plain fields apply to every other host. The jump chain (dialChain) and the host key question (hostKeyCheck) are shared by Dial and QueryKeys, so a key query never accepts an unknown host silently either.
  • Labmill's own ControlMaster. One authenticated connection carries everything: commands run as separate sessions on it, and SFTP runs on it. Each process gets that from x/crypto/ssh for free. To share the connection across processes (TUI, GUI and Agent) and keep it alive briefly after Labmill exits, the first Labmill process starts a small helper that holds the connection. The helper listens on a per-user local socket (a Unix socket, or AF_UNIX on Windows 10+), and exits after about 10 minutes idle, like OpenSSH ControlPersist. Later Labmill processes connect to the socket instead of logging in again. Before the helper hands a held login to a new client (also to ConnectCached), it checks that the login still answers a keepalive request within the keepalive timeout (15 s), unless a keepalive answered within the last keepalive interval (30 s; measured on the wall clock, so time spent suspended counts). A login that doesn't answer (the laptop slept, the network changed) is dropped and its local clients closed, and the client gets a fresh login with its prompts (ConnectCached gets ErrNotConnected). A busy but working link is never dropped this way while its keepalives answer. A local client's own keepalive request is answered only once the cluster login answered one, so a caller can tell a stale login (drop the cached connection, connect again) from a merely slow command (keep it). Commands on a held connection honor their context also while the session opens (backend.SSHRunner), so a stale link times out instead of hanging.
  • SSH settings are kept in OpenSSH format, so the user can switch to a plain terminal later. Users may later want direct terminal access, so every SSH setting Labmill creates goes in an OpenSSH-compatible config file, and ssh <alias> / sftp <alias> simply works afterwards. This file is the single source of truth: Labmill reads its own connections from it, and clusters.yaml only refers to the alias.
  • Labmill owns ~/.ssh/config.d/labmill (on Windows, %USERPROFILE%\.ssh\config.d\labmill). It adds one Include config.d/labmill line at the top of ~/.ssh/config. The top matters because an Include placed after a Host/Match block only applies inside that block. If ~/.ssh/config is missing, create it, with ~/.ssh at 0700 and the files at 0600. Back up the existing file before the one-line edit, and never rewrite the user's own entries.
  • Directives written per host: Host <alias>, HostName, User, Port if it isn't 22, ProxyJump when there's a jump host, and ServerAliveInterval. On macOS/Linux also write ControlMaster auto, ControlPath ~/.ssh/controlmasters/%r@%h:%p (create that directory) and ControlPersist 10m, so later terminal use gets the same one-login behavior. Leave those three out on Windows, where Win32-OpenSSH errors on them.
  • Aliases: OpenSSH uses the first value it finds for each setting, and the Include sits at the top. If the user already has a Host entry for that hostname, reuse their alias instead of adding one. Otherwise pick an alias that doesn't clash with any of their entries.
  • The Go side parses this file plus the user's ~/.ssh/config with a Go ssh_config parser (check that it follows Include). It honors HostName, User, Port, ProxyJump and IdentityFile, and ignores directives it can't act on (Match … Exec, ControlMaster; Labmill's helper does that job).
  • Bastion vs. login host. At many sites the bastion is the Slurm login host, so it's a single hop. Where the login/submit host sits behind a separate bastion, Labmill does a ProxyJump.
  • x/crypto/ssh has no ProxyJump setting, but it supports the mechanism directly: dial the next hop over the existing connection (client.Dial("tcp", "login:22")), then run ssh.NewClientConn on that net.Conn. Chain this for several hops.
  • Each hop can need its own login, so the dialog may run once per hop.
  • The jump host is recorded as ProxyJump in the cluster's OpenSSH entry (see above), so terminal ssh takes the same route.

Interactive shell

labmill shell <cluster> [--dir <folder>] and the TUI's /shell <cluster> [--dir <folder>] open a shell on the login node over a new session of the helper's existing connection (sshx.RunShell: RequestPty with the local TERM (default xterm-256color) and size, then Shell, or with --dir the command cd -- <quoted dir> && exec $SHELL -l; window-change on resize: SIGWINCH on Unix, polling every 300 ms on Windows). sshx.ShellExec puts the local terminal in raw mode (golang.org/x/term; on Windows it also turns on VT input and output processing), restores it on every exit path including SIGTERM/SIGHUP, and reads stdin through a poll reader so no key typed after the shell ends is swallowed. It implements Bubble Tea's ExecCommand, so the TUI suspends with tea.Exec and resumes when the shell exits. The open session is a local connection to the helper, so its idle timer does not fire while a shell is open. Login prompts for a missing sign-in use the usual path (TUI: the login modal through the Bridge and the key setup; CLI: a terminal prompter that asks once and never retries a password). Privacy: the shell's bytes are never logged, shown in the chat or sent to Claude; labmill.log gets "shell opened/closed on " only.

Desktop terminal. The Wails app shows the same session in xterm.js tabs (internal/gui/terminal.go, frontend/terminal.js). PageAPI.OpenTerminal(cluster, dir, cols, rows) and OpenJobTerminal(jobID, cols, rows) (the folder comes from the job store, never from the page; the cluster and folder may not contain \n, \r or NUL) return an ID at once; launch.Runtime.RunTerminal then connects through the helper (the same forgetDenied/markDenied handling as OpenShell, so a missing sign-in shows the normal login dialog once), runs sshx.RunShell with the xterm.js size and closes its connection when the shell ends. Events to the page: labmill:term-out {id, data} (base64 of raw bytes, chunks of at most 16 KB, in order), labmill:term-state {id, state:"open"}, labmill:term-exit {id, code, message}. From the page: TerminalInput(id, seq, base64) (never blocks: typing before the connection is up is queued, 1 MB at most; bigger input fails with "paste too large"), TerminalResize(id, seq, cols, rows), TerminalAck(id, n), CloseTerminal(id) (no exit event). Ordering: Wails runs every bound call in its own goroutine, so the page sends one input call at a time (what is typed meanwhile is joined into the next call) and one resize at a time (newest size wins), and Go delivers input by sequence number (a reorder buffer of up to 64 calls; a rejected call still uses its number) and ignores a resize older than one already seen. Flow control: output stalls once 256 KB are unacknowledged; the page acknowledges what term.write has finished (batched every 10 ms), so a slow page pauses the stdout copy, the SSH window fills and the remote program waits; closing a terminal wakes a stalled writer. Sign-in: while a chat turn runs (gui.App.Busy) a terminal opens no dialog and clears no refusal; it uses a login the helper already holds (sshx.ConnectCached) or shows "Sign-in is possible once the assistant has finished; try again then" in its tab. Otherwise terminal connects to one host are serialized (terminalClient): the first clears an old refusal like /shell, a request that waited behind it does not and, if the password was rejected meanwhile, fails with the refusal instead of prompting; the page disables New terminal while a tab is connecting. At most 8 terminals; Terminals.CloseAll runs at shutdown. The connection to the helper stays open while a terminal exists, so the helper's idle timer does not fire. "Open in system terminal" (internal/gui/systerm.go) starts x-terminal-emulator/gnome-terminal/konsole/xfce4-terminal/xterm (Linux, the command wrapped in sh -c with the program and cluster as positional parameters), wt.exe or, without it, the CLI itself with CREATE_NEW_CONSOLE (Windows, systerm_windows.go; no cmd) or Terminal.app through osascript running exec /bin/sh -c '<script>' sh <cli> <cluster> (macOS; also for fish users) with labmill shell <cluster>; the cluster name must match [A-Za-z0-9][A-Za-z0-9._@-]*. The labmill binary is looked for next to the GUI binary (labmill or labmill-<GOOS>-<GOARCH>, .exe required on Windows; also beside a macOS .app), then on the PATH; the option is hidden when it isn't found. Privacy: the terminal's bytes only travel between Go and the page (no logging, no chat, no Claude); labmill.log gets "terminal opened/closed on " only. Vendored (MIT, license files beside them in frontend/vendor/xterm/): @xterm/xterm 6.0.0 (xterm.js, xterm.css) and @xterm/addon-fit 0.11.0 (addon-fit.js), copied unmodified from the npm packages; to upgrade, fetch the new lib/*.js, css/xterm.css and LICENSE files the same way and update these versions. xterm.js writes <style> elements for its layout and colors, so the page's Content-Security-Policy has style-src-elem 'self' 'unsafe-inline' (style attributes and scripts stay locked to 'self'; nothing in the page builds markup from untrusted text).

Details of the terminal handling. On Windows the console input is read as console input records (ReadConsoleInputW, key-down records with a character only, so a key-up or mouse record can never block a read and swallow the first key typed after the shell ends; Ctrl+Z is data). ShellExec.Run closes the input, waits for the read in flight (guardedReader) and only then restores the terminal. SIGTERM or SIGHUP while a shell is open ends it, restores the terminal and returns *sshx.ErrShellSignaled: the TUI quits, and labmill shell exits with 128 plus the signal number. A channel that closes without an exit status (dropped connection) is sshx.ErrShellClosed: labmill shell prints "the connection to closed" and exits 255. An explicit /shell clears only the "password rejected" refusal of its own host and the institution's other hosts (connCache.forgetDenied), never other institutions' refusals or the canceled-sign-in flag, and is still one login attempt.