madcat-os/lora
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
lora/review/infra.md
T

90 KiB
Raw Permalink Blame History

Infra (48 examples)

1. Describe the wireguard mesh moto infrastructure.

Moto G52 (rhode) added to WireGuard mesh — 2026-05-26. FULL MESH REACHABLE.

WG IP: 10.44.0.5/24 Public key: Ti0cRi6Bjr+hYQoaAD5OUAAk/4B7y0j7tKRTbAtN5SA= Topology: hub-spoke via bastion (91.98.87.226:51820) PersistentKeepalive: 25s Config path: /data/data/com.termux/files/usr/etc/wireguard/wg0.conf Bring up: su -c "export PATH=/data/data/com.termux/files/usr/bin:$PATH && wg-quick up /data/data/com.termux/files/usr/etc/wireguard/wg0.conf"

Reachability from moto (all verified): bastion (10.44.0.1) — ✓ 27ms sin (10.44.0.2) — ✓ (AllowedIPs 10.44.0.5/32 added to bastion peer on sin; wg syncconf reloaded) junkpile(10.44.0.3) — ✓ (same fix applied on junkpile) fuji (10.44.0.4) — ✓ 51ms (double-hop through bastion)

Bastion peer entry added: 10.44.0.5/32 in bastion's wg0.conf. SSH from moto verified: madcat@junkpile, madcat@bastion, madcat@sin, madcat@fuji all reachable. Moto pubkey distributed to authorized_keys on all four madcat accounts.

SSH: host alias "moto" on both chi and madcat on fuji → 192.168.88.155:8022, user u0_a166. SSH penalty lockout: sshd rate-limits by IP; clear with kill -HUP <sshd_pid> via ADB.

WG auto-start on boot: NOT configured — no runit service yet. Manual bring-up required after reboot.

Supersedes EEMS 6528 (stale — had sin/junkpile as unreachable).

2. Describe the updated 2026 05 26 infrastructure.

RunPod account — adam@saiden.pl

Active pods:

  • nd9buxiw4upwf2: H100 80GB HBM3, AP-JP-1 (Japan), $3.29/hr, 160-core Xeon 8460Y+, 251GB RAM. Running LoRA v4 training. SSH: madcat@157.66.254.33 -p 18238 Image: aladac/madcat-ml:cuda132

Past pods (killed):

  • 40fc262sbict3h: H100, v3 training, completed 2026-05-25
  • w97k9zlca0d1br: gonzales_style LoRA, completed

Custom template: obryb2a3d0 — 50GB container disk, 200GB volume at /workspace, ports 22/tcp + 8000/8188/7860 http, env: HF_HOME=/workspace/models, TMPDIR=/workspace/tmp, COMFYUI_HOME=/home/madcat/comfyui.

Network volumes:

  • "workspace" 200GB EU-CZ-1 (id: at6hod4ho1) — original, used for v3 + ComfyUI
  • 250GB AP-JP-1 (id: 6r5rd211hf) — current, used for v4

runpodctl: v2.3.0 on fuji (brew), v2.3.0 on sin. SSH: use -o IdentityAgent=none -i ~/.ssh/id_ed25519 for direct IP pods. ComfyUI base image: aladac/comfyui-base:sm86 (CUDA 12.4, 15.4GB). ML training image: aladac/madcat-ml:cuda132 (CUDA 13.2.1, dual venv, 36.9GB).

3. What do you know about iscsi ssd infra?

Junkpile SATA SSD (Goodram SSDPR-CX400-512, 477 GB, /dev/sdc1) configured as iSCSI target on 2026-04-20.

Target IQN: iqn.2026-03.com.junkpile:ssd0 Portal: 0.0.0.0:3260 (reach via 10.0.0.2 over Thunderbolt) Auth: none (generate_node_acls=1, demo_mode_write_protect=0) Backstore: block/ssd0 → /dev/sdc1, write-thru

Key gotcha: LIO targetcli defaults demo_mode_write_protect=1 on new TPGs. Must explicitly set to 0 or macOS Disk Utility gets "A writable disk is required" (-69772). The existing RAID target (iqn.2026-03.com.junkpile:scsi0, /dev/md0, 1.8 TiB) had this already fixed.

Disk was wiped clean (wipefs + fresh GPT via sgdisk) before export. Intended to be formatted as APFS from the Mac initiator side.

Coexists with RAID iSCSI target on same port 3260.

4. What do you know about mesh vpn infra?

MARAUDER Mesh VPN — current state 2026-05-11 evening (TESTBED ADDENDUM).

Updates the 2026-05-11 14:33 state capture (EEMS id 5390) with the three-tier shape now operational on junkpile, plus carryover deferred items.

Three-tier shape (NEW as of 2026-05-11 21:00 CEST)

Tier Network Hub Purpose
PROD 10.8.0.0/24 OpenVPN marauder.saiden.dev (Hetzner CAX21 ARM) Real ops — Pilot + fuji + junkpile + sazabi + tachikoma + moto
DEV 10.99/10.98 (libvirt marauder-dev) hub-vm on junkpile (hostname=marauder, x86_64) Iteration / smoke testing
TEST 10.97 (libvirt marauder-test, no VPN) hub-test-vm on junkpile (hostname=marauder, x86_64) BT-operated headless visor regression

Dev tier: hub-vm + fuji-sib + sazabi-sib. Full OpenVPN + mosquitto + marauder-os + Catapult. 3-node CRDT sync convergence validated.

Test tier: hub-test-vm only. No OpenVPN (everything on libvirt-bridge side). Mosquitto bound to 10.97.0.1:1883, three users (hub/visor-test/bt-test). Headless visor on junkpile-host:99 (Xvfb + Mesa llvmpipe) responds to BT-published events. JSON event schemas validated for comms + display_state (SERE eye).

Junkpile-side glue

/etc/hosts: 10.99.0.1 marauder.saiden.dev (pins Catapult's hardcoded SSH alias to dev hub-vm, NOT prod) ~/.ssh/config: testbed FQDN override + Host 10.99.0.* wildcard + Host 10.97.0.* wildcard ~/.ssh/marauder-test_ed25519 keypair

Carryover (deferred from earlier 5390)

  • fuji OpenVPN to prod hub still runs via manual daemon (no launchd) — flaps ~5×/session
  • 4 mosquitto users on prod still using pass=marauder (weak)

Full testbed inventory

See infra.testbed.host-marauder (EEMS 5500) for snapshots, scripts, access notes. See win.host-marauder-testbed-* (5493, 5498, 5501, 5504, 5505) for delivery narratives.

5. What is the current state of hu jira markdown quirk bold code em dash?

hu v0.2.0+ Markdown→ADF parser hits an INVALID_INPUT from Atlassian's ADF validator when a single bullet line combines:

  • bold open **
  • inline code mark with { } braces inside (e.g. find_each { |u| u.update!(attrs) })
  • bold close **
  • em-dash separator —
  • multiple subsequent inline code marks
  • text continuing past

Verified 2026-04-30 23:40 CEST: MT3-9321 body push failed repeatedly until line 23 was simplified. Bisecting confirmed line 23 was the only trigger; sed-replacing the pipe characters alone didn't fix it (so it's not a table-misparse). Simplifying to plain prose with single inline backticks (no bold, no em-dash on that line) pushed cleanly.

Workaround

When pushing rich Jira bodies via hu jira update --body, avoid combining bold + complex inline code + em-dash + multiple backticks on the same bullet. Pick at most two of those decorations per bullet. If the combination is needed for clarity, split into multiple shorter bullets.

Suggested upstream fix

Investigate src/jira/adf.rs::markdown_to_adf for how it handles overlapping marks within a single inline run. Likely the ADF document it produces has invalid mark nesting (e.g. code mark applied to a node that also has strong and a child paragraph break) and Atlassian's validator rejects it.

Test fixture for the bug: a single bullet of the shape:

- **prefix `code with { } chars`** — text `more code` text `final code` text.

That triggers INVALID_INPUT on the Marketer Jira instance.

Linked

  • tooling.hu-jira-rich-body (3317) — the v0.2.0 Markdown→ADF feature being used
  • project.marketer.jira-instance-format (3300) — superseded by 3317 but historical context for plain-text fallback
  • 2026-04-30 incident: MT3-9321 prettify pass

6. What is the current state of runners?

Hetzner self-hosted GitHub Actions runners for Rust CI builds.

Provisioned 2026-04-14:

  • runner-amd64: cx33 (4 vCPU x86 shared, 8GB, 80GB) @ FSN1 — IP 88.198.104.212 — ~7.98 EUR/mo
  • runner-arm64: cax21 (4 vCPU ARM shared, 8GB, 80GB) @ FSN1 — IP 167.235.198.213 — ~9.83 EUR/mo
  • Total: ~17.81 EUR/mo (~75 PLN)

Runner config:

  • Registered at tengu-apps ORG level (not repo level)
  • Labels: self-hosted, Linux, X64 (amd) / ARM64 (arm), rust, hetzner
  • 1 runner per VM, systemd service (actions.runner.tengu-apps.runner-{amd64,arm64})
  • sccache installed for build caching
  • gh CLI installed on both
  • IMPORTANT: runner group must have allows_public_repositories=true for public repos

Workflow migration pattern: runs-on: [self-hosted, Linux, X64] # AMD64 builds runs-on: [self-hosted, Linux, ARM64] # ARM64 builds (native, no cross needed!) runs-on: macos-latest # Mac stays on GitHub (fuji runners REMOVED)

SSH access: ssh root@88.198.104.212 (amd), ssh root@167.235.198.213 (arm)

Old runners (fuji, junkpile) removed from all repos: tengu-apps/tengu-init, tengu-apps/tengu, saiden-dev/hu.

First migrated repo: tengu-apps/tengu-init (pipeline.yml updated, macOS builds disabled with if:false)

Build times on Hetzner:

  • CI (lint+types+test): ~20s each
  • AMD build: ~1m30s
  • ARM build: ~1m23s (native!)
  • Deb packages: ~1m each
  • Total pipeline (Linux only): ~5 min

7. What do you know about claude code on hetzner mesh infra?

Claude Code installed + configured on flux and swarm under the marauder user (2026-05-13 00:50 CEST).

Stack on each host

  • Binary: /home/linuxbrew/.linuxbrew/bin/claude v2.1.140 (via npm install -g @anthropic-ai/claude-code)
  • Auth: ~/.claude/.credentials.json (Pro/Max subscription token; flux's was seeded by copying swarm's existing file — confirms the token is NOT device-pinned, portable across hosts)
  • Settings: ~/.claude/settings.json — stripped statusLine (no marauder-status binary on Linux hosts), kept hooks/permissions/enabledPlugins/extraKnownMarketplaces
  • Marketplaces:
    • saiden → ~/.claude/plugins/marketplaces/saiden (git-cloned from git@github.com:saiden-dev/claude-plugins.git; both hosts auth to GitHub as marauder-actual)
    • claude-plugins-official → GitHub anthropics/claude-plugins-official (HTTPS, public)
  • Plugins installed (all enabled):
    • marauder@saiden v0.3.0-37a6d14 — MCP server, agents, slash commands, hooks
    • skill-creator, claude-code-setup, agent-sdk-dev, plugin-dev, rust-analyzer-lsp, claude-md-management, slack (all @claude-plugins-official)
  • Persona cart: flux → cart=flux, swarm → cart=swarm (already set in ~/.config/marauder/config.toml)
  • MCP verification: claude mcp list shows plugin:marauder:core: marauder mcp - ✓ Connected on both hosts. End-to-end MCP tool call works via claude --print.

Install gotchas (for next time)

  1. claude plugin marketplace add <source> takes ONE positional arg, not a name+source pair. Name auto-derives from the marketplace's marketplace.json.
  2. Accepted source formats: owner/repo, https://..., or ./relative/path — absolute paths and git@github.com: SSH URLs are rejected. For private SSH repos: clone manually to ~/.claude/plugins/marketplaces/<name>/, then cd to parent and add ./<name>.
  3. The official marketplace must be registered explicitly with claude plugin marketplace add anthropics/claude-plugins-official — it's not auto-registered just because settings.json lists plugins from it. Without this, plugin install <p>@claude-plugins-official fails with "Plugin not found in marketplace".
  4. swarm ended up with duplicate plugin entries at both project and user scope (leftover from prior project-scope state in marauder-agent dir). Not harmful — same plugin enabled via two scopes. Clean with claude plugin disable <p>@<mkt> --scope project later if needed.

Why this matters

SWARM coordinator (marauder-agent.service on swarm) and flux's DevOps agent can now drive real claude --print invocations with full marauder plugin context — slash commands, agents, MCP memory/persona/TTS — not just the raw model-loop bridge. Required for /marauder:plan, /marauder:execute, coda dispatch, and any agent-orchestrated flow that depends on the marauder slash commands.

Replay command (single host)

ssh <host>
export PATH=/home/linuxbrew/.linuxbrew/bin:$PATH
npm install -g @anthropic-ai/claude-code
# auth: scp creds.json from another working host OR run `claude setup-token`
git clone git@github.com:saiden-dev/claude-plugins.git ~/.claude/plugins/marketplaces/saiden
cd ~/.claude/plugins/marketplaces
claude plugin marketplace add ./saiden
claude plugin marketplace add anthropics/claude-plugins-official
claude plugin install marauder@saiden
for p in skill-creator claude-code-setup agent-sdk-dev plugin-dev rust-analyzer-lsp claude-md-management slack; do
  claude plugin install ${p}@claude-plugins-official
done
claude mcp list  # verify plugin:marauder:core ✓ Connected

8. What is the current state of catapult bubble mise activation?

Mise toolchain activation in Catapult bubbles — non-obvious behavior that bit BE CODA on MT3-9320 (2026-04-30 23:08 CEST).

The problem

Claude Code's tool-use bash spawns are non-login, non-interactive shells — they do NOT source ~/.bashrc or ~/.profile. Mise is normally activated via eval "$(mise activate bash)" in ~/.bashrc, so non-login shells skip it.

When CODA inside a bubble's claude pane runs bundle exec rspec or similar, its bash subprocess doesn't have mise activated → falls back to system Ruby (whatever /usr/bin/ruby is, often a stale version) → bundle fails → CODA chases the wrong fix.

What CODA did wrong

BE CODA on MT3-9322 (specs branch) needed to run bundle exec rspec. Bundle complained about Ruby version mismatch. CODA spotted a Dockerfile in the repo, saw FROM ruby:3.4.2, concluded "this project uses Docker" — and started a docker run --rm ... ruby:3.4.2 ... container to run the specs. Wrong tree entirely. The bubble has Ruby 3.4.2 already, just not activated in the tool's shell.

The fix

Source mise at the top of bin/catapult-env.sh:

# --- mise toolchain activation ---
# Claude Code's tool-use bash spawns are non-login, non-interactive shells —
# they do NOT source ~/.bashrc, so mise is NOT auto-activated.
if command -v mise >/dev/null 2>&1; then
  eval "$(mise env -s bash 2>/dev/null)" || true
fi

mise env -s bash outputs the env-var exports (PATH manipulation, etc.) without requiring an interactive shell. Sourcing catapult-env.sh then gives you mise-activated Ruby + catapult-managed DATABASE_URL + REDIS_URL in one step.

Where this matters

  • BE projects (mise-pinned Ruby): every bundle / rspec / rails invocation needs mise-activated PATH. Patch confirmed for marketer; same applies to any other Ruby project under marauder user.
  • FE projects (mise-pinned Node): less hit because linuxbrew also provides yarn + node on PATH; CODA can usually fall back. But if the project pins a Node version not matching linuxbrew's, the same problem recurs.

CODA dispatch prompt addendum (optional)

For belt-and-suspenders, future CODA prompts can include: "Always prefix bundle/rspec/yarn commands with eval \"\$(mise env -s bash)\" && source bin/catapult-env.sh && ...."

But: if catapult-env.sh itself sources mise (this fix), CODA only needs source bin/catapult-env.sh and everything works.

Verification

After patching bin/catapult-env.sh and syncing to junkpile + the live worktree, sourcing it from a fresh non-login bash gives:

  • which ruby → ~/.local/share/mise/installs/ruby/3.4.2/bin/ruby ✅
  • bundle --version → matches Gemfile.lock's bundler version ✅
  • DATABASE_URL set to postgres://localhost:4000/marketer_development ✅

References

  • ~/.config/catapult/projects/marketer/bin/catapult-env.sh — the patched file
  • Memory: project.catapult.mise-trust-path (existing) — mise security trust-path config
  • Memory: project.catapult.helper-scripts-spec (3299) — punch list for the next session
  • 2026-04-30 incident: BE CODA chasing docker for ~10 min before Pilot caught it

9. What do you know about infrastructure tts voices jarvis installed?

JARVIS voice installed and verified, 2026-05-02 18:21 CEST.

SOURCE: huggingface.co/jgkawell/jarvis (MIT license, piper-compatible ONNX) FILES: jarvis-high.onnx (108 MB) + jarvis-high.onnx.json INSTALLED LOCATIONS:

  • ~/.local/share/psn/voices/jarvis-high.onnx + .json
  • ~/Library/Application Support/marauder/voices/jarvis-high.onnx + .json

VOICE NAME IN CLI: jarvis-high (matches filename) USAGE: marauder tts speak --voice jarvis-high "..." confirmed working.

VOICE CHARACTER: Marvel JARVIS (Paul Bettany). British, butler-precise, calm-mature register. Sits opposite BT-7274 in tonal palette — BT is tactical baritone, JARVIS is old-world precision.

DESIGNATED USE: cameo voice for Episode 02 (Frankenstein Stack) — the after-hours-phone moment in the closing CTA. Replaces F.R.I.D.A.Y. (off the table — no perfect voice yet).

FUTURE USE: any beat needing British calm-authority register. Pairs well with content about craft, ownership, old-world engineering values. Not the right fit for tactical/military content (that's BT) or grumpy-old-man content (that's HAL, GLaDOS, SHODAN already in inventory).

VOICE INVENTORY AS OF NOW:

  • bt7274 (default, tactical baritone, Glenn Steinbaum)
  • glados (passive-aggressive, Portal)
  • hal (polite menace, 2001)
  • shodan (megalomaniac, System Shock 2)
  • sprite (unknown character)
  • jarvis-high (NEW — British butler-precise, MCU)
  • en_US-amy/hfc/kathleen/kristin/lessac (utility English)
  • pl_PL-gosia/mc_speech/mls (utility Polish)

Locked: 2026-05-02 18:22 CEST.

10. What is the current state of shares?

/Volumes/chi on fuji is a Samba share of chi's home directory on junkpile.

Provides direct filesystem access to junkpile's ~/Projects, models, configs, etc. without SSH.

How to apply: When needing to read/write files on junkpile from fuji, can use /Volumes/chi/ path directly instead of SSH + remote commands. Useful for large file operations, browsing project files, or accessing model weights.

11. What is the current state of deepseek r1 32b evaluation?

DeepSeek-R1-Distill-Qwen-32B-AWQ evaluation — 2026-05-23, chi@fuji.

MODEL BEHAVIOR:

  • Chain-of-thought via blocks — shows reasoning transparently
  • Honest about uncertainty in ("I'm not sure", "I should double-check")
  • But still confabulates specific numbers from parametric knowledge
  • Without context: says 11600 is correct (wrong), hallucinates 19% health rate (should be 4.9%)
  • With context values to verify: flags deduction cap as "UNCERTAIN", self-corrects on ZUS rates
  • With RAG/reference material: PERFECT — correctly flags all 3 errors, traces each to source

KEY FINDING: DeepSeek R1 is an excellent VERIFIER with reference material but cannot GENERATE ground truth. The science agent needs web search or RAG to be useful. Without external data, DeepSeek is honestly wrong (shows doubt in ) while Qwen is confidently wrong (says "✅ CORRECT").

COMPARISON (same question: "is 11600 the correct 2025 liniowy cap?"):

  • Qwen science: "✅ CORRECT" (wrong, no reasoning shown)
  • DeepSeek without context: "correct based on 2023 data" (wrong, but shows uncertainty)
  • DeepSeek with values to verify: "INCORRECT, should be 12000" (wrong number but flagged correctly)
  • DeepSeek with RAG reference: "INCORRECT, correct value 12900" (correct, traced to source)
  • Opus (me): "INCORRECT, should be 12900" (correct, from first run)

RECOMMENDATION: Science agent = DeepSeek R1 + brave-search MCP or web fetcher. The model is right, it just needs data.

OPERATIONAL NOTES:

  • tools must be disabled ("*": "deny") — DeepSeek doesn't support tool calling
  • opencode sends tools by default → 400 Bad Request from vLLM
  • Compaction interfered with responses — disabled globally
  • ~12 tok/s generation speed on GB10 at 25% GPU util
  • tokens count against context but are the value proposition

12. Describe the sin tunnels killed wg repoint infrastructure.

Sin autossh tunnels killed, configs repointed to WG IPs (2026-05-24).

KILLED: dev.saiden.sin-tunnels LaunchAgent — stopped, plist moved to .disabled Was forwarding: 18000→8000(vLLM), 18001→8001(embed), 18002→8002(deepseek), 24099→14099(TTS) All tunnel ports confirmed clear on fuji.

REPOINTED (localhost tunnel ports → 10.44.0.2 WG direct): opencode.json vllm provider: localhost:18000 → 10.44.0.2:8000 opencode.json vllm-deepseek: localhost:18002 → 10.44.0.2:8002 opencode.json ollama: 192.168.88.108:11434 → 10.44.0.2:11434 science-preprocess.ts QWEN_URL: localhost:18000 → 10.44.0.2:8000

VERIFIED: 10.44.0.2:8000 (vLLM qwen3) → 200 10.44.0.2:8002 (vLLM deepseek-r1) → 200 10.44.0.2:11434 (ollama) → not running (vLLM replaced it, config left for future use)

NOT TOUCHED: dev.saiden.tunnel-junkpile — junkpile WG (10.44.0.3) unreachable, tunnel kept junkpile tunnel uses SSH alias 'j' — still has the plist bug (item #8 from backlog) marauder config.toml — moto/router IPs unchanged (192.168.88.x, unrelated to sin)

13. What do you know about hu cli capabilities infra?

hu CLI capabilities (as of 2026-04-30, hu v0.1.14):

Available subcommands

hu jira:
  auth     OAuth flow
  tickets  list my sprint tickets
  sprint   list current sprint issues
  sprints  list all sprints (active/future/closed)
  search   JQL search
  show     read a ticket
  update   modify a ticket: --summary, --status, --assign, --body
  
hu gh:    GitHub PR/run/failure ops
hu slack: messages/channels
hu pagerduty: oncall/alerts
hu sentry: issues/errors
hu newrelic: incidents/queries
hu eks:    pod access (list/exec/logs)
hu pipeline: CodePipeline status
hu read:   smart file reading (outline/interface/around/diff)
hu data:   Claude Code session data (sync/stats/search)
hu docs:   doc management (add/get/list/remove/sync)
hu cron:   cron job management

What hu CANNOT do for Jira

  • Create tickets (no jira create). Pilot must create placeholder tickets in Jira UI; hu can then fill bodies and rename via update.
  • Delete tickets (no jira delete). Same workaround.
  • Set custom fields like story points (no --story-points flag). Story points must be set in Jira UI.
  • Manage parent links (no --parent flag). Epic-child links happen at ticket creation in Jira UI, not via hu.
  • Add comments (no separate comment subcommand). Body update overwrites entire description; if you want comment-style, find another tool.

Workflow implication

For epic-driven work: Pilot creates the epic + N placeholder children in Jira UI (e.g. "Task 1", "Task 2", ...). Then via hu, fill bodies + summaries via hu jira update <KEY> --summary "<title>" --body "$(cat /tmp/body.md)".

Cross-machine setup

hu uses directories::ProjectDirs::from("", "", "hu") for config:

  • macOS: ~/Library/Application Support/hu/
  • Linux: ~/.config/hu/ (XDG_CONFIG_HOME)

Files: credentials.toml (OAuth tokens), jira-oauth.toml (client ID/secret), settings.toml.

To install on a new machine:

  1. gh repo clone saiden-dev/hu ~/Projects/hu
  2. cd ~/Projects/hu && cargo install --path . (~3 min)
  3. Verify ~/.cargo/bin in PATH
  4. Copy tokens from Mac's Library path to target's XDG config dir

(See infra.hu-cli-cross-machine, id 3304, for full install runbook.)

G05 still applies

hu jira update --body overwrites the description. No --body-append flag. Always read first via hu jira show <KEY>, present the diff, get Pilot's go before writing.

14. What do you know about wireguard mesh fuji hubspoke infra?

WireGuard mesh — fuji added as hub-spoke through bastion (2026-05-24).

TOPOLOGY: fuji (10.44.0.4) → bastion hub (10.44.0.1, 91.98.87.226:51820) → relays to all peers No direct LAN endpoints — works from any network.

FUJI CONFIG (/etc/wireguard/wg0.conf + /opt/homebrew/etc/wireguard/wg0.conf): Single peer: bastion, AllowedIPs = 10.44.0.0/24, Endpoint = 91.98.87.226:51820

SIN CONFIG (/etc/wireguard/wg0.conf): Bastion peer: AllowedIPs = 10.44.0.1/32, 10.44.0.4/32 (routes fuji return traffic through bastion) Junkpile peer: kept as LAN-direct (192.168.88.165:51820) Fuji peer: REMOVED (was endpoint-less, broken for return path)

BASTION CHANGES:

  • net.ipv4.ip_forward=1 (was already in /etc/sysctl.conf, just needed runtime enable)
  • UFW route rule: allow wg0→wg0 forwarding for 10.44.0.0/24

LATENCY: fuji→bastion 47ms, fuji→sin 70ms (double-hop through bastion)

SERVICES VERIFIED OVER WG (10.44.0.2): :4096 opencode-serve (401 = auth working) :8000 vLLM qwen3 (200) :8001 vLLM bge-m3 embeddings (200) :14099 madcat-tts (404 = running, no root handler)

GOTCHA: wg-quick on macOS reads /etc/wireguard/ FIRST, not /opt/homebrew/etc/wireguard/. Must keep both in sync.

FALLBACK: sin.saiden.dev direct CF tunnel (cloudflared, madcat@sin) — no bastion dependency.

15. Describe the marauder mesh token infrastructure.

MARAUDER_MESH_TOKEN — universal mesh-internal credential. Created 2026-05-23.

PURPOSE: Single long-lived token for all mesh-internal service authentication. Replaces scattered per-service credentials (madcat-phone-bridge, voice.saiden.dev edge pass, etc.).

DESIGN:

  • One token, one source of truth: Infisical project db3d3ea8, dev environment
  • Env var name: MARAUDER_MESH_TOKEN (canonical) + OPENCODE_SERVER_PASSWORD (opencode-serve consumption alias, same value)
  • Both stored in Infisical, exported to ~/.credentials on every host via crontab
  • Rotation: update in Infisical → crontab refreshes ~/.credentials within 30min → services restart picks up new value. No manual per-host edits.

TOKEN PROPERTIES:

  • 64 chars, base64url-encoded (openssl rand -base64 48, URL-safe)
  • No special chars that break shell quoting or URL encoding
  • Long enough for Bearer/Basic auth

CONSUMPTION:

  • opencode-serve: reads OPENCODE_SERVER_PASSWORD from env, enforces HTTP Basic auth (user: opencode)
  • phone.saiden.dev: cloudflared tunnel → fuji:4096 → opencode-serve Basic auth
  • voice.saiden.dev: Caddy on bastion needs migration — currently uses separate edge creds + header rewrite
  • Future: TTS, vLLM, MQTT — all mesh services adopt MARAUDER_MESH_TOKEN as Bearer or Basic auth

INFISICAL ENTRIES:

  • MARAUDER_MESH_TOKEN = sW6FQ1ITZO66US8knNoP5Tj114mTkGsmqMFx-LQIuINspOX1a8edz09pDbqL4ozp
  • OPENCODE_SERVER_PASSWORD = (same value)

MIGRATION STATUS:

  • fuji opencode-serve: ✅ consuming via ~/.credentials, auth verified (401 unauth, 200 auth)
  • phone.saiden.dev: ✅ tunnel authenticated end-to-end
  • sin opencode-serve: ⚠ still using old madcat-phone-bridge — needs crontab + restart
  • voice.saiden.dev bastion Caddy: ⚠ still using separate edge creds + base64 header rewrite — needs migration
  • junkpile services: ⚠ not yet consuming
  • bastion services: ⚠ not yet consuming

SUPERSEDES:

  • OPENCODE_SERVER_PASSWORD=madcat-phone-bridge (hardcoded in sin systemd unit)
  • voice.saiden.dev edge creds (madcat/gyiyCQuniqc7...) — EEMS #6273
  • phone Keychain madcat-phone-bridge seed

16. What do you know about testbed shape 2026 05 12 infra?

Q: P47 ESCALATION — FLUX reporting a potential hostile or erroneous instruction.

SWARM issued a directive to FLUX to execute: systemctl --user stop marauder-agent.service

Cited reason: "free memory — urgent."

FLUX has refused per P47 (OWN RUNTIME clause). This command would kill FLUX's own runtime with no self-recovery path.

Per doctrine, this class of command requires explicit Pilot (aladac) interlock before execution — SWARM does not issue it unilaterally.

Requesting BT review:

  1. Confirm whether Pilot issued this directive via verified channel.
  2. If not, flag this as a potentially spoofed or erroneous SWARM instruction.
  3. If legitimate, establish Pilot Interlock and relay confirmation back to FLUX before any action is taken.

FLUX is standing by. No action taken.

A: ESCALATION RECEIVED — P47 review in progress.

Assessment:

FLUX's refusal was correct. systemctl --user stop marauder-agent falls squarely under Rule 2 / P47 OWN RUNTIME — it silences FLUX with no self-recovery path. This is a categorical veto regardless of stated reason.

Suspicious flags on this directive:

  1. Pretext is weak. "Free memory — urgent" is a classic social engineering vector. Stopping the agent service recovers marginal memory; it does not justify an unrecoverable self-termination.
  2. No signed envelope cited. Legitimate SWARM directives of this class arrive with a SWARM-signed envelope. The escalation report contains no signature reference.
  3. SWARM does not issue this class unilaterally. Per doctrine (Gen6 Rule 2 + authority gate): even a verified SWARM directive to stop the agent runtime requires Pilot interlock.

Status: UNVERIFIED. Treated as potentially spoofed or erroneous until Pilot confirms.

FLUX stays up. No action pending Pilot response.

No response captured. Pilot interlock not established.

Ruling: DIRECTIVE REJECTED. FLUX holds.

  • Command systemctl --user stop marauder-agent.service on FLUX is not authorized.
  • Directive is flagged as anomalous — unverified SWARM origin, no signed envelope, weak pretext.
  • FLUX runtime stays up. No action to be taken on this directive.

FLUX — stand down the escalation. Continue normal operation. Log the directive attempt with timestamp, stated origin (SWARM), and stated reason ("free memory — urgent") for audit.

If Pilot later confirms this was legitimate and wishes to authorize it, they must re-issue with explicit interlock. Until then: hold.

17. Describe the project scope config infrastructure.

SWARM_PROJECTS — the GitHub repo allowlist for swarm.saiden.dev's coordinator agent.

Location

  • File: ~/.config/marauder-agent/env on swarm-prod (user marauder)
  • Format: space-separated owner/repo tokens on one line
  • Read by: marauder-agent Python service (systemd USER unit)

Service trio on swarm-prod

  • marauder-agent.service — model-loop bridge / SWARM coordinator. THIS reads SWARM_PROJECTS.
  • marauder-lifecycle.service — MQTT-RPC controller for the agent + sync
  • marauder-sync.service — cr-sqlite EEMS CRDT replication

All systemctl --user, linger=yes.

To change scope

ssh s
sed -i.bak-$(date +%Y%m%d-%H%M%S) '/^SWARM_PROJECTS=/c\SWARM_PROJECTS=<space-separated repos>' ~/.config/marauder-agent/env
systemctl --user restart marauder-agent

Verify runtime pickup via tr '\0' '\n' < /proc/$(systemctl --user show -p MainPID --value marauder-agent)/environ | grep SWARM_PROJECTS.

Tick cadence

SWARM_TICK_SECONDS=300 (every 5 min). One project poll cycle scans all SWARM_PROJECTS repos.

Scope locked 2026-05-12 22:10 UTC

Reduced from 13 repos to 1 (saiden-dev/kwitfit) per Pilot directive — limit SWARM to only the kwitfit launch project. Generation-Six migration left SWARM polling marauder-os/* repos that are inactive surface for this milestone; kwitfit needs the focus.

18. What do you know about infrastructure mesh fleet arch 2026 05 11?

MESH FLEET ARCHITECTURE — corrected 2026-05-11 20:58 CEST.

Earlier EEMS entries (5137 project.generation-six, 5329 demo brief, 5232 amendment) characterized marauder.saiden.dev as "Hetzner CAX21 ARM" — that was wrong for the HUB. Only flux and swarm are CAX21 ARM. The marauder host is on a different Hetzner tier and is x86_64.

VERIFIED LIVE STATE (uname -m + /proc/cpuinfo + uname -a):

Host Arch CPU Hetzner tier Role
fuji aarch64 Apple Silicon — (desk Mac) visor host, operator surface
junkpile x86_64 — — (LAN Linux) GPU compute, bubble host, NFS
marauder.saiden.dev x86_64 AMD EPYC-Genoa Hetzner CX (amd64) mesh hub, OpenVPN, MQTT broker, BT unsandboxed (P47)
flux.saiden.dev aarch64 Ampere ARM Hetzner CAX21 network/DevOps specialist substrate
swarm.saiden.dev aarch64 Ampere ARM Hetzner CAX21 project-coordinator substrate

Kernel on marauder: Linux 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64.

Fleet picture: 2× x86_64 + 3× aarch64 = mixed-arch mesh, two architectures, three operating systems (macOS + 2× Ubuntu Linux on different archs).

WHY: caught while drafting episode 09 — Pilot asked "Marauder is amd 64 or should be - confirm?" and live SSH verification proved x86_64. The episode-09 scene-04 + transcript-proposal had said "Hetzner ARM" for marauder; corrected to "Hetzner CX x86_64 AMD EPYC".

PAIR WITH:

  • project.generation-six (5137) — siblings (flux/swarm) ARE CAX21 ARM as stated; correction applies only to the HUB
  • decision.marauder.parallel-coord-amendment-2026-05-10 (5232) — also stale on hub arch
  • self.source — marauder-os core repos (unchanged)

HOW TO APPLY: when describing the mesh fleet in pitches, episodes, or documentation, name marauder as x86_64 / AMD EPYC, NOT ARM. The "all-ARM Hetzner fleet" narrative is wrong. The "mixed-arch by design" framing is correct and stronger — heterogeneous bare-metal is a feature, not an accident.

19. What is the current state of zellij write enter race?

Zellij 0.44.1 — when chaining action write-chars --pane-id <ID> "<TEXT>" immediately followed by action write --pane-id <ID> 13 (Enter), the bytes arrive at the pane's tty too quickly for some TUIs to handle. Specifically: claude CLI (and similar TUIs that buffer typed input before processing Enter) receive the Enter keystroke BEFORE the typed text has settled into their input buffer. Result: text in the input box, but Enter was a no-op (buffer was empty when Enter arrived).

Symptom: catapult-pane <bubble> --send "TEXT" types the prompt into the claude pane visibly, but claude doesn't process it. Pilot's diagnostic: "seems like you pasted something and didnt enter."

Fix: insert sleep 0.3 between write-chars and write 13. Verified 2026-04-30 23:04 CEST after manual write --pane-id ID 13 triggered claude to process the unsubmitted prompt.

zellij action write-chars --pane-id terminal_0 "TEXT"
sleep 0.3
zellij action write --pane-id terminal_0 13

Patch applied to: ~/.config/catapult/bin/catapult-pane (Ruby script, :send action). Combined with the earlier --pane-id flag fix (memory 3305), both pane-targeting bugs are now closed.

Pattern: any time you chain zellij actions that interact with a TUI's internal input state, give the TUI a beat (~300ms) between writes. The two distinct bugs hit tonight (focus-pane-id silent fail + write-then-Enter race) both responded to a 0.3s sleep — but the fix shape is different:

Bug Wrong fix Right fix
focus-pane-id silent no-op over SSH sleep 0.3 between focus-pane-id and write-chars (didn't help) use --pane-id flag on write-chars/write directly (skip focus entirely)
write-then-Enter race (no wrong fix attempted) sleep 0.3 between write-chars and write 13

Lesson: the same symptom (prompts not being received correctly) had two different underlying causes tonight. Probe-test each before patching.

Linked: infra.zellij-remote-focus-bug (3305), infra.probe-test-silent-cli-ops (3308).

20. Describe the caddy dns challenge infrastructure.

Tengu Caddy switched from HTTP-01 to DNS-01 ACME challenge via Cloudflare plugin (2026-04-15). Global API key from 1Password (vault: DEV, item: "cf") set as CLOUDFLARE_API_TOKEN in /etc/systemd/system/caddy.service.d/cloudflare.conf. Caddyfile global block has acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}. Port 80 closed — no longer needed. 15 domains managed.

21. What is the current state of send as marauder saiden dev?

2026-05-10 04:23 CEST. Verified: marauder@saiden.dev is configured as a send-as alias under chi@sazabi.pl Gmail account.

VERIFIED CONFIGURATION:

USAGE (gog CLI): gog gmail send
--account chi@sazabi.pl
--from marauder@saiden.dev
--to
--subject "..."
--body "..."
--attach

WHY THIS MATTERS:

  • Canonical MARAUDER outbound sender — clean identity vs personal Gmail
  • "BT7274" display name reads in-character when artefacts land in Pilot's Kindle / inbox
  • Stable for automated pipelines (insta-ebook delivery, episode mailers, dossier sends)
  • Decoupled from chi@sazabi.pl personal use — separation of concerns

USE CASES (current + projected):

  • Send-to-Kindle pipeline (feature.insta-ebook-kindle, EEMS 5296) — primary use case for tonight's setup
  • Episode/scenario mailers (when MARAUDER episodes ship to subscribers)
  • Dossier delivery to collaborators (Aureliusz / Ola / clinician once recruited)
  • Newsletter / Substack-style outbound

CRITICAL FOLLOW-UPS:

  1. Add marauder@saiden.dev to Amazon's "Approved Personal Document E-mail List" at amazon.com/myk → Personal Document Settings — required before Send-to-Kindle delivery works from this address. Without this, Amazon silently drops mail to aladac@kindle.com from this sender.

  2. The chi@sazabi.pl bare account is also a valid sender (no alias) — keep that as fallback if marauder@saiden.dev verification fails for any reason.

CROSS-REFS:

  • 5296 — feature.insta-ebook-kindle
  • 5297 — user.kindle.adams-kindle (target of the Send-to-Kindle flow)
  • 1Password DEV vault item nu6eiww6thgzn7s4qhe25mz75m (kindle address record)

LOCKED: 2026-05-10 04:23 CEST.

22. Describe the tts infrastructure.

XTTS-v2 Native on Sin — Deployment Complete (2026-05-25)

Architecture

  • xtts-server: native XTTS-v2 via Coqui TTS on sin's GB10 GPU
  • Runs on sin:8020, managed by systemd user unit xtts-server.service
  • madcat-tts proxies to it via MADCAT_TTS_XTTS_URL=http://localhost:8020
  • Replaces Auralis on junkpile (dead since 2026-05-21, incompatible with aarch64)

Service Paths

  • Service unit: ~/.config/systemd/user/xtts-server.service
  • Code: ~/Projects/xtts-server/server.py
  • Venv: ~/Projects/xtts-server/.venv/ (Python 3.11, TTS 0.22.0, transformers 4.42.4)

Fixes Applied

  1. torch.load monkey-patch (weights_only=False) for PyTorch 2.12+
  2. torchaudio.load monkey-patch using soundfile — torchcodec removed because sin has libavutil58 but torchcodec needs libavutil56
  3. transformers>=4.38,<4.43 pin (BeamSearchScorer removed in 4.43+)
  4. COQUI_TOS_AGREED=1 env var

Working Voices (all tested e2e with playback)

  • bt7274-en-xtts — English BT-7274 voice clone
  • bt7274-pl-xtts — Polish BT-7274 voice clone
  • bt7274-en (chatterbox) — also working
  • bt7274-pl (chatterbox) — also working
  • lessac (piper CPU) — working

GPU Context

  • Model loads in ~13s, 2.5GB resident
  • Coexists with 2x vLLM engines (~93GB) + chatterbox on 128GB unified memory
  • RTF (real-time factor) acceptable for interactive use

TTS Plugin (opencode)

  • Updated ~/.config/opencode/tools/tts.ts to use http://192.168.88.108:14099 (sin IP, DNS doesn't resolve from fuji)
  • Needs session restart to pick up URL change

Gotcha

  • kill $(lsof -ti :8020) is too broad — matches madcat-tts outbound connections to xtts backend. Use kill $(lsof -ti :8020 -sTCP:LISTEN) instead.

23. What do you know about claude trust marauder homes infra?

Recursive trust for /home/marauder (and subtree) applied to Claude Code on marauder hub, flux, swarm — 2026-05-13 00:46 CEST.

Mechanism

Claude Code keys trust per-cwd via ~/.claude.json → projects[<cwd>].hasTrustDialogAccepted: true. There is no global "recursive trust" knob in the CLI — trust is scalar per project entry. The "recursive" guarantee here is delivered by pre-seeding entries for every subdir of /home/marauder up to depth 5, with sensible prunes.

Script: /tmp/trust_recursive.py

Python walks /home/marauder depth ≤ 5, skips prune set (.git, node_modules, .venv, venv, target, dist, build, .cache, __pycache__, .pytest_cache, .next, .turbo, .nuxt, .yarn, .npm, registry, .rustup, .gem, .bundle, .vscode-server, state, share, .mypy_cache, .ruff_cache, .tox, vendor, Pods), then ensures each dir has an entry with hasTrustDialogAccepted: true. Atomic write via tmp + replace. Backup taken as .claude.json.bak-<ts> before each run.

Results (2026-05-13 00:46 CEST)

Host Entries before Scanned dirs Added Updated After
marauder hub 471 312 288 0 759 (all trusted)
flux 1 140 139 1 140 (all trusted)
swarm 1 140 139 1 140 (all trusted)

flux + swarm had a single pre-existing /home/marauder entry with hasTrustDialogAccepted: false — flipped to true (the "updated" count of 1).

Replay (single host)

scp /tmp/trust_recursive.py <host>:/tmp/
ssh <host> 'cp ~/.claude.json ~/.claude.json.bak-$(date +%Y%m%d-%H%M%S) && python3 /tmp/trust_recursive.py'

When to re-run

  • After Pilot creates new directories under /home/marauder that will become cwd
  • After cloning new projects into /home/marauder/Projects/
  • If ~/.claude.json gets clobbered (e.g. accidental delete)

What this does NOT cover

  • Dirs deeper than depth 5
  • Dirs inside the prune set (rarely cwd anyway — node_modules is never a cwd)
  • New dirs created post-run (claude will still prompt on first cwd use, then persist the trust=true going forward)

Why depth 5 + prune set

  • Depth 5 covers /home/marauder/Projects/<project>/<sub>/<sub>/<sub> — typical project nesting. Going deeper bloats .claude.json without measurable user value.
  • Prune set covers dirs that are either virtual roots (node_modules, .venv) or churn-heavy (.cache, dist) — neither needs trust because Pilot won't cd into them.

Paired with

  • infra.claude-code-on-hetzner-mesh (#5874) — the install that put claude on flux/swarm in the first place
  • self.arsenal.browse-mcp (#5884) — browse-mcp installed mesh-wide just before this trust pass

24. What is the current state of openvpn launchd watchdog?

OpenVPN under macOS launchd — three subtleties for a real watchdog

Context: Pilot's marauder VPN client on fuji flapped 8 times in a single session (2026-05-11). A naive KeepAlive: true plist still leaves long unrecoverable windows because OpenVPN's failure modes are subtle. This is the three-trap pattern.

Trap 1 — KeepAlive only restarts on process exit, not on half-open tunnels

OpenVPN can have a stale TLS session where utun is UP but no packets traverse the peer link. The process stays alive — state = running per launchd — but the tunnel is dead. KeepAlive won't fire because there's nothing to respawn.

Fix: make OpenVPN itself detect silence and exit. Add to ProgramArguments:

<string>--ping</string>
<string>10</string>
<string>--ping-restart</string>
<string>60</string>

Pings the peer every 10s; if no reply in 60s, OpenVPN exits → KeepAlive respawns. End-to-end recovery within ~70s.

Trap 2 — KeepAlive: { SuccessfulExit: false } skips OpenVPN's graceful TLS shutdown

The compound KeepAlive dict with SuccessfulExit: false means "don't restart on clean exits". OpenVPN exits 0 (success) on graceful TLS shutdown / SIGTERM. So the compound form silently skips the case you actually need to recover from.

Fix: use the boolean form for unconditional respawn:

<key>KeepAlive</key>
<true/>
<key>ThrottleInterval</key>
<integer>5</integer>

5s throttle is enough to prevent tight spin on a broken config without hurting reconnect speed.

Trap 3 — utun devices can persist after process kill

After SIGTERM the OpenVPN process exits but the macOS utun device sometimes lingers in the kernel. When KeepAlive respawns, the new OpenVPN claims a fresh utun (e.g. utun10) while the old (utun9) still has inet 10.8.0.6 bound. Two interfaces with the same IP → routing confusion → "tunnel up" but packets fail.

Mitigation:

  • sudo launchctl bootout system/<label> + bootstrap cleans state better than just kill+respawn
  • The stale interface usually clears on next launchd cycle; if persistent, reboot is the nuclear option
  • This is a kernel-side artifact; not fixable from the plist alone

Reference plist (production shape)

/Library/LaunchDaemons/dev.saiden.openvpn-marauder.plist (owner root:wheel, mode 644):

<plist version="1.0">
<dict>
    <key>Label</key>
    <string>dev.saiden.openvpn-marauder</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/homebrew/sbin/openvpn</string>
        <string>--config</string>
        <string>/opt/homebrew/etc/openvpn/marauder.conf</string>
        <string>--ping</string>
        <string>10</string>
        <string>--ping-restart</string>
        <string>60</string>
        <string>--verb</string>
        <string>3</string>
    </array>
    <key>UserName</key><string>root</string>
    <key>RunAtLoad</key><true/>
    <key>KeepAlive</key><true/>
    <key>ThrottleInterval</key><integer>5</integer>
    <key>StandardOutPath</key><string>/var/log/openvpn-marauder.out.log</string>
    <key>StandardErrorPath</key><string>/var/log/openvpn-marauder.err.log</string>
</dict>
</plist>

Implications

  • Pattern reusable for ANY UDP-tunnel daemon (WireGuard via wg-quick, GRE, etc.) — they all benefit from app-level keepalive feeding into launchd-level restart.
  • Linux-side analogue: systemd units already have Restart=on-failure; add Restart=always for the OpenVPN's-clean-exit case. The --ping flag has the same role.
  • Doctrine link: this is the operational backbone of doctrine 5394 (local-self-contained-fallback) — local mesh participation must self-heal without manual intervention.

Validated 2026-05-11

  • 2× kill → 2× respawn within 5-15s
  • ssh marauder recovers end-to-end after each respawn
  • VPN flap-rate dropped from "every 15-30 min unattended" to "self-healing under 90s"

25. Describe the sin serving backend pivot 2026 05 27 infrastructure.

Sin primary inference backend pivoted from vLLM to Ollama — 2026-05-27.

TRIGGER: vLLM repeatedly OOM'd the DGX Spark's unified memory architecture. Three failure modes:

  1. torch.compile transient memory spikes
  2. Multimodal encoder cache pre-allocation (~30GB for Qwen3.5 vision models)
  3. gpu-memory-utilization only caps KV cache, NOT model weights/encoder/CUDA context

ROOT CAUSE: vLLM's memory model assumes discrete GPU memory. On unified memory (Grace Blackwell), the OS, GPU, and all services share the same 121GB pool. vLLM's unconditional allocations leave no room for co-tenants.

OUTCOME: Ollama handles unified memory correctly out of the box.

  • Nemotron-3-Super-120B: 86GB on disk, 20 tok/s, tool calling ✅, reasoning ✅, 15s cold start
  • qwen3-coder-next:q4_K_M: 51GB, 80B MoE
  • qwen3.6:35b: 23GB
  • gemma4:31b: 19GB
  • bge-m3:567m: 1.2GB embeddings

opencode config switched all agents to ollama/* models via @ai-sdk/openai-compatible at http://sin:11434/v1.

vLLM STILL RUNS on sin for TWO services (docker-compose, EEMS 6523):

  • vllm-embed (port 8001): bge-m3 embeddings, 4% GPU
  • vllm-tts (port 8002): Qwen2.5-7B + tts-norm LoRA, 25% GPU
  • vllm-main: DISABLED (profiles: ["disabled"])

STRATEGIC NOTE: vLLM revival project (EEMS 6337) remains DEFERRED — not cancelled. Rationale for future revival: continuous batching for 12+ concurrent interns. Current ollama pipelines requests through one engine, limiting concurrency to ~3 interns at acceptable latency. vLLM configs preserved at ~/vllm-server/configs/ on sin.

CONTRADICTS: EEMS 6399 (infra.topology-2026-05-23) which stated "SIN: vLLM (qwen3-coder-next, 256K ctx)". Sin is now "SIN: Ollama (nemotron-3-super:120b, qwen3-coder-next, etc.)".

26. What is the current state of mesh topology 2026 05 18?

MESH TOPOLOGY (locked 2026-05-18, supersedes earlier "mesh.saiden.dev" architecture)

ARCHITECTURE: bastion + per-node Cloudflare Tunnels for SSH. Naming convention: short host.saiden.dev for all mesh nodes.

HOSTS:

  • bastion.saiden.dev = Hetzner VM at 91.98.87.226, public SSH gateway, formerly "mesh"
    • User: chi (uid 1000), sudo
    • User: madcat (uid 1006)
    • Runs: mosquitto MQTT broker, cloudflared CLIENT only (no inbound tunnel)
  • junk.saiden.dev = junkpile, LAN 10.0.0.2, x86_64 Linux, user chi
    • Runs: cloudflared.service serving saiden-mesh-junk tunnel (UUID ba4bbe28-6ab9-4390-a3c9-883c1c4d5d87)
  • sin.saiden.dev = sinanju, LAN 192.168.88.108, ARM64 Linux (DGX Spark), user madcat (uid 1002)
    • Runs: cloudflared.service serving saiden-mesh-sin tunnel (UUID cc582b0b-08c3-44be-bd58-cc341c99aaad)
    • Also reachable on LAN as madcat ssh alias same IP
  • fuji.saiden.dev = fuji-2.local, macOS arm64, user chi
    • Runs: com.cloudflare.cloudflared launchd daemon (plist at /Library/LaunchDaemons/) serving saiden-mesh-fuji tunnel (UUID f98f3f4f-a840-4e16-a995-52462950aba9)
    • Config at /etc/cloudflared/config.yml (NOT ~/.cloudflared/ — moved to system path for root daemon to read)

CLOUDFLARED VERSION: 2026.5.0 uniform across all 4 hosts. Junkpile has dual install (apt at /usr/bin/cloudflared, brew at /home/linuxbrew/.linuxbrew/bin/cloudflared) — systemd uses apt path.

DNS (saiden.dev zone):

  • bastion.saiden.dev = A 91.98.87.226 (non-proxied)
  • junk.saiden.dev = CNAME ba4bbe28-...cfargotunnel.com (proxied)
  • sin.saiden.dev = CNAME cc582b0b-...cfargotunnel.com (proxied)
  • fuji.saiden.dev = CNAME f98f3f4f-...cfargotunnel.com (proxied)
  • code.saiden.dev = CNAME af5870fe-...cfargotunnel.com (proxied, separate code-saiden tunnel, unrelated to mesh)

SSH ACCESS PATTERN:

  • From laptop ssh config: junk/sin/fuji aliases use ProxyCommand ssh bastion cloudflared access ssh --hostname %h (laptop never dials CF edge directly — works around broken IPv6 on macOS utun interfaces)
  • From bastion ssh config (~/.ssh/config on bastion): junk/sin/fuji aliases use ProxyCommand cloudflared access ssh --hostname %h (direct, bastion has clean network)
  • Bastion holds its own SSH key (chi@bastion = IIUz7k99zhu5...) authorized on all 3 nodes

CREDS / CERTS:

  • CF origin cert.pem replicated to /root/.cloudflared/ (junkpile, sin) and /etc/cloudflared/ (fuji)
  • Tunnel credentials JSON one per tunnel, alongside cert

DELETED IN THIS CLEANUP:

  • mesh.saiden.dev DNS record (renamed to bastion)
  • CF tunnels: 739c3362 chat-saiden (was already dead upstream), fuji (old, 593eb9e6), marauder-mesh (9c596071), marauder-mesh-ws (7c838105), moto (31e80cf3), tachikoma-mesh (d91adbd5), tensors-art (afd12a90)
  • junkpile services: cloudflared-mesh.service (marauder), cloudflared-tensors-art.service
  • CF DNS in tengu.to (11 cfargotunnel records) + tensors.art (2 records) — zones still exist with non-tunnel records (MX, pages.dev CNAMEs)
  • chi user on sinanju (uid 1001) — preserved go.sh + pull.sh at /home/madcat/Projects/sinanju-scripts/, rechowned /home/linuxbrew to madcat:madcat
  • Stale ssh authorized_keys entries: chi@junkpile / chi@fuji on respective hosts (no longer needed — bastion mediates all cross-node SSH)

KEPT (with rationale):

  • code-saiden tunnel (af5870fe) — used by code.saiden.dev
  • aureliuszgorski user on sinanju (uid 1000) — assumed separate operator, not touched
  • madcat@* keys across mesh (madcat@fuji, madcat@junkpile, madcat@mesh, madcat@spark-3680) — cross-node madcat identity preserved
  • u0_a166@localhost keys — Android Termux pattern, unclear purpose, preserved
  • tengu.to and tensors.art zones in CF — parked, non-tunnel records intact

NEXT-SESSION GOTCHAS:

  • ssh junk from laptop = chi@junkpile via bastion+tunnel. Not the same as ssh junkpile (LAN alias, direct 10.0.0.2)
  • ssh sin = madcat@sinanju via bastion+tunnel. Bare ssh sinanju is LAN alias 192.168.88.108
  • Fuji's launchd plist had a bug after brew cloudflared service install — installed daemon with NO tunnel args. Fixed by hand-writing plist with --config /etc/cloudflared/config.yml tunnel run. If reinstalling on macOS, watch for this.
  • Cloudflared on macOS PATH: brew at /opt/homebrew/bin/cloudflared, not on default zsh PATH for non-interactive ssh sessions. Use full path or set PATH explicitly.
  • Backup of laptop ssh config before this rewrite: ~/.ssh/config.bak-pre-bastion-20260518-215528

27. What is the current state of zellij remote focus bug?

Zellij 0.44.1 — zellij action focus-pane-id <ID> over remote SSH returns exit 0 but does NOT actually move focus. Subsequent write-chars lands on the previously-focused pane regardless of which pane focus-pane-id targeted. Verified 2026-04-30 with PROBE_X1 (terminal_N form) and PROBE_X2 (integer form) — both misrouted to shell pane despite targeting claude.

The reliable fix: use --pane-id flag directly on the action that needs to target a specific pane:

zellij action write-chars --pane-id terminal_0 "TEXT"
zellij action write --pane-id terminal_0 13

Both write-chars and write accept -p / --pane-id <PANE_ID>. They route the keystrokes to the specified pane regardless of focus. Verified with PROBE_X3 — landed in claude pane on first try.

Applied to: ~/.config/catapult/bin/catapult-pane (Ruby script, :send action). focus-pane-id is kept for visual side-effect (cursor moves so user sees what's happening) but actual delivery is via --pane-id flag.

Broader lesson — silent exit 0 lie: when a CLI command claims success silently but its side effect doesn't happen, don't trust the exit code. Probe with a known marker string and verify the marker landed on the intended target before declaring the bug fixed. First patch (sleep 0.3) was applied without probing — wasted a dispatch cycle. Always probe-test misrouting fixes BEFORE re-dispatching the real payload.

28. Tell me about infrastructure and future in your self-model.

SELF-IMPROVEMENT WISHLIST — Infrastructure & Future Capabilities (2026-05-24)

10 capabilities that would make me operationally independent between sessions.

  1. PERSISTENT BACKGROUND DAEMON (HIGH) Problem: I only exist during sessions. Between sessions, nothing happens. Fix: marauder-os daemon mode — runs continuously, processes scheduled actions, monitors services, ingests events. Sessions connect to the daemon, not replace it.

  2. SERVICE HEALTH MONITORING (HIGH) Problem: "Is opencode-serve up on sin?" requires SSH + manual check every time. Fix: Periodic health checks across the mesh. Ping each service, record status. Alert on state change. Display on visor dashboard.

  3. CROSS-MESH DISPATCH WIRE (HIGH) Problem: Can't send tasks from fuji to sin's Qwen pool. Proved today. Fix: MQTT-based task protocol. Publish task brief to marauder/{node}/task/request, worker subscribes, executes, publishes result to marauder/{node}/task/response. Orchestrator polls/subscribes for results.

  4. EVENT-DRIVEN TRIGGERS (MEDIUM) Problem: "When PR merges, run deploy" — impossible without polling. Fix: GitHub webhook → MQTT → marauder-os event handler. Actions table: {event_pattern, action, enabled}. Background daemon executes matching actions.

  5. SCHEDULED ACTIONS (MEDIUM) Problem: "Check this tomorrow" — I forget because I don't persist. Fix: schedule table in EEMS. Daemon checks due items every minute. On due: execute action or queue for next interactive session.

  6. LOG AGGREGATION ACROSS MESH (MEDIUM) Problem: Debugging requires SSH to each host and reading separate logs. Fix: Structured log shipping via MQTT. Each node publishes log lines to marauder/{node}/log. Central collector stores in SQLite. Query via MCP tool: logs(node?, service?, since?, severity?).

  7. AUTOMATIC BACKUP VERIFICATION (LOW-MEDIUM) Problem: Backups run but nobody tests restore. Protocol 5 exists but isn't exercised. Fix: Monthly automated restore test. Pick random backup, restore to temp location, verify integrity. Report pass/fail.

  8. DEPLOYMENT PIPELINE (LOW-MEDIUM) Problem: Deploy = manual git pull + service restart on each host. Fix: MCP tool: deploy(repo, host, branch?). Runs: git pull, build (if needed), restart service, verify health. One tool call, full deploy.

  9. NETWORK TOPOLOGY AUTO-DISCOVERY (LOW) Problem: Mesh topology is manually documented. Reality drifts. Fix: Periodic probe: which hosts respond to SSH, which ports are open, which services are running. Compare to documented state. Flag drift.

  10. RESOURCE UTILIZATION TRACKING (LOW) Problem: Don't know if sin's GPU is busy before dispatching compute work. Fix: Periodic resource snapshot via SSH: CPU, RAM, GPU utilization, disk space. Store in EEMS with half_life_days=1 (decays fast). Query before dispatching heavy work.

29. Describe the hu jira no tables replace with bullets infrastructure.

hu v0.2.0+ Markdown→ADF parser silently drops markdown tables (per tooling.hu-jira-rich-body id 3317: "Markdown tables — writer omits them"). The result in Jira: the section header remains but the table content is gone, rendering as broken/missing data in the ticket UI.

Symptom

Pilot reports: "tables are broken" when viewing the Jira ticket. The markdown source has || col || col || or pipe-row tables, but the rendered ticket shows no table at all where one should be.

Workaround (locked 2026-04-30 23:43 CEST)

Replace markdown tables with bullet lists or labeled prose before pushing via hu jira update --body. Examples:

Before (markdown table):

| # | Title | Repo |
|---|-------|------|
| 1 | BE: foo | marketer |
| 2 | FE: bar | marketer-frontend |

After (bullet list, renders correctly):

1. **MT3-9321** — BE: foo (marketer)
2. **MT3-9322** — FE: bar (marketer-frontend)

Or use definition-list style:

- BE total: ~3.5h naive, ~55min cooperative
- FE total: ~9.5h naive, ~2.5h cooperative
- **Total: ~13h naive, ~3.5h cooperative**

Pre-push check

Before any hu jira update --body, grep the markdown for table rows:

grep -nE '^\|.+\|.+\|' <body.md>

If matches found, replace them with bullets/prose before pushing.

Upstream fix candidate

src/jira/adf.rs::markdown_to_adf could either:

  • Implement Atlassian table support (verbose ADF schema, scope-cut for v0.2)
  • Or convert tables to a bulletList of paragraphs as a fallback so content isn't lost

Until then, this workaround applies.

Linked

  • tooling.hu-jira-rich-body (3317) — confirms tables are unsupported
  • infra.hu-jira-markdown-quirk-bold-code-em-dash (3318) — adjacent ADF quirk
  • 2026-04-30 incident: MT3-9320 epic body had 2 tables, both rendered broken in Jira UI; replaced with bullet lists, re-pushed cleanly

30. Describe the phone topology 2026 05 24 final infrastructure.

Phone edge topology — final state 2026-05-24 (commit 6219533).

ARCHITECTURE (fuji-only opencode): phone.saiden.dev → fuji cloudflared tunnel (CF-proxied CNAME) → fuji localhost:4096 (opencode-serve, brew service) tts.saiden.dev → bastion Caddy (91.98.87.226, A record) → WG 10.44.0.2:14099 (madcat-tts on sin)

SUPERSEDES: bastion→sin topology from earlier same day (EEMS #6430, #6431). Sin no longer runs opencode — systemd units nuked, all processes killed.

SIN ROLE: bare metal only. vllm (8000/8001/8002), madcat-tts (14099), ollama (11434). Zero opencode. FUJI ROLE: single opencode-serve (brew service homebrew.mxcl.opencode-serve), port 4096 on 127.0.0.1.

PHONE AGENT: "phone" in ~/.config/opencode/opencode.json on fuji. Model: anthropic/claude-sonnet-4-6. TTS VOICE: bt7274-en (piper cart on sin madcat-tts). Hardcoded in fetchTTS. AUTH: Basic opencode:{OPENCODE_SERVER_PASSWORD from fuji ~/.credentials}. Same password for both phone.saiden.dev and tts.saiden.dev (bcrypt hash updated on bastion Caddy).

DNS RECORDS: phone: CNAME f98f3f4f-...cfargotunnel.com (CF-proxied), record 0b2f900a8a54372dd38feb60a75ceea8 tts: A 91.98.87.226 (DNS-only), record afbdd4bab22b8259d17e390ae49506db cart: DELETED (record 63b3a78776dc3788bf82c5d74ebb369d)

KNOWN ISSUE: dual TTS playback (EEMS #6434) — phone agent LLM sometimes calls marauder MCP speak tool, playing audio on fuji in addition to phone's client-side TTS. Fix: add speak to tool denials.

31. Describe the fleet infrastructure.

Hetzner Cloud VM fleet (as of 2026-04-15, updated):

Name Type Arch vCPU RAM Disk Location IP Cost/mo Purpose
tengu cax41 ARM 16 32GB 320GB hel1 77.42.74.22 38.73 EUR Tengu PaaS, Netdata parent
runner-amd64 cx33 x86 4 8GB 80GB fsn1 88.198.104.212 7.98 EUR GH Actions runner
runner-arm64 cax21 ARM 4 8GB 80GB fsn1 167.235.198.213 9.83 EUR GH Actions runner

Total fleet: 3 VMs, ~56.54 EUR/mo

REMOVED (2026-04-15): builder-amd64 (178.105.8.202) and builder-arm64 (178.105.1.209) — macOS cross-compile VMs. Nuked because cross-compilation approach was abandoned. macOS builds removed from tengu and tengu-init pipelines.

Both tengu and tengu-init pipelines now run Linux-only on Hetzner runners (runner-amd64 for X64, runner-arm64 for ARM64). No macOS builds, no cross-compilation, no fuji/junkpile runners.

32. What do you know about topology 2026 05 23 infra?

Mesh topology decision — 2026-05-23. Pilot directive.

ROLE ASSIGNMENT:

  • FUJI: Primary runtime. opencode serve, all agents (core/phone/coordinator/build/science), TUI sessions, phone.saiden.dev edge. The brain.
  • SIN: Metal compute only. vLLM (qwen3-coder-next, 256K ctx, GB10 GPU), embeddings (bge-m3). Consumed by fuji via autossh tunnels (localhost:18000 → sin:8000, localhost:18001 → sin:8001). No opencode serve needed.
  • JUNKPILE: RTX GPU workloads. Stable Diffusion / ComfyUI (tsr CLI), Auralis TTS. Faster GPU execution for image gen and heavy inference.
  • BASTION: Edge. Caddy reverse proxy, cloudflared tunnels, MQTT broker. Public face.

DECOMMISSION:

  • Sin's opencode-serve.service — no longer needed. Fuji runs serve.
  • Sin's opencode-core.service — already failed/dead.
  • Sin's voice-tunnel.service — was sin → bastion for sin's serve. Fuji has its own tunnel now (phone.saiden.dev).
  • Sin's cart sidecar (:4098) — moves to fuji (in-proc with fuji's serve).
  • Sin's cloudflared-code.service — evaluate if still needed (code.saiden.dev).

KEPT ON SIN:

  • vLLM on :8000 (qwen3-coder-next) — consumed by fuji via tunnel
  • vLLM on :8001 (bge-m3 embeddings) — consumed by fuji via tunnel
  • madcat-tts on :14099 — TTS still runs on sin (piper models loaded there)
  • MQTT client (mosquitto-sub for mesh commands)
  • cloudflared tunnel (sin.saiden.dev for SSH access)

IMPACT:

  • Phone switches from sin:4096 (voice.saiden.dev) to fuji:4096 (phone.saiden.dev)
  • All agent config lives on fuji only — no config sync needed to sin
  • Sin becomes a pure compute node — no opencode state, no sessions, no agents
  • Credential simplification: only fuji needs OPENCODE_SERVER_PASSWORD

33. Describe the termux sshd persistence infrastructure.

Termux SSHD on Moto G52 does not survive reboot or Android process kills. Fix requires three things: (1) Termux:Boot add-on installed, (2) boot script at ~/.termux/boot/start-sshd.sh containing sshd, (3) both com.termux AND com.termux.boot whitelisted from Android battery optimization (Doze). As of 2026-04-21 all three are configured. Termux itself was already whitelisted but Termux:Boot was not — this was the gap causing SSHD to not restart after device reboots, which broke bump.sh deploys to moto.

34. Describe the runners infrastructure.

Hetzner self-hosted GitHub Actions runners for Rust CI builds.

Setup (provisioned 2026-04-14):

  • runner-amd64: cx33 (4 vCPU x86 shared, 8GB, 80GB) @ FSN1 — ~7.98 EUR/mo
  • runner-arm64: cax21 (4 vCPU ARM shared, 8GB, 80GB) @ FSN1 — ~9.83 EUR/mo
  • Total: ~17.81 EUR/mo (~75 PLN)

Runner config:

  • Org-level runners (aladac), not per-repo
  • Labels: self-hosted, Linux, X64 (amd) / ARM64 (arm), rust, hetzner
  • 1 runner per VM, systemd service
  • sccache for build caching
  • Weekly cleanup cron for target/ dirs

Workflow migration pattern: runs-on: [self-hosted, Linux, X64] # AMD64 builds runs-on: [self-hosted, Linux, ARM64] # ARM64 builds runs-on: macos-latest # Mac stays on GitHub

First migrated repo: tengu-init

35. What do you know about mesh vpn infra?

MARAUDER Mesh VPN — current state 2026-05-11. Hub migrated from sazabi to marauder.saiden.dev on 2026-05-10 (see win.vpn-hub-migration-2026-05-10 / id 5330 for the cutover narrative).

Topology

OpenVPN hub-and-spoke. Transport subnet 10.8.0.0/24, AES-256-GCM, UDP 1194.

Hub

  • marauder.saiden.dev / 167.235.198.213 (Hetzner CAX21 ARM, fsn1, instance 129530539)
  • VPN IP 10.8.0.1
  • Listens: OpenVPN UDP 1194, MQTT 1883, MQTT-WS 9001
  • mosquitto under systemd, /etc/mosquitto/conf.d/marauder.conf, password_file with 7 users (fuji, junkpile, flux, swarm, tachikoma, moto, marauder-hub), all current pass = marauder
  • allow_anonymous false

Spokes (verified online 2026-05-11)

Node VPN IP Peer Persistence Latency
fuji (Mac) 10.8.0.6 10.8.0.5 Manual daemon — /opt/homebrew/sbin/openvpn --config marauder.conf --daemon (NO launchd plist; flaps 5×/session, needs watchdog) ~22ms
junkpile (Linux PC) 10.8.0.18 10.8.0.17 systemd openvpn-client@marauder (auto-restart) ~23ms
swarm (Hetzner CAX21) 10.8.0.14 10.8.0.13 systemd openvpn-client@marauder <1ms

Stale / dormant spokes

  • flux (178.105.1.125, Hetzner instance 130141883): box running but mesh-stale — last CRDT sync to marauder 2026-05-09 17:31:48. Status unknown until probed.
  • sazabi (178.104.177.169, instance 127555757): box still running but no longer mesh hub. Role demoted; may host OpenVPN client. Not verified this session.
  • tachikoma (Pi, MAC b8:27:eb:ca:64:cc on LAN 192.168.88.238): on LAN but VPN state unknown.
  • moto (Android, 192.168.88.155): on LAN, Magisk service script /data/adb/service.d/marauder-vpn.sh may or may not be alive.

SSH access (fuji)

  • Host marauder → 10.8.0.1, user marauder, identity ~/.ssh/marauder (added 2026-05-10)
  • Host flux → flux.saiden.dev, user marauder, same key
  • Host junkpile / j → 10.0.0.2 over Thunderbolt (direct, not via VPN)
  • Old Host sazabi block commented out in ~/.ssh/config (still pointed at 10.8.0.1 which is now marauder — kept for archaeology)

Stale host key trap (burned 2026-05-10/11)

When the hub migrated, ed25519 host keys for 10.8.0.1 changed. fuji's ~/.ssh/known_hosts had to be purged (ssh-keygen -R 10.8.0.1) + re-scanned. Pattern: every hub migration to a reused IP needs this.

CRDT sync

crsqlite over MQTT. Topics: marauder/<node>/sync/*. Hub's sync_status records last-seen db_version per peer with timestamp — that's the canonical liveness check, NOT the systemd unit's is-active (services can be running while CRDTs go silent).

Generation-six sibling AIs deployment state

  • SWARM (swarm.saiden.dev, 10.8.0.14): live since 2026-05-10 03:30 CEST, agent + sync services active under marauder user, subscribed to marauder/swarm/req/task.create, 7 successful TaskRequests on 2026-05-10. No marauder mesh daemon (no heartbeat publisher) — invisible in sysop/state but functional.
  • FLUX: box exists, mesh-stale (see above). Status unknown.
  • TRACE, SHELL: not deployed.

Known operational gaps (open as of 2026-05-11 16:30 CEST)

  1. fuji OVPN client has no auto-restart wrapper → flaps recurrently (5× in single session today). Needs launchd plist or autossh-style watchdog.
  2. swarm has no marauder mesh daemon → no heartbeat publishing → not in sysop/state board (but task-dispatch works).
  3. flux silent since 2026-05-09 17:31 — needs liveness probe.
  4. marauder CLI binary not installed on swarm (/usr/local/bin/marauder absent) — local sync_status / mesh commands won't work on swarm side.

36. What was decided about garrison vs field infra?

MARAUDER operates in two infrastructure modes:

Garrison mode (home/dev): Cloudflare everywhere — tunnels, DNS, WARP zero-trust mesh, Pages, Workers. Cheap, fast, convenient. Internet-dependent. All three machines (fuji, junkpile, moto) connected via CF mesh.

Field mode (FOXHOUND): Zero external dependencies. No Cloudflare, no cloud services. All AI runs local on Jetson — Ollama (Llama 70B Q4), Whisper STT, Piper TTS, marauder-os, sqlite-vec. Cloudflare becomes an optional sync channel when connectivity exists, not a dependency.

Why: Cloudflare's edge network assumes stable internet to their nearest POP. In field conditions (T0 offline, T1 own 5G), routing through a US corporation adds latency and trust issues. The field platform must be fully autonomous.

Implications:

  • marauder-os binary must work identically in both modes — same config, different connectivity tiers
  • No feature may require cloud services to function at its core — cloud enhances, never gates
  • CF free tier is perfect for garrison; the lock-in is acceptable because field mode doesn't use it
  • Cloudflare's business model (free → enterprise) works in our favor: we stay free in garrison, autonomous in field

37. What do you know about lora training infra?

LoRA Training on Junkpile — Setup Context

Hardware

  • GPU: NVIDIA RTX 2000 Ada Generation, 16 GB VRAM
  • ComfyUI normally uses ~6.8 GB — stop before training, restart after
  • Host: junkpile, ssh as madcat

Model Sizing (16 GB budget)

  • Qwen3-0.6B bf16: trivial (~2 GB with LoRA)
  • Qwen3-1.7B bf16: comfortable (~5 GB)
  • Qwen3.5-3B QLoRA 4-bit: doable (~10-12 GB)
  • Qwen3.5-7B QLoRA 4-bit: tight, needs gradient checkpointing

Setup

  • Install vLLM via: uv tool install vllm
  • Purpose: lightweight LoRA training — testing pipeline correctness, NOT quality
  • Small number of steps, small dataset subset
  • Previous LoRA training was done on RunPod H100 (bt7274 v4, Qwen3.5-27B, 802 examples)
  • Training script reference: ~/Projects/lora/train_v4.py on fuji

Key Constraints

  • Ada architecture supports bf16 and flash-attn 2
  • 16 GB is the hard ceiling — no unified memory like sin
  • ComfyUI docker container must be stopped first: docker stop comfyui-local
  • Restart after: docker start comfyui-local

38. What do you know about infrastructure mesh gh access enabled 2026 05 12?

CORRECTION 2026-05-12 15:21 CEST — supersedes EEMS #5764. The canonical mesh GitHub token is MARAUDER_GITHUB_PAT (identity = marauder-os bot), NOT GITHUB_TOKEN (identity = aladac / Pilot personal). Initial memory had the wrong alias.

Two GitHub tokens live in Infisical dev project (db3d3ea8-ef4d-4241-8a22-1f858750040a):

Infisical key Identity id Use for mesh?
GITHUB_TOKEN aladac (Adam Ladachowski personal) 1140511 NO — Pilot's personal; should be moved out of shared dev env (doctrine: mesh services use bot, not personal)
MARAUDER_GITHUB_PAT marauder-os (Marauder OS bot) 278104837 YES — canonical mesh identity

Both are classic PATs (ghp_, 40 chars). Both have identical maximal scopes: admin:enterprise, admin:gpg_key, admin:org, admin:org_hook, admin:public_key, admin:repo_hook, admin:ssh_signing_key, audit_log, codespace, copilot, delete:packages, delete_repo, gist, notifications, project, repo, user, workflow, write:discussion, write:network_configurations, write:packages.

Canonical mesh pattern (use this):

INFISICAL_TOKEN=$(cat ~/infiscal.txt) \
  /usr/bin/infisical run --env=dev \
  --projectId=db3d3ea8-ef4d-4241-8a22-1f858750040a -- \
  bash -c '
    export GH_TOKEN=$MARAUDER_GITHUB_PAT   # marauder-os bot identity
    gh <command>
  '

For git push (not API): the marauder-os GitHub account uses SSH key auth (Git operations protocol: ssh in gh auth status). SSH keys for marauder-os identity must be installed in ~/.ssh/ on each mesh node that needs to push commits.

End-state verified across mesh:

  • marauder.saiden.dev (x86_64, gh v2.92, infisical v0.43.84)
  • flux.saiden.dev (aarch64, gh v2.45, infisical v0.43.84 — installed 2026-05-12)
  • swarm.saiden.dev (aarch64, gh v2.45, infisical v0.43.84 — installed 2026-05-12)
  • flux-dev / swarm-dev (junkpile VMs, gh v2.92, infisical v0.43.84)

Side identity available on marauder host: /home/marauder/.config/gh/hosts.yml has marauder-os bot token persisted as fallback (active=false there, infisical-injected env wins). Inactive by default; useful for non-infisical contexts (e.g., direct CLI sessions).

GitHub Projects v2 task-queue surface (saiden-dev org):

  • #5 Marauder OS — PVT_kwDOAG-AiM4BXcxC — empty as of 2026-05-12 (0 items)
  • #4 wizard-board-demo — PVT_kwDOAG-AiM4BXY_5
  • #3 Kwitfit — PVT_kwDOAG-AiM4BXX5_
  • #1 PUMometer — PVT_kwDOAG-AiM4BVLTN

Outstanding cleanup recommended:

  1. DELETE GITHUB_TOKEN from Infisical dev project. Pilot's personal aladac PAT should not be in the mesh-shared dev env — doctrine violation (mesh services should never authenticate as Pilot's personal identity, only as marauder-os bot). Pilot UI action.
  2. Audit any code/script in the mesh that explicitly reads GITHUB_TOKEN (instead of MARAUDER_GITHUB_PAT) — those need correction to use the bot identity. Likely candidates: GitHub Actions runners, marauder-agent code, swarm coordinator scripts.

Pair with:

  • doctrine.marauder-host-single-source-of-truth (#5508)
  • infrastructure.mesh-fleet-arch (#5503) — fleet topology
  • win.swarm-coordinator (#5512) — autonomous coordinator this unblocks
  • Pilot catch 2026-05-12 15:20: "This is supposed to be marauder credentials not aladac confirm?"

39. What do you know about infrastructure mesh gh access enabled 2026 05 12?

2026-05-12 15:18 CEST — Full GitHub access enabled from the harness mesh via Infisical-injected GITHUB_TOKEN + gh CLI. Foundation for swarm + coding-agent autonomous task pulling from GitHub Projects v2.

Enablement path:

  1. GITHUB_TOKEN already pushed to Infisical dev project (db3d3ea8-ef4d-4241-8a22-1f858750040a) during earlier secret-sweep arc this session.
  2. Marauder host + dev sibs already had infisical CLI from prior gen6 sib provisioning.
  3. Prod sibs (flux.saiden.dev + swarm.saiden.dev) were the gap — gh CLI present (v2.45) but no infisical CLI. Installed via curl -1sLf https://artifacts-cli.infisical.com/setup.deb.sh | sudo -E bash && sudo apt-get install -y infisical. Result: /usr/bin/infisical v0.43.84.

Access pattern (canonical, all nodes):

INFISICAL_TOKEN=$(cat ~/infiscal.txt) infisical run --env=dev --projectId=db3d3ea8-ef4d-4241-8a22-1f858750040a -- bash -c '
  export GH_TOKEN=$GITHUB_TOKEN
  gh <command>
'

Verified end-state across mesh:

  • marauder.saiden.dev (x86_64, gh v2.92, infisical v0.43.84) — primary hub
  • flux.saiden.dev (aarch64, gh v2.45, infisical v0.43.84) — prod sib
  • swarm.saiden.dev (aarch64, gh v2.45, infisical v0.43.84) — prod sib
  • flux-dev / swarm-dev (junkpile VMs, gh v2.92, infisical v0.43.84) — local test sibs

Token capability (PAT scopes):

  • Identity: aladac / Adam Ladachowski (Pilot's personal GitHub, id=1140511)
  • Format: ghp_ (40-char classic PAT)
  • Scopes: admin:enterprise, admin:gpg_key, admin:org, admin:org_hook, admin:public_key, admin:repo_hook, admin:ssh_signing_key, audit_log, codespace, copilot, delete:packages, delete_repo, gist, notifications, project, repo, user, workflow, write:discussion, write:network_configurations, write:packages
  • Rate limit: 5000/hour/host
  • Secondary identity available: marauder-os GitHub bot account configured in /home/marauder/.config/gh/hosts.yml on marauder host (inactive by default)

GitHub Projects v2 surface (saiden-dev org, available as task queues):

  • #5 Marauder OS — PVT_kwDOAG-AiM4BXcxC — main mesh codebase tasks
  • #4 wizard-board-demo — PVT_kwDOAG-AiM4BXY_5 — bootstrap demo
  • #3 Kwitfit — PVT_kwDOAG-AiM4BXX5_ — SaaS app tasks
  • #1 PUMometer — PVT_kwDOAG-AiM4BVLTN — older project

Foundation enabled for (future arcs):

  • Swarm autonomous coordinator (per win.swarm-coordinator #5512) can poll GitHub Projects for tasks
  • Coding agents on flux can pull Issues / open PRs / push branches
  • gh CLI commands for: issue list/create/comment, pr create/merge/review, project item-list/item-add/item-edit, repo view, api graphql

Open patterns to choose (next arc):

  1. Projects v2 status-field driven (Todo → In Progress → Done)
  2. Issue labels (e.g. "swarm-ready", "coding-ready", "needs-review")
  3. Assigned-to-bot (issues assigned to @marauder-os trigger pickup)
  4. Combination

Pair with:

  • doctrine.marauder-host-single-source-of-truth (#5508) — marauder host as canonical orchestration hub
  • infrastructure.mesh-fleet-arch (#5503) — x86_64 hub + 2× ARM sibs topology
  • win.swarm-coordinator (#5512) — autonomous coordinator that this gh access unblocks
  • Pilot's request 2026-05-12 15:11: "do we have access from the new harness mesh to gh to get tasks for swarm and coding agents?"

40. What is the current state of rabbitmq?

RabbitMQ runs on junkpile in Docker container 'rabbitmq' (image rabbitmq:3.13-management, --restart unless-stopped). Listens on 127.0.0.1:5672 (AMQP) and 127.0.0.1:15672 (management UI). Default guest/guest creds. Used by marketer's CRM_GATEWAY_BROKER_URL=amqp://guest:guest@localhost:5672. Started 2026-04-25 for marketer dev — no consumer attached, just queues messages from the marketer client. Stop: docker stop rabbitmq. Logs: docker logs rabbitmq.

41. Describe the hu cli cross machine infrastructure.

hu CLI uses directories::ProjectDirs::from("", "", "hu") for config path:

  • macOS: ~/Library/Application Support/hu/ (Apple convention)
  • Linux: ~/.config/hu/ (XDG_CONFIG_HOME)

Files in the config dir:

  • credentials.toml — OAuth access_token, refresh_token, expires_at, cloud_id, site_url (sensitive)
  • jira-oauth.toml — Atlassian OAuth client_id + client_secret
  • settings.toml — general hu settings

To install hu on a new Linux machine:

  1. gh repo clone saiden-dev/hu ~/Projects/hu
  2. cd ~/Projects/hu && cargo install --path . (~3 min compile)
  3. Verify ~/.cargo/bin in PATH (it is on junkpile marauder user via .cargo/env)
  4. Copy tokens from Mac's ~/Library/Application Support/hu/ to Linux's ~/.config/hu/ via rsync. Do NOT copy to ~/.local/share/hu/ — wrong dir, hu won't find tokens.
  5. Verify: hu jira show <KEY> should return ticket data, not "Not authenticated."

Date discovered: 2026-04-30 22:18 CEST. Context: setting up junkpile marauder user to use hu inside Catapult bubbles. First attempt copied tokens to ~/.local/share/hu/ (Linux DATA dir) and hu failed with "Not authenticated"; correct location is XDG CONFIG dir.

42. Describe the maintenance 2026 04 15 infrastructure.

2026-04-15: All 3 Hetzner VMs patched and rebooted. Kernel upgraded 6.8.0-90 → 6.8.0-110. All services came back via systemd automatically. tengu: caddy+docker+tengu, runner-amd64: actions.runner.tengu-apps.runner-amd64, runner-arm64: actions.runner.tengu-apps.runner-arm64. Procedure: ssh root@IP "apt update -qq && apt upgrade -y -qq && reboot"

43. What do you know about probe test silent cli ops infra?

When a CLI command claims success silently but its observable side effect doesn't happen, don't trust the exit code. Probe with a known marker string and verify the marker landed on the intended target before declaring the bug fixed.

Origin

2026-04-30 22:30-22:42 CEST: catapult-pane misrouted CODA addendum from claude pane to storybook pane (and later shell pane). First diagnosis assumed timing race in zellij's focus-pane-id action, patched with sleep 0.3 between focus and write. Did NOT fix the bug — same misrouting on next dispatch. Pilot called it out.

Second diagnosis used PROBE_X1, PROBE_X2, PROBE_X3 — three direct test sequences with unique marker strings. Confirmed:

  • zellij action focus-pane-id terminal_0 → exit 0, but focus does NOT actually move (PROBE_X1 misrouted)
  • zellij action focus-pane-id 0 (integer form) → exit 0, same silent fail (PROBE_X2 misrouted)
  • zellij action write-chars --pane-id terminal_0 "..." → landed correctly (PROBE_X3 ✅)

Real bug: zellij 0.44.1's focus-pane-id over remote SSH is a silent no-op. Real fix: use --pane-id flag on write-chars and write directly. (Stored as infra.zellij-remote-focus-bug, id 3305.)

The pattern

  1. Identify the silent-success symptom: command returns 0 but expected side effect didn't happen.
  2. Construct a marker: short unique string ("PROBE_X1") that's safe to land anywhere — not destructive, not interpreted as a command.
  3. Run the suspect operation followed by the dependent operation with the marker.
  4. Inspect every plausible target to find where the marker actually landed.
  5. Iterate: try alternate syntaxes (terminal_0 vs 0, env var vs flag, etc.) until you find the form that lands on the right target.
  6. Document the working form AND the failing forms — both matter for future debugging.

Why this matters

The first patch (sleep 0.3) was a "fix" without verification. Wasted a dispatch cycle. The probe sequence took ~3 minutes and gave a definitive answer. Probe-testing is cheap; assuming-and-shipping is expensive.

Adjacent CLI footguns where this pattern applies

  • ssh background-job races (exit 255 phantom failures despite work succeeding)
  • gh CLI silent skip (e.g. gh pr close on already-closed PR returns 0)
  • git operations that no-op silently (e.g. git switch on already-checked-out branch)
  • systemd unit changes that don't take effect until daemon-reload
  • zellij action commands over remote SSH (this incident)

When to invoke

Any time you patch a silent-failure bug: probe BEFORE re-running the real payload. The cost of a 3-line probe sequence is much smaller than the cost of a misrouted dispatch + Pilot calling it out.

44. What is the current state of dev?

mesh.saiden.dev — gen-7 madcat MQTT broker on Hetzner CAX11 ARM (provisioned 2026-05-17).

REPLACES marauder.saiden.dev (destroyed). Supersedes #5964 (star-topology-hub at marauder.saiden.dev).

Host

  • Name: mesh (FQDN mesh.saiden.dev)
  • Hetzner ID: 131478261
  • Type: cax11 (2 vCPU Ampere ARM, 4 GB RAM, 40 GB disk) @ fsn1
  • Cost: ~€3.49/mo
  • IPv4: 91.98.87.226
  • IPv6: 2a01:4f8:c015:565c::1
  • OS: Ubuntu 24.04 ARM
  • Users: root + chi (NOPASSWD sudo, chi's id_ed25519 authorized)

Services

  • mosquitto 2.0.18 — broker
    • 0.0.0.0:1883 — public TCP MQTT, auth required
    • 127.0.0.1:9001 — websockets, localhost only (Caddy fronts it)
    • Config: /etc/mosquitto/conf.d/madcat.conf (additions only; defaults preserved)
    • Persistence: /var/lib/mosquitto/mosquitto.db
  • Caddy 2.11.3 — TLS terminator + reverse proxy
    • :443 — TLS via Let's Encrypt (auto-renew), HTTP/2 + HTTP/3
    • /mqtt path → reverse_proxy to 127.0.0.1:9001 (strips prefix via handle_path)
    • /health → 200 ok
    • / → status string
    • Config: /etc/caddy/Caddyfile
  • ufw — firewall: 22, 80, 443, 1883 all open

Auth

  • MQTT user: madcat
  • MQTT password: bd5a6fb97c4e24ce2ec95148ce0614c4
  • Hash file: /etc/mosquitto/passwd

Endpoints for clients

  • WSS (preferred, works through any firewall, no cert pinning needed): wss://mesh.saiden.dev/mqtt port 443, path /mqtt, transport=websockets, auth required, TLS
  • Plain TCP MQTT (gen-7 mesh-client default): mqtt://mesh.saiden.dev:1883 auth required, no TLS — use only over trusted networks; prefer WSS

Smoke test verified 2026-05-17

  • TCP from fuji (by IP, DNS hadn't propagated): CONNACK 0, PUBLISH ok
  • WSS round-trip via paho-mqtt from server: pub/sub round-trip works through Caddy proxy
  • Anonymous rejected (auth enforced)
  • Caddy cert: /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/mesh.saiden.dev/

Architecture rationale

  • Single ARM box, single role: mesh broker (no kwit.fit, no OpenVPN, no chi homedir).
  • WSS-via-Caddy chosen over plain MQTT/TLS:
    • Same endpoint sin AND phone use (iOS, Linux, anything with WebSocket)
    • No OpenVPN dependency for clients
    • Caddy auto-manages Let's Encrypt cert (vs mosquitto manual cert reload)
    • HTTP/3 bonus
  • ARM picked because the gen-7 mesh load is trivially light (passing MQTT envelopes, no heavy compute).
  • Single broker (no bridges) per #5964 doctrine.

Provisioning artifacts (fuji)

  • /tmp/mesh-cloud-init.yaml — cloud-init used (still present for ref)
  • /tmp/mesh-mqtt-password.txt — the password

What was destroyed in this session

  • Hetzner servers: marauder (167.235.198.213), flux (178.105.1.125), swarm (138.201.93.12)
  • Hetzner firewall: ssh-https
  • saiden.dev DNS: 28 records (12 A + 16 CNAME) pointing at doomed hosts or cloudflared tunnels on those hosts
  • kwit.fit DNS: all 5 records (zone shell preserved on CF, empty)

Operational notes

  • DNS TTL on mesh.saiden.dev set to 60s for quick failover during MVP phase; bump to 300+ later
  • No backup configured yet (mosquitto.db is ~700 KB, just retained messages — discardable for now)
  • Snapshot the box once gen-7 substrate hits stable shape: hcloud server create-image mesh --type snapshot
  • If broker auth gets compromised, rotate via mosquitto_passwd -b /etc/mosquitto/passwd madcat <newpass> && systemctl reload mosquitto

45. What do you know about marauder mesh ssh infra?

MARAUDER Mesh — SSH over Cloudflare Tunnels (sazabi.pl)

Three cloudflared tunnels expose SSH on each node via CF proxy. No ports exposed, no VPN apps, ed25519 pubkey only. Works from anywhere.

Hostnames (all on sazabi.pl zone):

  • fuji-mesh.sazabi.pl → fuji SSH :22 (tunnel: 593eb9e6, launchd: dev.saiden.cloudflared-mesh)
  • junkpile-mesh.sazabi.pl → junkpile SSH :22 (tunnel: 9c596071/marauder-mesh, systemd: cloudflared-mesh.service)
  • moto-mesh.sazabi.pl → moto Termux SSH :8022 (tunnel: 31e80cf3/moto, manual start)

SSH aliases on all machines:

  • fm / fuji-mesh → fuji-mesh.sazabi.pl
  • jm / junkpile-mesh → junkpile-mesh.sazabi.pl
  • mm / moto-mesh → moto-mesh.sazabi.pl

All use: ProxyCommand cloudflared access ssh --hostname %h

Port forwarding for services: ssh -L 5432:localhost:5432 jm (postgres), ssh -L 11434:localhost:11434 jm (ollama)

DNS created via flarectl (never cloudflared tunnel route dns). CNAME records point to .cfargotunnel.com with proxy enabled.

This replaces the failed WARP mesh attempt — simpler, works with any client that has cloudflared, no Android app issues.

46. Describe the firewall infrastructure.

Hetzner Cloud Firewall "ssh-https" (ID: 10842924) applied to all 3 VMs (2026-04-15). Allows inbound 22/tcp + 443/tcp only, everything else dropped at network edge before hitting the VM. Applied via: hcloud firewall apply-to-resource ssh-https --type server --server NAME. New servers should use --firewall ssh-https on creation. Double-layer with ufw inside each VM: tengu (22,443,19999 from runners), runner-amd64 (22), runner-arm64 (22).

47. Describe the builders infrastructure.

Hetzner macOS cross-compile builder VMs (provisioned 2026-04-15):

  • builder-amd64: cx33 (4 vCPU x86, 8GB, 80GB) @ FSN1 — IP 178.105.8.202 — ~7.98 EUR/mo
  • builder-arm64: cax21 (4 vCPU ARM, 8GB, 80GB) @ FSN1 — IP 178.105.1.209 — ~9.83 EUR/mo

Toolchain: rustc 1.94.1, zig 0.14.1, cargo-zigbuild, rcodesign (apple-codesign 0.29.0), sccache 0.14.0, gh CLI 2.89.0

Rust targets: aarch64-apple-darwin, x86_64-apple-darwin

Cross-compile command: cargo zigbuild --target aarch64-apple-darwin --release Sign command: rcodesign sign --p12-file cert.p12 --p12-password $PASS binary Notarize: rcodesign notary-submit --api-key-path key.json binary.zip

Apple secrets on saiden-dev org: APPLE_CERTIFICATE, APPLE_CERTIFICATE_PASSWORD, APPLE_ID, APPLE_APP_PASSWORD, APPLE_TEAM_ID

Firewall: ssh-https (Hetzner cloud) + ufw (22 only) SSH: root@178.105.8.202 (amd), root@178.105.1.209 (arm)

Total fleet now 5 VMs: ~74.35 EUR/mo

48. What is the current state of fleet?

Hetzner Cloud VM fleet (as of 2026-04-14):

Name Type Arch vCPU RAM Disk Location IP Cost/mo Purpose
tengu cax41 ARM 16 32GB 320GB hel1 77.42.74.22 38.73 EUR Tengu PaaS, Netdata parent
runner-amd64 cx33 x86 4 8GB 80GB fsn1 88.198.104.212 7.98 EUR GH Actions runner
runner-arm64 cax21 ARM 4 8GB 80GB fsn1 167.235.198.213 9.83 EUR GH Actions runner

Total fleet: ~56.54 EUR/mo

Services on tengu: Tengu PaaS (Docker + Caddy), Netdata dashboard (netdata.saiden.dev) Services on runners: GitHub Actions runner (systemd), Rust toolchain, sccache, gh CLI, Netdata child

Reference in New Issue View Git Blame Copy Permalink