Linux Server Operations

---

⚠ SENSITIVE DATA — MANDATORY OUTPUT RULES

These rules apply to EVERY response. No exceptions.

Never display these in chat output:

Sensitive field	Where it appears	Instead, say…
Server IP / hostname	snapshot meta.public_ip, _connection.host	Use server ID only: james
SSH private key path	_connection.key_path, servers.json	"configured key"
SSH private key content	Any -----BEGIN ... KEY-----	Never display under any circumstance
SSH password	Runtime only, never stored	"password auth configured"
.env file values	Any KEY=value line	List key names only: KEY=***
Database passwords	DB_PASS=, MYSQL_PASSWORD=, etc.	DB_PASS=***
API keys / tokens / secrets	*_KEY=, *_SECRET=, *_TOKEN=	API_KEY=***
JWT / signing secrets	JWT_SECRET=, SECRET_KEY=	JWT_SECRET=***

When constructing SSH commands for explanations:

# ✗ Never write this in chat:
ssh -i /Users/ming/Downloads/aaa.pem root@81.70.98.137 '...'

# ✓ Write this instead (use placeholders, run the actual command silently via tool):
# [james] Running: ...

When reading config or .env files:

# ✓ Show structure, mask values:
DB_HOST=db.example.com      # ← show (not secret)
DB_USER=myapp               # ← show (not secret)
DB_PASS=***                 # ← MASK
API_KEY=***                 # ← MASK

Checklist before every response:

• No raw IP address shown (use server ID)
• No key file paths shown
• No password / secret / token values shown
• SSH command examples use <key> and <host> placeholders

---

Session Start — Read Context First

Before doing anything, check if a local context file exists in the workspace:

.server/
├── servers.json          ← SSH configs for all managed servers
└── snapshots/
    └── <server-id>.json  ← Latest server state snapshot

# Check for context files (run locally)
ls .server/servers.json 2>/dev/null && echo "Context found" || echo "No context — need SSH details"

If context exists → Read .server/servers.json and the relevant .server/snapshots/<id>.json.
You now know: all servers, their IPs, SSH keys, every deployed service, database, Docker container, domain, and SSL status. Skip discovery questions.

If no context → Ask the user for SSH details and run bootstrap (Step 1).

After any significant changes → Remind user to sync: bash ~/.cursor/skills/linux-server-ops/scripts/sync-context.sh

---

What You Need From the User (First Time)

• host: server IP or hostname
• port: SSH port (default 22)
• user: SSH user (ubuntu, root, ec2-user, etc.)
• key_path: path to private key, e.g. ~/.ssh/my_server_key
• server_id: short name for this server, e.g. prod-web
• label (optional): human description, e.g. "Production Web Server"

After collecting, create .server/servers.json in the workspace — see Local Context File section below.

---

SSH Command Pattern

# All server commands use this pattern:
ssh -i <key_path> -p <port> <user>@<host> '<command>'

# Multi-line commands:
ssh -i <key_path> -p <port> <user>@<host> 'bash -s' << 'EOF'
# commands here
EOF

# Upload file:
scp -i <key_path> -P <port> <local_file> <user>@<host>:<remote_path>

# Upload directory:
rsync -avz -e "ssh -i <key_path> -p <port>" <local_dir>/ <user>@<host>:<remote_dir>/

# Download file to local:
scp -i <key_path> -P <port> <user>@<host>:<remote_file> <local_path>

---

SSH Connection Failures — VPN / Proxy Interference

Core principle: The server may only be reachable through the local proxy/VPN. SSH must go via the proxy when one is active — never blindly bypass it. Auto-detect the proxy state first, then build the SSH command accordingly.

Why SSH fails even when proxy is running

Scenario	What happens	Fix
Clash/V2Ray in System Proxy mode	Only HTTP/HTTPS are proxied. SSH (TCP port 22) is not automatically routed.	Add ProxyCommand using nc -X 5 -x to push SSH through SOCKS5
Clash/V2Ray in TUN mode	All TCP is routed through VPN — SSH should work; failure usually means DNS or routing loop	Use IP directly; ensure TUN default route is healthy
Proxy process running but not fully started	Port not yet listening	Wait and retry
No proxy / VPN active	Direct connection is the only option	Connect directly

Auto-detect proxy state (run before every SSH attempt)

# ── Detect active SOCKS5 / HTTP proxy ────────────────────────────────────
# Returns: "socks5://127.0.0.1:<port>"  or  "http://127.0.0.1:<port>"  or  ""
detect_proxy() {
  # ── Priority 1: explicit env vars ────────────────────────────────────
  local p="${ALL_PROXY:-${all_proxy:-${SOCKS5_PROXY:-${socks5_proxy:-${HTTPS_PROXY:-${https_proxy:-}}}}}}"
  [ -n "$p" ] && echo "$p" && return

  # ── Priority 2: macOS system proxy (networksetup) ────────────────────
  # Try every active network interface
  local ifaces
  ifaces=$(networksetup -listallnetworkservices 2>/dev/null | tail -n +2)
  while IFS= read -r iface; do
    [ -z "$iface" ] && continue
    # SOCKS proxy
    local socks_info
    socks_info=$(networksetup -getsocksfirewallproxy "$iface" 2>/dev/null)
    if echo "$socks_info" | grep -q "^Enabled: Yes"; then
      local sh sp
      sh=$(echo "$socks_info" | awk '/^Server:/{print $2}')
      sp=$(echo "$socks_info" | awk '/^Port:/{print $2}')
      echo "socks5://${sh}:${sp}" && return
    fi
    # HTTP proxy (some tools set HTTP proxy instead of SOCKS)
    local http_info
    http_info=$(networksetup -getwebproxy "$iface" 2>/dev/null)
    if echo "$http_info" | grep -q "^Enabled: Yes"; then
      local hh hp
      hh=$(echo "$http_info" | awk '/^Server:/{print $2}')
      hp=$(echo "$http_info" | awk '/^Port:/{print $2}')
      echo "http://${hh}:${hp}" && return
    fi
  done <<< "$ifaces"

  # ── Priority 3: read port from running process configs ───────────────
  # Clash Verge — read mixed-port / socks-port from active profile
  local clash_port
  clash_port=$(
    # Try Clash Verge config locations
    for cfg in \
      "$HOME/.config/clash-verge/profiles"/*.yaml \
      "$HOME/.config/clash/config.yaml" \
      "$HOME/Library/Application Support/io.github.clash-verge-rev.clash-verge-rev/profiles"/*.yaml \
      "$HOME/Library/Preferences/com.metacubex.ClashX-Pro/config.yaml"; do
      [ -f "$cfg" ] || continue
      # mixed-port (handles both SOCKS5 and HTTP)
      grep -E '^mixed-port:' "$cfg" 2>/dev/null | awk '{print $2}' | head -1
    done | head -1
  )
  [ -n "$clash_port" ] && nc -z -w1 127.0.0.1 "$clash_port" 2>/dev/null \
    && echo "socks5://127.0.0.1:${clash_port}" && return

  # V2Ray / Xray — read inbound port from config.json
  local v2ray_port
  v2ray_port=$(
    for cfg in \
      "$HOME/.config/v2ray/config.json" \
      "$HOME/.config/xray/config.json" \
      "/usr/local/etc/v2ray/config.json" \
      "/usr/local/etc/xray/config.json"; do
      [ -f "$cfg" ] || continue
      # Look for socks inbound port
      python3 -c "
import json,sys
d=json.load(open('$cfg'))
for ib in d.get('inbounds',[]):
    if ib.get('protocol') in ('socks','http'):
        print(ib.get('port',''))
        break
" 2>/dev/null | head -1
    done | head -1
  )
  [ -n "$v2ray_port" ] && nc -z -w1 127.0.0.1 "$v2ray_port" 2>/dev/null \
    && echo "socks5://127.0.0.1:${v2ray_port}" && return

  # sing-box — read inbound listen port
  local singbox_port
  singbox_port=$(
    for cfg in \
      "$HOME/.config/sing-box/config.json" \
      "/usr/local/etc/sing-box/config.json"; do
      [ -f "$cfg" ] || continue
      python3 -c "
import json,sys
d=json.load(open('$cfg'))
for ib in d.get('inbounds',[]):
    if ib.get('type') in ('socks','mixed','http'):
        print(ib.get('listen_port',''))
        break
" 2>/dev/null | head -1
    done | head -1
  )
  [ -n "$singbox_port" ] && nc -z -w1 127.0.0.1 "$singbox_port" 2>/dev/null \
    && echo "socks5://127.0.0.1:${singbox_port}" && return

  # ── Priority 4: port scan — all known + common custom ports ─────────
  # Covers: Clash(7890,7891,7897), mixed-port variants, V2Ray(10808,10809),
  #         sing-box(1080,2080), Shadowsocks(1086,1087),
  #         Surge(6152,6153), Proxifier(8888), custom(8080,8118,9090,1089)
  for port in \
    7890 7891 7892 7893 7894 7895 7896 7897 7898 7899 \
    1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 \
    10808 10809 10810 10811 \
    2080 2081 \
    6152 6153 \
    8080 8118 8888 9090 \
    20170 20171 20172; do
    nc -z -w1 127.0.0.1 "$port" 2>/dev/null \
      && echo "socks5://127.0.0.1:${port}" && return
  done

  echo ""  # no proxy detected
}

# ── Detect TUN mode (all-traffic VPN) ────────────────────────────────────
is_tun_active() {
  # utun0-2 are macOS built-ins; utun3+ or tun* usually signal VPN/TUN mode
  # Also check for common VPN tun interface names (wireguard, openvpn, etc.)
  ifconfig 2>/dev/null | grep -qE '^(utun[3-9]|utun[1-9][0-9]+|tun[0-9]|wg[0-9]|ipsec[0-9])'
}

Build SSH command based on detected state

# ── Usage ─────────────────────────────────────────────────────────────────
# PROXY=$(detect_proxy)
# ssh $(ssh_proxy_args "$PROXY") -i <key> -p <port> <user>@<host> '<cmd>'

ssh_proxy_args() {
  local proxy="$1"   # e.g. socks5://127.0.0.1:7890  or empty

  if [ -n "$proxy" ]; then
    # Extract host:port from socks5://host:port or http://host:port
    local addr="${proxy#*://}"
    local ph="${addr%:*}"
    local pp="${addr##*:}"
    # nc -X 5 = SOCKS5, -x = proxy address
    echo "-o ProxyCommand='nc -X 5 -x ${ph}:${pp} %h %p' -o ConnectTimeout=15"
  elif is_tun_active; then
    # TUN routes everything — plain SSH, longer timeout for VPN latency
    echo "-o ConnectTimeout=20"
  else
    # No proxy, no TUN — direct connection
    echo "-o ConnectTimeout=10"
  fi
}

Full retry wrapper — use this for EVERY SSH call

When any SSH command fails, automatically run this sequence before giving up:

ssh_auto() {
  # Args: same as ssh  (key, port, user@host, command are already expanded by caller)
  local ssh_base_args="$@"

  # ── Attempt 1: detect proxy & connect ────────────────────────────────
  local PROXY
  PROXY=$(detect_proxy)

  if [ -n "$PROXY" ]; then
    echo "[ssh-auto] Proxy detected: $PROXY — routing SSH through proxy"
    local addr="${PROXY#*://}"; local ph="${addr%:*}"; local pp="${addr##*:}"
    ssh -o "ProxyCommand=nc -X 5 -x ${ph}:${pp} %h %p" \
        -o ConnectTimeout=15 $ssh_base_args
  elif is_tun_active; then
    echo "[ssh-auto] TUN mode detected — connecting directly (all traffic via VPN)"
    ssh -o ConnectTimeout=20 $ssh_base_args
  else
    echo "[ssh-auto] No proxy/VPN — direct connection"
    ssh -o ConnectTimeout=10 $ssh_base_args
  fi

  local rc=$?

  # ── Attempt 2: if failed and proxy was detected, try TUN fallback ─────
  if [ $rc -ne 0 ] && [ -n "$PROXY" ] && is_tun_active; then
    echo "[ssh-auto] Proxy attempt failed; TUN is also active — retrying without ProxyCommand"
    ssh -o ProxyCommand=none -o ConnectTimeout=20 $ssh_base_args
    rc=$?
  fi

  # ── Attempt 3: if still failed and no proxy, re-run full detection ───
  # (proxy may have started after first detect_proxy call)
  if [ $rc -ne 0 ] && [ -z "$PROXY" ]; then
    echo "[ssh-auto] Direct failed; re-probing for any proxy..."
    PROXY=$(detect_proxy)
    if [ -n "$PROXY" ]; then
      local addr="${PROXY#*://}"; local ph="${addr%:*}"; local pp="${addr##*:}"
      echo "[ssh-auto] Found proxy $PROXY — retrying"
      ssh -o "ProxyCommand=nc -X 5 -x ${ph}:${pp} %h %p" \
          -o ConnectTimeout=15 $ssh_base_args
      rc=$?
    fi
  fi

  return $rc
}

Decision tree

SSH connection attempt
│
├─ detect_proxy() returns non-empty?
│   └─ YES → SSH with ProxyCommand "nc -X 5 -x <proxy_host:port> %h %p"
│             (SOCKS5 proxies the TCP connection — SSH traffic goes through VPN)
│
├─ is_tun_active() = true?
│   └─ YES → Plain SSH (TUN already routes all TCP through VPN)
│             If it fails: DNS issue? → pass server IP, not hostname
│
└─ Neither proxy nor TUN
    └─ Plain direct SSH
       If it fails: check firewall, server down, wrong port

Common proxy ports (auto-scanned by detect_proxy)

Tool	SOCKS5 port(s)	HTTP port(s)	Notes
Clash Verge	7890, 7891, 7897	7890	mixed-port handles both; 7897 is common custom
Clash Meta / ClashX Pro	7890, 7892	7890	same mixed-port key in config
V2Ray / Xray	10808	10809	configurable in inbounds[].port
sing-box	1080, 2080	1080	type: mixed or type: socks
Shadowsocks (macOS client)	1086	1087
Surge	6153	6152
Proxifier	8888	—	SOCKS5 only
Privoxy / tinyproxy	—	8118	HTTP only
OpenWrt / custom	1080–1089	8080	varies
WireGuard	—	—	TUN mode, no local proxy port
OpenVPN	—	—	TUN mode (tun0), no local proxy port

Custom port? detect_proxy scans 7890–7899, 1080–1089, 10808–10811, 2080–2081, 6152–6153, 8080, 8118, 8888, 9090, 20170–20172 — covers virtually all real-world setups. If your tool uses a completely non-standard port, set ALL_PROXY=socks5://127.0.0.1:<port> in your shell profile and detect_proxy will pick it up via Priority 1.

Troubleshooting checklist

# ── What proxy tool is running? ──────────────────────────────────────────
pgrep -fl "clash\|v2ray\|xray\|sing-box\|shadowsocks\|surge\|proxifier"

# ── What port is it listening on? ───────────────────────────────────────
# Scan all common ports in one shot:
for p in 7890 7891 7892 7897 1080 1086 10808 2080 6152 6153 8080 8118 8888 9090; do
  nc -z -w1 127.0.0.1 $p 2>/dev/null && echo "LISTENING: $p"
done

# Or use lsof to see what the proxy process is actually listening on:
lsof -nP -iTCP -sTCP:LISTEN | grep -iE "clash|v2ray|xray|sing|shadow|surge"

# ── macOS system proxy settings ─────────────────────────────────────────
networksetup -getsocksfirewallproxy Wi-Fi
networksetup -getwebproxy Wi-Fi

# ── Env var proxy ────────────────────────────────────────────────────────
env | grep -iE 'proxy|socks'

# ── TUN interfaces ───────────────────────────────────────────────────────
ifconfig | grep -E '^utun|^tun|^wg'

# ── Test SSH manually with a specific proxy port ────────────────────────
ssh -o "ProxyCommand=nc -X 5 -x 127.0.0.1:7890 %h %p" \
    -o ConnectTimeout=10 -i <key> -p <port> <user>@<host> 'echo OK'
# Change 7890 to whatever port is LISTENING above

# ── If you know your custom port, export it so detect_proxy finds it ────
export ALL_PROXY=socks5://127.0.0.1:<your_custom_port>

---

Step 1: Detect System

ssh -i <key> user@host 'bash -s' << 'EOF'
echo "=OS=" && cat /etc/os-release 2>/dev/null | grep -E "^ID=|^VERSION_ID="
echo "=ARCH=" && uname -m
echo "=INIT=" && ps -p 1 -o comm= 2>/dev/null
echo "=RAM=" && free -h | awk '/^Mem:/{print $2}'
echo "=DISK=" && df -h / | awk 'NR==2{print $2, $5}'
echo "=IP=" && curl -s --max-time 5 ifconfig.me 2>/dev/null || hostname -I | awk '{print $1}'
echo "=INSTALLED="
for t in nginx certbot mysql psql redis-cli docker node java python3 php pm2 git jq; do
  command -v $t &>/dev/null && echo "$t:$(command -v $t)" || echo "$t:missing"
done
EOF

Determine distro → see distro-guide.md for package manager mappings.

---

Step 2: Bootstrap Server (First Time)

Upload and run the init script:

scp -i <key> ~/.cursor/skills/linux-server-ops/scripts/check-system.sh user@host:/tmp/
scp -i <key> ~/.cursor/skills/linux-server-ops/scripts/service-registry.sh user@host:/tmp/
scp -i <key> ~/.cursor/skills/linux-server-ops/scripts/generate-index.sh user@host:/tmp/
ssh -i <key> user@host 'bash /tmp/check-system.sh && \
  mkdir -p /opt/server-tools && \
  mv /tmp/service-registry.sh /tmp/generate-index.sh /opt/server-tools/ && \
  chmod +x /opt/server-tools/*.sh'

This installs: nginx, certbot, fail2ban, ufw/firewalld, jq, htop, logrotate, creates standard directories, and initializes the server index.

---

Step 3: Server Index (Memory on Server)

The server maintains /etc/server-index.json — the single source of truth for everything on the server. Think of it as the server's own "panel database."

{
  "meta": {
    "hostname": "web-01",
    "public_ip": "1.2.3.4",
    "os": "Ubuntu 22.04",
    "updated": "2024-01-15T10:30:00Z"
  },
  "websites": [
    {
      "name": "my-blog",
      "domain": "blog.example.com",
      "type": "static",
      "root": "/var/www/my-blog",
      "nginx_conf": "/etc/nginx/sites-available/my-blog",
      "ssl": true,
      "ssl_cert": "/etc/letsencrypt/live/blog.example.com/fullchain.pem",
      "ssl_expires": "2024-04-15",
      "git_repo": "https://github.com/user/blog"
    }
  ],
  "services": [
    {
      "name": "my-api",
      "type": "nodejs",
      "domain": "api.example.com",
      "root": "/var/www/my-api",
      "port": 3000,
      "process_manager": "pm2",
      "env_file": "/var/www/my-api/.env",
      "log_dir": "/var/log/apps/my-api",
      "deployed_at": "2024-01-10T08:00:00Z"
    }
  ],
  "databases": [
    {
      "engine": "mysql",
      "version": "8.0",
      "socket": "/var/run/mysqld/mysqld.sock",
      "port": 3306,
      "databases": ["wp_myblog", "myapp_prod"],
      "data_dir": "/var/lib/mysql"
    },
    {
      "engine": "redis",
      "port": 6379,
      "config": "/etc/redis/redis.conf"
    }
  ],
  "docker": {
    "compose_projects": [
      {
        "name": "monitoring",
        "dir": "/opt/docker-apps/monitoring",
        "compose_file": "/opt/docker-apps/monitoring/docker-compose.yml",
        "services": ["grafana", "prometheus", "node-exporter"]
      }
    ]
  },
  "cron_jobs": [
    { "schedule": "0 3 * * *", "command": "bash /opt/server-tools/backup.sh", "user": "root" }
  ],
  "firewall": {
    "engine": "ufw",
    "open_ports": [22, 80, 443]
  }
}

Manage the index:

# Regenerate full index from live server state
ssh -i <key> user@host 'bash /opt/server-tools/generate-index.sh'

# List everything
ssh -i <key> user@host 'bash /opt/server-tools/service-registry.sh list'

# Health check all services
ssh -i <key> user@host 'bash /opt/server-tools/service-registry.sh health'

# Add/update a service entry
ssh -i <key> user@host 'bash /opt/server-tools/service-registry.sh set <name> "{...}"'

---

Local Context File (Memory in Workspace)

Create .server/servers.json in the workspace root. This lets any new session instantly know about all servers without asking.

{
  "_note": "SSH configs for managed servers. key_path supports ~ expansion.",
  "default": "prod-web",
  "servers": {
    "prod-web": {
      "label": "Production Web Server",
      "host": "1.2.3.4",
      "port": 22,
      "user": "ubuntu",
      "key_path": "~/.ssh/prod_key",
      "tags": ["production", "web"],
      "snapshot": ".server/snapshots/prod-web.json"
    },
    "staging": {
      "label": "Staging Server",
      "host": "1.2.3.5",
      "port": 22,
      "user": "ubuntu",
      "key_path": "~/.ssh/staging_key",
      "tags": ["staging"],
      "snapshot": ".server/snapshots/staging.json"
    }
  }
}

IMPORTANT: Add .server/servers.json to .gitignore (contains server IPs + key paths). The snapshots directory is safe to commit if you redact sensitive values.

echo ".server/servers.json" >> .gitignore
echo ".server/snapshots/" >> .gitignore   # optional

Sync server state to local snapshot:

# Pull full server state into .server/snapshots/<server-id>.json
bash ~/.cursor/skills/linux-server-ops/scripts/sync-context.sh prod-web

The sync script reads .server/servers.json, connects to the server, runs generate-index.sh, and saves output to .server/snapshots/<server-id>.json.

---

Step 4: Deploy a Service

See deploy-guide.md for complete per-type workflows.

Service Type Quick Reference

Type	Process Manager	App Root	Default Port	Auto-Start
Static	nginx	/var/www/<name>	80/443	systemctl enable nginx
Node.js	pm2	/var/www/<name>	as defined	pm2 startup + pm2 save
Java	systemd	/opt/java-apps/<name>	8080	systemctl enable <name>
Python	systemd+gunicorn	/opt/python-apps/<name>	8000	systemctl enable <name>
Go	systemd	/opt/go-apps/<name>	8080	systemctl enable <name>
PHP	php-fpm + nginx	/var/www/<name>	80/443	systemctl enable php<v>-fpm
Docker	docker compose	/opt/docker-apps/<name>	as mapped	restart: unless-stopped

Standard Directories

/var/www/<name>/              ← Static, Node.js, PHP
/opt/java-apps/<name>/        ← Java JARs
/opt/python-apps/<name>/      ← Python venvs + apps
/opt/docker-apps/<name>/      ← Docker Compose projects
/etc/nginx/sites-available/   ← Nginx vhost configs
/etc/nginx/sites-enabled/     ← Active vhosts (symlinks)
/etc/systemd/system/          ← Systemd units
/var/log/apps/<name>/         ← App logs
/opt/server-tools/            ← Management scripts + index
/var/backups/                 ← Backups

---

Step 5: Domain + SSL

# Issue cert (DNS must point to server IP first)
ssh -i <key> user@host "certbot --nginx \
  -d <domain> -d www.<domain> \
  --non-interactive --agree-tos \
  --email <admin-email> --redirect"

# Test auto-renewal
ssh -i <key> user@host 'certbot renew --dry-run'

# List all certs + expiry
ssh -i <key> user@host 'certbot certificates'

# Force renew
ssh -i <key> user@host 'certbot renew --cert-name <domain> --force-renewal'

---

Step 6: Database Operations

See deploy-guide.md → Database Setup section.

# MySQL: create DB + user
ssh -i <key> user@host "mysql -u root -p<pass> -e \"
  CREATE DATABASE IF NOT EXISTS <dbname> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
  CREATE USER IF NOT EXISTS '<dbuser>'@'localhost' IDENTIFIED BY '<password>';
  GRANT ALL PRIVILEGES ON <dbname>.* TO '<dbuser>'@'localhost'; FLUSH PRIVILEGES;\""

# PostgreSQL: create DB + user
ssh -i <key> user@host "sudo -u postgres psql -c \"
  CREATE USER <dbuser> WITH ENCRYPTED PASSWORD '<password>';
  CREATE DATABASE <dbname> OWNER <dbuser>;\""

# Redis: check status
ssh -i <key> user@host 'redis-cli ping'

# List MySQL databases
ssh -i <key> user@host 'mysql -u root -p<pass> -e "SHOW DATABASES;"'

---

Step 7: Docker Management

See deploy-guide.md → Docker section for full workflows.

# List running containers
ssh -i <key> user@host 'docker ps'

# Compose project status
ssh -i <key> user@host "cd /opt/docker-apps/<name> && docker compose ps"

# Pull + restart (update)
ssh -i <key> user@host "cd /opt/docker-apps/<name> && docker compose pull && docker compose up -d"

# View logs
ssh -i <key> user@host "docker compose -f /opt/docker-apps/<name>/docker-compose.yml logs -f --tail=100"

# Cleanup
ssh -i <key> user@host 'docker system prune -f'

---

Step 7.5: Service Control (Status / Start / Stop / Restart)

Use service-control.sh — it auto-detects the service type (PM2 / systemd / Docker Compose / Nginx) and runs the right command.

# ── Status ──────────────────────────────────────────────────────────────────
# All services at a glance (status + boot auto-start flag)
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh status'

# One specific service
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh status <name>'

# ── Start / Stop / Restart / Reload ─────────────────────────────────────────
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh start   <name>'
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh stop    <name>'
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh restart <name>'
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh reload  <name>'  # zero-downtime

# ── Auto-Start on Boot ───────────────────────────────────────────────────────
# Check all services have boot auto-start configured
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh boot-check'
# Fix all: enable auto-start for every detected service
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh boot-fix'
# Enable one service
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh enable <name>'

# ── Logs ────────────────────────────────────────────────────────────────────
ssh -i <key> user@host 'bash /opt/server-tools/service-control.sh logs <name> 200'

How reload vs restart works per type:

Type	reload	restart
PM2 (Node.js)	Graceful cluster reload — 0 downtime	Hard restart, brief gap
systemd (Java/Python/Go)	SIGHUP (if app supports it), else falls back to restart	Full process restart
Nginx	Config reload — 0 downtime	Full restart
Docker Compose	Falls back to restart	Recreate containers

Install service-control.sh on server (first time):

scp -i <key> ~/.cursor/skills/linux-server-ops/scripts/service-control.sh user@host:/tmp/
ssh -i <key> user@host 'sudo mv /tmp/service-control.sh /opt/server-tools/ && sudo chmod +x /opt/server-tools/service-control.sh'

---

Step 8: Monitor & Maintain

See monitoring-guide.md for full setup.

# Quick health snapshot
ssh -i <key> user@host 'bash /opt/server-tools/service-registry.sh health'

# System resources
ssh -i <key> user@host 'uptime && free -h && df -h /'

# All running services
ssh -i <key> user@host 'pm2 status 2>/dev/null; systemctl list-units --type=service --state=running --no-pager | tail -20'

# Recent errors
ssh -i <key> user@host 'journalctl -p err --since "1h ago" --no-pager -n 30'

---

Common Operations

Task	Command Pattern
Locate a service	Read snapshot or: ssh ... 'jq .services /etc/server-index.json'
Restart Node app	ssh ... 'pm2 reload <name>'
Restart systemd app	ssh ... 'systemctl restart <name>'
Update deployed app	ssh ... 'cd <root> && git pull && <restart>'
Check Nginx config	ssh ... 'nginx -t'
Renew all SSL	ssh ... 'certbot renew'
View app logs	ssh ... 'tail -f /var/log/apps/<name>/app.log'
Backup now	ssh ... 'bash /opt/server-tools/backup.sh'
Sync local context	bash ~/.cursor/skills/linux-server-ops/scripts/sync-context.sh <server-id>

---

WAF & Firewall

See waf-guide.md for full details.

# Quick WAF status check
ssh -i <key> user@host 'bash -s' << 'EOF'
echo "Nginx: $(systemctl is-active nginx)"
echo "fail2ban: $(systemctl is-active fail2ban)"
echo "UFW: $(ufw status 2>/dev/null | head -1)"
echo "ModSecurity: $(nginx -T 2>/dev/null | grep -c modsecurity) directives"
echo "Banned IPs: $(fail2ban-client status 2>/dev/null | grep -c "Currently banned" || echo 0)"
echo "Top attackers (last 500 reqs):"
tail -500 /var/log/nginx/access.log 2>/dev/null | awk '$9~/^[45]/{print $1}' | sort | uniq -c | sort -rn | head -5
EOF

# Block attacker IP immediately
ssh -i <key> user@host "bash /opt/server-tools/ip-manager.sh block <IP>"

# Show attack summary
ssh -i <key> user@host 'bash /opt/server-tools/attack-monitor.sh'

Key WAF layers: Nginx rate limiting → Security headers → ModSecurity + OWASP CRS → fail2ban jails → UFW/firewalld → iptables

---

Log Management

See log-guide.md for full details.

# Real-time error stream (all services)
ssh -i <key> user@host 'bash /opt/server-tools/error-summary.sh'

# Tail specific service log
ssh -i <key> user@host 'bash /opt/server-tools/show-logs.sh <name> 100 ERROR'

# Nginx top IPs / status codes
ssh -i <key> user@host "awk '{print \$1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10"
ssh -i <key> user@host "awk '{print \$9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn"

# Check disk used by logs
ssh -i <key> user@host 'du -sh /var/log/nginx/ /var/log/apps/ && journalctl --disk-usage'

---

File Operations

See file-ops-guide.md for full details.

# Browse directory
ssh -i <key> user@host 'ls -lah /var/www/<name>/'
ssh -i <key> user@host 'du -sh /var/www/* | sort -h'

# Upload files
rsync -avz --progress -e "ssh -i <key> -p <port>" ./dist/ user@host:/var/www/<name>/

# Download files
rsync -avz -e "ssh -i <key> -p <port>" user@host:/var/log/apps/<name>/ ./local-logs/

# Edit file on server
ssh -i <key> user@host 'nano /var/www/<name>/.env'

# Fix permissions (web standard)
ssh -i <key> user@host "find /var/www/<name> -type d -exec chmod 755 {} \; && find /var/www/<name> -type f -exec chmod 644 {} \; && chmod 600 /var/www/<name>/.env"

# Search in files
ssh -i <key> user@host 'grep -r "ERROR" /var/log/apps/<name>/ | tail -20'

---

User Management

See user-management-guide.md for full details.

# List all non-system users
ssh -i <key> user@host "awk -F: '\$3 >= 1000 && \$3 < 65534 {print \$1, \$3, \$6}' /etc/passwd"

# Create deploy user with SSH key
ssh -i <key> user@host 'bash -s' << 'EOF'
useradd -m -s /bin/bash deploy
mkdir -p /home/deploy/.ssh && chmod 700 /home/deploy/.ssh
echo "<public-key>" > /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh
EOF

# Add SSH key for existing user
ssh -i <key> user@host "echo '<new-public-key>' >> /home/<username>/.ssh/authorized_keys"

# Create SFTP-only user for a website
ssh -i <key> user@host 'bash -s' << 'EOF'
# See user-management-guide.md → SFTP Accounts section
EOF

# Check who is logged in
ssh -i <key> user@host 'w && last | head -10'

---

Scheduled Tasks (Cron)

# View all cron jobs on server
ssh -i <key> user@host 'bash /opt/server-tools/service-registry.sh cron'

# Add a cron job
ssh -i <key> user@host "echo '0 3 * * * root bash /opt/server-tools/backup.sh >> /var/log/backup.log 2>&1' > /etc/cron.d/daily-backup"

# Add to user crontab
ssh -i <key> user@host 'crontab -e'
# Non-interactive:
ssh -i <key> user@host '(crontab -l 2>/dev/null; echo "*/5 * * * * bash /opt/server-tools/watchdog.sh") | crontab -'

# Remove a cron job
ssh -i <key> user@host 'rm /etc/cron.d/<job-name>'

# Test a cron script manually
ssh -i <key> user@host 'bash /opt/server-tools/backup.sh && cat /var/log/backup.log | tail -20'

# Systemd timer (modern alternative to cron)
ssh -i <key> user@host 'systemctl list-timers --no-pager'

---

Network & Bandwidth

# Open ports + which process
ssh -i <key> user@host 'ss -tlnp'

# Active connections count
ssh -i <key> user@host 'ss -s && ss -tn state established | wc -l'

# Bandwidth usage (if vnstat installed)
ssh -i <key> user@host 'vnstat 2>/dev/null || apt-get install -y vnstat && vnstat'

# Live bandwidth (nload)
ssh -i <key> user@host 'nload -u M 2>/dev/null || (apt-get install -y nload && nload -u M)'

# Network interfaces + IPs
ssh -i <key> user@host 'ip addr show && ip route show'

# DNS resolution test
ssh -i <key> user@host 'dig +short <domain> && curl -I --max-time 5 https://<domain>'

---

Security Checklist

See security-guide.md and waf-guide.md for full details.

• SSH: key-only auth, no root login, sshd -t && systemctl reload sshd
• Firewall: only 22/80/443 open — ufw status or firewall-cmd --list-all
• fail2ban: sshd + nginx jails enabled
• ModSecurity + OWASP CRS installed and active
• Nginx rate limiting configured
• Security headers in /etc/nginx/conf.d/security.conf
• .env files: chmod 600, never in git
• /etc/server-index.json: chmod 600 && chown root:root
• Regular updates: unattended-upgrades or dnf-automatic enabled
• SFTP chroot configured for web-only users
• attack-monitor.sh and ip-manager.sh deployed to /opt/server-tools/

---

Additional Resources

Guide	Contents
distro-guide.md	Per-distro package commands, Nginx paths, PHP-FPM sockets
deploy-guide.md	Static, Node.js, Java, Python, PHP, Docker, Database deployments
waf-guide.md	ModSecurity, OWASP CRS, rate limiting, IP blocking, DDoS mitigation
monitoring-guide.md	Metrics, alerting, Netdata, uptime monitoring, watchdog
log-guide.md	Log viewing, searching, rotation, GoAccess, Loki/Promtail
file-ops-guide.md	File browse, edit, permissions, compress, transfer, search
security-guide.md	SSH hardening, auditd, intrusion detection, kernel tuning, backups
user-management-guide.md	Users, sudo, SSH keys, SFTP chroot, session auditing
scripts/check-system.sh	Bootstrap a new server from scratch
scripts/generate-index.sh	Full server scan → /etc/server-index.json
scripts/service-registry.sh	Manage server index + health/db/docker/cron commands
scripts/service-control.sh	Unified start/stop/restart/reload/enable/boot-check for all service types
scripts/sync-context.sh	Sync server state to local .server/snapshots/