Roamtty

← All posts

How mosh works: the client, the server and the protocols between them

mosh (mobile shell) is a remote terminal released in 2012 by Keith Winstein and colleagues at MIT. Its key difference from SSH fits in one sentence: SSH carries a byte stream, mosh synchronizes screen state. The server keeps the real screen, the client holds a copy, and the two keep bringing the copy up to date. Switching networks, locking the phone or losing packets only delays that; the connection itself never breaks.

This post starts with the overall shape, then takes apart the login, the datagram layer, state synchronization and echo prediction in turn. Details were checked against the mosh 1.4.0 source and the paper Mosh: An Interactive Remote Shell for Mobile Clients.

The big picture: two programs, two channels

Phone / laptop mosh-client a copy of the screen local echo prediction heartbeat · resend · roam Your terminal keyboard in · screen out Server mosh-server the real screen state built-in terminal emulator UDP port from 60001 shell / tmux / vim running in a pty keystrokes · window size encrypted UDP (SSP) screen diffs SSH: one login at the start Phone / laptop mosh-client a copy of the screen local echo prediction heartbeat · resend · roam Terminal keys · screen Server mosh-server the real screen state built-in emulator UDP port from 60001 shell tmux · vim in a pty ↓ keystrokes · window size ↑ screen diffs encrypted UDP (SSP) SSH: one login at the start
Figure 1: the two ends of mosh. SSH only opens the door; all terminal traffic after that is encrypted UDP.

mosh is two programs:

  • mosh-server runs on the server. It starts your shell in a pseudo-terminal (pty) and feeds the output into its own terminal emulator, so it always knows what the screen should look like right now.
  • mosh-client runs on your phone or laptop. It sends keystrokes to the server, applies screen changes to its copy, and draws that copy in your terminal.

The mosh user@server you type is actually a Perl script. It logs in with SSH to start mosh-server, then replaces itself with mosh-client. So the server needs mosh installed, but no root access and no daemon.

Step one: log in over SSH, get a port and a key

mosh (client) sshd mosh-server ① ssh user@server TCP 22, a normal SSH login ② run mosh-server new bind a UDP port random 128-bit key ③ MOSH CONNECT 60001 4NeCCgvZFe2RnPgrcU1PQw sent back inside the SSH channel ④ SSH exits, mosh-server detaches ⑤ first encrypted UDP packet (with that key) ⑥ screen state… UDP only from here on client sshd mosh-server ① ssh login TCP 22 ② mosh-server new bind a UDP port make a random key ③ MOSH CONNECT 60001 4NeCCgvZFe2RnPgrcU1PQw sent back over SSH ④ SSH exits, mosh-server detaches ⑤ first encrypted UDP packet ⑥ screen state over UDP
Figure 2: startup. The port and key ride along on SSH, and authentication is entirely SSH's job.

The script runs roughly:

ssh -t user@server -- mosh-server new -s -l LANG=en_US.UTF-8

mosh-server looks for a free UDP port starting at 60001 (range 60000–61000), generates a random AES-128 key, prints this line to standard output and detaches into the background:

MOSH CONNECT 60001 4NeCCgvZFe2RnPgrcU1PQw
[mosh-server detached, pid = 12345]

The 22 characters are the 128-bit key in base64. The client reads the line from SSH's output, SSH exits, and the script runs:

MOSH_KEY=4NeCCgvZFe2RnPgrcU1PQw mosh-client 203.0.113.5 60001

The benefit: passwords, public keys and two-factor prompts are all handled by mature SSH, so mosh has no login code of its own; the key travels once, inside the encrypted SSH channel, and is different for every session.

Note that mosh-server does not know where the client is yet. It just waits on its port for the first UDP packet that was encrypted with this key and passes authentication.

The protocol layers

Once SSH has exited, only UDP is left between the two ends. mosh stacks a few layers on top, each with one job:

What is synced User input (client → server) every keystroke must arrive Screen state (server → client) frames in between can be skipped Transport: SSP Instruction { old_num, new_num, ack_num, throwaway_num, diff } "take your state old_num, apply diff, and you have state new_num" Compress, split protobuf → zlib → small fragments that avoid IP fragmentation Datagram layer AES-128-OCB encryption · sequence numbers · timestamps for RTT follows the source address of the newest valid packet: roaming UDP / IP User input client → server every key must arrive Screen state server → client may skip frames Transport: SSP old_num · new_num · ack_num throwaway_num · diff state old_num + diff = state new_num Compress and split protobuf → zlib → small fragments Datagram layer AES-128-OCB · sequence · timestamps follows the newest valid packet (roaming) UDP / IP
Figure 3: the mosh protocol stack. There is no TCP; SSP provides reliability itself.

From the bottom up: the datagram layer makes sure a packet came from the peer and was not altered, and keeps track of where the peer is; the fragment layer splits large updates; SSP makes the two ends converge; on top are the two things actually being synchronized. One at a time:

The datagram layer: encryption, sequence numbers, RTT and roaming

What is inside one UDP packet

plaintext encrypted (AES-128-OCB) authentication nonce 8 bytes time 2 bytes reply 2 bytes frag header 10 bytes fragment data compressed Instruction tag 16 bytes top bit: direction 63 bits: sequence send time (ms) peer's stamp + time held instruction id index · final bit one instruction may span several fragments flip any bit and the check fails The tag is checked first: a packet not sealed with the session key is dropped without reply. encrypted (AES-128-OCB) plaintext / tag nonce 8 bytes direction bit + sequence timestamp 2 bytes send time (ms) reply stamp 2 bytes peer's stamp + time held frag header 10 bytes id · index · final bit fragment data variable compressed Instruction slice tag 16 bytes flip a bit, check fails The tag is checked first; packets that fail are dropped silently.
Figure 4: the mosh datagram. Everything after the first 8 bytes is encrypted.
  • Nonce: OCB needs a nonce that never repeats. mosh uses a direction bit plus an incrementing sequence number: the top bit is 0 for client packets and 1 for server packets, each direction counts from 0, so the two never collide. The sequence number also tells the receiver which packet is newer.
  • Timestamps: like TCP's timestamp option. A writes its millisecond clock into each packet; B echoes it back in the reply field, adding how long it held on to it. A subtracts and gets the round-trip time (RTT), then computes the smoothed RTT and retransmission timeout with TCP's formulas. The send rate and echo prediction below both depend on this number.
  • Authentication tag: OCB encrypts and authenticates in one pass. Port scans, forged packets and modified packets fail the tag check and are dropped.

Roaming: the server trusts the newest valid packet

Phone mosh-client Server mosh-server Wi-Fi public IP 203.0.113.7 old path: gone once you leave the Wi-Fi Cellular public IP 198.51.100.24 seq 1042 authentic + highest sequence → reply to the new address Phone mosh-client Server mosh-server Wi-Fi public 203.0.113.7 old path: gone Cellular public 198.51.100.24 seq 1042 authentic + highest sequence → reply to the new address
Figure 5: roaming. The server keeps no connection, only the address of the latest valid new packet.

A TCP connection is identified by a four-tuple (both IPs and ports); when the phone's IP changes, the connection is dead. mosh-server has no such binding: whenever it receives a packet that authenticates and has a higher sequence number than any before, it sends its replies to that packet's source address. Move from Wi-Fi to 4G, and the next packet comes from the new address; the server checks the tag and follows. You notice nothing.

An attacker without the key cannot forge a packet that authenticates. Replaying a captured packet does not help either: its sequence number is not the newest, so the server does not move, and SSP updates are idempotent, so a duplicate does no harm.

So that roaming takes effect promptly, the client sends a heartbeat every 3 seconds even when idle, which also keeps NAT mappings alive. If the client hears nothing from the server for 10 seconds, it switches to a new local UDP port and keeps sending, in case a router has quietly dropped the NAT mapping and replies to the old port will never arrive.

The State Synchronization Protocol: only chase the latest screen

Why a TCP byte stream is the wrong fit

SSH over TCP: a byte stream, every byte in order Client Server 1 2 3 4 5 6 7 8 … cat a big log: megabytes are already queued in buffers, so Ctrl-C waits until they all arrive mosh: the server moves the client's state straight to the latest Client holds #1 screens on the server, a new state per output #1 #2 #3 #4 #5 #2–#4 never sent diff: #1 → #5 SSH over TCP: every byte, in order Client Server 1 2 3 4 … cat a big log: megabytes queue up, Ctrl-C waits until they all arrive mosh: jump straight to the latest #1 #2 #3 #4 #5 #2–#4 never sent screens on the server, one per output diff: #1 → #5 Client holds #1, becomes #5
Figure 6: a byte stream must deliver everything in order; state sync only cares which version the peer has and which is newest.

What a terminal user cares about is what is on the screen now, not every frame in between. TCP cannot know that: it delivers every byte in order, so one lost packet stalls everything behind it, and on a slow link the old output piled up in the send buffer keeps Ctrl-C from taking effect.

SSP (State Synchronization Protocol) takes another approach: each side numbers its states and only ever sends the diff from the version the peer has to the newest one. On a slow network it sends less often, with bigger jumps; when a packet is lost, the next diff simply includes what was missed, with no need to resend old packets one by one.

What an instruction carries

SSP passes Instruction messages (protobuf) back and forth:

FieldMeaning
old_numThe base of the diff: "apply this to your state number …"
new_numThe state number after applying it
diffThe change itself. When empty, the instruction is just a heartbeat or an ack
ack_numPiggybacked ack: "of your states, I have received up to this number"
throwaway_num"I will never again use a state older than this as a base; you can drop them"
protocol_versionProtocol version, currently 2; mismatched ends refuse to talk
chaffRandom padding, so packet lengths leak less about what you type

On receiving an instruction, the receiver checks whether it has state old_num. If it does, it applies diff to get new_num; if not (say the packet with that base was lost), it ignores the instruction and waits for a resend. To speed this up, the sender optimistically assumes recently sent states have arrived and diffs against them; if it guessed wrong, it falls back to the last acknowledged state after a timeout. throwaway_num lets the receiver delete old states safely, so memory does not grow.

Pacing

SSP does not send a packet on every change; it sets a "frame rate" to suit the network:

  • Sends are at least half an RTT apart, clamped to 20–250 ms. On a good network that is at most about 50 frames per second; on a bad one it slows down rather than filling the link with stale frames.
  • When the screen changes, the server waits a few milliseconds to collect output that follows immediately, so one ls does not become several packets.
  • Acks are delayed up to 100 ms and ride on data packets when possible.
  • When nothing happens, a heartbeat goes out every 3 seconds so each side knows the other is still there.

Two directions, two kinds of state

SSP itself is generic; the layer above decides what is synchronized. In mosh the two directions carry different things:

DirectionWhat is syncedWhat the diff containsCan states be skipped?
Client → serverThe user input streamKeystrokes (Keystroke) and window size changes (ResizeMessage) since the baseNo, every keystroke must arrive as typed
Server → clientThe whole screen: text, colours, cursor, titleTerminal escape sequences that redraw the old screen into the new one (HostBytes), plus window size (ResizeMessage) and the echo ack (EchoAck)Yes, as long as the last version is right

The server-side diff is neat: it compares the client's version of the screen with the newest one and produces a short run of escape sequences, such as "move the cursor to row 3, column 10 and write done". The client feeds that to its own terminal emulator, and its copy matches the server's.

Local echo prediction: typing that keeps up on a slow link

In SSH, every character you type appears only after the server echoes it back. With a 300 ms round trip, typing feels like wading through glue. The mosh client guesses instead: most of the time, a normal character typed at a prompt appears at the cursor and the cursor moves one cell right. So the client draws it without waiting for the server.

① you type l, s $ ls no echo yet, drawn anyway underlined on slow links: "a guess" ② the server confirms $ ls ls shows up on the server's screen underline removed, guess kept ③ wrong guess, e.g. a password Password: keys acked, but nothing on screen guess withdrawn, server wins Decided by EchoAck: about 50 ms after input arrives, its echo should be on the server's screen. $ ls ① you type l, s no echo yet, drawn anyway underlined on slow links $ ls ② server confirms server shows ls too underline goes, guess kept Password: ③ wrong guess acked, but not on screen withdrawn; server wins Decided by EchoAck: about 50 ms after input arrives, its echo should be visible
Figure 7: echo prediction. A right guess makes typing feel local; a wrong one flickers briefly, and the final screen is always the server's.

How does it know whether it guessed right? Through the server's EchoAck: about 50 ms after receiving a given input state, the server tells the client "everything up to input number N has had its echo reflected on the screen". With that, the client checks the server's screen: if the predicted characters really appeared, the prediction stands; if not (a password prompt does not echo, for instance), it is withdrawn and prediction pauses for that stretch of input.

A few details keep it from getting in the way:

  • The default --predict=adaptive only shows predictions when the RTT is high, so on a LAN you never see them; at a higher RTT predicted characters are underlined. always and never force it on or off.
  • Keys whose effect is hard to guess, such as Enter, Esc and Ctrl combinations, start a new epoch. Predictions in a new epoch stay hidden until the server has confirmed one guess, so jumping around in vim's normal mode does not fill the screen with stray letters.
  • Predictions are an overlay on top of the copy and are never written into it. The server's screen is always the final answer.

What this design costs

Synchronizing the screen instead of a byte stream makes mosh resistant to disconnects and packet loss, but it also differs from SSH in a few ways:

What you seeWhy
No scrollback; scrolling up does not show old outputOnly the current screen is synced, and skipped frames never reach the client. Run tmux inside for history
No port forwarding, X11 forwarding or ssh-agent forwardingSSH exits after startup, and the mosh protocol only carries a terminal
Firewalls must allow UDP 60000–61000Terminal traffic is UDP; a server with only TCP 22 open cannot be reached
mosh-server waits forever by default after the client disappearsThe server cannot tell "client gone" from "phone asleep". Set a limit with MOSH_SERVER_NETWORK_TMOUT
Both ends need a UTF-8 localemosh's terminal emulator decodes UTF-8; without the locale the server says needs a UTF-8 native locale
"mosh: Last contact … ago" at the top after a long silenceHeartbeats stopped and the client is telling you the network has a problem; it goes away when packets flow again, and Ctrl-^ . quits

For server setup, common errors and pairing with tmux, see SSH on your phone without disconnects: a mosh and tmux guide.

mosh in Roamtty

Roamtty implements the client side of the mosh protocol (version 2) in Kotlin: it also runs mosh-server new over SSH and reads MOSH CONNECT, uses the same AES-128-OCB datagrams, 3-second heartbeats and 10-second port hops, and hands the HostBytes it receives straight to the app's own terminal emulator. The server runs the official mosh-server, with nothing extra to install.

Roamtty does not do local echo prediction for now (it ignores EchoAck). In tmux mode mosh-server attaches to the tmux session as soon as it starts, and tmux provides the scrollback. If no UDP reply arrives within 8 seconds it falls back to SSH; see the Roamtty section of the guide for details.

References