I remember when mosh first came out and thinking "that sounds like a great idea, maybe I'll set it up sometime" and then never getting around to it. Several times over the years I've thought similarly, but it never seemed worth it. A week ago a coworker asked if there were good tools for handling remote terminal use over laggy internet connections, and I suggested mosh. They really liked it, which then had me feeling silly for never trying it, and got me to actually do it. So: I'm typing this over mosh, and it's excellent! I should have done this years ago.
With a traditional remote shell like ssh you are sending characters back and forth and the state of your terminal is maintained client-side. This means that when you press a 'q' you don't see it appear on the screen until a message has gone to the server saying "they pressed 'q'" and a response comes back saying "ok, put a 'q' on the screen". If you have a nice fast connection this happens so quickly you can't even tell, but if you're on flaky wifi it can be very painful.
Mosh turns this around, maintains the terminal on the server, and syncs the state of the terminal from the server to the client as efficiently as it can. Additionally, it echos your typing back as you go, and then overwrites it with the state from the server once it gets it. It even handles cases where you're typing text into the middle of an existing line, sliding the rest of the line forwards to make room. It shows locally echoed text with a little underline so you know it's just a prediction.
Working from my third floor bedroom connected to wifi in the basement, in a house with seven other people who might all be streaming video at once, with a workflow that is built entirely around being ssh'd into a remote server, this is just amazing.
It's funny looking back at the world before, and seeing people being dismissive of these ideas. For example, about six months before mosh came out someone asked on StackOverflow how to avoid the typing delay on SSH. The accepted answer is saying this is impossible and just suggesting that people type without seeing feedback (this probably works a lot better for people who are more accurate typists than I am). Similarly, the HackerNews discussion has several people saying that you can patch something similar together with existing tools.
This does leave me wondering whether mosh could make better predictions. When I hold down the up arrow, it only slowly moves my cursor from line to line, waiting for server round trips. Since mosh has my whole terminal state, it can probably figure out that I'm typing this in emacs from the minibuffer and how previous arrow keys and emacs keybindings were handled. It would be really neat if it could then use those predictions to let me navigate vertically as easily as horizontally. Similarly, when I type Alt-X emacs moves me to the minibuffer: after seeing a few rounds of this it seems like a fancy prediction system for mosh could do this as well? You would need to be careful about not echoing passwords, but I think that's still doable? I'm curious whether anyone has explored this direction, though for now I'm really very happy with its current abilities.