
Last Update: November 16, 2026 7 min read
BY
eric
On Monday morning a client told us their staff could not log in to the office server. Remote Desktop just kept failing.
The server was a Windows Server 2019 machine in Google Cloud. It runs the client's file shares and accounting software, and it is the domain controller and the Remote Desktop host for the whole team. From our management console it looked fine: online, checking in, no alarms. The Remote Desktop port was open and answering.
It had rebooted itself more than 100 times since Sunday morning, roughly every 15 minutes. And the thing pressing the restart button was our own RMM agent.
Nothing in this story was broken badly on its own. A disk that was too full, a Windows restart flag that kept coming back, an agent that trusted that flag too much, and a licensing setting that fell out of the registry during the chaos. Each one was small. Together they locked out every user for 29 hours. This is how the storm formed, how we broke it, and what we changed so it can't happen again.
What it looked like from the outside
The first checks all said "healthy":
- Our management console showed the server online, last seen seconds ago. It was: the agent checked in during each 10-minute window between reboots.
- Port 3389 was open, and a raw RDP handshake came back with a normal security negotiation.
- Google Cloud showed the VM running since 29 October. Google had not restarted anything.
The first real clue was the uptime field in the agent's own inventory: 17 minutes. A server that runs the office does not normally have 17 minutes of uptime on a Monday morning.
Google Cloud keeps the VM's serial console output even when you can't log in, and that told the story straight away:
2026-11-16T08:24:00 ... for "shutdown" event
2026-11-16T08:29:55 ... for "startup" event
2026-11-16T08:40:04 ... for "shutdown" event
2026-11-16T08:45:44 ... for "startup" event
...
Up for about ten minutes, a clean shutdown, five minutes to boot, repeat. The log buffer went back to Sunday afternoon and the pattern was already running.
Piece one: a disk at 95%
The C: drive had 14 GB free out of 300 GB. That number had been sitting in the agent's inventory for a while.
People treat 95% full as "getting full, deal with it soon". On a Windows server it is closer to "already failing in ways you can't see yet". Windows Update needs room to stage, unpack and roll back cumulative updates. Logs, profile caches and temp files all grow. When an update can't complete cleanly, Windows does not stop asking for it. It downloads it, tries again, and flags that it wants a restart.
We cannot prove the full disk was the single cause of what came next. The restart flag was transient and we never caught it in the act. But a server that keeps re-downloading an update ten minutes after every boot, on a nearly full disk, is the usual suspect.
Piece two: a restart flag that keeps coming back
Windows records "I need a restart" in a couple of well-known registry keys, under Component Based Servicing and Windows Update. Plenty of tools read these keys to decide whether a machine needs rebooting. Ours does too.
On this server something set one of those flags roughly ten minutes after each boot. That lines up with the Windows Update events in the same window ("found 1 update", "an update was downloaded"). When we checked at 9 minutes, 11 minutes and an hour of uptime, the flags were clear. So the flag was transient: up just long enough to be seen, then gone.
On its own, that's harmless. Windows would have nagged and waited for the Sunday maintenance window like it always does.
Piece three: our own agent took the bait
This is the uncomfortable part. We manage our clients' machines with our own RMM agent. It does inventory, monitoring, patching, remote support and policy enforcement. Its patch feature has a reboot coordinator, and in that version (0.3.9) the decision was essentially:
// a reboot is "needed" if we applied patches, OR the OS reports one pending
if needed && !hasUser {
return rebootNow // no maintenance window, no rate limit
}
The policy for this client approved no patches at all, so the agent applied nothing. But the coordinator also counted "Windows says a restart is pending" as a reason to reboot. When nobody was logged in, it rebooted immediately, at any hour, with no limit on how often.
That closes the loop:
- The server boots. Nobody is logged in yet.
- About ten minutes later, Windows flags a pending restart.
- The agent's policy cycle sees the flag and no active user, and runs
shutdown.exe /rwith the comment "Applying scheduled updates". - The server reboots. Users trying to connect get kicked out or never get in.
- Go back to step 1.
The server-side audit log made it obvious once we looked: 100 entries of "patch reboot ok", and zero patch installs. The agent was rebooting a machine it had not changed, over and over.
It was also invisible in the agent's own log file. The coordinator never wrote a line saying "I'm rebooting because of X". We found it through Windows event ID 1074, which records which process asked for the shutdown and with what comment. "Applying scheduled updates" existed in exactly one place in our codebase.
How we got in without Remote Desktop
With RDP down and the agent's remote command feature not yet enabled on this box, we used Google Cloud's serial console. Windows Server on GCE exposes the Special Administration Console (SAC) on a second serial port, which gives you a command prompt with no network login at all.
Driving it from a script has some sharp edges, worth knowing if you ever need it in a hurry:
- The SAC screen mangles long output, so we had PowerShell write results to the first serial port (COM1) and read them back with
gcloud compute instances get-serial-port-output. - Typing long scripts over the serial line drops characters. Putting the script in the VM's metadata and fetching it from the metadata server inside the guest was far more reliable.
- PowerShell's download progress bar crashes inside SAC, so set
$ProgressPreference = 'SilentlyContinue'before any web request.
Once we were in, the fix for the loop itself was one line: stop the agent and set it to manual start. The next reboot, due at 10:10, never came.
Piece four: the outage that hid behind the outage
With the server finally staying up, a user tried again from Remmina and got "Cannot connect". So did we.
The server's logs showed the login succeeding: network-level authentication passed. The session died a moment later. Reproducing it with xfreerdp3 gave the real reason: the connection was aborted during the RDP licensing step. Connecting with /admin, which skips licensing, worked fine. The user's own error message confirmed it: "The remote session was disconnected because there are no Remote Desktop License Servers available to provide a license."
The client had 50 per-user RDS licences on this same server, with 14 in use. They hadn't gone anywhere. What had gone was the setting telling Remote Desktop to use them:
The local Group Policy file still held the right values, but they were no longer applied. With no licensing mode and the 120-day grace period long gone, Windows refused every non-admin session.
Our best explanation is the reboot storm itself. Every boot logged DNS failures ("none of the configured DNS servers responded") on a machine that is its own domain controller, and Group Policy processing on a half-started DC that gets hard-cycled every 15 minutes is not going to be reliable. Licensing errors started on Sunday evening, well into the loop.
The fix was two commands: gpupdate /target:computer /force to reapply the policy, then a restart of the Remote Desktop service so it picked the setting up. A normal user session connected straight away.
What we fixed in the agent
We cut a new agent release as a hotfix on top of the version the fleet was running, so nothing else rode along with an urgent change:
Reboot safety
Visibility
There is also a regression test that replays this exact incident: an unattended machine, a restart flag that comes back every boot, and 15-minute cycles. The old code rebooted on every cycle. The new code reboots zero times outside the window, once inside it, and then holds.
We also grew the boot disk from 300 GB to 400 GB. On Google Cloud that is a single gcloud compute disks resize with the VM running, followed by extending the C: partition inside Windows. C: went from 14 GB free to 114 GB free.
The timeline
What we're taking away
For anyone running Windows servers
For anyone building automation
The humbling part is that every piece of this was ordinary. Disks fill up. Windows asks for restarts. Agents reboot machines. Group Policy occasionally fails to apply. What turned four ordinary things into a day-long outage was that they lined up, and the one piece we wrote ourselves had no brake on it.
If your monitoring is showing a server at 95% disk right now, that is the piece of the storm you can remove today.
Previous Article
Next Article
Nov 10, 2026








Comments (0)
Leave a Comment