preloader
post-thumb

Last Update: November 16, 2026 7 min read


BYauthor-thumberic

|Loading...

On Monday morning a client told us their staff could not log in to the office server. Remote Desktop just kept failing.

The server was a Windows Server 2019 machine in Google Cloud. It runs the client's file shares and accounting software, and it is the domain controller and the Remote Desktop host for the whole team. From our management console it looked fine: online, checking in, no alarms. The Remote Desktop port was open and answering.

It had rebooted itself more than 100 times since Sunday morning, roughly every 15 minutes. And the thing pressing the restart button was our own RMM agent.

Nothing in this story was broken badly on its own. A disk that was too full, a Windows restart flag that kept coming back, an agent that trusted that flag too much, and a licensing setting that fell out of the registry during the chaos. Each one was small. Together they locked out every user for 29 hours. This is how the storm formed, how we broke it, and what we changed so it can't happen again.

What it looked like from the outside

The first checks all said "healthy":

  • Our management console showed the server online, last seen seconds ago. It was: the agent checked in during each 10-minute window between reboots.
  • Port 3389 was open, and a raw RDP handshake came back with a normal security negotiation.
  • Google Cloud showed the VM running since 29 October. Google had not restarted anything.

The first real clue was the uptime field in the agent's own inventory: 17 minutes. A server that runs the office does not normally have 17 minutes of uptime on a Monday morning.

Google Cloud keeps the VM's serial console output even when you can't log in, and that told the story straight away:

text
2026-11-16T08:24:00 ... for "shutdown" event
2026-11-16T08:29:55 ... for "startup" event
2026-11-16T08:40:04 ... for "shutdown" event
2026-11-16T08:45:44 ... for "startup" event
...

Up for about ten minutes, a clean shutdown, five minutes to boot, repeat. The log buffer went back to Sunday afternoon and the pattern was already running.

Piece one: a disk at 95%

The C: drive had 14 GB free out of 300 GB. That number had been sitting in the agent's inventory for a while.

People treat 95% full as "getting full, deal with it soon". On a Windows server it is closer to "already failing in ways you can't see yet". Windows Update needs room to stage, unpack and roll back cumulative updates. Logs, profile caches and temp files all grow. When an update can't complete cleanly, Windows does not stop asking for it. It downloads it, tries again, and flags that it wants a restart.

We cannot prove the full disk was the single cause of what came next. The restart flag was transient and we never caught it in the act. But a server that keeps re-downloading an update ten minutes after every boot, on a nearly full disk, is the usual suspect.

Piece two: a restart flag that keeps coming back

Windows records "I need a restart" in a couple of well-known registry keys, under Component Based Servicing and Windows Update. Plenty of tools read these keys to decide whether a machine needs rebooting. Ours does too.

On this server something set one of those flags roughly ten minutes after each boot. That lines up with the Windows Update events in the same window ("found 1 update", "an update was downloaded"). When we checked at 9 minutes, 11 minutes and an hour of uptime, the flags were clear. So the flag was transient: up just long enough to be seen, then gone.

On its own, that's harmless. Windows would have nagged and waited for the Sunday maintenance window like it always does.

Piece three: our own agent took the bait

This is the uncomfortable part. We manage our clients' machines with our own RMM agent. It does inventory, monitoring, patching, remote support and policy enforcement. Its patch feature has a reboot coordinator, and in that version (0.3.9) the decision was essentially:

go
// a reboot is "needed" if we applied patches, OR the OS reports one pending
if needed && !hasUser {
    return rebootNow // no maintenance window, no rate limit
}

The policy for this client approved no patches at all, so the agent applied nothing. But the coordinator also counted "Windows says a restart is pending" as a reason to reboot. When nobody was logged in, it rebooted immediately, at any hour, with no limit on how often.

That closes the loop:

  1. The server boots. Nobody is logged in yet.
  2. About ten minutes later, Windows flags a pending restart.
  3. The agent's policy cycle sees the flag and no active user, and runs shutdown.exe /r with the comment "Applying scheduled updates".
  4. The server reboots. Users trying to connect get kicked out or never get in.
  5. Go back to step 1.

The server-side audit log made it obvious once we looked: 100 entries of "patch reboot ok", and zero patch installs. The agent was rebooting a machine it had not changed, over and over.

It was also invisible in the agent's own log file. The coordinator never wrote a line saying "I'm rebooting because of X". We found it through Windows event ID 1074, which records which process asked for the shutdown and with what comment. "Applying scheduled updates" existed in exactly one place in our codebase.

How we got in without Remote Desktop

With RDP down and the agent's remote command feature not yet enabled on this box, we used Google Cloud's serial console. Windows Server on GCE exposes the Special Administration Console (SAC) on a second serial port, which gives you a command prompt with no network login at all.

Driving it from a script has some sharp edges, worth knowing if you ever need it in a hurry:

  • The SAC screen mangles long output, so we had PowerShell write results to the first serial port (COM1) and read them back with gcloud compute instances get-serial-port-output.
  • Typing long scripts over the serial line drops characters. Putting the script in the VM's metadata and fetching it from the metadata server inside the guest was far more reliable.
  • PowerShell's download progress bar crashes inside SAC, so set $ProgressPreference = 'SilentlyContinue' before any web request.

Once we were in, the fix for the loop itself was one line: stop the agent and set it to manual start. The next reboot, due at 10:10, never came.

Piece four: the outage that hid behind the outage

With the server finally staying up, a user tried again from Remmina and got "Cannot connect". So did we.

The server's logs showed the login succeeding: network-level authentication passed. The session died a moment later. Reproducing it with xfreerdp3 gave the real reason: the connection was aborted during the RDP licensing step. Connecting with /admin, which skips licensing, worked fine. The user's own error message confirmed it: "The remote session was disconnected because there are no Remote Desktop License Servers available to provide a license."

The client had 50 per-user RDS licences on this same server, with 14 in use. They hadn't gone anywhere. What had gone was the setting telling Remote Desktop to use them:

Setting
In the Group Policy file
Live in the registry
Licensing mode
Per user (4)
not set
License server
this server
not set
Grace period left
0 days

The local Group Policy file still held the right values, but they were no longer applied. With no licensing mode and the 120-day grace period long gone, Windows refused every non-admin session.

Our best explanation is the reboot storm itself. Every boot logged DNS failures ("none of the configured DNS servers responded") on a machine that is its own domain controller, and Group Policy processing on a half-started DC that gets hard-cycled every 15 minutes is not going to be reliable. Licensing errors started on Sunday evening, well into the loop.

The fix was two commands: gpupdate /target:computer /force to reapply the policy, then a restart of the Remote Desktop service so it picked the setting up. A normal user session connected straight away.

What we fixed in the agent

We cut a new agent release as a hotfix on top of the version the fleet was running, so nothing else rode along with an urgent change:

Reboot safety

Respect the maintenance windowIf only Windows wants a restart (the agent changed nothing), wait for the patch window instead of rebooting now.
Reboot-loop guardNever initiate a second reboot within 6 hours of the last one. The time is saved to disk before the reboot fires, so it survives the restart.

Visibility

Log every decisionEvery reboot, deferral and suppression is logged with its reason.
Report, do not hideHeld or suppressed reboots now show as drift on the server, so a stuck machine is visible instead of quietly looping.

There is also a regression test that replays this exact incident: an unattended machine, a restart flag that comes back every boot, and 15-minute cycles. The old code rebooted on every cycle. The new code reboots zero times outside the window, once inside it, and then holds.

We also grew the boot disk from 300 GB to 400 GB. On Google Cloud that is a single gcloud compute disks resize with the VM running, followed by extending the C: partition inside Windows. C: went from 14 GB free to 114 GB free.

The timeline

When (AEST)
What happened
Sun 05:08
First agent-initiated reboot. The loop begins.
Sun 21:04
First Remote Desktop licensing error.
Mon 08:49
A manual reset from the cloud console. The loop carries on.
Mon 09:20
Serial log shows a reboot every ~15 minutes.
Mon 09:50
Event 1074 points at the RMM agent.
Mon 10:08
Agent stopped. The loop is broken.
Mon 10:36
Licensing failure reproduced. Admin sessions work, user sessions do not.
Mon 10:56
Group Policy reapplied. Users connect.
Mon 11:30
Agent 0.3.13 installed and back online, disk grown to 400 GB.

What we're taking away

For anyone running Windows servers

95% disk is an outage in waitingGive low disk space an owner and a deadline. It is the cheapest piece of this storm to remove.
"Online" is not "healthy"A machine that checks in every 15 minutes between reboots looks online. Watch uptime and boot counts too.
One symptom, two causesAfter fixing the obvious problem, reproduce what the user actually does before calling it done.
Reboot storms break other thingsAfter any loop of hard restarts, check Group Policy results, licensing and disk health.

For anyone building automation

Rate-limit every power actionAnything that can reboot a production machine needs a window, a minimum interval and a durable record written first.
OS state is a hint, not an orderAct at once only on your own intent. Things the OS reports on its own should wait for policy.
Log the whyIf your agent cannot tell you why it rebooted a machine, you will be reading Windows event logs at 9am on a Monday.
Alert on flapping100 reboots with zero patches is a pattern the server could have flagged on Sunday morning.

The humbling part is that every piece of this was ordinary. Disks fill up. Windows asks for restarts. Agents reboot machines. Group Policy occasionally fails to apply. What turned four ordinary things into a day-long outage was that they lined up, and the one piece we wrote ourselves had no brake on it.

If your monitoring is showing a server at 95% disk right now, that is the piece of the storm you can remove today.

Comments (0)

Leave a Comment
Your email won't be published. We'll only use it to notify you of replies to your comment.
Loading comments...

Keywords

  • windows server reboot loop
  • rmm agent
  • pending reboot
  • disk space alert
  • remote desktop licensing
  • rds cal
  • group policy
  • incident response
  • windows update
Previous Article
post-thumb

Oct 03, 2021

Setting up Ingress for a Web Service in a Kubernetes Cluster with NGINX Ingress Controller

A simple tutorial that helps configure ingress for a web service inside a kubernetes cluster using NGINX Ingress Controller

Next Article
post-thumb

Nov 10, 2026

What Profitable ASX Day Traders Have in Common

The closing post of the ASX order book series. After reading several Australian traders' public journals end to end, these are the habits and beliefs the profitable ones shared, the things they did not have in common at all, and an index of every post in the series.

agico

We transform visions into reality. We specializes in crafting digital experiences that captivate, engage, and innovate. With a fusion of creativity and expertise, we bring your ideas to life, one pixel at a time. Let's build the future together.

Copyright ©  2026  TYO Lab · v0.0.38