TLDR: Use PsExec and run “cmdkey /list” on affected systems. Occurs from a combination of saved credentials, DFS/domain accessing when installing software.
This was a case of a very odd account lockout that lead me to a cause I’ve never seen from Windows. Normally account lockouts are fairly simple to resolve, usually it is a mobile device with an outdated password. Most of the time checking the mobile device resolves lockouts.
My tech changed their password and was bombarded by lockout issues. After a few unlocks and having them check their mobile devices, I realized it was time to go in-depth into tracking the source.
Microsoft has an Account Lockout Tools that help with this. More specifically I typically use LockoutStatus.exe to pull information from my DCs.
The first column I look at is Orig Lock, which tells me which DC issued the lock. Then I look at Bad Pwd Count, which tells me if any other DCs received a bad password attempt. Finally Last Bad Pwd lets me know when it last received a bad attempt, which is helpful as I’m navigating logs.
Then onto each DC, which I use remote Event Viewer on. I have a Custom view that looks at the Security log and filters for event IDs “4625, 4740, 4771, 4776”. I also filter it to the last hour to limit the amount of results I need to look through.
When looking at Event Viewer, the lockouts are 4740 under the “User Account Management” category. Easy to spot. But these don’t always paint a good picture as I need to see all systems that issued the bad password. The Last Bad Pwd from the lockout tool helps here and gives me a starting point to work back from. Someday I shall build a powershell script to automate this.
In this case, the bad password attempts where occurring from dozens of endpoint devices. This seemed odd. I first checked that they where not logged into these devices. I do this using powershell script to query multiple places and also checking to see if loginui.exe exists for that user (indicating they are at the login screen). In my case, my tech was not logged into any of the machines. I even purged the techs account from the system and rebooted for good measure.
Luckily one of the machines was not in use, so I stopped looking around with powershell and logged into the system. My next step was to check Services on a machine, and I could not locate any running under my tech. Checked the task scheduler and couldn’t locate any tasks running under the account either. Finally checked to see if any persistent drives where attached not supplied by GPOs. Atlas nothing was found.
This had me at a loss. Far as I could think of was using their account. Now into to trusty Event Viewer, I saw that Group Policy was failing to update on the system. This struck me as odd and for sake of debugging, I rebooted the system. This brought up the same interesting data in the event viewer again, the group policy yet again failed to update. I looked more closely at the event viewer and discovered that it was failing because of bad credentials.
So digging more into event viewer, I found an interesting error, which was logged onto the system. This provided me the critical data I needed to know to start looking in the right direction.
The error indicates that it failed to log in using the affected account. I know the computer AD account should be the one doing the group policy updates. But it was right there indicating it was using a user account.
This was completely new to me. But it was using a saved user credentials. Still found it trying to use that account during gpupdate. It was indicating a potential for credential manager. Pulling it up as a admin user didn’t reveal anything useful. But what about the system? I didn’t think about it, but the system account could be using credential manager.
I was somewhat familiar with cmdkey.exe existing but haven’t really used it. None the less I fired up command prompt, issued a psexec with -s to start a cmd.exe as the system. I then issued a “cmdey /list”. Behold there was a interesting entry in the list.
The affected account right there. I issued a “cmdkey /delete domain:target=contoso.local” and removed the credentials. Then rebooted the system. Checking event logs, there was no errors doing gpupdate and it didn’t trigger any more bad password errors on the techs account.
I checked other systems and all had the same thing. Issuing the deletes resolved the issue. I’ve found the problem. But why? This left me at a loss for why. I had my tech rejoin a test system to the domain and it didn’t show the problem. I started to talk to my admin and mentioning all the systems that where affected. I had been at this a while and missed a key point that my admin pointed out. All these systems where recent installs/reloads.
After looking back at all systems we had deployed recently, sure enough they all where affected by this. My admin suggested the image was the problem. Like all good admins, I have a test system around and luckily its a virtual machine. After adjusting the settings to do a net boot, I booted it up and did a reload of the system.
I didn’t even need to join the domain, just pulled PSexec into the system and ran the checks I had found. Sure enough, the image had the account there. So now I have located the source of why systems had it. Our image had this credential.
This is now playing on a theory, I suspect since we use DFS and access the DFS via the domain name, that my tech hit our software repository to access installers. It first prompted them in the standard user context to access the shares. When the installer ran, it needs admin level access to do that. So it would prompt again for credentials, as only the current session user has access to the share, not the system. My tech then entered their credentials again. At this point the tech does not believe they told it to save their credentials. But they are not sure if the box was checked or not. I believe that windows defaulted to checking that box.
This then saved the credentials using the domain name as the “server” name would be when saving traditional credentials. When they finished installing all software, they generated the image with a sysprep and tested it out. As they had not changed their password, things worked as expected.
Once they changed their password, suddenly all of the systems started to fail gpupdate and kept trying every 90 minutes. Because the systems where accessing the domain to try and find a dc that can present them the netlogon and sysvol shares needed.
My tech has been setting up images this way for a long time, so that leaves me to believe something has changed in Windows 10 1909 that is causing this. It also leaves me to believe that sysprep is not clearing credentials like I would expect from the system account.
I’m unclear as to why Windows decides to use the stored credentials in cmdkey when accessing the domain shares. It doesn’t store the computer account credentials there but it does seem to decide to use those first and doesn’t try to go back to any others.
To put this into steps
- Admin installs fresh Windows 10 1909
- Admin accesses DFS/domain shares and is prompted for credentials
- Admin inputs credentials
- Admin navigates shares to install software.
- Windows prompts for credentials again to install software
- Windows saves these credentials
- Admin finishes prepping system and issues a sysprep
- Image is made and deployed to systems
- Admin changes their domain password
- Admin experiences account lockouts and systems fail gpupdate as they can no longer use the saved admin password.
- Use cmdkey as system account and delete saved credentials
- Everything works as expected