The mailing list for users of the eduroam Configuration Assistant Tool (CAT)

[Vittorio Gambaletta <noc.geantml AT toniolowifi.net>]

Hello everybody,

from what I can see at my deployment (I manage an eduroam Service Provider in Italy), after a quite extensive analysis of problems that were reported by users in the last couple of days, I can say that the main issue with the latest Windows 11 22H2 update does NOT seem to be with Credential Guard, but just with the newly enabled-by-default usage of TLS v1.3 for EAP-TTLS and PEAP.

Windows Defender Credential Guard should create problems only when using local account, AD account or maybe Microsoft account credentials remotely, which should never be the case with eduroam. It should not affect usage of credentials that are specific for one particular 802.1x network, at least for authentication using username/password. Unfortunately I haven't had a chance to try certificate-based authentication (EAP-TLS) on Windows 11 22H2, so I don't know if or how that could be impacted by Credential Guard.

I'd like to share my analysis of the issue with TLS v1.3, and the possible solutions I've found.


Usage of TLS v1.3 in EAP-TTLS and PEAP requires some specific implementation changes, both on the supplicant and on the RADIUS server of the Identity Provider; without those changes, in case TLS v1.3 gets negotiated anyway, the software on the non-updated side would fail to make key derivation computations correctly, so the authentication would always fail due to this.

There is not yet a finalized standard (RFC) for this; it's currently an Internet-Draft, and only very recent versions of some supplicants and RADIUS servers already implement this in a correct way. For example, only the newest FreeRADIUS v3.0.26 (just released some weeks ago) and v3.2.0+ support TLS v1.3 correctly... Please refer to the latest specification draft, at the time of writing https://datatracker.ietf.org/doc/html/draft-ietf-emu-tls-eap-types-09#section-5 for more details about this.


So, essentially, the problem arises when an upgraded supplicant tries to use TLS v1.3 for EAP, and the RADIUS server has it enabled by default (for example after an upgrade to OpenSSL v1.1.1+), but without implementing it correctly.

Since Windows 11 22H2 has now enabled TLS v1.3 by default also for EAP-TTLS and PEAP, the issue started emerging and diffusing quickly as the Windows update started being rolled out to users.

Modern Windows obviously isn't of any help as usual, because it does not show any useful error message to the user. It only asks to insert the credentials again and again; sometimes it stops the connection with just a cryptic message like "Can't connect to this network", without saying anything useful about what is actually happening under the hood. The Windows Event Log is of no help too.

By the way, I've also encountered a similar problem in the past few months on some flavours of Linux too, where OpenSSL got updated but wpa_supplicant did not; TLS v1.3 was again wrongly automatically enabled for EAP, so it was regularly failing when a RADIUS server was also willing to negotiate TLS v1.3 (that at the time was completely wrong too, due to the same problematic default in OpenSSL, and the missing correct implementation both in the supplicant and in the RADIUS server)...


So. The problem can be solved or worked around essentially in three ways:

• by upgrading the RADIUS server used by the Identity Provider (not by the Service Provider, which is merely opaquely proxying the EAP request to the FLR servers!);
  
  
• by disabling usage of TLS v1.3 on older RADIUS servers (again, of the IdP) which do not implement it correctly (if possible at all, because some older versions of RADIUS servers might just use the TLS library defaults, without letting the administrator choose which TLS versions to enable; also for OpenSSL the program actually needs to know about and handle a new C "#define" in order to be able to disable TLS v1.3... I'm not sure if this can be somehow worked around by leaving only TLS v1.2- ciphersuites enabled);
  
  
• by disabling usage of TLS v1.3 on the supplicant.

Until every impacted IdP applies the required updates or workaround on their side, the supplicant-side workaround can be applied on Windows 11 by simply adding this registry value:


Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\PPP\EAP\13]
"TLSVersion"=dword:00000fc0


which is a bitmask, telling the rastls.dll library to enable TLS v1.0, v1.1 and v1.2 only, and disable all other versions (so including TLS v1.3).

(This needs to be specified for "13" which is actually EAP-TLS, but it still also applies to EAP-TTLS and PEAP where the workaround is needed.)

No reboot should be needed after setting the new value, or at least, the connection here started working again straight away after forgetting the network, adding the registry value, trying to reconnect and reinserting the user credentials provided by an affected IdP.

Unfortunately this setting on Windows is global for the entire machine, so it requires Administrator privileges to apply... I'm not sure if there could be a way to apply it just to a particular connection profile, which could be a """good""" and effective workaround also for geteduroam or eduroam CAT.


I honestly fail to understand why Microsoft did push such an update implementing a "breaking" Internet-Draft in this way, without even implementing a mechanism to automatically fallback to older working TLS versions when the new version is completely failing with older servers that are actually out there in production today...


I hope all this can be helpful for you too!


Best regards,
Vittorio Gambaletta

On 10/10/2022 08:33:50 CEST, Stefan Winter wrote:

Hi all,

I have run connectivity tests form cat-test.eduroam.org for the realm univ-paris1.fr and I see that the negotiated TLS version is 1.2

 

Thanks for having run such extensive diagnostics so far, for ruling out Credential Guard, TLS 1.3 server-side and TLS 1.2 client-side as certainly being not the issue.

 

What remains is to figure out which TLS server-side in combination with TLS 1.3 client-side on Windows 11 22H2 is causing the trouble. I think what we can say with some certainty is that the newest FreeRADIUS servers (3.0.26 and 3.2.0+) handle TLS 1.3 correctly. I also recall seeing issues in TLS 1.3 server-side in earlier versions of FreeRADIUS but can't say which ones exactly, and which configurations are affected.

 

So, as more people find this particular issue, it would be very nice if you could share which RADIUS server product and version you deploy on your servers. Ideally, not only if you have issues with Win 22H2 and TLS 1.3, but also if things work fine on that OS: being able to exclude products, versions and configurations is also helping.

 

Finally, this looks like neither a CAT nor geteduroam specific problem then.

 

Greetings,

 

Stefan Winter

 

Tomasz

 

W dniu 07.10.2022 o 12:44, Paul Dekkers (via cat-users Mailing List) pisze:

Hi,

Glad we tested this; this indeed means you're not affected by Credential Guard (may still apply to others) but since your Windows update your client now tries to negotiate TLS 1.3 and the RADIUS servers of your university doesn't handle that properly and there is thus no fallback to TLS 1.2

If you have contacts with your university IT desk, I guess it makes sense to inform them: you can now exactly indicate what the issue is.

Now I hope we can still find someone where Credential Guard is indeed the issue, so we can get more clarity on that.

Regards,
Paul

 

On 07/10/2022 12:39, Timothée Peraldi wrote:

Hi there!

I've ran the tests you've asked me:

 

- This command line in Powershell gives me 0 as an answer, so it seems like I'm not on CredentialGuard.

 

- I've installed eduroam again (with the CAT installer) with the login you've sent me, and it works! So I guess the issue is with my university's configuration.

 

Should I forward this conversation to my university's IT desk to get them to fix this? I've seen on Twitter that other people are having the same issue after updating Windows in other French universities.

 

Thank you for your help,

Timothée

Paul Dekkers <paul.dekkers AT surf.nl> a écrit :

Hi,

Thanks for making those screenshots and performing tests. Let's get to the bottom of this;

There are differences in Windowws 11 Professional and Enterprise; and I believe it's only Enterprise that enables Credential Guard by default. You can see this in "System Information" (or msinfo32.exe) at the bottom of the list in "Virtualization-based security services configured" and "Virtualization-based security services running". It should list "Credential Guard" if it's enabled.

You can also run this in powershell:

(Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard).SecurityServicesRunning

and if the result is 0, you're not on CredentialGuard.

So what could be at play is that you're seeing the TLS 1.3 issue; Because Microsoft changed 2 things at the same time for eduroam... I sent you a separate mail with a username/password and I hope you are so kind to test this account on this particular client. I'm sure this account works with TLS 1.3

The TLS 1.3 issue means that some Identity Providers are not compatible with clients trying to do TLS 1.3 authentication: and the authentication then fails, it doesn't try to fallback to TLS 1.2. This is something that can be resolved by an upgrade at the IdP. With the proper configuration, clients do fallback to TLS 1.2.

Regards,
Paul

 

On 06/10/2022 18:10, Timothée Peraldi wrote:

Hi Tony and Paul,

Here are some screenshots of the issue:

 

The system configuration :

http://tim.othee.fr/temp/eduroam_system_config.png

Please note that I have the Professional edition of Windows 11, and not the Entreprise one (but I believe those mostly share the same security policies?).

 

When I try to connect to eduroam, it asks me for my username and password ("an action is required"), then it says "we couldn't connect to this network":

http://tim.othee.fr/temp/eduroam_trying_to_connect.png

If I try again, it will ask for my password again, and so on and so on.

 

I have also tried the geteduroam app, but it says "unable to connect to eduroam", then sends me back the same "couldn't connect to this network" error:

http://tim.othee.fr/temp/geteduroam_not_working.png

 

For reference, here is a screenshot of eduroam working normally on Android 12, with the same login and at the exact same place:

http://tim.othee.fr/temp/eduroam_working_android12.jpeg

 

Please let me know if you need any additional information.

 

Have a great day,

Timothée

Paul Dekkers <paul.dekkers AT surf.nl> a écrit :

Hi,

If someone has more experience with Credential Guard and/or this Windows update, I'd love to find out. If true, we may need to write up an advisory, but it will affect a lot of people.

So far, it looks like this affects Windows Enterprise edition users (maybe Timothée can confirm?) and it would be a rolling update: so some already get it, others may not yet.

There is word about some Cumulative Updates fixing some of the "save credentials" issues for users, but it's unclear to me if that resolves the PEAP-MSCHAPv2 authentication for users that entered the credentials manually, and did not get them via "AD user credentials".

If this truely affects all PEAP-MSCHAPv2 authentications on Windows, and the majority of our users has Enterprise editions for Windows, we need to investigate what options still work,

Timothée; could you in fact check if geteduroam for Windows would work? We have some reports of a strange error, maybe this is actually related. You could download from https://www.geteduroam.app/

Regards,
Paul

 

On 05/10/2022 18:20, Tony Skalski (via cat-users Mailing List) wrote:

Credential Guard is reportedly on by default in the latest W11 update (have not confirmed this myself). This will block the use of NTLM hashes and will prevent EAP-PEAP from working.

On Wed, Oct 5, 2022 at 10:29 AM Timothée Peraldi <cat-users AT lists.geant.org> wrote:

Hello,
I have updated my computer to the 22H2 update of Windows 11, that was 
released yesterday by Microsoft.
   
Since then, I cannot connect my computer to eduroam. My other devices 
(Android 12, ChromeOS) still work fine, and other WiFi networks still 
work on this computer.
   
I've tried uninstalling and reinstalling CAT but it still won't connect.
   
I just checked on Reddit and I'm seeing a few other similar threads, 
all posted in the last few days.
Is this a widespread issue?
   
Have a great day,
Timothée

To unsubscribe, send this message: mailto:sympa AT lists.geant.org?subject=unsubscribe%20cat-users
Or use the following link: https://lists.geant.org/sympa/sigrequest/cat-users

 

--

Tony Skalski (he/him/his)

System Administrator | IT

Office: 507-786-3227

1510 St. Olaf Avenue Northfield, MN 55057

stolaf.edu

 

To unsubscribe, send this message: MailScanner soupçonne le lien suivant d'être une tentative de fraude de la part de "lists.geant.org" MailScanner soupçonne le lien suivant d'être une tentative de fraude de la part de "lists.geant.org" MailScanner soupçonne le lien suivant d'être une tentative de fraude de la part de "lists.geant.org" MailScanner soupçonne le lien suivant d'être une tentative de fraude de la part de "lists.geant.org" mailto:sympa AT lists.geant.org?subject=unsubscribe%20cat-users
Or use the following link: https://lists.geant.org/sympa/sigrequest/cat-users

 

To unsubscribe, send this message: mailto:sympa AT lists.geant.org?subject=unsubscribe%20cat-users
Or use the following link: https://lists.geant.org/sympa/sigrequest/cat-users

-- 
Tomasz Wolniewicz    
          twoln AT umk.pl        http://www.home.umk.pl/~twoln

Uniwersyteckie Centrum Informatyczne   Information&Communication Technology Centre
Uniwersytet Mikolaja Kopernika         Nicolaus Copernicus University,
pl. Rapackiego 1, Torun                pl. Rapackiego 1, Torun, Poland
            tel: +48-56-611-2750; tel kom.: +48-693-032-576

To unsubscribe, send this message: mailto:sympa AT lists.geant.org?subject=unsubscribe%20cat-users
Or use the following link: https://lists.geant.org/sympa/sigrequest/cat-users

-- 
This email may contain information for limited distribution only, please treat accordingly.

Fondation Restena, Stefan WINTER
Chief Technology Officer
2, avenue de l'Université
L-4365 Esch-sur-Alzette

To unsubscribe, send this message: mailto:sympa AT lists.geant.org?subject=unsubscribe%20cat-users
Or use the following link: https://lists.geant.org/sympa/sigrequest/cat-users