Using a Linux L2TP/IPsec VPN server with Windows Vista

eXTReMe Tracker
Last update: Jun 8, 2008


1.1 Introduction

I have made the following webpages on using L2TP/IPsec with Linux:

The page you are now reading describes how you can use Windows Vista with Openswan. It assumes you are already familiar with setting up Openswan with L2TP/IPsec. If you are not, it's probably a good idea to start with reading this page (the first one in the list). It includes information on setting up the Linux side. The other pages contain specifics on several L2TP/IPsec clients which are available for Windows and Mac OS X.

Windows Vista can also be configured to use IPsec without L2TP (see below). If you are looking for a third-party VPN client for Vista, check out that section too.

There are basically two methods of authenticating Windows 2000/XP/Vista clients in IPsec VPNs: Preshared Keys (PSKs) and X.509 certificates. Windows Vista also supports Kerberos V5, NTLMv2 and "health certificates" but these are Microsoft proprietary "standards" and not supported by Openswan. If you have a Windows XP or Vista client, I recommend that you start with a PSK and test that first. It should be easier to do than setting up certificates. Once you got a basic setup working, you can proceed with certificates. On Windows 2000 clients, however, it is more difficult to configure a PSK. So if you only have Windows 2000 clients, I suggest that you skip PSKs altogether and use certificates. Note that if you want to use PSKs then either each client shares this single PSK (not recommended for security reasons!) or each client has its own PSK but is also required to have a fixed IP address. For testing purposes this should not be a problem. Fixed IP addresses are not required for clients which use certificates. Don't forget to disable the PSK configuration files once you switch to certificates! Openswan will get confused if there are PSK and certificate configuration files for the same client / IP address. The PSK configuration will get the upper hand.

There is a snag. Microsoft believes that certificates should only be used for authenticating computers, not users (never mind that third-party IPsec clients authenticate users with certificates just fine!). This means that the certificate of the user has to be imported as a 'local computer certificate' (requires Administrator privileges). The result is that the certificate is shared by all users on the client machine (if there is only one user this should not be a problem of course). You can use different usernames and PAP/CHAP passwords to distinguish the users on a client, but passwords provide weaker security than user certificates. One final remark: I believe it is not possible to store the 'local computer certificate' on a smartcard or USB token, unfortunately (please correct me if I am wrong!).

You can import a certificate to Windows Vista manually (through MMC) but it can also be automated. See the "Advanced" section for that.

1.2 Author

The author of this document is Jacco de Leeuw. Corrections, additions, extra information etc. are much appreciated.

Back to Contents


2. Contents

3. Installation and configuration

The L2TP/IPsec client is installed on Windows Vista by default. You might want to check the Microsoft Knowledgebase to see if there are any Hotfixes or other bugfixes for Vista.

3.1 Personal firewall issues

I have not encountered problems with Vista's built-in firewall and its default settings when used with L2TP/IPsec. But there is one issue that you have to keep in mind: the default settings block incoming "ping" requests (ICMP echo). This is a bit annoying if you are testing and troubleshooting connections. Ping can be enabled on Vista but I could not find any official documentation on that. What worked for me is the following procedure:

Click "Start", and then "Control panel". Click "System and Maintenance" (or switch to "Classic View", which I prefer anyway). Click "Administrative Tools". Click "Windows Firewall with Advanced Security". Select "Inbound Rules". Select "File and Printer Sharing (Echo Request - ICMPv4-In". Use the right mouse button to enable the rule.

However, problems have been reported with the personal firewall included with Microsoft OneCare. This is a complete different firewall made by another team at Microsoft. It should not be confused with Vista's built-in personal firewall. It appears that VPN ports have to be opened for the OneCare firewall (note that opening TCP 1723 is not required for L2TP/IPsec, only for PPTP). The easiest solution would be to tick off the checkbox "VPN - connect to another computer over a virtual private network" in the OneCare settings ("Firewall connection tool"). This was reported on Alec Saunders' blog.

Back to Contents


4. Setting up connections with Preshared keys

We'll start with Preshared Keys (PSKs). The IPsec client included with Windows Vista supports PSKs out of the box.  The routine for authenticating Windows Vista through certificates is almost the same, but more work in advance because you have to generate and import a certificate. PSKs are easier to use than certificates but they have the disadvantage that in practice you can use them only for clients which have fixed IP addresses. Preferably, the PSK is distributed out-of-band, e.g. you write it down on a piece of paper and walk to the other machine.

Here is the procedure for configuring a Preshared Key based L2TP/IPsec connection on Windows Vista. Click on the links for screenshots. (Note that Vista supports various themes and styles. There may be some GUI differences on your screen).

On the Linux server you should successively get an IPsec connection, L2TP connection and then a PPP connection. Check /var/log/secure and /var/log/messages on the Linux server for errors. If everything works fine, you might want to consider upgrading to certificates. See below.

Back to Contents


5. Importing certificates

At this point I assume you have made certificates for your users. If not, see my other webpage.

In most cases certificates come in the form of PKCS#12 files. These files contain the user's certificate, the corresponding private key and one or more CA root certificates. There are two ways of importing certificates: manual and automated. If you want to do an automated import, you need to download and install a utility on the client (see below).

Here is 's how you can do a manual import to Windows Vista. (For an automated import of the certificate, see below).

Back to Contents

6. Using certificates

6.1 Setting up connections with certificates

Once you have imported the user certificate, you can configure an L2TP/IPsec connection which uses this certificate to authenticate against a Linux VPN server. 

The procedure for creating an L2TP/IPsec connection which uses a certificate is almost the same as the procedure for PSKs mentioned above. The difference is this:
You cannot select which client certificate to use for a particular L2TP/IPsec connection. How does the Windows Vista client know what certificate to use if multiple certificates from different CAs have been imported? Well, the client requires the certificates on both sides to be signed by the same Certificate Authority (CA). The client verifies that the server has a valid certificate, but you don't actually install the server's certificate on the client. Instead, the client verifies whether the certificate presented by the server was issued by a CA that the client trusts. This certificate is typically included in the PKCS#12 file. This PKCS#12 file contains the user's private key, the corresponding certificate and one or more CA certificates. The Linux server's certificate is not included in the PKCS#12 file.

6.2 "Verify Name and Usage attributes"

Vista will receive the server's certificate that the the server itself sends once the IPsec negotiations start. It then verifies the validity of this certificate with the CA certificate it has in its certificate store (i.e. was it signed by the CA or has the certificate expired). Vista then verifies if the hostname in the server's certificate matches the hostname that you entered in the VPN connection settings. Note that Windows 2000 and XP do not perform this hostname check (MacOS X does). See also this section which includes instructions on how to generate such server certificates with OpenSSL.

If you want to restore the original Windows 2000/XP behaviour (i.e. the subjectAltName in the certificate is not checked) you need to disable the option "Verify the Name and Usage attributes of the server's certificate" in the "IPsec Settings" window. Be warned though that this makes it easier for adversaries to impersonate the server if they have compromised a valid (client) certificate, e.g. from a stolen laptop.

The "Verify Name" option seems to imply that Vista also verifies the "Extended Key Usage" restrictions that a server certificate may have. The log messages seem to confirm this but I have not verified it yet.

Vista does not seem to verify the hostname in a server certificate when an IP address (instead of a hostname) was entered in the client's VPN connection settings. I have not checked whether an IP address in a server certificate (i.e. subjectAltName=IP:160.85.22.3 in OpenSSL) is actually verified by Vista. IP addresses in certificates are very inflexible so they do not make much sense anyway.

Back to Contents

7. NAT-Traversal

Windows Vista supports NAT-Traversal out of the box, both the official standard RFC 3947 and draft-02. More information on NAT-T in general can be found here.

7.1.1 Issue with Windows Vista and server-side NAT

Windows Vista has the same issue as Windows XP with ServicePack 2. This issue was discovered by George Ou.  Vista does not support establishing IPsec connections to servers behind NAT. Apparently Microsoft considers this a security risk because of an (uncommon) scenario which is described here. Microsoft even says that a VPN server behind NAT is "not recommended" (see also KB Q885348), although the NAT-T RFC describes it as a normal setup that should be supported. The problem occurs with both Windows Server 2003 and Openswan, so it is an issue in Vista, not Openswan.

The Windows XP SP2 workaround described in Knowledge Base article Q885407 does not work because this registry key does not exist under Windows Vista. The correct registry key for Vista has been obtained by George Ou through his Microsoft contacts. Since then, it has also been published in Microsoft KB article 926179. The key is:

  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent\AssumeUDPEncapsulationContextOnSendRule

This key should be created and set to 2 (DWORD) to allow server-side NAT. I have modified George's original script for Windows XP SP2. You can download the equivalent script for Windows Vista here. If you wish to restore Windows Vista to its default behaviour, use this script. After applying either script, remember to reboot your computer or to restart the "Remote Access Connection Manager" (Rasman, located in "Settings" -> "Control Panel" -> "Administrative Tools" -> "Services"), or you can use this batch script.

7.1.2 PSK and NAT-T in Vista

In most (production) cases you will want to use certificate authentication instead of a Preshared Key (PSK). Certificates provide better security and work better when NAT is involved. However, in some cases you may be forced to use a PSK.

Windows Vista is very similar to the L2TP/IPsec client included with Windows XP/2003, but there is an additional requirement when a PSK is used and NAT is involved. You have to add this line to your L2TP-PSK section:

  rightsubnet=vhost:%no,%priv

Windows XP/2003 support PSKs and NAT-T but they are based on draft-02 of the NAT-T standard ("draft-ietf-ipsec-nat-t-ike-02"). Vista also supports this draft-02 but when connecting to recent versions of Openswan it prefers RFC 3947 over draft-02. Apparently these implementations use different identifiers when NAT is involved: when a Windows XP/2003 client connects, Openswan reports the following:

  Main mode peer ID is ID_FQDN: '@blabla.example.com'

But when a Vista client connects (or probably any other RFC 3947 compliant client), the following is reported:

  Main mode peer ID is ID_IPV4_ADDR: 'x.x.x.x'

(Where x.x.x.x is the IP address that the client has on the NATed network).

7.1.3 Server-side NAT with pure IPsec or AuthIP

If you only wish to use L2TP/IPsec with Vista you can skip this section.

It appears that Windows Vista can not only use L2TP/IPsec but also IPsec without L2TP. This has the same server-side NAT issue as in the L2TP/IPsec scenario mentioned above. According to information obtained by George Ou from his Microsoft contacts, the registry key for the IPsec-without-L2TP scenario is:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPsec\AssumeUDPEncapsulationContextOnSendRule

However, I noticed that the \IPsec key does not exist so I am not quite sure if this will work. I have not investigated it.

A third scenario where Vista can be used as a client to connect to a server behind NAT is AuthIP (see below). The following command should enable the server-behind-NAT scenario with AuthIP:

netsh advfirewall set global ipsec ipsecthroughnat serverandclientbehindnat

This sets the following equivalent registry key to 2 (DWORD):

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy

Back to Contents


8. IPv6

In Windows Vista, IPv6 is installed and enabled by default. It is included as the "Internet Protocol Version 6 (TCP/IPv6)" protocol component in the properties page of network connections (in the Connections and Adapters folder).

Many applications in Vista now support IPv6, including the L2TP/IPsec VPN client. This would allow you to get a secure IPv6 connection, wherever you are, as long as you have a working (IPv4) Internet connection and your VPN server is able to provide access to an IPv6 network.

First of all, your Linux kernel and your PPP daemon must support IPv6. Most recent ones do (check 'man pppd'). You probably do not need IPv6 support in your L2TP daemon, assuming your Internet connection is IPv4. Exceptions may be l2tpd and xl2tpd because these maintain their own IPv4 address pools for remote clients so you won't be able to hand out IPv6 addresses with (x)l2tpd. However, like the other L2TP daemons, you should be able to use RADIUS through a plug-in and hand out IPv6 addresses to remote clients. Alternatively, you can configure (x)l2tpd manually to assign one IPv6 address in /etc/ppp/options.l2tpd:

ipcp-accept-local
ipcp-accept-remote
ipv6 ::dead:beef,::dead:beee
noccp
auth
(etc.)

This should come in handy for testing purposes or if you want to support only one client.

As far as I know, l2tpns is the only Open Source implementation with built-in IPv6 support.

Back to Contents


9. Advanced IPsec features

9.1 Weak crypto

If you connect to an Openswan error and you receive the following error:
"Error 741: The local computer does not support the required data encryption type."
then the Openswan server uses weak cryptographic algorithms. This is not the default configuration for Openswan so the best solution is to configure Openswan to use the standard (strong) algorithms. However, if you still want to use weak cryptography (not recommended!) you can change a registry setting in Vista ("AllowL2TPWeakCrypto"). This is described in KB article 929856.

9.2 Stronger crypto: AES, SHA-1, CRL

By default Windows Vista proposes ISAKMP SAs with 3DES encryption, HMAC authentication based on SHA-1 hashes and Diffie-Hellman group 14 (MODP2048). These are reasonable defaults and Openswan will accept them. Vista also supports ISAKMP SAs with AES but only in combination with ECC Diffie-Hellman groups, which Openswan does not support. (Note: you get more control over crypto algorithms if you use pure IPsec instead of L2TP/IPsec. See below).

Vista also proposes IPsec SAs with either 128-bit AES or 168-bit 3DES encryption and HMAC authentication based on SHA-1. Recent versions of Openswan default to AES, so in most cases AES will be used for the bulk encryption. AES is much faster than 3DES so this is good for the throughput of the L2TP/IPsec connection.

If you want 256-bit AES encryption instead of 128-bit AES encryption you need to modify the properties of the L2TP/IPsec connection on the Vista client.

Older versions of Openswan by default use 3DES and SHA-1 for IPsec SAs. These are good defaults for Windows 2000/XP clients that use the built-in IPsec stack. AES is much faster than 3DES and Vista supports AES, so you might want to explicitly enable AES support if you are using an older version of Openswan. You could for instance add the following lines to the connection section(s) in your ipsec.conf:

ike=aes-sha,3des-sha
esp=aes-sha1,3des-sha1

9.3 Perfect Forward Secrecy

I don't know how to enable PFS with L2TP/IPsec. That is, when the "New Connection Wizard" is used to create an L2TP/IPsec connection. There is still no "PFS" setting in the "New Connection Wizard". Perhaps the default security policy of the Wizard can be hacked so that PFS is enabled, but I haven't tried that.

There are valid reasons for using PFS. If you need it, you will have to use IPsec without L2TP. Vista supports PFS when pure IPsec is used (i.e. without L2TP), but only when the command-line is used to configure the connection, and when PSK authentication is used. PFS currently does not work with certificate authentication on Vista.

9.4 Strong CRL checking

The following command seems to indicate that Windows Vista can verify the revocation status of a certificate that is presented by a server:

netsh advfirewall set global ?
This results in:

      strongcrlcheck    - Configures how CRL checking is enforced.
                          0: Disable CRL checking
                          1: Fail if cert is revoked (default)
                          2: Fail on any error
                          notconfigured: Returns the value to its not
                          configured state.

But according to "netsh advfirewall show global", it appears that 0 is the default, not 1! You can change this with:
netsh advfirewall set global ipsec strongcrlcheck 1

I have not verified if this actually works.

9.5 IPCOMP and DPD

IPsec compression (IPCOMP) and Dead Peer Detection (DPD) are still not supported by Windows Vista.

Back to Contents


10. IPsec without L2TP

10.1 Discussion

10.1.1 Vista's built-in configuration utilities

Windows versions before Vista were very difficult to configure for IPsec without L2TP. Microsoft boasts that they have reduced the complexity: they say that in Windows 2000/XP it required more than 100 mouseclicks to set up an IPsec VPN connection, and in Vista it requires "only" 15 mouseclicks. There is also slightly more help info in Vista compared to XP, such as "What is a VPN?" but this is generally very basic info. The help info does say that IPsec without L2TP is not to be used for Road Warrior-style VPNs. They advise to use L2TP/IPsec or PPTP for that.

There are two new configuration utilities in Windows Vista that attempt to make IPsec without L2TP easier:

Unfortunately, both these configuration utilities experience a couple of problems.

The first problem is that there is almost no documentation about either "netsh advfirewall" or the IPsec client in WFwAS. Problem #2 is that there is a bug in Vista: when certificate-based authentication is involved Vista currently cannot process packets that it receives from the Openswan server. This problem is reported to be fixed in Vista SP1. The third problem is that things don't work at all if NAT is involved. A fourth problem is that you can only specify server IP addresses in the new Vista configuration utilities. You cannot specify the hostname of the server, so if the IP address of the IPsec server changes, all clients will have to be informed of this new IP address (this also rules out servers that addressed by DynDNS or something similar).

10.1.2 Third-party clients for Vista

An annoyance is that you need to know the actual IP address of the Vista client when you configure WFwAS or use netsh advfirewall. For example, check the Network Connection details of your Vista client's network adapter to see what IP address was received from the DHCP server. Another option is to find this IP address by running "ipconfig /all" on the command prompt.

An alternative is to use a 'wrapper' around Vista's IPsec configuration framework called Windows Filtering Platform (WFP). Here is a list of such clients for Windows 2000/XP. I don't know whether these will work on Vista as well. But IPsec configuration management has changed significantly in Vista and I suspect that these tools won't work. Work is under way to extend TauVPN / iVPN (by Nejc Skoberne and Stefan Markowitz) with Vista support. QuickVPNplus is reported to work with Vista.

Another alternative is to buy a third-party IPsec client. Microsoft maintains a list of third-party VPN software that is compatible with Vista. A complication is that VPN clients may require access to the Vista kernel. Some anti-virus vendors are in trouble because Microsoft does not give them access to the Vista kernel, so I would not be surprised if VPN client vendors are in the same boat.

Note that if you use IPsec without L2TP, remote clients will not be able to obtain virtual IP addresses from the internal (i.e. protected) subnet. I.e. to other computers the remote client will not appear to be located on the internal network.

10.2 WFwAS

10.2.1 WFwAS with PSK authentication

As mentioned in the previous section, Vista's WFwAS does not work when used with certificates, but it does work with Preshared Key (PSK) authentication. Of course in most real-world setups you do not want to use a PSK for authentication, because it is less secure and does not scale well. But for relatively simple setups this can be an option. The configuration is described in the following sections.

10.2.2 WFwAS with PSK: server side configuration

Here is a sample Openswan configuration for use with Vista's WFwAS client. As you can see it is a relatively simple configuration file. There is no L2TP involved.


# Configuration supporting multiple users with any type of
# IPsec client. This includes the built-in client (WFwAS)
# in Windows Vista.
#
# Authenticates through a Pre-Shared Key. Supports clients that
# are behind NAT or not.

conn IPSEC-PSK
     #
     authby=secret
     pfs=no
     rekey=no
     keyingtries=3
     #
     # ----------------------------------------------------------
     # The VPN server.
     #
     # Allow incoming connections on this IP address.
     #
     left=123.123.123.123
     #
     # Our internal (protected) network
     leftsubnet=192.168.1.0/24
     #
     # ----------------------------------------------------------
     # The remote user(s).
     #
     # Allow incoming connections from any IP address.
     right=%any
     rightsubnet=vhost:%no,%priv
     #
     # ----------------------------------------------------------
     # Change 'ignore' to 'add' to enable this configuration.
     #
     auto=ignore


Note: by default my example configuration file is not activated by default (for security reasons). Change auto=ignore to auto=add to enable the connection. Also, if you use different IP addresses and subnets in your network, change these accordingly in the example above. Then, enter your PSK in the ipsec.secrets file:


123.123.123.123: PSK "keysharedbyallclients"

Restart or reload Openswan to enable the new configuration.

10.2.3 WFwAS with PSK: client side configuration

After you have configured your Openswan server, you create an IPsec connection rule on your Windows Vista client. Here is the procedure for configuring a Preshared Key based IPsec connection on Windows Vista:

The new IPsec connection rule shows up in "Connection Security Rules". Later on, should you want to modify the settings of this connection rule, select the rule and right-click "Properties".

The IPsec connection rule is immediately active. But IPsec connections are not. An IPsec connection will be automatically set up when you try to reach an IP address in the remote subnet. In the example, the subnet is 192.168.1.0/24, so if you "ping 192.168.1.1" the connection will be set up. You do not have to click "Connect" anywhere. The Vista client will contact the Linux server and you should see an incoming IPsec connection in the logfiles. Check /var/log/secure and /var/log/messages on the Linux server for messages.

IPsec connections are automatically disconnected after a while. If you want to disconnect immediately (or [temporarily] disable the IPsec connection rule), right-click the rule and select "Disable Rule" from the context menu.

10.2.4 Stronger crypto with WFwAS: AES, SHA-1

Unlike L2TP/IPsec, you can modify the IPsec settings of the WFwAS client and make it use stronger cryptographic settings.

In WFwAS, select the top option, also called "WFwAS". Select "Action" from the menu and then "Properties". Go to the "IPsec Settings" tab and then click "Customize". In the "Customize IPSec Settings" window, select the option "Advanced" for both "Key exchange (Main Mode)" and "Data protection (Quick Mode)". Click the top "Customize" button. The "Customize Advanced Key Exchange Settings" window appears. Select Diffie-Hellman group 14 and click "Add" to add AES-256-SHA1 to the IKE proposals. Select the new AES-256-SHA1 entry and use the arrow buttons to move AES-256-SHA1 to the top of the list. This will make sure that they are tried first. Click "OK".

Click the bottom "Customize" button. The "Customize Data Protection Settings" window appears. Click "Add" and add AES-256-SHA1 to the IPsec proposals. Move these too to the top by selecting them and clicking the "Up" arrow. In the "Data Protection (Quick Mode)" settings you may also want to enable "Require encryption for all connection security rules that use these settings". If not, Vista will also propose "ESP Null" and AH, i.e. no encryption at all. Openswan will not accept proposals without encryption (thankfully!) but it will log these messages: "kernel algorithm does not like: no alg" and "unsupported ESP Transform ESP_NULL from x.x.x.x".

Click "OK" after you added AES-256-SHA1.  You're back at the "Customize IPSec settings" window. The "Authentication method" does not have to be changed. Apparently it is not used for IPsec Connection Security rules.

10.3 netsh advfirewall

10.3.1 netsh advfirewall with PSK authentication

Read this section if you prefer to use the command line instead of a GUI to set up the IPsec connection (for example, because you want to run it from a script).

Of course the setup on the server side is the same as with WFwAS, only the client side is different. You don't run the WFwAS GUI but instead use the command prompt to run these two commands (requires Administrator privileges):


netsh advfirewall set global mainmode mmsecmethods dhgroup14:aes256-sha1,aes128-sha1,3des-sha1

netsh advfirewall consec add rule name="LinuxVPNrule" enable=yes mode=tunnel localtunnelendpoint=192.168.0.2 remotetunnelendpoint=123.123.123.123 endpoint1=192.168.0.2 endpoint2=192.168.1.0/24 action=requireinrequireout auth1=computerpsk auth1psk="keysharedbyallclients" qmsecmethods=esp:sha1-aes256,esp:sha1-aes128,esp:sha1-3des qmpfs=none

You probably use different IP addresses and subnets in your setup, so change the IP addresses that are displayed in bold according to the ones you are using. Also use your own PSK instead of my example "keysharedbyallclients". If you want to use Perfect Forward Secrecy (you probably do, if you are connecting to a Linux VPN server), see below.

After you have run these commands the IPsec connection rule is immediately active. But IPsec connections are not. An IPsec connection will be automatically set up when you try to reach an IP address in the remote subnet. In the example, the subnet is 192.168.1.0/24, so if you "ping 192.168.1.1" the connection will be set up.

Other commands that may be useful:


# Show all IPsec rules:
netsh advfirewall consec show rule name=all

# Delete all IPsec rules (be careful):
netsh advfirewall consec delete rule name=all

# List all available parameters when adding a rule:
netsh advfirewall consec add rule /?

# Dump all rules to a script (does nothing; does not work?):
netsh advfirewall consec dump

10.3.2 netsh advfirewall with PSK and PFS
 
Perfect Forward Secrecy (PFS) is a feature that can only be enabled on the command line, not with WFwAS. You enable PFS with the parameter qmpfs=. If it is not present when you add a rule, it defaults to qmpfs=none which means that PFS is disabled. To enable PFS you would use qmpfs=dhgroup14 or qmpfs=dhgroup2 (the other values for the ECC DH groups are not supported by Openswan and DH group 1 is too weak). Probably the best option is to use qmpfs=mainmode. This tells Vista to use the same PFS group as configured for Main Mode (the "mmsecmethods" command). Then use the following commands instead of the commands mentioned previously:


netsh advfirewall set global mainmode mmsecmethods dhgroup14:aes256-sha1,aes128-sha1,3des-sha1

netsh advfirewall consec add rule name="LinuxVPNrule" enable=yes mode=tunnel localtunnelendpoint=192.168.0.2 remotetunnelendpoint=123.123.123.123 endpoint1=192.168.0.2 endpoint2=192.168.1.0/24 action=requireinrequireout auth1=computerpsk auth1psk="keysharedbyallclients" qmsecmethods=esp:sha1-aes256,esp:sha1-aes128,esp:sha1-3des qmpfs=mainmode

The parameter that was changed for PFS is displayed in bold.

10.4 Certificate authentication

10.4.1 WFwAS with certificate authentication: server side configuration

Here is a sample Openswan configuration for use with Vista's and certificate authentication. It is slightly more complex than PSK authentication.

# Configuration supporting multiple users with any type of
# IPsec client. This includes the built-in client (WFwAS)
# in Windows Vista.
#
# Authenticates through certificates. Supports clients that
# are behind NAT or not.

conn IPSEC-CERT
     #
     authby=rsasig
     pfs=no
     rekey=no
     keyingtries=3
     #
     # ----------------------------------------------------------
     # The VPN server.
     #
     # Allow incoming connections on this IP address.
     #
     left=123.123.123.123
     #
     # Our internal (protected) network
     leftsubnet=192.168.1.0/24
     leftrsasigkey=%cert
     leftcert=/etc/ipsec.d/localCERT.pem
     #
     # ----------------------------------------------------------
     # The remote user(s).
     #
     # Allow incoming connections from any IP address.
     right=%any
     rightsubnet=vhost:%no,%priv
     rightca=%same
     rightrsasigkey=%cert
     #
     # ----------------------------------------------------------
     # Change 'ignore' to 'add' to enable this configuration.
     #
     auto=ignore

This example configuration file is also not activated by default. Change auto=ignore to auto=add if to enable the connection. Restart or reload Openswan to enable the new configuration.

Note that this is just one example. Certificate authentication can be implemented in several ways. See also the "Certificates" section on my other page.

10.4.2 WFwAS with certificate authentication: client side configuration

There is currently a bug in Vista if you use the WFwAS client with certificate authentication. The connection is set up and packets are exchanged between the Vista client and the Openswan server, but for some reason Vista does not process incoming packets. I have been informed that this bug has been fixed in Beta 6001.17036 v.652 of ServicePack 1.

To use certificate authentication with WFwAS, install a client certificate, then follow the same procedure as for PSKs, until you get to the "Authentication method" window. Select "Computer certificate". Click "Browse" and select the root certificate of the CA that issued the client certificate. Or, alternatively, manually enter the subject name of the root certificate at "CA name". Then continue the procedure for PSKs.

WARNING: Vista's IPsec client does not seem to verify the subjectAltname in the server's certificate (unlike Vista's L2TP/IPsec client). And on the IPsec client you cannot specify a subject of the server certificate either (unlike Linux clients where you can specify rightid="CN=vpnserver.example.com", for example). This means that a stolen client certificate can impersonate a server to Vista clients. There is a "Certificate mapping" option in the Vista IPsec client but I don't know if it can be used to thwart this attack, and besides, this Certificate mapping option seems to require ActiveDirectory which is not supported on Linux.

10.4.3 netsh advfirewall with certificate authentication

The following procedure can be used if you want to use the command line and authenticate to the server with a client certificate. Again, the setup on the server side is the same as with WFwAS, only the client side is different. Run these two commands (requires Administrator privileges):


netsh advfirewall set global mainmode mmsecmethods dhgroup14:aes256-sha1,aes128-sha1,3des-sha1

netsh advfirewall consec add rule name="LinuxVPNrule" enable=yes mode=tunnel localtunnelendpoint=192.168.0.2 remotetunnelendpoint=123.123.123.123 endpoint1=192.168.0.2 endpoint2=192.168.1.0/24 action=requireinrequireout auth1=computercert auth1ca="C=NL, S=ST, L=L, O=TESTORG, CN=TESTCA certmapping:no excludecaname:no" auth1healthcert=no qmsecmethods=esp:sha1-aes256,esp:sha1-aes128,esp:sha1-3des qmpfs=none

You probably use different IP addresses and subnets in your setup, so change the IP addresses that are displayed in bold according to the ones you are using. Also change the parameter auth1ca and enter your CA's subject name (if the name contains spaces, you may need to escape these or put the name between quotes. Or perhaps you don't have to do anything. I have not looked into this yet).

If you want to use PFS (you probably do, if you are connecting to a Linux VPN server), see above.

10.5. NAT with WFwAS or netsh advfirewall

Unfortunately the procedures described above do not work when NAT is involved. Openswan responds with a QuickMode proposal to the Vista client but for some reason Vista does not like a parameter: "IkeIsSaValidForTunnel failed with Windows error 87(ERROR_INVALID_PARAMETER)". Vista then sends an "IKE Informational Mode" message to the server and disconnects:

pluto[5863]: "IPSEC-PSK"[1] 192.168.15.1 #3: ignoring informational payload, type INVALID_PAYLOAD_TYPE
pluto[5863]: "IPSEC-PSK"[1] 192.168.15.1 #3: received and ignored informational message
pluto[5863]: "IPSEC-PSK"[1] 192.168.15.1 #3: received Delete SA payload: deleting ISAKMP State #3

But from the IKE log I cannot deduce which parameter exactly is unacceptable to Vista.

The same problem also occurs when ipsec-tools (racoon) is used instead of Openswan. Therefore I suspect that this is a problem in Vista. The problem has not been resolved in Vista SP1. According to Knowledge Base article Q944335, it's not a bug, it's a feature! Microsoft writes: "Behind an NAT device, if more than one computer shares the same source port, a conflict may occur." But that's why the IETF  invented NAT-Traversal! It looks like Microsoft is not compliant with the NAT-T RFC, and they prefer their new proprietary VPN protocol SSTP which they advertise as the VPN solution for nasty NAT problems...

Back to Contents


11. Troubleshooting

11.1 Dial-Up Networking error messages

The error messages generated by Windows 2000/XP are not terribly clear. Microsoft claims to have improved and clarified the error messages and help info in Windows Vista. There is a list of Dial-Up Networking (DUNS) error messages for Vista in KB article 923944. To some extend, Vista uses the same Dial-Up Networking (DUNS) error messages as Windows 2000/XP. But I noticed that some errors that were reported in Windows 2000/XP are not displayed in Vista at all, fooling the user into thinking that the connection is still up. If you get an error message not listed here or if you just want more information on why a particular connection fails, enable the Oakley log and/or the PPP log on Windows Vista.

Vista also has a new "network diagnostics framework" but so far I have found it utterly useless for troubleshooting VPN problems (with advice such as "Reset the network adapter" etc.).

11.1.1 "Error 732: Your computer and the remote computer could not agree on PPP control protocols."

See this page.

11.1.2 "Error 741: The local computer does not support the required data encryption type" or
"Error 742: The remote server does not support encryption".

See this section.

11.1.2 "Error 766: a certificate could not be found"

This appears to be a new error message that was not present in Windows 2000/XP (those already have error 781/786). You get the following error message:

"Error 766: A certificate could not be found. Connections that use the L2TP protocol over IPSec require the installation of a machine certificate, also known as a computer certificate."

See the remarks under Error 781. I don't know exactly the difference between these two.

11.1.3 "Error 787: could not authenticate"

You get the following error message:

"Error 766: The L2TP connection attempt failed because the security layer could not authenticate the remote computer."
This appears to be similar to Error 835, except you have probably disabled the option "Verify the Name and Usage attributes" and the server certificate is still not acceptable to the client. This could mean that the server does not have a valid certificate, the client does not have a root certificate of the CA that issued the server's certificate or perhaps there is a man-in-the-middle or some other security problem. You may have to look at the Oakley log for more detailed information (see below).

11.1.4 "Error 809: server not responding"

You get the following error message:

"Error 809: The network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices (e.g., firewalls, NAT, routers, etc.) between your computer and the remote server is not configured to allow VPN connections. Please contact your Administrator or your service provider to determine which device may be causing the problem. "
You may get this error when the server is behind NAT but you have not applied the Vista registry patch as described here. The IPsec phase seems to be working but the L2TP/PPP packets do not get through. There may be other causes for this error. Check the firewall settings, i.e. allow IKE (UDP 500), NAT-T (UDP 4500) and ESP (IP 50).

11.1.4 "Error 835: could not authenticate"

This too appears to be a new error message that was not present in Windows 2000/XP. You get the following error message:

"Error 835: The L2TP connection attempt failed because the security layer could not authenticate the remote computer. This could be because one or more fields of the certificate presented by the remote server could not be validated as belonging to the target destination."

This error occurs under the following condition: 1) you are using certificates 2) the server certificate contains a hostname and 3) this hostname does not match the hostname configured in the VPN connection properties. Disabling "Verify the Name and Usage attributes of the server's certificate" in the "IPsec Settings" window is a quick workaround for the problem, but this is probably a bad idea because it reduces security. A better solution is to make sure that the hostname in your VPN settings matches the hostname in the server certificate. For more details, see this section.

11.2 "OAKLEY_GROUPs supported"

You may get the following warning:
"only OAKLEY_GROUP_MODP1024 and OAKLEY_GROUP_MODP1536 supported. Attribute OAKLEY_GROUP_DESCRIPTION"
This is just a harmless message and not an error. By default one of the IKE proposals by Windows Vista is a Diffie-Hellman MODP2048 (group 14) key exchange which is not supported by your version of Openswan. Both sides will agree upon one of the other IKE proposals with either MODP1024 (group 2) or MODP1536 (group 5). Nothing to worry about. If you upgrade to a more recent version of Openswan you will get MODP2048 support.

You may get the following warning:
"Diffie-Hellman group 20 is not a supported modp group.  Attribute OAKLEY_GROUP_DESCRIPTION"
"Diffie-Hellman group 19 is not a supported modp group.  Attribute OAKLEY_GROUP_DESCRIPTION"
Vista also supports DH group 19, which is an elliptic curve algorithm using a 256-bit random curve group (NIST identifier P-256), and DH group 20, which is an elliptic curve algorithm using a 384-bit random curve group (NIST identifier P-384). Openswan does not support these, possibly due to software patent issues.

11.3 Windows Vista picks the wrong certificate

I have not seen this problem on Vista yet, because I always use rightca=%same anyway :-). For more information, see my Windows 2000/XP page.

Back to Contents


12. Troubleshooting: examining the Windows logs

It may not not be very obvious in the Windows Vista L2TP/IPsec client but there are two phases: the IPsec phase and the L2TP/PPP phase. Each phase has its own logs.

Information about troubleshooting on the Linux side can be found on my main L2TP/IPsec page.

12.1 Oakley logs

If there is a problem with your L2TP/IPsec VPN connection, you may need to look at the IPsec logs on your Windows workstation. Examples are troublesome firewalls, broken NAT devices or IPsec authentication problems. These cannot be resolved by looking up the DUNS error code or by examining the PPP (RAS) logs.

High level logging is enabled by setting the DisableIKEAudits registry key. For more details, see Basic Troubleshooting for IPsec based VPN’s by Stefaan Pouseele.

For more detailed logs you need to enable IPsec (IKE) logging. On Windows 2000 and XP you enable this by setting a registry key. This procedure does not work on Windows Vista. I have been in contact with Microsoft Support and they provided the following procedure:
This ETL file is a binary file. You will have to convert it to human readable output with tracefmt.exe, a tool that is included in the Windows XP Support Tools pack.
Note that the files %SystemRoot%\System32\Ikeext.etl and %TEMP%\wfpdiag.txt contain the Preshared Key (PSK) if your VPN connection uses a PSK. Change your PSK if you send these files to a third party for analysis. You may also want to delete these files after you have examined them.

I don't know if this same routine can be used on Windows Server 2008 as well.

12.2 PPP logs

On Windows 2000/XP, "netsh ras set tracing * enabled" command enables PPP logging. Although this command still works on Vista, Microsoft recommends using "netsh ras diagnostics" instead. This entry on the Microsoft Routing and Remote Access Blog provides the following procedure:
Back to Contents


13. Advanced

13.1 Automated certificate installation

A number of tools have been described on my other webpage. I have tried certimport by Xelerance and it works fine on Windows Vista. CertImport by Kiko Vives Aragonés fails to run because of a missing file MFC42D.DLL. I suspect that recompiling the source code with Visual Studio 2005 (Express Edition) may resolve this. Note that all tools used to import a machine certificate require Administrator privileges.

13.2 Automated L2TP/IPsec configuration on Windows Vista

I don't know if the Connection Manager Administration Kit (CMAK) can be used with Windows Vista. Again, it does not matter because it probably does not make (financial) sense to use CMAK with Openswan. See also my other webpage.

There is some documentation on the Microsoft website about the Windows Filtering Platform (WFP), which is the underlying platform for Windows Firewall with Advanced Security (WFwAS). Possibly this WFP API can be used to create IPsec connections automatically.

13.3 Joining a domain

Matt Berther has written about joining a domain with Vista. I have not verified this.

13.4 MS-CHAPv1 not supported

As you can see from this screenshot, the first version of MS-CHAP is not supported by Vista. It supports MS-CHAPv2 only. This is discussed in Microsoft Knowledge Base article Q926170. Windows 2000/XP do support MS-CHAPv1. In my opinion it is a bit silly to drop support for MS-CHAPv1 because the security of an L2TP/IPsec connection does not depend on MS-CHAPv1 (unless you are using a group shared key that is public knowledge, which you should not do anyway). Besides, PAP and CHAP are still supported in Vista and these are even less secure than MS-CHAP. The only reason I can think of is that it is a server-side issue. Microsoft wants an authentication scheme for NTLM which uses irreversible passwords and MS-CHAPv1 was not strong enough. I think you cannot disable MS-CHAPv1 on Windows Server 2000/2003.

Back to Contents


14 AuthIP

14.1 AuthIP discussion

Microsoft has implemented "AuthIP", a proprietary extension (see also: "embrace and extend") to IKE, an important part of the IPsec standard. There is no in depth information about AuthIP. There is mainly high level product overview blurb on the Microsoft website (such as this, this, this and two Cable Guy colums). Of course this Microsoft proprietary extension is not supported by any Open Source implementation. It is undocumented and no doubt heavily patented. AuthIP is only supported by Windows Vista and Windows Server 2008. Microsoft seems to prefer AuthIP over IPsec, according to this presentation. In AuthIP there is a "second authentication" and the first authentication (the one defined by the official IKE standards) is optional, or "anonymous". (Why didn't Microsoft adopt IKEv2 instead if they needed 'legacy' authentication methods?). AuthIP adds an optional second authentication phase based on NTLMv2, user certificates or Kerberos. AuthIP is never used with L2TP/IPsec, only with 'pure' IPsec.

When you connect, Vista logs the following Vendor IDs:

packet from x.x.x.x:500: ignoring Vendor ID payload [MS-MamieExists]
packet from x.x.x.x:500: ignoring Vendor ID payload [MS NT5 ISAKMPOAKLEY 00000005]
packet from x.x.x.x:500: received Vendor ID payload [RFC 3947] method set to=110
packet from x.x.x.x:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n] meth=106, but already using method 110
packet from x.x.x.x:500: ignoring Vendor ID payload [FRAGMENTATION]
packet from x.x.x.x:500: ignoring Vendor ID payload [MS-Negotiation Discovery Capable]
packet from x.x.x.x:500: ignoring Vendor ID payload [Vid-Initial-Contact]
packet from x.x.x.x:500: ignoring Vendor ID payload [IKE CGA version 1]

The Vendor IDs "MS-Negotiation Discovery Capable" (MD5 hash: fb1de3cdf341b7ea16b7e5be0855f120), "IKE CGA version 1" (MD5 hash: e3a5966a76379fe707228231e5ce8652) and "MS-MamieExists" (MD5 hash: 214ca4faffa7f32d6748e5303395ae83) probably announce AuthIP support.

14.2 AuthIP bug in Vista (pre-ServicePack)

Due to a bug in Vista, the WFwAS client and netsh advfirewall insist on doing a second authentication, even if you have not asked for it. Vista sends IKE packets containing 'Next Payload type' 133 with 'Exchange type' 243 (Microsoft's proprietary version of Main Mode) and Exchange type 246 (the AuthIP version of Informational Message reporting a disconnect). These types are unknown to Openswan because they are non-standard numbers reserved for private (i.e. in-house) use. Normally an IPsec implementation should negotiate proprietary extensions by sending and receiving Vendor IDs. But Microsoft ignores this and sends these non-standard packets right away. So Openswan has no choice but to reject these packets:
packet from x.x.x.x:500: next payload type of ISAKMP Message has an unknown value: 133
Although the first authentication succeeds, the second authentication fails because Openswan does not support Microsoft's proprietary AuthIP scheme. So Vista will fail to connect.

This bug has been fixed in Vista SP1.

Back to Contents


15.1 PPTP authentication: PEAP-EAP-MSCHAPv2 and PEAP-EAP-TLS

A new feature in Windows Vista is support for stronger user authentication in PPTP and L2TP/IPsec VPNs. Previous versions of Windows only supported passwords and the encryption keys were derived from these passwords which is not good practice. Vista now also supports PEAP in PPTP VPNs. Windows 2000/XP supports PEAP only for 802.1x wireless networks. The authentication mechanisms supported in Vista's PEAP are EAP-MSCHAPv2 (passwords) and EAP-TLS (smartcards and local certificates), which are considerable stronger than the MSCHAPv2 passwords that were the only option in previous versions of Windows.

PPTP with PEAP will initially not ask for a username and password but only show a "Connect" button. When the client connects, it will try to validate the root certificate name and the server certificate, respectively. Only then the client will request the user for his username and password (in the case of EAP-MSCHAPv2).

This could be an alternative for L2TP/IPsec. Advantages that I can think of:
Disadvantages:
15.2 L2TP/IPsec with PEAP

Windows Vista also supports the use of PEAP with L2TP/IPsec. I have not investigated this. I don't think it adds much security or features, because L2TP/IPsec already has strong authentication for both client and server. Smartcards are already supported through L2TP/IPsec with EAP-TLS, and passwords through L2TP/IPsec with PAP/CHAP/MSCHAP. No need to add an extra PEAP layer.

Back to Contents


16. Split tunnelling

If you don't want to send regular Internet traffic through the VPN tunnel you may want to enable split tunnelling. See this section for more details about split tunnelling and its advantages and disadvantages.

On Windows Vista, you enable or disable split tunnelling by modifying the 'Advanced' TCP/IP settings of the VPN connection you created. You have to uncheck the box called "Use default gateway on remote network" to enable split tunnelling.

Back to Contents


17. Acknowledgements

Thanks to George Ou of TechRepublic for helping resolve the NAT-T problem in Vista. Thanks also go to his Microsoft contacts for releasing this information.

Thanks to Okan C. of Microsoft Support for the info on obtaining an Oakley.log equivalent on Vista.

Back to Contents


18. Related links

Microsoft has done some interoperability testing between a beta version of Vista and racoon. They don't have nice things to say about racoon, and strangely enough there is no information on how to set up a Vista client.

Back to Contents


19. Revision history

Jun 8, 2008: Vista actually does support AES-256 for IPsec SAs.
Feb 18, 2008: Pure IPsec bug not fixed in SP1. Microsoft says it's not a bug, it's a feature...
Dec 25, 2007: Possibly another bug in Vista when pure IPsec is used with NAT.
Nov 12, 2007: Confirmed: bug in Vista when certificate authentication is used with pure IPsec. (Fixed in SP1).
Sep 18, 2007: Added info on pure IPsec client in Vista.
Mar 9, 2007: Added info on getting an Oakley.log equivalent.
Nov 21, 2006: MS-CHAPv1 not supported.
Nov 9, 2006: NATed server problem resolved.
Nov 5, 2006: Page created. Based on the Windows 2000/XP page.

Jacco de Leeuw