Wireshark-users: Re: [Wireshark-users] L2TP-over-IPsec (may be off topic)
From: Kok-Yong Tan <ktan@xxxxxxxxxxxxxxxxxxx>
Date: Tue, 14 Sep 2010 14:15:03 -0400

On Sep 14, 2010, at 13:59, Sake Blok wrote:

On 14 sep 2010, at 19:01, Kok-Yong Tan wrote:

However, I have a physically separate hardware firewall in between
the endpoints (a L2TP-over-IPsec client and a L2TP-over-IPsec server)
and I've discovered that the L2TP-over-IPsec VPN will only
successfully connect if UDP port 1701 is open on the firewall.

What do you mean by successfully connect? If that means the L2TP- over-IPsec client and the L2TP-over-IPsec server can communicate with each other? Did you check whether there is actually a tunnel formed? If not, it's just a L2TP connection and that will work, but it will not be encrypted.

By successful, I mean that I can ping the server from the client as well as ping any other device on the server side from the client. The reverse is also true (i.e., any device on the server side, including the server, can ping the client and only the client since it's a host-to-network VPN). When it's unsuccessful, I can see from the client side that the IPsec tunnel forms but I get the error that the "L2TP server is not responding". So it only comes up "halfway" until I create that WAN to LAN rule on the firewall. Note that as per my follow-up post, no port forwarding rule for port 1701 exists, only port forwarding rules for ports 500 and 4500 to the server.

It seems like the L2TP tunnel just does not trigger the IPsec encapsulation to kick in. What does a network trace say? Only traffic on UDP port 1701, no UDP-500, no ip proto 50 and no UDP port 4500? That would be in sync with the above.

This will be the next step but I haven't done that yet.

What type of L2TP-over-IPsec client and L2TP-over-IPsec server are involved?

I'm trying various Macintoshes at OS versions 10.5.8 and 10.6.4 to an Xserve running OS version 10.4.11.

--
Reality Artisans, Inc.             #   Network Wrangling and Delousing
P.O. Box 565, Gracie Station       #   Apple Certified Consultant
New York, NY 10028-0019            #   Apple Consultants Network member
<http://www.realityartisans.com>   #   Apple Developer Connection member
(212) 369-4876 (Voice) # My PGP public key can be found at <https://keyserver.pgp.com>