Wireshark · Wireshark-users: Re: [Wireshark-users] Huge VoIP Problem :(

[Mark Jeffers <mantramark@xxxxxxxxx>]

Wow - Great comments/ideas guys.

Sorry, I misspoke - the phones do NOT tag packets for VLAN1 --- they come through untagged.
Yes - the phone server is attached untagged to the switch on VLAN9 only.
Only two VLANs on the network.

As for your other questions...
Multiple switch links  - on the entire network, we have 7 switches.  The main site where the phone switch is located has three switches, setup in what Dell calls a "stack" - all three switches are configured as one and act as one (supposedly).  The other 4 switches are distributed to 3 other nearby buildings connected to the main stack with fiber (one building has 2 of the switches also configured as a unified stack).  No device is more the one switch link from the phone server -- assuming the Dell "stack" truly functions as one, which is a large assumption.  (Physically, no device is more than two switch links away from the phone server)
I know what you're all thinking right now... but the irony is that the worst problems occur at the main site where the three-stack switch is.  The site with the two stack has almost no problems, and the other two sites report intermittent problems.  In other words, the fiber links and the configurations of the ports are identical and done correctly (at least according to Dell engineers).  For... uh... "science", I experimented with changing which switch on the main network stack the remote sites link in to, thinking maybe, for example, the site with the 2-stack would report more problems and one of the other sites would experience some relief...  no change, though.  Uggg!

The firewall does indeed tag packets for VLAN9, but I just did that change a couple of days ago.  Before, I was using the DMZ port on the firewall configured on the VLAN9 subnet - so the firewall was plugged into the switch twice -- one port for VLAN1, one port for VLAN9 with a policy allowing the PCs to access the phone server on VLAN9 for the call center software.  In this configuration, the switch allowed only the right VLAN traffic to access the right firewall port - the firewall did no tagging at all.   I thought maybe the "double-plugging" of the firewall into the network was somehow causing a loop so I changed to the current configuration... zero change in performance.  Is the firewall tagging VLAN9 a problem?

Different subnets for the two VLANs, yes.  The PC VLAN1 is on 10.25.247.0/24 and the Phone VLAN9 is 10.25.249.0/24.   A third party vendor chose the PC subnet - we rely on them for a huge database system, so I can't change it.   There is DHCP for the PCs on VLAN1.  All devices on VLAN9 are statically assigned.

By the way, I've neither seen nor heard of any problems with any PC data - not the database, not the Internet.  Only phone problems.  Not a big surprise though, cause the you can hear packet loss in an audio stream.  The computers may slow down a bit too, but the users aren't noticing it.

I'm going to comb through the network, and make sure nothing besides the firewall is routing between the VLANs.  I'm pretty sure, but it's time to bust out the 'fine-tooth' comb.

Ryan - I'm going to try your suggestion on the ring-buffered capture.

Unfortunately guys, I'm heading out tomorrow morning to do some completely-of-the-grid deep woods camping for a week.  I'll have to revisit this issue a week from Monday.  This site has been limping along for months, so one more week won't hurt 'em too much.

Thank you, thank you, thank you for all your suggestions.  I'll report back as soon as I can.

Cheers,
mj

On Fri, Jun 19, 2009 at 5:47 PM, Marc Luethi <netztier@xxxxxxxxxx> wrote:

On Fri, 2009-06-19 at 09:40 -0400, Mark Jeffers wrote:
> The phones actually act as a level 2 switch themselves.   They tag
> their own packets for VLAN9 (the voice VLAN on my network) and tag the
> packets of the PC attached to them (if there is one) as VLAN1.

Do they actually _tag_ the PC's packets to VLAN 1? On 802.1q trunks,
VLAN 1 generally is the "native" VLAN and does not have the extra 4
bytes and therefore no tag. It is however possible to tag the native
VLAN as well - in general, this option must be activated, though.

> Attaching a phone and a pc to the same switch port has made me nervous
> from day one, but the vendor swore up and down it would work no
> problem.

It definitely isn't. Provided you configure the switch port as 802.1q
link (being a cisco-i-fied persion, I use the term "802.1q trunk" often,
although other vendors use "trunk" to refer to multiple parallel
ethernet links)

In your case, the switch port for a Phone+PC should then be configured
to send packets for VLAN 9 with tags, and packets for VLAN 1 without (or
with, depending if the phone is configured to send VLAN 1 packets with
tags or without).

Make sure that no packets for/from other VLANs (than 1 and 9) go out of
the switch on those Phone+PC ports.

> Also, one thing that has me shaking my head in disbelief is that while
> Allworx built their phones with VLAN tagging abilities, their main
> phone server can't tag its own packets.

Perfectly allright. Just make sure that the server's switch port is
"VLAN 9 only" and sends untagged frames. (Cisco speak: "switchport
access vlan 9)

> But anyway, I was of course suspicious of the pc/phone combo, but some
> of my most problematic phones have no pc attached to them.  Plus, I
> figured building the VLANs would solve any problem related to that.
> Perhaps I was wrong?

Well, VLANing is a good way to separate traffic, but some consideration
is necessary to make it work.

Questions:

- are there any inter switch links between the voice server's switch
 port and the Phone+PC's switch ports?
- if yes, is that/these interswitch link properly configured as 802.1q links
 to carry the needed VLANs (and if possible, exclusively the needed ones)?
- actual voice traffic is between the phones directly, not between
 the phone and the SIP server; phone-to-server is only used for call
 setup and registration.
- are the firewall's switch ports configured properly as untagged for
 VLAN 1 and VLAN 9, respectively (assuming that the FW does not do
 tagging itself).
- since VLAN 1 and VLAN 9 are meant to be different broadcast domains,
 do they have different IP subnets?
- is there any other device (besides the firewall) that has each
 a "leg" into VLAN 1 and VLAN 9? Make sure that it does not "bridge"
 nor "route" between these VLANs.


regards

Marc

___________________________________________________________________________
Sent via:    Wireshark-users mailing list <wireshark-users@xxxxxxxxxxxxx>
Archives:    http://www.wireshark.org/lists/wireshark-users
Unsubscribe: https://wireshark.org/mailman/options/wireshark-users
            mailto:wireshark-users-request@xxxxxxxxxxxxx?subject=unsubscribe