Wireshark-users: Re: [Wireshark-users] Experiencing Packet Loss in High Volume Packet Capture App
From: Jeff Morriss <jeff.morriss.ws@xxxxxxxxx>
Date: Mon, 26 Nov 2012 17:08:10 -0500
John Powell wrote:
Hi Jeff,

Thanks for your input - sorry about the Microsoft document - for future reference - what type of document would suggest using to detail such information?

No worries, just letting you know...

Image files (JPEGs, PNGs) work well as do PDFs.

Up until a few months ago the vendors we looked at were unable to provide the flexibility of my solution using wireshark.

Now one vendor appears to be able to come some ways to what we need, the problem is the the vendor can not decode one of our VoIP signalling protocols because it is proprietary even though Wireshark does a decent job. For the SIP protocols the vendor solution will work.

The problem I have is that the RTP packets come from the same gateway for both SIP and the proprietary protocol so I can not strip them from being captured.

That being said, I am still interested in any suggestions to may have to mitigate the issue.

Well, given your disk utilization, I'd say that speeding up your disks somehow would be a good first step. Some ideas (these from someone who has never paid much attention to this kind of thing except from mild interest):

- Are your disks 15K RPM (or do they have faster than that now?)? Interestingly, I'm pretty sure I saw the CACE Tech (now Riverbed) guys saying SSDs are NOT good for this kind of thing.

- Are your disks being used for anything else besides capturing? At your utilization rates if the disk is also used for the OS, simple paging ("hey, someone ran 'awk', I've got to go load that program from the disk to the memory!") would cause the disk to do something other than focus on writing as fast as it can.

- Try RAID (0 or 10?).

- I'm sure there's something to be said about the disk's transport (SATA vs SAS vs. whatever else) but I wouldn't know what that thing would be ;-).

- Do some file system research: I'm sure someone's done performance comparisons. If there's a difference in sustained write speed, choose the fastest one.

Regards,
-Jeff

On Mon, Nov 26, 2012 at 2:49 PM, Jeff Morriss <jeff.morriss.ws@xxxxxxxxx <mailto:jeff.morriss.ws@xxxxxxxxx>> wrote:

    John Powell wrote:

        Hi Everyone,

        I am running CentOS 6.3 on a HP 8200 using 3TB WD Green drives
        using a EXT4 file system.

        I am using Wireshark 1.8.2 compiled from source.

        I am using DUMPCAP to rotate and store historical Packet Captures.

        Whether I capture the packets with Wireshark or view the DUMPCAP
        created file, I see dropouts in the packets being captured.

        I tried to turning off journalling but this did not seem to help
        much:

        umount /dev/mapper/VolGroup00-LogVol___Data

        /sbin/tune2fs -o journal_data_writeback
        /dev/mapper/VolGroup00-LogVol___Data

        /sbin/tune2fs -O ^has_journal /dev/mapper/VolGroup00-LogVol___Data

        /sbin/e2fsck -f /dev/mapper/VolGroup00-LogVol___Data


        I have a attached a couple of IOGraphs from Wireshark showing
        the packet drops.


    (Note that Microsoft documents aren't the most portable way of
    sharing...  Many of us don't natively have a way to open them.
    Fortunately, Google frequently can...)

    The document indicates that your disks are 71% busy writing about 38
    Mbytes/sec and that you're periodically getting periods where almost
    *nothing* is captured and that those periods can be quite long (in
    one case it looks like about 500 msec).

    In my mind, you're crossing into the territory where a dedicated
    capture device (which has been engineered for this kind of
    high-speed capture) is needed.  You may be able to make some
    progress but you'll be reinventing a wheel that's already been
    solved (probably with much effort) by several vendors.