Wireshark · Ethereal-users: RE: [Ethereal-users] duplicate packet removal

[Alistair.McGlinchy@xxxxxxxxxxxxxxxxxxxxx]

Title: Message

John,
 
>  I am assuming that frames are unique, and a retransmission will be different to the original packet.

 

I'm not so sure this assumption is correct. Try tracing a web request to a server that doesn't talk port 80. You'll probably see three outbound SYN  frames  (and three RSTs) .  These frames will be identical (including IP and TCP checksums).   There are also numerious services running on my local network (esp MAC layer protocols such as ARP) which contiually squirt the same or similar frames onto the LAN all the time. This may or may not be important to you but goes some way to explains why this is a non-tirivial task.

 

> It might be a simple thing to do to modify the code to keep a circular buffer of src, dst, CRC
> and then scan it for a duplicates before allowing a packet to be merged into the output file.
> The length of the buffer will be the correlation window for the duplicate packet check.

 
> Read from input files
>    Pick most recent packet
>    Check against buffer
>    If in buffer, then ignore
>    If not in buffer
>        put src, dst, CRC into buffer
>        write to output file
> Loop until all input files read

 

Even if you are confindent the duplicates listed above are not a problem for you. The algorithm will need a little more work.  What is going to prompt you to shift frames out of your buffer (size, time, # of frames?) . Also, the TCP layer checksum is only two bytes long (=65536 possible checksums). This gets you into the shared birthday problem  

 http://www.cut-the-knot.com/do_you_know/coincidence.shtml . There's a 50% chance of a duplicate checksum every 301 or so packets.  So you'd need to compare a little more than the TCP checksum. Depending on how large your trace is I'd probably opt for storing the whole frame and comparing the whole packet.

 

Nastiest of all IMHO you need to make a decision on the frame's timestamp.(Again maybe this isn't important to you...)  Which  all leads back to my last question:
>> It may be easier to advise you if we understood what sort of packets were being duplicated? Eg is it MAC 

>> level broadcasts, RIP updates etc, HSRP hellos ?   

 

You say you are running "capture processes on one system". Does this mean
1) you're tracing two NICs on the same machine,
2) the same NIC with different filters,
3) or your're running the same filter on the same NIC with different trace start times.

 

If the answer is #3 (and depending on the filter #2 too) you will capture exactly the same frames in exactly the same order with the same relative time stamps, so your algorithm will be fine, in fact you needn't worry about a buffer at all. For anything else you'll need to put your thinking hat on. :-)

 

That having been said I'm quite tempted to see if I can write something that would do this sort thing.At least until I convince myself it really is too tricky a problem, :-)

 

Cheers,

 
Alistair

---

Registered Office:
Marks & Spencer p.l.c
Michael House, Baker Street,
London, W1U 8EP
Registered No. 214436 in England and Wales.

Telephone (020) 7935 4422
Facsimile (020) 7487 2670

www.marksandspencer.com

Please note that electronic mail may be monitored.

This e-mail is confidential. If you received it by mistake, please let us know and then delete it from your system; you should not copy, disclose, or distribute its contents to anyone nor act in reliance on this e-mail, as this is prohibited and may be unlawful.

The registered office of Marks and Spencer Financial Services PLC, Marks and Spencer Unit Trust Management Limited, Marks and Spencer Life Assurance Limited and Marks and Spencer Savings and Investments Limited is Kings Meadow, Chester, CH99 9FB.