Ethereal-dev: Re: [Ethereal-dev] Performance. Ethereal is slow when using large captures.
Not a programmer, but would be happy to help in anyway.
As to testing the speed, I would suggest that you specify the gprof,
ps and any other values you wish to capture.
Kevin
On Thursday, November 13, 2003, at 09:54 PM, Ronnie Sahlberg wrote:
Ethereal is very slow when working with very large captures containing
hundred of thousands of packets.
Every operation we do that makes ethereal rescan teh capture file taks
a
long time.
I would like to start looking at methods to make ethereal faster,
especially
for large captures trading a larger memory footprint
for faster refiltering speeds but would like to know if there are
others
interested in helping.
(you dont have to be a programmer, someone that knows how to run
gprof and
read the data would be very very helpful as well)
If there is interest maybe we should try to ask Gerald to set up an
additional mailinglist as well : ethereal-speed@xxxxxxxxxxxx but
that is a
question for later
depending on whether there is interest.
If there is interest in this, I propose we take 0.9.16 as baseline, and
measure the execution speed and memory footprint for ethereal
for this version to start with.
A good example capture might be setting snaplen to 100 and
downloading
ethereal from the website over and over until the capture file has
100.000
packets (10MB).
Create three color filters : "ip" "tcp" and "frame.pkt_len>80"
(random
filter strings from the top of my head)
Open the UPD and the TCP conversation list.
Load the capture into ethereal.
then measure the time it takes to refilter the capture as fast as
possible
10 times using the display filter "tcp.port==80"
After the 10th refilter of the trace, measure the memory footprint of
ethereal using ps.
Repeat this using a gzipped copy of the capture file.
This could be the baseline.
(the time to perform the test should be a couple of minutes at least
so that
we can see any differences in test implementations easier)
After this it would be good if someone did this test and collected
gprof
data to see where most of the cpu is spent.
I guess but am not sure it is in the actual dissection of the packet.
It would be great to have this verified.
Everything below this line assumes the theory that the dissection of
packets
are taking up a significant part of the time to refilter a packet.
Hypothesis: a lot of time is spent redissecting the packets over and
over
everytime we refilter the trace.
Possible Solution: reduce the number of times we redissect the packets.
(We would need a preference option called "Faster refiltering at the
expense
of memory footprint". Memory is cheap but CPU power is finite)
currently:
Everytime we rescan a capture file we loop over every single packet
and:
dissect the packet completely and build an edt tree (in file.c)
prune the edt tree to only contain those fields that are part of any
of the
filters we use (display filter, color filter, the filter used by taps
etc)
evaluate each filter one at a time using the now pruned edt tree.
discard the edt tree and go to the next packet.
possible optimization:
When we dissect the packet the first time, keep the full edt tree
around
and never prune it.
The loop would then be:
Loop over all the packets:
For this packet, find the unpruned edt tree (dont dissect the
packet at
all)
apply all filters one at a time on the unpruned edt tree.
This would at the expense of keeping the potentially very memory
consuming
edt tree around eliminate the need to redissect the packet.
The memory required by the edt tree could probably be optimized over
time.
The risk is, how will the time it takes to evaluate a filter be
affected by
the tree not being pruned? What is the execution time compared to
number
of fields held in the edt tree ? Ordo(n) or Ordo(n^2) ? Maybe we
would
have to do major brainsurgery there as well?
Maybe it turns out to be unfeasible? Who knows until we have tried.
Still, SOME things we do would require a redissect of the packet
anyway such
as when clicking on the packet in the packet list but that is not
performance critical.
Can anyone see any obvious flaws in the reasoning above which would
make it
unfeasible or the approach irrelevant?
Would anyone that knows gprof be interesting is doing the measurements
on
the "baseline" as suggested? and keep testing the same sample file
for all further performacne tests we do so we know if we are going in
the
right direction or not?
_______________________________________________
Ethereal-dev mailing list
Ethereal-dev@xxxxxxxxxxxx
http://www.ethereal.com/mailman/listinfo/ethereal-dev
