Wireshark · Ethereal-dev: Re: [Ethereal-dev] Performance. Ethereal is slow when using largecaptures.

["Ronnie Sahlberg" <ronnie_sahlberg@xxxxxxxxxxxxxx>]

Good point.
I never noticed that the number of sessions affected the refilter speed.
The type of traffic (some reassembled PDUs take much longer than others to
dissect) I know would affect it but never knew the number of sessions
did.

The problem now is that fore VERY large captures, ethereal is always slow
under all circumstances.
So let us start with just a simple random generic capture and measure for it
to try to keep the number of variables low.
(If it is as you say the number of sessions affect it as well,  do you mean
the number of TCP sessions or what kind of sessions?
 At some point, when the worst performance problem has been addressed this
would be a very interesting area to look at.
 (I could create different synthetic capture files to measure with,   same
number of packets, same payload just different number of sessions)
 Make a note that you have observed the number of sessions to possibly have
an effect on the dissection speed so we dont forget to look at
  it furhter down the track
)


I currently belive that during refiltering of a capture, most time would be
spent inside file.c/add_packet_to_packet_list().
It would be VERY VERY useful to verify that this assumption is correct.
I would really like someone to look at gprof data and analyze where most
time is consumed to either verify my claim  add_packet_to_packet_list()
or to invalidate it.

The thing inside this function I think consumes the most cpu I belive would
be where we call epan_dissect_run() and perform a full dissection of the
packet.


As I see it, apart from the initial time we encounter the packet during file
read (or live capture) there are not that many instances where we really
must
dissect the packet at all.
OK. If we select a packet in the list so it gets displayed in the dissect
pane that might be an exception but that is not something that we do 100.000
times
per capture anyway so the performance of that is irrelevant.
We might also need to do a full rescan/redissect of all packets IF we have
changed the preferences in such a way that the packets will be dissected
differently  or when we have changed stuff using  DecodeAs.

However, for me and many other users, the MAIN reason ethereal rescans the
packet list is because we have applied or changed a filter. Some users will
filter and refilter a capture file over and over and over, ten, twenty,
thirty if not more times for each capture they work with.
Or see when a ConversationList dislog or a ServiceResponseTime dialog is
opened.


Well enough of that. To my idea:

Hypothesis:  A significant part of the slowness of ethereal when refiltering
a capture file comes from the expensive calls to epan_dissect_run() called
from add_packet_to_packet_list() in file.c
Potential fix: Reduce the number of calls made to epan_dissect_run() at the
expense of additional memory requirements (enabled by a preference)

Assuming that most of the time we perform a full rescan/redissect of the
capture file is when we really just want to reapply a display filter. (and
are not doing anything that affects how a packet is dissected).

What do we need in order to refilter the packet list  if we do not allow
calling epan_dissect_run()?
1, We need to remember all COL values for all packets so that we can just
reapply them when adding the packet to the packetlist without calling the
dissector and recreating them that way.   This will consume additional
memory.
2, For every packet we need to keep a list of all the hf_fields that were
encountered in the packet.
    This list contains the index of the hf variable as well as the value it
has.
    Nothing else needs to be stored there (in order to reduce the impact on
memory)
    This list may NOT be pruned as the edt structs are. This is because we
want to be able to still use this list even after the filters have changed
and thus
the pruning would be different.   No pruning.
The "ApplyFilterToEdtStructure" fucntions would need to be changed (or
duplicated) so they could operate on the list in 2 instead of the edt
structure.
This function might also need to be looked at so that it would be efficient
even for very large lists (no pruning)


1 would allow us to rebuild the packet list without needing to call the
dissector (?)
2 would allow us to refilter the entire trace without calling any
dissectors.

ideas, comments?


Right now it would be nice if someone could create a capture as I proposed
earlier and use GPROF to check where most of the CPU is spent when
refiltering the capture. To verify if my assumptions are correct or
invalidate them.


(
As a nice benefit in the future, IF we were to have that list of fields for
each packet, easily available, we could do things like merging this list
between packets.
Say #6 is the Call and #27 is the Response.
Since these packets are paired we could merge the lists from these two
packets into a single one.
Then when searcing for something that occured in the Response packet, we
would automatically also pick up the matching Call packet sinte their lists
were merged.
I.e filtering for smb.error==foo   would both find the Response that barfed
saying foo  but also teh matched Call to this Response.
That would also be useful.
)


----- Original Message ----- 
From: "Kevin"
Sent: Saturday, November 15, 2003 12:02 PM
Subject: Re: [Ethereal-dev] Performance. Ethereal is slow when using
largecaptures.


> An observation on performance with large files.
>
> I have noticed that trace files with a large number of sessions or
> flows take longer to load than a similar trace with just a few
> connections.  This makes sense because of the work needed for each
> flow.  As part of this testing, should we add the number of connections
> as well as number of frames to the mix?
>
> As part of this testing, we should use a series of standard traces that
> let us test and compare various configurations.  I would be happy to
> contribute some traces.
>
> Kevin
>
> On Friday, November 14, 2003, at 05:02 PM, Ian Schorr wrote:
>
> > Hi Ronnie,
> >
> > This is great, sounds like very important work towards the usability
> > of Ethereal.  The captures I work with on a daily basis tend to be
> > fairly large (200,000+ frames, 800,000 or more per capture isn't
> > unheard of), and speed is by far my #1 concern at the moment.
> >
> > I'm not sure what I can do, but I'd love to help however I can -
> > perhaps with the gprof stuff?
> >
> > Ian
> >
>
> _______________________________________________
> Ethereal-dev mailing list
> Ethereal-dev@xxxxxxxxxxxx
> http://www.ethereal.com/mailman/listinfo/ethereal-dev
>