Wireshark · Ethereal-dev: Re: [Ethereal-dev] RFC framework for graphical extensions like t he recent rate_

["Ronnie Sahlberg" <sahlberg@xxxxxxxxxxxxxxxx>]

Follow up on the comments to my previous mail.
Basically I saw two major preceived problems with the design:

1, filtering
Not sufficiently advanced filtering.
True, by only feeding what the display filter shows to the graph extension
is not infinitely
configurable. You can not have different filters for different graphs.
This can be added and hidden inside the infrastructure at a later stage.
Adding this feature does not need
any changes in the interface to the dissectors or the extensions.

2, GNUplot
GNUplot is not infinitely configurable. However, it is VERY easy to use for
the majority of uses that
only need simple graphical output and no interaction with it.
That GNUplot is external to ethereal is actually a very strong point and not
a weakness.
It would allow us to generate extensions WITHOUT tying up ethereal to GTK
forever.
By writing all the extensions using direct GTK calls, I fear that would
practically mean that the KDE/Qt/nativeW32/curses ports will NEVER happen.
But it does not have to be GNUplot, it is whatever program is places on the
other side of the pipe created in
(*start_collecting)(), it could just as well be Matlab,POV,... or something
else.

Even if MY examples focus on GNUplot, there is nothing stopping anyone from
writing something using only GTK calls and putting it inside
(*draw_graph)(), the design allows that.
The design is not meant for the people who WANTS to write everything with
direct GTK calls,
those people can develop those extensions anyway without any of this
support.
The design is to make it easy for someone with very basic C knowledge by
looking at an example extension
to develop a new but simple extension. It is aimed to help those who will
never be able to use GTK calls, or want a useful extension but are not
willing to do so if it takes more than 100 lines to code.

If you dont want GNUplot, well dont create a pipe, dont exec gnuplot. Put
everything yourself inside
(*draw_graph)()  Its your choice, you can do that if you want to. But you
dont have to.


I will try to hack together a prototype during the weekend so we can analyze
it.


----- Original Message -----
From: "Ronnie Sahlberg"
Sent: Thursday, June 27, 2002 9:28 PM
Subject: Re: [Ethereal-dev] RFC framework for graphical extensions like t he
recent rate_


> Hi,
>
> First let me state that I have read each and every followup on the initial
> post
> carefully and have gained many valuable insights and ideas.
> I especially want to thank Joerg Mayer for his first post. It gave me a
lot
> of insights on
> things I had not thought through properly.
>
> I will not respond to any post individually but will address the result of
> applying the more importnant
> (as i thought) points in this very mail. I may have forgotten some points,
> sorry for that.
>
> I will refer to every extension as graphs, eventhough it may be used for
> other things as well.
>
> First some short responses:
>
> Why not update the graph whenever a new packet arrives?
> Arbitrary desicion to reduce CPU load. I think it is enough if the graph
is
> updated once every second,
> if there is new data. I image on live captures one might have 1000s of
> packets coming in every second.
>
> Why not use e.g. GNUPlot to print the graphs instead of coding it in GTK?
> Actually an excellent idea. The rest of the proposed draft design will be
> based on using something that opens a pipe to write commands into GNUPlot
> and let gnuplot do all the drawing (which will then make sure the graphs
> will always look really nice and professional).
> Another advantage is that we could then start GNUPlot with nice-ing it to
> very low priority on each and every (unix) platform so that the graph
> updates will not steal valuable cpu cycles if ehtereal needs it for
> something more important. To me its preferable to have jerky updates of
the
> graph than to get higher packetloss if capturing.
>
> Why a separate thread to process data for the graphs and why low-priority?
> A thread would run in the same memory space as the main ethereal
> application, this means it would
> be able to take simple and cheap pointers as arguments to the
datacollection
> subroutine. This graph
> data collection routine would also be able to follow pointers if it needed
> to if we were to pass
> pinfo or whatever to it.
> I.e. we would get very cheap ipc (since we only pass a pointer and dont
need
> to marshal anything) and
> we would have maximum flexibility. We pass *pinfo and a *void, the
> collection thread could potentially
> access anything and everything inside ethereal. We thus do not create any
> restrictions limitations
> on what kind of data we can handla and what we can not.
> Why low-priority?  Well, data processing to update a graph window once
every
> second or so is not as timecritical as the rest of ethereal like capturing
> packets and updating the packetlist. Lowpriority
> would allow us to make sure the important things get the cpu when they
need
> it and the graph thingy
> gets the cpu if nothing else is happening.
>
> Different types of filtering?
> I thought without analyzing too closely on if the current implementation
> would allow it, that new packets/events
> would only be fed to whatever graph thingy if it first passed the
> displayfilter.
> Something like if(tree && !pinfo->fd->flags.visited){
> This would mean that the graphs would only be based on whatever packets
the
> displayfilters would pass.
> Entering the displayfilter "ssh"  would automatically change all graphs to
> only show statistics for "ssh"
> since only these packets would be sent to the graph thingies.
> No new filtering, just apply proper displayfilters and the graphs will do
> the right thing.
>
> We should not write too much new code!
> Agreed. Especially lotsa code for one specific toolkit and at the same
time
> killing off KDE/Qt/W32native/Curses
> ports. Definitely. I am ashamed I did not see this, perhaps the most
> important requirement.
> I would therefore propose that NO widget specific code is implemented in
> ethereal, instead the graphies should
> only process the data and push it down a pipe to GNUplot and let GNUPlot
> draw it.
> This also applies to the request to not add lots of new bloat to ethereal.
> By offloading all the graphics manipulation to external tools (that does
> these thing really well) the amount
> of bloat will be limited. I still will aim at keeping the interface clean
so
> it can easily be changed to dlopen() stuff
> for thse that dont want any bloat at all.
>
>
>
> Ok, new proposal:
> All graphs are managed by separate GNUPlot processes. Ethereal will
> communicate to teh command interface
> by writing commands to a pipe to the cli of gnuplot. This pipe is one-way.
> Ethereal graphie thing writes and gnuplot reads.
>
>
> The dissectors are updated in the following way:
> -------------------------------------------------
> Each dissector (as we need to) has the following two new changes:
> proto_reg_handoff_xxx
>      will also do   tap_handle=register_packet_tap("xxx");
> dissect_xxx
>      the main dissection function for the protocol will add, after the
> entire packet is dissected (and all structures
>      updated and all child dissectors have returned) something like this
> just before they return :
>      if(tree && !pinfo->fd->flags.visited){
>            tap_packet(tap_handle, packet_info *pinfo, void
> *protocol_specific_structure);
>      }     /* better names for the functions are WELCOME */
>
>
> register_packet_tap() is used to tell the infrastructure that this
dissector
> can be tapped.
> tap_packet() will then call each datacollector subscribing to this tap in
> turn. For this it is importnant to
> keep these datacollectors as lightweight as possible.
>
> So, the API between the dissector and the wanted infrastructure is the two
> functions
> tap_handle *register_packet_tap(char *tap_name)
> void tap_packet(tap_handle *hnd, packet_info *pinfo, void
> *protocol_specific_data)
>
> We can enhance dissectors on a need basis, only addint this functionality
to
> it whenever there is an extension that needs it.
>
>
> magic infrastructure inside ethereal
> -----------------------------------
> the infrastructure will make all of it work and is what we must implement.
> it is not important though what it looks
> like. what is important is the interface to use it so people can use it.
>
>
> The datacollection and graph processing units. I.e. the extensions
> -----------------------------------------------------------------
> The extensions are just a bunch of subroutines. These are not implemented
as
> threads. If need be for separate threads running in parallell to the main
> ethereal thread this fact can be hidden in the infrastructure above in
that
> case.
> The following functions are required for an extension (No other calls
across
> main ethereal nor global variables are allowed to make it easy to dlopen()
> convert it later to reduce bloat)
> (you dont like the name graph? neither do I, come up with something
better)
>
> The extencion .c file should contain a function
> register_graph_xxx()
>     used to initialize the extension. This function is called in some
manner
> (initially similar to register.c since it is
>     easy?) once when ethereal starts to initialize all state and prepare
the
> extension for action.
>     One thing it would need to do would be to register itself as a
> datacollector with the infrastructure with
> register_data_collector(
>     "FrameSizeHistogram",  /* Name of this extension as it will appear in
> the tools menu where on/off is toggled */
>     void(*start_collecting)(void),
>     void(*stop_collecting)(void),
>     void(*reset_counters)(void),
>     void(*draw_graph)(void)
> );
>
> The first two functions are called whenever the menuitem is ticked or
> unticked:
> start_collecting()   will clean up all state variable (by calling
> reset_counters?) and then do things like
>     starting GNUPlot with an open pipe to ewwrite commands to it.
>     When all state is prepared we will here also register which taps we
want
> to listen to by calling
>     open_tap(
>         "frame",   /* name of the tap when it was created in teh dissector
> above */
>         col_handle=(*collect_data)(packet_info *pinfo, void
> *protocol_specific_data)
>     )
>     collect_data is what is called whenever tap_packet() is called in the
> frame dissector.
> stop_collecting() will kill gnuplot and clean up all state.
>     It will also remove all taps we have placed by calling
>     close_tap(col_handle);
>     which will stop tap_packet() from sending more stuff out way.
>
> We can thus have multiple collect_data() functions in one and the same
> extension, in case we want to
> assemble data from multiple protocols in teh same thingy. (protocol
> statistics replacement?)
> collect_data() is called whenever tap_packet() is called from the
> dissector(s) we have registered
>     we want to subscribe to.
>     For each and every frame that passes the display filter, we will get
one
> call to collect_date() for each
>     protocol we have opened the tap to.
>     collect_data() should be as cheap as possible not to interfer with the
> important parts of ethereal.
>
> void (*reset_counters)(void)
>    this function is called whenever the infrastructure wants us to reset
all
> state and all counters (but let whatever
>    GNUPlot process or pipe we have remain active. To be called when the
> capture is restarted or rescanned.
>
> For ALL the functions above, the infrastructure will guarantee that they
> will never be called concurrently.
> The following, last, function may be called concurrently to any of those
> above.
> void (*draw_graph)()   If there has been packets coming from a specific
tap
> during the last second or so, then
>    the infrastructure will call this function to get it to (ask gnuplot
to)
> redraw the window.
>    This function would be called for anyone that has had packets coming
> their way about once a second or so.
>    This part of the infrastructure is a prime candidate to implement as a
> low-priority thread.
>
>
>
>
> Thats the updated design proposal. Its difficult to explain properly since
> its all in my head but i think it is improving a lot compared to the
initial
> design.
> This design feels quite good and right.
> I will await anyones comments on this new design.
> If it gets clean and easy enough to use we might get a lot of people
> producing extensions.
>
> As a test to see if the final design will work I propose we do two simple
> extensions such as
> FileSizeHistogram
> and something that prints similar output as
> nfsstat
> does.
> These need not look great or be used at all in other ways than to just
> verify that the design works and the API
> is easy to use.
>
>
> best regards
>     ronnie sahlberg
>
>
>
> _______________________________________________
> Ethereal-dev mailing list
> Ethereal-dev@xxxxxxxxxxxx
> http://www.ethereal.com/mailman/listinfo/ethereal-dev