Ethereal-dev: [Ethereal-dev] Re: NTAR - PCAP next generation dump file format implementation
If I understand well, what you ask is a way to "group" a set of packets
to quickly navigate among them. In this case, ntar already has a
solution to the problem: the file format subdivides the capture in
sections, whose number and size is controlled by the calling program (or
library, in case of libpcap) using the ntar_create_section() and
ntar_close_section(). It is possible, for example, to close the current
sections and create a new one after 10.000 packets. When reading the
file ntar_get_next_section() or ntar_get_previous_section() can be used
to navigate through the sections and reach the desired one.
Regarding advanced ways to speed-up Ethereal capture handling, this is
the precise goal of the directory block
(http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html#anchor15),
which however is not been implemented yet.
Loris
ronnie sahlberg wrote:
The draft describes two different blocks for holding a captured packet:
Packet Block: it contains a single captured packet, or a portion of it.
Simple Packet Block: it contains a single captured packet, or a
portion of it, with only a minimal set of information about it.
I often work with very very large capture files and often want to only
extract a very small subset (packets captured between time X and time
Y).
This is very very slow with the current fileformats doe to the massive
amount of data that has to be processed.
I would propose to introduce a new type of block and encapsulate both
PacketBlock and SimplePacketBlock in it :
Packet
======
This the block you refer to in the draft as PacketBlock. A Packet MUST
be encapsulated inside a PacketBlock.
SimplePacket
============
This the block you refer to in the draft as SimplePacketBlock. A
SimplePacket MUST be encapsulated inside a PacketBlock.
Then introduce a new PacketBlock :
PacketBlock
===========
A block that contains one or (preferrably) more Packet or SimplePacket
structures.
Format of PacketBlock (in units of 4 bytes):
Version: 1 byte, version of packetblock structure.
Num packets: 3 bytes, number of Packet or SimplePacket structures in
this PacketBlock, or zero (==use offline tool to calculate this and
populate the field)
Length : 8 bytes, total length of this block in bytes.
TSofFirstPacket: timestamp of the first packet or simplepacket in this
PacketBlock or zero (==run some offline tool to populate this field if
you need it)
TSofLastPacket: timestamp of last packet or simplepacket in this
PaketBlock or zero.
A PCAP writer should aggregate PacketBlocks/SimplePacketBlocks into
blocks of approximately 10MByte.
This default approximate block-size should be controllable by the user
but i think 10MByte should be reasonable.
Assuming we have a 10GByte capture file, then this would make it possible to :
1, parse the cap file and collect all PacketBlock headers, timestamps
and number of packets in each block. fseek() to the next block
instead of parsing it packet by packet.
2, present to the user a list :
Block 1: start:x:y stop:x:y packets:z
Block 2: ...
...
3, Allow the user to ONLY read/extract a subset of the PacketBlocks in
teh capture file.
Having this kind of encapsulation structure would allow users to use
large captures (and skip the portions that are of no interest) while
the current design would require the user to read the entire 10GB
file.
1GB and larger capture files are becoming more and more standard to
many users and reading through this massive amount of data is slow and
painful.
On 6/25/05, Gianluca Varenni <gianluca.varenni@xxxxxxxxx> wrote:
Hi all.
This mail is to announce the birth of the NTAR project. NTAR stands for
Network Trace Archival and Retrieval library, and is an implementation of
the PCAP next generation dump file format, that was proposed and discussed
last year by several folks on the libpcap/tcpdump and WinPcap mailing
lists.
The library is released under the 3-clause/BSD license.
The URL of the project is:
http://www.winpcap.org/ntar
On this website you can find
- the source file of the library (both for windows and *nix)
- the HTML documentation of the API (generated with doxygen from the
commented source files), both for the user and for someone wanting to
extend it. The docs contain some tests/examples that you can look to
get an idea of how the library works.
- An updated version of the PCAP draft specifying the file format.
The original draft of the file format is available at
http://www.tcpdump.org/pcap/pcap.html
I'm seeking contributors to improve the library in terms of
- testing the library on different platforms and operating systems
- reviewing the API
- implementing new extensions to the library.
A mailing list, ntar-workers@xxxxxxxxxxx, has been created for NTAR-related
discussions. People interested in this project are welcome to join it, the
mailman web interface to subscribe is available at
https://www.winpcap.org/mailman/listinfo/ntar-workers
Have a nice day
Gianluca Varenni
_______________________________________________
Ethereal-dev mailing list
Ethereal-dev@xxxxxxxxxxxx
http://www.ethereal.com/mailman/listinfo/ethereal-dev
_______________________________________________
Ethereal-dev mailing list
Ethereal-dev@xxxxxxxxxxxx
http://www.ethereal.com/mailman/listinfo/ethereal-dev
