Wireshark · Wireshark-dev: [Wireshark-dev] layers of reassembly

["Julian Cable" <julian.cable@xxxxxxxxx>]

Hi,

  I'm trying to reassemble packets meeting the ETSI ES 201 980 Packet Mode specification:

====================================
6.6 Packet mode

Data services generally consist of either streams of information, in either synchronous or asynchronous form, or files of
information. A generalized packet delivery system allows the delivery of asynchronous streams and files for various
services in the same data stream and allows the bit rate of the (synchronous) data stream to be shared on a
frame-by-frame basis between the various services. Services can be carried by a series of single packets or as a series of
data units. A data unit is a series of packets that are considered as one entity with regard to error handling - one received
errored packet within a data unit causes the whole data unit to be rejected. This mechanism can be used to transfer files
and also to allow simpler synchronization of asynchronous streams. The carriage of data applications is described in
TS 101 968 [8].

The size of a packet mode data logical frame shall be a multiple of the packet size. The maximum length of a data unit
is 8215 bytes.

6.6.1 Packet structure
The packet is made up as follows:
- header 8 bits.
- data field n bytes.
- CRC 16 bits.

The header contains information to describe the packet. The data field contains the data intended for a particular service. The length of the data field is indicated by use of data entity 5, see clause 6.4.3.6. Cyclic Redundancy Check (CRC): this 16-bit CRC shall be calculated on the header and the data field. It shall use the generator polynomial G16 (x) = x16 + x12 + x5 + 1 (see annex D).

6.6.1.1 Header

The header consists of the following fields:
- first flag 1 bit.
- last flag 1 bit.
- packet Id 2 bits.
- Padded Packet Indicator (PPI) 1 bit.
- Continuity Index (CI) 3 bits.

The following definitions apply:

First flag, Last flag: these flags are used to identify particular packets which form a succession of packets. The flags are assigned as follows:
First flag Last flag The packet is:
  0            0 : an intermediate packet;
  0            1 : the last packet of a data unit;
  1            0 : the first packet of a data unit;
  1            1 : the one and only packet of a data unit.
Packet Id: this 2-bit field indicates the Packet Id of this packet.
Padded Packet Indicator: this 1-bit flag indicates whether the data field carries padding or not, as follows:
  0: no padding is present: all data bytes in the data field are useful;
  1: padding is present: the first byte gives the number of useful data bytes in the data field.

Continuity index: this 3-bit field shall increment by one modulo-8 for each packet with this packet Id.

6.6.1.2 Data field

The data field contains the useful data intended for a particular service. If the Padded Packet Indicator (PPI) field of the header is 0, then all bytes of the data field are useful bytes. If the PPI is 1 then the first byte indicates the number of useful bytes that follow, and the data field is completed with padding bytes of value 0x00.

Packets with no useful data are permitted if no packet data is available to fill the logical frame. The PPI shall be set to 1
and the first byte of the data field shall be set to 0 to indicate no useful data. The first and last flags shall be set to 1. The
continuity index shall be incremented for these empty packets. If less than 4 sub-streams are used within the data stream
then an unused packet id shall be used. Empty packets using a packet id of <p> shall not be inserted during the
transmission of a DRM data unit using the same packet id <p>.
=====================================================

The packet id is a stream multiplexing feature, it is the same for all packets in a stream.

There are no per-Data Unit sequence numbers, the continuity index is a running sequence number across all packets. All the packets making up a data unit must be in sequence with no other packets with the same packet id in-between.

This protocol is carried on top of other protocols, including UDP. A Data Unit can span multiple ethernet frames and there can be zero or more packets in each ethernet frame.

I tried using fragment_add_seq_next but it gets confused. I looked in reassemble.c and it uses frame-layer state (the "visited" flag). This is part, but not all of the problem. I looked at the only other code that uses fragment_add_seq_next in stream.c and its very complicated.

I thought of using a conversation per fragmented Data Unit - is this appropriate?

The next layer up (MOT) that I haven't started yet, will also need its own reassembly functions, so I need something that can work "layer clean".

Do you have any advice on this?

Julian Cable