Wireshark · Wireshark-dev: Re: [Wireshark-dev] Reduce memory consumption by re-reading data from file for reassembled packets.

[Jeff Morriss <jeff.morriss.ws@xxxxxxxxx>]

Jakub Zawadzki wrote:
> On Wed, Nov 03, 2010 at 03:00:10PM -0400, Jeff Morriss wrote:
> > Memory mapped files count as part of the process' address space so 
> > mapping the capture file means that, for reassembly, we'd be trading 
> > malloc'd memory for mmap()'d memory *and* it would mean we're holding 
> > the whole file--including the bits not used in reassembly--in memory. 
> > (We don't currently do this.)
>
> Well I'm not expert in mmap()'ed files, but afair they won't be read until page fault, 

They may not be loaded into memory, but AFAIK the mapped memory will 
immediately count towards the process' (used) address space.

> and madvise(..., MADV_DONTNEED) should free memory.

Never heard of that until now; interesting...  But DONTNEED just says "I 
won't need it soon" (so feel free to page it out--but that won't reduce 
our used address space).  REMOVE would mean "unmap the memory" but the 
Linux man page says that only works with shmfs/tmpfs--and anyway we will 
probably need it again.  And I'm guessing it's not portable (the Linux 
man page makes a reference to Posix 2001).

> Still it's lot easier to work with than fread() + fseek()
>
> > Memory-mapped files also are problematic when doing live captures: as 
> > the file grows we'd have to remap the memory (yikes!)
>
> Ok, I agree this is problem. But why yikes?

If we attempt to remap we're not guaranteed that the mapping will land 
at the same address again.  If not, all our pointers into the map are 
now invalid.  There may be other problems too (if that's not enough ;-)).