Wireshark · Wireshark-dev: Re: [Wireshark-dev] Fuzz-test.sh crashes that can't be reproduced

[Nora Sandler <nsandler@xxxxxxxxxxxxxxxxxxxxxx>]

Hi Peter,

I did take a look at a few different core dumps. I don't *think* it's running out of memory - looking at the backtraces, it sometimes fails when deallocating, sometimes when allocating. And the tool's memory usage seems to stay pretty constant.

Here's a backtrace where it fails on allocation:

Core file '/cores/core.10792' (x86_64) was loaded.
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #1: tid = 0x0000, 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160, stop reason = signal SIGSTOP
  * frame #0: 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160
    frame #1: 0x00007fffbb152131 libsystem_malloc.dylib`malloc_zone_check + 58
    frame #2: 0x00007fffbb151e8a libsystem_malloc.dylib`internal_check + 17
    frame #3: 0x00007fffbb14231d libsystem_malloc.dylib`malloc_zone_malloc + 86
    frame #4: 0x00007fffbb1412b0 libsystem_malloc.dylib`malloc + 24
    frame #5: 0x000000010da0e746 libglib-2.0.0.dylib`g_malloc + 24
    frame #6: 0x000000010fbda70d libwireshark.0.dylib`wmem_strict_alloc(private_data=0x00007fbc0ed02bf0, size=40) + 29 at wmem_allocator_strict.c:93 [opt]
    frame #7: 0x000000010fbd8d58 libwireshark.0.dylib`wmem_alloc0 [inlined] wmem_alloc(allocator=<unavailable>) + 27 at wmem_core.c:58 [opt]
    frame #8: 0x000000010fbd8d3d libwireshark.0.dylib`wmem_alloc0(allocator=<unavailable>, size=40) + 13 at wmem_core.c:66 [opt]
    frame #9: 0x000000010f195030 libwireshark.0.dylib`proto_register_erf [inlined] erf_meta_tag_info_new(allocator=<unavailable>, tag=0x00000001109f4bb0) + 13 at packet-erf.c:902 [opt]
    frame #10: 0x000000010f195023 libwireshark.0.dylib`proto_register_erf [inlined] init_tag_fields(hfri_table=0x00007fbc115694c0, ett_table=0x00007fbc11569590, tag=0x00000001109f4bb0) + 5 at packet-erf.c:1008 [opt]
    frame #11: 0x000000010f19501e libwireshark.0.dylib`proto_register_erf [inlined] init_meta_tags + 148 at packet-erf.c:1088 [opt]
    frame #12: 0x000000010f194f8a libwireshark.0.dylib`proto_register_erf + 362 at packet-erf.c:3269 [opt]
    frame #13: 0x000000010fb3ac2d libwireshark.0.dylib`register_all_protocols(cb=0x0000000000000000, cb_data=0x0000000000000000) + 33757 at register.c:2243 [opt]
    frame #14: 0x000000010eea4cd4 libwireshark.0.dylib`proto_init(register_all_protocols_func=(libwireshark.0.dylib`register_all_protocols at register.c:1465), register_all_handoffs_func=(libwireshark.0.dylib`register_all_protocol_handoffs at register.c:5689), cb=0x0000000000000000, client_data=0x0000000000000000) + 404 at proto.c:534 [opt]
    frame #15: 0x000000010ee87abf libwireshark.0.dylib`epan_init(register_all_protocols_func=(libwireshark.0.dylib`register_all_protocols at register.c:1465), register_all_handoffs_func=(libwireshark.0.dylib`register_all_protocol_handoffs at register.c:5689), cb=0x0000000000000000, client_data=0x0000000000000000) + 239 at epan.c:166 [opt]
    frame #16: 0x000000010d67b352 tshark`main(argc=<unavailable>, argv=<unavailable>) + 994 at tshark.c:902 [opt]
    frame #17: 0x00007fffbafc0255 libdyld.dylib`start + 1
    frame #18: 0x00007fffbafc0255 libdyld.dylib`start + 1

And one where it's failing on free:

Core file '/cores/core.10458' (x86_64) was loaded.
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #1: tid = 0x0000, 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160, stop reason = signal SIGSTOP
  * frame #0: 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160
    frame #1: 0x00007fffbb152131 libsystem_malloc.dylib`malloc_zone_check + 58
    frame #2: 0x00007fffbb151e8a libsystem_malloc.dylib`internal_check + 17
    frame #3: 0x00007fffbb143feb libsystem_malloc.dylib`free + 358
    frame #4: 0x00000001057a88ff libwireshark.0.dylib`wmem_strict_free_all [inlined] wmem_strict_free(private_data=<unavailable>) + 47 at wmem_allocator_strict.c:139 [opt]
    frame #5: 0x00000001057a88dc libwireshark.0.dylib`wmem_strict_free_all(private_data=0x00007fa42650d670) + 12 at wmem_allocator_strict.c:194 [opt]
    frame #6: 0x00000001057a6efa libwireshark.0.dylib`wmem_destroy_allocator [inlined] wmem_free_all_real(allocator=0x00007fa426527eb0, final=1) + 26 at wmem_core.c:118 [opt]
    frame #7: 0x00000001057a6ee9 libwireshark.0.dylib`wmem_destroy_allocator(allocator=0x00007fa426527eb0) + 9 at wmem_core.c:137 [opt]
    frame #8: 0x00000001057a9cdf libwireshark.0.dylib`wmem_cleanup_scopes + 95 at wmem_scopes.c:159 [opt]
    frame #9: 0x00000001049edf4b tshark`main(argc=<unavailable>, argv=<unavailable>) + 12251 at tshark.c:2087 [opt]
    frame #10: 0x00007fffbafc0255 libdyld.dylib`start + 1
    frame #11: 0x00007fffbafc0255 libdyld.dylib`start + 1

Thanks,
Nora

> 
> Hey Nora,
> 
> On Wed, Dec 07, 2016 at 04:07:45PM -0800, Nora Sandler wrote:
>> Hi list!
>> 
>> I'm trying to do some fuzzing with fuzz-test.sh, and I'm seeing some strange behavior that I hope someone can help me figure out. For every pcap I try, I get a crash pretty quickly, usually in less than 10 passes. But then I can't reproduce the crash using test-captures.sh and the fuzzed output file. 
>> 
>> I'm seeing this behavior using my own pcaps as well as captures from https://wiki.wireshark.org/SampleCaptures#Sample_Captures. I'm running the latest code from the master branch on OS X 10.12. It seems like it's heap corruption related since I stop getting crashes if I comment out the following lines test-common.sh: 
>> 
>> export MallocCheckHeapStart=1000
>> export MallocCheckHeapEach=1000
>> 
>> Here's some sample output:
>> 
>> Fuzzing:
>> 
>> $ ./tools/fuzz-test.sh -b ./build/run/ ~/Downloads/dhcp.pcap
>> expr(15011,0x7fff9ab623c0) malloc: protecting edges
>> expr(15011,0x7fff9ab623c0) malloc: enabling scribbling to detect mods to free blocks
>> expr(15011,0x7fff9ab623c0) malloc: checks heap after 1000th operation and each 1000 operations
>> expr(15011,0x7fff9ab623c0) malloc: will abort on heap corruption
>> [...]
>> mv(15166,0x7fff9ab623c0) malloc: will abort on heap corruption
>> ./tools/fuzz-test.sh: line 203: 15155 Segmentation fault: 11  (core dumped) "$RUNNER" $COMMON_ARGS $ARGS $TMP_DIR/$TMP_FILE > /dev/null 2>> $TMP_DIR/$ERR_FILE.$SUBSHELL_PID
>> [...]
>> 
>> ERROR
>> Processing failed. Capture info follows:
>> 
>>  Input file: /Users/me/Downloads/dhcp.pcap
>>  Output file: /tmp/fuzz-2016-12-07-15007.pcap
>>  Pass: 6
>> [...]
>> stderr follows:
>> 
>> cat(15181,0x7fff9ab623c0) malloc: protecting edges
>> cat(15181,0x7fff9ab623c0) malloc: enabling scribbling to detect mods to free blocks
>> cat(15181,0x7fff9ab623c0) malloc: checks heap after 1000th operation and each 1000 operations
>> cat(15181,0x7fff9ab623c0) malloc: will abort on heap corruption
>> Input file: /Users/me/Downloads/dhcp.pcap
>> 
>> Build host information:
>> Darwin my-machine.local 16.1.0 Darwin Kernel Version 16.1.0: Thu Oct 13 21:26:57 PDT 2016; root:xnu-3789.21.3~60/RELEASE_X86_64 x86_64
>> [...]
>> Return value:  0
>> 
>> Dissector bug:  0
>> 
>> Valgrind error count:  0
>> [...]
>> 
>> And trying to reproduce the crash:
>> 
>> $ ./tools/test-captures.sh -b ./build/run /tmp/fuzz-2016-12-07-15007.pcap
>> Testing file /tmp/fuzz-2016-12-07-15007.pcap...
>> - with tree... sh(15256,0x7fff9ab623c0) malloc: protecting edges
>> [...]
>> OK
>> [...]
>> - without tree... sh(15262,0x7fff9ab623c0) malloc: protecting edges
>> [...]
>> OK
>> - without tree but with a read filter... sh(15268,0x7fff9ab623c0) malloc: protecting edges
>> [...]
>> OK
>> 
>> Is this an actual memory corruption bug in wireshark? A problem with the fuzzing script? Or am I doing something wrong? I'd appreciate any suggestions you have.
>> 
>> Thanks,
>> Nora Sandler
>> 
> 
> Is it possible that the tool consumes a lot of memory, eventually
> running out of memory and failing on allocations? Can you try to obtain
> a coredump for post-mortem analysis?
> -- 
> Kind regards,
> Peter Wu
> https://lekensteyn.nl