[0/3] Fix crash if connection drops in scoped_restore_current_thread's ctor

Message ID 20200708233125.1030-1-pedro@palves.net
Headers show
Series
  • Fix crash if connection drops in scoped_restore_current_thread's ctor
Related show

Message

Pedro Alves July 8, 2020, 11:31 p.m.
(I have internet again: found a sim card of a different operator that
works.  This will do until the communications tower near me is
repaired and get I fiber back...)

This series fixes the crashes exposed by the
gdb.multi/multi-target.exp testcase when run against an Asan-enabled
GDB build, initially reported by Simon here:

  https://sourceware.org/pipermail/gdb-patches/2020-July/170222.html

The first two patches fix the crashes, and we should probably put them
in GDB 10.

The last patch is a follow up that avoids swallowing exceptions in
scoped_restore_current_thread's dtor that I'm thinking would be a bit
too invasive to put in GDB 10, I think it could do with a longer
baking period in master.

Pedro Alves (3):
  Fix crash if connection drops in scoped_restore_current_thread's ctor,
    part 1
  Fix crash if connection drops in scoped_restore_current_thread's ctor,
    part 2
  Make scoped_restore_current_thread's cdtors exception free (RFC)

 gdb/blockframe.c            |  6 +---
 gdb/dwarf2/frame-tailcall.c | 18 +++++++++--
 gdb/frame.c                 | 73 ++++++++++++++++++++++++++++++-------------
 gdb/frame.h                 | 22 ++++++++++---
 gdb/gdbthread.h             |  4 +++
 gdb/stack.c                 |  9 +++---
 gdb/thread.c                | 76 ++++++++++++++++-----------------------------
 gdb/value.c                 | 13 +++++++-
 8 files changed, 132 insertions(+), 89 deletions(-)


base-commit: ad8464f799a4c96c7ab8bdfec3f95846cf54f9b0
prerequisite-patch-id: 32ffdda7d7d774bc4df88bf848bcb796559b53ce
prerequisite-patch-id: 02021b74355b70debd344a6e445285c67dfef7d6
prerequisite-patch-id: c87fcf5a54f6805967cbf8ab107606c57d9ecf52
prerequisite-patch-id: ac7dee583d0ffa519c9d1cd89d27664bca68d8c1
prerequisite-patch-id: eac59ae2ea85d2d51e5be1b03e88a5641cc12c22
prerequisite-patch-id: 13da42ad04dc8e2e3bd6a556a0be0e17cf23669b
prerequisite-patch-id: fd3f09fdb58ddc1c595ea014716851f4c8fca48c
-- 
2.14.5

Comments

Pedro Alves July 10, 2020, 11:02 p.m. | #1
On 7/9/20 12:31 AM, Pedro Alves wrote:
> (I have internet again: found a sim card of a different operator that

> works.  This will do until the communications tower near me is

> repaired and get I fiber back...)

> 

> This series fixes the crashes exposed by the

> gdb.multi/multi-target.exp testcase when run against an Asan-enabled

> GDB build, initially reported by Simon here:

> 

>   https://sourceware.org/pipermail/gdb-patches/2020-July/170222.html

> 

> The first two patches fix the crashes, and we should probably put them

> in GDB 10.

> 

> The last patch is a follow up that avoids swallowing exceptions in

> scoped_restore_current_thread's dtor that I'm thinking would be a bit

> too invasive to put in GDB 10, I think it could do with a longer

> baking period in master.

> 

> Pedro Alves (3):

>   Fix crash if connection drops in scoped_restore_current_thread's ctor,

>     part 1

>   Fix crash if connection drops in scoped_restore_current_thread's ctor,

>     part 2

>   Make scoped_restore_current_thread's cdtors exception free (RFC)


I've now merged patches 1 and 2.  Patch 3 will wait until after the branch
is cut.
Simon Marchi July 22, 2020, 7:37 p.m. | #2
On 2020-07-10 7:02 p.m., Pedro Alves wrote:
> On 7/9/20 12:31 AM, Pedro Alves wrote:

>> (I have internet again: found a sim card of a different operator that

>> works.  This will do until the communications tower near me is

>> repaired and get I fiber back...)

>>

>> This series fixes the crashes exposed by the

>> gdb.multi/multi-target.exp testcase when run against an Asan-enabled

>> GDB build, initially reported by Simon here:

>>

>>   https://sourceware.org/pipermail/gdb-patches/2020-July/170222.html

>>

>> The first two patches fix the crashes, and we should probably put them

>> in GDB 10.

>>

>> The last patch is a follow up that avoids swallowing exceptions in

>> scoped_restore_current_thread's dtor that I'm thinking would be a bit

>> too invasive to put in GDB 10, I think it could do with a longer

>> baking period in master.

>>

>> Pedro Alves (3):

>>   Fix crash if connection drops in scoped_restore_current_thread's ctor,

>>     part 1

>>   Fix crash if connection drops in scoped_restore_current_thread's ctor,

>>     part 2

>>   Make scoped_restore_current_thread's cdtors exception free (RFC)

> 

> I've now merged patches 1 and 2.  Patch 3 will wait until after the branch

> is cut.

> 


I now see this other ASan failure when running gdb.multi/multi-target.exp, it's in the
attached asan.log.  There are colors, so it's easier to read if you "cat" it in your
terminal.  It looks familiar, because it happens in scoped_restore_current_thread's dtor
(not ctor), but maybe it just happens to be there but could happen at any other point.

It happens when starting test_continue with non-stop on, just after having completed
test_continue with non-stop off.  It's when GDB does "monitor exit".

Unfortunately, the "freed by thread T0 here" stack trace is again truncated, probably
because the stack is too deep for the portion of the stack ASan captures.  But I managed
to attach to GDB with GDB using gdb_interact and capture it (I broke on unpush_and_perror),
here's the equivalent GDB backtrace:

#0  xfree<void> (ptr=0x621004a5d900) at /home/smarchi/src/binutils-gdb/gdb/../gdbsupport/common-utils.h:63
#1  0x0000000001626260 in call_freefun (h=0x20f8da0 <frame_cache_obstack>, old_chunk=0x621004a5d900) at /home/smarchi/src/binutils-gdb/libiberty/obstack.c:103
#2  0x0000000001626c87 in _obstack_free (h=0x20f8da0 <frame_cache_obstack>, obj=0x0) at /home/smarchi/src/binutils-gdb/libiberty/obstack.c:280
#3  0x000000000098ae26 in reinit_frame_cache () at /home/smarchi/src/binutils-gdb/gdb/frame.c:1856
#4  0x0000000001098adf in switch_to_no_thread () at /home/smarchi/src/binutils-gdb/gdb/thread.c:1301
#5  0x0000000000acf544 in switch_to_inferior_no_thread (inf=0x615000244d00) at /home/smarchi/src/binutils-gdb/gdb/inferior.c:626
#6  0x0000000000e7c38c in remote_unpush_target (target=0x6170000c0c00) at /home/smarchi/src/binutils-gdb/gdb/remote.c:5521
#7  0x0000000000e92db6 in unpush_and_perror (target=0x6170000c0c00, string=0x191d400 "Remote communication error.  Target disconnected.") at /home/smarchi/src/binutils-gdb/gdb/remote.c:9101
#8  0x0000000000e930c7 in remote_target::readchar (this=0x6170000c0c00, timeout=2) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9141
#9  0x0000000000e9576f in remote_target::getpkt_or_notif_sane_1 (this=0x6170000c0c00, buf=0x6170000c0c18, forever=0, expecting_notif=0, is_notif=0x0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9683
#10 0x0000000000e961c9 in remote_target::getpkt_sane (this=0x6170000c0c00, buf=0x6170000c0c18, forever=0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9790
#11 0x0000000000e95545 in remote_target::getpkt (this=0x6170000c0c00, buf=0x6170000c0c18, forever=0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9623
#12 0x0000000000e91ba3 in remote_target::remote_read_bytes_1 (this=0x6170000c0c00, memaddr=0x7ffff78bc38d, myaddr=0x7fff7dca59a0 "", len_units=1, unit_size=1, xfered_len_units=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:8860
#13 0x0000000000e9240c in remote_target::remote_read_bytes (this=0x6170000c0c00, memaddr=0x7ffff78bc38d, myaddr=0x7fff7dca59a0 "", len=1, unit_size=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:8987
#14 0x0000000000e9b821 in remote_target::xfer_partial (this=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, readbuf=0x7fff7dca59a0 "", writebuf=0x0, offset=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:10987
#15 0x000000000104fd3a in raw_memory_xfer_partial (ops=0x6170000c0c00, readbuf=0x7fff7dca59a0 "", writebuf=0x0, memaddr=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:918
#16 0x0000000001050425 in memory_xfer_partial_1 (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, readbuf=0x7fff7dca59a0 "", writebuf=0x0, memaddr=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1047
#17 0x0000000001050608 in memory_xfer_partial (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, readbuf=0x7fff7dca59a0 "", writebuf=0x0, memaddr=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1076
#18 0x0000000001050b92 in target_xfer_partial (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, readbuf=0x7fff7dca59a0 "", writebuf=0x0, offset=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1133
#19 0x0000000001051a7b in target_read_partial (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, buf=0x7fff7dca59a0 "", offset=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1379
#20 0x0000000001051c59 in target_read (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, buf=0x7fff7dca59a0 "", offset=140737346519949, len=1) at /home/smarchi/src/binutils-gdb/gdb/target.c:1419
#21 0x0000000001051178 in target_read_memory (memaddr=0x7ffff78bc38d, myaddr=0x7fff7dca59a0 "", len=1) at /home/smarchi/src/binutils-gdb/gdb/target.c:1222
#22 0x00000000004b4731 in amd64_stack_frame_destroyed_p (gdbarch=0x6210027e8510, pc=0x7ffff78bc38d) at /home/smarchi/src/binutils-gdb/gdb/amd64-tdep.c:2909
#23 0x00000000004b4822 in amd64_epilogue_frame_sniffer (self=0x169df00 <amd64_epilogue_frame_unwind>, this_frame=0x621004a5d9e0, this_prologue_cache=0x621004a5d9f8) at /home/smarchi/src/binutils-gdb/gdb/amd64-tdep.c:2924
#24 0x0000000000981048 in frame_unwind_try_unwinder (this_frame=0x621004a5d9e0, this_cache=0x621004a5d9f8, unwinder=0x169df00 <amd64_epilogue_frame_unwind>) at /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:128
#25 0x000000000098126d in frame_unwind_find_by_frame (this_frame=0x621004a5d9e0, this_cache=0x621004a5d9f8) at /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:186
#26 0x0000000000983c9d in compute_frame_id (fi=0x621004a5d9e0) at /home/smarchi/src/binutils-gdb/gdb/frame.c:546
#27 0x0000000000984167 in get_frame_id (fi=0x621004a5d9e0) at /home/smarchi/src/binutils-gdb/gdb/frame.c:582
#28 0x0000000001098eef in restore_selected_frame (a_frame_id=..., frame_level=0) at /home/smarchi/src/binutils-gdb/gdb/thread.c:1355
#29 0x00000000010992f8 in scoped_restore_current_thread::restore (this=0x7fff7dca5f30) at /home/smarchi/src/binutils-gdb/gdb/thread.c:1411
#30 0x0000000001099355 in scoped_restore_current_thread::~scoped_restore_current_thread (this=0x7fff7dca5f30, __in_chrg=<optimized out>) at /home/smarchi/src/binutils-gdb/gdb/thread.c:1420
#31 0x0000000000aeab84 in do_target_wait (wait_ptid=..., ecs=0x7fff7dca6290, options=1) at /home/smarchi/src/binutils-gdb/gdb/infrun.c:3670
#32 0x0000000000aecbe3 in fetch_inferior_event () at /home/smarchi/src/binutils-gdb/gdb/infrun.c:3965
#33 0x0000000000aa8097 in inferior_event_handler (event_type=INF_REG_EVENT) at /home/smarchi/src/binutils-gdb/gdb/inf-loop.c:42
#34 0x0000000000eab8b7 in remote_async_inferior_event_handler (data=0x6170000d6a00) at /home/smarchi/src/binutils-gdb/gdb/remote.c:14166
#35 0x00000000004ca110 in check_async_event_handlers () at /home/smarchi/src/binutils-gdb/gdb/async-event.c:295
#36 0x00000000015bef41 in gdb_do_one_event () at /home/smarchi/src/binutils-gdb/gdbsupport/event-loop.cc:194
#37 0x0000000000bfd50e in start_event_loop () at /home/smarchi/src/binutils-gdb/gdb/main.c:356
#38 0x0000000000bfd816 in captured_command_loop () at /home/smarchi/src/binutils-gdb/gdb/main.c:416
#39 0x0000000000c00c25 in captured_main (data=0x7fff7dca65d0) at /home/smarchi/src/binutils-gdb/gdb/main.c:1253
#40 0x0000000000c00cb5 in gdb_main (args=0x7fff7dca65d0) at /home/smarchi/src/binutils-gdb/gdb/main.c:1268
#41 0x0000000000414d9e in main (argc=5, argv=0x7fff7dca6738) at /home/smarchi/src/binutils-gdb/gdb/gdb.c:32


The problem seems to be:

- We create a new frame_info object in restore_selected_frame (by calling find_relative_frame)
- The frame is allocated on the frame_cache_obstack
- In frame_unwind_try_unwinder, we try to find an unwinder for that frame
- While trying unwinders, memory read fails because the remote target closes, because of "monitor exit"
- That calls reinit_frame_cache (as shown above), which resets frame_cache_obstack
- When handling the exception in frame_unwind_try_unwinder, we try to set some things on the frame_info
  object (like *this_cache, which in fact tries to write into frame_info::prologue_cache), but the
  frame_info object is no more, it went away with the obstack.

Simon
==4074==ERROR: AddressSanitizer: heap-use-after-free on address 0x621004a4fdf8 at pc 0x0000009810b3 bp 0x7fff38bbca90 sp 0x7fff38bbca80
WRITE of size 8 at 0x621004a4fdf8 thread T0
    #0 0x9810b2 in frame_unwind_try_unwinder /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:134
    #1 0x98126c in frame_unwind_find_by_frame(frame_info*, void**) /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:186
    #2 0x983c9c in compute_frame_id /home/smarchi/src/binutils-gdb/gdb/frame.c:546
    #3 0x984166 in get_frame_id(frame_info*) /home/smarchi/src/binutils-gdb/gdb/frame.c:582
    #4 0x1098eee in restore_selected_frame /home/smarchi/src/binutils-gdb/gdb/thread.c:1355
    #5 0x10992f7 in scoped_restore_current_thread::restore() /home/smarchi/src/binutils-gdb/gdb/thread.c:1411
    #6 0x1099354 in scoped_restore_current_thread::~scoped_restore_current_thread() /home/smarchi/src/binutils-gdb/gdb/thread.c:1420
    #7 0xaeab83 in do_target_wait /home/smarchi/src/binutils-gdb/gdb/infrun.c:3670
    #8 0xaecbe2 in fetch_inferior_event() /home/smarchi/src/binutils-gdb/gdb/infrun.c:3965
    #9 0xaa8096 in inferior_event_handler(inferior_event_type) /home/smarchi/src/binutils-gdb/gdb/inf-loop.c:42
    #10 0xeab8b6 in remote_async_inferior_event_handler /home/smarchi/src/binutils-gdb/gdb/remote.c:14166
    #11 0x4ca10f in check_async_event_handlers() /home/smarchi/src/binutils-gdb/gdb/async-event.c:295
    #12 0x15bef40 in gdb_do_one_event() /home/smarchi/src/binutils-gdb/gdbsupport/event-loop.cc:194
    #13 0xbfd50d in start_event_loop /home/smarchi/src/binutils-gdb/gdb/main.c:356
    #14 0xbfd815 in captured_command_loop /home/smarchi/src/binutils-gdb/gdb/main.c:416
    #15 0xc00c24 in captured_main /home/smarchi/src/binutils-gdb/gdb/main.c:1253
    #16 0xc00cb4 in gdb_main(captured_main_args*) /home/smarchi/src/binutils-gdb/gdb/main.c:1268
    #17 0x414d9d in main /home/smarchi/src/binutils-gdb/gdb/gdb.c:32
    #18 0x7fc78984e83f in __libc_start_main ../csu/libc-start.c:291
    #19 0x414b98 in _start (/home/smarchi/build/binutils-gdb/gdb/gdb+0x414b98)

0x621004a4fdf8 is located 248 bytes inside of 4064-byte region [0x621004a4fd00,0x621004a50ce0)
freed by thread T0 here:
    #0 0x7fc78c385c7f in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10bc7f)
    #1 0x98f9f6 in xfree<void> /home/smarchi/src/binutils-gdb/gdb/../gdbsupport/common-utils.h:62
    #2 0x162625f in call_freefun /home/smarchi/src/binutils-gdb/libiberty/obstack.c:103
    #3 0x1626c86 in _obstack_free /home/smarchi/src/binutils-gdb/libiberty/obstack.c:280
    #4 0x98ae25 in reinit_frame_cache() /home/smarchi/src/binutils-gdb/gdb/frame.c:1856
    #5 0x1098ade in switch_to_no_thread() /home/smarchi/src/binutils-gdb/gdb/thread.c:1301
    #6 0xacf543 in switch_to_inferior_no_thread(inferior*) /home/smarchi/src/binutils-gdb/gdb/inferior.c:626
    #7 0xe7c38b in remote_unpush_target /home/smarchi/src/binutils-gdb/gdb/remote.c:5521
    #8 0xe92db5 in unpush_and_perror /home/smarchi/src/binutils-gdb/gdb/remote.c:9101
    #9 0xe930c6 in remote_target::readchar(int) /home/smarchi/src/binutils-gdb/gdb/remote.c:9141
    #10 0xe9576e in remote_target::getpkt_or_notif_sane_1(std::vector<char, gdb::default_init_allocator<char, std::allocator<char> > >*, int, int, int*) /home/smarchi/src/binutils-gdb/gdb/remote.c:9683
    #11 0xe961c8 in remote_target::getpkt_sane(std::vector<char, gdb::default_init_allocator<char, std::allocator<char> > >*, int) /home/smarchi/src/binutils-gdb/gdb/remote.c:9790
    #12 0xe95544 in remote_target::getpkt(std::vector<char, gdb::default_init_allocator<char, std::allocator<char> > >*, int) /home/smarchi/src/binutils-gdb/gdb/remote.c:9623
    #13 0xe91ba2 in remote_target::remote_read_bytes_1(unsigned long, unsigned char*, unsigned long, int, unsigned long*) /home/smarchi/src/binutils-gdb/gdb/remote.c:8860
    #14 0xe9240b in remote_target::remote_read_bytes(unsigned long, unsigned char*, unsigned long, int, unsigned long*) /home/smarchi/src/binutils-gdb/gdb/remote.c:8987
    #15 0xe9b820 in remote_target::xfer_partial(target_object, char const*, unsigned char*, unsigned char const*, unsigned long, unsigned long, unsigned long*) /home/smarchi/src/binutils-gdb/gdb/remote.c:10987
    #16 0x104fd39 in raw_memory_xfer_partial(target_ops*, unsigned char*, unsigned char const*, unsigned long, long, unsigned long*) /home/smarchi/src/binutils-gdb/gdb/target.c:918
    #17 0x1050424 in memory_xfer_partial_1 /home/smarchi/src/binutils-gdb/gdb/target.c:1047
    #18 0x1050607 in memory_xfer_partial /home/smarchi/src/binutils-gdb/gdb/target.c:1076
    #19 0x1050b91 in target_xfer_partial(target_ops*, target_object, char const*, unsigned char*, unsigned char const*, unsigned long, unsigned long, unsigned long*) /home/smarchi/src/binutils-gdb/gdb/target.c:1133
    #20 0x1051a7a in target_read_partial /home/smarchi/src/binutils-gdb/gdb/target.c:1379
    #21 0x1051c58 in target_read(target_ops*, target_object, char const*, unsigned char*, unsigned long, long) /home/smarchi/src/binutils-gdb/gdb/target.c:1419
    #22 0x1051177 in target_read_memory(unsigned long, unsigned char*, long) /home/smarchi/src/binutils-gdb/gdb/target.c:1222
    #23 0x4b4730 in amd64_stack_frame_destroyed_p /home/smarchi/src/binutils-gdb/gdb/amd64-tdep.c:2909
    #24 0x4b4821 in amd64_epilogue_frame_sniffer /home/smarchi/src/binutils-gdb/gdb/amd64-tdep.c:2924
    #25 0x981047 in frame_unwind_try_unwinder /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:128
    #26 0x98126c in frame_unwind_find_by_frame(frame_info*, void**) /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:186
    #27 0x983c9c in compute_frame_id /home/smarchi/src/binutils-gdb/gdb/frame.c:546
    #28 0x984166 in get_frame_id(frame_info*) /home/smarchi/src/binutils-gdb/gdb/frame.c:582
    #29 0x1098eee in restore_selected_frame /home/smarchi/src/binutils-gdb/gdb/thread.c:1355

previously allocated by thread T0 here:
    #0 0x7fc78c386078 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c078)
    #1 0x4a1ad3 in xmalloc /home/smarchi/src/binutils-gdb/gdb/alloc.c:60
    #2 0x162616d in call_chunkfun /home/smarchi/src/binutils-gdb/libiberty/obstack.c:94
    #3 0x1626318 in _obstack_begin_worker /home/smarchi/src/binutils-gdb/libiberty/obstack.c:141
    #4 0x16265cb in _obstack_begin /home/smarchi/src/binutils-gdb/libiberty/obstack.c:164
    #5 0x98ae44 in reinit_frame_cache() /home/smarchi/src/binutils-gdb/gdb/frame.c:1857
    #6 0x1098b4c in switch_to_thread(thread_info*) /home/smarchi/src/binutils-gdb/gdb/thread.c:1316
    #7 0x1099146 in scoped_restore_current_thread::restore() /home/smarchi/src/binutils-gdb/gdb/thread.c:1399
    #8 0x1099354 in scoped_restore_current_thread::~scoped_restore_current_thread() /home/smarchi/src/binutils-gdb/gdb/thread.c:1420
    #9 0xaeab83 in do_target_wait /home/smarchi/src/binutils-gdb/gdb/infrun.c:3670
    #10 0xaecbe2 in fetch_inferior_event() /home/smarchi/src/binutils-gdb/gdb/infrun.c:3965
    #11 0xaa8096 in inferior_event_handler(inferior_event_type) /home/smarchi/src/binutils-gdb/gdb/inf-loop.c:42
    #12 0xeab8b6 in remote_async_inferior_event_handler /home/smarchi/src/binutils-gdb/gdb/remote.c:14166
    #13 0x4ca10f in check_async_event_handlers() /home/smarchi/src/binutils-gdb/gdb/async-event.c:295
    #14 0x15bef40 in gdb_do_one_event() /home/smarchi/src/binutils-gdb/gdbsupport/event-loop.cc:194
    #15 0xbfd50d in start_event_loop /home/smarchi/src/binutils-gdb/gdb/main.c:356
    #16 0xbfd815 in captured_command_loop /home/smarchi/src/binutils-gdb/gdb/main.c:416
    #17 0xc00c24 in captured_main /home/smarchi/src/binutils-gdb/gdb/main.c:1253
    #18 0xc00cb4 in gdb_main(captured_main_args*) /home/smarchi/src/binutils-gdb/gdb/main.c:1268
    #19 0x414d9d in main /home/smarchi/src/binutils-gdb/gdb/gdb.c:32
    #20 0x7fc78984e83f in __libc_start_main ../csu/libc-start.c:291

SUMMARY: AddressSanitizer: heap-use-after-free /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:134 in frame_unwind_try_unwinder
Shadow bytes around the buggy address:
  0x0c4280941f60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4280941f70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4280941f80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4280941f90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4280941fa0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c4280941fb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]
  0x0c4280941fc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4280941fd0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4280941fe0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4280941ff0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4280942000: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==4074==ABORTING
Pedro Alves July 22, 2020, 8:37 p.m. | #3
On 7/22/20 8:37 PM, Simon Marchi wrote:
> On 2020-07-10 7:02 p.m., Pedro Alves wrote:

>> On 7/9/20 12:31 AM, Pedro Alves wrote:

>>> (I have internet again: found a sim card of a different operator that

>>> works.  This will do until the communications tower near me is

>>> repaired and get I fiber back...)

>>>

>>> This series fixes the crashes exposed by the

>>> gdb.multi/multi-target.exp testcase when run against an Asan-enabled

>>> GDB build, initially reported by Simon here:

>>>

>>>   https://sourceware.org/pipermail/gdb-patches/2020-July/170222.html

>>>

>>> The first two patches fix the crashes, and we should probably put them

>>> in GDB 10.

>>>

>>> The last patch is a follow up that avoids swallowing exceptions in

>>> scoped_restore_current_thread's dtor that I'm thinking would be a bit

>>> too invasive to put in GDB 10, I think it could do with a longer

>>> baking period in master.

>>>

>>> Pedro Alves (3):

>>>   Fix crash if connection drops in scoped_restore_current_thread's ctor,

>>>     part 1

>>>   Fix crash if connection drops in scoped_restore_current_thread's ctor,

>>>     part 2

>>>   Make scoped_restore_current_thread's cdtors exception free (RFC)

>>

>> I've now merged patches 1 and 2.  Patch 3 will wait until after the branch

>> is cut.

>>

> 

> I now see this other ASan failure when running gdb.multi/multi-target.exp, it's in the

> attached asan.log.  There are colors, so it's easier to read if you "cat" it in your

> terminal.  It looks familiar, because it happens in scoped_restore_current_thread's dtor

> (not ctor), but maybe it just happens to be there but could happen at any other point.

> 

> It happens when starting test_continue with non-stop on, just after having completed

> test_continue with non-stop off.  It's when GDB does "monitor exit".

> 

> Unfortunately, the "freed by thread T0 here" stack trace is again truncated, probably

> because the stack is too deep for the portion of the stack ASan captures.  But I managed

> to attach to GDB with GDB using gdb_interact and capture it (I broke on unpush_and_perror),

> here's the equivalent GDB backtrace:

> 

> #0  xfree<void> (ptr=0x621004a5d900) at /home/smarchi/src/binutils-gdb/gdb/../gdbsupport/common-utils.h:63

> #1  0x0000000001626260 in call_freefun (h=0x20f8da0 <frame_cache_obstack>, old_chunk=0x621004a5d900) at /home/smarchi/src/binutils-gdb/libiberty/obstack.c:103

> #2  0x0000000001626c87 in _obstack_free (h=0x20f8da0 <frame_cache_obstack>, obj=0x0) at /home/smarchi/src/binutils-gdb/libiberty/obstack.c:280

> #3  0x000000000098ae26 in reinit_frame_cache () at /home/smarchi/src/binutils-gdb/gdb/frame.c:1856

> #4  0x0000000001098adf in switch_to_no_thread () at /home/smarchi/src/binutils-gdb/gdb/thread.c:1301

> #5  0x0000000000acf544 in switch_to_inferior_no_thread (inf=0x615000244d00) at /home/smarchi/src/binutils-gdb/gdb/inferior.c:626

> #6  0x0000000000e7c38c in remote_unpush_target (target=0x6170000c0c00) at /home/smarchi/src/binutils-gdb/gdb/remote.c:5521

> #7  0x0000000000e92db6 in unpush_and_perror (target=0x6170000c0c00, string=0x191d400 "Remote communication error.  Target disconnected.") at /home/smarchi/src/binutils-gdb/gdb/remote.c:9101

> #8  0x0000000000e930c7 in remote_target::readchar (this=0x6170000c0c00, timeout=2) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9141

> #9  0x0000000000e9576f in remote_target::getpkt_or_notif_sane_1 (this=0x6170000c0c00, buf=0x6170000c0c18, forever=0, expecting_notif=0, is_notif=0x0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9683

> #10 0x0000000000e961c9 in remote_target::getpkt_sane (this=0x6170000c0c00, buf=0x6170000c0c18, forever=0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9790

> #11 0x0000000000e95545 in remote_target::getpkt (this=0x6170000c0c00, buf=0x6170000c0c18, forever=0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:9623

> #12 0x0000000000e91ba3 in remote_target::remote_read_bytes_1 (this=0x6170000c0c00, memaddr=0x7ffff78bc38d, myaddr=0x7fff7dca59a0 "", len_units=1, unit_size=1, xfered_len_units=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:8860

> #13 0x0000000000e9240c in remote_target::remote_read_bytes (this=0x6170000c0c00, memaddr=0x7ffff78bc38d, myaddr=0x7fff7dca59a0 "", len=1, unit_size=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:8987

> #14 0x0000000000e9b821 in remote_target::xfer_partial (this=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, readbuf=0x7fff7dca59a0 "", writebuf=0x0, offset=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/remote.c:10987

> #15 0x000000000104fd3a in raw_memory_xfer_partial (ops=0x6170000c0c00, readbuf=0x7fff7dca59a0 "", writebuf=0x0, memaddr=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:918

> #16 0x0000000001050425 in memory_xfer_partial_1 (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, readbuf=0x7fff7dca59a0 "", writebuf=0x0, memaddr=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1047

> #17 0x0000000001050608 in memory_xfer_partial (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, readbuf=0x7fff7dca59a0 "", writebuf=0x0, memaddr=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1076

> #18 0x0000000001050b92 in target_xfer_partial (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, readbuf=0x7fff7dca59a0 "", writebuf=0x0, offset=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1133

> #19 0x0000000001051a7b in target_read_partial (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, buf=0x7fff7dca59a0 "", offset=140737346519949, len=1, xfered_len=0x7fff7dca58b0) at /home/smarchi/src/binutils-gdb/gdb/target.c:1379

> #20 0x0000000001051c59 in target_read (ops=0x6170000c0c00, object=TARGET_OBJECT_MEMORY, annex=0x0, buf=0x7fff7dca59a0 "", offset=140737346519949, len=1) at /home/smarchi/src/binutils-gdb/gdb/target.c:1419

> #21 0x0000000001051178 in target_read_memory (memaddr=0x7ffff78bc38d, myaddr=0x7fff7dca59a0 "", len=1) at /home/smarchi/src/binutils-gdb/gdb/target.c:1222

> #22 0x00000000004b4731 in amd64_stack_frame_destroyed_p (gdbarch=0x6210027e8510, pc=0x7ffff78bc38d) at /home/smarchi/src/binutils-gdb/gdb/amd64-tdep.c:2909

> #23 0x00000000004b4822 in amd64_epilogue_frame_sniffer (self=0x169df00 <amd64_epilogue_frame_unwind>, this_frame=0x621004a5d9e0, this_prologue_cache=0x621004a5d9f8) at /home/smarchi/src/binutils-gdb/gdb/amd64-tdep.c:2924

> #24 0x0000000000981048 in frame_unwind_try_unwinder (this_frame=0x621004a5d9e0, this_cache=0x621004a5d9f8, unwinder=0x169df00 <amd64_epilogue_frame_unwind>) at /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:128

> #25 0x000000000098126d in frame_unwind_find_by_frame (this_frame=0x621004a5d9e0, this_cache=0x621004a5d9f8) at /home/smarchi/src/binutils-gdb/gdb/frame-unwind.c:186

> #26 0x0000000000983c9d in compute_frame_id (fi=0x621004a5d9e0) at /home/smarchi/src/binutils-gdb/gdb/frame.c:546

> #27 0x0000000000984167 in get_frame_id (fi=0x621004a5d9e0) at /home/smarchi/src/binutils-gdb/gdb/frame.c:582

> #28 0x0000000001098eef in restore_selected_frame (a_frame_id=..., frame_level=0) at /home/smarchi/src/binutils-gdb/gdb/thread.c:1355

> #29 0x00000000010992f8 in scoped_restore_current_thread::restore (this=0x7fff7dca5f30) at /home/smarchi/src/binutils-gdb/gdb/thread.c:1411

> #30 0x0000000001099355 in scoped_restore_current_thread::~scoped_restore_current_thread (this=0x7fff7dca5f30, __in_chrg=<optimized out>) at /home/smarchi/src/binutils-gdb/gdb/thread.c:1420

> #31 0x0000000000aeab84 in do_target_wait (wait_ptid=..., ecs=0x7fff7dca6290, options=1) at /home/smarchi/src/binutils-gdb/gdb/infrun.c:3670

> #32 0x0000000000aecbe3 in fetch_inferior_event () at /home/smarchi/src/binutils-gdb/gdb/infrun.c:3965

> #33 0x0000000000aa8097 in inferior_event_handler (event_type=INF_REG_EVENT) at /home/smarchi/src/binutils-gdb/gdb/inf-loop.c:42

> #34 0x0000000000eab8b7 in remote_async_inferior_event_handler (data=0x6170000d6a00) at /home/smarchi/src/binutils-gdb/gdb/remote.c:14166

> #35 0x00000000004ca110 in check_async_event_handlers () at /home/smarchi/src/binutils-gdb/gdb/async-event.c:295

> #36 0x00000000015bef41 in gdb_do_one_event () at /home/smarchi/src/binutils-gdb/gdbsupport/event-loop.cc:194

> #37 0x0000000000bfd50e in start_event_loop () at /home/smarchi/src/binutils-gdb/gdb/main.c:356

> #38 0x0000000000bfd816 in captured_command_loop () at /home/smarchi/src/binutils-gdb/gdb/main.c:416

> #39 0x0000000000c00c25 in captured_main (data=0x7fff7dca65d0) at /home/smarchi/src/binutils-gdb/gdb/main.c:1253

> #40 0x0000000000c00cb5 in gdb_main (args=0x7fff7dca65d0) at /home/smarchi/src/binutils-gdb/gdb/main.c:1268

> #41 0x0000000000414d9e in main (argc=5, argv=0x7fff7dca6738) at /home/smarchi/src/binutils-gdb/gdb/gdb.c:32

> 


Bah.

> 

> The problem seems to be:

> 

> - We create a new frame_info object in restore_selected_frame (by calling find_relative_frame)

> - The frame is allocated on the frame_cache_obstack

> - In frame_unwind_try_unwinder, we try to find an unwinder for that frame

> - While trying unwinders, memory read fails because the remote target closes, because of "monitor exit"

> - That calls reinit_frame_cache (as shown above), which resets frame_cache_obstack

> - When handling the exception in frame_unwind_try_unwinder, we try to set some things on the frame_info

>   object (like *this_cache, which in fact tries to write into frame_info::prologue_cache), but the

>   frame_info object is no more, it went away with the obstack.


I'm thinking that to fix this we will need a generation counter in
reinit_frame_cache.  Then in frame_unwind_try_unwinder, don't call
frame_cleanup_after_sniffer if the generation is not the same as it was
on entry.

Something like this.  Does it fix it for you?  I can't seem to reproduce
the crash here.

From 202b20db082969cfa156468c3443888761629dee Mon Sep 17 00:00:00 2001
From: Pedro Alves <pedro@palves.net>

Date: Wed, 22 Jul 2020 20:53:59 +0100
Subject: [PATCH] Fix yet another bug exposed by ASAN + multi-target.exp

---
 gdb/frame-unwind.c | 13 ++++++++++---
 gdb/frame.c        | 13 +++++++++++++
 gdb/frame.h        |  4 ++++
 3 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/gdb/frame-unwind.c b/gdb/frame-unwind.c
index 3334c472d02..ba25e19172e 100644
--- a/gdb/frame-unwind.c
+++ b/gdb/frame-unwind.c
@@ -121,6 +121,8 @@ frame_unwind_try_unwinder (struct frame_info *this_frame, void **this_cache,
 {
   int res = 0;
 
+  unsigned int entry_generation = get_frame_cache_generation ();
+
   frame_prepare_for_sniffer (this_frame, unwinder);
 
   try
@@ -130,9 +132,14 @@ frame_unwind_try_unwinder (struct frame_info *this_frame, void **this_cache,
   catch (const gdb_exception &ex)
     {
       /* Catch all exceptions, caused by either interrupt or error.
-	 Reset *THIS_CACHE.  */
-      *this_cache = NULL;
-      frame_cleanup_after_sniffer (this_frame);
+	 Reset *THIS_CACHE, unless something reinitialized the frame
+	 cache meanwhile, in which case THIS_FRAME is now
+	 dangling.  */
+      if (get_frame_cache_generation () == entry_generation)
+	{
+	  *this_cache = NULL;
+	  frame_cleanup_after_sniffer (this_frame);
+	}
 
       if (ex.error == NOT_AVAILABLE_ERROR)
 	{
diff --git a/gdb/frame.c b/gdb/frame.c
index ac1016b083f..4ac958a1e95 100644
--- a/gdb/frame.c
+++ b/gdb/frame.c
@@ -53,6 +53,17 @@
 
 static struct frame_info *sentinel_frame;
 
+/* Number of calls to reinit_frame_cache.  */
+static unsigned int frame_cache_generation = 0;
+
+/* See frame.h.  */
+
+unsigned int
+get_frame_cache_generation ()
+{
+  return frame_cache_generation;
+}
+
 /* The values behind the global "set backtrace ..." settings.  */
 set_backtrace_options user_set_backtrace_options;
 
@@ -1843,6 +1854,8 @@ reinit_frame_cache (void)
 {
   struct frame_info *fi;
 
+  ++frame_cache_generation;
+
   /* Tear down all frame caches.  */
   for (fi = sentinel_frame; fi != NULL; fi = fi->prev)
     {
diff --git a/gdb/frame.h b/gdb/frame.h
index cfc15022ed5..8d029cc065d 100644
--- a/gdb/frame.h
+++ b/gdb/frame.h
@@ -949,6 +949,10 @@ extern const gdb::option::option_def set_backtrace_option_defs[2];
 /* The values behind the global "set backtrace ..." settings.  */
 extern set_backtrace_options user_set_backtrace_options;
 
+/* Get the number of calls to reinit_frame_cache.  */
+
+unsigned int get_frame_cache_generation ();
+
 /* Mark that the PC value is masked for the previous frame.  */
 
 extern void set_frame_previous_pc_masked (struct frame_info *frame);

base-commit: 32fa152e3bfcf021ce49767be547fae5129d922b
-- 
2.14.5
Simon Marchi July 22, 2020, 8:47 p.m. | #4
On 2020-07-22 4:37 p.m., Pedro Alves wrote:
> I'm thinking that to fix this we will need a generation counter in

> reinit_frame_cache.  Then in frame_unwind_try_unwinder, don't call

> frame_cleanup_after_sniffer if the generation is not the same as it was

> on entry.

> 

> Something like this.  Does it fix it for you?  I can't seem to reproduce

> the crash here.


I don't have time to try, but the approach makes sense.  It also crossed my mind,
but I thought it would be more complicated to implement than that.

Simon