Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-3002] using DispatchData.enumerateBytes on Linux leaks the block #715

Closed
weissi opened this issue Oct 20, 2016 · 4 comments
Closed

[SR-3002] using DispatchData.enumerateBytes on Linux leaks the block #715

weissi opened this issue Oct 20, 2016 · 4 comments

Comments

@weissi
Copy link
Member

weissi commented Oct 20, 2016

Previous ID SR-3002
Radar None
Original Reporter @weissi
Type Bug
Status Resolved
Resolution Done

Attachment: Download

Additional Detail from JIRA
Votes 0
Component/s libdispatch
Labels Bug, Leak, Linux
Assignee dgrove-oss (JIRA)
Priority Medium

md5: 323c438d5e940e5a5a3e6dc87ed9b40f

is blocked by:

  • SR-2313 Add withoutActuallyEscaping

Issue Description:

Description

When using DispatchData.enumerateBytes on Linux, it leaks the block which leaks to pretty bad memory leaks...

The example program is

import Dispatch

while true {
    let d = DispatchData.empty
    d.enumerateBytes { _ in }
}

just leave that running on Linux and it'll leak a lot of memory, on Darwin (with the Apple Swift from Xcode it's fine).

If you're lazy, just run the attached test.sh with bash test.sh under Linux or Darwin and it'll create, compile, run (for 10s), and print statistics for the test program.

Numbers

See below for some numbers:

Linux

this is created with Swift from https://swift.org/builds/development/ubuntu1604/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04.tar.gz but the same problem happens with all Swift versions I have tested.

The kernel is Linux thing 4.4.0-36-generic #​55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ bash test.sh
swift source in '/tmp/swift-dispatch-mem-leak_MgQ7Lq/test.swift'
swift binary in '/tmp/swift-dispatch-mem-leak_MgQ7Lq/test'
=====
import Dispatch

while true {
    let d = DispatchData.empty
    d.enumerateBytes { _ in }
}
=====
Swift version 3.0-dev (LLVM 2f56c9717e, Clang b0df436efa, Swift 25d58f9228)
Target: x86_64-unknown-linux-gnu
=====
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jweiss   18062  100 62.5 10309652 10253740 pts/3 R+ 18:51   0:10 ./test
test.sh: line 28: 18062 Killed                  ./test
OK

that's 10253740 kb resident in 10s, that's a 10GB leak.

Darwin

$ bash test.sh
swift source in '/tmp/swift-dispatch-mem-leak_iorCwW/test.swift'
swift binary in '/tmp/swift-dispatch-mem-leak_iorCwW/test'
=====
import Dispatch

while true {
    let d = DispatchData.empty
    d.enumerateBytes { _ in }
}
=====
Apple Swift version 3.0 (swiftlang-800.0.46.2 clang-800.0.38)
Target: x86_64-apple-macosx10.9
=====
USER       PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
johannes 34757 100.0  0.0  2445456   6832 s002  R+    6:50pm   0:09.99 ./test
test.sh: line 28: 34757 Killed: 9               ./test
OK

that's 6MB resident in 10s, much better.

Valgrind output

$ valgrind --leak-check=full /tmp/swift-dispatch-mem-leak_yqOZxx/test
==18530== Memcheck, a memory error detector
==18530== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==18530== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==18530== Command: /tmp/swift-dispatch-mem-leak_yqOZxx/test
==18530== 
^C==18530== 
==18530== Process terminating with default action of signal 2 (SIGINT)
==18530==    at 0x400C73: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== HEAP SUMMARY:
==18530==     in use at exit: 59,364,626 bytes in 1,482,300 blocks
==18530==   total heap usage: 1,482,310 allocs, 10 frees, 59,444,330 bytes allocated
==18530== 
==18530== 57 (16 direct, 41 indirect) bytes in 1 blocks are definitely lost in loss record 4 of 9
==18530==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x515EBFA: _swift_stdlib_getUnsafeArgvArgc (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x507139F: globalinit_33_FD9A49A256BEB6AF7C48013347ADC3BA_func3 (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x5E98AD8: __pthread_once_slow (pthread_once.c:116)
==18530==    by 0x515C4A9: swift_once (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x400C5D: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== 192 bytes in 6 blocks are possibly lost in loss record 5 of 9
==18530==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x514F9E5: swift_slowAlloc (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x514FA1E: _swift_allocObject_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x4102463: _TFV8Dispatch12DispatchData14enumerateBytesfT5blockFTGSRVs5UInt8_SiRSb_T__T_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x400C9B: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== 240 bytes in 5 blocks are possibly lost in loss record 6 of 9
==18530==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x41006D8: _Block_copy_internal (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x41024C0: _TFV8Dispatch12DispatchData14enumerateBytesfT5blockFTGSRVs5UInt8_SiRSb_T__T_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x400C9B: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== 59,291,376 (35,574,864 direct, 23,716,512 indirect) bytes in 741,143 blocks are definitely lost in loss record 9 of 9
==18530==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x41006D8: _Block_copy_internal (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x41024C0: _TFV8Dispatch12DispatchData14enumerateBytesfT5blockFTGSRVs5UInt8_SiRSb_T__T_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x400C9B: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== LEAK SUMMARY:
==18530==    definitely lost: 35,574,880 bytes in 741,144 blocks
==18530==    indirectly lost: 23,716,553 bytes in 741,142 blocks
==18530==      possibly lost: 432 bytes in 11 blocks
==18530==    still reachable: 72,761 bytes in 3 blocks
==18530==         suppressed: 0 bytes in 0 blocks
==18530== Reachable blocks (those to which a pointer was found) are not shown.
==18530== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==18530== 
==18530== For counts of detected and suppressed errors, rerun with: -v
==18530== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)
@swift-ci
Copy link

Comment by David Grove (JIRA)

I can reproduce.

The fix is that we really need to be calling CDispatch.dispatch_data_apply (not swift_dispatch_data_apply). Trying to do that however results in a Swift compiler error because it thinks the block argument being captured in the closure argument to dispatch_data_apply is going to escape. We need the functionality of https://bugs.swift.org/browse/SR-2313 to cleanly fix the leak. Will have to figure out the least ugly way to kludge around it not being available...

@swift-ci
Copy link

Comment by David Grove (JIRA)

Submitted #186 using unsafeBitCast in place of withoutActuallyEscaping as suggested by Jordan. Once the PR is merged, I will resolve this issue and create a new one to track that we need to update this code once withoutActuallyEscaping is implemented.

@swift-ci
Copy link

Comment by David Grove (JIRA)

Fix merged into Swift3 branch: #189 Leaving issue open until merged to master as well.

@swift-ci
Copy link

swift-ci commented Dec 8, 2016

Comment by David Grove (JIRA)

fixed now merged to master as well.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 5, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants