New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SR-10252] swift build --sanitize=address failed #52652
Comments
I believe the reason that you're seeing this issue is that the Swift compiler and the clang on your system have a different address sanitizer version an they're incompatible. That was a longstanding bug in Swift that it relied on the system's clang. So stuff (like ASan) that is relying on a particular clang version would only work if your system happened to have the right one. Swift 5 now finally fixed that by shipping a matching clang compiler :slight_smile:. I can also confirm that the attached example project works just fine with ASan for me in Swift 5.
|
This means the version of Clang on your system is mismatched with the version of Swift, and therefore they expect a different ASan runtime. I think the Swift 5 toolchains have a Clang included with them to fix this issue! cc @kubamracek |
Comment by Christophe Braud (JIRA) @weissi I confirm no problem with Swift 5 As soon as I can I will recompile my project in Vapor 3 with Swift 5 |
Sorry, I meant that before Swift 5, the Swift Docker image would have used a system version of Clang, which is almost certainly out of sync with the Swift compiler. I guess it would be reasonable for the package manager to not try to sanitize the C libraries in the process under those conditions, but I'm not sure that's worth changing in Swift 4.2.x now that Swift 5 is out. Or am I missing something? |
cc also @aciidb0mb3r |
Comment by Christophe Braud (JIRA) > Sorry, I meant that before Swift 5 |
Tof (JIRA User) Just to comment here: The memory fragmentation isn't really a SwiftNIO issue. SwiftNIO offers all the tools to be more tighter in the case. However, there are a number of very straightforward cases where SwiftNIO could help the user through heuristics. For example if someone extracts a 10 byte In the case of the Vapor database driver however, this wouldn't have helped. If all buffers are fairly large, the SwiftNIO can't automatically decide if a copy or zero-copy is better. That's up to the Vapor mysql driver really 🙂. |
Comment by Christophe Braud (JIRA) > The memory fragmentation isn't really a SwiftNIO issue. SwiftNIO offers all the tools to be more tighter in the case. However, there are a number of very straightforward cases where SwiftNIO could help the user through heuristics. @weissi I understand your point and all tools or improved you can provider with Swift NIO are welcome. 😉 > That's up to the Vapor mysql driver really 🙂. This fragmentation problem reminds me of a case: Several years ago I had a similar memory fragmentation problem on a projet. On this project we had a large number of objects that were allocated and deallocated intensively. Almost all objects of this project had the same size. To reduce fragmentation and make the best use of memory we had set up several heaps. Each heap being dedicated to objects with same size. Because the objects in a heap always had the same size, we could maximize the use of the memory pages with a minimum of fragmentation and a noticeable increase the performance. It's just an idea, but I wonder if this kind of approach could be interesting in terms of how Swift manages his memory. The advantage of doing so at the language level is that it would be available by default for all applications compile with Swift. Anyway, it's just an idea 😉 |
Tof (JIRA User) Right, that's getting interesting. Right now it wouldn't even be possible to use different heaps for certain {{Data}}s. Eventually the language could provide more support for custom allocators... |
Tof (JIRA User) regarding your application with Postgres: Do you have any stats on how much peak memory should be required vs. how much peak memory is required? I'm sure the postgres driver etc could also be optimised that we get tighter in memory here. Which driver are you using btw? |
Comment by Christophe Braud (JIRA) @weissi currently the application run in a micro service environment with no debug tool, a minimalist version of ubuntu (we would have preferred Alpine Linux but Swift is not available on this micro Linux) and the application. We don't have the permission to change this environment. we have just the right to update the application. With the service administration we can see what he consumes in memory real time but without history. At startup the service takes 8 MiB. After that he takes more and more memory. As soon as the application reaches 100 MiB the service is automatically restarted. The speed at which I get to the 100 MiB depends a lot on the data flow that the service has to deal with. The traces that we put in the app allowed us to identify that as soon as we access database the service increases its memory consumption and never release it. To do well we would have to look at the vapor side to see if we can have more accurate stats. |
Tof (JIRA User) Alright, a couple of things here. First of all, in your case it sounds like we have no idea what the real issue is, the only similarity with the other issue is that it's memory-related, correct? Can you change the code of your application? Maybe you could add a call to Regarding what you write about Futures/Promises. It sounds like you believe that futures/promises add inefficiencies that async/await will resolve. That is not the case. Vapor/NIO can be very efficient because of the Futures/Promises and not despite them 🙂. NIO/Vapor use an asynchronous programming model. That allows us to not use one kernel thread per connection which increases efficiency. If you do asynchronous programming you need some forms of callbacks, they can be for example real callbacks (closures), futures/promises, or other techniques. Callbacks get you into 'callback hell' and therefore we chose futures/promises. But futures/promises are not less efficient than closures or anything, in both cases you will need to allocate and there's no way out of that. Btw, the same applies to the coroutines, for asynchronous programming, you need to allocate a 'continuation' where the program will continue when the result of a certain operation (say a For LLVM to be able to completely get rid of any async/await traces, the code needs to be able to run synchronously but that's impossible for network programming because you often need for the peer on the other side to do something for you to be able to make progress. Does that make sense? We might want to move this discussion over to the forums btw 🙂 |
Attachment: Download
Environment
Swift: 4.2.3
Linux: Ubuntu 16.04
Docker Desktop Community: 2.0.0.3
Vapor: 3.3.0
Additional Detail from JIRA
md5: e55cd21289406cd52a9b80fe1c842ce6
Issue Description:
As describe on swift forum Memory leaking in Vapor app try to build a project with
Steps to reproduce:
1. unzip test project (see attach file)
2. generate linux image with command
3. deploy image on docker swarm
4. attach to the image
5. build the project with
Results:
Swift build generate this output:
Expected results:
Build the project without errors
The text was updated successfully, but these errors were encountered: