Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-9603] NSFastEnumerationIterator doesn't detect mutations #3564

Open
lilyball mannequin opened this issue Jan 5, 2019 · 5 comments
Open

[SR-9603] NSFastEnumerationIterator doesn't detect mutations #3564

lilyball mannequin opened this issue Jan 5, 2019 · 5 comments

Comments

@lilyball
Copy link
Mannequin

lilyball mannequin commented Jan 5, 2019

Previous ID SR-9603
Radar None
Original Reporter @lilyball
Type Bug
Environment

Verified with source inspection on both Swift 4.2.1 and latest Swift master.

Additional Detail from JIRA
Votes 0
Component/s Foundation
Labels Bug
Assignee None
Priority Medium

md5: e8dc81adcdb9ed8602c6d95a9d12783b

Issue Description:

The implementation of NSFastEnumerationIterator doesn't detect mutations, which can lead to hard-to-debug crashes if the no-mutations invariant is violated.

The current implementation initializes the NSFastEnumerationState such that the mutationsPtr is pointing at a known-zero value, and then it never touches the mutationsPtr again. This doesn't match the way NSFastEnumeration is used in Obj-C.

Based on my reading of emitted LLVM IR, it appears that in Obj-C, a for-in loop desugars to the following code:

// for (id elt in collection) { body(elt); }
NSFastEnumerationState stackState = {0};
__unsafe_unretained id stackObjects[16];
NSUInteger count = [collection countByEnumeratingWithState:&stackState objects:stackObjects count:16];
if (count != 0) {
    unsigned long savedMutationsPtr = *stackState.mutationsPtr;
    do {
        for (NSUInteger i = 0; i < count; i += 1) {
            if (*stackState.mutationsPtr != savedMutationsPtr) {
                objc_enumerationMutation(collection);
            }
            body(stackState.itemsPtr[i])
        }
        count = [collection countByEnumeratingWithState:&stackState objects:stackObjects count:16];
    } while (count != 0);
}

There's two major differences between this code and what Swift is doing:

  1. The Swift code initializes mutationsPtr to a non-zero value, which seems to be pointless, and then never reads it, which means it can't detect mutations.

  2. For some reason the Swift code checks if the itemsPtr matches the stack-allocated array of objects, and if so, it then switches over the index and reads from that stack array. I see no reason for it to do this instead of just reading from the itemsPtr always.

@lilyball
Copy link
Mannequin Author

lilyball mannequin commented Jan 5, 2019

Also there's code in here that copies the struct properties to local variables, with a comment // ensure NO ivars of self are actually captured, and it's not clear why it does this either as the only blocks involved here are withUnsafeMutablePointer which is noescaping.

@lilyball
Copy link
Mannequin Author

lilyball mannequin commented Jan 5, 2019

Another potential concern: if I have an NSFastEnumerationIterator I can access next() after it's already returned nil, which will cause it to invoke countByEnumerating(with:objects:count:) repeatedly. The problem with this is NSFastEnumeration isn't designed with the expectation that you'll ever invoke the method again (with the same state) after it's returned 0. While this is probably ok with the built-in collections, it seems potentially problematic to assume it's always safe.

@lilyball
Copy link
Mannequin Author

lilyball mannequin commented Jan 5, 2019

The following seems like a much more faithful translation of the Obj-C version (completely untested of course):

public struct NSFastEnumerationIterator : IteratorProtocol {
    var enumerable: NSFastEnumeration
    var objects: (Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?, Unmanaged<AnyObject>?) = (nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil)
    var state = NSFastEnumerationState(state: 0, itemsPtr: nil, mutationsPtr: nil, extra: (0, 0, 0, 0, 0))
    var index = 0
    var count = 0
    var mutationsPtr: UInt? = nil

    public init(_ enumerable: NSFastEnumeration) {
        self.enumerable = enumerable
    }

    public mutating func next() -> Any? {
        if index >= count {
            index = 0
            count = withUnsafeMutablePointer(to: &self.objects) {
                let buffer = AutoreleasingUnsafeMutablePointer<AnyObject?>($0)
                return enumerable.countByEnumerating(with: &state, objects: buffer, count: 16)
            }
            if count == 0 { return nil }
            if mutationsPtr == nil {
                mutationsPtr = state.mutationsPtr.unsafelyUnwrapped.pointee
            }
        }
        if state.mutationsPtr.unsafelyUnwrapped.pointee != mutationsPtr.unsafelyUnwrapped {
            objc_enumerationMutation(enumerable)
        }
        defer { index += 1 }
        return state.itemsPtr.unsafelyUnwrapped[index]
    }
}

@belkadan
Copy link

belkadan commented Jan 7, 2019

Yeah, this looks a lot better. I assume state.mutationsPtr was being set to something non-nil because of incorrect nullability at one point, but it might have just been a misunderstanding of the API from the beginning.

The one reason I can think of to not always go through itemsPtr is that the optimizer can tell that's an exclusive access, but when we're talking about NSFastEnumeration I'm not sure how much that matters in practice. You only access each item once anyway.

If we want to avoid the "calling next() at the end will call countByEnumerating(with:objects:count:) again" problem, we could check if count == 0 && mutationsPtr != nil. That means we've called it at least once already. (Also, the local value isn't really a pointer; it's more a mutationState or something.)

cc @millenomi, @Catfish-Man

@Catfish-Man
Copy link
Member

This looks good to me, without having done a full comparison to the existing code. Thanks Lily!

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants