Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-7195] JSONSerialization encodes Double differently on Darwin and Linux #3727

Closed
florianreinhart opened this issue Mar 14, 2018 · 6 comments

Comments

@florianreinhart
Copy link

Previous ID SR-7195
Radar None
Original Reporter @florianreinhart
Type Bug
Status Resolved
Resolution Done
Environment

Swift 4.0.3, macOS 10.13.3 and Ubuntu 16.04

Additional Detail from JIRA
Votes 0
Component/s Foundation
Labels Bug
Assignee None
Priority Medium

md5: e38af2c3f6bfc9b1fe10fcfba2bf9763

relates to:

  • SR-5961 JSONSerialization does not properly convert 1.1

Issue Description:

`JSONSerialization` and thus `JSONEncoder` encode floating point values in a different way on Linux than on Darwin. This can cause a loss of precision and decoding a different value than the one that was encoded on Linux.

Running the following code demonstrates the issue.

import Foundation

let value1 = 7.7087009966199993
let value2 = 7.7087009966200002

// Encode using JSONSerialization
do {
    print("Encoding using JSONSerialization")
    let dict1 = ["value": value1]
    let dict2 = ["value": value2]
    let jsonData1 = try! JSONSerialization.data(withJSONObject: dict1)
    let jsonData2 = try! JSONSerialization.data(withJSONObject: dict2)
    let jsonString1 = String(decoding: jsonData1, as: UTF8.self)
    let jsonString2 = String(decoding: jsonData2, as: UTF8.self)
    print("json1 is: \(jsonString1). Dump of input value follows.")
    dump(dict1)
    print("json2 is: \(jsonString2). Dump of input value follows.")
    dump(dict2)
    
    // Decode using JSONSerialization
    print("\nDecoding using JSONSerialization")
    let decodedDict1 = try! JSONSerialization.jsonObject(with: jsonData1) as! [String : Double]
    let decodedDict2 = try! JSONSerialization.jsonObject(with: jsonData2) as! [String : Double]
    print("decoded dictionary 1 is\(dict1 == decodedDict1 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedDict1)
    print("decoded dictionary 2 is\(dict2 == decodedDict2 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedDict2)
}

struct DoubleWrapper: Equatable, Codable {
    let value: Double
    
    static func ==(lhs: DoubleWrapper, rhs: DoubleWrapper) -> Bool {
        return lhs.value == rhs.value
    }
}

do {
    // Encode using JSONEncoder
    print("\n\n\nEncoding using JSONEncoder")
    let wrappedValue1 = DoubleWrapper(value: value1)
    let wrappedValue2 = DoubleWrapper(value: value2)
    let jsonEncoder = JSONEncoder()
    let jsonData1 = try! jsonEncoder.encode(wrappedValue1)
    let jsonData2 = try! jsonEncoder.encode(wrappedValue2)
    let jsonString1 = String(decoding: jsonData1, as: UTF8.self)
    let jsonString2 = String(decoding: jsonData2, as: UTF8.self)
    print("json1 is: \(jsonString1). Dump of input follows.")
    dump(wrappedValue1)
    print("json2 is: \(jsonString2). Dump of input follows.")
    dump(wrappedValue2)
    
    // Decode using JSONDecoder
    print("\nDecoding using JSONDecoder")
    let jsonDecoder = JSONDecoder()
    let decodedWrappedValue1 = try! jsonDecoder.decode(DoubleWrapper.self, from: jsonData1)
    let decodedWrappedValue2 = try! jsonDecoder.decode(DoubleWrapper.self, from: jsonData2)
    print("decoded value 1 is\(wrappedValue1 == decodedWrappedValue1 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedWrappedValue1)
    print("decoded value 1 is\(wrappedValue2 == decodedWrappedValue2 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedWrappedValue2)
}

Swift 4.0.3 on macOS 10.13.3 produces the following output and encodes and decodes all values as expected.

Encoding using JSONSerialization
json1 is: {"value":7.7087009966199993}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966199993
json2 is: {"value":7.7087009966200002}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002

Decoding using JSONSerialization
decoded dictionary 1 is equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966199993
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002



Encoding using JSONEncoder
json1 is: {"value":7.7087009966199993}. Dump of input follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966199993
json2 is: {"value":7.7087009966200002}. Dump of input follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966200002

Decoding using JSONDecoder
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966199993
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966200002

And Swift 4.0.3 on Ubuntu 16.04. Both input double values are encoded as 7.70870099662 in the resulting JSON.

Encoding using JSONSerialization
json1 is: {"value":7.70870099662}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966199993
json2 is: {"value":7.70870099662}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002

Decoding using JSONSerialization
decoded dictionary 1 is not equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002



Encoding using JSONEncoder
json1 is: {"value":7.70870099662}. Dump of input follows.
▿ test.DoubleWrapper
- value: 7.7087009966199993
json2 is: {"value":7.70870099662}. Dump of input follows.
▿ test.DoubleWrapper
- value: 7.7087009966200002

Decoding using JSONDecoder
decoded value 1 is not equal to the input. Dump of decoded value follows.
▿ test.DoubleWrapper
- value: 7.7087009966200002
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ test.DoubleWrapper
- value: 7.7087009966200002
@belkadan
Copy link

cc @itaiferber. There may have been a dup of this already.

@florianreinhart
Copy link
Author

@itaiferber, I am still seeing this issue in Swift 4.2 on Linux. Could you take a look at this?

@spevans
Copy link
Collaborator

spevans commented Oct 9, 2018

On Linux, JSONSerialization uses CFNumberFormatterCreateStringWithNumber to encode Float and Double and this ultimately uses ICU underneath. Interestingly in ICU-61 the Double -> String algorithm has changed (Ive been looking at upgrading the ICU on Linux) and was thinking that a better fix for scl-foundation would be to just use String to encode Double and Float. I have a patch that implements this if we want to make this change.

Note that Ubuntu18 currently uses ICU-60 so upgrading ICU to a later version would change the output format of JSONSerialization anyway but not in a way guaranteed to match Darwin.

@itaiferber
Copy link
Contributor

Looks like floating-point numbers are printed out via

let string = CFNumberFormatterCreateStringWithNumber(nil, _numberformatter, num._cfObject)._swiftObject
writer(string) 

with _numberformatter defined as

private lazy var _numberformatter: CFNumberFormatter = {
    let formatter: CFNumberFormatter
    formatter = CFNumberFormatterCreate(nil, CFLocaleCopyCurrent(), kCFNumberFormatterNoStyle)
    CFNumberFormatterSetProperty(formatter, kCFNumberFormatterMaxFractionDigits, NSNumber(value: 15))
    CFNumberFormatterSetFormat(formatter, "0.###############"._cfObject)
    return formatter
}() 

This indeed doesn't match what we do on Darwin, which is snprintf(bufPtr, NUMBERBUF_SIZE, "%0.*g", DBL_DECIMAL_DIG, [obj doubleValue]) (i.e. on Darwin we print with more digits of precision). However, this has been true on swift-corelibs-foundation since before Swift 4, so I'm not sure what changed here in Swift 4.0.3 specifically on Linux.

cc @spevans — you've worked more on JSONSerialization on swift-corelibs-foundation than I have. Any recollection why this may have happened? I don't have a Linux machine to test on at the moment but I can try to get something running.

@florianreinhart
Copy link
Author

@itaiferber, I just tested this with Swift 3.1.1 on Linux and issues exists there as well. So this not a regression in Swift 4.

@spevans, Using String to encode Double/Float should be a good solution for now. This ensures round-trip correctness, which is more important than having the exact same JSON on both platforms, I guess.

@spevans
Copy link
Collaborator

spevans commented Oct 14, 2018

Fixed in #1722

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 5, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants