Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-5278] JSON number should be decodable/encodable to Decimal value in Swift 4 encoding/decoding #4286

Closed
swift-ci opened this issue Jun 22, 2017 · 10 comments

Comments

@swift-ci
Copy link
Contributor

Previous ID SR-5278
Radar rdar://problem/32926804
Original Reporter JesperL (JIRA User)
Type Bug
Status Resolved
Resolution Done
Environment

The Swift 4 version shipped with Xcode Version 9.0 beta 2 (9M137d)

Additional Detail from JIRA
Votes 0
Component/s Foundation, Standard Library
Labels Bug
Assignee @itaiferber
Priority Medium

md5: 1fb127ee67f4b6f2d60cfb60b8149793

relates to:

  • SR-5249 Codable throws error when decoding an Int/Float from a JSON string

Issue Description:

The JSON standard does not prescribe that numbers must be decoded as binary floating point (Float and Double). Because of its lossy nature, it would be very useful to be able to avoid conversion between decimal and binary floating point, and thus to maintain precision.

The following playground:

let json = "{\"num\": 0.1}"
let jsonData = json.data(using: .utf8)!

struct PayloadDecimal: Codable {
    let num: Decimal
}
struct PayloadDouble: Codable {
    let num: Double
}

let jsonDecoder = JSONDecoder()
do {
    let decodedObjectDecimal = try jsonDecoder.decode(PayloadDecimal.self, from: jsonData)
    let num = decodedObjectDecimal.num
} catch {
    error // this is reached
}
do {
    let decodedObjectDouble = try jsonDecoder.decode(PayloadDouble.self, from: jsonData)
    let num = decodedObjectDouble.num // this is reached
} catch {
    error
}

produces a correctly decoded number (0.1) for the Double case and an error for the Decimal case: Expected to decode Dictionary<String, Any> but found a number instead., seemingly because Decimal only supports keyed encoding.

I understand that while Decimal's implementation of Encodable can detect a JSONEncoder, there's no mechanism to access the raw values from within the JSON structure. One alternative is to add this. Another alternative is special-casing Decimal: public func decode(_ type: Decimal.Type) throws -> Decimal could be added to SingleValueDecodingContainer, and so on for the other protocols.

My motivations why this is a bug are:

  • Decoding to and encoding from the customary decimal arithmetic type is supported by similar JSON handling in popular libraries for other languages.

  • It is hard to work around without transforming the JSON data, which is either invasive or impossible.

  • The way in which it fails is surprising, since the reason for the keyed archiving behavior for Decimal is not obvious in the context of JSON.

  • The potentially user-actionable "workaround" of declaring a Codable type that encodes or decodes a decimal value from the JSON by actually encoding or decoding a Double only works by forcing the conversion to binary arithmetic, which defeats the purpose of using Decimal in the first place.

@belkadan
Copy link

cc @itaiferber

@itaiferber
Copy link
Contributor

It's possible for us to attempt a few things in Decimal's constructor — attempting to initialize from a Double, Float, String, etc. We can also allow JSONEncoder/JSONDecoder to intercept Decimal values to output correctly.

@itaiferber
Copy link
Contributor

@swift-ci create

@swift-ci
Copy link
Contributor Author

Comment by Jesper Lindholm (JIRA)

That sounds like a good idea. The following is a good example of why going through an IEEE 754 binary floating point type like Double or Float is best avoided if at all possible:

let json = "{\"num\": 0.1, \"otherNum\": 0.10000000000000001}"
let jsonData = json.data(using: .utf8)!

struct PayloadDouble: Codable {
    let num: Double
    let otherNum: Double
}

let jsonDecoder = JSONDecoder()
do {
    let decodedObjectDouble = try jsonDecoder.decode(PayloadDouble.self, from: jsonData)
    let num = decodedObjectDouble.num
    let otherNum = decodedObjectDouble.num
    let areEqual = num == otherNum // -> true !!!
} catch {
    error
}

The example above is Double because that's all that will run now, but if the type above was Decimal and decoding meant going through Double, that would still happen even for Decimal. Even if you are in the odd position that you use Decimal but don't care about the precision, re-encoding the JSON wouldn't roundtrip to the same data.

@bobergj
Copy link

bobergj commented Jun 28, 2017

It's possible for us to attempt a few things in Decimal's constructor — attempting to initialize from a Double, Float, String, etc.

For decimal numbers, it would be best if it could construct a Decimal directly from a NSDecimalNumber. As far as I know the underlying NSJsonSerialization library will attempt to parse numbers that do not fit in `long long` as NSDecimalNumber.

Regarding initialisation from Strings, I don't think that is a good idea, because what locale would it attempt to use? The locale would have to be configurable, but that also only covers a restricted set of use cases where the string has a format supported by Decimal.
Decimal has

init?(string: String, locale: Locale? = default), 

But leaving the locale as default is only really recommended for user input, not for parsing values in a known format, such as from a Json API.

@itaiferber
Copy link
Contributor

Completed with apple/swift#10553

@itaiferber
Copy link
Contributor

@bobergj Bridging to/from NSDecimalNumber is exactly the way to go, and this is what I've done. Should maintain compatibility in a wide variety of cases.

Also, in the case of string conversion, we would likely use the en_US_POSIX locale by default — for non-standard cases, it seems reasonable to require custom handling of such values.

@bobergj
Copy link

bobergj commented Jun 28, 2017

@itaiferber Should you decide to go for "en_US_POSIX" by default, you may want to fix https://bugs.swift.org/browse/SR-5326

@swift-ci
Copy link
Contributor Author

Comment by Jesper Lindholm (JIRA)

Sweet - this seems to solve the problem. Looking forward to seeing this in an upcoming beta. Quick work!

@swift-ci
Copy link
Contributor Author

swift-ci commented Sep 1, 2017

Comment by Bill Boland (JIRA)

I'm sorry in advance if this is not the proper forum for such a question. I was looking to see how Decimal may have adopted the Codable protocol as I have many JSON objects using Decimal values. We found that they are not always representable accurately as Double/Float. For example, in the Playground example to this issue, if the value is 0.81,

let json = "{\"num\": 0.81}"

the resulting decoded Decimal is printed as 0.8100000000000002048 (with Xcode 9 beta 6). So, we tend to encode and decode to/from Strings instead.

I was intrigued by the comments about using a String representation (which is what we would desire to maintain accuracy). But this did not seem to work (resulting in errors) when changing the playground json value to:

let json = "{\"num\": \"0.81\"}"

Is decoding from (and encoding to) a string something that is in the works or I need to implement myself? It wasn't clear from the discussion. Note: It would be nice, akin to Date, to have some control over this (decimalDecodingStrategy/decimalEncodingStrategy)

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 5, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants