New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SR-871] Benchmark runner misreports minimum values #43483
Comments
Any ideas, Michael? |
For what it's worth, I haven't yet seen such output when I run |
This is bizarre since the code that calculates the minimum is: var samples = [UInt64](count: c.numSamples, repeatedValue: 0)
let sampler = SampleRunner()
for s in 0..<c.numSamples {
let time_per_sample: UInt64 = 1_000_000_000 * UInt64(c.iterationScale)
var scale : UInt
var elapsed_time : UInt64 = 0
// Compute the scaling factor if a fixed c.fixedNumIters is not specified.
scale = c.fixedNumIters
// Rerun the test with the computed scale factor.
elapsed_time = sampler.run(name, fn: fn, num_iters: scale)
...
// save result in microseconds or k-ticks
samples[s] = elapsed_time / UInt64(scale) / 1000
...
}
...
// Return our benchmark results.
return BenchResults(delim: c.delim, sampleCount: UInt64(samples.count),
min: samples.minElement()!, max: samples.maxElement()!,
mean: mean, sd: sd, median: internalMedian(samples))
} If the min element is greater than the maximum element then it would suggest that Array.minElement/Array.maxElement are incorrect. EDITED: Formatted the pasted code. |
It may be higher than the raw swift driver though. I will add some asserts that checks bounds to the swift driver. If it doesn't trigger it means that a script on top is misunderstanding the output of the low level driver. |
Gotcha! It's because
The cases I've seen confirm that a lexical comparison caused them. |
I'll create a pull request in a few moments. (My Python is a bit rusty.) |
Simple unit testing of the script would have caught this bug. I filed: |
Submitted fix as PR #1530 |
Attachment: Download
Additional Detail from JIRA
md5: c88582f5d81eeaf1671d9f61a060a88f
Issue Description:
While looking at some benchmarking anomalies for PR #1194, I noticed that benchmark results are sometimes logged with minimum values that are far greater than either the maximum or mean.
I've noticed these appearing on stdout before, but I assumed they were an artifact of the code that prints the results, and would not lead to miscalculated results. I was wrong.
I attached an example log containing numerous such mistakes. (I increased the number of samples to 25 to get more stable results; otherwise I haven't touched the benchmarking infrastructure.)
For example, look at the
Histogram
line in the snippet below:Since benchmark comparisons compare minimum values only, this bug makes all results produced by the current version of the benchmark suite highly suspect. For example, here is a diff for the snippet above:
The text was updated successfully, but these errors were encountered: