cc @lplarson

I came to realize benchmark runs are pretty taxing for CI infrastructure, but I think it makes sense to run at least one benchmark per day, like a nightly build and store those results for the purposes of SR-4591.

I have noticed one of the benchmarks that affects SR-4499 suddenly improved in the last week.

SuffixAnySequence                | 27665    | 105      | -99.6%    | **263.47x** 
DropLastAnySequence              | 34460    | 105      | -99.7%    | **328.19x** 

I’m trying to find a commit that fixed it. At the moment the only process I could think of is to go through commit history in github.com/apple/swift, opening those associated with PRs and looking if the reviewers did request a benchmark. So far, there was none in the last 4 days...

I got lucky and found two benchmark runs in PR #8489 in a span of two days that show the improvement in those tests. But I’m sure we can do better… And, I still don’t know which commit introduced the improvement.

@lplarson The Benchmark_Driver has command submit that uploads results of benchmark to something called LNT host. It also refers to SVN revision. Is this some kind of legacy infrastructure that did what I’m requesting here? Or is this somehow still running somewhere?

It's legacy infrastructure that works with LNT. What kind of centralized system do you have in mind?

One that will allow me to solve the above mentioned scenario of searching for particular commit that caused a performance change. See SR-4591. Having a publicly accessible storage of benchmark logs (with git hashes, not just dates) would be a good start. One nightly run on master, when doing PRs with benchmark runs, store those too… There probably isn’t direct equivalence between a merge commit to master and the benchmark run on the merged branch - there could have been other merges/commits on master in between the benchmark run and the merge. But having some reference is better than none. But one benchmark per night might suffice.

Basically something that allows us to build to later add some kind of visualization on top.

BTW, what is LNT?

http://llvm.org/docs/lnt/