@swift-ci create

I don’t think you can use Resident Set Size as an indicator for leaks. If we keep asking and freeing memory the OS does necessearily need to reclaim those map pages until it is under memory pressure … 22MB is probably not creating that pressure.

Did you mean to say "OS does not necessarily need to reclaim"?

What explains the variance (12MB vs 17MB) between runs?

Yes I meant to say "does not necessarily need to reclaim".

Given that MAX_RSS is not good enough to detect leaks, I’m not insisting it is a leak. I’m trying to get the performance tests to behave deterministically – consistent memory usage between runs seems like a sensible requirement to me.

What explains the huge variance in MAX_RSS between runs here? 12 MB vs 17 MB, identical number of iterations. I’m pointing at MonteCarloE as most exacerbated example, but all the Array benchmarks have this instability. That’s my core concern. Leak was just a guess.

Try running with a consistent number of iterations (vs. samples).

The benchmark suite will run a test for varying number of iterations if you don't to that.

$ /usr/bin/time -lp $PWD/bin/Benchmark_O --num-iters=110 MonteCarloE 2>&1 | grep max
  46313472  maximum resident set size
$ /usr/bin/time -lp $PWD/bin/Benchmark_O --num-iters=110 MonteCarloE 2>&1 | grep max
  46313472  maximum resident set size
$ /usr/bin/time -lp $PWD/bin/Benchmark_O --num-iters=110 MonteCarloE 2>&1 | grep max
  46313472  maximum resident set size
$ /usr/bin/time -lp $PWD/bin/Benchmark_O --num-iters=110 MonteCarloE 2>&1 | grep max
  46313472  maximum resident set size
$ /usr/bin/time -lp $PWD/bin/Benchmark_O --num-iters=110 MonteCarloE 2>&1 | grep max
  46309376  maximum resident set size
$ /usr/bin/time -lp $PWD/bin/Benchmark_O --num-iters=110 MonteCarloE 2>&1 | grep max
  46313472  maximum resident set size

--verbose will show you how many iterations it runs (I don't know why we called it "scale" in the verbose output)

$ bin/Benchmark_O --verbose MonteCarloE
Verbose
--- CONFIG ---
NumSamples: 1
Verbose: true
IterScale: 1
Tests Filter: ["MonteCarloE"]
Tests to run: MonteCarloE, 

--- DATA ---
#,TEST,SAMPLES,MIN(us),MAX(us),MEAN(us),SD(us),MEDIAN(us)
Running MonteCarloE for 1 samples.
    Measuring with scale 109.
    Sample 0,9086
124,MonteCarloE,1,9086,9086,9086,0,9086

Totals,1,9086,9086,9086,0,0
$ bin/Benchmark_O --verbose MonteCarloE
Verbose
--- CONFIG ---
NumSamples: 1
Verbose: true
IterScale: 1
Tests Filter: ["MonteCarloE"]
Tests to run: MonteCarloE, 

--- DATA ---
#,TEST,SAMPLES,MIN(us),MAX(us),MEAN(us),SD(us),MEDIAN(us)
Running MonteCarloE for 1 samples.
    Measuring with scale 106.
    Sample 0,9141
124,MonteCarloE,1,9141,9141,9141,0,9141

Totals,1,9141,9141,9141,0,0

Wow! Thank you, that explains the variance between run.

I read somewhere that Benchmark_0 will time 1 iteration and derive N to make it run for approximately 1 second. So between different invocations, it is choosing different Ns, because the timing baseline varies, too.

So, the whole business with averaging MAX_RSS is really misguided - this (combined with delayed memory releases for autoreleased objects) is the source of variance. Different number of iterations will accumulate different amount of garbage in the pool - which manifests as different MAX_RSS between runs.
🤦

Still… am I correct in assuming that this issue exist only in code that interoperates with ObjC? Or does pure Swift also have some autoreleased objects?

autoreleased objects come from having to deal with the return calling convention of objective c apis. They return their references at +0 (unowned) but because they need to relinquish ownership but at the same time keep the object alive for the duration of the call they put the object into the autorelease pool for deferred release.

And Array is an ObjC object abiding by this convention under the hood?