> On Wed 25 Apr 2012 22:39, ludo@gnu.... (Ludovic Courtès) writes:
>>> So, those are the problems: benchmarks running for inappropriate,
>>> inconsistent durations;
>> I don’t really see such a problem. It doesn’t matter to me if
>> ‘arithmetic.bm’ takes 2mn while ‘vlists.bm’ takes 40s, since I’m not
>> comparing them.
> Running a benchmark for 2 minutes is not harmful to the results, but it
> is a bit needless. One second is enough.
Well, duration has to be chosen such that the jitter is small enough.
Sometimes it could be 2mn, sometimes 1s.
>>> and benchmarks being optimized out.
>> That should be fixed.
> In what way? It would make those benchmarks different.
> Thesis: anything for which you would want to turn off the optimizer is
> not a good benchmark anyway.
Yes, it depends on the benchmarks. For instance, I once added
benchmarks for ‘1+’ and ‘1-’, because I wanted to see the impact of an
optimization to the corresponding VM instructions.
Nowadays peval would optimize those benchmarks out. Yet, the fact is
that I was interested in the performance of the underlying VM
instructions, regardless of what the compiler might be doing.