Comparing languages wrt energy, speed, and memory use

Comparing languages wrt energy, speed, and memory use

[Jerry]

This paper comparing 27 languages with respect to energy use, speed, and memory use is interesting. Of course Ada fares very well.

https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleFinal.pdf

It is linked from this Slashdot page which I'm sure is full of useless chatter.

https://developers.slashdot.org/story/22/02/20/0143226/is-it-more-energy-efficient-to-program-in-rust

Re: Comparing languages wrt energy, speed, and memory use

[Fernando Oleo Blanco]

Hello everybody,

I am going to leave a few comments regarding this paper that I believe
everybody should know. Most if not all of these points are known and
have been discussed pretty much everywhere; but a lot of people still
don't know them or decide to not know.

The programs are taken from the Programming Language Benchmark Game. It
is a really cool place that has been providing relevant performance
data for a lot of languages and comparisons between them.

Here are a few issues:

1. Quite a few languages are not using heavily optimised code. Ada is
   one of them. Some of those programs are written as direct
   translations from other languages from people that did not know the
   target language.
2. Quite a few of those implementations have not been touched in years.
   Some of the improvements that may have taken place in the
   language/compiler/tools may not be taken advantage of. For example,
   the Ada examples are compiled with -gnatNp. Can anybody say what
   that flag does? x)
3. C/C++/Rust program are competing on getting the best results. Other
   languages are lagging behind. For example, Fortran could do much
   better. For a couple of years, the Fortran community has been
   improving the code little by little and they have managed to improve
   their results.
4. There are a few controversies. Some languages are not allowed to use
   higher performance libraries while others are allowed their stl or
   equivalent that do actually use the same tools as those libraries.
   Thre are a few other examples.

As the very Game page says, do not take the benchmark seriously. But
the communities whose languages are on top, they do not care. Ada has
been left behind since very few or nobody is actually taking a look at
the code and optimising it...

We may want to improve some of these tests as a community :)

Here are some relevant links:
- Benchmark game:
https://benchmarksgame-team.pages.debian.net/benchmarksgame/
- Source code:
https://salsa.debian.org/benchmarksgame-team/benchmarksgame

Regards,
Fer

Am Sun, 20 Feb 2022 14:59:29 -0800 (PST)
schrieb Jerry <list_email@icloud.com>:

> This paper comparing 27 languages with respect to energy use, speed,
> and memory use is interesting. Of course Ada fares very well.
> 
> https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleFinal.pdf
> 
> It is linked from this Slashdot page which I'm sure is full of
> useless chatter.
> 
> https://developers.slashdot.org/story/22/02/20/0143226/is-it-more-energy-efficient-to-program-in-rust
> 

Re: Comparing languages wrt energy, speed, and memory use

[J-P. Rosen]

Le 22/02/2022 à 21:10, Fernando Oleo Blanco a écrit :
> I am going to leave a few comments regarding this paper that I believe
> everybody should know. Most if not all of these points are known and
> have been discussed pretty much everywhere; but a lot of people still
> don't know them or decide to not know.
[good remarks snipped]

Let me add another one: this benchmark does not consider the energy 
(electrical and human) needed to write and debug the program... That 
could also make a difference for Ada!

Real ecological balance, taking everything into account, is tricky...

-- 
J-P. Rosen
Adalog
2 rue du Docteur Lombard, 92441 Issy-les-Moulineaux CEDEX
Tel: +33 1 45 29 21 52
https://www.adalog.fr

Re: Comparing languages wrt energy, speed, and memory use

[G.B.]

On 22.02.22 21:10, Fernando Oleo Blanco wrote:
> Hello everybody,
> 
> I am going to leave a few comments regarding this paper that I believe
> everybody should know. Most if not all of these points are known and
> have been discussed pretty much everywhere; but a lot of people still
> don't know them or decide to not know.
> 
> The programs are taken from the Programming Language Benchmark Game. It
> is a really cool place that has been providing relevant performance
> data for a lot of languages and comparisons between them.
> 
> Here are a few issues:

One issue is Isaac Gouy's clever approach. (Not complaining. I sometimes
didn't see the point, though, of adopting another new thing. For example,
when a new regex library was introduced (at some point) that wins hands
down by using optimization techniques you'd associate with JIT compilers
or with data based optimization. Worth knowing about, but how does it
help comparing languages when all you can do is link it to your program?)


> 1. Quite a few languages are not using heavily optimised code. Ada is
>     one of them. Some of those programs are written as direct
>     translations from other languages from people that did not know the
>     target language.

Can you be specific? For example, at least one program currently
leads by making extensive use of x86 intrinsic ops.

Some use OMP with intrinsic 128bit ops. Does GNAT have a similar parallel
loop in the language yet?

  
> 2. Quite a few of those implementations have not been touched in years.

Yet, some Ada program versions #N+m used to run faster than #N. They now
have their speed difference wiped out or even reversed...
I see -march=ivybridge now, so the hardware has likely changed.

>     Some of the improvements that may have taken place in the
>     language/compiler/tools may not be taken advantage of. For example,
>     the Ada examples are compiled with -gnatNp. Can anybody say what
>     that flag does? x)

GNAT User's Guide explains. (su-p-press and front end i-N-lining)

> 3. C/C++/Rust program are competing on getting the best results. Other
>     languages are lagging behind. For example, Fortran could do much
>     better.

How would Fortran do much better? Can Ada learn from that?

> 4. There are a few controversies. Some languages are not allowed to use
>     higher performance libraries while others are allowed their stl or
>     equivalent that do actually use the same tools as those libraries.
>     Thre are a few other examples.

Controversies add to the management's keeping this a lively benchmark.

> Ada has
> been left behind since very few or nobody is actually taking a look at
> the code and optimising it...

Care to be specific how it is not optimized? There is always room for
better algorithms. Looking at fasta, the leading programs all use threading
now, with the exception of the one at the top, which has an introductory
comment.

Re: Comparing languages wrt energy, speed, and memory use

[Fernando Oleo Blanco]

Hi,

answer is interlaced with your text.

Am Thu, 24 Feb 2022 08:42:40 +0100
schrieb "G.B." <bauhaus@notmyhomepage.invalid>:

> One issue is Isaac Gouy's clever approach. (Not complaining. I
> sometimes didn't see the point, though, of adopting another new
> thing. For example, when a new regex library was introduced (at some
> point) that wins hands down by using optimization techniques you'd
> associate with JIT compilers or with data based optimization. Worth
> knowing about, but how does it help comparing languages when all you
> can do is link it to your program?)
> 

True, that is the case of GMP for some multiprecission test... Some
languages use it, other implement the algorithms themselves...

> 
> Can you be specific? For example, at least one program currently
> leads by making extensive use of x86 intrinsic ops.
> 
> Some use OMP with intrinsic 128bit ops. Does GNAT have a similar
> parallel loop in the language yet?
> 

Yes, take a look at
https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-gnat-2.html
it is taken from the Pascal implementation and uses intrinsics. My
point is that some of these programs are not very Ada-like. As far as I
remember, there was one ported from Lua.

Ada 2022 will have a parallel keyword. However, it is still not
supported in FSF GNAT, which is the one being used. Also, the
benchmarks are Ada 2012.

> 
> Yet, some Ada program versions #N+m used to run faster than #N. They
> now have their speed difference wiped out or even reversed...
> I see -march=ivybridge now, so the hardware has likely changed.
> 
> >     Some of the improvements that may have taken place in the
> >     language/compiler/tools may not be taken advantage of. For
> > example, the Ada examples are compiled with -gnatNp. Can anybody
> > say what that flag does? x)
> 
> GNAT User's Guide explains. (su-p-press and front end i-N-lining)
> 

Correct, but that switch has been deprecated for years, it is no longer
documented anywhere in the new GNAT releases:
https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gnat_ugn.pdf

> > 3. C/C++/Rust program are competing on getting the best results.
> > Other languages are lagging behind. For example, Fortran could do
> > much better.
> 
> How would Fortran do much better? Can Ada learn from that?
> 

Fortran is using Intel's compiler, which is known to be one of the best.
Fortran compilers can much more easily generate SIMD code and
parallelise loops automatically if the code is idiomatic.

Also, Fortran was not fourth in the race a while ago. For example Ada
overtook Fortran for a small while. December 2018:
https://web.archive.org/web/20181204085050/https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fast.html
Ada is fourth; while it was fifth in April of that same year
https://web.archive.org/web/20180406194535/https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html
A year later, December 2019, Fortran could be fourth it it were not for
that outlier
https://web.archive.org/web/20191225172425/https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html

These are the current results:
https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html
Take a look at the evolution of the language podium. It has always been
C/C++/Rust, but starting from the fourth position there has been quite
a bit or rivalry.

> > 4. There are a few controversies. Some languages are not allowed to
> > use higher performance libraries while others are allowed their stl
> > or equivalent that do actually use the same tools as those
> > libraries. Thre are a few other examples.
> 
> Controversies add to the management's keeping this a lively benchmark.
> 
> > Ada has
> > been left behind since very few or nobody is actually taking a look
> > at the code and optimising it...
> 
> Care to be specific how it is not optimized? There is always room for
> better algorithms. Looking at fasta, the leading programs all use
> threading now, with the exception of the one at the top, which has an
> introductory comment.
> 

Some Ada programs could use better algorithms, data structures, more
up-to-date syntax and parallelism. Some programs could also be made a
bit prettier.


The crux of the issue is that you can pretty much always get peak
performance for non-GC languages if you use the same techniques,
libraries, algos, state of the art compilers, etc. And in a lot of real
world cases, even GC languages are not an issue, see Go, Erlang, Julia,
Lisp (SBCL), Nim...

But as someone (I believe it was the dean of TUM (Technische Universität
München)) once said: "Everybody knows that rankings are flawed, but it
is always better to be on top." The benchmark game, is after all, a
game. But some people took it to seriously. It is just like Football
hooligans.


Regards,
Fer

Re: Comparing languages wrt energy, speed, and memory use

[25.BX943]

On 2/20/22 5:59 PM, Jerry wrote:
> This paper comparing 27 languages with respect to energy use, speed, and memory use is interesting. Of course Ada fares very well.
> 
> https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleFinal.pdf
> 
> It is linked from this Slashdot page which I'm sure is full of useless chatter.
> 
> https://developers.slashdot.org/story/22/02/20/0143226/is-it-more-energy-efficient-to-program-in-rust

   After 30+ years, I started messing around with FORTRAN again.
   One of the things I noticed in the various help notes online
   was that programmers were actually comparing the numbers of
   cycles and executables size for various ways of solving any
   particular problem.

   This sort of thinking is rarely seen these days except in the
   microcontroller universe - and less even there because the
   RAM/ROM and speed of those devices has increased.

   ADA is another language where overall "efficiency" gets at
   least some consideration.

   With energy costs rising, maybe it's time to see MORE of these
   discussions and comparisons. Global warming be damned - this is
   a MONEY issue !  :-)

   Oh, and rising power costs may disappear the crypto sector.
   Those boxes full of GPUs calculating like mad - the power
   usage is stupendous. Once the energy in begins to exceed the
   value of the Bitcoins out - it's all over.

Re: Comparing languages wrt energy, speed, and memory use

[Robin Vowels]

On Monday, February 21, 2022 at 9:59:32 AM UTC+11, Jerry wrote:
> This paper comparing 27 languages with respect to energy use, speed, and memory use is interesting. Of course Ada fares very well. 
> 
> https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleFinal.pdf 
.
Has this anything to do with reality?
.
What of the design, testing, and maintainability of programs?

Re: Comparing languages wrt energy, speed, and memory use

[Jeffrey R.Carter]

On 2022-02-27 09:05, Robin Vowels wrote:
> On Monday, February 21, 2022 at 9:59:32 AM UTC+11, Jerry wrote:
>> This paper comparing 27 languages with respect to energy use, speed, and memory use is interesting. Of course Ada fares very well.
>>
>> https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleFinal.pdf
> .
> Has this anything to do with reality?
> .
> What of the design, testing, and maintainability of programs?

There are a couple of obvious problems with this study. First, the same data 
structures, algorithms, and checks for validity of input and so on, in any 
imperative language, should give very similar machine code. Robert Dewar 
famously had a collection of equivalent Ada and C programs that produced 
identical machine code when compiled with gcc. The kind of differences reported 
between C and Ada or C++ shows that they are comparing apples to orangutans.

Second, there are hard data that show that, compared to low-level languages like 
C, Ada requires 1/2 the effort to reach deployment, and 1/40 the effort to 
correct post-deployment errors. The energy consumption for that additional 
effort should swamp the kind of small differences during execution that this 
study concentrates on.

-- 
Jeff Carter
“A key difference between Ada and most other
software development languages is that Ada is
deigned as an engineering tool as well as a
programming tool.”
Ada Distilled
209