Frustrated by Magento? Then you’ll love
In recent years there’s this image that will make the rounds with a ranking of programming languages by energy efficiency. If you haven’t seen it, it looks like this.
I want to be careful here because I’m not a researcher. I don’t design studies and I don’t know how to design studies or anything about the environment folks will typically design studies in. This is intended as a critique of the methodology not a canceling of the research — if I’m off base here please let me know.
The first thing that stood out to me was JavaScript had a score of 4.45 and TypeScript had a significantly higher score of 21.50 which seemed — weird. TypeScript code is compiled into javascript code, so how could it be any less efficient? What exactly was this study measuring?
That led me to dig up the study itself and take a look at their methods. These numbers appear to come from this study, entitled Ranking Programming Languages by Energy Efficiency. Reading through the study, I learned their methodology included running a set of “well-known, diverse programming problems” in each of the target languages. The source of these “well-known, diverse programming problems” is a project called The Computer Language Benchmarks Game. Then I dug up their GitHub repo and I started to understand how that JavaScript/TypeScript number could be so wildly different.
The quality of the implementation of these benchmarks is — inconsistent. For example — the PHP implementation of the binary tree algorithm is pretty wild. It forks a few processes with pcntl_fork (typically considered an expensive operation) and uses the shmop extension. Neither of these things is really a thing in run-of-the-mill PHP code. The version of PHP also appears to be PHP 5 and not the more recently mega-optimized PHPs 7 and 8.
Similarly, the javascript implementation and TypeScript implementation of the binary tree algorithm end up using very different approaches. Also, digging into the Makefiles reveals the tests were run with Node.js version 7.9 — the odd numbered Node.js versions are the experimental versions and may not be particularly optimized for newer features like the class keyword present in only the compiled TypeScript.
Rather than being a generic test of a programming language’s energy efficiency, the study ends up being a test of a programming language’s energy efficiency combined with the efficiency of algorithm implementations of varying quality and using runtimes that don’t represent real-world use of the languages.
It’s still an interesting study and a lot of its work could be the basis for further research, but I wouldn’t draw too many conclusions from it.