Updated benchmark data for Dalvik and ART running apps on Android 4.4.2 Kit Kat on a Google Nexus 5

This article provides an update to the performance results from benchmarks running on the new Android ART runtime on Android 4.4.2.  This is not a review.  ART is only meant for developers at this time, so don’t use it if you have critical apps on your phone.

My previous article is still available: Dalvik versus ART benchmarks on a Nexus 5.

I also made a 40-minute video of all of the benchmarks running on the Nexus 5:

Test setup and brief introduction to ART:

If you have a smartphone running Android 4.4.2 “Kit Kat”, and you have activated the developer options in settings, you may have noticed a new option to select the runtime.  In all previous versions of Android, the runtime was “Dalvik”.  Google now offers “ART”.  ART will one day let you run all of your apps faster and with less power by using ahead-of-time compiling.  As of May 2014, ART is still not at that point, but Google wants developers to try out the new runtime and give feedback.

ART is not meant for the general public yet.  There are still bugs, and some apps won’t work.

Once again, I wanted to see how ART does at running benchmarks with the updated Android 4.4.2 release.  I used two Google Nexus 5 smartphones running Android 4.4.2.  This time, I ran all of the tests “unplugged” (on the battery). The specific hardware and firmware is shown in the screenshot below.

When the “Developer options” are activated, they show up in “Settings” just above “About phone”. You can switch to ART under the “Select runtime” option. Without “Developer options”, the phone defaults to “Dalvik”. The screenshot below shows one phone running Dalvik and one running ART.

All of the benchmark apps are available for free on the Play Store.  I wanted apps that people could try on their current phones for comparison.

I ran all of the tests on the Dalvik and ART phones.  I only ran each test one time, and the phones were not reinitialized, so I may have some incorrect results.

Quadrant Standard Edition by Aurora Softworks:

According to the developer, “Quadrant is a CPU, I/O and 3D graphics benchmark.”

I did not use airplane mode for this benchmark, as it requires a network connection.

Quadrant Standard Edition is available on Google Play:

https://play.google.com/store/apps/details?id=com.aurorasoftworks.quadrant.ui.standard

 

GFXBench 2 by Kishonti Ltd.:

According to the developer, “Using console-quality gaming content, GFXBench (formerly GLBenchmark 2.7) provides a tool for unified cross-platform graphics benchmarking on major consumer platforms including Android, iOS and Windows.”

I did not use airplane mode for this benchmark, as it requires a network connection.

GFXBench 3 is available on Google Play;

https://play.google.com/store/apps/details?id=com.glbenchmark.glbenchmark27

AndEBench by EEMBC:

According to the developer, “The AndEBench™ benchmark provides a standardized, industry-accepted method of evaluating Android platform performance.”

I ran this benchmark in airplane mode.

AndEBench is available on Google Play;

https://play.google.com/store/apps/details?id=com.eembc.coremark

3DMark by Futuremark Oy:

According to the developer, 3DMark is “the world’s most popular performance test”.

I ran this benchmark in airplane mode.

3DMark is available on Google Play:

https://play.google.com/store/apps/details?id=com.futuremark.dmandroid.application

AnTuTu Benchmark by Antutu:

According to the developer, “AnTuTu Benchmark is free benchmarking app that helps users know deeper into their Android devices through hard numbers and solid facts.”

I ran this benchmark in airplane mode.

AnTuTu Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=com.antutu.ABenchMark

Basemark X by Rightware:

According to the developer, “Basemark® X by Rightware® is professional performance evaluation tool for objective cross platform comparisons of game graphics performance between Android, iOS and Windows Phone 8 devices.”

I ran this benchmark in airplane mode.

Basemark X is available on Google Play:

https://play.google.com/store/apps/details?id=com.rightware.BasemarkX_Free

Vellamo Mobile Benchmark by Qualcomm Connected Experiences:

According to the developer, “Vellamo is designed to be an accurate, easy-to-use suite of system-level benchmarks for devices based on Android 2.3 forward.”

I ran this benchmark in airplane mode.

Vellamo Mobile Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=com.quicinc.vellamo

Pi Benchmark by Sasa D. Markovic:

According to the developer, “Pi Benchmark is a tool to measure the real speed of CPU and memory of your phone.”

I ran this benchmark in airplane mode.

Pi Benchmark is available on Google Play:

https://play.google.com/store/apps/details?id=rs.in.luka.android.pi

Prime Number Generator by Adrian Campos:

According to the developer, Prime Number Generator is “Just a prime number generator. Nothing special. It will allow you to enter any number to start with and to end with, and then share the results.”

I ran this app in airplane mode.  It was not designed as a benchmark, so I counted video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I generated prime numbers from 2 to 100 000. Also, you may have noticed that these are old screenshots from the earlier tests. I forgot to take new screenshots for the Prime Number Generator, but I did insert the updated times.

Prime Number Generator is available on Google Play;

https://play.google.com/store/apps/details?id=com.adriancampos.primenumbers

Lame4Android by Intervigil:

According to the developer, “Lame4Android brings the LAME MP3 encoder to the Android smartphone platform.”

I ran this app in airplane mode.  It was not designed as a benchmark, so I counted video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I encoded a ten minute mono 44 KHz audio file (local French news program) at the “Extreme” setting.

Lame4Android is available on Google Play;

https://play.google.com/store/apps/details?id=com.intervigil.lame

ASTRO File Manager with Clouds by Metago:

According to the developer, “ASTRO Cloud & File Manager has over 70 million downloads worldwide!”

I ran this app in airplane mode.  It was not designed as a benchmark, so I counted video frames to determine the run times.  (I inserted the blue text on the screenshots.)  I used the built-in zip function to compress a 380 MB file.  The test file was the map of France in Mappy GPS Free by Mappy S.A.

ASTRO File Manager with Clouds is available on Google Play:

https://play.google.com/store/apps/details?id=com.metago.astro

Summary of results:

Benchmark	Editor	Sub-test	Dalvik result	ART result
Quadrant Standard Edition	Aurora Softworks		8932	12583
GFXBench 3	Kishonti Ltd.	T-Rex HD Onscreen	1361 (24 FPS)	1365 (24 FPS)
AndEBench	EEMBC	Native	14123	13267
AndEBench	EEMBC	Java	506	295
3DMark	Futuremark Oy		17152	16610
AnTuTu Benchmark	AnTuTu		23774	21999
Basemark X	Rightware	On-Screen	17.242	17.147
Basemark X	Rightware	Off-Screen	13.165	13.569
Vellamo Mobile Benchmark	Qualcomm Connected Experiences	HTMl 5	1257	1189
Vellamo Mobile Benchmark	Qualcomm Connected Experiences	Metal	1178	1143
Pi Benchmark	Sasa D. Markovic	100000 digits	18.04 seconds	11.51 seconds
Prime Number Generator	Adrian Campos	Primes from 2-100000	8.1 seconds	6.6 seconds
Lame4Android	Intervigil	Compression of a 10 minute audio file	16.5 seconds	15.9 seconds
ASTRO File Manager with Clouds	Metago	zip function on GPS map file	122.2 seconds	121.0 seconds

Dalvik versus Art benchmark results on Android 4.4.2

All of the apps in this study ran flawlessly on the new ART runtime.  Most of the benchmarks ran at similar times, but some did show a significant difference in performance.  Google will certainly continue to work on improving ART before possibly making it the main runtime.  Until then, it would be a good idea for developers to test their apps on ART.  You can let me know what you think of the results in your comments below.