flutter test: concurrency and performance is very low #168268
Description
Steps to reproduce
Running flutter tests is very slow. tests that take only milliseconds to run, are run slowly in the test runner.
It seems that as well as a pretty inefficient run process, the concurrency settings are not honoured.
Lets take a look at the run process - as i understand it - firstly from the entry point:
flutter test
On different machines that I have access to, for the same test suite, the tests (about 2500) take between 1m30 and 10m to run.
Using my desktop as an example, its an Intel i9-14900KF - It has 24cores, 32 threads, peak freq of 6GHz. This machine also has 128GB RAM, and for storage, NVMe Samsung 990 Pro - its no slouch.
However, when running the tests with any concurrency setting, its impossible to make the CPU use more than 2 cores.
time flutter test
real 1m38.025s
user 4m9.351s
sys 1m1.180s
time flutter test -j 32
real 1m38.110s
user 4m9.876s
sys 1m1.392s
time flutter test -j 100
real 1m37.919s
user 4m8.537s
sys 1m1.240s
It seems firstly that concurrency has no impact at all, but actually the performance hits a maximum at 4
time flutter test -j 1
real 3m44.300s
user 3m52.690s
sys 0m59.870s
time flutter test -j 4
real 1m39.793s
user 4m7.636s
sys 1m0.708s
Running the tests we can see that there is some disk io, but nowhere near any sort of limit- disk reads are basically 0 (cached), disk writes are quite a bit, but still no real issue. On this machine /tmp is on nvme0n1.
03/05/25 13:35:29
r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util Device
0.00 0.0k 0.00 0.0% 0.00 0.0k 1442.50 170.9M 1.50 0.1% 12.28 121.3k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 17.72 7.6% nvme0n1
We can also see that there is little I/O Wait, and that the CPU is just chilling:
sar -u 1
Linux 6.11.0-21-generic (minnow) 03/05/25 _x86_64_ (32 CPU)
14:01:08 CPU %user %nice %system %iowait %steal %idle
14:01:09 all 6.34 0.00 2.16 0.00 0.00 91.50
14:01:10 all 6.59 0.00 2.23 0.03 0.00 91.15
14:01:11 all 5.67 0.00 2.22 0.00 0.00 92.11
14:01:12 all 7.71 0.00 1.86 0.00 0.00 90.43
14:01:13 all 8.22 0.00 2.17 0.00 0.00 89.61
14:01:14 all 6.14 0.00 1.83 0.00 0.00 92.04
14:01:15 all 5.55 0.00 2.17 0.00 0.00 92.28
14:01:16 all 6.22 0.00 1.92 0.00 0.00 91.86
14:01:17 all 6.12 0.00 1.98 0.00 0.00 91.90
14:01:18 all 5.90 0.00 2.23 0.00 0.00 91.87
Using execsnoop and tcpconnect we can see what's happening under the hood:
87.130 dart 3066280 3060994 0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json
87.133 flutter_tester 3066284 3060994 0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /tmp/flutter_tools.VFBOSQ/flutter_test_listener.QRNYMJ/listener.dart.dill
87.453 dart 3066298 3060994 0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json
87.456 flutter_tester 3066300 3060994 0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /tmp/flutter_tools.VFBOSQ/flutter_test_listener.IQKCTK/listener.dart.dill
87.774 dart 3066318 3060994 0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json
87.777 flutter_tester 3066320 3060994 0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /tmp/flutter_tools.VFBOSQ/flutter_test_listener.SLQPWE/listener.dart.dill
3049415 flutter_test 4 127.0.0.1 127.0.0.1 42051
3049439 flutter_test 4 127.0.0.1 127.0.0.1 43303
3049456 flutter_test 4 127.0.0.1 127.0.0.1 33745
3049476 flutter_test 4 127.0.0.1 127.0.0.1 36363
3049495 flutter_test 4 127.0.0.1 127.0.0.1 46323
3049514 flutter_test 4 127.0.0.1 127.0.0.1 34249
3049538 flutter_test 4 127.0.0.1 127.0.0.1 35085
3049556 flutter_test 4 127.0.0.1 127.0.0.1 41743
3049576 flutter_test 4 127.0.0.1 127.0.0.1 35867
3049594 flutter_test 4 127.0.0.1 127.0.0.1 35403
So for each test file - we are running pub deps, creating a source file in /tmp, compiling it, and running it, and this program is then connecting to a local webserver to do something..
Finding 1 - The test runner or the compiler is running pub deps many many times - it runs it 3 times before dartaotruntime and then once for each test file. This takes a long time - even on a very fast computer.
time /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json > /dev/null
real 0m0.218s
user 0m0.303s
sys 0m0.060s
However, this isn't a limiting factor: we just are not running the right number of test processes:
(with flutter test -j 32)
while true; do count=$(ps -elf | grep flutter_tester | grep -v grep | wc -l ); echo $(date +"%T.%3N") Count = $count; sleep 0.1; done
13:55:16.578 Count = 0
13:55:16.703 Count = 1
13:55:16.829 Count = 1
13:55:16.950 Count = 2
13:55:17.076 Count = 2
13:55:17.201 Count = 2
13:55:17.326 Count = 2
13:55:17.448 Count = 1
13:55:17.582 Count = 2
13:55:17.712 Count = 2
Finding 2 - For whatever reason, the specified concurrency is not honoured. or, Ahmdahl getting in the way, and a thing that should be done in parallel is being done serially by the job submission loop.
Even if we accept that each test must be independently compiled, and forked and connect using web sockets back to the main test co-ordinator, each of which things is a source of slowness... there is something happening in the test runner that means the tests are not running efficiently.
Expected results
Running tests should saturate either CPU or disk if its running properly.
Actual results
System is idle - tests take far too long to run.
Code sample
lots of system outputs here- the specific test code is not very relevant - its applicable to all tests.
Screenshots or Video
No response
Logs
Logs
[Paste your logs here]Flutter Doctor output
✓] Flutter (Channel stable, 3.29.2, on Ubuntu 24.04.2 LTS 6.11.0-21-generic, locale en_GB.UTF-8) [40ms]
• Flutter version 3.29.2 on channel stable at /home/richja/fvm/versions/3.29.2
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision c236373 (7 weeks ago), 2025-03-13 16:17:06 -0400
• Engine revision 18b71d6
• Dart version 3.7.2
• DevTools version 2.42.3
[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [53ms]
• Android SDK at /home/richja/Android/Sdk
✗ cmdline-tools component is missing
Run path/to/sdkmanager --install "cmdline-tools;latest"
See https://developer.android.com/studio/command-line for more details.
✗ Android license status unknown.
Run flutter doctor --android-licenses to accept the SDK licenses.
See https://flutter.dev/to/linux-android-setup for more details.
[✓] Chrome - develop for the web [15ms]
• Chrome at google-chrome
[✗] Linux toolchain - develop for Linux desktop [44ms]
✗ clang++ is required for Linux development.
It is likely available from your distribution (e.g.: apt install clang), or can be downloaded from https://releases.llvm.org/
✗ CMake is required for Linux development.
It is likely available from your distribution (e.g.: apt install cmake), or can be downloaded from https://cmake.org/download/
✗ ninja is required for Linux development.
It is likely available from your distribution (e.g.: apt install ninja-build), or can be downloaded from
https://github.com/ninja-build/ninja/releases
• pkg-config version 1.8.1
✗ GTK 3.0 development libraries are required for Linux development.
They are likely available from your distribution (e.g.: apt install libgtk-3-dev)
[!] Android Studio (not installed) [11ms]
• Android Studio not found; download from https://developer.android.com/studio/index.html
(or visit https://flutter.dev/to/linux-android-setup for detailed instructions).
[✓] IntelliJ IDEA Ultimate Edition (version 2025.1) [11ms]
• IntelliJ at /home/richja/.local/share/JetBrains/Toolbox/apps/intellij-idea-ultimate
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
[✓] Connected device (2 available) [68ms]
• Linux (desktop) • linux • linux-x64 • Ubuntu 24.04.2 LTS 6.11.0-21-generic
• Chrome (web) • chrome • web-javascript • Google Chrome 135.0.7049.114
[✓] Network resources [206ms]
• All expected network resources are available.