MATSim Benchmark

The performance of MATSim depends on a lot of different factors:

  • CPU-speed (although by far not always the limiting factor!)
  • Memory-Bus / -Controller (we’re moving huge amounts of memory, the faster the better)
  • File system (local Hard drive vs. RAID vs. NFS vs …)

To get a better understanding, under which circumstances MATSim performs best, we created a simple benchmark (performance test) that runs 20 iterations of a sample scenario with different settings. If you run the benchmark on your machine, we would be happy if you could send us your results.

Download and installation

Download the zip-file containing the benchmark: benchmark.zip  (35MB).

Unzip the downloaded file.

Running the benchmark

On the command line, run the following: 

java -Xmx500m -jar Benchmark.jar

This will generate a directory output with some files in it from the run. The test will usually run between 25 and 40 minutes. The benchmark requires Java 1.5 or newer and 150MB free disk space. If you want to re-run the benchmark, rename or delete the ./output/ directory and run the test again.

Submitting benchmark results

Please send an email to benchmark AT matsim DOT org containing:

  • the file output/stopwatch.txt
  • the file output/logfile.log
  • a description of your benchmark environment, including:
    • vendor of machine (e.g. HP, Dell, Apple, etc.)
    • processor-type (vendor (Intel, AMD, etc.), model-number, clock-speed, cache-size, number of processors and cores, …)
    • memory (bus-speed, memory-controller, etc.)
    • storage system (rpm, cache size, …, type: e.g. local hard drive, RAID, …)
    • operation system
  • any other information you think might be of interest to us

We collected some results and did a short analysis on them. Have a look at the results in the next section. If you want your results included as well or have some interesting findings yourself, please submit us your benchmark results.

 

00:05:46 Supermicro-system 2016

Supermicro XDRH-i motherboard

1x Intel Xeon E5-2690v4 (2.6GHz, 14 cores, HT, 35MB cache)

4x 32GB DDR4 memory PC2400 ECC (2400MHz)

SAMSUNG SSD 960 pro NVMe (1TB)

fresh Debian Jessie 8.7 64bit stable

$ java -version
java version "1.7.0_121"
OpenJDK Runtime Environment (IcedTea 2.6.8) (7u121-2.6.8-2~deb8u1)
OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)

############00:05:46#################
>>  java -Xmx500m -jar Benchmark.jar <<
$ head output/logfile.log
2017-03-07 17:30:42,781  INFO Gbl:158 JVM: 1.7.0_121; Oracle Corporation; mixed mode; 64-bit
2017-03-07 17:30:42,783  INFO Gbl:162 OS: Linux; 3.16.0-4-amd64; amd64
2017-03-07 17:30:42,783  INFO Gbl:165 CPU cores: 28
2017-03-07 17:30:42,785  INFO Gbl:166 max. Memory: 479.5MB (502792192B)
2017-03-07 17:30:42,785  INFO Gbl:188 MATSim-Build: 6528 (2009-04-22 01:01)

STOPWATCH.txt

Iteration       BEGIN iteration BEGIN replanning        END replanning  BEGIN dump all plans    END dump all plans      BEGIN mobsim    END mobsim      END iteration           replanning      dump all plans  mobsim  iteration
0       17:30:45                        17:30:45        17:30:46        17:30:46        17:31:01        17:31:01                        00:00:00        00:00:15        00:00:16
1       17:31:01        17:31:01        17:31:04        17:31:04        17:31:05        17:31:05        17:31:17        17:31:17                00:00:02        00:00:01        00:00:12        00:00:15
2       17:31:17        17:31:17        17:31:19                        17:31:19        17:31:32        17:31:32                00:00:02                00:00:13        00:00:15
3       17:31:32        17:31:32        17:31:34                        17:31:34        17:31:48        17:31:48                00:00:01                00:00:13        00:00:15
4       17:31:48        17:31:48        17:31:50                        17:31:50        17:32:03        17:32:03                00:00:02                00:00:13        00:00:15
5       17:32:03        17:32:03        17:32:05                        17:32:05        17:32:21        17:32:21                00:00:01                00:00:16        00:00:18
6       17:32:21        17:32:21        17:32:23                        17:32:23        17:32:36        17:32:36                00:00:02                00:00:12        00:00:14
7       17:32:36        17:32:36        17:32:38                        17:32:38        17:32:52        17:32:52                00:00:02                00:00:13        00:00:15
8       17:32:52        17:32:52        17:32:54                        17:32:54        17:33:08        17:33:08                00:00:02                00:00:14        00:00:16
9       17:33:08        17:33:08        17:33:10                        17:33:10        17:33:24        17:33:24                00:00:02                00:00:13        00:00:15
10      17:33:24        17:33:24        17:33:26        17:33:26        17:33:29        17:33:29        17:33:45        17:33:46                00:00:02        00:00:02        00:00:15        00:00:22
11      17:33:46        17:33:46        17:33:48                        17:33:48        17:34:02        17:34:02                00:00:02                00:00:13        00:00:15
12      17:34:02        17:34:02        17:34:04                        17:34:04        17:34:17        17:34:17                00:00:02                00:00:12        00:00:15
13      17:34:17        17:34:17        17:34:19                        17:34:19        17:34:32        17:34:32                00:00:02                00:00:12        00:00:15
14      17:34:32        17:34:32        17:34:34                        17:34:34        17:34:47        17:34:47                00:00:01                00:00:13        00:00:15
15      17:34:47        17:34:47        17:34:49                        17:34:49        17:35:05        17:35:05                00:00:01                00:00:15        00:00:17
16      17:35:05        17:35:05        17:35:07                        17:35:07        17:35:20        17:35:20                00:00:01                00:00:13        00:00:15
17      17:35:20        17:35:20        17:35:22                        17:35:22        17:35:36        17:35:37                00:00:02                00:00:14        00:00:16
18      17:35:37        17:35:37        17:35:39                        17:35:39        17:35:53        17:35:53                00:00:01                00:00:14        00:00:16
19      17:35:53        17:35:53        17:35:55                        17:35:55        17:36:08        17:36:09                00:00:01                00:00:13        00:00:15
20      17:36:09        17:36:09        17:36:11        17:36:11        17:36:13        17:36:13        17:36:30        17:36:31                00:00:01        00:00:02        00:00:16        00:00:22

 

############00:05:08#################
>> java -Xmx16g -jar Benchmark.jar <<

2017-03-07 17:50:36,846  INFO Gbl:158 JVM: 1.7.0_121; Oracle Corporation; mixed mode; 64-bit
2017-03-07 17:50:36,848  INFO Gbl:162 OS: Linux; 3.16.0-4-amd64; amd64
2017-03-07 17:50:36,849  INFO Gbl:165 CPU cores: 28
2017-03-07 17:50:36,850  INFO Gbl:166 max. Memory: 14563.5MB (15270936576B)

STOPWATCH.txt

Iteration       BEGIN iteration BEGIN replanning        END replanning  BEGIN dump all plans    END dump all plans      BEGIN mobsim    END mobsim      END iteration           replanning      dump all plans  mobsim  iteration
0       17:50:39                        17:50:39        17:50:40        17:50:40        17:50:58        17:50:58                        00:00:01        00:00:17        00:00:18
1       17:50:58        17:50:58        17:51:00        17:51:00        17:51:01        17:51:01        17:51:11        17:51:11                00:00:02        00:00:01        00:00:09        00:00:13
2       17:51:11        17:51:11        17:51:13                        17:51:13        17:51:23        17:51:23                00:00:02                00:00:09        00:00:12
3       17:51:23        17:51:23        17:51:25                        17:51:25        17:51:35        17:51:35                00:00:02                00:00:10        00:00:12
4       17:51:35        17:51:35        17:51:38                        17:51:38        17:51:48        17:51:48                00:00:02                00:00:10        00:00:12
5       17:51:48        17:51:48        17:51:50                        17:51:50        17:52:06        17:52:06                00:00:02                00:00:15        00:00:17
6       17:52:06        17:52:06        17:52:08                        17:52:08        17:52:18        17:52:19                00:00:02                00:00:10        00:00:12
7       17:52:19        17:52:19        17:52:21                        17:52:21        17:52:31        17:52:31                00:00:02                00:00:10        00:00:12
8       17:52:31        17:52:31        17:52:34                        17:52:34        17:52:44        17:52:44                00:00:02                00:00:10        00:00:12
9       17:52:44        17:52:44        17:52:46                        17:52:46        17:52:57        17:52:57                00:00:02                00:00:10        00:00:12
10      17:52:57        17:52:57        17:52:59        17:52:59        17:53:02        17:53:02        17:53:17        17:53:18                00:00:02        00:00:02        00:00:15        00:00:21
11      17:53:18        17:53:18        17:53:21                        17:53:21        17:53:31        17:53:31                00:00:02                00:00:10        00:00:12
12      17:53:31        17:53:31        17:53:33                        17:53:33        17:53:46        17:53:46                00:00:02                00:00:12        00:00:15
13      17:53:46        17:53:46        17:53:48                        17:53:48        17:54:02        17:54:02                00:00:02                00:00:13        00:00:15
14      17:54:02        17:54:02        17:54:04                        17:54:04        17:54:14        17:54:14                00:00:02                00:00:10        00:00:12
15      17:54:14        17:54:14        17:54:16                        17:54:16        17:54:32        17:54:32                00:00:02                00:00:15        00:00:17
16      17:54:32        17:54:32        17:54:34                        17:54:34        17:54:44        17:54:44                00:00:02                00:00:10        00:00:12
17      17:54:44        17:54:44        17:54:46                        17:54:46        17:54:59        17:54:59                00:00:02                00:00:12        00:00:15
18      17:54:59        17:54:59        17:55:02                        17:55:02        17:55:12        17:55:12                00:00:02                00:00:10        00:00:12
19      17:55:12        17:55:12        17:55:15                        17:55:15        17:55:27        17:55:27                00:00:02                00:00:12        00:00:14
20      17:55:27        17:55:27        17:55:29        17:55:29        17:55:32        17:55:32        17:55:46        17:55:47                00:00:02        00:00:02        00:00:13        00:00:20

 

00:05:47 MacBook Pro Late 2013

2.6 GHz Intel Core i7

16 GB 1600 MHz DDR3

SSD

Iteration       BEGIN iteration BEGIN replanning        END replanning  BEGIN dump all plans    END dump all plans      BEGIN mobsim    END mobsim      END iteration           replanning      dump all plans  mobsim  iteration

0       20:04:29                        20:04:29        20:04:29        20:04:29        20:04:44        20:04:45                        00:00:00        00:00:15        00:00:15

1       20:04:45        20:04:45        20:04:47        20:04:47        20:04:47        20:04:47        20:05:00        20:05:00                00:00:01        00:00:00        00:00:12        00:00:15

2       20:05:00        20:05:00        20:05:02                        20:05:02        20:05:14        20:05:14                00:00:02                00:00:12        00:00:14

3       20:05:14        20:05:14        20:05:16                        20:05:16        20:05:29        20:05:29                00:00:01                00:00:12        00:00:14

4       20:05:29        20:05:29        20:05:31                        20:05:31        20:05:44        20:05:44                00:00:02                00:00:12        00:00:14

5       20:05:44        20:05:44        20:05:46                        20:05:46        20:06:01        20:06:01                00:00:01                00:00:15        00:00:17

6       20:06:01        20:06:01        20:06:03                        20:06:03        20:06:17        20:06:17                00:00:02                00:00:13        00:00:15

7       20:06:17        20:06:17        20:06:19                        20:06:19        20:06:33        20:06:33                00:00:01                00:00:14        00:00:16

8       20:06:33        20:06:33        20:06:35                        20:06:35        20:06:49        20:06:50                00:00:02                00:00:14        00:00:16

9       20:06:50        20:06:50        20:06:52                        20:06:52        20:07:06        20:07:06                00:00:01                00:00:14        00:00:16

10      20:07:06        20:07:06        20:07:08        20:07:08        20:07:10        20:07:10        20:07:27        20:07:28                00:00:02        00:00:02        00:00:16        00:00:22

11      20:07:28        20:07:28        20:07:30                        20:07:30        20:07:45        20:07:45                00:00:02                00:00:15        00:00:17

12      20:07:45        20:07:45        20:07:47                        20:07:47        20:08:01        20:08:01                00:00:02                00:00:13        00:00:15

13      20:08:01        20:08:01        20:08:03                        20:08:03        20:08:16        20:08:16                00:00:01                00:00:13        00:00:15

14      20:08:16        20:08:16        20:08:18                        20:08:18        20:08:31        20:08:31                00:00:02                00:00:12        00:00:15

15      20:08:31        20:08:31        20:08:33                        20:08:33        20:08:48        20:08:48                00:00:01                00:00:14        00:00:16

16      20:08:48        20:08:48        20:08:50                        20:08:50        20:09:04        20:09:04                00:00:02                00:00:14        00:00:16

17      20:09:04        20:09:04        20:09:06                        20:09:06        20:09:21        20:09:21                00:00:02                00:00:14        00:00:16

18      20:09:21        20:09:21        20:09:23                        20:09:23        20:09:38        20:09:38                00:00:01                00:00:14        00:00:16

19      20:09:38        20:09:38        20:09:40                        20:09:40        20:09:54        20:09:54                00:00:01                00:00:13        00:00:15

20      20:09:54        20:09:54        20:09:56        20:09:56        20:09:58        20:09:58        20:10:14        20:10:16                00:00:01        00:00:02        00:00:16        00:00:21