HDD benchmark results
Results of simple benchmarks using mkfile(1M), dd(1) and filebench. The RW test is used to get a feeling for max. performance wrt. to sequential reads and writes. It writes out 4x32GiB (so ARC should be poissioned enough) using mkfile(1M) and after that it reads in those files using dd(1) in chunks of 128KiB. All other tests are made using the workflow definitions coming with filebench (fb) with run 60. The only parameter changed in the fb tests is the directory to point to the appropriate directory in the tested ZFS pool. We avoided explicitly changing other parameters to have "something comparable" out of the box. So depending on the workload/test, the results may not necessarily represent the max. performance the system + storage is able to provide! So to properly interprete the results, it is recommended to have a look at the workflow definition in question and related semantics.
For more test details, have a look at *.txt files and related sub directories.
NOTE: On Linux/Ubuntu we had to remove useism (use local instead of shared memory) from the oltp.f workload to get it work. Otherwise threads dead lock with FUTEX_WAIT when trying to attach the shared memory segment (ipc.c:ipc_ismmalloc).
RW
| config | write [MiB/s] | read [MiB/s] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 2482.5 | 10780.0 |
|---|
| 2x2-default | 2500.5 | 11164.6 |
|---|
| 2x2-disksort-nvcache-throttle32 | 2496.6 | 10469.0 |
|---|
| 2x2-disksort-nvcache | 2462.2 | 10886.5 |
|---|
bringover
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 6002.0 | 6001.4 | 1001.0 | 1000.0 | 30.0 | 113.5 | 0.0 |
|---|
| 2x2-default | 6002.0 | 6001.4 | 1001.0 | 1000.0 | 30.0 | 68.8 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 6002.0 | 6001.3 | 1001.0 | 1000.0 | 30.0 | 57.3 | 0.0 |
|---|
| 2x2-disksort-nvcache | 6002.0 | 5999.8 | 1000.8 | 999.8 | 30.0 | 152.5 | 0.0 |
|---|
copyfiles
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 6002.0 | 5999.9 | 1000.8 | 999.8 | 30.0 | 99.3 | 0.0 |
|---|
| 2x2-default | 6002.0 | 6001.3 | 1001.0 | 1000.0 | 30.0 | 59.8 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 6002.0 | 6001.3 | 1001.0 | 1000.0 | 30.0 | 96.3 | 0.0 |
|---|
| 2x2-disksort-nvcache | 6002.0 | 6001.4 | 1001.0 | 1000.0 | 30.0 | 67.0 | 0.0 |
|---|
createfiles
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 149974.8 | 149817.5 | 0.0 | 49936.8 | 780.2 | 250.5 | 0.3 |
|---|
| 2x2-default | 149978.0 | 149820.7 | 0.0 | 49938.5 | 780.2 | 253.0 | 0.3 |
|---|
| 2x2-disksort-nvcache-throttle32 | 149977.5 | 149818.4 | 0.0 | 49937.0 | 780.2 | 254.0 | 0.3 |
|---|
| 2x2-disksort-nvcache | 149979.5 | 149822.1 | 0.0 | 49938.5 | 780.2 | 259.0 | 0.3 |
|---|
deletefiles
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 50000.0 | 43721.2 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-default | 50000.0 | 12498.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 50000.0 | 37484.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache | 50000.0 | 49956.4 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
fileserver
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 13896035.0 | 230532.2 | 20957.5 | 41915.5 | 5595.7 | 338.5 | 0.4 |
|---|
| 2x2-default | 13917794.0 | 231243.5 | 21022.0 | 42044.5 | 5614.4 | 338.0 | 0.4 |
|---|
| 2x2-disksort-nvcache-throttle32 | 13858491.0 | 230286.8 | 20935.0 | 41871.0 | 5589.0 | 338.5 | 0.4 |
|---|
| 2x2-disksort-nvcache | 13882754.5 | 230867.8 | 20988.0 | 41977.0 | 5605.9 | 337.5 | 0.4 |
|---|
listdirs
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 23621515.0 | 393391.6 | 0.0 | 0.0 | 81.7 | 0.0 | 0.0 |
|---|
| 2x2-default | 23492385.0 | 391153.7 | 0.0 | 0.0 | 81.3 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 23431101.5 | 390187.2 | 0.0 | 0.0 | 81.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache | 23371997.0 | 389180.3 | 0.0 | 0.0 | 80.8 | 0.0 | 0.0 |
|---|
makedirs
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 10000.0 | 9999.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-default | 10000.0 | 9998.8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 10000.0 | 9998.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache | 10000.0 | 9999.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
mongo
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 96.8 | 0.0 |
|---|
| 2x2-default | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 58.3 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 89.5 | 0.0 |
|---|
| 2x2-disksort-nvcache | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 57.8 | 0.0 |
|---|
multistreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 819225.5 | 13651.9 | 13652.0 | 0.0 | 13638.5 | 296.0 | 0.3 |
|---|
| 2x2-default | 818920.5 | 13647.2 | 13647.5 | 0.0 | 13633.9 | 296.5 | 0.3 |
|---|
| 2x2-disksort-nvcache-throttle32 | 821146.0 | 13683.1 | 13683.0 | 0.0 | 13669.6 | 296.0 | 0.3 |
|---|
| 2x2-disksort-nvcache | 819422.0 | 13655.5 | 13655.0 | 0.0 | 13642.1 | 296.5 | 0.3 |
|---|
multistreamreaddirect
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 819253.5 | 13652.7 | 13653.0 | 0.0 | 13639.3 | 296.5 | 0.3 |
|---|
| 2x2-default | 825638.5 | 13758.4 | 13758.0 | 0.0 | 13744.9 | 294.0 | 0.3 |
|---|
| 2x2-disksort-nvcache-throttle32 | 820681.0 | 13675.8 | 13676.0 | 0.0 | 13662.5 | 296.0 | 0.3 |
|---|
| 2x2-disksort-nvcache | 820853.5 | 13678.4 | 13678.5 | 0.0 | 13665.0 | 296.0 | 0.3 |
|---|
multistreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 433007.5 | 7214.6 | 0.0 | 7214.5 | 7214.5 | 1261.5 | 0.5 |
|---|
| 2x2-default | 430987.5 | 7181.1 | 0.0 | 7181.5 | 7181.1 | 1281.0 | 0.6 |
|---|
| 2x2-disksort-nvcache-throttle32 | 425212.0 | 7085.7 | 0.0 | 7086.0 | 7085.6 | 1291.0 | 0.6 |
|---|
| 2x2-disksort-nvcache | 435494.0 | 7257.0 | 0.0 | 7257.0 | 7256.9 | 1256.0 | 0.5 |
|---|
multistreamwritedirect
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 441981.5 | 7364.8 | 0.0 | 7365.0 | 7364.7 | 1233.0 | 0.5 |
|---|
| 2x2-default | 440312.5 | 7337.3 | 0.0 | 7337.5 | 7337.2 | 1232.5 | 0.5 |
|---|
| 2x2-disksort-nvcache-throttle32 | 442060.0 | 7366.2 | 0.0 | 7366.0 | 7366.1 | 1229.0 | 0.5 |
|---|
| 2x2-disksort-nvcache | 441406.5 | 7355.2 | 0.0 | 7355.0 | 7355.1 | 1240.5 | 0.5 |
|---|
networkfs
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 36661.0 | 611.0 | 90.0 | 90.0 | 3.6 | 619.0 | 0.1 |
|---|
| 2x2-default | 36661.0 | 610.9 | 90.0 | 90.0 | 3.8 | 704.0 | 0.1 |
|---|
| 2x2-disksort-nvcache-throttle32 | 36661.0 | 611.0 | 90.0 | 90.0 | 3.6 | 702.5 | 0.1 |
|---|
| 2x2-disksort-nvcache | 36661.0 | 611.0 | 90.0 | 90.0 | 3.6 | 655.5 | 0.1 |
|---|
oltp
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 10071136.5 | 167696.3 | 83420.0 | 83416.0 | 332.3 | 47.5 | 0.0 |
|---|
| 2x2-default | 10290484.5 | 171336.3 | 85232.5 | 85225.5 | 339.4 | 49.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 10128198.0 | 168647.3 | 83896.5 | 83886.0 | 334.2 | 51.0 | 0.0 |
|---|
| 2x2-disksort-nvcache | 10149192.0 | 168993.3 | 84066.0 | 84061.0 | 334.8 | 50.5 | 0.0 |
|---|
openfiles
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 39035788.3 | 649939.8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-default | 38995139.0 | 649253.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 39243371.3 | 653390.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache | 39469074.0 | 657147.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
randomfileaccess
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 152838.0 | 2546.7 | 566.0 | 283.0 | 7341.6 | 5978.5 | 5.9 |
|---|
| 2x2-default | 151323.5 | 2521.4 | 560.5 | 280.0 | 7207.2 | 6042.0 | 5.9 |
|---|
| 2x2-disksort-nvcache-throttle32 | 152677.5 | 2543.9 | 565.0 | 282.5 | 7231.0 | 5986.0 | 5.8 |
|---|
| 2x2-disksort-nvcache | 154093.5 | 2567.7 | 570.5 | 285.5 | 7292.2 | 5929.5 | 5.8 |
|---|
randomread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 11992419.5 | 199842.6 | 199842.5 | 0.0 | 1561.3 | 8.0 | 0.0 |
|---|
| 2x2-default | 11833213.5 | 197192.5 | 197192.5 | 0.0 | 1540.6 | 8.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 11813746.0 | 196871.0 | 196871.0 | 0.0 | 1538.1 | 8.5 | 0.0 |
|---|
| 2x2-disksort-nvcache | 12032945.0 | 200510.1 | 200510.0 | 0.0 | 1566.4 | 8.0 | 0.0 |
|---|
randomrw
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 13176790.5 | 219542.8 | 151258.0 | 68285.0 | 1715.1 | 11.0 | 0.0 |
|---|
| 2x2-default | 12989551.0 | 216399.1 | 148652.5 | 67747.0 | 1690.6 | 12.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 12920202.5 | 215285.2 | 151365.5 | 63920.0 | 1681.9 | 11.5 | 0.0 |
|---|
| 2x2-disksort-nvcache | 13123285.5 | 218628.5 | 149464.0 | 69164.5 | 1708.1 | 11.5 | 0.0 |
|---|
randomwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 5684800.5 | 94736.2 | 0.0 | 94736.5 | 740.1 | 19.5 | 0.0 |
|---|
| 2x2-default | 5540208.0 | 92327.4 | 0.0 | 92327.0 | 721.3 | 20.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 5658993.5 | 94301.5 | 0.0 | 94301.5 | 736.7 | 20.0 | 0.0 |
|---|
| 2x2-disksort-nvcache | 5691039.0 | 94834.4 | 0.0 | 94834.5 | 740.9 | 20.0 | 0.0 |
|---|
ratelimcopyfiles
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4254.8 | 0.1 |
|---|
| 2x2-default | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4380.5 | 0.1 |
|---|
| 2x2-disksort-nvcache-throttle32 | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4291.8 | 0.1 |
|---|
| 2x2-disksort-nvcache | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4292.0 | 0.1 |
|---|
removedirs
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 10000.0 | 9998.6 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-default | 10000.0 | 9998.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache-throttle32 | 10000.0 | 9998.7 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
| 2x2-disksort-nvcache | 10000.0 | 9998.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|---|
singlestreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 256063.0 | 4267.2 | 4267.0 | 0.0 | 4266.4 | 249.5 | 0.2 |
|---|
| 2x2-default | 258475.5 | 4307.6 | 4307.5 | 0.0 | 4306.6 | 251.5 | 0.2 |
|---|
| 2x2-disksort-nvcache-throttle32 | 264057.0 | 4400.5 | 4400.5 | 0.0 | 4399.6 | 244.5 | 0.2 |
|---|
| 2x2-disksort-nvcache | 256141.0 | 4268.6 | 4269.0 | 0.0 | 4267.7 | 252.0 | 0.2 |
|---|
singlestreamreaddirect
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 255320.5 | 4254.9 | 4254.5 | 0.0 | 4254.1 | 250.0 | 0.2 |
|---|
| 2x2-default | 260241.5 | 4337.0 | 4337.0 | 0.0 | 4336.1 | 245.5 | 0.2 |
|---|
| 2x2-disksort-nvcache-throttle32 | 259342.5 | 4322.1 | 4322.0 | 0.0 | 4321.2 | 247.5 | 0.2 |
|---|
| 2x2-disksort-nvcache | 261852.5 | 4363.8 | 4364.0 | 0.0 | 4363.0 | 242.5 | 0.2 |
|---|
singlestreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 168197.5 | 2802.8 | 0.0 | 2803.0 | 2802.8 | 929.5 | 0.4 |
|---|
| 2x2-default | 167674.5 | 2794.1 | 0.0 | 2794.0 | 2794.1 | 938.0 | 0.4 |
|---|
| 2x2-disksort-nvcache-throttle32 | 167621.5 | 2793.1 | 0.0 | 2793.0 | 2793.1 | 932.0 | 0.4 |
|---|
| 2x2-disksort-nvcache | 168907.5 | 2814.7 | 0.0 | 2814.5 | 2814.6 | 936.0 | 0.4 |
|---|
singlestreamwritedirect
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 168527.0 | 2808.3 | 0.0 | 2808.5 | 2808.3 | 930.0 | 0.4 |
|---|
| 2x2-default | 168296.5 | 2804.3 | 0.0 | 2804.0 | 2804.3 | 938.0 | 0.4 |
|---|
| 2x2-disksort-nvcache-throttle32 | 170271.0 | 2837.3 | 0.0 | 2837.5 | 2837.3 | 914.0 | 0.3 |
|---|
| 2x2-disksort-nvcache | 169197.0 | 2819.4 | 0.0 | 2819.0 | 2819.3 | 929.5 | 0.4 |
|---|
tpcso
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
|
|---|
varmail
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 9153719.5 | 152405.2 | 23446.5 | 23447.5 | 539.2 | 125.5 | 0.3 |
|---|
| 2x2-default | 10514806.0 | 175066.2 | 26933.0 | 26933.0 | 631.3 | 140.5 | 0.3 |
|---|
| 2x2-disksort-nvcache-throttle32 | 12480558.0 | 207795.7 | 31968.5 | 31968.5 | 750.5 | 145.0 | 0.2 |
|---|
| 2x2-disksort-nvcache | 12499507.0 | 208111.5 | 32017.0 | 32017.0 | 751.4 | 143.0 | 0.2 |
|---|
videoserver
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 23191.0 | 386.4 | 386.0 | 0.0 | 752.3 | 2204.5 | 1.8 |
|---|
| 2x2-default | 23176.0 | 386.3 | 386.0 | 0.0 | 752.3 | 1801.0 | 1.5 |
|---|
| 2x2-disksort-nvcache-throttle32 | 23196.0 | 386.5 | 386.0 | 0.0 | 752.3 | 2119.0 | 1.5 |
|---|
| 2x2-disksort-nvcache | 23176.0 | 386.2 | 386.0 | 0.0 | 752.3 | 1832.5 | 1.7 |
|---|
webproxy
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 24180633.5 | 400539.0 | 105408.5 | 21082.5 | 993.1 | 139.5 | 0.3 |
|---|
| 2x2-default | 24015230.5 | 398337.6 | 104829.0 | 20966.5 | 987.9 | 141.0 | 0.3 |
|---|
| 2x2-disksort-nvcache-throttle32 | 23945860.0 | 396378.3 | 104313.5 | 20863.5 | 983.4 | 141.5 | 0.3 |
|---|
| 2x2-disksort-nvcache | 24256011.0 | 401975.3 | 105786.5 | 21158.0 | 996.8 | 138.0 | 0.3 |
|---|
webserver
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| 1x2-disksort-nvcache | 35236048.5 | 586862.6 | 189310.0 | 18931.5 | 3111.6 | 71.0 | 0.1 |
|---|
| 2x2-default | 35478607.5 | 590888.0 | 190608.5 | 19062.0 | 3132.8 | 70.5 | 0.1 |
|---|
| 2x2-disksort-nvcache-throttle32 | 34996918.5 | 582927.9 | 188041.0 | 18805.0 | 3090.6 | 71.5 | 0.1 |
|---|
| 2x2-disksort-nvcache | 35034111.0 | 583531.6 | 188236.0 | 18824.0 | 3094.1 | 71.0 | 0.1 |
|---|