HDD benchmark results
Results of simple benchmarks using mkfile(1M), dd(1) and filebench. The RW test is used to get a feeling for max. performance wrt. to sequential reads and writes. It writes out 4x32GiB (so ARC should be poissioned enough) using mkfile(1M) and after that it reads in those files using dd(1) in chunks of 128KiB. All other tests are made using the workflow definitions coming with filebench (fb) with run 60. The only parameter changed in the fb tests is the directory to point to the appropriate directory in the tested ZFS pool. We avoided explicitly changing other parameters to have "something comparable" out of the box. So depending on the workload/test, the results may not necessarily represent the max. performance the system + storage is able to provide! So to properly interprete the results, it is recommended to have a look at the workflow definition in question and related semantics.
For more test details, have a look at *.txt files and related sub directories.
NOTE: On Linux/Ubuntu we had to remove useism (use local instead of shared memory) from the oltp.f workload to get it work. Otherwise threads dead lock with FUTEX_WAIT when trying to attach the shared memory segment (ipc.c:ipc_ismmalloc).
RW
config | write [MiB/s] | read [MiB/s] |
fb 1 |
1x2-disksort-nvcache | 2482.5 | 10780.0 |
2x2-default | 2500.5 | 11164.6 |
2x2-disksort-nvcache-throttle32 | 2496.6 | 10469.0 |
2x2-disksort-nvcache | 2462.2 | 10886.5 |
bringover
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 6002.0 | 6001.4 | 1001.0 | 1000.0 | 30.0 | 113.5 | 0.0 |
2x2-default | 6002.0 | 6001.4 | 1001.0 | 1000.0 | 30.0 | 68.8 | 0.0 |
2x2-disksort-nvcache-throttle32 | 6002.0 | 6001.3 | 1001.0 | 1000.0 | 30.0 | 57.3 | 0.0 |
2x2-disksort-nvcache | 6002.0 | 5999.8 | 1000.8 | 999.8 | 30.0 | 152.5 | 0.0 |
copyfiles
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 6002.0 | 5999.9 | 1000.8 | 999.8 | 30.0 | 99.3 | 0.0 |
2x2-default | 6002.0 | 6001.3 | 1001.0 | 1000.0 | 30.0 | 59.8 | 0.0 |
2x2-disksort-nvcache-throttle32 | 6002.0 | 6001.3 | 1001.0 | 1000.0 | 30.0 | 96.3 | 0.0 |
2x2-disksort-nvcache | 6002.0 | 6001.4 | 1001.0 | 1000.0 | 30.0 | 67.0 | 0.0 |
createfiles
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 149974.8 | 149817.5 | 0.0 | 49936.8 | 780.2 | 250.5 | 0.3 |
2x2-default | 149978.0 | 149820.7 | 0.0 | 49938.5 | 780.2 | 253.0 | 0.3 |
2x2-disksort-nvcache-throttle32 | 149977.5 | 149818.4 | 0.0 | 49937.0 | 780.2 | 254.0 | 0.3 |
2x2-disksort-nvcache | 149979.5 | 149822.1 | 0.0 | 49938.5 | 780.2 | 259.0 | 0.3 |
deletefiles
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 50000.0 | 43721.2 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-default | 50000.0 | 12498.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 50000.0 | 37484.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache | 50000.0 | 49956.4 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
fileserver
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 13896035.0 | 230532.2 | 20957.5 | 41915.5 | 5595.7 | 338.5 | 0.4 |
2x2-default | 13917794.0 | 231243.5 | 21022.0 | 42044.5 | 5614.4 | 338.0 | 0.4 |
2x2-disksort-nvcache-throttle32 | 13858491.0 | 230286.8 | 20935.0 | 41871.0 | 5589.0 | 338.5 | 0.4 |
2x2-disksort-nvcache | 13882754.5 | 230867.8 | 20988.0 | 41977.0 | 5605.9 | 337.5 | 0.4 |
listdirs
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 23621515.0 | 393391.6 | 0.0 | 0.0 | 81.7 | 0.0 | 0.0 |
2x2-default | 23492385.0 | 391153.7 | 0.0 | 0.0 | 81.3 | 0.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 23431101.5 | 390187.2 | 0.0 | 0.0 | 81.0 | 0.0 | 0.0 |
2x2-disksort-nvcache | 23371997.0 | 389180.3 | 0.0 | 0.0 | 80.8 | 0.0 | 0.0 |
makedirs
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 10000.0 | 9999.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-default | 10000.0 | 9998.8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 10000.0 | 9998.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache | 10000.0 | 9999.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
mongo
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 96.8 | 0.0 |
2x2-default | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 58.3 | 0.0 |
2x2-disksort-nvcache-throttle32 | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 89.5 | 0.0 |
2x2-disksort-nvcache | 7006.0 | 7005.2 | 1001.0 | 1001.0 | 23.0 | 57.8 | 0.0 |
multistreamread
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 819225.5 | 13651.9 | 13652.0 | 0.0 | 13638.5 | 296.0 | 0.3 |
2x2-default | 818920.5 | 13647.2 | 13647.5 | 0.0 | 13633.9 | 296.5 | 0.3 |
2x2-disksort-nvcache-throttle32 | 821146.0 | 13683.1 | 13683.0 | 0.0 | 13669.6 | 296.0 | 0.3 |
2x2-disksort-nvcache | 819422.0 | 13655.5 | 13655.0 | 0.0 | 13642.1 | 296.5 | 0.3 |
multistreamreaddirect
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 819253.5 | 13652.7 | 13653.0 | 0.0 | 13639.3 | 296.5 | 0.3 |
2x2-default | 825638.5 | 13758.4 | 13758.0 | 0.0 | 13744.9 | 294.0 | 0.3 |
2x2-disksort-nvcache-throttle32 | 820681.0 | 13675.8 | 13676.0 | 0.0 | 13662.5 | 296.0 | 0.3 |
2x2-disksort-nvcache | 820853.5 | 13678.4 | 13678.5 | 0.0 | 13665.0 | 296.0 | 0.3 |
multistreamwrite
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 433007.5 | 7214.6 | 0.0 | 7214.5 | 7214.5 | 1261.5 | 0.5 |
2x2-default | 430987.5 | 7181.1 | 0.0 | 7181.5 | 7181.1 | 1281.0 | 0.6 |
2x2-disksort-nvcache-throttle32 | 425212.0 | 7085.7 | 0.0 | 7086.0 | 7085.6 | 1291.0 | 0.6 |
2x2-disksort-nvcache | 435494.0 | 7257.0 | 0.0 | 7257.0 | 7256.9 | 1256.0 | 0.5 |
multistreamwritedirect
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 441981.5 | 7364.8 | 0.0 | 7365.0 | 7364.7 | 1233.0 | 0.5 |
2x2-default | 440312.5 | 7337.3 | 0.0 | 7337.5 | 7337.2 | 1232.5 | 0.5 |
2x2-disksort-nvcache-throttle32 | 442060.0 | 7366.2 | 0.0 | 7366.0 | 7366.1 | 1229.0 | 0.5 |
2x2-disksort-nvcache | 441406.5 | 7355.2 | 0.0 | 7355.0 | 7355.1 | 1240.5 | 0.5 |
networkfs
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 36661.0 | 611.0 | 90.0 | 90.0 | 3.6 | 619.0 | 0.1 |
2x2-default | 36661.0 | 610.9 | 90.0 | 90.0 | 3.8 | 704.0 | 0.1 |
2x2-disksort-nvcache-throttle32 | 36661.0 | 611.0 | 90.0 | 90.0 | 3.6 | 702.5 | 0.1 |
2x2-disksort-nvcache | 36661.0 | 611.0 | 90.0 | 90.0 | 3.6 | 655.5 | 0.1 |
oltp
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 10071136.5 | 167696.3 | 83420.0 | 83416.0 | 332.3 | 47.5 | 0.0 |
2x2-default | 10290484.5 | 171336.3 | 85232.5 | 85225.5 | 339.4 | 49.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 10128198.0 | 168647.3 | 83896.5 | 83886.0 | 334.2 | 51.0 | 0.0 |
2x2-disksort-nvcache | 10149192.0 | 168993.3 | 84066.0 | 84061.0 | 334.8 | 50.5 | 0.0 |
openfiles
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 39035788.3 | 649939.8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-default | 38995139.0 | 649253.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 39243371.3 | 653390.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache | 39469074.0 | 657147.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
randomfileaccess
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 152838.0 | 2546.7 | 566.0 | 283.0 | 7341.6 | 5978.5 | 5.9 |
2x2-default | 151323.5 | 2521.4 | 560.5 | 280.0 | 7207.2 | 6042.0 | 5.9 |
2x2-disksort-nvcache-throttle32 | 152677.5 | 2543.9 | 565.0 | 282.5 | 7231.0 | 5986.0 | 5.8 |
2x2-disksort-nvcache | 154093.5 | 2567.7 | 570.5 | 285.5 | 7292.2 | 5929.5 | 5.8 |
randomread
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 11992419.5 | 199842.6 | 199842.5 | 0.0 | 1561.3 | 8.0 | 0.0 |
2x2-default | 11833213.5 | 197192.5 | 197192.5 | 0.0 | 1540.6 | 8.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 11813746.0 | 196871.0 | 196871.0 | 0.0 | 1538.1 | 8.5 | 0.0 |
2x2-disksort-nvcache | 12032945.0 | 200510.1 | 200510.0 | 0.0 | 1566.4 | 8.0 | 0.0 |
randomrw
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 13176790.5 | 219542.8 | 151258.0 | 68285.0 | 1715.1 | 11.0 | 0.0 |
2x2-default | 12989551.0 | 216399.1 | 148652.5 | 67747.0 | 1690.6 | 12.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 12920202.5 | 215285.2 | 151365.5 | 63920.0 | 1681.9 | 11.5 | 0.0 |
2x2-disksort-nvcache | 13123285.5 | 218628.5 | 149464.0 | 69164.5 | 1708.1 | 11.5 | 0.0 |
randomwrite
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 5684800.5 | 94736.2 | 0.0 | 94736.5 | 740.1 | 19.5 | 0.0 |
2x2-default | 5540208.0 | 92327.4 | 0.0 | 92327.0 | 721.3 | 20.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 5658993.5 | 94301.5 | 0.0 | 94301.5 | 736.7 | 20.0 | 0.0 |
2x2-disksort-nvcache | 5691039.0 | 94834.4 | 0.0 | 94834.5 | 740.9 | 20.0 | 0.0 |
ratelimcopyfiles
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4254.8 | 0.1 |
2x2-default | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4380.5 | 0.1 |
2x2-disksort-nvcache-throttle32 | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4291.8 | 0.1 |
2x2-disksort-nvcache | 6002.0 | 60.0 | 10.0 | 10.0 | 0.3 | 4292.0 | 0.1 |
removedirs
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 10000.0 | 9998.6 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-default | 10000.0 | 9998.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache-throttle32 | 10000.0 | 9998.7 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2x2-disksort-nvcache | 10000.0 | 9998.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
singlestreamread
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 256063.0 | 4267.2 | 4267.0 | 0.0 | 4266.4 | 249.5 | 0.2 |
2x2-default | 258475.5 | 4307.6 | 4307.5 | 0.0 | 4306.6 | 251.5 | 0.2 |
2x2-disksort-nvcache-throttle32 | 264057.0 | 4400.5 | 4400.5 | 0.0 | 4399.6 | 244.5 | 0.2 |
2x2-disksort-nvcache | 256141.0 | 4268.6 | 4269.0 | 0.0 | 4267.7 | 252.0 | 0.2 |
singlestreamreaddirect
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 255320.5 | 4254.9 | 4254.5 | 0.0 | 4254.1 | 250.0 | 0.2 |
2x2-default | 260241.5 | 4337.0 | 4337.0 | 0.0 | 4336.1 | 245.5 | 0.2 |
2x2-disksort-nvcache-throttle32 | 259342.5 | 4322.1 | 4322.0 | 0.0 | 4321.2 | 247.5 | 0.2 |
2x2-disksort-nvcache | 261852.5 | 4363.8 | 4364.0 | 0.0 | 4363.0 | 242.5 | 0.2 |
singlestreamwrite
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 168197.5 | 2802.8 | 0.0 | 2803.0 | 2802.8 | 929.5 | 0.4 |
2x2-default | 167674.5 | 2794.1 | 0.0 | 2794.0 | 2794.1 | 938.0 | 0.4 |
2x2-disksort-nvcache-throttle32 | 167621.5 | 2793.1 | 0.0 | 2793.0 | 2793.1 | 932.0 | 0.4 |
2x2-disksort-nvcache | 168907.5 | 2814.7 | 0.0 | 2814.5 | 2814.6 | 936.0 | 0.4 |
singlestreamwritedirect
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 168527.0 | 2808.3 | 0.0 | 2808.5 | 2808.3 | 930.0 | 0.4 |
2x2-default | 168296.5 | 2804.3 | 0.0 | 2804.0 | 2804.3 | 938.0 | 0.4 |
2x2-disksort-nvcache-throttle32 | 170271.0 | 2837.3 | 0.0 | 2837.5 | 2837.3 | 914.0 | 0.3 |
2x2-disksort-nvcache | 169197.0 | 2819.4 | 0.0 | 2819.0 | 2819.3 | 929.5 | 0.4 |
tpcso
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
|
varmail
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 9153719.5 | 152405.2 | 23446.5 | 23447.5 | 539.2 | 125.5 | 0.3 |
2x2-default | 10514806.0 | 175066.2 | 26933.0 | 26933.0 | 631.3 | 140.5 | 0.3 |
2x2-disksort-nvcache-throttle32 | 12480558.0 | 207795.7 | 31968.5 | 31968.5 | 750.5 | 145.0 | 0.2 |
2x2-disksort-nvcache | 12499507.0 | 208111.5 | 32017.0 | 32017.0 | 751.4 | 143.0 | 0.2 |
videoserver
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 23191.0 | 386.4 | 386.0 | 0.0 | 752.3 | 2204.5 | 1.8 |
2x2-default | 23176.0 | 386.3 | 386.0 | 0.0 | 752.3 | 1801.0 | 1.5 |
2x2-disksort-nvcache-throttle32 | 23196.0 | 386.5 | 386.0 | 0.0 | 752.3 | 2119.0 | 1.5 |
2x2-disksort-nvcache | 23176.0 | 386.2 | 386.0 | 0.0 | 752.3 | 1832.5 | 1.7 |
webproxy
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 24180633.5 | 400539.0 | 105408.5 | 21082.5 | 993.1 | 139.5 | 0.3 |
2x2-default | 24015230.5 | 398337.6 | 104829.0 | 20966.5 | 987.9 | 141.0 | 0.3 |
2x2-disksort-nvcache-throttle32 | 23945860.0 | 396378.3 | 104313.5 | 20863.5 | 983.4 | 141.5 | 0.3 |
2x2-disksort-nvcache | 24256011.0 | 401975.3 | 105786.5 | 21158.0 | 996.8 | 138.0 | 0.3 |
webserver
config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
fb 1 |
1x2-disksort-nvcache | 35236048.5 | 586862.6 | 189310.0 | 18931.5 | 3111.6 | 71.0 | 0.1 |
2x2-default | 35478607.5 | 590888.0 | 190608.5 | 19062.0 | 3132.8 | 70.5 | 0.1 |
2x2-disksort-nvcache-throttle32 | 34996918.5 | 582927.9 | 188041.0 | 18805.0 | 3090.6 | 71.5 | 0.1 |
2x2-disksort-nvcache | 35034111.0 | 583531.6 | 188236.0 | 18824.0 | 3094.1 | 71.0 | 0.1 |