HDD benchmark results
Results of simple benchmarks using mkfile(1M), dd(1) and filebench. The RW test is used to get a feeling for max. performance wrt. to sequential reads and writes. It writes out 4x32GiB (so ARC should be poissioned enough) using mkfile(1M) and after that it reads in those files using dd(1) in chunks of 128KiB. All other tests are made using the workflow definitions coming with filebench (fb) with run 60. The only parameter changed in the fb tests is the directory to point to the appropriate directory in the tested ZFS pool. We avoided explicitly changing other parameters to have "something comparable" out of the box. So depending on the workload/test, the results may not necessarily represent the max. performance the system + storage is able to provide! So to properly interprete the results, it is recommended to have a look at the workflow definition in question and related semantics.
For more test details, have a look at *.txt files and related sub directories.
NOTE: On Linux/Ubuntu we had to remove useism (use local instead of shared memory) from the oltp.f workload to get it work. Otherwise threads dead lock with FUTEX_WAIT when trying to attach the shared memory segment (ipc.c:ipc_ismmalloc).
RW
| config | write [MiB/s] | read [MiB/s] |
| fb 1 |
| 01_10x900G-HGST-mirror | 772.7 | 3036.9 |
| 02_10x900G-HGST-mirror-compressed | 1066.7 | 5189.0 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 1065.0 | 5354.3 |
| 04_10x900G-HGST-mirror-1SSD | 773.7 | 3003.5 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 1068.8 | 5234.5 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 1059.3 | 5080.4 |
| 07_10x900G-HGST-mirror-2SSD | 783.4 | 2974.9 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 1062.6 | 5285.2 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 940.0 | 5310.9 |
| 10_8xSSD-mirror | 917.5 | 2957.4 |
| 11_8xSSD-mirror-compressed | 1092.2 | 5205.5 |
| 12_8xSSD-mirror-atime=off-compressed | 1075.3 | 5302.3 |
| 13_8xSSD-mirror-atime=off | 997.5 | 2985.7 |
fileserver
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 1463047.5 | 24175.1 | 2198.0 | 4396.0 | 585.3 | 5014.5 | 7.0 |
| 02_10x900G-HGST-mirror-compressed | 3714805.5 | 61699.4 | 5609.0 | 11218.5 | 1494.8 | 1893.5 | 2.5 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 2509086.0 | 41503.6 | 3773.0 | 7546.0 | 1005.8 | 2731.5 | 3.6 |
| 04_10x900G-HGST-mirror-1SSD | 3312540.0 | 54985.8 | 4998.5 | 9997.5 | 1332.4 | 2268.5 | 3.0 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 4192791.0 | 69662.0 | 6332.5 | 12666.0 | 1686.9 | 1590.0 | 1.9 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 4465711.0 | 74117.6 | 6738.0 | 13476.5 | 1796.8 | 1481.0 | 1.8 |
| 07_10x900G-HGST-mirror-2SSD | 3435726.0 | 57077.6 | 5189.0 | 10378.0 | 1383.1 | 1981.5 | 2.7 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 4807087.5 | 79862.6 | 7260.0 | 14520.5 | 1934.9 | 1360.5 | 1.7 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 3327584.0 | 55269.8 | 5024.5 | 10049.0 | 1339.7 | 2027.5 | 2.6 |
| 10_8xSSD-mirror | 4158249.0 | 69041.7 | 6276.5 | 12553.0 | 1672.8 | 1605.0 | 2.0 |
| 11_8xSSD-mirror-compressed | 4442315.0 | 73830.7 | 6711.5 | 13424.0 | 1790.2 | 1488.5 | 1.9 |
| 12_8xSSD-mirror-atime=off-compressed | 4147501.5 | 68957.8 | 6268.5 | 12538.0 | 1670.8 | 1601.0 | 2.0 |
| 13_8xSSD-mirror-atime=off | 4151422.5 | 69008.9 | 6273.5 | 12547.5 | 1672.7 | 1601.5 | 2.1 |
multistreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 331332.5 | 5519.6 | 5519.5 | 0.0 | 5514.1 | 723.5 | 0.7 |
| 02_10x900G-HGST-mirror-compressed | 363859.0 | 6061.4 | 6061.5 | 0.0 | 6055.5 | 658.0 | 0.7 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 363994.0 | 6062.7 | 6063.0 | 0.0 | 6056.7 | 658.0 | 0.7 |
| 04_10x900G-HGST-mirror-1SSD | 330510.5 | 5506.6 | 5506.5 | 0.0 | 5501.2 | 725.0 | 0.7 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 363678.0 | 6059.6 | 6059.5 | 0.0 | 6053.7 | 658.0 | 0.7 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 363808.0 | 6057.2 | 6057.5 | 0.0 | 6051.3 | 658.0 | 0.7 |
| 07_10x900G-HGST-mirror-2SSD | 331050.0 | 5516.0 | 5516.0 | 0.0 | 5510.6 | 724.0 | 0.7 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 363522.0 | 6055.0 | 6055.0 | 0.0 | 6049.0 | 659.0 | 0.7 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 364032.5 | 6064.7 | 6064.5 | 0.0 | 6058.8 | 657.5 | 0.7 |
| 10_8xSSD-mirror | 330582.0 | 5509.3 | 5509.0 | 0.0 | 5503.8 | 724.5 | 0.7 |
| 11_8xSSD-mirror-compressed | 363378.0 | 6051.9 | 6052.0 | 0.0 | 6046.0 | 658.5 | 0.7 |
| 12_8xSSD-mirror-atime=off-compressed | 363696.5 | 6058.5 | 6058.5 | 0.0 | 6052.5 | 658.5 | 0.7 |
| 13_8xSSD-mirror-atime=off | 330324.0 | 5505.0 | 5505.0 | 0.0 | 5499.6 | 725.0 | 0.7 |
multistreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 77354.5 | 1288.2 | 0.0 | 1288.0 | 1288.1 | 2757.5 | 3.1 |
| 02_10x900G-HGST-mirror-compressed | 243968.5 | 4063.9 | 0.0 | 4064.0 | 4063.8 | 2839.5 | 1.0 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 244429.5 | 4071.9 | 0.0 | 4072.0 | 4071.9 | 2828.5 | 1.0 |
| 04_10x900G-HGST-mirror-1SSD | 61234.0 | 1019.7 | 0.0 | 1019.5 | 1019.6 | 2191.0 | 3.9 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 241585.0 | 4024.5 | 0.0 | 4024.5 | 4024.5 | 2797.5 | 1.0 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 240850.0 | 4012.1 | 0.0 | 4012.0 | 4012.0 | 2854.0 | 1.0 |
| 07_10x900G-HGST-mirror-2SSD | 61142.5 | 1018.2 | 0.0 | 1018.5 | 1018.1 | 2922.0 | 4.0 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 244312.0 | 4069.8 | 0.0 | 4069.5 | 4069.8 | 2790.0 | 1.0 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 228330.0 | 3803.5 | 0.0 | 3804.0 | 3803.5 | 2933.0 | 1.0 |
| 10_8xSSD-mirror | 134162.5 | 2234.2 | 0.0 | 2234.0 | 2234.1 | 2166.5 | 1.8 |
| 11_8xSSD-mirror-compressed | 244713.5 | 4076.3 | 0.0 | 4076.5 | 4076.3 | 2856.0 | 1.0 |
| 12_8xSSD-mirror-atime=off-compressed | 242501.5 | 4039.9 | 0.0 | 4040.0 | 4039.8 | 2905.5 | 1.0 |
| 13_8xSSD-mirror-atime=off | 132781.0 | 2211.3 | 0.0 | 2211.5 | 2211.3 | 2149.5 | 1.8 |
oltp
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 3197914.5 | 53263.8 | 26512.5 | 26478.5 | 105.5 | 162.0 | 0.0 |
| 02_10x900G-HGST-mirror-compressed | 3239166.0 | 53957.0 | 26856.0 | 26825.0 | 106.9 | 172.5 | 0.0 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 3281461.0 | 54659.3 | 27205.5 | 27174.5 | 108.3 | 164.5 | 0.0 |
| 04_10x900G-HGST-mirror-1SSD | 3262066.0 | 54339.4 | 27039.0 | 27021.5 | 107.6 | 188.0 | 0.0 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 3267195.5 | 54417.8 | 27082.0 | 27056.5 | 107.8 | 188.5 | 0.0 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 3352258.5 | 55852.1 | 27794.0 | 27772.0 | 110.6 | 173.0 | 0.0 |
| 07_10x900G-HGST-mirror-2SSD | 3285595.5 | 54732.7 | 27235.5 | 27216.5 | 108.4 | 184.5 | 0.0 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 3262790.0 | 54354.5 | 27050.0 | 27025.5 | 107.6 | 188.5 | 0.0 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 3255185.5 | 54230.0 | 26983.5 | 26968.5 | 107.4 | 192.5 | 0.0 |
| 10_8xSSD-mirror | 3264053.5 | 54381.7 | 27061.5 | 27041.5 | 107.7 | 188.0 | 0.0 |
| 11_8xSSD-mirror-compressed | 3375637.5 | 56241.7 | 27989.5 | 27964.5 | 111.4 | 168.5 | 0.0 |
| 12_8xSSD-mirror-atime=off-compressed | 3226407.5 | 53750.8 | 26750.0 | 26725.5 | 106.5 | 196.5 | 0.0 |
| 13_8xSSD-mirror-atime=off | 3372874.0 | 56192.3 | 27962.0 | 27943.0 | 111.3 | 169.5 | 0.0 |
singlestreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 84158.5 | 1401.2 | 1401.5 | 0.0 | 1401.0 | 721.0 | 0.7 |
| 02_10x900G-HGST-mirror-compressed | 91849.0 | 1529.3 | 1529.5 | 0.0 | 1529.0 | 661.0 | 0.6 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 91764.5 | 1528.4 | 1528.5 | 0.0 | 1528.0 | 661.0 | 0.6 |
| 04_10x900G-HGST-mirror-1SSD | 84094.5 | 1400.1 | 1400.0 | 0.0 | 1399.9 | 722.5 | 0.7 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 91799.0 | 1528.9 | 1529.0 | 0.0 | 1528.6 | 662.0 | 0.6 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 91918.5 | 1531.6 | 1532.0 | 0.0 | 1531.3 | 660.0 | 0.6 |
| 07_10x900G-HGST-mirror-2SSD | 84108.0 | 1400.8 | 1401.0 | 0.0 | 1400.7 | 721.5 | 0.7 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 91601.5 | 1525.1 | 1525.0 | 0.0 | 1524.8 | 663.0 | 0.6 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 91897.0 | 1530.5 | 1530.0 | 0.0 | 1530.2 | 675.0 | 0.6 |
| 10_8xSSD-mirror | 83809.5 | 1395.4 | 1395.0 | 0.0 | 1395.1 | 724.5 | 0.7 |
| 11_8xSSD-mirror-compressed | 91503.5 | 1524.0 | 1524.0 | 0.0 | 1523.7 | 663.0 | 0.7 |
| 12_8xSSD-mirror-atime=off-compressed | 91639.0 | 1526.8 | 1527.0 | 0.0 | 1526.6 | 662.0 | 0.6 |
| 13_8xSSD-mirror-atime=off | 83505.0 | 1390.3 | 1390.5 | 0.0 | 1390.0 | 727.0 | 0.7 |
singlestreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 59107.0 | 984.5 | 0.0 | 984.5 | 984.5 | 2326.0 | 1.0 |
| 02_10x900G-HGST-mirror-compressed | 65218.0 | 1085.9 | 0.0 | 1085.5 | 1085.9 | 2837.5 | 0.9 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 65335.0 | 1088.3 | 0.0 | 1088.5 | 1088.3 | 2827.5 | 0.9 |
| 04_10x900G-HGST-mirror-1SSD | 57807.0 | 962.4 | 0.0 | 962.5 | 962.4 | 2304.0 | 1.0 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 64631.5 | 1076.5 | 0.0 | 1076.5 | 1076.5 | 2857.0 | 0.9 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 64737.0 | 1078.1 | 0.0 | 1078.0 | 1078.1 | 2831.5 | 0.9 |
| 07_10x900G-HGST-mirror-2SSD | 57571.0 | 958.7 | 0.0 | 958.5 | 958.7 | 2446.5 | 1.0 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 64820.5 | 1079.6 | 0.0 | 1079.5 | 1079.6 | 2836.5 | 0.9 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 64518.5 | 1074.8 | 0.0 | 1075.0 | 1074.8 | 2901.0 | 0.9 |
| 10_8xSSD-mirror | 64164.5 | 1068.7 | 0.0 | 1069.0 | 1068.7 | 2965.5 | 0.9 |
| 11_8xSSD-mirror-compressed | 65228.0 | 1086.4 | 0.0 | 1086.5 | 1086.3 | 2820.0 | 0.9 |
| 12_8xSSD-mirror-atime=off-compressed | 65209.0 | 1086.0 | 0.0 | 1086.0 | 1086.0 | 2835.0 | 0.9 |
| 13_8xSSD-mirror-atime=off | 63954.0 | 1065.4 | 0.0 | 1065.5 | 1065.4 | 2372.0 | 0.9 |
varmail
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 589598.0 | 9820.4 | 1511.0 | 1511.0 | 30.6 | 255.5 | 5.2 |
| 02_10x900G-HGST-mirror-compressed | 584404.0 | 9733.7 | 1497.5 | 1498.0 | 30.6 | 273.0 | 5.3 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 576128.5 | 9593.8 | 1476.0 | 1476.0 | 29.8 | 242.0 | 5.3 |
| 04_10x900G-HGST-mirror-1SSD | 2443761.0 | 40701.8 | 6262.0 | 6262.0 | 146.4 | 891.0 | 1.5 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 4366713.0 | 72737.3 | 11190.0 | 11190.5 | 260.8 | 267.0 | 0.7 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 4340613.5 | 72295.0 | 11122.0 | 11122.5 | 259.0 | 279.0 | 0.7 |
| 07_10x900G-HGST-mirror-2SSD | 3395930.5 | 56567.2 | 8702.5 | 8703.0 | 203.4 | 439.0 | 0.9 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 4375884.5 | 72879.0 | 11212.5 | 11212.5 | 260.4 | 265.0 | 0.7 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 4236526.0 | 70549.4 | 10853.5 | 10854.0 | 252.2 | 277.5 | 0.7 |
| 10_8xSSD-mirror | 3773822.0 | 62863.4 | 9671.5 | 9671.5 | 226.2 | 332.5 | 0.7 |
| 11_8xSSD-mirror-compressed | 3472921.5 | 57839.1 | 8898.0 | 8898.0 | 208.5 | 398.0 | 0.9 |
| 12_8xSSD-mirror-atime=off-compressed | 3986246.0 | 66394.2 | 10214.5 | 10214.5 | 239.3 | 298.5 | 0.7 |
| 13_8xSSD-mirror-atime=off | 4182150.0 | 69658.4 | 10716.5 | 10716.5 | 251.5 | 266.0 | 0.7 |
webproxy
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
| fb 1 |
| 01_10x900G-HGST-mirror | 3332464.5 | 55506.9 | 14606.5 | 2921.5 | 142.0 | 1075.5 | 5.7 |
| 02_10x900G-HGST-mirror-compressed | 5210873.0 | 86799.8 | 22842.0 | 4568.5 | 219.1 | 701.0 | 3.5 |
| 03_10x900G-HGST-mirror-atime=off-compressed | 5044350.5 | 83904.8 | 22080.0 | 4416.0 | 212.3 | 753.0 | 3.7 |
| 04_10x900G-HGST-mirror-1SSD | 4036241.0 | 67206.3 | 17685.5 | 3537.0 | 170.5 | 942.5 | 4.7 |
| 05_10x900G-HGST-mirror-1SSD-compressed | 4860656.5 | 80961.5 | 21305.5 | 4261.0 | 204.8 | 792.0 | 3.8 |
| 06_10x900G-HGST-mirror-1SSD-atime=off-compressed | 5127065.0 | 85353.6 | 22461.0 | 4492.0 | 215.6 | 668.5 | 3.6 |
| 07_10x900G-HGST-mirror-2SSD | 4480355.5 | 74565.7 | 19622.0 | 3924.5 | 189.7 | 894.0 | 4.2 |
| 08_10x900G-HGST-mirror-2SSD-compressed | 5149636.5 | 85766.4 | 22569.5 | 4514.0 | 216.7 | 733.5 | 3.6 |
| 09_10x900G-HGST-mirror-2SSD-atime=off-compressed | 4792396.0 | 79789.2 | 20996.5 | 4199.0 | 201.7 | 748.5 | 3.9 |
| 10_8xSSD-mirror | 4668269.0 | 77732.8 | 20455.5 | 4091.0 | 197.1 | 852.0 | 4.0 |
| 11_8xSSD-mirror-compressed | 4967519.5 | 82745.6 | 21775.0 | 4355.5 | 209.5 | 780.5 | 3.8 |
| 12_8xSSD-mirror-atime=off-compressed | 5120106.5 | 85247.3 | 22433.0 | 4486.5 | 215.4 | 760.0 | 3.6 |
| 13_8xSSD-mirror-atime=off | 4656061.5 | 77532.0 | 20402.5 | 4081.0 | 196.5 | 838.0 | 4.0 |