HDD benchmark results
Results of simple benchmarks using mkfile(1M), dd(1) and filebench. The RW test is used to get a feeling for max. performance wrt. to sequential reads and writes. It writes out 4x32GiB (so ARC should be poissioned enough) using mkfile(1M) and after that it reads in those files using dd(1) in chunks of 128KiB. All other tests are made using the workflow definitions coming with filebench (fb) with run 60. The only parameter changed in the fb tests is the directory to point to the appropriate directory in the tested ZFS pool. We avoided explicitly changing other parameters to have "something comparable" out of the box. So depending on the workload/test, the results may not necessarily represent the max. performance the system + storage is able to provide! So to properly interprete the results, it is recommended to have a look at the workflow definition in question and related semantics.
For more test details, have a look at *.txt files and related sub directories.
RW
| config | write [MiB/s] | read [MiB/s] |
|---|
| fb 1 |
|---|
| default | 2094.0 | 1649.5 |
|---|
| disksort-nvcache-throttle32 | 2076.5 | 1662.1 |
|---|
| disksort-nvcache | 2077.3 | 1642.5 |
|---|
| disksort-throttle32 | 2064.8 | 1643.7 |
|---|
| disksort | 2071.3 | 1641.3 |
|---|
| nvcache-throttle32 | 2065.4 | 1657.9 |
|---|
| nvcache | 2046.1 | 1631.5 |
|---|
| throttle32 | 2075.9 | 1644.6 |
|---|
| fb 2 |
|---|
| default | 1033.4 | 231.8 |
|---|
| disksort-nvcache-throttle32 | 1035.8 | 232.9 |
|---|
| disksort-nvcache | 1054.2 | 231.1 |
|---|
| disksort-throttle32 | 1040.5 | 234.4 |
|---|
| disksort | 1038.2 | 233.1 |
|---|
| nvcache-throttle32 | 1050.3 | 234.0 |
|---|
| nvcache | 1098.2 | 230.9 |
|---|
| throttle32 | 1046.7 | 233.3 |
|---|
fileserver
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 5108457.5 | 84700.9 | 7700.5 | 15400.0 | 2052.9 | 1361.0 | 1.9 |
|---|
| disksort-nvcache-throttle32 | 7827292.0 | 129760.5 | 11796.5 | 23593.0 | 3147.4 | 896.5 | 1.2 |
|---|
| disksort-nvcache | 6023043.5 | 99984.4 | 9089.5 | 18179.0 | 2423.6 | 1137.5 | 1.5 |
|---|
| disksort-throttle32 | 6071659.0 | 100898.5 | 9172.5 | 18345.5 | 2446.1 | 1151.5 | 1.6 |
|---|
| disksort | 5884578.0 | 97808.9 | 8891.5 | 17783.5 | 2372.3 | 1168.5 | 1.5 |
|---|
| nvcache-throttle32 | 5873522.0 | 97428.5 | 8857.0 | 17714.5 | 2360.9 | 1181.5 | 1.5 |
|---|
| nvcache | 6076387.5 | 100952.3 | 9177.5 | 18355.0 | 2446.9 | 1128.5 | 1.5 |
|---|
| throttle32 | 6469860.0 | 107460.8 | 9769.0 | 19538.5 | 2603.5 | 1052.0 | 1.3 |
|---|
| fb 2 |
|---|
| default | 4558014.0 | 75644.9 | 6877.0 | 13754.0 | 1834.6 | 1528.0 | 2.1 |
|---|
| disksort-nvcache-throttle32 | 4825638.0 | 80038.5 | 7276.5 | 14552.5 | 1941.2 | 1440.5 | 2.0 |
|---|
| disksort-nvcache | 4046449.5 | 67181.1 | 6107.5 | 12215.0 | 1629.1 | 1727.5 | 2.4 |
|---|
| disksort-throttle32 | 3933811.0 | 65171.5 | 5924.5 | 11849.5 | 1579.3 | 1782.5 | 2.4 |
|---|
| disksort | 4267438.5 | 70744.9 | 6431.5 | 12863.0 | 1714.8 | 1631.5 | 2.2 |
|---|
| nvcache-throttle32 | 3620539.5 | 59997.9 | 5454.0 | 10909.0 | 1454.3 | 1930.5 | 2.7 |
|---|
| nvcache | 4391189.0 | 72898.4 | 6627.0 | 13254.5 | 1766.0 | 1593.5 | 2.1 |
|---|
| throttle32 | 4345629.5 | 72127.9 | 6557.0 | 13114.5 | 1750.4 | 1602.0 | 2.2 |
|---|
multistreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 859694.5 | 14324.6 | 14324.5 | 0.0 | 14310.6 | 278.0 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 857757.0 | 14288.6 | 14288.5 | 0.0 | 14274.5 | 278.5 | 0.3 |
|---|
| disksort-nvcache | 860028.0 | 14320.1 | 14320.0 | 0.0 | 14306.1 | 277.5 | 0.3 |
|---|
| disksort-throttle32 | 860014.5 | 14323.5 | 14323.5 | 0.0 | 14309.5 | 277.5 | 0.3 |
|---|
| disksort | 858736.5 | 14297.9 | 14298.0 | 0.0 | 14283.9 | 278.0 | 0.3 |
|---|
| nvcache-throttle32 | 859448.0 | 14309.5 | 14309.0 | 0.0 | 14295.5 | 278.0 | 0.3 |
|---|
| nvcache | 856355.0 | 14259.1 | 14259.0 | 0.0 | 14245.3 | 279.0 | 0.3 |
|---|
| throttle32 | 859240.5 | 14320.0 | 14320.0 | 0.0 | 14306.1 | 278.0 | 0.3 |
|---|
| fb 2 |
|---|
| default | 862295.5 | 14360.8 | 14361.0 | 0.0 | 14346.8 | 276.5 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 855055.0 | 14237.4 | 14237.5 | 0.0 | 14223.5 | 279.0 | 0.3 |
|---|
| disksort-nvcache | 857303.0 | 14278.4 | 14278.5 | 0.0 | 14264.5 | 278.0 | 0.3 |
|---|
| disksort-throttle32 | 855582.0 | 14245.6 | 14245.5 | 0.0 | 14231.6 | 278.5 | 0.3 |
|---|
| disksort | 854799.0 | 14232.1 | 14232.5 | 0.0 | 14218.3 | 279.0 | 0.3 |
|---|
| nvcache-throttle32 | 857562.0 | 14278.1 | 14278.0 | 0.0 | 14264.2 | 278.0 | 0.3 |
|---|
| nvcache | 871856.5 | 14516.1 | 14516.5 | 0.0 | 14502.0 | 273.5 | 0.3 |
|---|
| throttle32 | 859161.5 | 14311.6 | 14311.5 | 0.0 | 14297.7 | 277.0 | 0.3 |
|---|
multistreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 533088.0 | 8884.1 | 0.0 | 8884.0 | 8884.0 | 1323.0 | 0.4 |
|---|
| disksort-nvcache-throttle32 | 536322.0 | 8934.7 | 0.0 | 8934.5 | 8934.5 | 1313.5 | 0.4 |
|---|
| disksort-nvcache | 542781.0 | 9039.7 | 0.0 | 9039.5 | 9039.6 | 1313.5 | 0.4 |
|---|
| disksort-throttle32 | 540661.0 | 9009.9 | 0.0 | 9010.0 | 9009.8 | 1314.0 | 0.4 |
|---|
| disksort | 539619.0 | 8987.6 | 0.0 | 8987.5 | 8987.6 | 1313.0 | 0.4 |
|---|
| nvcache-throttle32 | 531315.5 | 8848.0 | 0.0 | 8848.0 | 8847.9 | 1324.0 | 0.4 |
|---|
| nvcache | 534808.0 | 8910.0 | 0.0 | 8910.0 | 8910.0 | 1324.0 | 0.4 |
|---|
| throttle32 | 540732.5 | 9011.5 | 0.0 | 9011.5 | 9011.4 | 1310.0 | 0.4 |
|---|
| fb 2 |
|---|
| default | 191948.5 | 3198.9 | 0.0 | 3199.0 | 3198.9 | 6042.5 | 1.2 |
|---|
| disksort-nvcache-throttle32 | 187658.0 | 3126.2 | 0.0 | 3126.0 | 3126.1 | 6196.5 | 1.3 |
|---|
| disksort-nvcache | 191597.5 | 3190.9 | 0.0 | 3191.0 | 3190.8 | 6082.5 | 1.3 |
|---|
| disksort-throttle32 | 187193.0 | 3119.3 | 0.0 | 3119.5 | 3119.3 | 6191.0 | 1.3 |
|---|
| disksort | 185801.5 | 3095.1 | 0.0 | 3095.0 | 3095.0 | 6170.5 | 1.3 |
|---|
| nvcache-throttle32 | 187180.5 | 3117.1 | 0.0 | 3117.0 | 3116.9 | 6175.5 | 1.3 |
|---|
| nvcache | 187904.0 | 3129.9 | 0.0 | 3130.0 | 3129.9 | 6193.0 | 1.2 |
|---|
| throttle32 | 187210.0 | 3120.0 | 0.0 | 3120.0 | 3120.0 | 6107.0 | 1.3 |
|---|
oltp
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 8357280.0 | 139207.7 | 69258.0 | 69236.0 | 275.8 | 77.0 | 0.0 |
|---|
| disksort-nvcache-throttle32 | 8274197.0 | 137790.1 | 68548.5 | 68535.0 | 273.0 | 78.0 | 0.0 |
|---|
| disksort-nvcache | 8153071.5 | 135767.3 | 67545.0 | 67526.5 | 269.0 | 82.0 | 0.0 |
|---|
| disksort-throttle32 | 8162089.5 | 135939.6 | 67628.5 | 67614.0 | 269.4 | 82.0 | 0.0 |
|---|
| disksort | 8192715.0 | 136429.2 | 67874.0 | 67856.0 | 270.3 | 80.5 | 0.0 |
|---|
| nvcache-throttle32 | 8275547.5 | 137799.3 | 68551.0 | 68541.5 | 273.0 | 77.5 | 0.0 |
|---|
| nvcache | 8117105.5 | 135166.4 | 67245.0 | 67228.5 | 267.8 | 82.0 | 0.0 |
|---|
| throttle32 | 8205416.5 | 136713.0 | 68009.5 | 68002.5 | 270.9 | 79.0 | 0.0 |
|---|
| fb 2 |
|---|
| default | 7763357.5 | 129324.1 | 64338.0 | 64323.0 | 256.2 | 94.0 | 0.0 |
|---|
| disksort-nvcache-throttle32 | 8366725.5 | 139308.3 | 69306.0 | 69288.0 | 276.0 | 78.0 | 0.0 |
|---|
| disksort-nvcache | 8008972.5 | 133379.4 | 66357.5 | 66338.5 | 264.3 | 87.5 | 0.0 |
|---|
| disksort-throttle32 | 8099517.5 | 134882.3 | 67101.5 | 67088.5 | 267.2 | 84.5 | 0.0 |
|---|
| disksort | 7930383.5 | 132064.9 | 65702.0 | 65686.0 | 261.7 | 90.0 | 0.0 |
|---|
| nvcache-throttle32 | 8646889.0 | 143986.1 | 71630.5 | 71617.5 | 285.3 | 69.5 | 0.0 |
|---|
| nvcache | 7976741.5 | 132828.2 | 66084.5 | 66063.0 | 263.1 | 87.5 | 0.0 |
|---|
| throttle32 | 8072012.5 | 134487.3 | 66911.5 | 66886.0 | 266.5 | 85.0 | 0.0 |
|---|
singlestreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 191850.0 | 3195.9 | 3196.0 | 0.0 | 3195.3 | 313.5 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 192662.5 | 3207.8 | 3207.5 | 0.0 | 3207.1 | 312.5 | 0.3 |
|---|
| disksort-nvcache | 191530.0 | 3190.6 | 3190.5 | 0.0 | 3189.9 | 314.0 | 0.3 |
|---|
| disksort-throttle32 | 191175.5 | 3183.5 | 3183.5 | 0.0 | 3182.8 | 314.5 | 0.3 |
|---|
| disksort | 191634.5 | 3190.6 | 3190.5 | 0.0 | 3190.0 | 314.0 | 0.3 |
|---|
| nvcache-throttle32 | 191568.0 | 3189.5 | 3189.5 | 0.0 | 3188.8 | 313.5 | 0.3 |
|---|
| nvcache | 190596.5 | 3173.4 | 3173.5 | 0.0 | 3172.7 | 316.0 | 0.3 |
|---|
| throttle32 | 192329.0 | 3202.9 | 3203.0 | 0.0 | 3202.3 | 312.5 | 0.3 |
|---|
| fb 2 |
|---|
| default | 214839.5 | 3578.8 | 3579.0 | 0.0 | 3578.0 | 280.0 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 215168.0 | 3582.5 | 3582.5 | 0.0 | 3581.8 | 280.5 | 0.3 |
|---|
| disksort-nvcache | 188552.5 | 3139.8 | 3139.5 | 0.0 | 3139.2 | 319.5 | 0.3 |
|---|
| disksort-throttle32 | 224860.0 | 3744.9 | 3745.0 | 0.0 | 3744.2 | 267.0 | 0.3 |
|---|
| disksort | 197727.5 | 3292.1 | 3292.0 | 0.0 | 3291.5 | 305.5 | 0.3 |
|---|
| nvcache-throttle32 | 216031.5 | 3596.8 | 3597.0 | 0.0 | 3596.1 | 279.0 | 0.3 |
|---|
| nvcache | 205700.0 | 3424.9 | 3425.0 | 0.0 | 3424.2 | 294.0 | 0.3 |
|---|
| throttle32 | 198472.5 | 3306.6 | 3307.0 | 0.0 | 3305.9 | 303.5 | 0.3 |
|---|
singlestreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 127390.5 | 2122.4 | 0.0 | 2122.0 | 2122.3 | 1283.5 | 0.5 |
|---|
| disksort-nvcache-throttle32 | 127061.5 | 2116.0 | 0.0 | 2116.0 | 2116.0 | 1296.5 | 0.5 |
|---|
| disksort-nvcache | 127647.0 | 2125.4 | 0.0 | 2125.5 | 2125.4 | 1282.5 | 0.5 |
|---|
| disksort-throttle32 | 127772.0 | 2128.3 | 0.0 | 2128.0 | 2128.3 | 1278.0 | 0.5 |
|---|
| disksort | 126857.5 | 2112.2 | 0.0 | 2112.0 | 2112.2 | 1281.5 | 0.5 |
|---|
| nvcache-throttle32 | 127735.0 | 2126.8 | 0.0 | 2127.0 | 2126.8 | 1282.5 | 0.5 |
|---|
| nvcache | 127122.5 | 2117.0 | 0.0 | 2117.0 | 2117.0 | 1282.5 | 0.5 |
|---|
| throttle32 | 126241.0 | 2103.7 | 0.0 | 2103.5 | 2103.7 | 1286.0 | 0.5 |
|---|
| fb 2 |
|---|
| default | 69126.0 | 1151.8 | 0.0 | 1152.0 | 1151.7 | 5844.0 | 0.9 |
|---|
| disksort-nvcache-throttle32 | 69821.0 | 1162.7 | 0.0 | 1162.5 | 1162.7 | 5863.0 | 0.9 |
|---|
| disksort-nvcache | 69987.0 | 1165.4 | 0.0 | 1165.5 | 1165.3 | 5847.0 | 0.9 |
|---|
| disksort-throttle32 | 69813.5 | 1163.3 | 0.0 | 1163.0 | 1163.3 | 5798.5 | 0.9 |
|---|
| disksort | 68795.5 | 1145.4 | 0.0 | 1145.5 | 1145.4 | 5858.5 | 0.9 |
|---|
| nvcache-throttle32 | 69914.5 | 1164.1 | 0.0 | 1164.0 | 1164.1 | 5862.0 | 0.9 |
|---|
| nvcache | 71164.0 | 1185.0 | 0.0 | 1185.0 | 1185.0 | 5847.5 | 0.8 |
|---|
| throttle32 | 69981.0 | 1166.1 | 0.0 | 1166.0 | 1166.1 | 5820.0 | 0.9 |
|---|
varmail
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 5061097.5 | 84341.0 | 12975.5 | 12975.5 | 304.1 | 291.5 | 0.5 |
|---|
| disksort-nvcache-throttle32 | 5332565.5 | 88838.1 | 13667.5 | 13667.5 | 320.4 | 265.0 | 0.5 |
|---|
| disksort-nvcache | 5592862.0 | 93177.3 | 14335.0 | 14335.0 | 336.7 | 244.0 | 0.5 |
|---|
| disksort-throttle32 | 5556750.5 | 92607.9 | 14247.0 | 14247.5 | 333.5 | 246.5 | 0.5 |
|---|
| disksort | 4494252.0 | 74858.5 | 11516.5 | 11517.0 | 270.4 | 345.0 | 0.7 |
|---|
| nvcache-throttle32 | 5255513.0 | 87520.7 | 13465.0 | 13465.0 | 314.9 | 262.0 | 0.5 |
|---|
| nvcache | 5166997.5 | 86103.6 | 13246.5 | 13247.0 | 310.7 | 268.0 | 0.5 |
|---|
| throttle32 | 4741532.5 | 79017.1 | 12156.5 | 12156.5 | 284.7 | 318.5 | 0.6 |
|---|
| fb 2 |
|---|
| default | 3946652.0 | 65760.9 | 10117.0 | 10117.0 | 237.7 | 360.0 | 0.7 |
|---|
| disksort-nvcache-throttle32 | 4281309.0 | 71308.6 | 10970.5 | 10970.5 | 257.1 | 317.0 | 0.7 |
|---|
| disksort-nvcache | 4359746.0 | 72615.2 | 11171.5 | 11171.5 | 262.3 | 314.0 | 0.7 |
|---|
| disksort-throttle32 | 3974716.5 | 66225.4 | 10188.5 | 10188.5 | 239.4 | 364.5 | 0.8 |
|---|
| disksort | 3932673.0 | 65488.8 | 10075.0 | 10075.0 | 236.3 | 375.0 | 0.8 |
|---|
| nvcache-throttle32 | 4403006.0 | 73319.0 | 11280.0 | 11280.0 | 264.8 | 308.0 | 0.7 |
|---|
| nvcache | 4388277.5 | 73110.4 | 11247.5 | 11248.0 | 263.9 | 304.0 | 0.7 |
|---|
| throttle32 | 3894139.0 | 64890.0 | 9983.0 | 9983.0 | 233.9 | 373.0 | 0.8 |
|---|
webproxy
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 9766060.5 | 162598.1 | 42788.5 | 8558.0 | 406.1 | 396.5 | 1.9 |
|---|
| disksort-nvcache-throttle32 | 9666092.0 | 160839.2 | 42326.0 | 8465.0 | 401.4 | 421.0 | 2.0 |
|---|
| disksort-nvcache | 10339984.5 | 172163.3 | 45306.0 | 9061.5 | 429.6 | 381.5 | 1.8 |
|---|
| disksort-throttle32 | 10198377.5 | 169787.2 | 44681.0 | 8936.0 | 424.0 | 376.5 | 1.9 |
|---|
| disksort | 9351531.0 | 155718.3 | 40978.5 | 8196.0 | 389.6 | 422.5 | 2.0 |
|---|
| nvcache-throttle32 | 9890540.5 | 164612.1 | 43319.0 | 8664.0 | 411.0 | 397.5 | 1.9 |
|---|
| nvcache | 9745276.0 | 162212.9 | 42687.5 | 8538.0 | 404.8 | 399.5 | 2.0 |
|---|
| throttle32 | 9967429.5 | 166014.6 | 43688.0 | 8738.0 | 414.6 | 401.5 | 1.9 |
|---|
| fb 2 |
|---|
| default | 9617363.5 | 160119.9 | 42136.5 | 8427.5 | 400.2 | 428.5 | 2.0 |
|---|
| disksort-nvcache-throttle32 | 8586164.0 | 142924.3 | 37611.0 | 7522.5 | 357.4 | 437.0 | 2.2 |
|---|
| disksort-nvcache | 9461755.5 | 157535.1 | 41456.5 | 8291.5 | 393.4 | 421.0 | 2.0 |
|---|
| disksort-throttle32 | 9562013.0 | 159195.1 | 41893.0 | 8378.5 | 398.1 | 409.5 | 2.0 |
|---|
| disksort | 9607139.0 | 159943.7 | 42090.5 | 8418.0 | 399.2 | 416.5 | 2.0 |
|---|
| nvcache-throttle32 | 9747348.5 | 162261.5 | 42700.5 | 8540.5 | 405.9 | 400.5 | 1.9 |
|---|
| nvcache | 9734116.5 | 161957.4 | 42620.0 | 8524.0 | 405.0 | 416.5 | 1.9 |
|---|
| throttle32 | 9280652.5 | 154618.1 | 40688.5 | 8138.0 | 386.1 | 410.0 | 2.1 |
|---|