HDD benchmark results
Results of simple benchmarks using mkfile(1M), dd(1) and filebench. The RW test is used to get a feeling for max. performance wrt. to sequential reads and writes. It writes out 4x32GiB (so ARC should be poissioned enough) using mkfile(1M) and after that it reads in those files using dd(1) in chunks of 128KiB. All other tests are made using the workflow definitions coming with filebench (fb) with run 60. The only parameter changed in the fb tests is the directory to point to the appropriate directory in the tested ZFS pool. We avoided explicitly changing other parameters to have "something comparable" out of the box. So depending on the workload/test, the results may not necessarily represent the max. performance the system + storage is able to provide! So to properly interprete the results, it is recommended to have a look at the workflow definition in question and related semantics.
For more test details, have a look at *.txt files and related sub directories.
RW
| config | write [MiB/s] | read [MiB/s] |
|---|
| fb 1 |
|---|
| default | 2073.9 | 1655.0 |
|---|
| disksort-nvcache-throttle32 | 2071.3 | 1653.7 |
|---|
| disksort-nvcache | 2075.3 | 1650.8 |
|---|
| disksort-throttle32 | 2069.3 | 1640.1 |
|---|
| disksort | 2073.9 | 1652.0 |
|---|
| nvcache-throttle32 | 2034.5 | 1654.1 |
|---|
| nvcache | 2075.3 | 1634.7 |
|---|
| throttle32 | 2070.7 | 1633.5 |
|---|
| fb 2 |
|---|
| default | 1083.8 | 231.3 |
|---|
| disksort-nvcache-throttle32 | 1060.1 | 233.0 |
|---|
| disksort-nvcache | 1076.7 | 233.9 |
|---|
| disksort-throttle32 | 1058.7 | 233.6 |
|---|
| disksort | 1024.1 | 236.5 |
|---|
| nvcache-throttle32 | 1059.3 | 234.8 |
|---|
| nvcache | 1067.4 | 234.1 |
|---|
| throttle32 | 1052.7 | 231.4 |
|---|
fileserver
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 6760850.0 | 112259.9 | 10205.5 | 20411.0 | 2720.8 | 1010.5 | 1.3 |
|---|
| disksort-nvcache-throttle32 | 7585082.0 | 126090.5 | 11463.0 | 22925.5 | 3056.6 | 891.0 | 1.1 |
|---|
| disksort-nvcache | 7106208.5 | 118139.3 | 10740.0 | 21480.5 | 2865.3 | 977.5 | 1.3 |
|---|
| disksort-throttle32 | 5281640.5 | 87767.1 | 7978.5 | 15957.5 | 2126.2 | 1311.0 | 1.8 |
|---|
| disksort | 6627934.5 | 110213.2 | 10019.0 | 20039.0 | 2670.4 | 1054.5 | 1.3 |
|---|
| nvcache-throttle32 | 5784186.0 | 96100.4 | 8736.5 | 17473.0 | 2329.6 | 1211.0 | 1.6 |
|---|
| nvcache | 6022072.0 | 100065.9 | 9097.0 | 18194.0 | 2425.0 | 1141.0 | 1.5 |
|---|
| throttle32 | 5593520.5 | 92925.8 | 8447.5 | 16896.0 | 2253.9 | 1267.5 | 1.7 |
|---|
| fb 2 |
|---|
| default | 4161244.0 | 69021.6 | 6274.5 | 12549.5 | 1672.3 | 1684.5 | 2.3 |
|---|
| disksort-nvcache-throttle32 | 7585082.0 | 126090.5 | 11463.0 | 22925.5 | 3056.6 | 891.0 | 1.1 |
|---|
| disksort-nvcache | 4407914.5 | 73162.2 | 6651.0 | 13302.5 | 1775.3 | 1594.0 | 2.1 |
|---|
| disksort-throttle32 | 4303719.5 | 71413.2 | 6492.0 | 12984.5 | 1730.0 | 1649.0 | 2.2 |
|---|
| disksort | 4740466.5 | 78752.3 | 7159.5 | 14318.5 | 1908.7 | 1476.5 | 2.0 |
|---|
| nvcache-throttle32 | 4358977.5 | 72324.0 | 6575.0 | 13150.5 | 1752.1 | 1599.5 | 2.1 |
|---|
| nvcache | 5425337.5 | 90231.8 | 8202.5 | 16406.0 | 2189.0 | 1307.0 | 1.7 |
|---|
| throttle32 | 4205793.0 | 69851.9 | 6350.0 | 12700.5 | 1693.0 | 1656.5 | 2.2 |
|---|
multistreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 858486.0 | 14304.9 | 14305.0 | 0.0 | 14290.9 | 278.0 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 859188.0 | 14310.0 | 14310.0 | 0.0 | 14296.0 | 278.0 | 0.3 |
|---|
| disksort-nvcache | 858733.0 | 14310.0 | 14310.0 | 0.0 | 14296.0 | 278.0 | 0.3 |
|---|
| disksort-throttle32 | 860791.5 | 14332.8 | 14333.0 | 0.0 | 14318.8 | 277.5 | 0.3 |
|---|
| disksort | 860048.0 | 14328.7 | 14329.0 | 0.0 | 14314.7 | 277.5 | 0.3 |
|---|
| nvcache-throttle32 | 857711.0 | 14280.6 | 14280.5 | 0.0 | 14266.7 | 278.5 | 0.3 |
|---|
| nvcache | 857774.5 | 14281.6 | 14281.5 | 0.0 | 14267.7 | 278.5 | 0.3 |
|---|
| throttle32 | 858086.5 | 14286.7 | 14287.0 | 0.0 | 14272.8 | 278.5 | 0.3 |
|---|
| fb 2 |
|---|
| default | 860021.5 | 14329.8 | 14330.0 | 0.0 | 14315.7 | 277.0 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 859188.0 | 14310.0 | 14310.0 | 0.0 | 14296.0 | 278.0 | 0.3 |
|---|
| disksort-nvcache | 855095.0 | 14237.0 | 14237.0 | 0.0 | 14223.0 | 279.0 | 0.3 |
|---|
| disksort-throttle32 | 864767.0 | 14398.0 | 14398.0 | 0.0 | 14384.0 | 275.5 | 0.3 |
|---|
| disksort | 851753.0 | 14187.1 | 14187.0 | 0.0 | 14173.2 | 280.0 | 0.3 |
|---|
| nvcache-throttle32 | 864224.5 | 14389.1 | 14389.0 | 0.0 | 14375.0 | 276.5 | 0.3 |
|---|
| nvcache | 859993.5 | 14319.3 | 14319.5 | 0.0 | 14305.4 | 277.5 | 0.3 |
|---|
| throttle32 | 862572.0 | 14362.9 | 14363.0 | 0.0 | 14348.9 | 276.5 | 0.3 |
|---|
multistreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 538794.0 | 8977.0 | 0.0 | 8977.0 | 8977.0 | 1311.0 | 0.4 |
|---|
| disksort-nvcache-throttle32 | 541575.0 | 9024.5 | 0.0 | 9024.5 | 9024.4 | 1311.5 | 0.4 |
|---|
| disksort-nvcache | 541856.5 | 9028.1 | 0.0 | 9028.0 | 9028.1 | 1312.5 | 0.4 |
|---|
| disksort-throttle32 | 542015.5 | 9027.8 | 0.0 | 9028.0 | 9027.9 | 1314.5 | 0.4 |
|---|
| disksort | 524701.0 | 8743.1 | 0.0 | 8743.0 | 8743.0 | 1335.5 | 0.4 |
|---|
| nvcache-throttle32 | 538403.0 | 8968.0 | 0.0 | 8968.0 | 8967.9 | 1313.0 | 0.4 |
|---|
| nvcache | 540803.5 | 9005.0 | 0.0 | 9005.0 | 9004.9 | 1310.0 | 0.4 |
|---|
| throttle32 | 539735.0 | 8988.8 | 0.0 | 8989.0 | 8988.6 | 1314.5 | 0.4 |
|---|
| fb 2 |
|---|
| default | 190310.5 | 3171.1 | 0.0 | 3171.0 | 3171.0 | 6110.0 | 1.3 |
|---|
| disksort-nvcache-throttle32 | 541575.0 | 9024.5 | 0.0 | 9024.5 | 9024.4 | 1311.5 | 0.4 |
|---|
| disksort-nvcache | 189571.0 | 3156.4 | 0.0 | 3156.5 | 3156.4 | 6121.0 | 1.3 |
|---|
| disksort-throttle32 | 179674.0 | 2992.2 | 0.0 | 2992.0 | 2992.1 | 6343.5 | 1.3 |
|---|
| disksort | 188389.5 | 3139.6 | 0.0 | 3139.5 | 3139.6 | 6169.5 | 1.3 |
|---|
| nvcache-throttle32 | 189612.0 | 3157.8 | 0.0 | 3157.5 | 3157.6 | 6144.0 | 1.3 |
|---|
| nvcache | 188524.0 | 3141.6 | 0.0 | 3141.5 | 3141.5 | 6182.0 | 1.3 |
|---|
| throttle32 | 185474.0 | 3091.0 | 0.0 | 3091.0 | 3091.0 | 6149.0 | 1.3 |
|---|
oltp
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 7832931.5 | 130458.4 | 64908.0 | 64882.0 | 258.4 | 77.0 | 0.0 |
|---|
| disksort-nvcache-throttle32 | 8208188.0 | 136708.3 | 68009.5 | 67998.5 | 270.8 | 75.5 | 0.0 |
|---|
| disksort-nvcache | 8249370.5 | 137388.0 | 68351.0 | 68332.5 | 272.2 | 75.5 | 0.0 |
|---|
| disksort-throttle32 | 7742870.5 | 128937.6 | 64145.0 | 64132.0 | 255.5 | 81.0 | 0.0 |
|---|
| disksort | 7694815.0 | 128173.4 | 63773.0 | 63743.5 | 254.0 | 82.0 | 0.0 |
|---|
| nvcache-throttle32 | 8075789.5 | 134475.6 | 66898.5 | 66887.5 | 266.5 | 80.5 | 0.0 |
|---|
| nvcache | 8075014.0 | 134464.6 | 66896.5 | 66878.5 | 266.4 | 79.5 | 0.0 |
|---|
| throttle32 | 7820861.5 | 130235.4 | 64797.0 | 64771.5 | 258.0 | 77.0 | 0.0 |
|---|
| fb 2 |
|---|
| default | 7616213.5 | 126851.5 | 63107.0 | 63094.0 | 251.3 | 85.5 | 0.0 |
|---|
| disksort-nvcache-throttle32 | 8208188.0 | 136708.3 | 68009.5 | 67998.5 | 270.8 | 75.5 | 0.0 |
|---|
| disksort-nvcache | 7943819.0 | 132281.1 | 65812.5 | 65790.5 | 262.1 | 86.0 | 0.0 |
|---|
| disksort-throttle32 | 7647658.0 | 127339.0 | 63353.5 | 63332.5 | 252.3 | 83.0 | 0.0 |
|---|
| disksort | 7558846.0 | 125945.6 | 62659.0 | 62641.0 | 249.5 | 86.0 | 0.0 |
|---|
| nvcache-throttle32 | 7695608.5 | 128134.5 | 63749.0 | 63728.5 | 253.8 | 93.5 | 0.0 |
|---|
| nvcache | 8219326.0 | 136884.8 | 68099.0 | 68083.5 | 271.2 | 76.5 | 0.0 |
|---|
| throttle32 | 7675907.0 | 127838.7 | 63606.5 | 63577.0 | 253.3 | 83.5 | 0.0 |
|---|
singlestreamread
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 192756.5 | 3210.0 | 3210.0 | 0.0 | 3209.3 | 312.0 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 191969.0 | 3197.5 | 3197.5 | 0.0 | 3196.8 | 313.5 | 0.3 |
|---|
| disksort-nvcache | 191661.0 | 3191.7 | 3192.0 | 0.0 | 3191.1 | 314.0 | 0.3 |
|---|
| disksort-throttle32 | 191052.5 | 3180.9 | 3181.0 | 0.0 | 3180.3 | 315.0 | 0.3 |
|---|
| disksort | 191575.5 | 3190.7 | 3190.5 | 0.0 | 3190.0 | 314.5 | 0.3 |
|---|
| nvcache-throttle32 | 192803.5 | 3210.1 | 3210.0 | 0.0 | 3209.5 | 312.5 | 0.3 |
|---|
| nvcache | 191217.0 | 3183.7 | 3183.5 | 0.0 | 3183.0 | 314.5 | 0.3 |
|---|
| throttle32 | 191035.0 | 3180.6 | 3180.5 | 0.0 | 3180.0 | 315.0 | 0.3 |
|---|
| fb 2 |
|---|
| default | 227131.5 | 3783.9 | 3784.0 | 0.0 | 3783.1 | 264.5 | 0.3 |
|---|
| disksort-nvcache-throttle32 | 191969.0 | 3197.5 | 3197.5 | 0.0 | 3196.8 | 313.5 | 0.3 |
|---|
| disksort-nvcache | 227363.0 | 3785.5 | 3785.5 | 0.0 | 3784.8 | 264.5 | 0.3 |
|---|
| disksort-throttle32 | 219761.5 | 3658.9 | 3659.0 | 0.0 | 3658.2 | 274.0 | 0.3 |
|---|
| disksort | 201786.5 | 3362.1 | 3362.0 | 0.0 | 3361.5 | 302.0 | 0.3 |
|---|
| nvcache-throttle32 | 209411.0 | 3486.6 | 3486.5 | 0.0 | 3485.9 | 287.5 | 0.3 |
|---|
| nvcache | 203178.5 | 3384.7 | 3384.5 | 0.0 | 3384.0 | 296.0 | 0.3 |
|---|
| throttle32 | 225146.0 | 3749.2 | 3749.5 | 0.0 | 3748.4 | 267.0 | 0.3 |
|---|
singlestreamwrite
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 126789.5 | 2112.1 | 0.0 | 2112.0 | 2112.1 | 1285.5 | 0.5 |
|---|
| disksort-nvcache-throttle32 | 127798.0 | 2129.0 | 0.0 | 2129.0 | 2129.0 | 1286.0 | 0.5 |
|---|
| disksort-nvcache | 128012.0 | 2132.3 | 0.0 | 2132.0 | 2132.3 | 1294.5 | 0.5 |
|---|
| disksort-throttle32 | 127279.0 | 2119.6 | 0.0 | 2119.5 | 2119.6 | 1283.0 | 0.5 |
|---|
| disksort | 127443.0 | 2122.9 | 0.0 | 2123.0 | 2122.9 | 1279.0 | 0.5 |
|---|
| nvcache-throttle32 | 127260.5 | 2119.2 | 0.0 | 2119.0 | 2119.2 | 1295.5 | 0.5 |
|---|
| nvcache | 127870.5 | 2129.0 | 0.0 | 2129.0 | 2129.0 | 1281.0 | 0.5 |
|---|
| throttle32 | 127727.0 | 2126.6 | 0.0 | 2126.5 | 2126.6 | 1277.5 | 0.5 |
|---|
| fb 2 |
|---|
| default | 70058.0 | 1167.0 | 0.0 | 1167.0 | 1167.0 | 5834.0 | 0.9 |
|---|
| disksort-nvcache-throttle32 | 127798.0 | 2129.0 | 0.0 | 2129.0 | 2129.0 | 1286.0 | 0.5 |
|---|
| disksort-nvcache | 68895.5 | 1147.1 | 0.0 | 1147.0 | 1147.1 | 5860.0 | 0.9 |
|---|
| disksort-throttle32 | 70057.5 | 1166.5 | 0.0 | 1166.5 | 1166.5 | 5848.5 | 0.9 |
|---|
| disksort | 69124.0 | 1151.9 | 0.0 | 1152.0 | 1151.9 | 5788.5 | 0.9 |
|---|
| nvcache-throttle32 | 69162.5 | 1151.7 | 0.0 | 1151.5 | 1151.7 | 5902.0 | 0.9 |
|---|
| nvcache | 70296.5 | 1171.4 | 0.0 | 1171.5 | 1171.4 | 5793.0 | 0.9 |
|---|
| throttle32 | 70221.0 | 1169.9 | 0.0 | 1169.5 | 1169.9 | 5774.5 | 0.9 |
|---|
varmail
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 1434577.0 | 23905.5 | 3677.5 | 3678.0 | 76.1 | 292.0 | 2.1 |
|---|
| disksort-nvcache-throttle32 | 4844161.5 | 80731.4 | 12420.0 | 12420.5 | 290.9 | 238.0 | 0.6 |
|---|
| disksort-nvcache | 4801939.5 | 80023.8 | 12311.0 | 12311.0 | 289.1 | 238.5 | 0.6 |
|---|
| disksort-throttle32 | 1422917.5 | 23704.4 | 3647.0 | 3647.0 | 77.1 | 288.0 | 2.1 |
|---|
| disksort | 1472858.0 | 24546.1 | 3776.5 | 3776.5 | 77.3 | 266.0 | 2.0 |
|---|
| nvcache-throttle32 | 4701172.0 | 78303.0 | 12046.5 | 12047.0 | 282.6 | 251.0 | 0.6 |
|---|
| nvcache | 4679266.0 | 77907.0 | 11985.5 | 11985.5 | 280.9 | 249.0 | 0.6 |
|---|
| throttle32 | 1480112.5 | 24654.3 | 3792.5 | 3793.5 | 77.1 | 264.0 | 2.0 |
|---|
| fb 2 |
|---|
| default | 1481053.5 | 24680.3 | 3797.0 | 3797.0 | 75.2 | 284.0 | 2.0 |
|---|
| disksort-nvcache-throttle32 | 4844161.5 | 80731.4 | 12420.0 | 12420.5 | 290.9 | 238.0 | 0.6 |
|---|
| disksort-nvcache | 4025407.0 | 67035.6 | 10313.0 | 10313.5 | 242.2 | 300.0 | 0.7 |
|---|
| disksort-throttle32 | 1455158.0 | 24236.3 | 3728.5 | 3729.0 | 76.3 | 291.5 | 2.1 |
|---|
| disksort | 1490696.0 | 24843.9 | 3822.0 | 3822.0 | 77.0 | 290.0 | 2.0 |
|---|
| nvcache-throttle32 | 4262577.0 | 71001.2 | 10923.0 | 10923.0 | 256.3 | 275.5 | 0.7 |
|---|
| nvcache | 4303881.0 | 71725.0 | 11034.5 | 11035.0 | 259.4 | 267.0 | 0.7 |
|---|
| throttle32 | 1454617.0 | 24240.6 | 3729.5 | 3729.5 | 76.0 | 289.5 | 2.1 |
|---|
webproxy
| config | ops | ops/s | reads | writes | mb/s | CPU µs/op | latency [ms] |
|---|
| fb 1 |
|---|
| default | 8940325.0 | 148868.0 | 39176.0 | 7835.0 | 372.3 | 450.5 | 2.2 |
|---|
| disksort-nvcache-throttle32 | 9776139.5 | 162786.3 | 42838.5 | 8568.0 | 407.2 | 407.5 | 2.0 |
|---|
| disksort-nvcache | 10410094.5 | 173219.2 | 45584.0 | 9116.5 | 431.9 | 375.0 | 1.8 |
|---|
| disksort-throttle32 | 9749354.5 | 162289.2 | 42707.5 | 8541.5 | 405.8 | 411.0 | 2.0 |
|---|
| disksort | 9422634.5 | 156865.9 | 41280.5 | 8256.5 | 392.0 | 411.0 | 2.0 |
|---|
| nvcache-throttle32 | 9579483.5 | 159492.3 | 41971.5 | 8394.0 | 398.2 | 412.0 | 2.0 |
|---|
| nvcache | 9383971.0 | 156244.0 | 41116.5 | 8223.5 | 390.8 | 431.0 | 2.0 |
|---|
| throttle32 | 9978756.5 | 166157.4 | 43725.5 | 8745.0 | 415.0 | 389.0 | 1.9 |
|---|
| fb 2 |
|---|
| default | 10031995.5 | 167083.0 | 43969.0 | 8794.0 | 417.5 | 392.5 | 1.9 |
|---|
| disksort-nvcache-throttle32 | 9776139.5 | 162786.3 | 42838.5 | 8568.0 | 407.2 | 407.5 | 2.0 |
|---|
| disksort-nvcache | 9794290.5 | 163048.3 | 42907.5 | 8581.5 | 407.0 | 405.0 | 1.9 |
|---|
| disksort-throttle32 | 9354047.0 | 155639.5 | 40958.0 | 8191.5 | 389.0 | 426.5 | 2.0 |
|---|
| disksort | 9540100.0 | 158781.8 | 41784.5 | 8357.0 | 396.8 | 415.5 | 2.0 |
|---|
| nvcache-throttle32 | 9477777.0 | 157766.1 | 41517.5 | 8303.5 | 394.5 | 421.0 | 2.0 |
|---|
| nvcache | 9716752.0 | 161765.3 | 42570.0 | 8514.0 | 404.4 | 417.5 | 2.0 |
|---|
| throttle32 | 9457364.5 | 157413.6 | 41424.5 | 8285.0 | 393.1 | 424.5 | 2.0 |
|---|