SuperMicro X9DRH-iF (CSE-219A-R920LPB, BPN-SAS-213A) 2x E5-2690 @ 2.90GHz (8 cores with 2 HTs) 128 GB 1600MHz ECC LSI-620J 1x ESM 2x WidePort Uplink 24x Seagate ST9146852SS via LSI SAS 9207-8e (PCIe 3.0 x8, IT P15) 2x WidePort Uplink Solaris 11.1 SRU 5.5, mxpio=enabled setenv JBOD1 /dev/chassis/JBOD1 + zpool create [-f] -o failmode=continue -O compression=on -O atime=off pool1 \ mirror $JBOD1/SLOT_01_/disk $JBOD1/SLOT_02_/disk \ mirror $JBOD1/SLOT_03_/disk $JBOD1/SLOT_04_/disk \ mirror $JBOD1/SLOT_05_/disk $JBOD1/SLOT_06_/disk \ mirror $JBOD1/SLOT_07_/disk $JBOD1/SLOT_08_/disk \ mirror $JBOD1/SLOT_09_/disk $JBOD1/SLOT_10_/disk \ mirror $JBOD1/SLOT_11_/disk $JBOD1/SLOT_12_/disk \ mirror $JBOD1/SLOT_13_/disk $JBOD1/SLOT_14_/disk \ mirror $JBOD1/SLOT_15_/disk $JBOD1/SLOT_16_/disk \ mirror $JBOD1/SLOT_17_/disk $JBOD1/SLOT_18_/disk \ mirror $JBOD1/SLOT_19_/disk $JBOD1/SLOT_20_/disk \ mirror $JBOD1/SLOT_21_/disk $JBOD1/SLOT_22_/disk \ mirror $JBOD1/SLOT_23_/disk $JBOD1/SLOT_24_/disk \ spare /dev/chassis/SYS/HDD1/disk /dev/chassis/SYS/HDD9/disk Simple TESTs to check upper bounds: + time sh -c 'mkfile 32g bla ; sync' ; \ + time sh -c 'mkfile 32g blabla ; sync' ; \ + time sh -c 'mkfile 32g blablabla ; sync' ; \ + time sh -c 'mkfile 32g blablablabla ; sync' + chmod 644 b* + time dd if=bla of=/dev/null bs=128k ; \ + time dd if=blabla of=/dev/null bs=128k ; \ + time dd if=blablabla of=/dev/null bs=128k ; \ + time dd if=blablablabla of=/dev/null bs=128k --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=on, atime=off, recordsize=128K) --------------------------------------------------------------------------- write 1986.0 2074.0 2087.1 2074.0 MiB/s read 1630.3 1646.6 1638.4 1638.4 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.6 1001/1000 30.0 80 0.1 6002 6001.6 1001/1000 30.0 93 0.1 6002 6001.5 1001/1000 30.0 93 0.1 6002 6001.5 1001/1000 30.0 80 0.1 copyfiles: 6002 6001.6 1001/1000 30.0 119 0.1 6002 5995.5 1000/999 30.0 105 0.1 6002 5995.7 1000/999 30.0 84 0.1 6002 6001.4 1001/1000 30.0 103 0.1 createfiles: 149977 74982.8 0/24994 390.5 463 0.5 149980 74948.1 0/24984 390.3 374 0.4 149976 74946.0 0/24981 390.3 322 0.4 149978 74910.1 0/24970 390.1 329 0.4 deletefiles: 50000 49945.8 0/0 0.0 0 0.0 50000 24986.2 0/0 0.0 0 0.0 50000 24986.1 0/0 0.0 0 0.0 50000 49996.2 0/0 0.0 0 0.0 fileserver: 7702506 128024.0 11638/23277 3106.1 878 1.1 6581950 109378.9 9944/19887 2653.7 1034 1.4 6623465 109915.0 9992/19985 2664.9 1030 1.3 7451809 123626.1 11239/22478 2998.3 904 1.2 listdirs: 18039295 300389.5 0/0 62.4 0 0.0 18574360 309370.9 0/0 64.3 0 0.0 18240619 303751.1 0/0 63.1 0 0.0 18125306 301826.8 0/0 62.7 0 0.0 makedirs: 10000 9999.2 0/0 0.0 0 0.0 10000 9999.4 0/0 0.0 0 0.0 10000 9989.4 0/0 0.0 0 0.0 10000 9999.1 0/0 0.0 0 0.0 mongo: 7006 6998.3 1000/1000 23.0 93 0.0 7006 7005.4 1001/1001 23.0 65 0.0 7006 7005.4 1001/1001 23.0 94 0.0 7006 6998.6 1000/1000 23.0 60 0.0 multistreamread: 862093 14355.5 14356/0 14341.5 277 0.3 858251 14290.0 14290/0 14276.0 278 0.3 859108 14306.8 14307/0 14292.8 278 0.3 860985 14339.7 14340/0 14325.6 277 0.3 multistreamreaddirect: 860840 14332.5 14333/0 14318.5 278 0.3 859886 14319.5 14320/0 14305.5 278 0.3 861154 14337.8 14338/0 14323.7 278 0.3 859138 14307.8 14308/0 14293.8 278 0.3 multistreamwrite: 511924 8527.3 0/8527 8527.3 1333 0.5 535243 8915.6 0/8916 8915.5 1320 0.4 538600 8971.3 0/8971 8971.3 1321 0.4 539336 8984.1 0/8984 8984.0 1323 0.4 multistreamwritedirect: 546448 9103.6 0/9104 9103.5 1305 0.4 546379 9102.9 0/9103 9102.8 1306 0.4 545917 9094.7 0/9095 9094.6 1321 0.4 545729 9089.4 0/9089 9089.3 1319 0.4 networkfs: 36661 610.8 90/90 3.7 255 0.1 36661 610.9 90/90 3.5 284 0.1 36661 610.6 90/90 3.7 278 0.1 36661 610.9 90/90 3.8 264 0.1 oltp: 5297298 88230.9 43908/43871 174.8 97 0.0 5497098 91538.2 45555/45514 181.3 88 0.0 5295451 88188.6 43886/43851 174.7 87 0.0 5433989 90499.7 45037/44999 179.3 98 0.0 openfiles: 25818695 429883.7 0/0 0.0 0 0.0 25895192 431144.4 0/0 0.0 0 0.0 25164799 418989.5 0/0 0.0 0 0.0 25270523 420757.5 0/0 0.0 0 0.0 randomfileaccess: 135595 2257.7 502/251 5804.9 6623 6.6 132794 2212.0 491/246 5632.8 6762 6.8 135802 2261.3 502/251 5737.3 6610 6.6 135629 2259.3 502/251 5776.5 6605 6.6 randomread: 12082918 201272.2 201272/0 1572.4 8 0.0 11984883 199632.7 199633/0 1559.6 8 0.0 11761281 195905.1 195905/0 1530.5 8 0.0 11860690 197542.0 197542/0 1543.3 8 0.0 randomrw: 12627355 210240.9 134963/75278 1642.5 12 0.0 12400605 206466.2 131953/74513 1613.0 12 0.0 12564568 209196.0 137327/71869 1634.3 12 0.0 12295437 204714.5 133226/71489 1599.3 12 0.0 randomwrite: 7348179 122345.4 0/122345 955.8 14 0.0 7274825 121205.7 0/121206 946.9 15 0.0 7283488 121266.3 0/121266 947.4 15 0.0 7220903 120226.2 0/120226 939.3 15 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 833 0.1 6002 60.0 10/10 0.3 838 0.1 6002 60.0 10/10 0.3 894 0.1 6002 60.0 10/10 0.3 859 0.1 removedirs: 10000 9999.0 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 10000 9989.0 0/0 0.0 0 0.0 10000 9999.1 0/0 0.0 0 0.0 singlestreamread: 191947 3195.8 3196/0 3195.2 313 0.3 191943 3197.1 3197/0 3196.4 314 0.3 191405 3186.8 3187/0 3186.2 315 0.3 191212 3183.6 3184/0 3182.9 315 0.3 singlestreamreaddirect: 191475 3188.0 3188/0 3187.3 314 0.3 191993 3196.7 3197/0 3196.1 313 0.3 190754 3178.5 3178/0 3177.8 316 0.3 190364 3169.5 3169/0 3168.8 317 0.3 singlestreamwrite: 128164 2134.1 0/2134 2134.1 1286 0.5 127692 2126.6 0/2127 2126.6 1282 0.5 127541 2124.2 0/2124 2124.1 1279 0.5 126757 2110.8 0/2111 2110.8 1297 0.5 singlestreamwritedirect: 127926 2130.9 0/2131 2130.9 1281 0.5 127582 2124.5 0/2124 2124.4 1290 0.5 126726 2110.3 0/2110 2110.3 1297 0.5 127726 2127.3 0/2127 2127.3 1289 0.5 tpcso: varmail: 710159 11833.2 1820/1821 36.5 230 4.3 709755 11824.5 1819/1819 37.7 232 4.3 689768 11491.7 1768/1768 34.9 251 4.5 679462 11320.0 1741/1742 35.3 249 4.5 videoserver: 23200 386.4 386/0 1028.0 3313 1.4 23200 386.5 386/0 1028.3 3167 1.4 23200 386.5 386/0 1028.1 3272 1.4 23200 386.5 386/0 1028.1 3193 1.4 webproxy: 10386770 172933.7 45509/9102 430.7 386 1.8 9971714 165940.7 43669/8734 414.5 394 1.9 9866073 164246.5 43223/8645 410.3 380 1.9 10389138 172922.9 45506/9101 431.9 362 1.8 webserver: 36019795 600048.9 193564/19357 3181.3 94 0.2 36183940 602865.6 194472/19448 3197.0 94 0.2 32300139 538144.0 173595/17361 2854.7 132 0.3 36152910 602200.8 194258/19427 3193.6 94 0.2 --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=off, atime=off, recordsize=128K) --------------------------------------------------------------------------- write 1343.0 1348.5 1371.1 1382.6 MiB/s read 1400.3 1376.8 1376.8 1365.3 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 5995.6 1000/999 30.0 94 0.1 6002 6001.5 1001/1000 30.0 127 0.4 6002 6001.5 1001/1000 30.0 132 0.4 6002 6001.4 1001/1000 30.0 151 0.4 copyfiles: 6002 6001.4 1001/1000 30.0 108 0.1 6002 6001.4 1001/1000 30.0 142 0.4 6002 6001.3 1001/1000 30.0 151 0.5 6002 6001.6 1001/1000 30.0 147 0.5 createfiles: 149973 74943.8 0/24979 390.3 338 0.4 149974 74980.6 0/24992 390.5 426 0.5 149978 74947.0 0/24981 390.3 510 0.5 149974 74945.0 0/24980 390.3 503 0.5 deletefiles: 50000 49995.9 0/0 0.0 0 0.0 50000 16654.5 0/0 0.0 0 0.0 50000 16660.2 0/0 0.0 0 0.0 50000 16649.2 0/0 0.0 0 0.0 fileserver: 6524258 108406.9 9855/19711 2627.9 1040 1.4 5684884 94440.4 8585/17171 2289.4 1166 1.6 4656115 77150.1 7014/14027 1869.5 1458 2.0 6185875 102631.3 9330/18660 2487.7 1086 1.5 listdirs: 17999106 299699.7 0/0 62.2 0 0.0 18261784 304064.0 0/0 63.2 0 0.0 17809991 296551.8 0/0 61.6 0 0.0 17953579 298971.5 0/0 62.1 0 0.0 makedirs: 10000 9999.2 0/0 0.0 0 0.0 10000 9999.0 0/0 0.0 0 0.0 10000 9989.3 0/0 0.0 0 0.0 10000 9989.5 0/0 0.0 0 0.0 mongo: 7006 6998.2 1000/1000 23.0 91 0.0 7006 7005.3 1001/1001 23.0 154 0.5 7006 7005.4 1001/1001 23.0 96 0.4 7006 7005.3 1001/1001 23.0 46 0.0 multistreamread: 770018 12831.8 12832/0 12819.2 310 0.3 769925 12831.5 12832/0 12819.0 310 0.3 745479 12423.5 12423/0 12411.3 310 0.3 767907 12795.2 12795/0 12782.7 311 0.3 multistreamreaddirect: 758591 12641.6 12642/0 12629.2 310 0.3 739792 12318.4 12318/0 12306.3 310 0.3 768754 12799.3 12799/0 12786.8 311 0.3 752941 12537.3 12537/0 12525.0 310 0.3 multistreamwrite: 137864 2296.0 0/2296 2295.9 1195 1.7 123855 2063.3 0/2063 2063.2 1184 1.9 121452 2022.4 0/2022 2022.4 1179 2.0 123163 2051.1 0/2051 2051.0 1336 1.9 multistreamwritedirect: 130776 2178.0 0/2178 2177.9 1189 1.8 131619 2191.7 0/2192 2191.7 1173 1.8 127583 2124.8 0/2125 2124.7 1177 1.9 125375 2088.0 0/2088 2088.0 1209 1.9 networkfs: 36661 610.9 90/90 3.7 337 0.5 36661 610.6 90/90 3.6 447 1.4 36661 610.9 90/90 3.4 467 1.5 36661 610.9 90/90 3.9 478 1.6 oltp: 5153045 85809.5 42707/42663 170.0 87 0.0 5075883 84539.4 42074/42032 167.5 86 0.0 5201208 86604.9 43100/43061 171.6 87 0.0 5369155 89408.3 44481/44469 177.1 95 0.0 openfiles: 26152110 435462.9 0/0 0.0 0 0.0 25866663 430743.0 0/0 0.0 0 0.0 25893075 431116.2 0/0 0.0 0 0.0 25976629 432513.5 0/0 0.0 0 0.0 randomfileaccess: 135632 2258.8 502/251 5760.5 6613 6.6 131870 2196.6 488/244 5547.0 6552 6.8 131906 2197.2 488/244 5547.0 6523 6.8 132112 2199.8 489/244 5502.7 6580 6.8 randomread: 11621778 193623.8 193624/0 1512.7 8 0.0 11573263 192786.8 192787/0 1506.1 8 0.0 11394515 189778.4 189778/0 1482.6 9 0.0 11593061 193093.3 193093/0 1508.5 8 0.0 randomrw: 12360089 205791.8 132760/73031 1607.7 12 0.0 12623818 210183.6 135006/75178 1642.0 12 0.0 12253470 204123.2 130032/74091 1594.7 12 0.0 12358080 205758.4 131927/73832 1607.5 12 0.0 randomwrite: 7095559 118139.1 0/118139 923.0 15 0.0 7318385 121847.4 0/121847 951.9 15 0.0 7174555 119452.5 0/119452 933.2 15 0.0 7192575 119760.1 0/119760 935.6 15 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 913 0.1 6002 60.0 10/10 0.3 1161 1.7 6002 60.0 10/10 0.3 1150 1.9 6002 60.0 10/10 0.3 1146 2.0 removedirs: 10000 9999.2 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 10000 9998.5 0/0 0.0 0 0.0 10000 9999.0 0/0 0.0 0 0.0 singlestreamread: 177324 2952.4 2952/0 2951.8 339 0.3 175841 2927.7 2928/0 2927.1 344 0.3 176703 2942.1 2942/0 2941.5 344 0.3 176743 2942.7 2943/0 2942.1 342 0.3 singlestreamreaddirect: 73264 1220.4 1220/0 1220.1 444 0.8 176342 2936.0 2936/0 2935.4 346 0.3 174445 2904.4 2904/0 2903.8 352 0.3 94232 1569.5 1570/0 1569.2 420 0.6 singlestreamwrite: 116925 1947.0 0/1947 1947.0 1134 0.5 116668 1943.1 0/1943 1943.1 1137 0.5 113728 1893.7 0/1894 1893.7 1178 0.5 117112 1950.5 0/1950 1950.5 1149 0.5 singlestreamwritedirect: 112135 1867.4 0/1867 1867.4 1148 0.5 113757 1894.2 0/1894 1894.2 1139 0.5 112793 1878.3 0/1878 1878.2 1194 0.5 110649 1843.0 0/1843 1842.9 1163 0.5 tpcso: varmail: 695507 11589.2 1783/1783 36.9 242 4.4 709606 11822.0 1819/1819 37.0 238 4.3 667158 11115.0 1710/1710 35.7 245 4.6 676618 11273.7 1734/1735 35.2 259 4.5 videoserver: 23189 386.3 386/0 752.0 2850 10.2 23198 386.4 386/0 752.0 3009 11.1 23153 385.7 385/0 751.9 3155 10.2 23179 386.0 386/0 751.8 3262 7.4 webproxy: 9377629 156093.4 41077/8215 390.9 407 2.0 8071895 134378.2 35363/7073 336.5 503 2.3 8579814 142877.2 37599/7520 357.0 482 2.2 9617367 159870.0 42071/8414 400.1 404 2.0 webserver: 35267861 587581.7 189542/18955 3116.3 97 0.2 35549237 592239.9 191045/19105 3141.0 96 0.2 35656195 594032.3 191623/19163 3149.5 95 0.2 35276975 587720.5 189587/18960 3116.3 97 0.2 --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=off, atime=off, sync=disabled, recordsize=128K) --------------------------------------------------------------------------- write 1394.4 1400.3 1400.3 1400.3 MiB/s read 1222.7 1310.7 1295.2 1316.0 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 5995.5 1000/999 30.0 151 0.4 6002 6001.7 1001/1000 30.0 187 0.4 6002 5995.7 1000/999 30.0 75 0.1 6002 6001.4 1001/1000 30.0 197 0.4 copyfiles: 6002 6001.6 1001/1000 30.0 137 0.4 6002 6001.6 1001/1000 30.0 137 0.4 6002 5995.6 1000/999 30.0 132 0.4 6002 6001.4 1001/1000 30.0 172 0.4 createfiles: 149975 74945.4 0/24980 390.3 342 0.4 149973 74944.1 0/24980 390.3 466 0.5 149976 74945.7 0/24981 390.3 440 0.5 149974 74906.3 0/24967 390.1 340 0.4 deletefiles: 50000 12487.0 0/0 0.0 0 0.0 50000 9991.6 0/0 0.0 0 0.0 50000 12490.1 0/0 0.0 0 0.0 50000 12490.1 0/0 0.0 0 0.0 fileserver: 7544638 125155.4 11378/22756 3037.0 888 1.2 6022160 100022.5 9093/18186 2425.6 1125 1.5 6620628 109919.3 9993/19986 2666.3 1025 1.4 6422386 106556.7 9687/19374 2581.9 1057 1.4 listdirs: 18515736 308382.8 0/0 64.0 0 0.0 18462635 307438.0 0/0 63.8 0 0.0 18102333 301404.6 0/0 62.6 0 0.0 18152275 302339.2 0/0 62.8 0 0.0 makedirs: 10000 9999.4 0/0 0.0 0 0.0 10000 9999.4 0/0 0.0 0 0.0 10000 9999.4 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 mongo: 7006 7005.5 1001/1001 23.0 73 0.2 7006 7005.4 1001/1001 23.0 47 0.0 7006 6998.5 1000/1000 23.0 41 0.0 7006 6998.6 1000/1000 23.0 38 0.0 multistreamread: 769352 12818.6 12819/0 12806.1 311 0.3 767501 12788.4 12788/0 12775.8 311 0.3 769336 12809.4 12809/0 12796.9 311 0.3 767519 12791.0 12791/0 12778.5 311 0.3 multistreamreaddirect: 769050 12809.6 12810/0 12797.0 311 0.3 766834 12767.4 12767/0 12754.9 311 0.3 767680 12793.4 12793/0 12780.9 310 0.3 769160 12808.8 12809/0 12796.3 311 0.3 multistreamwrite: 118402 1972.0 0/1972 1971.9 1268 2.0 116071 1933.2 0/1933 1933.1 1241 2.1 116717 1944.1 0/1944 1944.0 1241 2.1 118698 1976.8 0/1977 1976.8 1245 2.0 multistreamwritedirect: 120053 1999.4 0/1999 1999.4 1236 2.0 121482 2023.1 0/2023 2023.0 1233 2.0 120916 2013.9 0/2014 2013.8 1237 2.0 120679 2009.5 0/2009 2009.4 1243 2.0 networkfs: 36661 610.6 90/90 3.7 420 1.0 36333 605.5 89/89 3.4 415 1.0 36661 610.7 90/90 3.5 408 1.1 36661 610.5 90/90 3.6 453 1.3 oltp: 9848829 164009.8 81585/81584 324.9 48 0.0 9505148 158278.1 78737/78729 313.6 55 0.0 9538467 158833.2 79016/79003 314.7 51 0.0 9642325 160573.7 79876/79874 318.1 53 0.0 openfiles: 25284069 420996.4 0/0 0.0 0 0.0 25741298 428629.3 0/0 0.0 0 0.0 26193248 436119.0 0/0 0.0 0 0.0 26003763 433020.7 0/0 0.0 0 0.0 randomfileaccess: 132779 2210.8 491/246 5610.1 6681 6.8 133670 2225.6 495/247 5614.8 6622 6.7 135189 2251.4 500/250 5771.9 6534 6.6 133786 2228.1 495/248 5588.6 6631 6.7 randomread: 11315207 188493.4 188493/0 1472.6 9 0.0 11485008 191313.5 191313/0 1494.6 9 0.0 11617442 193513.3 193513/0 1511.8 8 0.0 11701434 194905.8 194906/0 1522.7 8 0.0 randomrw: 12633573 210344.6 136022/74323 1643.3 12 0.0 12512109 208428.4 132555/75873 1628.3 12 0.0 12445823 207219.2 131471/75748 1618.9 12 0.0 12561766 209149.7 132270/76880 1634.0 12 0.0 randomwrite: 7542071 125647.8 0/125648 981.6 14 0.0 7735221 128890.5 0/128890 1007.0 14 0.0 7643296 127321.6 0/127322 994.7 14 0.0 7752864 129205.7 0/129206 1009.4 13 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 1205 1.6 6002 60.0 10/10 0.3 1166 1.8 6002 60.0 10/10 0.3 1086 1.7 6002 60.0 10/10 0.3 1140 1.6 removedirs: 10000 9999.2 0/0 0.0 0 0.0 10000 9998.9 0/0 0.0 0 0.0 10000 9999.1 0/0 0.0 0 0.0 10000 9999.1 0/0 0.0 0 0.0 singlestreamread: 176818 2943.9 2944/0 2943.3 344 0.3 175825 2927.4 2927/0 2926.8 345 0.3 174534 2905.9 2906/0 2905.3 349 0.3 176259 2934.6 2935/0 2934.1 344 0.3 singlestreamreaddirect: 175412 2920.6 2921/0 2920.0 347 0.3 175900 2928.7 2929/0 2928.1 345 0.3 176293 2937.5 2937/0 2936.9 345 0.3 173688 2891.9 2892/0 2891.3 352 0.3 singlestreamwrite: 114169 1899.8 0/1900 1899.7 1185 0.5 113208 1885.1 0/1885 1885.1 1165 0.5 114312 1903.7 0/1904 1903.7 1148 0.5 114393 1905.1 0/1905 1905.1 1173 0.5 singlestreamwritedirect: 124117 2066.9 0/2067 2066.9 1078 0.5 113500 1890.6 0/1891 1890.6 1182 0.5 117982 1965.2 0/1965 1965.2 1083 0.5 113059 1883.2 0/1883 1883.2 1157 0.5 tpcso: varmail: 14311684 238317.5 36664/36664 858.8 177 0.2 17176915 286029.8 44005/44005 1031.6 140 0.2 14675843 244377.5 37596/37597 879.0 183 0.2 14275759 237719.0 36572/36572 856.4 184 0.2 videoserver: 23176 386.0 386/0 1027.9 3231 9.5 23167 385.9 386/0 1027.9 3241 8.9 23216 386.7 386/0 752.0 2889 11.6 23184 386.2 386/0 1028.1 3271 11.7 webproxy: 8482563 141139.9 37142/7428 354.0 466 2.2 9388016 156239.7 41116/8223 390.8 428 2.0 7451856 124096.9 32657/6531 311.0 471 2.5 9811255 163315.0 42978/8596 408.7 418 1.9 webserver: 35663278 594228.2 191686/19170 3151.0 96 0.2 33527750 558477.2 180154/18016 2961.3 130 0.3 35359490 589149.6 190048/19006 3124.0 96 0.2 24612284 410035.0 132269/13228 2175.0 192 0.5 --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=on, atime=off, sync=disabled, recordsize=128K) --------------------------------------------------------------------------- write 2087.1 2073.9 2087.1 2048.0 MiB/s read 1638.4 1654.9 1630.3 1646.6 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.5 1001/1000 30.0 172 0.3 6002 6001.3 1001/1000 30.0 177 0.2 6002 6001.6 1001/1000 30.0 167 0.2 6002 6001.5 1001/1000 30.0 198 0.2 copyfiles: 6002 5995.6 1000/999 30.0 143 0.3 6002 6001.6 1001/1000 30.0 98 0.1 6002 6001.2 1001/1000 30.0 158 0.2 6002 6001.4 1001/1000 30.0 181 0.3 createfiles: 149975 74945.0 0/24980 390.3 443 0.5 149977 74946.3 0/24981 390.3 343 0.4 149973 74944.8 0/24980 390.3 343 0.4 149981 74947.7 0/24983 390.3 467 0.5 deletefiles: 50000 16649.3 0/0 0.0 0 0.0 50000 12487.0 0/0 0.0 0 0.0 50000 24998.2 0/0 0.0 0 0.0 50000 12490.1 0/0 0.0 0 0.0 fileserver: 6594381 109486.9 9953/19907 2656.5 1031 1.4 7364175 122183.8 11108/22215 2961.5 921 1.2 8204700 136295.5 12390/24781 3303.5 816 1.1 5780792 95956.2 8723/17447 2328.1 1186 1.6 listdirs: 17870866 297560.0 0/0 61.8 0 0.0 18028752 300194.6 0/0 62.3 0 0.0 18284431 304465.3 0/0 63.2 0 0.0 17323777 288508.2 0/0 59.9 0 0.0 makedirs: 10000 9989.4 0/0 0.0 0 0.0 10000 9999.3 0/0 0.0 0 0.0 10000 9999.5 0/0 0.0 0 0.0 10000 9989.4 0/0 0.0 0 0.0 mongo: 7006 7005.5 1001/1001 23.0 67 0.1 7006 7005.5 1001/1001 23.0 84 0.1 7006 7005.5 1001/1001 23.0 114 0.2 7006 6998.5 1000/1000 23.0 56 0.0 multistreamread: 856375 14259.4 14259/0 14245.4 279 0.3 855177 14252.2 14252/0 14238.2 279 0.3 855898 14264.1 14264/0 14250.1 279 0.3 855277 14241.1 14241/0 14227.2 279 0.3 multistreamreaddirect: 856554 14261.2 14261/0 14247.3 279 0.3 856616 14262.9 14263/0 14249.0 279 0.3 856402 14258.9 14259/0 14245.0 279 0.3 857314 14276.9 14277/0 14262.9 279 0.3 multistreamwrite: 550400 9168.8 0/9169 9168.8 1311 0.4 545846 9090.7 0/9091 9090.7 1315 0.4 546519 9102.8 0/9103 9102.8 1314 0.4 546156 9096.6 0/9097 9096.6 1319 0.4 multistreamwritedirect: 551249 9181.3 0/9181 9181.3 1310 0.4 545446 9088.2 0/9088 9088.2 1319 0.4 548878 9140.8 0/9141 9140.7 1312 0.4 550629 9169.3 0/9169 9169.3 1307 0.4 networkfs: 36395 606.2 89/89 3.4 393 0.8 36420 606.8 90/90 3.6 426 0.9 36661 610.7 90/90 3.7 420 0.6 36661 610.9 90/90 3.6 387 0.5 oltp: 9729250 162025.3 80602/80593 321.0 50 0.0 10045013 167286.5 83221/83208 331.4 43 0.0 9516704 158480.7 78838/78830 314.0 55 0.0 9418997 156832.7 78016/78013 310.7 54 0.0 openfiles: 25614704 426501.4 0/0 0.0 0 0.0 26022543 433292.6 0/0 0.0 0 0.0 25424548 423328.5 0/0 0.0 0 0.0 25886217 431105.6 0/0 0.0 0 0.0 randomfileaccess: 132310 2203.2 490/245 5509.1 6734 6.8 132840 2212.1 492/246 5621.3 6725 6.8 132939 2213.4 492/246 5439.5 6691 6.8 134333 2236.7 497/249 5715.0 6628 6.7 randomread: 11701350 194864.5 194864/0 1522.4 8 0.0 11689959 194765.1 194765/0 1521.6 8 0.0 11443247 190709.5 190710/0 1489.9 8 0.0 11755279 195785.6 195786/0 1529.6 8 0.0 randomrw: 12304042 204859.2 132282/72578 1600.5 12 0.0 12499357 208109.9 132899/75211 1625.9 12 0.0 12364242 205861.0 130246/75615 1608.3 12 0.0 12343742 205519.4 130170/75350 1605.6 12 0.0 randomwrite: 7591705 126469.9 0/126470 988.0 14 0.0 7705288 128371.3 0/128371 1002.9 14 0.0 7568190 126051.8 0/126052 984.8 14 0.0 7659223 127561.3 0/127561 996.6 14 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 1046 0.7 6002 60.0 10/10 0.3 1035 0.7 6002 60.0 10/10 0.3 976 0.3 6002 60.0 10/10 0.3 1008 0.4 removedirs: 10000 9989.2 0/0 0.0 0 0.0 10000 9999.0 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 10000 9999.3 0/0 0.0 0 0.0 singlestreamread: 183126 3049.0 3049/0 3048.4 332 0.3 181545 3022.9 3023/0 3022.3 336 0.3 184308 3068.8 3069/0 3068.2 331 0.3 180748 3009.4 3009/0 3008.8 338 0.3 singlestreamreaddirect: 185963 3096.2 3096/0 3095.6 327 0.3 189349 3155.0 3155/0 3154.4 318 0.3 188401 3136.8 3137/0 3136.2 322 0.3 190262 3167.9 3168/0 3167.3 318 0.3 singlestreamwrite: 126509 2106.8 0/2107 2106.8 1291 0.5 127074 2116.9 0/2117 2116.9 1296 0.5 127535 2123.4 0/2123 2123.4 1297 0.5 128162 2134.4 0/2134 2134.4 1282 0.5 singlestreamwritedirect: 128568 2141.2 0/2141 2141.2 1280 0.5 129364 2154.1 0/2154 2154.0 1282 0.5 126925 2113.9 0/2114 2113.8 1300 0.5 127326 2119.9 0/2120 2119.9 1287 0.5 tpcso: varmail: 15442252 257139.1 39560/39560 924.0 160 0.2 15121585 251795.7 38738/38738 904.0 172 0.2 14809565 246600.2 37938/37938 885.1 178 0.2 12350963 205650.9 31639/31639 740.8 217 0.2 videoserver: 23200 386.4 386/0 1028.1 3258 1.4 23204 386.5 386/0 1028.0 3202 1.4 23200 386.5 386/0 1028.1 3370 1.4 23200 386.6 386/0 1028.4 3369 1.4 webproxy: 9929238 165194.0 43472/8694 412.6 399 1.9 10387418 172822.6 45480/9096 431.5 365 1.8 9342758 155544.3 40933/8187 389.1 440 2.0 9061058 150915.1 39714/7943 377.1 439 2.1 webserver: 35939119 598676.3 193121/19313 3174.3 94 0.2 35734564 595251.9 192016/19203 3156.4 95 0.2 35791803 596259.3 192341/19235 3161.4 95 0.2 36080056 601080.0 193896/19391 3187.3 94 0.2