SuperMicro X9DRH-iF (CSE-219A-R920LPB, BPN-SAS-213A) 2x E5-2690 @ 2.90GHz (8 cores with 2 HTs) 128 GB 1600MHz ECC LSI-620J 1x ESM 2x WidePort Uplink 24x Seagate ST9146852SS via LSI SAS 9207-8e (PCIe 3.0 x8, IT P15) 2x WidePort Uplink via LSI SAS 9207-8i (PCIe 3.0 x8, IT P15) Solaris 11.1 SRU 5.5, mxpio=enabled 12x 2-way mirrors ZFS + 1x Talos2 log setenv JBOD1 /dev/chassis/JBOD1 + zpool create [-f] -o failmode=continue -O compression=on -O atime=off pool1 \ mirror $JBOD1/SLOT_01_/disk $JBOD1/SLOT_02_/disk \ mirror $JBOD1/SLOT_03_/disk $JBOD1/SLOT_04_/disk \ mirror $JBOD1/SLOT_05_/disk $JBOD1/SLOT_06_/disk \ mirror $JBOD1/SLOT_07_/disk $JBOD1/SLOT_08_/disk \ mirror $JBOD1/SLOT_09_/disk $JBOD1/SLOT_10_/disk \ mirror $JBOD1/SLOT_11_/disk $JBOD1/SLOT_12_/disk \ mirror $JBOD1/SLOT_13_/disk $JBOD1/SLOT_14_/disk \ mirror $JBOD1/SLOT_15_/disk $JBOD1/SLOT_16_/disk \ mirror $JBOD1/SLOT_17_/disk $JBOD1/SLOT_18_/disk \ mirror $JBOD1/SLOT_19_/disk $JBOD1/SLOT_20_/disk \ mirror $JBOD1/SLOT_21_/disk $JBOD1/SLOT_22_/disk \ mirror $JBOD1/SLOT_23_/disk $JBOD1/SLOT_24_/disk \ spare /dev/chassis/SYS/HDD1/disk /dev/chassis/SYS/HDD9/disk \ log /dev/chassis/SYS/HDD13/disk # Talos2 Simple TESTs to check upper bounds: + time sh -c 'mkfile 32g bla ; sync' ; \ + time sh -c 'mkfile 32g blabla ; sync' ; \ + time sh -c 'mkfile 32g blablabla ; sync' ; \ + time sh -c 'mkfile 32g blablablabla ; sync' + chmod 644 b* + time dd if=bla of=/dev/null bs=128k ; \ + time dd if=blabla of=/dev/null bs=128k ; \ + time dd if=blablabla of=/dev/null bs=128k ; \ + time dd if=blablablabla of=/dev/null bs=128k --------------------------------------------------------------------------- compression=on, atime=off, recordsize=128K --------------------------------------------------------------------------- write 2010.3 2060.9 2073.9 2127.8 MiB/s read 1663.4 1646.6 1646.6 1638.4 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.5 1001/1000 30.0 330 0.1 6002 5995.6 1000/999 30.0 75 0.1 6002 6001.6 1001/1000 30.0 98 0.1 6002 6001.7 1001/1000 30.0 84 0.1 copyfiles: 6002 5995.7 1000/999 30.0 103 0.1 6002 5995.7 1000/999 30.0 89 0.1 6002 5995.5 1000/999 30.0 95 0.1 6002 6001.7 1001/1000 30.0 70 0.1 createfiles: 149973 74942.1 0/24979 390.3 431 0.5 149974 74946.0 0/24980 390.3 415 0.4 149974 74945.8 0/24980 390.3 341 0.4 149977 74909.7 0/24969 390.1 338 0.4 deletefiles: 50000 49996.1 0/0 0.0 0 0.0 50000 24973.8 0/0 0.0 0 0.0 50000 12487.1 0/0 0.0 0 0.0 50000 12487.1 0/0 0.0 0 0.0 fileserver: 6732596 111717.2 10156/20312 2708.7 1008 1.3 7099982 117767.8 10706/21413 2856.8 949 1.3 5223514 86607.4 7873/15747 2098.6 1321 1.8 5684680 94442.7 8586/17172 2289.7 1203 1.6 listdirs: 15842224 263767.3 0/0 54.8 0 0.0 15804471 263159.2 0/0 54.6 0 0.0 15807122 263182.5 0/0 54.7 0 0.0 15777977 262713.6 0/0 54.6 0 0.0 makedirs: 10000 9989.5 0/0 0.0 0 0.0 10000 9989.6 0/0 0.0 0 0.0 10000 9989.4 0/0 0.0 0 0.0 10000 9989.2 0/0 0.0 0 0.0 mongo: 7006 6998.7 1000/1000 23.0 36 0.0 7006 6998.6 1000/1000 23.0 45 0.0 7006 7005.5 1001/1001 23.0 42 0.0 7006 7005.5 1001/1001 23.0 68 0.0 multistreamread: 858019 14285.7 14286/0 14271.8 279 0.3 856549 14261.3 14261/0 14247.3 279 0.3 854730 14231.0 14231/0 14217.1 280 0.3 853581 14211.8 14212/0 14197.9 280 0.3 multistreamreaddirect: 856450 14259.6 14260/0 14245.7 279 0.3 857295 14274.5 14275/0 14260.5 279 0.3 855647 14247.3 14247/0 14233.4 279 0.3 853618 14212.5 14212/0 14198.6 280 0.3 multistreamwrite: 509242 8480.3 0/8480 8480.3 1343 0.5 541638 9019.8 0/9020 9019.7 1313 0.4 537155 8950.5 0/8950 8950.4 1309 0.4 539532 8982.6 0/8983 8982.5 1313 0.4 multistreamwritedirect: 542332 9030.6 0/9031 9030.6 1313 0.4 542608 9035.7 0/9036 9035.6 1323 0.4 543454 9049.8 0/9050 9049.7 1314 0.4 543222 9049.5 0/9049 9049.4 1316 0.4 networkfs: 36661 610.5 90/90 3.4 267 0.3 36661 610.7 90/90 4.0 331 0.3 36661 610.8 90/90 3.8 306 0.4 36661 610.7 90/90 3.6 312 0.4 oltp: 7901397 131571.6 65459/65438 260.6 84 0.0 8256307 137484.1 68393/68386 272.4 75 0.0 8271916 137755.8 68533/68517 272.9 76 0.0 7755552 129140.1 64246/64233 255.8 89 0.0 openfiles: 26094541 434471.9 0/0 0.0 0 0.0 26429405 440040.4 0/0 0.0 0 0.0 25248617 420379.5 0/0 0.0 0 0.0 25776460 429170.7 0/0 0.0 0 0.0 randomfileaccess: 135218 2251.4 500/250 5702.1 6630 6.6 132698 2209.4 491/246 5500.8 6765 6.8 135449 2255.2 501/251 5677.1 6605 6.6 135916 2263.0 503/251 5752.6 6604 6.6 randomread: 11841896 197214.7 197215/0 1540.7 8 0.0 12166316 202633.1 202633/0 1583.1 8 0.0 11864541 197562.6 197563/0 1543.4 8 0.0 11972964 199374.6 199375/0 1557.6 8 0.0 randomrw: 12431518 206981.8 128387/78595 1617.0 12 0.0 12706916 211567.1 132409/79159 1652.9 12 0.0 12242665 203837.6 132433/71404 1592.5 12 0.0 12538861 208768.9 132851/75917 1631.0 12 0.0 randomwrite: 7174906 119461.3 0/119461 933.3 15 0.0 7310621 121720.0 0/121720 950.9 14 0.0 7181641 119573.5 0/119574 934.2 15 0.0 7229958 120378.1 0/120378 940.5 15 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 813 0.1 6002 60.0 10/10 0.3 798 0.1 6002 60.0 10/10 0.3 846 0.1 6002 60.0 10/10 0.3 763 0.1 removedirs: 10000 9989.1 0/0 0.0 0 0.0 10000 9999.3 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 10000 9999.1 0/0 0.0 0 0.0 singlestreamread: 189266 3151.2 3151/0 3150.6 319 0.3 189801 3160.1 3160/0 3159.5 318 0.3 190025 3163.9 3164/0 3163.2 317 0.3 186910 3112.0 3112/0 3111.4 325 0.3 singlestreamreaddirect: 191602 3190.2 3190/0 3189.6 314 0.3 191085 3181.5 3181/0 3180.8 316 0.3 189427 3153.9 3154/0 3153.3 319 0.3 189993 3163.3 3163/0 3162.7 317 0.3 singlestreamwrite: 126843 2112.1 0/2112 2112.0 1282 0.5 126768 2110.9 0/2111 2110.9 1288 0.5 127166 2117.3 0/2117 2117.3 1291 0.5 127877 2129.4 0/2129 2129.3 1279 0.5 singlestreamwritedirect: 127682 2125.9 0/2126 2125.9 1291 0.5 127930 2130.0 0/2130 2130.0 1289 0.5 127775 2127.8 0/2128 2127.8 1282 0.5 127822 2128.6 0/2129 2128.6 1290 0.5 tpcso: varmail: 3298562 54936.4 8452/8452 196.7 360 0.9 3367477 56087.0 8629/8629 200.4 339 0.9 3335048 55532.1 8543/8544 199.2 353 0.9 3292103 54829.8 8435/8436 197.6 359 0.9 videoserver: 23200 386.3 386/0 1027.8 3268 1.4 23240 387.0 387/0 1028.0 3193 1.4 23200 386.3 386/0 1027.8 3273 1.4 23200 386.4 386/0 1027.9 3324 1.4 webproxy: 9920717 165186.5 43470/8694 413.0 398 1.9 8950855 149049.1 39223/7845 372.8 452 2.1 10152994 169068.1 44492/8898 422.3 397 1.8 8951983 148814.9 39162/7832 372.4 459 2.1 webserver: 35707039 594904.1 191904/19192 3154.4 96 0.2 36090095 601230.6 193945/19396 3188.4 94 0.2 35662263 594041.5 191626/19164 3149.7 96 0.2 35542234 592130.7 191010/19102 3139.9 96 0.2 --------------------------------------------------------------------------- compression=off, atime=off, recordsize=128K --------------------------------------------------------------------------- write 1337.5 1382.6 1382.6 1388.5 MiB/s read 1388.5 1365.3 1348.5 1321.3 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.5 1001/1000 30.0 113 0.2 6002 5995.5 1000/999 30.0 132 0.4 6002 6001.5 1001/1000 30.0 157 0.4 6002 5995.6 1000/999 30.0 146 0.4 copyfiles: 6002 5995.5 1000/999 30.0 89 0.1 6002 6001.6 1001/1000 30.0 147 0.4 6002 6001.6 1001/1000 30.0 132 0.4 6002 6001.4 1001/1000 30.0 152 0.4 createfiles: 149980 74910.9 0/24971 390.1 356 0.4 149973 74944.4 0/24980 390.3 431 0.5 149976 74909.1 0/24969 390.1 353 0.4 149973 74907.6 0/24968 390.1 440 0.5 deletefiles: 50000 24973.8 0/0 0.0 0 0.0 50000 24986.3 0/0 0.0 0 0.0 50000 12493.3 0/0 0.0 0 0.0 50000 24973.7 0/0 0.0 0 0.0 fileserver: 5946521 98608.1 8964/17929 2389.2 1150 1.5 5793917 96024.2 8730/17459 2327.1 1174 1.6 5385751 89483.0 8135/16270 2168.8 1266 1.7 5831020 96877.9 8807/17614 2349.7 1164 1.6 listdirs: 15799391 263053.2 0/0 54.6 0 0.0 15720789 261744.7 0/0 54.4 0 0.0 15794425 262970.9 0/0 54.6 0 0.0 15769317 262552.8 0/0 54.5 0 0.0 makedirs: 10000 9999.5 0/0 0.0 0 0.0 10000 9989.4 0/0 0.0 0 0.0 10000 9999.4 0/0 0.0 0 0.0 10000 9989.5 0/0 0.0 0 0.0 mongo: 7006 7005.3 1001/1001 23.0 48 0.0 7006 6998.4 1000/1000 23.0 68 0.0 7006 6998.4 1000/1000 23.0 38 0.0 7006 7005.5 1001/1001 23.0 42 0.0 multistreamread: 767157 12784.7 12785/0 12772.2 311 0.3 768651 12801.5 12802/0 12789.0 311 0.3 764629 12741.4 12741/0 12729.0 311 0.3 760703 12665.4 12665/0 12653.0 312 0.3 multistreamreaddirect: 764604 12733.1 12733/0 12720.6 311 0.3 765351 12750.8 12751/0 12738.3 311 0.3 765523 12756.8 12757/0 12744.3 311 0.3 762824 12713.1 12713/0 12700.6 312 0.3 multistreamwrite: 119198 1985.3 0/1985 1985.3 1246 2.0 118883 1979.5 0/1980 1979.4 1359 2.0 117077 1949.6 0/1950 1949.5 1242 2.0 117038 1948.6 0/1949 1948.6 1356 2.0 multistreamwritedirect: 118325 1970.2 0/1970 1970.1 1277 2.0 117483 1956.1 0/1956 1956.0 1266 2.0 117368 1954.4 0/1954 1954.3 1252 2.0 118151 1967.4 0/1967 1967.3 1238 2.0 networkfs: 36661 610.4 90/90 3.8 429 1.1 36661 610.4 90/90 3.8 459 1.3 36661 610.5 90/90 3.7 452 1.4 36661 610.5 90/90 3.6 466 1.5 oltp: 7931975 132092.0 65718/65697 261.7 84 0.0 7844560 130625.0 64987/64968 258.8 87 0.0 7966728 132660.9 65997/65984 262.8 84 0.0 8265150 137635.4 68477/68453 272.7 75 0.0 openfiles: 26201555 436247.2 0/0 0.0 0 0.0 26159925 435554.2 0/0 0.0 0 0.0 26890365 447714.8 0/0 0.0 0 0.0 25548167 425368.4 0/0 0.0 0 0.0 randomfileaccess: 135657 2258.6 502/251 5752.0 6579 6.6 133019 2214.8 492/246 5652.6 6640 6.8 129168 2150.7 478/239 5371.5 6816 7.0 129770 2160.6 480/240 5274.9 6766 6.9 randomread: 11811975 196713.3 196713/0 1536.8 8 0.0 11619303 193531.8 193532/0 1512.0 8 0.0 11112557 185085.8 185086/0 1446.0 9 0.0 11659322 194156.1 194156/0 1516.8 8 0.0 randomrw: 12646422 210560.5 137800/72760 1645.0 12 0.0 12540550 208797.8 137015/71783 1631.2 12 0.0 12376713 206070.0 135942/70128 1609.9 12 0.0 12587031 209571.3 136508/73064 1637.3 12 0.0 randomwrite: 7404256 123277.5 0/123277 963.1 14 0.0 7220073 120212.6 0/120213 939.1 15 0.0 7147853 119012.1 0/119012 929.8 15 0.0 7264714 120956.6 0/120957 945.0 15 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 1016 1.1 6002 60.0 10/10 0.3 1170 1.7 6002 60.0 10/10 0.3 1175 2.0 6002 60.0 10/10 0.3 1180 1.6 removedirs: 10000 9999.3 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 10000 9999.3 0/0 0.0 0 0.0 singlestreamread: 168000 2797.1 2797/0 2796.6 344 0.4 177217 2950.6 2951/0 2950.0 340 0.3 173704 2892.1 2892/0 2891.5 351 0.3 176283 2935.1 2935/0 2934.5 343 0.3 singlestreamreaddirect: 175571 2923.2 2923/0 2922.6 348 0.3 177834 2960.9 2961/0 2960.3 339 0.3 11797 196.5 196/0 196.4 1040 5.1 176465 2938.5 2938/0 2937.9 343 0.3 singlestreamwrite: 115396 1921.4 0/1921 1921.4 1137 0.5 114853 1912.3 0/1912 1912.3 1181 0.5 55288 920.5 0/921 920.5 1364 1.1 114676 1909.4 0/1909 1909.3 1154 0.5 singlestreamwritedirect: 114823 1911.9 0/1912 1911.9 1162 0.5 114529 1907.0 0/1907 1906.9 1192 0.5 54992 915.6 0/916 915.6 1351 1.1 114494 1906.5 0/1907 1906.5 1139 0.5 tpcso: varmail: 3030555 50478.0 7766/7766 181.7 423 1.0 3247878 54081.5 8320/8320 193.4 366 0.9 3589397 59777.2 9196/9197 215.0 301 0.8 3413845 56870.8 8749/8749 204.8 336 0.9 videoserver: 23225 386.7 386/0 1027.9 3247 8.8 23174 385.9 386/0 1027.7 3305 8.3 23185 386.1 386/0 751.7 2939 6.6 23238 387.0 387/0 752.0 2971 6.3 webproxy: 9541336 158798.5 41789/8358 397.0 416 2.0 10001571 166506.3 43817/8764 415.8 406 1.9 10566052 175812.8 46266/9253 439.5 368 1.8 7060046 117540.6 30931/6186 294.7 456 2.7 webserver: 35589796 592924.5 191266/19127 3144.0 95 0.2 35353620 588979.7 189993/19000 3123.5 97 0.2 35344317 588870.8 189958/18997 3123.0 97 0.2 35749276 595532.7 192107/19212 3158.2 95 0.2 TBD: --------------------------------------------------------------------------- compression=off, atime=off, sync=disabled, recordsize=128K --------------------------------------------------------------------------- write 1382.6 1412.4 1382.6 1388.5 MiB/s read 1290.1 1359.7 1359.7 1371.1 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': --------------------------------------------------------------------------- compression=on, atime=off, sync=disabled, recordsize=128K --------------------------------------------------------------------------- write 2155.8 2141.7 2087.1 2073.9 MiB/s read 1655.0 1630.3 1646.6 1655.0 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test':