SuperMicro X9DRH-iF (CSE-219A-R920LPB, BPN-SAS-213A) 2x E5-2690 @ 2.90GHz (8 cores with 2 HTs) 128 GB 1600MHz ECC LSI-620J 1x ESM 2x WidePort Uplink 24x Seagate ST9146852SS via LSI SAS 9207-8e (PCIe 3.0 x8, IT P15) 2x WidePort Uplink via LSI SAS 9207-8i (PCIe 3.0 x8, IT P15) Solaris 11.1 SRU 5.5, mxpio=enabled 12x 2-way mirrors ZFS + 2x Talos2 log setenv JBOD1 /dev/chassis/JBOD1 + zpool create [-f] -o failmode=continue -O compression=on -O atime=off pool1 \ mirror $JBOD1/SLOT_01_/disk $JBOD1/SLOT_02_/disk \ mirror $JBOD1/SLOT_03_/disk $JBOD1/SLOT_04_/disk \ mirror $JBOD1/SLOT_05_/disk $JBOD1/SLOT_06_/disk \ mirror $JBOD1/SLOT_07_/disk $JBOD1/SLOT_08_/disk \ mirror $JBOD1/SLOT_09_/disk $JBOD1/SLOT_10_/disk \ mirror $JBOD1/SLOT_11_/disk $JBOD1/SLOT_12_/disk \ mirror $JBOD1/SLOT_13_/disk $JBOD1/SLOT_14_/disk \ mirror $JBOD1/SLOT_15_/disk $JBOD1/SLOT_16_/disk \ mirror $JBOD1/SLOT_17_/disk $JBOD1/SLOT_18_/disk \ mirror $JBOD1/SLOT_19_/disk $JBOD1/SLOT_20_/disk \ mirror $JBOD1/SLOT_21_/disk $JBOD1/SLOT_22_/disk \ mirror $JBOD1/SLOT_23_/disk $JBOD1/SLOT_24_/disk \ spare /dev/chassis/SYS/HDD1/disk /dev/chassis/SYS/HDD9/disk \ log /dev/chassis/SYS/HDD13/disk /dev/chassis/SYS/HDD14/disk # Talos2 Simple TESTs to check upper bounds: + time sh -c 'mkfile 32g bla ; sync' ; \ + time sh -c 'mkfile 32g blabla ; sync' ; \ + time sh -c 'mkfile 32g blablabla ; sync' ; \ + time sh -c 'mkfile 32g blablablabla ; sync' + chmod 644 b* + time dd if=bla of=/dev/null bs=128k ; \ + time dd if=blabla of=/dev/null bs=128k ; \ + time dd if=blablabla of=/dev/null bs=128k ; \ + time dd if=blablablabla of=/dev/null bs=128k --------------------------------------------------------------------------- compression=on, atime=off, recordsize=128K --------------------------------------------------------------------------- write 2141.7 1998.1 2127.8 2073.9 MiB/s read 1638.4 1638.4 1646.6 1646.6 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.4 1001/1000 30.0 155 0.1 6002 5995.7 1000/999 30.0 80 0.1 6002 5995.6 1000/999 30.0 84 0.1 6002 5995.5 1000/999 30.0 84 0.1 copyfiles: 6002 5995.4 1000/999 30.0 108 0.1 6002 5995.7 1000/999 30.0 93 0.1 6002 6001.5 1001/1000 30.0 99 0.1 6002 5995.7 1000/999 30.0 74 0.1 createfiles: 149973 74944.9 0/24980 390.3 508 0.6 149977 74909.5 0/24970 390.1 425 0.5 149975 74908.7 0/24969 390.1 447 0.5 149974 74907.9 0/24968 390.1 343 0.4 deletefiles: 50000 24973.7 0/0 0.0 0 0.0 50000 24973.6 0/0 0.0 0 0.0 50000 12487.1 0/0 0.0 0 0.0 50000 49945.7 0/0 0.0 0 0.0 fileserver: 6119076 101709.1 9246/18493 2467.7 1118 1.5 7298854 121045.9 11004/22008 2936.9 925 1.2 5323806 88480.1 8044/16088 2146.0 1293 1.7 6002682 99637.7 9058/18116 2415.0 1135 1.5 listdirs: 17718296 295002.6 0/0 61.3 0 0.0 17511676 291562.1 0/0 60.6 0 0.0 17760134 295697.5 0/0 61.4 0 0.0 17105850 284805.1 0/0 59.2 0 0.0 makedirs: 10000 9989.5 0/0 0.0 0 0.0 10000 9989.2 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 10000 9988.8 0/0 0.0 0 0.0 mongo: 7006 6998.6 1000/1000 24.0 47 0.0 7006 7005.2 1001/1001 23.0 42 0.0 7006 6998.4 1000/1000 23.0 57 0.0 7006 7005.3 1001/1001 23.0 54 0.0 multistreamread: 856773 14265.0 14265/0 14251.0 279 0.3 857262 14273.0 14273/0 14259.1 279 0.3 858810 14298.7 14299/0 14284.8 278 0.3 855017 14235.7 14236/0 14221.8 279 0.3 multistreamreaddirect: 858708 14297.2 14297/0 14283.2 278 0.3 856684 14263.4 14263/0 14249.4 279 0.3 856689 14263.5 14264/0 14249.6 279 0.3 854388 14225.3 14225/0 14211.4 280 0.3 multistreamwrite: 525529 8751.5 0/8751 8751.4 1336 0.5 538704 8971.0 0/8971 8971.0 1318 0.4 543277 9045.3 0/9045 9045.2 1317 0.4 541333 9014.2 0/9014 9014.1 1322 0.4 multistreamwritedirect: 544493 9067.1 0/9067 9067.1 1314 0.4 539530 8982.9 0/8983 8982.8 1325 0.4 547603 9117.3 0/9117 9117.2 1319 0.4 546537 9101.8 0/9102 9101.7 1310 0.4 networkfs: 36661 610.4 90/90 4.0 279 0.2 36661 610.4 90/90 3.6 304 0.3 36661 610.4 90/90 3.5 298 0.3 36661 610.6 90/90 3.6 322 0.4 oltp: 8260692 137559.7 68439/68416 272.5 75 0.0 8313473 138423.4 68861/68853 274.2 75 0.0 8234563 137117.8 68217/68198 271.7 76 0.0 8266363 137647.7 68489/68453 272.7 76 0.0 openfiles: 25735501 428488.7 0/0 0.0 0 0.0 25507913 424697.2 0/0 0.0 0 0.0 26037644 433517.9 0/0 0.0 0 0.0 25973716 432453.8 0/0 0.0 0 0.0 randomfileaccess: 132680 2209.1 491/245 5530.3 6780 6.8 135828 2261.5 502/251 5861.8 6610 6.6 130456 2172.0 483/241 5500.4 6882 6.9 136234 2268.2 504/252 5813.8 6591 6.6 randomread: 11947491 198940.3 198940/0 1554.2 8 0.0 11858629 197456.6 197457/0 1542.6 8 0.0 11936431 198760.0 198760/0 1552.8 8 0.0 11731634 195366.5 195366/0 1526.3 8 0.0 randomrw: 12652407 210659.6 131500/79159 1645.8 12 0.0 12660774 210798.9 134040/76759 1646.9 12 0.0 12482530 207831.3 134260/73572 1623.7 12 0.0 12254191 204029.3 132646/71383 1594.0 12 0.0 randomwrite: 7279545 121201.3 0/121201 946.9 15 0.0 7278027 121176.4 0/121176 946.7 15 0.0 7264560 120951.4 0/120951 944.9 15 0.0 7153111 119097.3 0/119097 930.4 15 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 848 0.1 6002 60.0 10/10 0.3 896 0.1 6002 60.0 10/10 0.3 869 0.1 6002 60.0 10/10 0.3 829 0.1 removedirs: 10000 9989.0 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 10000 9989.0 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 singlestreamread: 191284 3184.8 3185/0 3184.2 315 0.3 189771 3159.6 3160/0 3159.0 319 0.3 189074 3148.0 3148/0 3147.4 321 0.3 186909 3112.0 3112/0 3111.3 325 0.3 singlestreamreaddirect: 190279 3168.0 3168/0 3167.4 316 0.3 190216 3167.0 3167/0 3166.4 317 0.3 191040 3180.7 3181/0 3180.1 316 0.3 190238 3167.4 3167/0 3166.8 317 0.3 singlestreamwrite: 127720 2126.5 0/2126 2126.5 1278 0.5 127019 2114.9 0/2115 2114.9 1298 0.5 127582 2124.2 0/2124 2124.2 1279 0.5 126391 2104.4 0/2104 2104.4 1299 0.5 singlestreamwritedirect: 127649 2125.4 0/2125 2125.4 1296 0.5 127809 2128.0 0/2128 2127.9 1283 0.5 127094 2116.1 0/2116 2116.1 1302 0.5 127288 2119.3 0/2119 2119.3 1293 0.5 tpcso: varmail: 3747097 62389.7 9598/9599 224.6 340 0.8 3484503 58014.9 8925/8925 210.3 390 0.9 4191131 69779.2 10735/10735 251.4 278 0.7 3724108 62003.9 9539/9539 223.5 342 0.8 videoserver: 23200 386.3 386/0 1027.7 3258 1.4 23200 386.3 386/0 1027.8 3330 1.4 23200 386.3 386/0 1027.7 3294 1.4 23200 386.3 386/0 1027.7 3272 1.4 webproxy: 9162947 152474.2 40125/8025 380.4 442 2.1 9720336 161848.8 42592/8518 404.1 394 1.9 9224088 153597.5 40420/8084 384.2 416 2.0 9125279 151944.0 39985/7997 379.9 455 2.1 webserver: 35796699 596286.8 192350/19236 3162.4 95 0.2 35762696 595670.0 192151/19216 3158.7 95 0.2 35891914 597872.8 192862/19287 3170.9 95 0.2 35881472 597614.0 192778/19279 3169.0 94 0.2 --------------------------------------------------------------------------- compression=off, atime=off, recordsize=128K --------------------------------------------------------------------------- write 1371.1 1376.8 1382.6 1382.6 MiB/s read 1412.4 1365.3 1365.3 1365.3 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 5995.5 1000/999 30.0 94 0.1 6002 5995.4 1000/999 30.0 127 0.5 6002 5995.6 1000/999 30.0 132 0.4 6002 5995.6 1000/999 30.0 136 0.4 copyfiles: 6002 5995.6 1000/999 30.0 85 0.1 6002 5995.5 1000/999 30.0 131 0.5 6002 5995.6 1000/999 30.0 122 0.4 6002 5995.6 1000/999 30.0 151 0.4 createfiles: 149973 74945.2 0/24980 390.3 344 0.4 149972 74907.7 0/24967 390.1 450 0.5 149975 74908.2 0/24969 390.1 337 0.4 149974 74945.9 0/24980 390.3 334 0.4 deletefiles: 50000 9989.7 0/0 0.0 0 0.0 50000 24973.7 0/0 0.0 0 0.0 50000 24986.3 0/0 0.0 0 0.0 50000 16649.3 0/0 0.0 0 0.0 fileserver: 5251819 87213.1 7928/15857 2111.7 1307 1.8 5987417 99395.7 9036/18072 2408.7 1137 1.5 6522917 108203.3 9837/19674 2622.3 1039 1.4 6735428 111872.9 10170/20341 2713.2 989 1.3 listdirs: 17727372 295153.5 0/0 61.3 0 0.0 17779555 296022.4 0/0 61.5 0 0.0 17756076 295631.3 0/0 61.4 0 0.0 17781592 296055.0 0/0 61.5 0 0.0 makedirs: 10000 9989.2 0/0 0.0 0 0.0 10000 9989.4 0/0 0.0 0 0.0 10000 9989.2 0/0 0.0 0 0.0 10000 9989.4 0/0 0.0 0 0.0 mongo: 7006 6998.4 1000/1000 23.0 29 0.0 7006 6998.2 1000/1000 23.0 29 0.0 7006 6998.3 1000/1000 23.0 50 0.0 7006 6998.2 1000/1000 23.0 58 0.0 multistreamread: 766186 12762.9 12763/0 12750.4 312 0.3 750515 12495.7 12496/0 12483.5 311 0.3 766857 12768.0 12768/0 12755.5 311 0.3 758656 12631.3 12631/0 12618.9 312 0.3 multistreamreaddirect: 765387 12744.9 12745/0 12732.5 312 0.3 761113 12674.6 12675/0 12662.2 311 0.3 766548 12766.8 12767/0 12754.3 311 0.3 763527 12712.4 12712/0 12700.0 312 0.3 multistreamwrite: 123213 2051.5 0/2051 2051.4 1231 1.9 117001 1948.0 0/1948 1948.0 1242 2.0 118901 1979.7 0/1980 1979.6 1336 2.0 114971 1914.2 0/1914 1914.1 1273 2.1 multistreamwritedirect: 121176 2017.5 0/2018 2017.5 1242 2.0 119663 1992.3 0/1992 1992.3 1249 2.0 121226 2018.4 0/2018 2018.3 1245 2.0 119051 1982.1 0/1982 1982.1 1282 2.0 networkfs: 36661 610.5 90/90 3.7 376 0.8 36661 610.5 90/90 3.6 460 1.3 36661 610.4 90/90 3.5 430 1.3 36661 610.4 90/90 3.9 453 1.4 oltp: 8089045 134695.3 67010/66995 266.8 81 0.0 8221080 136884.1 68103/68080 271.2 78 0.0 7936518 132149.7 65741/65731 261.8 85 0.0 8240232 137207.2 68259/68245 271.8 77 0.0 openfiles: 25267119 420688.2 0/0 0.0 0 0.0 26475790 440813.5 0/0 0.0 0 0.0 25412208 423104.7 0/0 0.0 0 0.0 26292222 437755.9 0/0 0.0 0 0.0 randomfileaccess: 133037 2215.0 492/246 5585.3 6722 6.8 131642 2191.8 487/244 5541.6 6674 6.8 131276 2185.7 486/243 5481.8 6663 6.8 132130 2199.9 489/244 5683.9 6661 6.8 randomread: 11451923 190675.5 190676/0 1489.7 9 0.0 11326369 188584.0 188584/0 1473.3 9 0.0 11661625 194199.4 194199/0 1517.2 8 0.0 11553096 192386.0 192386/0 1503.0 8 0.0 randomrw: 12653752 210681.9 133230/77452 1645.9 12 0.0 12228859 203608.1 134523/69085 1590.7 12 0.0 12537770 208751.0 135504/73247 1630.9 12 0.0 12611364 209976.3 131164/78812 1640.4 12 0.0 randomwrite: 7159255 119198.1 0/119198 931.2 15 0.0 7101248 118231.8 0/118232 923.7 15 0.0 7405563 123298.5 0/123298 963.3 14 0.0 7050395 117385.7 0/117386 917.1 15 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 940 0.1 6002 60.0 10/10 0.3 1130 1.6 6002 60.0 10/10 0.3 1195 1.9 6002 60.0 10/10 0.3 1116 1.8 removedirs: 10000 9989.2 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 10000 9989.1 0/0 0.0 0 0.0 singlestreamread: 174937 2912.6 2913/0 2912.0 347 0.3 10521 175.2 175/0 175.1 1432 5.7 174963 2913.1 2913/0 2912.5 347 0.3 173868 2894.8 2895/0 2894.3 353 0.3 singlestreamreaddirect: 175175 2916.6 2917/0 2916.0 349 0.3 14168 235.9 236/0 235.9 1357 4.2 174995 2913.6 2914/0 2913.0 349 0.3 173982 2896.7 2897/0 2896.2 354 0.3 singlestreamwrite: 115235 1918.6 0/1919 1918.6 1151 0.5 55407 922.5 0/922 922.5 1328 1.1 114911 1913.2 0/1913 1913.2 1199 0.5 114877 1912.6 0/1913 1912.6 1183 0.5 singlestreamwritedirect: 114298 1903.0 0/1903 1903.0 1167 0.5 53947 898.2 0/898 898.2 1358 1.1 112651 1875.6 0/1876 1875.6 1203 0.5 114043 1898.7 0/1899 1898.7 1180 0.5 tpcso: varmail: 3663776 61000.5 9385/9385 220.0 350 0.8 3459518 57600.7 8862/8862 207.7 383 0.9 3309375 55099.9 8477/8477 199.0 415 0.9 3459219 57594.8 8861/8861 208.2 391 0.9 videoserver: 23189 386.1 386/0 1027.6 3226 6.8 23232 386.8 387/0 751.9 2848 10.2 23198 386.3 386/0 1027.7 3392 7.1 23219 386.6 386/0 751.8 2960 9.1 webproxy: 8646917 143951.7 37882/7576 360.3 460 2.2 9644517 160558.6 42252/8450 401.5 426 1.9 5881451 97923.6 25769/5154 246.8 526 3.2 9859713 164160.5 43200/8640 410.6 404 1.9 webserver: 35720379 594857.7 191889/19190 3154.4 95 0.2 35763491 595628.1 192138/19215 3158.5 95 0.2 22461474 374104.2 120679/12069 1984.3 215 0.6 35721444 594971.5 191926/19193 3154.9 95 0.2 TBD: --------------------------------------------------------------------------- compression=off, atime=off, sync=disabled, recordsize=128K --------------------------------------------------------------------------- write 1382.6 1412.4 1382.6 1388.5 MiB/s read 1290.1 1359.7 1359.7 1371.1 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': --------------------------------------------------------------------------- compression=on, atime=off, sync=disabled, recordsize=128K --------------------------------------------------------------------------- write 2155.8 2141.7 2087.1 2073.9 MiB/s read 1655.0 1630.3 1646.6 1655.0 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test':