SuperMicro X8DAH+iF (CSE-219A-R920LPB, BPN-SAS-213A) 2x X5680 @ 3.33GHz (6 cores with 2 HTs) 96 GB 1333MHz ECC LSI-620J 1x ESM 2x WidePort Uplink 24x Seagate ST9146852SS via LSI SAS 9200-8e (PCIe 2.0 x8, IT P15) 2x WidePort Uplink Solaris 11.1 SRU 5.5, mxpio=enabled setenv JBOD1 /dev/chassis/JBOD1 + zpool create [-f] -o failmode=continue -O compression=on -O atime=off pool1 \ mirror $JBOD1/SLOT_01_/disk $JBOD1/SLOT_02_/disk \ mirror $JBOD1/SLOT_03_/disk $JBOD1/SLOT_04_/disk \ mirror $JBOD1/SLOT_05_/disk $JBOD1/SLOT_06_/disk \ mirror $JBOD1/SLOT_07_/disk $JBOD1/SLOT_08_/disk \ mirror $JBOD1/SLOT_09_/disk $JBOD1/SLOT_10_/disk \ mirror $JBOD1/SLOT_11_/disk $JBOD1/SLOT_12_/disk \ mirror $JBOD1/SLOT_13_/disk $JBOD1/SLOT_14_/disk \ mirror $JBOD1/SLOT_15_/disk $JBOD1/SLOT_16_/disk \ mirror $JBOD1/SLOT_17_/disk $JBOD1/SLOT_18_/disk \ mirror $JBOD1/SLOT_19_/disk $JBOD1/SLOT_20_/disk \ mirror $JBOD1/SLOT_21_/disk $JBOD1/SLOT_22_/disk \ mirror $JBOD1/SLOT_23_/disk $JBOD1/SLOT_24_/disk \ spare /dev/chassis/SYS/HDD1/disk /dev/chassis/SYS/HDD9/disk Simple TESTs to check upper bounds: + time sh -c 'mkfile 32g bla ; sync' ; \ + time sh -c 'mkfile 32g blabla ; sync' ; \ + time sh -c 'mkfile 32g blablabla ; sync' ; \ + time sh -c 'mkfile 32g blablablabla ; sync' + chmod 644 b* + time dd if=bla of=/dev/null bs=128k ; \ + time dd if=blabla of=/dev/null bs=128k ; \ + time dd if=blablabla of=/dev/null bs=128k ; \ + time dd if=blablablabla of=/dev/null bs=128k --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=on, atime=off, recordsize=128K) --------------------------------------------------------------------------- write 2391.8 2501.4 2409.4 2427.3 MiB/s read 1638.4 1646.6 1646.6 1646.6 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.3 1001/1000 30.0 79 0.1 6002 6001.3 1001/1000 30.0 90 0.1 6002 6001.3 1001/1000 30.0 113 0.1 6002 6001.5 1001/1000 30.0 69 0.1 copyfiles: 6002 6001.4 1001/1000 30.0 89 0.1 6002 6001.2 1001/1000 30.0 144 0.1 6002 6001.3 1001/1000 30.0 74 0.1 6002 6001.4 1001/1000 30.0 83 0.1 createfiles: 149980 74947.9 0/24982 390.3 399 0.4 149979 74946.1 0/24982 390.3 352 0.4 149976 74945.6 0/24981 390.3 425 0.5 149981 74947.1 0/24983 390.3 347 0.4 deletefiles: 50000 49947.6 0/0 0.0 0 0.0 50000 24986.0 0/0 0.0 0 0.0 50000 24973.8 0/0 0.0 0 0.0 50000 24986.0 0/0 0.0 0 0.0 fileserver: 7394799 123016.4 11183/22367 2981.8 677 1.0 5311110 88244.4 8022/16045 2140.0 966 1.5 5455737 90689.5 8244/16489 2198.5 938 1.5 5554125 92342.2 8395/16790 2238.6 919 1.5 listdirs: 23237464 386890.7 0/0 80.4 0 0.0 23260152 387463.4 0/0 80.5 0 0.0 23168216 385895.9 0/0 80.1 0 0.0 23208513 386560.9 0/0 80.3 0 0.0 makedirs: 10000 9999.1 0/0 0.0 0 0.0 10000 9999.1 0/0 0.0 0 0.0 10000 9998.7 0/0 0.0 0 0.0 10000 9999.1 0/0 0.0 0 0.0 mongo: 7006 7005.1 1001/1001 23.0 81 0.0 7006 7005.1 1001/1001 23.0 33 0.0 7006 7005.2 1001/1001 23.0 98 0.0 7006 7005.4 1001/1001 23.0 43 0.0 multistreamread: 838307 13970.0 13970/0 13956.4 286 0.3 832950 13880.9 13881/0 13867.3 287 0.3 825730 13760.3 13760/0 13746.9 290 0.3 824937 13747.5 13748/0 13734.1 290 0.3 multistreamreaddirect: 828456 13805.7 13806/0 13792.2 289 0.3 825257 13752.6 13753/0 13739.2 290 0.3 826028 13765.7 13766/0 13752.2 290 0.3 835604 13925.3 13925/0 13911.7 286 0.3 multistreamwrite: 338862 5644.9 0/5645 5644.9 2213 0.7 352166 5867.1 0/5867 5867.0 2184 0.7 347545 5790.1 0/5790 5790.0 2177 0.7 350104 5832.6 0/5833 5832.6 2186 0.7 multistreamwritedirect: 350136 5833.1 0/5833 5833.1 2188 0.7 350093 5833.9 0/5834 5833.8 2179 0.7 348729 5811.2 0/5811 5811.2 2191 0.7 351826 5861.4 0/5861 5861.3 2199 0.7 networkfs: 36661 611.0 90/90 3.8 289 0.1 36661 610.9 90/90 3.6 282 0.1 36661 611.0 90/90 3.8 289 0.1 36661 610.9 90/90 3.9 307 0.2 oltp: 5093681 84809.2 42211/42164 168.0 55 0.0 5028536 83727.5 41673/41626 165.8 54 0.0 4961195 82603.3 41112/41068 163.6 56 0.0 4942259 82292.7 40959/40912 163.0 64 0.0 openfiles: 38041672 633375.2 0/0 0.0 0 0.0 37392268 622568.4 0/0 0.0 0 0.0 38163217 635401.7 0/0 0.0 0 0.0 38434217 639914.8 0/0 0.0 0 0.0 randomfileaccess: 149861 2497.4 555/278 7144.2 6002 6.0 149762 2495.6 555/277 6885.2 6000 6.0 149490 2491.0 553/277 6887.9 6024 6.0 149058 2483.6 552/276 6884.1 6029 6.0 randomread: 12448088 207453.6 207454/0 1620.7 8 0.0 12545549 209077.7 209078/0 1633.4 8 0.0 12340416 205658.7 205659/0 1606.7 8 0.0 12364533 206060.1 206060/0 1609.8 8 0.0 randomrw: 13320167 221988.8 138210/83779 1734.3 11 0.0 13697510 228259.5 143809/84451 1783.3 11 0.0 13335746 222245.2 140418/81827 1736.3 11 0.0 13574221 226213.2 143419/82795 1767.3 11 0.0 randomwrite: 7321755 122020.3 0/122020 953.3 15 0.0 7483515 124716.4 0/124716 974.3 14 0.0 7230242 120495.4 0/120495 941.4 15 0.0 7360416 122664.5 0/122664 958.3 14 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 1246 0.1 6002 60.0 10/10 0.3 1231 0.1 6002 60.0 10/10 0.3 1197 0.1 6002 60.0 10/10 0.3 1189 0.1 removedirs: 10000 9998.7 0/0 0.0 0 0.0 10000 9999.0 0/0 0.0 0 0.0 10000 9999.0 0/0 0.0 0 0.0 10000 9998.5 0/0 0.0 0 0.0 singlestreamread: 260499 4341.3 4341/0 4340.5 232 0.2 260415 4339.9 4340/0 4339.1 232 0.2 258540 4308.7 4309/0 4307.9 235 0.2 268216 4469.9 4470/0 4469.0 225 0.2 singlestreamreaddirect: 270902 4514.7 4515/0 4513.8 223 0.2 260527 4341.7 4342/0 4340.9 231 0.2 259421 4323.4 4323/0 4322.5 233 0.2 269927 4498.5 4498/0 4497.6 223 0.2 singlestreamwrite: 162099 2701.3 0/2701 2701.3 1679 0.4 159887 2664.4 0/2664 2664.3 1630 0.4 160249 2670.1 0/2670 2670.1 1598 0.4 160358 2672.0 0/2672 2672.0 1607 0.4 singlestreamwritedirect: 160609 2676.5 0/2676 2676.5 1629 0.4 160103 2668.0 0/2668 2668.0 1612 0.4 160310 2671.1 0/2671 2671.1 1623 0.4 161688 2694.0 0/2694 2693.9 1657 0.4 tpcso: varmail: 690692 11510.3 1771/1771 41.3 212 4.5 672158 11200.3 1723/1723 40.1 218 4.6 702414 11702.1 1800/1800 42.1 207 4.4 695763 11593.4 1783/1784 40.6 209 4.5 videoserver: 23160 385.9 386/0 1028.3 4036 1.1 23164 386.0 386/0 1055.6 3902 1.0 23160 386.0 386/0 1028.4 4023 1.1 23160 385.9 386/0 1028.4 3980 1.1 webproxy: 9931156 165292.6 43498/8700 413.0 310 1.9 9686470 161255.9 42436/8487 403.3 331 1.9 9830562 163668.8 43071/8614 408.8 335 1.9 9606297 159879.8 42074/8415 399.4 334 2.0 webserver: 38016032 633217.3 204263/20427 3357.4 70 0.2 37994878 632846.5 204144/20415 3355.8 70 0.2 37811908 629807.0 203163/20317 3339.1 70 0.2 38525832 641658.9 206986/20700 3401.8 69 0.2 --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=off, atime=off, recordsize=128K) --------------------------------------------------------------------------- write 1456.4 1406.4 1400.3 1371.1 MiB/s read 1290.1 1270.1 1295.2 1316.0 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.3 1001/1000 30.0 84 0.1 6002 6001.5 1001/1000 30.0 156 0.4 6002 3000.7 500/500 15.0 176 0.5 6002 6001.5 1001/1000 30.0 84 0.1 copyfiles: 6002 6001.3 1001/1000 30.0 99 0.1 6002 6001.5 1001/1000 30.0 89 0.1 6002 6001.2 1001/1000 30.0 75 0.1 6002 6001.4 1001/1000 30.0 79 0.1 createfiles: 149975 74908.0 0/24968 390.1 490 0.5 149975 74945.7 0/24981 390.3 351 0.4 149977 74946.3 0/24981 390.3 344 0.4 149981 74948.5 0/24983 390.3 326 0.4 deletefiles: 50000 49994.7 0/0 0.0 0 0.0 50000 12490.1 0/0 0.0 0 0.0 50000 24973.4 0/0 0.0 0 0.0 50000 12487.0 0/0 0.0 0 0.0 fileserver: 6758096 112483.6 10226/20452 2727.1 741 1.2 6235382 103706.6 9428/18856 2515.9 796 1.3 6390026 106266.3 9661/19321 2580.3 773 1.3 5176322 86106.3 7828/15656 2088.6 964 1.6 listdirs: 22065437 367526.8 0/0 76.3 0 0.0 22515621 374878.9 0/0 77.9 0 0.0 22537363 375234.5 0/0 77.9 0 0.0 22692655 378015.2 0/0 78.5 0 0.0 makedirs: 10000 9999.0 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 10000 9998.8 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 mongo: 7006 7005.3 1001/1001 23.0 123 0.0 7006 7005.2 1001/1001 23.0 83 0.4 7006 7005.3 1001/1001 23.0 122 0.3 7006 7005.4 1001/1001 23.0 96 0.4 multistreamread: 731194 12185.1 12185/0 12173.2 324 0.3 727959 12131.3 12131/0 12119.4 323 0.3 731469 12189.7 12190/0 12177.8 327 0.3 706714 11777.2 11777/0 11765.7 324 0.3 multistreamreaddirect: 735234 12252.5 12253/0 12240.6 326 0.3 721518 12024.1 12024/0 12012.4 323 0.3 731016 12181.9 12182/0 12170.0 325 0.3 731868 12196.6 12197/0 12184.7 324 0.3 multistreamwrite: 116313 1937.2 0/1937 1937.1 1643 2.1 112186 1868.5 0/1868 1868.4 1577 2.1 118157 1967.8 0/1968 1967.7 1663 2.0 110239 1835.8 0/1836 1835.7 1818 2.2 multistreamwritedirect: 115767 1928.0 0/1928 1927.9 1605 2.1 117717 1960.4 0/1960 1960.3 1586 2.0 119004 1981.8 0/1982 1981.7 1593 2.0 119202 1985.0 0/1985 1984.9 1604 2.0 networkfs: 36661 610.9 90/90 3.8 418 0.8 36570 609.4 90/90 3.8 461 1.5 36661 611.0 90/90 3.7 471 1.5 36661 610.9 90/90 4.0 493 1.8 oltp: 4896210 81526.2 40581/40528 161.5 56 0.0 4975268 82837.2 41228/41184 164.1 57 0.0 5011759 83457.7 41534/41496 165.3 64 0.0 4879904 81254.8 40444/40394 160.9 63 0.0 openfiles: 38398208 639313.7 0/0 0.0 0 0.0 38585907 642441.6 0/0 0.0 0 0.0 38553814 641916.5 0/0 0.0 0 0.0 38279762 632862.3 0/0 0.0 0 0.0 randomfileaccess: 148464 2474.0 550/275 6827.3 6062 6.0 148619 2476.1 550/275 6717.0 5870 6.0 146342 2438.5 542/271 6773.3 5946 6.1 147789 2462.2 547/274 6645.4 5869 6.1 randomread: 12231437 203842.0 203842/0 1592.5 8 0.0 12129079 202136.2 202136/0 1579.2 8 0.0 12139798 202311.5 202311/0 1580.6 8 0.0 12042485 200692.9 200693/0 1567.9 8 0.0 randomrw: 13408366 223447.4 143532/79916 1745.7 11 0.0 13349013 222441.2 142672/79769 1737.8 11 0.0 13452516 224195.3 140706/83489 1751.5 11 0.0 13501252 225007.6 141381/83626 1757.9 11 0.0 randomwrite: 7605046 126741.7 0/126742 990.2 14 0.0 7551132 125841.0 0/125841 983.1 14 0.0 7439628 123978.0 0/123978 968.6 14 0.0 7436235 123927.7 0/123928 968.2 14 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 1396 0.1 6002 60.0 10/10 0.3 1484 2.0 6002 60.0 10/10 0.3 1480 2.2 6002 60.0 10/10 0.3 1639 2.5 removedirs: 10000 9998.9 0/0 0.0 0 0.0 10000 9998.9 0/0 0.0 0 0.0 10000 9989.5 0/0 0.0 0 0.0 10000 9998.7 0/0 0.0 0 0.0 singlestreamread: 228525 3808.5 3808/0 3807.7 264 0.3 226274 3771.0 3771/0 3770.2 266 0.3 214621 3576.8 3577/0 3576.1 267 0.3 226900 3781.3 3781/0 3780.6 266 0.3 singlestreamreaddirect: 229639 3827.0 3827/0 3826.3 265 0.3 226721 3778.4 3778/0 3777.7 268 0.3 9265 154.4 154/0 154.4 1429 6.5 233653 3893.9 3894/0 3893.2 259 0.3 singlestreamwrite: 115217 1918.8 0/1919 1918.8 1293 0.5 114935 1914.2 0/1914 1914.2 1214 0.5 114610 1909.0 0/1909 1909.0 1227 0.5 114902 1913.8 0/1914 1913.8 1224 0.5 singlestreamwritedirect: 114788 1912.3 0/1912 1912.3 1245 0.5 114290 1904.0 0/1904 1904.0 1245 0.5 114793 1912.5 0/1913 1912.5 1243 0.5 114559 1907.8 0/1908 1907.8 1221 0.5 tpcso: varmail: 693897 11561.9 1779/1779 40.6 205 4.5 689928 11494.4 1768/1769 41.2 207 4.5 673338 11219.6 1726/1726 39.6 207 4.6 634956 10579.7 1628/1628 38.1 204 4.9 videoserver: 23194 386.4 386/0 752.1 3453 11.4 23196 386.4 386/0 752.1 3080 9.5 23196 386.5 386/0 752.2 2704 10.8 23127 385.3 385/0 751.9 3242 8.3 webproxy: 9859882 164072.3 43177/8635 409.9 329 1.9 9721985 161675.1 42546/8509 404.3 325 1.9 9411235 156643.3 41222/8244 391.8 336 2.0 10104510 167718.0 44136/8827 418.5 323 1.9 webserver: 37658187 627259.7 202341/20235 3326.1 71 0.2 37583364 626025.0 201943/20195 3318.7 71 0.2 37637365 626841.7 202207/20222 3324.4 71 0.2 37421299 623296.2 201063/20107 3304.4 71 0.2 --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=off, atime=off, sync=disabled, recordsize=128K) --------------------------------------------------------------------------- write 1365.3 1371.1 1382.6 1371.1 MiB/s read 1255.5 1348.5 1365.3 1354.1 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.2 1001/1000 30.0 132 0.4 6002 3000.6 500/500 15.0 185 0.5 6002 6001.4 1001/1000 30.0 142 0.4 6002 6001.4 1001/1000 30.0 122 0.4 copyfiles: 6002 6001.4 1001/1000 30.0 79 0.1 6002 6001.4 1001/1000 30.0 88 0.1 6002 6001.3 1001/1000 30.0 89 0.1 6002 6001.4 1001/1000 30.0 84 0.1 createfiles: 149973 74943.1 0/24980 390.3 326 0.4 149981 74948.3 0/24983 390.3 330 0.4 149973 74943.5 0/24980 390.3 361 0.4 149975 74945.4 0/24980 390.3 330 0.4 deletefiles: 50000 16649.5 0/0 0.0 0 0.0 50000 24974.1 0/0 0.0 0 0.0 50000 16654.6 0/0 0.0 0 0.0 50000 12490.1 0/0 0.0 0 0.0 fileserver: 4965625 82571.6 7506/15013 2001.9 1020 1.7 5646117 93891.6 8536/17072 2277.7 892 1.4 5972102 99338.2 9031/18062 2406.9 853 1.3 5872922 97711.0 8883/17766 2369.5 862 1.4 listdirs: 22488913 374596.1 0/0 77.8 0 0.0 22309621 371594.0 0/0 77.2 0 0.0 22414570 373195.8 0/0 77.5 0 0.0 22531898 375144.2 0/0 77.9 0 0.0 makedirs: 10000 9999.0 0/0 0.0 0 0.0 10000 9999.2 0/0 0.0 0 0.0 10000 9998.9 0/0 0.0 0 0.0 10000 9999.0 0/0 0.0 0 0.0 mongo: 7006 7005.3 1001/1001 23.0 37 0.0 7006 7005.3 1001/1001 23.0 33 0.0 7006 7005.1 1001/1001 23.0 33 0.0 7006 7005.3 1001/1001 23.0 52 0.0 multistreamread: 738502 12306.7 12307/0 12294.7 324 0.3 731856 12196.2 12196/0 12184.3 327 0.3 735136 12251.1 12251/0 12239.1 325 0.3 733919 12230.8 12231/0 12218.8 326 0.3 multistreamreaddirect: 738618 12308.5 12308/0 12296.4 324 0.3 740723 12343.9 12344/0 12331.8 323 0.3 736241 12269.3 12269/0 12257.3 325 0.3 727925 12130.6 12131/0 12118.8 329 0.3 multistreamwrite: 112685 1876.5 0/1876 1876.4 1732 2.1 112311 1870.5 0/1870 1870.4 1623 2.1 111579 1858.4 0/1858 1858.3 1622 2.1 113252 1886.0 0/1886 1886.0 1620 2.1 multistreamwritedirect: 115110 1916.9 0/1917 1916.8 1619 2.1 116043 1932.6 0/1933 1932.5 1615 2.1 116309 1937.1 0/1937 1937.0 1620 2.1 115294 1918.7 0/1919 1918.7 1615 2.1 networkfs: 36661 610.9 90/90 3.8 419 0.8 36089 601.4 89/89 3.7 433 1.0 36533 608.8 90/90 3.6 437 1.1 36661 611.0 90/90 3.6 416 1.0 oltp: 8280742 137881.5 68588/68586 273.2 37 0.0 8551124 142378.2 70831/70817 282.1 31 0.0 8544115 142256.7 70771/70757 281.8 31 0.0 8467425 140985.1 70134/70128 279.3 31 0.0 openfiles: 38342383 638386.8 0/0 0.0 0 0.0 37883642 630747.2 0/0 0.0 0 0.0 37854402 630260.4 0/0 0.0 0 0.0 38121486 634708.4 0/0 0.0 0 0.0 randomfileaccess: 147628 2459.9 547/273 6717.7 6056 6.1 149824 2496.6 555/277 6925.3 5950 6.0 149356 2488.9 553/277 6956.8 5973 6.0 150571 2509.1 558/279 7007.2 5928 6.0 randomread: 12026738 200430.7 200431/0 1565.8 8 0.0 12047022 200765.3 200765/0 1568.5 8 0.0 12084340 201390.8 201391/0 1573.4 8 0.0 12113364 201874.5 201874/0 1577.1 8 0.0 randomrw: 13638384 227292.9 141097/86196 1775.7 11 0.0 13790966 229827.5 142876/86952 1795.5 11 0.0 13605514 226725.5 141457/85269 1771.3 11 0.0 13546316 225754.3 137086/88668 1763.7 11 0.0 randomwrite: 7865634 131083.7 0/131084 1024.1 13 0.0 7924632 132063.1 0/132063 1031.7 13 0.0 7819938 130322.6 0/130323 1018.1 13 0.0 7895099 131560.1 0/131560 1027.8 13 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 1619 1.6 6002 60.0 10/10 0.3 1589 1.8 6002 60.0 10/10 0.3 1604 2.3 6002 60.0 10/10 0.3 1564 2.2 removedirs: 10000 9989.4 0/0 0.0 0 0.0 10000 9998.7 0/0 0.0 0 0.0 10000 9998.8 0/0 0.0 0 0.0 10000 9989.3 0/0 0.0 0 0.0 singlestreamread: 223690 3727.8 3728/0 3727.1 272 0.3 223759 3729.0 3729/0 3728.3 271 0.3 227994 3799.6 3800/0 3798.8 266 0.3 224883 3747.7 3748/0 3747.0 272 0.3 singlestreamreaddirect: 162012 2699.9 2700/0 2699.4 294 0.4 233574 3892.6 3893/0 3891.8 260 0.3 226081 3767.7 3768/0 3766.9 270 0.3 223099 3718.0 3718/0 3717.2 272 0.3 singlestreamwrite: 111134 1851.3 0/1851 1851.3 1295 0.5 109809 1828.6 0/1829 1828.6 1285 0.5 109920 1830.5 0/1830 1830.5 1303 0.5 110159 1834.8 0/1835 1834.8 1271 0.5 singlestreamwritedirect: 110621 1842.3 0/1842 1842.3 1265 0.5 111154 1851.1 0/1851 1851.1 1266 0.5 111302 1853.9 0/1854 1853.9 1272 0.5 114082 1899.8 0/1900 1899.7 1231 0.5 tpcso: varmail: 15918617 265038.1 40775/40775 956.4 158 0.2 15678177 261034.9 40159/40159 942.0 150 0.2 15007406 249870.1 38441/38442 902.1 167 0.2 16625693 276810.5 42586/42586 998.7 145 0.2 videoserver: 23198 386.5 386/0 1028.2 3630 12.9 23085 384.6 384/0 1027.7 3519 13.8 23160 385.8 386/0 1028.1 3551 11.9 23143 385.6 385/0 752.0 3331 11.0 webproxy: 7304178 121619.3 32005/6401 305.0 388 2.6 7424256 123639.1 32536/6507 309.5 468 2.5 9916864 165012.8 43425/8685 412.0 338 1.9 8000659 133213.0 35056/7011 333.7 427 2.4 webserver: 38195490 636093.6 205191/20520 3371.9 70 0.2 38124199 634975.0 204830/20484 3367.8 70 0.2 38101226 634573.9 204701/20471 3364.1 70 0.2 37251188 620446.6 200144/20015 3289.6 72 0.2 --------------------------------------------------------------------------- 12x 2-way mirrors ZFS (compression=on, atime=off, sync=disabled, recordsize=128K) --------------------------------------------------------------------------- write 2340.6 2445.4 2501.4 2391.8 MiB/s read 1638.4 1638.4 1638.4 1638.4 MiB/s filebench 1.4.8 (0.175.1.0.0.24.2) as is with 'set $dir=/pool1/test': ops ops/s r/w mb/s cpu/op latency [µs] [ms] bringover: 6002 6001.3 1001/1000 30.0 183 0.3 6002 6001.4 1001/1000 30.0 162 0.3 6002 6001.4 1001/1000 30.0 198 0.3 6002 6001.4 1001/1000 30.0 142 0.2 copyfiles: 6002 6001.5 1001/1000 30.0 189 0.3 6002 6001.5 1001/1000 30.0 132 0.3 6002 6001.5 1001/1000 30.0 151 0.3 6002 6001.4 1001/1000 30.0 192 0.3 createfiles: 149979 74947.4 0/24983 390.3 339 0.4 149979 74946.6 0/24983 390.3 348 0.4 149982 74913.3 0/24972 390.1 481 0.5 149973 74943.3 0/24979 390.3 339 0.4 deletefiles: 50000 24985.9 0/0 0.0 0 0.0 50000 16654.6 0/0 0.0 0 0.0 50000 12490.1 0/0 0.0 0 0.0 50000 16649.4 0/0 0.0 0 0.0 fileserver: 6105785 101507.9 9228/18456 2462.6 830 1.3 6832299 113538.5 10322/20644 2754.3 736 1.1 5346122 88946.9 8086/16173 2154.3 957 1.5 5123812 85213.1 7747/15494 2066.8 1000 1.6 listdirs: 22220533 369959.3 0/0 76.8 0 0.0 22402799 373000.3 0/0 77.5 0 0.0 21665865 360870.1 0/0 75.0 0 0.0 22287621 371076.0 0/0 77.1 0 0.0 makedirs: 10000 9989.3 0/0 0.0 0 0.0 10000 9999.0 0/0 0.0 0 0.0 10000 9998.8 0/0 0.0 0 0.0 10000 9989.2 0/0 0.0 0 0.0 mongo: 7006 7005.3 1001/1001 23.0 84 0.2 7006 7005.1 1001/1001 23.0 51 0.0 7006 7005.4 1001/1001 23.0 90 0.1 7006 7005.2 1001/1001 23.0 94 0.2 multistreamread: 842287 14036.6 14037/0 14022.9 284 0.3 837557 13957.8 13958/0 13944.2 286 0.3 843400 14055.0 14055/0 14041.3 284 0.3 838436 13972.5 13972/0 13958.8 285 0.3 multistreamreaddirect: 831533 13857.5 13857/0 13843.9 288 0.3 835469 13923.1 13923/0 13909.5 286 0.3 834066 13899.6 13900/0 13886.0 287 0.3 822891 13713.7 13714/0 13700.3 291 0.3 multistreamwrite: 351433 5856.2 0/5856 5856.1 2209 0.7 353519 5891.0 0/5891 5890.9 2176 0.7 351476 5857.1 0/5857 5857.1 2187 0.7 351176 5852.3 0/5852 5852.2 2195 0.7 multistreamwritedirect: 356102 5932.3 0/5932 5932.3 2171 0.7 355271 5920.5 0/5920 5920.4 2187 0.7 353626 5891.5 0/5891 5891.4 2183 0.7 354218 5903.0 0/5903 5903.0 2190 0.7 networkfs: 36218 603.6 89/89 3.6 443 0.6 36661 611.0 90/90 3.6 430 0.6 36661 610.9 90/90 3.5 411 0.6 36661 610.9 90/90 3.7 424 0.6 oltp: 8316122 138461.1 68883/68868 274.3 36 0.0 8596532 143132.3 71199/71199 283.6 31 0.0 8350206 139033.3 69169/69152 275.4 35 0.0 8271183 137714.2 68510/68498 272.8 35 0.0 openfiles: 38581415 642364.0 0/0 0.0 0 0.0 38201353 636038.3 0/0 0.0 0 0.0 38928315 644548.4 0/0 0.0 0 0.0 38398659 638527.2 0/0 0.0 0 0.0 randomfileaccess: 146438 2440.2 542/271 6720.8 6096 6.1 148391 2472.8 549/275 6862.0 6042 6.0 138160 2302.1 512/256 6188.2 6487 6.5 148726 2478.1 551/275 6953.5 6023 6.0 randomread: 12026008 200418.6 200419/0 1565.8 8 0.0 12269438 204475.1 204475/0 1597.4 8 0.0 12386370 206424.0 206424/0 1612.7 8 0.0 12162148 202684.7 202685/0 1583.5 8 0.0 randomrw: 13504522 225053.0 140799/84254 1758.2 11 0.0 13703197 228361.6 137721/90641 1784.1 11 0.0 13974484 230758.1 143246/87512 1802.8 11 0.0 13627783 227104.2 140319/86786 1774.3 11 0.0 randomwrite: 7829596 130483.9 0/130484 1019.4 13 0.0 8066271 134428.4 0/134428 1050.2 13 0.0 7962913 132705.3 0/132705 1036.8 13 0.0 7844003 130711.7 0/130712 1021.2 13 0.0 ratelimcopyfiles: 6002 60.0 10/10 0.3 1369 0.6 6002 60.0 10/10 0.3 1355 0.6 6002 60.0 10/10 0.3 1305 0.5 6002 60.0 10/10 0.3 1395 0.6 removedirs: 10000 9988.9 0/0 0.0 0 0.0 10000 9998.7 0/0 0.0 0 0.0 10000 9999.3 0/0 0.0 0 0.0 10000 9998.7 0/0 0.0 0 0.0 singlestreamread: 252723 4211.7 4212/0 4210.8 242 0.2 256071 4267.4 4267/0 4266.5 237 0.2 266186 4436.0 4436/0 4435.2 228 0.2 249843 4163.8 4164/0 4163.0 245 0.2 singlestreamreaddirect: 256170 4269.2 4269/0 4268.3 237 0.2 267613 4459.8 4460/0 4459.0 226 0.2 260032 4333.6 4334/0 4332.7 232 0.2 257781 4296.0 4296/0 4295.1 235 0.2 singlestreamwrite: 160341 2671.6 0/2672 2671.6 1663 0.4 162488 2707.3 0/2707 2707.3 1659 0.4 162097 2700.9 0/2701 2700.8 1682 0.4 162055 2700.6 0/2701 2700.6 1666 0.4 singlestreamwritedirect: 161225 2686.3 0/2686 2686.3 1651 0.4 162488 2707.8 0/2708 2707.8 1639 0.4 159945 2665.3 0/2665 2665.3 1604 0.4 161062 2683.9 0/2684 2683.8 1661 0.4 tpcso: varmail: 11170259 185988.8 28614/28614 671.2 245 0.3 14562127 242457.1 37301/37301 874.0 178 0.2 13380121 222776.5 34273/34273 803.2 194 0.2 14265941 237521.8 36542/36542 855.5 182 0.2 videoserver: 23160 386.0 386/0 1028.4 4001 1.1 23202 386.6 386/0 1028.5 3972 1.1 23202 386.6 386/0 1028.5 3965 1.1 23164 386.0 386/0 1028.4 4027 1.1 webproxy: 7938918 132027.0 34744/6949 330.1 362 2.4 9357386 155820.3 41005/8201 389.4 350 2.0 9257292 153921.1 40506/8101 384.8 353 2.0 7907454 131573.1 34624/6925 330.0 430 2.4 webserver: 37589864 625828.5 201880/20189 3318.3 71 0.2 37543287 625294.6 201707/20172 3314.7 71 0.2 32342485 538776.4 173798/17381 2856.7 106 0.3 37393514 622785.4 200898/20091 3302.2 71 0.2