Continuation of: ZFS Raid-Z3 Performance with Zstandard (ZSTD) - Part 1 - Benchmark Background Information
Read & Random Read Results:
Note: This first set of results are with 16GB ARC
The command used for the tests was:
fio --name=test --numjobs=1 --allrandrepeat=1 --ioengine=libaio --filesize=80G --iodepth=64 --direct=1 --buffered=0 --time_based --runtime=300 --directory=VOLUME_MOUNTPOINT
| bs=4k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 18.9k | 73.7MiB/s | 906 | 3625KiB/s |
| 16k | 34.6k | 135MiB/s | 735 | 2942KiB/s |
| 32k | 43.0k | 168MiB/s | 408 | 1632KiB/s |
| 64k | 45.0k | 180MiB/s | 380 | 1524KiB/s |
| 128k | 46.9k | 183MiB/s | 360 | 1442KiB/s |
| 256k | 47.9k | 187MiB/s | 320 | 1283KiB/s |
| 512k | 43.7k | 171MiB/s | 296 | 1187KiB/s |
| 1024k | 42.7k | 167MiB/s | 274 | 1097KiB/s |
| bs=16k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 2679.0 | 41.9MiB/s | 445 | 7127KiB/s |
| 16k | 3101.0 | 48.5MiB/s | 815 | 12.7MiB/s |
| 32k | 3767.0 | 58.9MiB/s | 796 | 12.5MiB/s |
| 64k | 8429.0 | 132MiB/s | 683 | 10.7MiB/s |
| 128k | 8729.0 | 136MiB/s | 590 | 9455KiB/s |
| 256k | 11.0k | 173MiB/s | 427 | 6846KiB/s |
| 512k | 8746.0 | 137MiB/s | 364 | 5831KiB/s |
| 1024k | 9861.0 | 154MiB/s | 293 | 4698KiB/s |
| bs=64k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 901.0 | 56.3MiB/s | 236 | 14.8MiB/s |
| 16k | 965.0 | 60.4MiB/s | 234 | 14.6MiB/s |
| 32k | 973.0 | 60.8MiB/s | 224 | 14.0MiB/s |
| 64k | 1229.0 | 76.8MiB/s | 320 | 20.0MiB/s |
| 128k | 1864.0 | 117MiB/s | 505 | 31.6MiB/s |
| 256k | 3072.0 | 192MiB/s | 436 | 27.3MiB/s |
| 512k | 3013.0 | 188MiB/s | 368 | 23.0MiB/s |
| 1024k | 3172.0 | 198MiB/s | 281 | 17.6MiB/s |
As can be seen clearly, random read performance peaks when bs value is same as volblocksize. Although the performance peaked at 128k volblocksize for the 64k bs value.
Clearly the sequential read is performing better with larger volblocksize settings.
So, depending on your workload read size and read requirements you may want to select a volblocksize as close as the demand requirements. If your only concern is raw read speed, then it is best to use the larger volblocksize.
Write & Random Write Results (with and without fsync):
The same command as in read tests was used with --rw option having write and randwrite setting. Also the --fsync setting was usedWrite:
| bs=4k | fsync=0 | fsync=1 | ||
| volblocksize | Write IOPS | Write BW | Write IOPS | Write BW |
| 8k | 10.5k | 41.2MiB/s | 65 | 260KiB/s |
| 16k | 14.3k | 55.0MiB/s | 34 | 136KiB/s |
| 32k | 14.1k | 55.3MiB/s | 39 | 156KiB/s |
| 64k | 12.0k | 47.0MiB/s | 44 | 177KiB/s |
| 128k | 10.6k | 41.6MiB/s | 45 | 180KiB/s |
| 256k | 13.2k | 51.4MiB/s | 45 | 182KiB/s |
| 512k | 3901 | 15.2MiB/s | 64 | 258KiB/s |
| 1024k | 5633 | 22.0MiB/s | 121 | 486KiB/s |
| bs=16k | fsync=0 | fsync=1 | ||
| volblocksize | Write IOPS | Write BW | Write IOPS | Write BW |
| 8k | 3494 | 54.6MiB/s | 51 | 828KiB/s |
| 16k | 5216 | 81.5MiB/s | 35 | 561KiB/s |
| 32k | 2020 | 31.6MiB/s | 55 | 889KiB/s |
| 64k | 1563 | 24.4MiB/s | 43 | 689KiB/s |
| 128k | 1690 | 26.4MiB/s | 42 | 682KiB/s |
| 256k | 1568 | 24.5MiB/s | 43 | 689KiB/s |
| 512k | 1058 | 16.5MiB/s | 40 | 642KiB/s |
| 1024k | 1441 | 22.5MiB/s | 42 | 675KiB/s |
| bs=64k | fsync=0 | fsync=1 | ||
| volblocksize | Write IOPS | Write BW | Write IOPS | Write BW |
| 8k | 1020 | 63.8MiB/s | 59 | 3823KiB/s |
| 16k | 1441 | 90.1MiB/s | 36 | 2344KiB/s |
| 32k | 1834 | 115MiB/s | 37 | 2394KiB/s |
| 64k | 4019 | 251MiB/s | 43 | 2784KiB/s |
| 128k | 1006 | 62.9MiB/s | 49 | 3188KiB/s |
| 256k | 996 | 62.3MiB/s | 49 | 3160KiB/s |
| 512k | 902 | 56.4MiB/s | 46 | 2997KiB/s |
| 1024k | 854 | 53.4MiB/s | 49 | 3199KiB/s |
Random Write:
| bs=4k | fsync=0 | fsync=1 | ||
| volblocksize | RandWrite IOPS | RandWrite BW | RandWrite IOPS | RandWrite BW |
| 8k | 1593 | 6372KiB/s | 47 | 188KiB/s |
| 16k | 1329 | 5318KiB/s | 40 | 162KiB/s |
| 32k | 753 | 3014KiB/s | 50 | 200KiB/s |
| 64k | 650 | 2600KiB/s | 43 | 175KiB/s |
| 128k | 582 | 2332KiB/s | 58 | 235KiB/s |
| 256k | 261 | 1048KiB/s | 59 | 237KiB/s |
| 512k | 233 | 934KiB/s | 70 | 281KiB/s |
| 1024k | 181 | 726KiB/s | 108 | 435KiB/s |
| bs=16k | fsync=0 | fsync=1 | ||
| volblocksize | RandWrite IOPS | RandWrite BW | RandWrite IOPS | RandWrite BW |
| 8k | 4540 | 70.9MiB/s | 91 | 1458KiB/s |
| 16k | 6510 | 102MiB/s | 36 | 581KiB/s |
| 32k | 1013 | 15.8MiB/s | 37 | 600KiB/s |
| 64k | 865 | 13.5MiB/s | 59 | 946KiB/s |
| 128k | 416 | 6667KiB/s | 47 | 757KiB/s |
| 256k | 353 | 5661KiB/s | 63 | 1020KiB/s |
| 512k | 291 | 4671KiB/s | 80 | 1296KiB/s |
| 1024k | 204 | 3273KiB/s | 80 | 1288KiB/s |
| bs=64k | fsync=0 | fsync=1 | ||
| volblocksize | RandWrite IOPS | RandWrite BW | RandWrite IOPS | RandWrite BW |
| 8k | 1330 | 83.2MiB/s | 55 | 3531KiB/s |
| 16k | 1975 | 123MiB/s | 41 | 2672KiB/s |
| 32k | 2065 | 129MiB/s | 39 | 2524KiB/s |
| 64k | 4228 | 264MiB/s | 40 | 2594KiB/s |
| 128k | 513 | 32.1MiB/s | 48 | 3073KiB/s |
| 256k | 358 | 22.4MiB/s | 54 | 3504KiB/s |
| 512k | 286 | 17.9MiB/s | 67 | 4343KiB/s |
| 1024k | 209 | 13.1MiB/s | 65 | 4213KiB/s |
Read & Random Read Results With & Without Cache:
The cache was occupied little over 80GB in size so presumably it had all the data cached. The used command for cached reads was:
fio --name=test --loops=8 --numjobs=1 --allrandrepeat=1 --ioengine=libaio --filesize=10G --iodepth=64 --direct=1 --buffered=0 --directory=VOLUME_MOUNTPOINT
and for uncached reads there was no reason to loop multiple times:
fio --name=test --numjobs=1 --allrandrepeat=1 --ioengine=libaio --filesize=10G --iodepth=64 --direct=1 --buffered=0 --directory=VOLUME_MOUNTPOINT
The results are with cache:
| Cache: MK0100GCTYU | ||||
| bs=4k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 34.1k | 133MiB/s | 34.2k | 134MiB/s |
| 16k | 40.0k | 160MiB/s | 5509 | 21.5MiB/s |
| 32k | 45.5k | 178MiB/s | 5215 | 20.4MiB/s |
| 64k | 49.9k | 195MiB/s | 5423 | 21.2MiB/s |
| 128k | 55.9k | 218MiB/s | 3624 | 14.2MiB/s |
| 256k | 53.7k | 210MiB/s | 2020 | 8081KiB/s |
| 512k | 52.5k | 205MiB/s | 1043 | 4174KiB/s |
| 1024k | 47.7k | 186MiB/s | 529 | 2118KiB/s |
| bs=16k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 16.1k | 251MiB/s | 19.3k | 301MiB/s |
| 16k | 21.9k | 342MiB/s | 22.4k | 350MiB/s |
| 32k | 23.2k | 362MiB/s | 13.4k | 209MiB/s |
| 64k | 26.3k | 410MiB/s | 7077 | 111MiB/s |
| 128k | 30.8k | 481MiB/s | 3950 | 61.7MiB/s |
| 256k | 26.4k | 413MiB/s | 2083 | 32.6MiB/s |
| 512k | 26.5k | 414MiB/s | 1070 | 16.7MiB/s |
| 1024k | 26.0k | 421MiB/s | 539 | 8633KiB/s |
| bs=64k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 3606 | 225MiB/s | 4704 | 294MiB/s |
| 16k | 5046 | 315MiB/s | 5482 | 343MiB/s |
| 32k | 5666 | 354MiB/s | 5833 | 365MiB/s |
| 64k | 6064 | 379MiB/s | 6092 | 381MiB/s |
| 128k | 7101 | 444MiB/s | 3705 | 232MiB/s |
| 256k | 8264 | 517MiB/s | 2079 | 130MiB/s |
| 512k | 8752 | 547MiB/s | 1082 | 67.6MiB/s |
| 1024k | 7878 | 492MiB/s | 550 | 34.4MiB/s |
Without Cache:
| No Cache | ||||
| bs=4k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 34.5k | 135MiB/s | 791 | 3167KiB/s |
| 16k | 42.3k | 165MiB/s | 796 | 3187KiB/s |
| 32k | 46.1k | 180MiB/s | 476 | 1906KiB/s |
| 64k | 50.6k | 198MiB/s | 445 | 1781KiB/s |
| 128k | 61.7k | 241MiB/s | 412 | 1650KiB/s |
| 256k | 54.7k | 214MiB/s | 501 | 2006KiB/s |
| 512k | 54.7k | 214MiB/s | 375 | 1501KiB/s |
| 1024k | 51.2k | 200MiB/s | 293 | 1173KiB/s |
| bs=16k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 12.7k | 199MiB/s | 679 | 10.6MiB/s |
| 16k | 15.0k | 250MiB/s | 717 | 11.2MiB/s |
| 32k | 19.2k | 300MiB/s | 440 | 7042KiB/s |
| 64k | 24.9k | 389MiB/s | 416 | 6657KiB/s |
| 128k | 30.0k | 484MiB/s | 392 | 6281KiB/s |
| 256k | 26.6k | 416MiB/s | 485 | 7763KiB/s |
| 512k | 27.3k | 427MiB/s | 365 | 5842KiB/s |
| 1024k | 26.4k | 412MiB/s | 293 | 4703KiB/s |
| bs=64k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 3080 | 193MiB/s | 499 | 31.2MiB/s |
| 16k | 4395 | 275MiB/s | 572 | 35.8MiB/s |
| 32k | 4847 | 303MiB/s | 397 | 24.8MiB/s |
| 64k | 6568 | 411MiB/s | 394 | 24.6MiB/s |
| 128k | 8692 | 543MiB/s | 386 | 24.2MiB/s |
| 256k | 8110 | 507MiB/s | 495 | 30.0MiB/s |
| 512k | 8773 | 548MiB/s | 367 | 22.9MiB/s |
| 1024k | 7680 | 480MiB/s | 294 | 18.4MiB/s |
Write & Random Write Results (with and without fsync) + SLOG:
Same command as the previous write tests was used with and without fsync and with the addition of SLOG device.
Write:
| SLOG: INTEL SSDSC2KG480G8 | ||||
| bs=4k | fsync=0 | fsync=1 | ||
| volblocksize | Write IOPS | Write BW | Write IOPS | Write BW |
| 8k | 9343 | 36.5MiB/s | 686 | 2747KiB/s |
| 16k | 14.5k | 56.7MiB/s | 668 | 2674KiB/s |
| 32k | 9775 | 38.2MiB/s | 873 | 3493KiB/s |
| 64k | 8245 | 32.2MiB/s | 1027 | 4108KiB/s |
| 128k | 11.8k | 46.1MiB/s | 1087 | 4349KiB/s |
| 256k | 12.3k | 48.1MiB/s | 1175 | 4703KiB/s |
| 512k | 13.8k | 53.8MiB/s | 1085 | 4344KiB/s |
| 1024k | 16.6k | 64.0MiB/s | 1230 | 4923KiB/s |
| bs=16k | fsync=0 | fsync=1 | ||
| volblocksize | Write IOPS | Write BW | Write IOPS | Write BW |
| 8k | 6007 | 93.9MiB/s | 452 | 7236KiB/s |
| 16k | 9875 | 154MiB/s | 681 | 10.6MiB/s |
| 32k | 4804 | 75.1MiB/s | 564 | 9030KiB/s |
| 64k | 3691 | 57.7MiB/s | 730 | 11.4MiB/s |
| 128k | 5233 | 81.8MiB/s | 755 | 11.8MiB/s |
| 256k | 5479 | 85.6MiB/s | 653 | 10.2MiB/s |
| 512k | 4888 | 76.4MiB/s | 669 | 10.5MiB/s |
| 1024k | 5894 | 92.1MiB/s | 542 | 8677KiB/s |
| bs=64k | fsync=0 | fsync=1 | ||
| volblocksize | Write IOPS | Write BW | Write IOPS | Write BW |
| 8k | 1765 | 110MiB/s | 456 | 28.6MiB/s |
| 16k | 2530 | 158MiB/s | 499 | 31.2MiB/s |
| 32k | 2815 | 176MiB/s | 594 | 37.2MiB/s |
| 64k | 5298 | 331MiB/s | 214 | 13.4MiB/s |
| 128k | 1555 | 97.2MiB/s | 538 | 33.6MiB/s |
| 256k | 1819 | 114MiB/s | 634 | 39.7MiB/s |
| 512k | 1751 | 109MiB/s | 666 | 41.6MiB/s |
| 1024k | 1915 | 120MiB/s | 601 | 37.6MiB/s |
Random Write:
| SLOG: INTEL SSDSC2KG480G8 | ||||
| bs=4k | fsync=0 | fsync=1 | ||
| volblocksize | RandWrite IOPS | RandWrite BW | RandWrite IOPS | RandWrite BW |
| 8k | 524 | 2098KiB/s | 719 | 2879KiB/s |
| 16k | 785 | 3140KiB/s | 721 | 2885KiB/s |
| 32k | 502 | 2011KiB/s | 713 | 2853KiB/s |
| 64k | 662 | 2651KiB/s | 826 | 3305KiB/s |
| 128k | 546 | 2185KiB/s | 593 | 2372KiB/s |
| 256k | 417 | 1671KiB/s | 469 | 1877KiB/s |
| 512k | 330 | 1321KiB/s | 343 | 1373KiB/s |
| 1024k | 271 | 1085KiB/s | 265 | 1063KiB/s |
| bs=16k | fsync=0 | fsync=1 | ||
| volblocksize | RandWrite IOPS | RandWrite BW | RandWrite IOPS | RandWrite BW |
| 8k | 5198 | 81.2MiB/s | 454 | 7272KiB/s |
| 16k | 9380 | 147MiB/s | 440 | 7042KiB/s |
| 32k | 958 | 14.0MiB/s | 501 | 8020KiB/s |
| 64k | 995 | 15.6MiB/s | 643 | 10.1MiB/s |
| 128k | 733 | 11.5MiB/s | 537 | 8600KiB/s |
| 256k | 553 | 8858KiB/s | 505 | 8081KiB/s |
| 512k | 368 | 5900KiB/s | 343 | 5504KiB/s |
| 1024k | 260 | 4168KiB/s | 248 | 3972KiB/s |
| bs=64k | fsync=0 | fsync=1 | ||
| volblocksize | RandWrite IOPS | RandWrite BW | RandWrite IOPS | RandWrite BW |
| 8k | 1531 | 95.7MiB/s | 378 | 23.6MiB/s |
| 16k | 2545 | 159MiB/s | 439 | 27.5MiB/s |
| 32k | 2577 | 161MiB/s | 453 | 28.4MiB/s |
| 64k | 4998 | 312MiB/s | 205 | 12.8MiB/s |
| 128k | 743 | 46.5MiB/s | 444 | 27.8MiB/s |
| 256k | 527 | 32.0MiB/s | 377 | 23.6MiB/s |
| 512k | 387 | 24.2MiB/s | 331 | 20.7MiB/s |
| 1024k | 270 | 16.9MiB/s | 238 | 14.9MiB/s |
Read Results With 1GB ARC + With & Without 100GB L2ARC
| No Cache | ||||
| bs=4k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 34.5k | 135MiB/s | 791 | 3167KiB/s |
| 16k | 42.3k | 165MiB/s | 796 | 3187KiB/s |
| 32k | 46.1k | 180MiB/s | 476 | 1906KiB/s |
| 64k | 50.6k | 198MiB/s | 445 | 1781KiB/s |
| 128k | 61.7k | 241MiB/s | 412 | =1650KiB/s |
| 256k | 54.7k | 214MiB/s | 501 | 2006KiB/s |
| 512k | 54.7k | 214MiB/s | 375 | 1501KiB/s |
| 1024k | 51.2k | 200MiB/s | 293 | 1173KiB/s |
| bs=16k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 12.7k | 199MiB/s | 679 | 10.6MiB/s |
| 16k | 15.0k | 250MiB/s | 717 | 11.2MiB/s |
| 32k | 19.2k | 300MiB/s | 440 | 7042KiB/s |
| 64k | 24.9k | 389MiB/s | 416 | 6657KiB/s |
| 128k | 30.0k | 484MiB/s | 392 | 6281KiB/s |
| 256k | 26.6k | 416MiB/s | 485 | 7763KiB/s |
| 512k | 27.3k | 427MiB/s | 365 | 5842KiB/s |
| 1024k | 26.4k | 412MiB/s | 293 | 4703KiB/s |
| bs=64k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 3080 | 193MiB/s | 499 | 31.2MiB/s |
| 16k | 4395 | 275MiB/s | 572 | 35.8MiB/s |
| 32k | 4847 | 303MiB/s | 397 | 24.8MiB/s |
| 64k | 6568 | 411MiB/s | 394 | 24.6MiB/s |
| 128k | 8692 | 543MiB/s | 386 | 24.2MiB/s |
| 256k | 8110 | 507MiB/s | 495 | 30.0MiB/s |
| 512k | 8773 | 548MiB/s | 367 | 22.9MiB/s |
| 1024k | 7680 | 480MiB/s | 294 | 18.4MiB/s |
| Cache: MK0100GCTYU | ||||
| bs=4k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 34.1k | 133MiB/s | 34.2k | 134MiB/s |
| 16k | 40.0k | 160MiB/s | 5509 | 21.5MiB/s |
| 32k | 45.5k | 178MiB/s | 5215 | 20.4MiB/s |
| 64k | 49.9k | 195MiB/s | 5423 | 21.2MiB/s |
| 128k | 55.9k | 218MiB/s | 3624 | 14.2MiB/s |
| 256k | 53.7k | 210MiB/s | 2020 | 8081KiB/s |
| 512k | 52.5k | 205MiB/s | 1043 | 4174KiB/s |
| 1024k | 47.7k | 186MiB/s | 529 | 2118KiB/s |
| Cache: MK0100GCTYU | ||||
| bs=16k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 16.1k | 251MiB/s | 19.3k | 301MiB/s |
| 16k | 21.9k | 342MiB/s | 22.4k | 350MiB/s |
| 32k | 23.2k | 362MiB/s | 13.4k | 209MiB/s |
| 64k | 26.3k | 410MiB/s | 7077 | 111MiB/s |
| 128k | 30.8k | 481MiB/s | 3950 | 61.7MiB/s |
| 256k | 26.4k | 413MiB/s | 2083 | 32.6MiB/s |
| 512k | 26.5k | 414MiB/s | 1070 | 16.7MiB/s |
| 1024k | 26.0k | 421MiB/s | 539 | 8633KiB/s |
| Cache: MK0100GCTYU | ||||
| bs=64k | ||||
| volblocksize | Read IOPS | Read BW | RandRead IOPS | RandRead BW |
| 8k | 3606 | 225MiB/s | 4704 | 294MiB/s |
| 16k | 5046 | 315MiB/s | 5482 | 343MiB/s |
| 32k | 5666 | 354MiB/s | 5833 | 365MiB/s |
| 64k | 6064 | 379MiB/s | 6092 | 381MiB/s |
| 128k | 7101 | 444MiB/s | 3705 | 232MiB/s |
| 256k | 8264 | 517MiB/s | 2079 | 130MiB/s |
| 512k | 8752 | 547MiB/s | 1082 | 67.6MiB/s |
| 1024k | 7878 | 492MiB/s | 550 | 34.4MiB/s |
No comments:
Post a Comment