ZFS Raid-Z3 Performance with Zstandard (ZSTD) - Part 2 - In-compressible Data Benchmarks

Continuation of: ZFS Raid-Z3 Performance with Zstandard (ZSTD) - Part 1 - Benchmark Background Information

 

Read & Random Read Results:

Note: This first set of results are with 16GB ARC 

 

The command used for the tests was:

fio --name=test --numjobs=1 --allrandrepeat=1 --ioengine=libaio --filesize=80G --iodepth=64 --direct=1 --buffered=0 --time_based --runtime=300 --directory=VOLUME_MOUNTPOINT

bs=4k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	18.9k	73.7MiB/s	906	3625KiB/s
16k	34.6k	135MiB/s	735	2942KiB/s
32k	43.0k	168MiB/s	408	1632KiB/s
64k	45.0k	180MiB/s	380	1524KiB/s
128k	46.9k	183MiB/s	360	1442KiB/s
256k	47.9k	187MiB/s	320	1283KiB/s
512k	43.7k	171MiB/s	296	1187KiB/s
1024k	42.7k	167MiB/s	274	1097KiB/s
bs=16k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	2679.0	41.9MiB/s	445	7127KiB/s
16k	3101.0	48.5MiB/s	815	12.7MiB/s
32k	3767.0	58.9MiB/s	796	12.5MiB/s
64k	8429.0	132MiB/s	683	10.7MiB/s
128k	8729.0	136MiB/s	590	9455KiB/s
256k	11.0k	173MiB/s	427	6846KiB/s
512k	8746.0	137MiB/s	364	5831KiB/s
1024k	9861.0	154MiB/s	293	4698KiB/s
bs=64k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	901.0	56.3MiB/s	236	14.8MiB/s
16k	965.0	60.4MiB/s	234	14.6MiB/s
32k	973.0	60.8MiB/s	224	14.0MiB/s
64k	1229.0	76.8MiB/s	320	20.0MiB/s
128k	1864.0	117MiB/s	505	31.6MiB/s
256k	3072.0	192MiB/s	436	27.3MiB/s
512k	3013.0	188MiB/s	368	23.0MiB/s
1024k	3172.0	198MiB/s	281	17.6MiB/s

As can be seen clearly, random read performance peaks when bs value is same as volblocksize. Although the performance peaked at 128k volblocksize for the 64k bs value.

Clearly the sequential read is performing better with larger volblocksize settings.

 

So, depending on your workload read size and read requirements you may want to select a volblocksize as close as the demand requirements. If your only concern is raw read speed, then it is best to use the larger volblocksize.

Write & Random Write Results (with and without fsync):

The same command as in read tests was used with --rw option having write and randwrite setting. Also the --fsync setting was used

Write:

bs=4k	fsync=0		fsync=1
volblocksize	Write IOPS	Write BW	Write IOPS	Write BW
8k	10.5k	41.2MiB/s	65	260KiB/s
16k	14.3k	55.0MiB/s	34	136KiB/s
32k	14.1k	55.3MiB/s	39	156KiB/s
64k	12.0k	47.0MiB/s	44	177KiB/s
128k	10.6k	41.6MiB/s	45	180KiB/s
256k	13.2k	51.4MiB/s	45	182KiB/s
512k	3901	15.2MiB/s	64	258KiB/s
1024k	5633	22.0MiB/s	121	486KiB/s
bs=16k	fsync=0		fsync=1
volblocksize	Write IOPS	Write BW	Write IOPS	Write BW
8k	3494	54.6MiB/s	51	828KiB/s
16k	5216	81.5MiB/s	35	561KiB/s
32k	2020	31.6MiB/s	55	889KiB/s
64k	1563	24.4MiB/s	43	689KiB/s
128k	1690	26.4MiB/s	42	682KiB/s
256k	1568	24.5MiB/s	43	689KiB/s
512k	1058	16.5MiB/s	40	642KiB/s
1024k	1441	22.5MiB/s	42	675KiB/s
bs=64k	fsync=0		fsync=1
volblocksize	Write IOPS	Write BW	Write IOPS	Write BW
8k	1020	63.8MiB/s	59	3823KiB/s
16k	1441	90.1MiB/s	36	2344KiB/s
32k	1834	115MiB/s	37	2394KiB/s
64k	4019	251MiB/s	43	2784KiB/s
128k	1006	62.9MiB/s	49	3188KiB/s
256k	996	62.3MiB/s	49	3160KiB/s
512k	902	56.4MiB/s	46	2997KiB/s
1024k	854	53.4MiB/s	49	3199KiB/s

Random Write:

bs=4k	fsync=0		fsync=1
volblocksize	RandWrite IOPS	RandWrite BW	RandWrite IOPS	RandWrite BW
8k	1593	6372KiB/s	47	188KiB/s
16k	1329	5318KiB/s	40	162KiB/s
32k	753	3014KiB/s	50	200KiB/s
64k	650	2600KiB/s	43	175KiB/s
128k	582	2332KiB/s	58	235KiB/s
256k	261	1048KiB/s	59	237KiB/s
512k	233	934KiB/s	70	281KiB/s
1024k	181	726KiB/s	108	435KiB/s
bs=16k	fsync=0		fsync=1
volblocksize	RandWrite IOPS	RandWrite BW	RandWrite IOPS	RandWrite BW
8k	4540	70.9MiB/s	91	1458KiB/s
16k	6510	102MiB/s	36	581KiB/s
32k	1013	15.8MiB/s	37	600KiB/s
64k	865	13.5MiB/s	59	946KiB/s
128k	416	6667KiB/s	47	757KiB/s
256k	353	5661KiB/s	63	1020KiB/s
512k	291	4671KiB/s	80	1296KiB/s
1024k	204	3273KiB/s	80	1288KiB/s
bs=64k	fsync=0		fsync=1
volblocksize	RandWrite IOPS	RandWrite BW	RandWrite IOPS	RandWrite BW
8k	1330	83.2MiB/s	55	3531KiB/s
16k	1975	123MiB/s	41	2672KiB/s
32k	2065	129MiB/s	39	2524KiB/s
64k	4228	264MiB/s	40	2594KiB/s
128k	513	32.1MiB/s	48	3073KiB/s
256k	358	22.4MiB/s	54	3504KiB/s
512k	286	17.9MiB/s	67	4343KiB/s
1024k	209	13.1MiB/s	65	4213KiB/s

Read & Random Read Results With & Without Cache:

The cache was occupied little over 80GB in size so presumably it had all the data cached. The used command for cached reads was:

fio --name=test --loops=8 --numjobs=1 --allrandrepeat=1 --ioengine=libaio --filesize=10G --iodepth=64 --direct=1 --buffered=0 --directory=VOLUME_MOUNTPOINT

 

and for uncached reads there was no reason to loop multiple times:

fio --name=test --numjobs=1 --allrandrepeat=1 --ioengine=libaio --filesize=10G --iodepth=64 --direct=1 --buffered=0 --directory=VOLUME_MOUNTPOINT

 

The results are with cache:

Cache: MK0100GCTYU
bs=4k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	34.1k	133MiB/s	34.2k	134MiB/s
16k	40.0k	160MiB/s	5509	21.5MiB/s
32k	45.5k	178MiB/s	5215	20.4MiB/s
64k	49.9k	195MiB/s	5423	21.2MiB/s
128k	55.9k	218MiB/s	3624	14.2MiB/s
256k	53.7k	210MiB/s	2020	8081KiB/s
512k	52.5k	205MiB/s	1043	4174KiB/s
1024k	47.7k	186MiB/s	529	2118KiB/s
bs=16k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	16.1k	251MiB/s	19.3k	301MiB/s
16k	21.9k	342MiB/s	22.4k	350MiB/s
32k	23.2k	362MiB/s	13.4k	209MiB/s
64k	26.3k	410MiB/s	7077	111MiB/s
128k	30.8k	481MiB/s	3950	61.7MiB/s
256k	26.4k	413MiB/s	2083	32.6MiB/s
512k	26.5k	414MiB/s	1070	16.7MiB/s
1024k	26.0k	421MiB/s	539	8633KiB/s
bs=64k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	3606	225MiB/s	4704	294MiB/s
16k	5046	315MiB/s	5482	343MiB/s
32k	5666	354MiB/s	5833	365MiB/s
64k	6064	379MiB/s	6092	381MiB/s
128k	7101	444MiB/s	3705	232MiB/s
256k	8264	517MiB/s	2079	130MiB/s
512k	8752	547MiB/s	1082	67.6MiB/s
1024k	7878	492MiB/s	550	34.4MiB/s

Without Cache:

No Cache
bs=4k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	34.5k	135MiB/s	791	3167KiB/s
16k	42.3k	165MiB/s	796	3187KiB/s
32k	46.1k	180MiB/s	476	1906KiB/s
64k	50.6k	198MiB/s	445	1781KiB/s
128k	61.7k	241MiB/s	412	1650KiB/s
256k	54.7k	214MiB/s	501	2006KiB/s
512k	54.7k	214MiB/s	375	1501KiB/s
1024k	51.2k	200MiB/s	293	1173KiB/s
bs=16k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	12.7k	199MiB/s	679	10.6MiB/s
16k	15.0k	250MiB/s	717	11.2MiB/s
32k	19.2k	300MiB/s	440	7042KiB/s
64k	24.9k	389MiB/s	416	6657KiB/s
128k	30.0k	484MiB/s	392	6281KiB/s
256k	26.6k	416MiB/s	485	7763KiB/s
512k	27.3k	427MiB/s	365	5842KiB/s
1024k	26.4k	412MiB/s	293	4703KiB/s
bs=64k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	3080	193MiB/s	499	31.2MiB/s
16k	4395	275MiB/s	572	35.8MiB/s
32k	4847	303MiB/s	397	24.8MiB/s
64k	6568	411MiB/s	394	24.6MiB/s
128k	8692	543MiB/s	386	24.2MiB/s
256k	8110	507MiB/s	495	30.0MiB/s
512k	8773	548MiB/s	367	22.9MiB/s
1024k	7680	480MiB/s	294	18.4MiB/s

Write & Random Write Results (with and without fsync) + SLOG:

Same command as the previous write tests was used with and without fsync and with the addition of SLOG device.

Write:

SLOG: INTEL SSDSC2KG480G8
bs=4k	fsync=0		fsync=1
volblocksize	Write IOPS	Write BW	Write IOPS	Write BW
8k	9343	36.5MiB/s	686	2747KiB/s
16k	14.5k	56.7MiB/s	668	2674KiB/s
32k	9775	38.2MiB/s	873	3493KiB/s
64k	8245	32.2MiB/s	1027	4108KiB/s
128k	11.8k	46.1MiB/s	1087	4349KiB/s
256k	12.3k	48.1MiB/s	1175	4703KiB/s
512k	13.8k	53.8MiB/s	1085	4344KiB/s
1024k	16.6k	64.0MiB/s	1230	4923KiB/s
bs=16k	fsync=0		fsync=1
volblocksize	Write IOPS	Write BW	Write IOPS	Write BW
8k	6007	93.9MiB/s	452	7236KiB/s
16k	9875	154MiB/s	681	10.6MiB/s
32k	4804	75.1MiB/s	564	9030KiB/s
64k	3691	57.7MiB/s	730	11.4MiB/s
128k	5233	81.8MiB/s	755	11.8MiB/s
256k	5479	85.6MiB/s	653	10.2MiB/s
512k	4888	76.4MiB/s	669	10.5MiB/s
1024k	5894	92.1MiB/s	542	8677KiB/s
bs=64k	fsync=0		fsync=1
volblocksize	Write IOPS	Write BW	Write IOPS	Write BW
8k	1765	110MiB/s	456	28.6MiB/s
16k	2530	158MiB/s	499	31.2MiB/s
32k	2815	176MiB/s	594	37.2MiB/s
64k	5298	331MiB/s	214	13.4MiB/s
128k	1555	97.2MiB/s	538	33.6MiB/s
256k	1819	114MiB/s	634	39.7MiB/s
512k	1751	109MiB/s	666	41.6MiB/s
1024k	1915	120MiB/s	601	37.6MiB/s

 

Random Write:

SLOG: INTEL SSDSC2KG480G8
bs=4k	fsync=0		fsync=1
volblocksize	RandWrite IOPS	RandWrite BW	RandWrite IOPS	RandWrite BW
8k	524	2098KiB/s	719	2879KiB/s
16k	785	3140KiB/s	721	2885KiB/s
32k	502	2011KiB/s	713	2853KiB/s
64k	662	2651KiB/s	826	3305KiB/s
128k	546	2185KiB/s	593	2372KiB/s
256k	417	1671KiB/s	469	1877KiB/s
512k	330	1321KiB/s	343	1373KiB/s
1024k	271	1085KiB/s	265	1063KiB/s
bs=16k	fsync=0		fsync=1
volblocksize	RandWrite IOPS	RandWrite BW	RandWrite IOPS	RandWrite BW
8k	5198	81.2MiB/s	454	7272KiB/s
16k	9380	147MiB/s	440	7042KiB/s
32k	958	14.0MiB/s	501	8020KiB/s
64k	995	15.6MiB/s	643	10.1MiB/s
128k	733	11.5MiB/s	537	8600KiB/s
256k	553	8858KiB/s	505	8081KiB/s
512k	368	5900KiB/s	343	5504KiB/s
1024k	260	4168KiB/s	248	3972KiB/s
bs=64k	fsync=0		fsync=1
volblocksize	RandWrite IOPS	RandWrite BW	RandWrite IOPS	RandWrite BW
8k	1531	95.7MiB/s	378	23.6MiB/s
16k	2545	159MiB/s	439	27.5MiB/s
32k	2577	161MiB/s	453	28.4MiB/s
64k	4998	312MiB/s	205	12.8MiB/s
128k	743	46.5MiB/s	444	27.8MiB/s
256k	527	32.0MiB/s	377	23.6MiB/s
512k	387	24.2MiB/s	331	20.7MiB/s
1024k	270	16.9MiB/s	238	14.9MiB/s

Read Results With 1GB ARC + With & Without 100GB L2ARC

 

No Cache
bs=4k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	34.5k	135MiB/s	791	3167KiB/s
16k	42.3k	165MiB/s	796	3187KiB/s
32k	46.1k	180MiB/s	476	1906KiB/s
64k	50.6k	198MiB/s	445	1781KiB/s
128k	61.7k	241MiB/s	412	=1650KiB/s
256k	54.7k	214MiB/s	501	2006KiB/s
512k	54.7k	214MiB/s	375	1501KiB/s
1024k	51.2k	200MiB/s	293	1173KiB/s
bs=16k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	12.7k	199MiB/s	679	10.6MiB/s
16k	15.0k	250MiB/s	717	11.2MiB/s
32k	19.2k	300MiB/s	440	7042KiB/s
64k	24.9k	389MiB/s	416	6657KiB/s
128k	30.0k	484MiB/s	392	6281KiB/s
256k	26.6k	416MiB/s	485	7763KiB/s
512k	27.3k	427MiB/s	365	5842KiB/s
1024k	26.4k	412MiB/s	293	4703KiB/s
bs=64k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	3080	193MiB/s	499	31.2MiB/s
16k	4395	275MiB/s	572	35.8MiB/s
32k	4847	303MiB/s	397	24.8MiB/s
64k	6568	411MiB/s	394	24.6MiB/s
128k	8692	543MiB/s	386	24.2MiB/s
256k	8110	507MiB/s	495	30.0MiB/s
512k	8773	548MiB/s	367	22.9MiB/s
1024k	7680	480MiB/s	294	18.4MiB/s

 

 

 

Cache: MK0100GCTYU
bs=4k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	34.1k	133MiB/s	34.2k	134MiB/s
16k	40.0k	160MiB/s	5509	21.5MiB/s
32k	45.5k	178MiB/s	5215	20.4MiB/s
64k	49.9k	195MiB/s	5423	21.2MiB/s
128k	55.9k	218MiB/s	3624	14.2MiB/s
256k	53.7k	210MiB/s	2020	8081KiB/s
512k	52.5k	205MiB/s	1043	4174KiB/s
1024k	47.7k	186MiB/s	529	2118KiB/s
Cache: MK0100GCTYU
bs=16k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	16.1k	251MiB/s	19.3k	301MiB/s
16k	21.9k	342MiB/s	22.4k	350MiB/s
32k	23.2k	362MiB/s	13.4k	209MiB/s
64k	26.3k	410MiB/s	7077	111MiB/s
128k	30.8k	481MiB/s	3950	61.7MiB/s
256k	26.4k	413MiB/s	2083	32.6MiB/s
512k	26.5k	414MiB/s	1070	16.7MiB/s
1024k	26.0k	421MiB/s	539	8633KiB/s
Cache: MK0100GCTYU
bs=64k
volblocksize	Read IOPS	Read BW	RandRead IOPS	RandRead BW
8k	3606	225MiB/s	4704	294MiB/s
16k	5046	315MiB/s	5482	343MiB/s
32k	5666	354MiB/s	5833	365MiB/s
64k	6064	379MiB/s	6092	381MiB/s
128k	7101	444MiB/s	3705	232MiB/s
256k	8264	517MiB/s	2079	130MiB/s
512k	8752	547MiB/s	1082	67.6MiB/s
1024k	7878	492MiB/s	550	34.4MiB/s

 

 

 

 Continue reading - Part 3 - Compressible Data Benchmarks

 

 

ZFS Raid-Z3 Performance with Zstandard (ZSTD) - Part 1 - Benchmark Background Information

Background

I got my hands on a few unwanted HP Gen9 servers with lots of reasonably sized SAS drives. Don't laugh, but it was like in a sci-fi movie where the army would try to kill all the aliens as default action. The people who owned the hardware were literally scared of touching the hardware.

 

The consensus in the company was to hammer the drives and destroy the hardware because it will make too much noise, too much heat and get broken. Perhaps it could be standard response from people who does not understand the hardware well enough.

 

Perhaps a good example of one man's trash is another man's treasure :)

Test System

The test system consists of a Proxmox VE installation over a machine with Intel E5-2620 v4 and 32GB DDR4 memory. This means the default ARC size is about 16GB. The decision to use such a low memory configuration was to reduce the effect of the ARC cache.

The installation was made on an 8x 1.8TB 12G SAS HDD (EG1800JEMDB) with RAID-Z3 and ashift value of 12. The drives were connected to a SAS expander which was connected to P440ar with firmware version 7.00-0 and running in HBA mode.

Tests were executed inside a VM with 4 CPU cores and 4GB RAM. Several volumes of 100GB size, and volblocksize ranging from 8k to 1024k was mounted within the VM into different folders and fio command was used for tests.

Tests Performed

The  basic read, random read (randread), write and random write (randwrite)) tests were performed. The write tests were performed with and without fsync. The file size used was 80GB and created ready prior to running the tests. The tests were performed with fio bs sizes of 4, 16 and 64

For the second part of the read tests with cache. The ARC size was reduced to 1GB. It was the minimum value which seemed to work according to this bug report.The size was manipulated by overriding it using a  modprobe.d file

options zfs zfs_arc_max=1048067201

 

The read tests were repeated again with 1GB ARC value so the results would be comparable with the cache tests. In addition, the read tests were modified to read 10GB with 8 times loop of same test to be able to train the cache first.

 

Test Data

The in-compressible test data was created using the following fio command:

fio --name=test --create_only=1 --filesize=80G --directory=VOLUME_MOUNTPOINT

The compressible test data was created using the following fio command:

fio --name=test --buffer_pattern=0xdeadbeef --buffer_compress_percentage=75 --create_only=1 --filesize=80G --directory=VOLUME_MOUNTPOINT

The compressible data was found to be compressible to roughly 1/4 size of the original.

volblocksize	ratio	used	lused	compress	actual savings
8K	1.99x	92.4G	80.9G	zstd	-1.14x
16K	2.00x	92.0G	80.7G	zstd	-1.14x
32K	2.67x	46.0G	80.6G	zstd	1.75x
64K	3.20x	23.0G	80.6G	zstd	3.5x
128K	3.57x	22.9G	80.5G	zstd	3.5x
256K	3.78x	22.9G	80.5G	zstd	3.5x
512K	3.90x	20.0G	80.6G	zstd	4.03x
1024K	3.90x	19.3G	80.6G	zstd	4.17x

In some cases actual savings is more than the compression ratio reported. Perhaps this could be due to zeros accepted as null blocks instead of being in the compression statistics.

Continue reading - Part 2 - In-compressible Data Read Benchmark

ZFS zstd (zstd_compress) Feature Compression Performance

Zstandard (ZSTD) is a relatively new compression method introduced to ZFS.

Below are test results from a Proxmox host with kernel 5.4.106-1-pve. The pool consisted of 8 drives in RAID-Z3 configuration.

The comparison is between default compression=on (presumably LZ4) vs compression=zstd

The test was simply making a tar archive of the /usr partition from a Ubuntu 20.04 Desktop installation and then making more copies of it to fill about 100GB of space.

This was done on a VM running within Proxmox with virtual disks of different volblocksize mounted on different paths.

There is small discrepancy between filesystems. It could be due to fstrim not cleaning all the blocks.

Here is the results with compression=on 

volblocksize	ratio	volblocksize	used	refer	lused	lrefer	actual savings
8K	1.28x	8k	168G	168G	94.6G	94.6G	-1.77x
16K	1.54x	16k	102G	102G	95.1G	95.1G	-1.07x
32K	1.75x	32k	65.9G	65.9G	95.9G	95.9G	1.46x
64K	1.89x	64k	54.9G	54.9G	96.2G	96.2G	1.75x
128K	1.96x	128k	49.4G	49.4G	96.5G	96.5G	1.95x
256K	2.01x	256k	46.0G	46.0G	96.8G	96.8G	2.10x

The ratio is the ratio reported by zfs list command. The actual savings is the calculated savings base on diffeence between used and lused.

Similarly below is the results with compression=zstd

volblocksize	ratio	volblocksize	used	refer	lused	lrefer	actual savings
8K	1.49x	8k	145G	145G	94.6G	94.6G	-1.53x
16K	1.90x	16k	92.0G	92.0G	95.1G	95.1G	1.03x
32K	2.21x	32k	58.1G	58.1G	95.9G	95.9G	1.66x
64K	2.43x	64k	44.2G	44.2G	96.2G	96.2G	2,17x
128K	2.58x	128k	38.3G	38.3G	96.5G	96.5G	2,51x
256K	2.73x	256k	34.3G	34.3G	96.8G	96.8G	2,82x

The results are quite interesting as the default 8k size seems to be performing negatively and 16k is a borderline case. It is not clear why the default is 8k for volumes.

In general it looks like zstd seems to be providing better than the default compression and 128k and 256k block sizes seem to give good results.

In some of my future articles I will have a table going up to 1024k volblocksize.