Monday, April 5, 2021

ZFS Raid-Z3 Performance with Zstandard (ZSTD) - Part 1 - Benchmark Background Information

Background

I got my hands on a few unwanted HP Gen9 servers with lots of reasonably sized SAS drives. Don't laugh, but it was like in a sci-fi movie where the army would try to kill all the aliens as default action. The people who owned the hardware were literally scared of touching the hardware.
 
The consensus in the company was to hammer the drives and destroy the hardware because it will make too much noise, too much heat and get broken. Perhaps it could be standard response from people who does not understand the hardware well enough.
 
Perhaps a good example of one man's trash is another man's treasure :)

Test System

The test system consists of a Proxmox VE installation over a machine with Intel E5-2620 v4 and 32GB DDR4 memory. This means the default ARC size is about 16GB. The decision to use such a low memory configuration was to reduce the effect of the ARC cache.

The installation was made on an 8x 1.8TB 12G SAS HDD (EG1800JEMDB) with RAID-Z3 and ashift value of 12. The drives were connected to a SAS expander which was connected to P440ar with firmware version 7.00-0 and running in HBA mode.

Tests were executed inside a VM with 4 CPU cores and 4GB RAM. Several volumes of 100GB size, and volblocksize ranging from 8k to 1024k was mounted within the VM into different folders and fio command was used for tests.

Tests Performed

The  basic read, random read (randread), write and random write (randwrite)) tests were performed. The write tests were performed with and without fsync. The file size used was 80GB and created ready prior to running the tests. The tests were performed with fio bs sizes of 4, 16 and 64

For the second part of the read tests with cache. The ARC size was reduced to 1GB. It was the minimum value which seemed to work according to this bug report.The size was manipulated by overriding it using a  modprobe.d file

options zfs zfs_arc_max=1048067201
 
The read tests were repeated again with 1GB ARC value so the results would be comparable with the cache tests. In addition, the read tests were modified to read 10GB with 8 times loop of same test to be able to train the cache first.
 

Test Data

The in-compressible test data was created using the following fio command:
fio --name=test --create_only=1 --filesize=80G --directory=VOLUME_MOUNTPOINT
The compressible test data was created using the following fio command:
fio --name=test --buffer_pattern=0xdeadbeef --buffer_compress_percentage=75 --create_only=1 --filesize=80G --directory=VOLUME_MOUNTPOINT
The compressible data was found to be compressible to roughly 1/4 size of the original.

volblocksize ratio used lused compress actual savings
8K 1.99x 92.4G 80.9G zstd -1.14x
16K 2.00x 92.0G 80.7G zstd -1.14x
32K 2.67x 46.0G 80.6G zstd 1.75x
64K 3.20x 23.0G 80.6G zstd 3.5x
128K 3.57x 22.9G 80.5G zstd 3.5x
256K 3.78x 22.9G 80.5G zstd 3.5x
512K 3.90x 20.0G 80.6G zstd 4.03x
1024K 3.90x 19.3G 80.6G zstd 4.17x

In some cases actual savings is more than the compression ratio reported. Perhaps this could be due to zeros accepted as null blocks instead of being in the compression statistics.


ZFS zstd (zstd_compress) Feature Compression Performance

Zstandard (ZSTD) is a relatively new compression method introduced to ZFS.

Below are test results from a Proxmox host with kernel 5.4.106-1-pve. The pool consisted of 8 drives in RAID-Z3 configuration.

The comparison is between default compression=on (presumably LZ4) vs compression=zstd

The test was simply making a tar archive of the /usr partition from a Ubuntu 20.04 Desktop installation and then making more copies of it to fill about 100GB of space.

This was done on a VM running within Proxmox with virtual disks of different volblocksize mounted on different paths.

There is small discrepancy between filesystems. It could be due to fstrim not cleaning all the blocks.

Here is the results with compression=on 

volblocksize ratio volblocksize used refer lused lrefer actual savings
8K 1.28x 8k 168G 168G 94.6G 94.6G -1.77x
16K 1.54x 16k 102G 102G 95.1G 95.1G -1.07x
32K 1.75x 32k 65.9G 65.9G 95.9G 95.9G 1.46x
64K 1.89x 64k 54.9G 54.9G 96.2G 96.2G 1.75x
128K 1.96x 128k 49.4G 49.4G 96.5G 96.5G 1.95x
256K 2.01x 256k 46.0G 46.0G 96.8G 96.8G 2.10x

The ratio is the ratio reported by zfs list command. The actual savings is the calculated savings base on diffeence between used and lused.

Similarly below is the results with compression=zstd

volblocksize ratio volblocksize used refer lused lrefer actual savings
8K 1.49x 8k 145G 145G 94.6G 94.6G -1.53x
16K 1.90x 16k 92.0G 92.0G 95.1G 95.1G 1.03x
32K 2.21x 32k 58.1G 58.1G 95.9G 95.9G 1.66x
64K 2.43x 64k 44.2G 44.2G 96.2G 96.2G 2,17x
128K 2.58x 128k 38.3G 38.3G 96.5G 96.5G 2,51x
256K 2.73x 256k 34.3G 34.3G 96.8G 96.8G 2,82x

The results are quite interesting as the default 8k size seems to be performing negatively and 16k is a borderline case. It is not clear why the default is 8k for volumes.

In general it looks like zstd seems to be providing better than the default compression and 128k and 256k block sizes seem to give good results.

In some of my future articles I will have a table going up to 1024k volblocksize.

Tuesday, November 3, 2020

Golf MK4 Air Fillter Box Heated Air Flap Wax Thermostate Repair

The engine cover doubles as air filter box and also houses the heated air intake flap.

The heated air intake is through the round hole under the box. It can be seen when the filter cover is removed from the bottom.

Thermostat

After many years of operation, the heated air intake flap usually gets stuck in hot air side fully open.This is not a big problem but there is no point in taking in warmer air than necessary.

Unfortunately it is not so easy to find replacement thermostats. The only one I could find was some Volvo replacements. Although there are 2 different Volvo models, with different prices, they seem to be exactly same thermostat. So buy the cheaper one.

Here is the link to thermostat:
https://www.skandix.de/en/spare-parts/engine/fuel-mixture-formation/injection-system/idle-control/thermostat-air-temperature/1006579/

Volvo reference numbers are 1266826 and 9142616

The Volvo thermostat itself is slightly longer than the original and also the pin is much longer. So the pin must be cut smaller. The old pin can't be used because it is not same thickness.

Had to cut about 4mm to be able to bring it to correct size. There is some slack space when the pin is completely out. One can cut it even 6mm and it should still be fine. I just cut the minimum required amount so the flap can open completely.

Thermostat Removal

The thermostat can be seen once the other cover is also removed.

Removing the thermostat is not an easy task. If you are not careful the flap will get broken! The flap may be brittle from extended periods of hot air.

It is possible to remove the thermostat without removing the flap. However the new thermostat was longer and I could not install it without removing the whole spring system. But, there are people who claim to have done it.

For removing the old thermostat, you can simply push the spring as far as it goes and let the thermostat fall out. I removed the flap first.

For removing the flap, first you will need to pry open the little cover piece in the middle. It is a ~1cm length rectangular piece. You can see it removed in the picture below.

Next step is to move the spring to side, which is shown on the picture above. After this point, the flap can be removed with a little bit wiggling and bending of the black upper part. Then the spring assembly can be removed.


Once the spring is removed, one can remove the thermostat and replace it easily. I stored the new thermostat in freezer first to be able to push the pin all the way in. It makes installation much easier.

Thermostat Installation

The installation is reverse of removal. You will need to glue/melt the rectangular plastic cap back after installation.


Testing

The mormal operation is:

      Above +23 °C, flap must close warm air connection.

      Below +10 °C, flap opens warm air connection.

The Volvo thermostat starts opening warm air at about 9C and fully opens it at about 3C. It is fully closed at about 16C. Which are pretty good values all things considered.
 
I tested mine in fridge.
 






Saturday, April 11, 2020

How to assign remote keys to key combinations in KODI / X

I have tried to use the chrome launcher on KODI and it was not so easy to exit the browser.

I decided to map CTRL-W to a previously unused button on the remote control.

The first step is to find out what the system sees when you press the button on the remote. I used xev for this purpose. However running it from shell directly did not lead to any results.

I had to create a startup script for KODI and first start openbox window manager. It was necessary to install openbox package anyway because the chrome plugin for KODI was using it so the browser can start full screen.

#!/bin/sh

/usr/bin/openbox &
/usr/bin/kodi &
wait %2

Then had to edit /usr/share/xsessions/kodi.desktop to execute this script. Only after that, I have been able to capture the input from my remote. However this setting can eventually be reversed.

KeyPress event, serial 44, synthetic NO, window 0x400001,
    root 0x4e0, subw 0x0, time 1226119813, (1262,603), root:(1263,623),
    state 0x0, keycode 72 (keysym 0xffc3, F6), same_screen YES,
    XLookupString gives 0 bytes:
    XmbLookupString gives 0 bytes:
    XFilterEvent returns: False

KeyRelease event, serial 47, synthetic NO, window 0x400001,
    root 0x4e0, subw 0x0, time 1226119885, (1262,603), root:(1263,623),
    state 0x0, keycode 72 (keysym 0xffc3, F6), same_screen YES,
    XLookupString gives 0 bytes:
    XFilterEvent returns: False


According to output, the remote sent F6 to the HTPC. Now we can repurpose F6. The easiest way is to use xbindkeys. For this purpose you need to create /home/kodi/.xbindkeysrc file and put the following in it.

"xdotool key --clearmodifiers ctrl+w"
    F6 + release

Then we can start xbindkeys from the .xprofile file by simply adding /usr/bin/xbindkeys into it.

Now, when I press the remote key, the chrome browser window closes and I return back to KODI.

Tuesday, December 17, 2019

RL-3331 Emergency Light



RL-3331 is a cheap emergency light from a company with a strange web address http://www.yxdz88.com/

It has 30 LEDs although versions with 60 and 90 LEDs with slightly different model numbers exist.

The package lists specifications as:

Name of Product: Emergency Light
Item No.: RL-3331
Recharging Voltage Rate: AC220V-240V
Battery Voltage Rate: DC 4V
Battery Capacity: 1100mAh
Lighting Source: 30 LED
Duration time: >= 5 hours
Standard No: Q/STLM 0016-2011

As usual for cheap items from AliExpress. Not all these specifications are correct.

Some sellers also sell this item as "with lithium battery" and it is not true. Some list this device as 6W and it is also not true. The item has lead acid battery and LED power is much less....

General Construction

The general construction quality is quite high. The case is very tight and difficult to open even after removing the screws. The board inside has good solder joints, although it looks like it was made by hand. Therefore the quality can vary between units. There are not many component and it is a simple 4.82V transformer with a transistor to switch LEDs ON/OFF.

Charging voltage is 4.82V when the switch is at OFF position, it goes down from 4.82V to 4.76V if the control switch is set to Low or High modes.

Battery

Some sellers advertise this product as with lithium rechargeable batteries. However this is not true. It simply has a 2 cell lead-acid battery. Battery dimensions are 35mm 22mm 96mm
The small black cap can be removed to expose the rubber caps (valves) for this type of battery. It use a mat between the plates so no liquid can leak. After years of use, if the battery becomes weak, you may try to add some distilled water. Usually battery loses some liquid slowly in this type of simple construction.

Based on the 1100mAh specification. I first ran a 1A discharge test from full 4.8V voltage to empty 3V voltage
The battery did go from full to empty in about 42 minutes which totaled to 706mAh capacity. It could have given different results with different discharge currents. However it took over 24 hours to charge back 706mAh back into the battery!
After this test, I have tested the high power mode on the lamp. It took about 1h50m until it dipped under 3V
It is quite visible from the drop on the curve to see when battery was totally empty. However I continued this test much longer just to see what will happen. This also shows that clearly the lamp is much less than 6W as the 1A discharge test resulted in average 3.62V which makes a 3.62W usage. But it was much much more lighter load when you compare the graphs.
The voltage becomes more stable after the first drop but this is the deep-discharge territory and probably harm the battery in the long run.  It is not good to use the lamp over 2 hours on high power mode as this will damage the battery.
The charging back to 4.82V took over 55 hours! Yes, the recovery was really really slow.

The discharge of the light with low power mode results in a much more sensible discharge graph and it achieves about 4h30m before dipping under 3V. So the >=5 hour rating is also imaginary.

I ran the test for an extended period of 7 hours. I do need to get it to recharge for 2+ days for the lead acid battery to recover.

Conclusion

RL-3331 is a cheap product with cheap components. It will probably last about 2-3 years before the liquids inside the lead-acid battery vaporize. The seals are not always 100% airtight. Overall, it is a good product when the price is considered.

Once the battery is empty, you can change it with an actual lithium battery using a lithium BMS. With about 4.8V maximum voltage, you should be able to charge a 4.2V lithium battery fully. Perhaps an 18650 battery with 3000mAh can be installed to improve the performace dramatically. However it is sad that the charging back can take several days with such slow speeds.

Wednesday, November 13, 2019

Unbranding ZYXEL VMG3625-T20A (Telia/Sonera) Router

This operation can possibly brick your router. So do it only if you know what you are doing. Your router is your own responsibility. There are many so called YouTube videos etc. which have unnecessary and dangerous steps. So use your brain to decide what you need and what you do not. I will stick to bare minimum and as safe as possible steps for this specific router model.

Sometimes ISPs brand the routers and disable features, maybe mess with their settings and perhaps spy on you.

Maybe the router is locked into a specific provider and you need to use it with another. There are many reasons why you may want to install generic ZYXEL firmware to an xDSL router.

Luckily it is rather easy to unbrnad the ZYXEL routers. I have performed this operation on a ZYXEL VMG3625-T20A router which was provided free with (also free) VDSL service. Albeit, free service is limited to 10Mbit/s speed but that is a good enough speed for most people.

What Do We Need?

We need few items for this operation.
  • USB TTL Serial adapter
  • Putty or any terminal program which supports serial communication
  • Tftp client (Windows 10 comes with one!) 
 

Opening The Router Case

This router has only 2 screws under the sticker. Some other routers may have 4 screws. You can feel the placements of he screw holes with your finger if you move on the sticker. You can lift the corners of the sticker with a sharp knife and unscrew the screws. I left the screws under the sticker, so they would not go far.

Once the screws are removed, you can use a plastic prying tool to open the clips. The parts are quite tight, so use patience.

I did not make pictures of this as the operation is vey simple. If you can't manage to open a simple plastic cover, you should stop now.

Here is how the router looks like inside. For our purposes, the most important piece is th 4pin connector.






The 4pin connector...


Connecting Rouer To Serial Adapter

We need to connect the router to serial adapter for communication. As you can see, the 4pin connector pins are marked as GND, NC, RX, TX, and 3.3V. For our purposes, we only need to connect RX, TX, and GND pins.

Important: Now you need to set your TTL adapter to use 3.3v. They usually have a jumper or switch for this purpose.

Modem Side - Serial Adapter Side
RX         - TX
TX         - RX
GND        - GND 

The only thing you need to be careful is to connect TX to RX. There is little which can go wrong.

Open your serial communications program and set it to work with 115200 8N1 setting. If everything is corect, you should see boot messages printed to screen when you turn on the router.

Connecting Router To Ethernet

You will also need to connect the router to ethernet as we will use TFTP to push files to the router. You should set your PCs IP address statically to 192.168.1.10 (or anyting else than 192.168.1.1 in 192.168.1.0/24 network).

Getting Ready To Send Firmware

When you turn on the router, it will give you few seconds to abort the boot process. Press a key to abort it!

You should now be at a prompt which says ZHAL> Now we can start some magic.

First run ATSH command to extract some important information. We will not need this information, but it is good to have in case something goes wrong.

The output will look like following
Firmware Version : V5.30(ABIE.6)b2_E0 Bootbase Version : V1.32 | 03/22/2018 16:56:20 Vendor Name : Zyxel Communications Corp. Product Model : VMG3625-T20A Serial Number : SXXXXXXXXXXXX First MAC Address : XXXXXXXXXXX0 Last MAC Address : XXXXXXXXXXXF MAC Address Quantity : 16 Default Country Code : FF Boot Module Debug Flag : 01 RootFS Checksum : e4d28a4b Kernel Checksum : eeba2049 Main Feature Bits : 00 Other Feature Bits : 840291b4: 04050305 00000100 00000000 00000000 840291c4: 00000000 00000000 00000000 0000
Now we need to get the login code. The login code is required for dangerous operaitons. For this purpose we should feed the Product Model from above output usin ATSE command. We will need to decode this. Luckily some nice people provided a C program which can do it. You can compile it with GCC easily. Obviously this program can be used for other ZYXEL routers! Just see the example in comments section of the program to figure out how to proceed!

/* ZyXEL prestige 660HW series password calculator by brainstorm 
  * Thanks to http://www.adslayuda.com/Zyxel650-9.html authors
  *
  * Example usage:
  *
  * Router:
  * ======
  *
  * ATSE
  * 0028D6DF1C03
  * OK
  *
  * Computer:
  * ========
  *
  * ./zyxel 0028D6DF1C03
  * ATEN 1,221E3111
  *
  * Router:
  * ======
  * ATEN 1,221E3111
  * OK
  *
  * "Dangerous" commands enabled :-)
  *
  * */

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define magic1  0x10F0A563L
#define magic2  7
#define atse_length 12  /* ATSE command, ZyNOS seed password length */

#define WORD_LENGTH (8*sizeof(value))
int ror(unsigned int value, int places)
{
  return (value>>places)|(value<<(WORD_LENGTH-places));
}


int main (int argc, char* argv[]) {

        char *seed, a[7], c[3];
        unsigned int b,d,e,password;

        if ( argc != 2 ) {
                printf("Only one argument is permitted: 00BDC8667E5B\n");
                exit(-1);

        } else if ( strlen(argv[1]) != atse_length ) {
                printf( "Incorrect parameter length, should be %d characters long\n", atse_length );
                exit (-2);
        }

        seed = argv[1];

        strncpy (a, seed , 6);  //a="ersten" 3Bytes vom seed
        e = strtol(a,NULL,16);  //e=a

        strncpy (c, seed + strlen(seed)-2, 2); //c= last 2 bytes of seed?
        d = strtol(c,NULL,16) & magic2; //d="last byte" AND 7
        b = e + magic1; //

        b = ror(b,d);
        password = b ^ e;
        printf("\nATEN 1,%X\n", password);

        return 0;
}


After entering ATEN command we can finally upload the firmware. If you downloaded the firmware files from ZYXEL site, you will see that there is a BIN file which is about 17Mbyte in size. I renamed this file as zyxel.bin just for easy typing.

Now we need to tell the router that we are about to send it firmware image named, you guessed it, zyxel.bin. The command for tis operation is ZHAL> ATUR zyxel.bin The router will wait you to send the file

Upgrade to rootfs partition 1
TFTP server is started, put your file 'zyxel.bin' to server (IP is 192.168.1.1).

Now from windows we can send the file in binary mode. We need ot use the -i flag for binary transfer with windows tftp client. So the command should look like tftp -i 192.168.1.1 PUT zyxel.bin. We already set the IP address of our machine statically so it should work just fine. So the complete output should look like below:

ZHAL> ATSE VMG3625-T20A
0EA420ED3340
ZHAL> ATEN 1,10F1EDA3
ZHAL> ATUR zyxel.bin
Upgrade to rootfs partition 1
TFTP server is started, put your file 'zyxel.bin' to server (IP is 192.168.1.1).
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Total 17498534 (0x10B01A6) bytes received

File download to memory address 0x80020000, length is 17498534

Please be patient, start to upgrade RAS!

...........................................................................................................................................................................................................................................................................................................................................................................................................................................

Update boot flag to 1
...
Auto reboot after 2 seconds
ZHAL>


Router will reboot by itself and you need to wait patiently as this will take few minutes at least. Meanwhile, not forget to remove the static IP setting from your ethernet adapter or you may regret not doing it later.

Next Step

Obviously the next step is closing up the router and tightenin the screws. I first installed 5.30(ABIE.4)C0 firmware from ZYXEL site then used the built-in web based upgrade to upgrade to 5.30(ABIE.5)C0 firmware. Also normal VDSL2 connectivity was working fine, including the 6RD tunnel that Telia/Sonera provides. So everything seems to be working fine. I am not sure why Telia/Sonera decided to make their own image, it is slightly suspicious.

Using same method, you can install any firmware to device (as long as it is compatible). For example if you want to brand it as another provider's device. I don't know why you would want to do this, but it is an option.

Caveats

The only problem is that I do not know how to backup existing firmware from the device. Therefore if you want to return back to old firmware, you will need to have access to it as BIN file. If you know how you can backup firmware from a ZYXEL device using serial communications, drop a line.

Saturday, November 9, 2019

Ubuntu 19.10 and Namespaces vs Netplan

I am using OpenVPN and namespaces to isolate access to the VPN connection. Originally, I wrote small scripts to bring up the connection and setup the necessary network interfaces.

I used a bridged ethernet inside the namespace as gateway to outside. For this purpose, normally I use brctl to edit bridge configurations. But now that Ubuntu is using netplan, things have changed a little bit.

Some namespace related operations are unclear and not documented at all.

Setting Up Namespace

First action is to edit /etc/systemd/network/25-veth.netdev and define 2 virtual ethernets.

[NetDev]
Name=veth0
Kind=veth
[Peer]
Name=veth1

Next action is to edit the /etc/netplan/50-cloud-init.yaml and assign one of the ethernets into bridge with main ethernet.

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        enp5s0:
            addresses: []
            dhcp4: false
        veth0: {}
        veth1: {}

    bridges:
      br0:
        dhcp4: yes
        interfaces:
          - enp5s0
          - veth0


So after this I still had to configure few setting using a script. If you know how to convert this to use netplan, please write to comments.
  ip netns add vpn
  ip netns exec vpn ip link set dev lo up
  ip link set veth1 netns vpn
  ip netns exec vpn ifconfig veth1 inet 192.168.1.252 netmask 255.255.255.0 up
  ip netns exec vpn route add default gw 192.168.1.1 dev veth1
  ip netns exec vpn route add -net 10.8.0.0 netmask 255.255.255.0 gw 192.168.1.1 dev veth1

You probably do not need the last line. I needed to add it so that packets for 10.8.0.0/24 network would not go out through VPN.

Name Resolution

One other problem is that Ubuntu 19.10 seem to be using systemd-resolved and therefore /etc/resolv.conf looks like this:

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search lan

Now the fun part is that it is extremely difficult to route localhost addresses from namespace. So I had to use external resolv.conf and hosts files. These files are put to /etc/netns/vpn/ (here "vpn" is my namespace).

resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4


hosts
127.0.0.1       localhost.localdomain   localhost ubuntu
::1             localhost6.localdomain6 localhost6 ubuntu

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Now the reason for using some outside DNS IP addresses is so that the DNS traffic is routed over VPN once it is established. I was not super paranoid about this. But somebody might want to use up/down scripts with openvpn to edit the nameservers.

I had to setup hosts file because sudo command was giving dns resolving errors for the hostname ubuntu