compression

zlib acceleration

With the z15 the CPU in System Z has gotten the ability to compress data into the zlib format really fast. The feature is officially called Integrated Accelerator for zEnterprise Data Compression (zEDC). This is know as zlib acceleration. Of course decompression is also faster, but the vast amount of CPU resources is usually spent in compression. This videos have a high level introduction.
I’ve been using this feature now for a year and I really like it. For a tar -czf ... you get a really fast response.  

As this is a new user space instruction, the only thing you need from an operating system perspective is the z15 enablement. The same is true for z/VM and KVM. As soon as they support the new CPU model the instruction is available. For container environments, the instruction is available if it’s available for the Linux image running the container. 

To enable this you need one of the supporting distributions. There is an environment variable controlling which compression levels should be accelerated. If this variable is not set only compression level 1 is accelerated.
The environment variable DFLTCC_LEVEL_MASK contains a binary value. Each bit corresponds to one compression level. Here are a few examples:

  • DFLTCC_LEVEL_MASK=0x0000 (disable hardware acceleration)
  • DFLTCC_LEVEL_MASK=0x0002 (enable hardware acceleration for compression level 1)
  • DFLTCC_LEVEL_MASK=0x007e (enable hardware acceleration for compression level 1-6)
  • DFLTCC_LEVEL_MASK=0x01ff (enable hardware acceleration for compression level 0-8)

Note that the compression ratio will not change by this setting! The ratio will always be roughly the same as compression level 1. My recommendation is to use 0x01ff as this will speed up every level but the one with the best compression ratio. So if a software intentionally calls compression level 9 to achieve optimal compression it will get it. 
This setting also includes the usage if software asks for no compression (level 0) for best speed. This feature is so fast that there should be no reason not to compress. If this creates functional problems choose 0x01fe. 

Java has z15 support of zlib acceleration starting with Java 8 SR6 Fix Pack 16. It automatically detects the new hardware and uses it. The OpenJDK variants (at least the jdk-11.0.6+10 that I tried) uses the system zlib. So you need a supporting distribution with the right environment setting to exploit. 

As there are slight distribution specific differences, I’ll list them in detail below for the supported distributions. This section will grow as more and more distributions will pick up the support. You also may want to check the documentation at GitHub. 

RHEL 7.7, 7.8, 7.9

No support of the zlib yet but Java 8 SR6 already works.

RHEL 8.1

No support of the zlib yet but Java 8 SR6, already works. Red Hat included gzip and gunzip support. If you don’t want to set the environment variable at every login, you can add a line like

export DFLTCC_LEVEL_MASK=0x01ff

to /etc/bashrc. Then it’s enabled for every user. Of course everyone can include it in their own .bashrc

RHEL 8.2

Everything included! 
If you don’t want to set the environment variable at every login, you can add a line like

export DFLTCC_LEVEL_MASK=0x01ff

to /etc/bashrc. Then it’s enabled for every user. Of course everyone can include it in their own .bashrc

RHEL 8.4 and newer

The default has been changed to DFLTCC_LEVEL_MASK=0x007e (enable hardware acceleration for compression level 1-6).

SLES 12 SP5

SUSE has put the exploitation into the popular gzip and gunzip commands as well as in the standard libz. If you don’t want to set the environment variable at every login, you can add a line like

export DFLTCC_LEVEL_MASK=0x01ff

to /etc/bash.bashrc.local. Then it’s enabled for every user. Of course everyone can include it in their own .bashrc

SLES 15 SP1

SUSE has released the zlib library (libz1-1.2.11-3.9.1.s390x).  If you don’t want to set the environment variable at every login, you can add a line like

export DFLTCC_LEVEL_MASK=0x01ff

to /etc/bash.bashrc.local. Then it’s enabled for every user. Of course everyone can include it in their own .bashrc

Ubuntu 20.04

Canonical has integrated the new zlib in their new distribution. They also have the exploitation in gzip and gunzip. Again you have to set the environment variables right. To change it for every bash login add a line like

export DFLTCC_LEVEL_MASK=0x01ff

to /etc/bash.bashrc. Then it’s enabled for every user. Of course everyone can include it in their own .bashrc

Summary Table

Distribution Instruction enabled (e.g. for Java usage) gzip/gunzip zlib
RHEL 7.7, 7.8, 7.9      
RHEL 8.1      
RHEL 8.2 and newer      
SLES 12 SP5      
SLES 15 SP1      
Ubuntu 20.04      

 

Example on how to try with a supported gzip

First let’s create a 1 GB file with ASCII characters. Make sure that you have enough memory, so that ecerything fits into the cache:

base64 /dev/urandom | head -c 1000000000 > /tmp/test.txt

Then let’s measure and compress with the unit

time gzip /tmp/test.txt

This will give you the user time to compress the file. If everything works the user time reported should not be larger thant 2s. Next step is to try without the hardware:

gunzip /tmp/test.txt.gz
export DFLTCC_LEVEL_MASK=0x0000
time gzip /tmp/test.txt

This should give you now a way longer time. Depending on what the random number generator did, usually more than 45s user time.

(updated 6/21/2023)

Leave a Comment

Your email address will not be published. Required fields are marked *

Data protection
I, Eberhard Pasch (Place of residence: Germany), process personal data to operate this website only to the extent technically necessary. All details in my privacy policy.
Data protection
I, Eberhard Pasch (Place of residence: Germany), process personal data to operate this website only to the extent technically necessary. All details in my privacy policy.