Huge performance difference on specific task

Issues related to applications and software problems and general support
Post Reply
maretodoric
Posts: 1
Joined: 2022/09/14 08:03:52

Huge performance difference on specific task

Post by maretodoric » 2022/09/14 10:40:59

Hello everyone,

i have this strange annoying problem that's happening on CentOS 9 and Fedora 35.
I'm trying to build alpine ISO using tools inside docker image. And when building the ISO (which consists of building kernel, squashfs and putting it all together in an ISO file) it takes 20-22 minutes for 160MB ISO file.

Now here comes the huge performance difference:
- Fedora 35 is on linux laptop with an i5-6300HQ, 16GB RAM, running 5.17.12-200 kernel and Docker version 20.10.17
- CentOS 9 is running on a VM located on a host with i7-4790k with 32GB of RAM in a VMWare Workstation. CentOS 9 has 4vCPUs assigned and 8G of RAM, has a kernel version 5.14.0 and Docker version 20.10.18

Both of those environments take same amount of time to build the ISO
I've cloned CentOS 9 VM and installed Debian Bullseye (on same host as CentOS 9). So literally same hardware setup as CentOS 9 , however running Kernel 5.10 and same Docker version - 20.10.18.
On Debian, same image, same source - it takes only 5-6 minutes to complete! That is huge difference!

At the moment i thought that it might be due to some weird kernel problem, so i changed repo on debian to use bullseye-backports and installed 5.14.0.
And build time still takes 5-6 minutes on Debian and 20-22 minutes on CentOS9.

Not really sure why is that much difference and i would like some help to figure it out.


Below is Dockerfile I've used to put together build environment

Code: Select all

FROM alpine:3.16.2

COPY chroot/entrypoint.sh /

RUN \
	apk add ifupdown-ng=0.12.1-r0 openrc=0.44.10-r7 \
		alpine-conf fakeroot=1.25.3-r3 \
		openssl libattr=2.5.1-r1 \
		attr=2.5.1-r1 libacl=2.3.1-r0 \
		tar=1.34-r0 pkgconf \
		patch=2.7.6-r7 libgcc=11.2.1_git20220219-r2 \
		libstdc++=11.2.1_git20220219-r2 lzip=1.23-r0 \
		ca-certificates brotli-libs=1.0.9-r6 \
		nghttp2-libs=1.47.0-r0 libcurl \
		curl abuild=3.9.0-r0 \
		binutils libmagic=5.41-r0 \
		file=5.41-r0 libgomp=11.2.1_git20220219-r2 \
		libatomic=11.2.1_git20220219-r2 gmp=6.2.1-r2 \
		isl22=0.22-r0 mpfr4=4.1.0-r0 \
		mpc1=1.2.1-r0 gcc=11.2.1_git20220219-r2 \
		musl-dev=1.2.3-r0 libc-dev=0.7.2-r3 \
		g++=11.2.1_git20220219-r2 make=4.3-r0 \
		fortify-headers=1.1-r1 build-base=0.5-r3 \
		expat=2.4.8-r0 pcre2=10.40-r0 \
		git alpine-sdk=1.0-r1 \
		ncurses-terminfo-base=6.3_p20220521-r0 ncurses-libs=6.3_p20220521-r0 \
		readline=8.1.2-r0 bash=5.1.16-r2 \
		dosfstools=4.2-r1 lddtree=1.26-r2 \
		xz-libs=5.2.5-r1 zstd-libs=1.5.2-r1 \
		kmod=29-r2 kmod-openrc=29-r2 \
		libblkid=2.38-r1 argon2-libs=20190702-r1 \
		device-mapper-libs=2.02.187-r2 json-c=0.16-r0 \
		libuuid=2.38-r1 cryptsetup-libs=2.4.3-r0 \
		kmod-libs=29-r2 mkinitfs \
		grub=2.06-r2 grub-efi=2.06-r2 \
		mtools=4.0.39-r0 lz4-libs=1.9.3-r1 \
		lzo=2.10-r3 squashfs-tools=4.5.1-r0 \
		sudo=1.9.10-r0 blkid=2.38-r1 \
		syslinux=6.04_pre1-r10 libburn=1.5.4-r1 \
		libedit=20210910.3.1-r0 libisofs=1.5.4-r2 \
		libisoburn=1.5.4-r2 xorriso=1.5.4-r2 && \
	adduser build \
		-G abuild \
		-D -s /bin/bash \
		-h /build && \
	echo "%abuild ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/abuild && \
	su build -c "abuild-keygen -a -i -n" && \
	mv /build/.abuild /

VOLUME build
USER build
ENV USER=build

ENTRYPOINT [ "/entrypoint.sh" ]
CMD [ "/build/aports/scripts/start.sh" ]
Build can be triggered by running following command inside docker container. Scripts above are just adjusted for additional environment check.

Code: Select all

./mkimage.sh --tag test --outdir ~/iso --arch x86_64 --repository http://dl-cdn.alpinelinux.org/alpine/v3.16/main --repository http://dl-cdn.alpinelinux.org/alpine/v3.16/community --profile base
And once again - comparison "table"

Code: Select all

OS  |   vCPU  |  Memory  |  Kernel  |  Docker Version  |  Build time  |
CentOS 9  |  4  |  8GB  |  5.14.0-160  |  20.10.18  |  20-22min  |
Debian Bullseye  |  4  |  8GB  |  5.14.0-0.bpo.2  |  20.10.18  |  5-6min  |

PryMar56
Posts: 17
Joined: 2020/04/20 14:56:31

Re: Huge performance difference on specific task

Post by PryMar56 » 2022/12/12 21:51:33

Could the Deb 11 env have a working proxy?
The mkimage.sh command also includes the time to run the Docker container? that involves package downloads and maybe
the Deb 11 is taking advantage of cached packages.

Debian 11 is quite good, the only shortcoming I've found is it won't support zstd dpkg compression that is now in Ubuntu Jammy and higher.

Post Reply