CentOS

Backup and compile jobs needs large times because at random files utimes() syscalls have a 90 second delay. (CentOS-7.6, xfs filesystem). What's going up here?

How are you measuring this?
Is it a VM? and if so, what is the underlying hardware and how overcommitted is it?
If it is not a VM then what hardware?
Also, what is the output from uname -r

Kernel is 3.10.0-957.12.1.el7.x86_64, Hardware is Intel S2600WFT Board with Dual Xeon Silver 4114, 96 GB RAM, filesystem is xfs (exported by nfs).

At a guess NFS will be the problem here ("NFS is always the problem" - my new trade mark).

utimes() updates the file's last access and modification times.

AFAIK NFS is always synchronous (and yes, many very clever people over many years have tried to solve this is a "generic" way and have failed IMO).
Start looking at your networking (that's what I would do).
Or don't depend on file change meta-data for your backup, which may be hard, depending on your use case.

You might be able to mask last access change (don't care about access in a backup case), rather than mask modification time. Off hand, I do not know how to do (and don't really care). Although, given they are the same syscall, it somehow seems unlikely - although somebody else may just know the answer (if I cared, I'd start looking at mount -o options).

Can you try mounting the NFS with 'noatime,nodiratime' options and try again ?
Yet, you will loose access times and that could pose a problem for some software.

mount options for most clients are always (from /proc/mounts): But some clients (with old mounts) have only relatime