CPU throttling and ATLAS compile.

Issues related to hardware problems
PhillHS
Posts: 10
Joined: 2017/04/03 09:28:10

CPU throttling and ATLAS compile.

Post by PhillHS » 2021/07/15 10:40:50

Hi all,

I'm trying to compile a custom version of ATLAS for our cluster. However it's falling over because I don't seem to be able to disable CPU throttling.

Centos is : CentOS Linux release 7.9.2009

I've tried adding intel_pstate=disable to /etc/default/grub :

Code: Select all

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_pstate=disable"
GRUB_DISABLE_RECOVERY="true"
Refreshing grub with : grub2-mkconfig -o /boot/grub2/grub2.cfg and re-booting

However though the intel_pstate is inserted into /boot/grub2/grub2.cfg :

Code: Select all

 grep intel_ /boot/grub2/grub2.cfg 
        linux16 /boot/vmlinuz-3.10.0-1160.15.2.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-1127.10.1.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-514.16.1.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-327.3.1.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-327.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-0-rescue-7a3a745b3dac481ba6b56dbda65ab39c root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
It seems to be being ignored at boot time :

Code: Select all

 cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.15.2.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet LANG=en_GB.UTF-8
And ATLAS config still moans about throttling being enabled.

I also tried running cpupower frequency-set -g performance which set all the cpu threads, that didn't help either.

For info

Code: Select all

cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.20 GHz - 2.90 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.20 GHz and 2.90 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 1.66 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes
    2800 MHz max turbo 4 active cores
    2800 MHz max turbo 3 active cores
    2900 MHz max turbo 2 active cores
    2900 MHz max turbo 1 active cores
Any idea what I'm doing wrong? Machine is a Dell Poweredge R420 with 2x Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz installed.

Cheers.

Phill.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CPU throttling and ATLAS compile.

Post by TrevorH » 2021/07/15 13:37:53

Why would it care? Sounds like a really stupid check in the build script and the author should be taken out behind the bike sheds and given a good kicking!

Meanwhile, this appears to be answered in viewtopic.php?t=68958
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

PhillHS
Posts: 10
Joined: 2017/04/03 09:28:10

Re: CPU throttling and ATLAS compile.

Post by PhillHS » 2021/07/15 14:50:03

TrevorH wrote:
2021/07/15 13:37:53
Why would it care? Sounds like a really stupid check in the build script and the author should be taken out behind the bike sheds and given a good kicking!
I believe that as part of the build process it needs to some timing calibration which requires it to be sure that the CPU is running as fast as it can.
Meanwhile, this appears to be answered in viewtopic.php?t=68958
I saw that thread and as can be seen from the logs in my original post I have *PUT* intel_pstate=disable in my grub config and rebooted, but it seems to be being ignored by the kernel.
It is possibe that something has changed between Centos 7.4 which was being used in that thread and 7.9 which I am using, certainly last time I did this it did work.

Cheers.

Phill.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CPU throttling and ATLAS compile.

Post by TrevorH » 2021/07/15 14:59:17

Read further down the thread, it says that if you disable that one then it loads another acpi version of the same thing that you have to modprobe -r.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

PhillHS
Posts: 10
Joined: 2017/04/03 09:28:10

Re: CPU throttling and ATLAS compile.

Post by PhillHS » 2021/07/19 09:05:53

TrevorH wrote:
2021/07/15 14:59:17
Read further down the thread, it says that if you disable that one then it loads another acpi version of the same thing that you have to modprobe -r.
But I have read all the way to the bottom, to recap :

I have intel_pstate=disable in my grub files, but it's not appearing in the linux command line.

Also the intel_pstate drive is still loaded.

Code: Select all

# cpupower frequency-info 
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.20 GHz - 2.90 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.20 GHz and 2.90 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 1.65 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes
    2800 MHz max turbo 4 active cores
    2800 MHz max turbo 3 active cores
    2900 MHz max turbo 2 active cores
    2900 MHz max turbo 1 active cores
The point is I *CAN'T* get pstate to disable even by putting the intel_pstate=disable in grub.

Cheers.

Phill.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CPU throttling and ATLAS compile.

Post by TrevorH » 2021/07/19 10:04:34

So the bit in that thread about needing to rmmod acpi-cpufreq only appears in my copy of it?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

PhillHS
Posts: 10
Joined: 2017/04/03 09:28:10

Re: CPU throttling and ATLAS compile.

Post by PhillHS » 2021/07/19 13:16:51

TrevorH wrote:
2021/07/19 10:04:34
So the bit in that thread about needing to rmmod acpi-cpufreq only appears in my copy of it?
I Had read and tried that but just to be sure I tried it again :

Code: Select all

[root@exec5r420 ~]# rmmod acpi-cpufreq
rmmod: ERROR: Module acpi_cpufreq is not currently loaded
Also in that thread after putting intel_pstate=disable in the grub config file, rebuilding and restarting they where getting the " driver: acpi-cpufreq" from cpupower frequency-info, I am *NOT* under the same circumstances.

This is what I currently have :

Code: Select all


# uname -a
Linux exec5r420 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

# grep intel_pstate /boot/grub2/grub2.cfg 
        linux16 /boot/vmlinuz-3.10.0-1160.15.2.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-1127.10.1.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-514.16.1.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-327.3.1.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-3.10.0-327.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 
        linux16 /boot/vmlinuz-0-rescue-7a3a745b3dac481ba6b56dbda65ab39c root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable 

# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.15.2.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet LANG=en_GB.UTF-8

# cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.20 GHz - 2.90 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.20 GHz and 2.90 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 2.46 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes
    2800 MHz max turbo 4 active cores
    2800 MHz max turbo 3 active cores
    2900 MHz max turbo 2 active cores
    2900 MHz max turbo 1 active cores

# rmmod acpi_cpufreq
rmmod: ERROR: Module acpi_cpufreq is not currently loaded
The other thred did these so I'll include them for completeness....

Code: Select all

# modprobe --showconfig | grep cpufreq
alias acpi acpi_cpufreq
alias acpi*:ACPI0007:* acpi_cpufreq
alias acpi*:ACPI0007:* pcc_cpufreq
alias acpi*:LNXCPU:* acpi_cpufreq
alias acpi*:LNXCPU:* pcc_cpufreq
alias x86cpu:vendor:*:family:*:model:*:feature:*0016* acpi_cpufreq
alias x86cpu:vendor:*:family:*:model:*:feature:*00E8* acpi_cpufreq

#  modprobe acpi-cpufreq -v
insmod /lib/modules/3.10.0-1160.15.2.el7.x86_64/kernel/drivers/cpufreq/acpi-cpufreq.ko.xz 
Which *seems* to be the same results as the person in the other thread was getting......

Oddly I have had this working before (the last time I compiled ATLAS), but it may have been on a different node in our cluster (though with identical hardware) which is perplexing.

Now here's the odd thing, I've found the node I did this on before and intel_pstate=disable *IS* appeaing in it's command line and on that node

Code: Select all

# cpupower frequency-info
analyzing CPU 0:
  no or unknown cpufreq driver is active on this CPU
  CPUs which run at the same hardware frequency: Not Available
  CPUs which need to have their frequency coordinated by software: Not Available
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: Not Available
  available cpufreq governors: Not Available
  Unable to determine current policy
  current CPU frequency: Unable to call hardware
  current CPU frequency:  Unable to call to kernel
  boost state support:
    Supported: yes
    Active: yes
    2800 MHz max turbo 4 active cores
    2800 MHz max turbo 3 active cores
    2900 MHz max turbo 2 active cores
    2900 MHz max turbo 1 active cores

# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.15.2.el7.x86_64 root=UUID=bbadbeee-a193-4284-8f95-5ee70502b12c ro crashkernel=auto rhgb quiet intel_pstate=disable LANG=en_GB.UTF-8
This at least means I should be able to get ATLAS compiled, but as to why it's failing on a machine with the same hardware and software configuration (as far as I can tell, the machines are at a remote location, so I can't check the bios settings).

Cheers.

Phill.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CPU throttling and ATLAS compile.

Post by TrevorH » 2021/07/19 14:49:46

What's the content of /proc/cmdline?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: CPU throttling and ATLAS compile.

Post by jlehtone » 2021/07/19 16:12:42

Do you have EFI in use? EFI has grub config in different place:

Code: Select all

# ls -ld /etc/grub2*
lrwxrwxrwx. 1 root root 22 Mar 19 12:25 /etc/grub2.cfg -> ../boot/grub2/grub.cfg
lrwxrwxrwx. 1 root root 31 Mar 19 12:25 /etc/grub2-efi.cfg -> ../boot/efi/EFI/centos/grub.cfg
Furthermore, you did add the option by editing /etc/default/grub and running
grub2-mkconfig -o $appropriate_grub_cfg, did you?

PhillHS
Posts: 10
Joined: 2017/04/03 09:28:10

Re: CPU throttling and ATLAS compile.

Post by PhillHS » 2021/07/19 18:10:53

jlehtone wrote:
2021/07/19 16:12:42
Do you have EFI in use? EFI has grub config in different place:

Code: Select all

# ls -ld /etc/grub2*
lrwxrwxrwx. 1 root root 22 Mar 19 12:25 /etc/grub2.cfg -> ../boot/grub2/grub.cfg
lrwxrwxrwx. 1 root root 31 Mar 19 12:25 /etc/grub2-efi.cfg -> ../boot/efi/EFI/centos/grub.cfg
Furthermore, you did add the option by editing /etc/default/grub and running
grub2-mkconfig -o $appropriate_grub_cfg, did you?
I'm pretty sure I'm not using EFI and yeah I edited /etc/default/grub and then ran grub2-mkconfig.

Cheers.

Phill.

Post Reply