Centos75 kernel-3.10.0-862 impact CPU max MHz on Intel E5-2620v4

[jizh]

We encountered a critical issue when upgrade kernel from 3.10.0-693.17.1.el7 to kernel-3.10.0-862.14.4.el7 on a server with CPU E5-2620 v4.
After upgrade, the CPU max MHz was changed from 3000 to 2100 and system performance downgrade so much.


Attempt the following :
All kernel running with performance governor.
cpupower frequency-set -u 3000MHz
OR
/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable acpi=force”
OR
cat /sys/devices/system/cpu/intel_pstate/no_turbo
1
Enabled Intel Turbo Boost
But looks not take effect for changing.



kernel-3.10.0-693
Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping: 1
CPU MHz: 2099.752
CPU max MHz: 3000.0000
CPU min MHz: 1200.0000



kernel-3.10.0-862
Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping: 1
CPU MHz: 2099.615
CPU max MHz: 2100.0000
CPU min MHz: 1200.0000


I tested on the following kernels, and only all 862 kernel will change the max MHZ.
3.10.0-693.21.1.el7 3.10.0-693.17.1.el7 3.10.0-862.el7.x86_64 3.10.0-862.el7.x86_64 kernel-3.10.0-862.14.4.el7.x86_64






In addition, tested on another CPU, no impact on CPU E5-2667 v4. I installed kernel 693 and upgrade to 862, there is no change on max MHZ.

Model name: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
Stepping: 1
CPU MHz: 1203.320
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000







Just wondering if there is some document explaining the issue, or is there a way we can adjust the CPU max MHZ after upgraded to 3.10.0-862 kernel ?

Appreciate for any help.
Thanks

[TrevorH]

Try changing your BIOS settings for power management to 'performance' or 'maximum' or whatever your BIOS vendor terms it.

[jizh]

Thanks a lot, I'll take a try.
But a bit confused why it is not appeared in kernel 3.10.0-693*, is there any changed on kernel 893 about that I can search on?

[jlehtone]

That is a good question, but seems like a waste of time. You should first check with kernel-3.10.0-957.1.3.el7.x86_64 for it is the current kernel version.

[jizh]

Thanks Jiehtone.

I tried the kernel 3.10.0-957.1.3.el7.x86_64 and seems same with kernel-3.10.0-862.

[jizh]

Looks the kernel in centos7_5 and centos7_6 always can limit the CPU max MHz without Turbo mode .

And I tested in BIOS, we have to forcely set the Turbo mode enabled. OS/Kernel doesn't seem to be able to adjust intel_pstate as needed.


Not quite sure why this happened since centos7_5?


But thanks for your suggestion TrevorH and Jlehtone

[jizh]

After enabled Turbo mode in BIOS, I just did sysbench test again on the following three of kernels.
3.10.0-693.21.1.el7.x86_64
3.10.0-862.14.4.el7.x86_64
3.10.0-957.1.3.el7.x86_64


Now, the CPU max MHz/cpu feq range on the three kernels are same.

Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping: 1
CPU MHz: 2299.992
CPU max MHz: 3000.0000
CPU min MHz: 1200.0000


analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 1.20 GHz - 3.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 1.20 GHz and 3.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: 2.30 GHz (asserted by call to hardware)
boost state support:
Supported: yes
Active: yes


BUT, no lucks, 3.10.0-862 and 3.10.0-957 still quit slower than 3.10.0-693.21.1.el7.x86_64.

Could anyone help on other methods we can attempt to adjust?

[jlehtone]

is there any changed on kernel 893 about that I can search on?

Have a look on:

Code: Select all

rpm -q --changelog kernel-3.10.0-862.14.4.el7.x86_64 | less

At least the Wed Sep 20 2017 [3.10.0-712.el7] commit is full of [cpufreq] intel_pstate notes
and it is not even the only such commit between 3.10.0-693.21.1 and 3.10.0-862.

[jizh]

Ah, seem I find the reason why 3.10.0-862 and 3.10.0-957 quit slower than 3.10.0-693.21.1.el7.x86_64 on my test.

It is the case that kernel set my NIC parameter "busy-poll" to OFF
Kernel 7_5 7_6:

Code: Select all

# ethtool -k eno2 | grep "busy-poll"
busy-poll: off [fixed]

Kernel 7_4:

Code: Select all

# ethtool -k eno2 | grep "busy-poll"
busy-poll: on [fixed]

Because I'm using network-latency profile, and there are two setting related with

Code: Select all

net.core.busy_read=50
net.core.busy_poll=50

If I remove them in profile or change the both to 0, the performance of sysbench test will clearly be back to normal.


So for now, at least I have had a way to adjust the performance on new kernel(7_5/7_6).

But still confused the following:
why kernel change busy-poll from on to off and how can enable busy-poll on new kernel(7_5/7_6) ?



Appreciate for any help !!

[chemal]

I'm using network-latency profile

You declare that minimal network latency is most important to you and then you benchmark something else and complain?