Fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Support for security such as Firewalls and securing linux
Post Reply
weej
Posts: 8
Joined: 2020/01/18 23:18:27
Location: Dorset, UK

Fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by weej » 2020/01/19 00:29:21

Hi, CentOS newbie and first time poster here. Have played with Ubuntu in the past but really a Linux novice by most standards. I have researched as much as I could about my issue and have educated myself on the concept of Meltdown & Spectre vulnerabilities and common mitigation measures.

I have bought the following 2nd hand server machine:
HP Proliant ML330 Gen6 E5504 2.0GHz Quad Core, 12GB RAM installed

I installed CentOS 8 on a SATA SSD connected to a SATA port on the motherboard, since I plan to buy the SAS drives for RAID array once I am happy that I can get the machine + OS working properly.

During OS installation from USB I did notice the GUI reacting slowly (2-3 sec page transitions). This was not the case during a trial installation of CentOS 8 using the same media on a laptop of far inferior spec, which operated quickly and smoothly.

Immediately following installation on the server I could see that it was running slow to the point of being unusable. I cannot move the mouse pointer more than a few millimeters at a time and typing speed in Terminal is about one character for every 5 seconds. This was without any other applications running (I haven't even gotten as far as installing any).

BIOS firmware version was dated 2010. I have now updated it to the very last version released (2018), but it has made no difference.

I just about managed to generate the following cpuinfo. Apologies for the bad photo. I might have been able to copy & paste but it would have taken a day... I've had to type this post on my Windoze PC.
IMG_2344.jpg
IMG_2344.jpg (90.19 KiB) Viewed 1364 times
Am I right in believing that the most likely cause of the slow performance is mitigation of the following vulnerability?

Code: Select all

cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
I see that patches were applied to CentOS 7 to employ mitigation measures. I see no mention of them in posts about CentOS 8, but I assume they have been included? I can try CentOS 7 and 6, however I'm concerned that I'll be using now out-of-date OS versions.

I see that it may be possible to disable Spectre, Meltdown and L1TF mitigations but surely that would put my machine at risk, so would there be any way of both preventing mitigation in the form of slowing down of CPU and keeping the system safe? If so can anyone please advise how?

I have read that vulnerability mitigation measures can slow the CPU down by up to 20% but this feels like 95%, so I'm sure there is something else to it...

Many thanks in advance.
Last edited by weej on 2020/01/19 21:26:44, edited 2 times in total.

hunter86_bg
Posts: 2016
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: 2nd Hand Server w/ fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by hunter86_bg » 2020/01/19 07:23:52

Hmm... About the screen shot -> you can go to iLo and if you have an advanced iLO license, you can then:
ssh to ilo -> textcons -> and because no gui -> you will be able to access the console without issues and copy/paste.

Did you use latest HPSUM dvd to flash firmare and/or kernel drivers ?

About the slow stuff. First, if possible check iLO event log for issues. AHS log analyzer is available for anyone with HPE account, so you can upload it and check the health of the system.

Check if you can run 'HP Insight Diagnostics' which can tell you if the system has any hardware issues.

Note: AHS log analyzer doesn't work with logs over 200MB, so you might have to reduce the analysis timeframe when generating the AHS.

Also test with CentOS 7, as I'm pretty sure RHEL8 was not validated by HPE and might have issues with newer kernels. If CentOS 7 is sluggish, check with CentOS 6 which should be OK.

weej
Posts: 8
Joined: 2020/01/18 23:18:27
Location: Dorset, UK

Re: 2nd Hand Server w/ fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by weej » 2020/01/19 09:54:25

Thanks for you reply hunter86_bg!
hunter86_bg wrote:
2020/01/19 07:23:52
Hmm... About the screen shot -> you can go to iLo and if you have an advanced iLO license, you can then:
ssh to ilo -> textcons -> and because no gui -> you will be able to access the console without issues and copy/paste.
I'll have to check if iLO is licensed and enabled and see if I can do this.
hunter86_bg wrote:
2020/01/19 07:23:52
Did you use latest HPSUM dvd to flash firmare and/or kernel drivers ?
I created a bootable USB and installed this https://support.hpe.com/hpsc/swd/public ... 885cd45e7c
Should I have also done something else?
hunter86_bg wrote:
2020/01/19 07:23:52
About the slow stuff. First, if possible check iLO event log for issues. AHS log analyzer is available for anyone with HPE account, so you can upload it and check the health of the system.

Check if you can run 'HP Insight Diagnostics' which can tell you if the system has any hardware issues.

Also test with CentOS 7, as I'm pretty sure RHEL8 was not validated by HPE and might have issues with newer kernels. If CentOS 7 is sluggish, check with CentOS 6 which should be OK.
Will have a look and try all of these.

Thanks

User avatar
TrevorH
Forum Moderator
Posts: 28884
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 2nd Hand Server w/ fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by TrevorH » 2020/01/19 11:33:18

The meltdown etc issues should be sucking ~20% or less from your machine and really shouldn't be noticeable on an otherwise idle system that's doing nothing. I do not think the problems you have are anything to do with this.

From a command line prompt, run `top` and see what is using the cpu, if anything. I'd also look at booting in command line only mode (append a space followed by the digit 3 to the kernel command line at the grub prompt at boot time) and see if the problems persist. I am mostly suspecting you have video issues rather than anything else - the video cards supplied with server class hardware generally dates from about the mid-1990s and is not suitable for running high res graphics.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

weej
Posts: 8
Joined: 2020/01/18 23:18:27
Location: Dorset, UK

Re: 2nd Hand Server w/ fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by weej » 2020/01/19 14:49:02

TrevorH wrote:
2020/01/19 11:33:18
The meltdown etc issues should be sucking ~20% or less from your machine and really shouldn't be noticeable on an otherwise idle system that's doing nothing. I do not think the problems you have are anything to do with this.

From a command line prompt, run `top` and see what is using the cpu, if anything. I'd also look at booting in command line only mode (append a space followed by the digit 3 to the kernel command line at the grub prompt at boot time) and see if the problems persist. I am mostly suspecting you have video issues rather than anything else - the video cards supplied with server class hardware generally dates from about the mid-1990s and is not suitable for running high res graphics.
Thanks TrevorH. I will try what you have suggested above. I don’t believe it’s just a graphics issue though, since I even struggle to enter commands in Terminal. For example ‘su -‘ can be taken as ‘ssss-’ (several seconds after typing it). Hence it takes me several attempts to authenticate for root as you don’t get blank character feedback.

hunter86_bg
Posts: 2016
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: 2nd Hand Server w/ fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by hunter86_bg » 2020/01/19 16:08:54

Keep in mind that by default HPE ship with power savings enabled to the max.
Find the relevant options and set it to 'Static High Performance' (or it's equivalent).
Also, ensure that tuned profile is set to 'throughput-performance'.

You have 2 options:
1. Boot from iLO the latest SPP
2. Copy the iso to the server , mount and copy the contents to another folder. Then start the hpsum.sh (not sure about the name) and access the port, it will tell you on the console, via web browser (avoid IE). Then open the web interface, imput root user and it's password and start the process to add localhost and select the baseline of the iso.

Once it analyzes your system - it will recommend firmware/software + kernel modules.

The procedure is also valid if you want to flash the firmware on multiple machines (avoid mixing localhost + other systems).

Edit: It seems yoiu didn't use the SPP. Find and boot from the iso, but be careful to select the iso for your system.

Edit2: Most probably this is the SPP for Gen6 systems, but I'm not sure - boot in interactive mode .

User avatar
TrevorH
Forum Moderator
Posts: 28884
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 2nd Hand Server w/ fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by TrevorH » 2020/01/19 17:38:32

I'm going to guess that something is generating an interrupt storm. Post the contents of /proc/interrupts along with how long the system has been up (uptime).
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

weej
Posts: 8
Joined: 2020/01/18 23:18:27
Location: Dorset, UK

Re: Fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by weej » 2020/01/19 21:59:14

Thanks to you both. I will try those things in the next couple of days.

I can't download the HP Service Pack for ProLiant because:
An active warranty or support agreement covering Proliant servers must be linked to your HPE Support Center profile to access this application.
...and I have neither warranty or a support agreement. I have tried accessing the the HP ftp server but cannot find anything in it.

Anyone know how I can get hold of the SPP without paying for a support contract?

weej
Posts: 8
Joined: 2020/01/18 23:18:27
Location: Dorset, UK

Re: Fresh Centos 8 installation unusably slow, reports Meltdown/Spectre

Post by weej » 2020/01/21 21:29:58

Ok, so given that:
1. I don't know if iLO is licensed or enabled
2. I'm allowed to use AHS log analyser
3. I'm not allowed to have the SPP
3. I really struggled to boot in command line mode, tried to add 3 to the startup command line as well as lots of other suggested way, unsuccessfully
4. I downloaded CentOS 7 ready to try installing it, but remembered I had a freshly installed Windows 10 hard disk sitting on the desk waiting to go into another PC...

I thought I'd slip the WIndows 10 HDD in and try to boot from it to see any difference as it was the quickest thing to try before running the CentOS 7 install.

Windows 10 booted very quickly, apart from having to install some hardware drivers during startup. Once booted, everything ran v smoothly, so great sign! The hardware must be OK.

I installed CentOS 7 and it now runs very well! If there is any vulnerability mitigation happening, it doesn't seem to noticeably impact the system.

So great, I can carry on with CentOS 7, however am I going to be at a disadvantage running an OS that is not the latest will at some point cease to be supported?

Thank you both for your advice.

Post Reply

Return to “CentOS 8 - Security Support”