SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Issues related to applications and software problems and general support
feacluster
Posts: 23
Joined: 2017/11/28 14:48:49

SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by feacluster » 2021/11/29 02:47:50

I am using google cloud and created a CentOS 8 stream VM. I ran the following commands and was able to launch a remote desktop on the VM. But after restarting the instance I can not ssh into it. The ssh times out. Here are the commands I entered. My suspicion is that it is booting into some graphical login mode and somehow ssh is disabled?

Code: Select all

gcloud compute firewall-rules create vncserver --allow tcp:5900-5902 --source-tags=instance-4 --source-ranges=0.0.0.0/0 --description="Server with GUI"
dnf groupinstall "Server with GUI"
dnf install tigervnc-server tigervnc-server-module -y

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by jlehtone » 2021/11/29 08:36:32

I don't know gcloud. Does it provide a firewall? If yes, does it allow ssh?
Can you ssh in before you install environment group "Server with GUI" and tigervnc?
Can you ssh in before restart?

feacluster
Posts: 23
Joined: 2017/11/28 14:48:49

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by feacluster » 2021/11/29 19:29:39

Yes, I was able to ssh into it without any issue until I installed "Server with GUI" and rebooted. I did some investigation and found out that upon reboot the network was not being enabled somehow.

It looks like dnf groupinstall "Server with GUI" installs the "networkmanager-config-server" package, which intentionally disables autoconfiguration of network interfaces. I'm not familiar with CentOS, but I'm guessing they intended for administrators to configure network interfaces explicitly on servers? In any case, eth0 simply receives no configuration, so the VM's networking becomes totally broken!

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by jlehtone » 2021/11/29 20:05:14

That's interesting.
Can you do:

Code: Select all

nmcli c s
nmcli -f connection c s name-of-troublesome-connection
The "connection.autoconnect" is of special interest.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by TrevorH » 2021/11/29 20:34:39

I did some investigation and found out that upon reboot the network was not being enabled somehow.
Unless the install process uses the network, that is how it is. The default is to not start any network connection. It's only if you enable it during the install that it is enabled after.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

feacluster
Posts: 23
Joined: 2017/11/28 14:48:49

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by feacluster » 2021/11/30 14:16:47

TrevorH wrote:
2021/11/29 20:34:39
I did some investigation and found out that upon reboot the network was not being enabled somehow.
Unless the install process uses the network, that is how it is. The default is to not start any network connection. It's only if you enable it during the install that it is enabled after.
The network was enabled when I created the VM. Somehow it got disabled after installing "Server with GUI". I will try and reproduce the issue with a new VM and report back.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by jlehtone » 2021/11/30 14:55:44

networkmanager-config-server wrote:This adds a NetworkManager configuration file to make it behave more like the old "network" service. In particular, it stops NetworkManager from automatically running DHCP on unconfigured ethernet devices, and allows connections with static IP addresses to be brought up even on ethernet devices with no carrier.
The network connection that was ok after installation is a "configured device" (configuration created by installer).

If there is a VM and one attaches (hotplugs) additional interfaces to it, then those are unconfigured devices.
NetworkManager will autoconfigure and start connections on them, if networkmanager-config-server is not installed.
You don't want that, if the defaults are not ok for you. On server they usually are not.

Installation of additional packages, like networkmanager-config-server, should have no effect on connections.

Unless ... what packages were installed before you add the ones listed in "Server with GUI" dnf group?
What if network and firewall are configured and managed by different services than what are implicitly pulled in by "dnf install lots_of_packages"?
The new services would be on defaults and might be mutually exclusive with original.

Example: Another OpenStack provider has images with no firewall (because OpenStack does the filtering). No firewalld. I did, by mistake, pull firewalld in as dependency. While ssh was still allowed, other traffic was cleanly cut.

You can also do:

Code: Select all

dnf group info "Server with GUI"
That lists the dnf groups that are "part of" Server with GUI environment. You might not need all of them, nor all packages from each group.
A dnf group is just a list, not hard requirement. One can:

Code: Select all

dnf group install foo --exclude=bar
Even if 'bar' is a mandatory package for group 'foo', the other packages (that do not require 'bar') still install without error.

feacluster
Posts: 23
Joined: 2017/11/28 14:48:49

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by feacluster » 2021/12/01 02:42:38

jlehtone wrote:
2021/11/29 20:05:14
That's interesting.
Can you do:

Code: Select all

nmcli c s
nmcli -f connection c s name-of-troublesome-connection
The "connection.autoconnect" is of special interest.
I created a fresh CentOS VM on google cloud and only ran "dnf groupinstall "Server with GUI"". Next I installed tigervnc-server followed by the commands you suggested below:

Code: Select all

[feacluster@centos8 ~]$ nmcli c s
NAME                UUID                                  TYPE      DEVICE
Wired connection 1  xxxx-xxx-xxxx-xxx-xxx ethernet  eth0
[feacluster@centos8 ~]$ nmcli -f connection c s 'Wired connection 1'
connection.id:                          Wired connection 1
connection.uuid:                        xxxx-xxx-xxxx-xxx-xxx
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              eth0
connection.autoconnect:                 yes
connection.autoconnect-priority:        -999
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1638326206
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
connection.llmnr:                       -1 (default)
connection.wait-device-timeout:         -1
[feacluster@centos8 ~]$

feacluster
Posts: 23
Joined: 2017/11/28 14:48:49

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by feacluster » 2021/12/01 03:16:33

Next, I added this line to: /etc/NetworkManager/NetworkManager.conf

Code: Select all

no-auto-default=
I rebooted several times and was able to ssh back in with no problem. I then commented that line out and did a reboot. Then it locked me out... Still not sure I understand if this is a bug or a feature :?

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: SSH times out after I installed "Server with GUI" on a CentOS 8 stream VM

Post by jlehtone » 2021/12/01 07:45:39

The 'networkmanager-config-server' adds only one file:

Code: Select all

# grep ^[^#] /usr/lib/NetworkManager/conf.d/00-server.conf
[main]
no-auto-default=*
ignore-carrier=*
man NetworkManager.conf wrote:If a default NetworkManager.conf is provided by your distribution's packages, you should not modify it, since your changes may get overwritten by package updates. Instead, you can add additional .conf files to the /etc/NetworkManager/conf.d directory. These will be read in order, with later files overriding earlier ones.
One should therefore create /usr/lib/NetworkManager/conf.d/01-my.conf rather than edit /etc/NetworkManager/NetworkManager.conf.

However, everything points out to 'Wired connection 1' being an autogenerated connection. Not the one that you had originally, before installing more packages.

When was NetworkManager installed? You see that with:

Code: Select all

dnf history list NetworkManager
All dnf transactions:

Code: Select all

dnf history
IF groupinstall "Server with GUI" is the earliest transaction that involved NetworkManager,
THEN the network was not configured with NetworkManager and you should not install NetworkManager at all.
That is, should not install the entire Server with GUI subset of packages.

Post Reply