Multiport Serial Card no longer seeing ports

Issues related to hardware problems
Commtech_Josh
Posts: 5
Joined: 2020/07/16 16:23:05

Multiport Serial Card no longer seeing ports

Post by Commtech_Josh » 2020/07/20 20:23:34

Hello!

My Linux knowledge is weak, passable at best, so please bare with me.

I'm currently supporting/maintaining the drivers for some of our devices (the 422/x-PCIe series, Vendor ID 18f7 and PCI IDs 0020 and 0021) and I've found that on CentOS 8 the serial ports no longer show up. The cards still appear and are properly identified by lspci, but the ports aren't detected correctly.

I spent some time looking through the kernel source and comparing the dmesg output with a working system running an older Ubuntu, and it looks like the card isn't getting properly detected. There's no detection of ttyS ports in CentOS 8 even after setting nr_uarts=20. The ttyS ports are generated and I can see them in /dev/ttyS*, but they aren't associated with our card in any way.

Here's information from our older Ubuntu system:
lspci:
02:00.0 Serial controller: Commtech, Inc. 422/4-PCIe Serial PCIe Adapter [Fastcom] (rev 03)
lspci -n:
02:00.0 0700: 18f7:0020 (rev 03)
dmesg | grep "tty":
[ 0.000000] console [tty0] enabled
[ 0.542114] 00:0a: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 0.542871] 0000:02:00.0: ttyS4 at MMIO 0xefd00000 (irq = 17, base_baud = 7812500) is a XR17V35X
[ 0.542928] 0000:02:00.0: ttyS5 at MMIO 0xefd00400 (irq = 17, base_baud = 7812500) is a XR17V35X
[ 0.542978] 0000:02:00.0: ttyS6 at MMIO 0xefd00800 (irq = 17, base_baud = 7812500) is a XR17V35X
[ 0.543028] 0000:02:00.0: ttyS7 at MMIO 0xefd00c00 (irq = 17, base_baud = 7812500) is a XR17V35X
uname --all:
Linux lab-All-Series 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

From CentOS 8:
lspci:
02:00.0 Serial controller: Commtech, Inc. 422/4-PCIe Serial PCIe Adapter [Fastcom] (rev 03)
lspci -n:
02:00.0 0700: 18f7:0020 (rev 03)
dmesg | grep "tty":
[ 0.001000] printk: console [tty0] enabled
[ 0.686825] 00:05: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 0.707552] 0000:00:16.3: ttyS4 at I/O 0xf0e0 (irq = 19, base_baud = 115200) is a 16550A
uname --all:
Linux localhost.localdomain 4.18.0-193.6.3.el8_2.x86_64 #1 SMP Wed Jun 10 11:09:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

This issue was brought to my attention by a customer and I'm having some difficulty understanding what the problem might be. I would appreciate it if someone could shed some insight or offer some suggestions on how to resolve this.

Thanks!

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Multiport Serial Card no longer seeing ports

Post by TrevorH » 2020/07/20 20:48:02

You're obviously missing a driver for the card or you need to tell the driver that it can support that card. The fact that it appears in lspci just means the card is installed in the PCI slot correctly - lspci lists (pci) hardware that is attached and doesn't care if it has a driver or not.

Since you didn't use the -nn switch to lspci I have to guess that it is 10e8:82c5 and as far as I can see, nothing supports that card in either CentOS 7 or 8.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Commtech_Josh
Posts: 5
Joined: 2020/07/16 16:23:05

Re: Multiport Serial Card no longer seeing ports

Post by Commtech_Josh » 2020/07/20 21:04:30

The card is 18f7:0020.

We are the driver manufacturers for this card, and yes, the final drivers weren't installed. From my (limited) understanding, the core functionality such as read/write should be handled by the default serial drivers, while our particular drivers provide support for functionality unique to our devices such as custom clock speed. This is how its worked since 2013, but suddenly it seems that the core serial driver is no longer recognizing our device(s).

For example, these lines in the Linux 8250_exar.c files in the linux repo:
https://github.com/torvalds/linux/blob/ ... exar.c#L40
https://github.com/torvalds/linux/blob/ ... xar.c#L805

This is why I'm surprised - I don't know enough to understand why it would no longer function like that.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Multiport Serial Card no longer seeing ports

Post by TrevorH » 2020/07/20 21:50:13

Well on Debian I see that 8250_exar.ko is supplied as part of the kernel. On CentOS that is not the case, at least as far as I can see on both CentOS 7 and 8. However, on 8 I see

Code: Select all

$ grep -i exar /boot/config-4.18.0-193.6.3.el8_2.x86_64 
# CONFIG_NET_VENDOR_EXAR is not set
CONFIG_SERIAL_8250_EXAR=y
# CONFIG_GPIO_EXAR is not set
So something is built for it, just not sure _what_. There's a bunch of exar_ symbols show up in /boot/System.map as well. I think a lot of the serial code is probably built as part of the kernel itself and not as modules. A quick grep for 'serial' through /boot/config-$(uname -r) shows a lot of SERIAL options are turned on and some are modules and some are built-in. I don't know enough about it to know which bits are which.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Commtech_Josh
Posts: 5
Joined: 2020/07/16 16:23:05

Re: Multiport Serial Card no longer seeing ports

Post by Commtech_Josh » 2020/07/22 18:57:18

I'm at a bit of a loss. If I understand the Makefiles, CONFIG_SERIAL_8250_EXAR should be enabling the exar part of the serial drivers, but the kernel doesn't seem to even notice the UARTs on our card. Even though it clearly and correctly identifies the card as our card (Commtech, Inc. 422/4-PCIe Serial PCIe Adapter).

As I mentioned before, I'm relatively inexperienced with Linux. Is there some way to see the final 'source' of CentOS 8 to see if the code we relied on has been removed or disabled? I've tried following the guide for "If you really need the full kernel source" section of the CentOS guide (https://wiki.centos.org/HowTos/I_need_the_Kernel_Source) for CentOS 7 because there was no guide for CentOS 8, but I just get Kconfig and makefiles and empty directories.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Multiport Serial Card no longer seeing ports

Post by TrevorH » 2020/07/22 22:32:31

Even though it clearly and correctly identifies the card as our card (Commtech, Inc. 422/4-PCIe Serial PCIe Adapter).
You mean from lspci output? If so that doesn't mean anything at all. That just reports the devices that exist on the PCI[e] bus and looks up their vendor/device ids in /usr/share/hwdata/pci.ids. It doesn't mean anything whatsoever about whether the kernel supports the devices or not unless you're running lspci -vvv and looking at the "kernel module in use" lines.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

ValmerH
Posts: 2
Joined: 2020/08/06 16:42:41

Re: Multiport Serial Card no longer seeing ports

Post by ValmerH » 2020/08/06 18:37:50

Commtech_Josh wrote:
2020/07/22 18:57:18
I'm at a bit of a loss. If I understand the Makefiles, CONFIG_SERIAL_8250_EXAR should be enabling the exar part of the serial drivers, but the kernel doesn't seem to even notice the UARTs on our card. Even though it clearly and correctly identifies the card as our card (Commtech, Inc. 422/4-PCIe Serial PCIe Adapter).
Hello Josh,

I have encountered the same issue using a Commtech 422/8-PCIe card and am currently working on a solution. Have you been able to make any progress on this problem?

Thank you,
Valmer

Commtech_Josh
Posts: 5
Joined: 2020/07/16 16:23:05

Re: Multiport Serial Card no longer seeing ports

Post by Commtech_Josh » 2020/08/19 14:42:43

I'm still at a loss on this. When I do lspci -vvv everything seems to be correct:

Code: Select all

02:00.0 Serial controller: Commtech, Inc. 422/4-PCIe Serial PCIe Adapter [Fastcom] (rev 03) (prog-if 02 [16550])
	Physical Slot: 1-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 27
	Region 0: Memory at f7c00000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee02004  Data: 4022
	Capabilities: [78] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] Express (v2) Endpoint, MSI 01
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Kernel driver in use: exar_serial
I even verified that /boot/config-xxx has
CONFIG_SERIAL_8250_NR_UARTS and CONFIG_SERIAL_8250_RUNTIME_UARTS greater than the number of ports I have in the system. Additionally, CONFIG_SERIAL_8250_EXAR is set.

I'm really baffled at why CentOS could still fail to detect the ports on the card, and I've repeatedly confirmed that the card is not damaged and the drivers detect the UARTs on other (and older CentOS) kernels. All it does it detect a ttyS0 and a ttyS4 that's a 16550A.

Commtech_Josh
Posts: 5
Joined: 2020/07/16 16:23:05

Re: Multiport Serial Card no longer seeing ports

Post by Commtech_Josh » 2020/08/28 18:16:49

According to the FAQ there's a 'bump topic' button, but I can't seem to find it, so here we are.

Does anyone have any insights into what might be causing this?

ValmerH
Posts: 2
Joined: 2020/08/06 16:42:41

Re: Multiport Serial Card no longer seeing ports

Post by ValmerH » 2020/08/31 17:38:36

Commtech_Josh wrote:
2020/07/20 21:04:30
We are the driver manufacturers for this card, and yes, the final drivers weren't installed. From my (limited) understanding, the core functionality such as read/write should be handled by the default serial drivers, while our particular drivers provide support for functionality unique to our devices such as custom clock speed. This is how its worked since 2013, but suddenly it seems that the core serial driver is no longer recognizing our device(s).

For example, these lines in the Linux 8250_exar.c files in the linux repo:
https://github.com/torvalds/linux/blob/ ... exar.c#L40
https://github.com/torvalds/linux/blob/ ... xar.c#L805

This is why I'm surprised - I don't know enough to understand why it would no longer function like that.
Hi Josh,

Since the last working release, prior to CentOS 8, there has been a rewrite of the 8250 driver. I have identified the issue in 8250_exar.c which assigns the wrong number of ports to the Fastcom 422/4 and 422/8 PCIe cards based on the device ID. If you revisit https://github.com/torvalds/linux/blob/ ... 250_exar.c you will see my latest commit upstream which resolves your issue:

"serial: 8250_exar: Fix number of ports for Commtech PCIe cards"

I'm not sure when this will be updated in CentOS 8 ( kernel 4.18 ).

Post Reply