tcp timeout issue

[ek1010]

Hello,

We ran into a strange issue with regards to tcp connectivity from a CentOS 7 server. The server has a network interface that is connected to a top of rack switch. We connect to a counterparty via TCP to request data. The counterparty has a few TCP endpoints. All of these endpoints are accessible via the same interface on our server. The initial transmission is a replay of the days data up until the time of the request (i.e. a burst of TCP data). The total transmission size is under 2gb. The NIC and switch are 10Gb so there should be no issues as far as the hardware.

The problem we noticed is that when we first establish the TCP connection other new tcp connections to the same counterparty (via the same interface to all endpoints) start timing out. For example, a basic nc command starts timing out. We tested this by running nc in loop. However, existing TCP connections continue working as usual. Sometime after the replay is done connectivity comes back to normal state and we can establish new tcp connections as usual. The counterparty and datacenter host both claim that they see no issues.

I was wondering if there is something at the OS level that can cause this kind of behavior? Some basic details below and happy to provide more as needed. Thank you.

$ uname -a
Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)