ISCSI target gets blocked with "iSCSI Login negotiation failed" error

[KundanJ]

Hi All,

I am running targetcli v2.1.53 on centos 8 with kernel v4.18.0-394. I am getting below messages in /var/log/messages every 3-4 seconds.

Code: Select all

kernel: iSCSI Login timeout on Network Portal 10.74.67.11:3260
kernel: iSCSI Login negotiation failed.

After this messages if I try to restart the target service, it goes into hang state. I can't even kill it. Need to restart the VM to make the service normal again.
I also found some call traces in the /var/log/messages as below.

Code: Select all

Dec 28 05:49:07 kernel: iSCSI Login timeout on Network Portal 10.74.67.11:3260
Dec 28 05:49:07 kernel: iSCSI Login negotiation failed.
Dec 28 05:49:23 kernel: iSCSI Login timeout on Network Portal 10.74.67.11:3260
Dec 28 05:49:23 kernel: tx_data returned -104, expecting 48.
Dec 28 05:49:23 kernel: iSCSI Login negotiation failed.
Dec 28 05:49:26 kernel: INFO: task kworker/0:2:110640 blocked for more than 120 seconds.
Dec 28 05:49:26 kernel:      Not tainted 4.18.0-394.el8.x86_64 #1
Dec 28 05:49:26 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 28 05:49:26 kernel: task:kworker/0:2     state:D stack:    0 pid:110640 ppid:     2 flags:0x80004080
Dec 28 05:49:26 kernel: Workqueue: events target_tmr_work [target_core_mod]
Dec 28 05:49:26 kernel: Call Trace:
Dec 28 05:49:26 kernel: __schedule+0x2d1/0x830
Dec 28 05:49:26 kernel: ? __switch_to_asm+0x35/0x70
Dec 28 05:49:26 kernel: ? __switch_to_asm+0x41/0x70
Dec 28 05:49:26 kernel: schedule+0x35/0xa0
Dec 28 05:49:26 kernel: schedule_timeout+0x274/0x300
Dec 28 05:49:26 kernel: ? __schedule+0x2d9/0x830
Dec 28 05:49:26 kernel: ? __irq_work_queue_local+0x4b/0x50
Dec 28 05:49:26 kernel: ? irq_work_queue+0x16/0x20
Dec 28 05:49:26 kernel: wait_for_completion+0x96/0x100
Dec 28 05:49:26 kernel: target_put_cmd_and_wait+0x6e/0x90 [target_core_mod]
Dec 28 05:49:26 kernel: core_tmr_abort_task.cold.7+0x197/0x212 [target_core_mod]
Dec 28 05:49:26 kernel: target_tmr_work+0x96/0x110 [target_core_mod]
Dec 28 05:49:26 kernel: process_one_work+0x1a7/0x360
Dec 28 05:49:26 kernel: worker_thread+0x30/0x390
Dec 28 05:49:26 kernel: ? create_worker+0x1a0/0x1a0
Dec 28 05:49:26 kernel: kthread+0x10a/0x120
Dec 28 05:49:26 kernel: ? set_kthread_struct+0x40/0x40
Dec 28 05:49:26 kernel: ret_from_fork+0x35/0x40

Is there any other solution other than rebooting of the system?

[BShT]

change your system to a productional version, Centos 8 is dead and stream is a beta version

look at your cables

[TrevorH]

The current el8 kernel on Rocky/Alma/OEL/RHEL is 4.18.0-425.3.1.el8.x86_64