CentOS 5.5 to 5.6 Updates - broken scsi-target-utils-1.0.8-0.1

[teleport]

After upgrading my storage server to 5.6 all iscsi clients refuses to connect with message:

[code]Apr 9 12:06:09 hw20 kernel: scsi6 : iSCSI Initiator over TCP/IP
Apr 9 12:06:09 hw20 kernel: Vendor: IET Model: Controller Rev: 0001
Apr 9 12:06:09 hw20 kernel: Type: RAID ANSI SCSI revision: 05
Apr 9 12:06:09 hw20 kernel: scsi 6:0:0:0: Attached scsi generic sg1 type 12
Apr 9 12:06:09 hw20 kernel: Vendor: IET Model: VIRTUAL-DISK Rev: 0001
Apr 9 12:06:09 hw20 kernel: Type: Direct-Access ANSI SCSI revision: 05
Apr 9 12:06:09 hw20 kernel: sdb : very big device. try to use READ CAPACITY(16).
Apr 9 12:06:09 hw20 kernel: sdb : unsupported sector size 1229280288.
Apr 9 12:06:09 hw20 kernel: SCSI device sdb: 0 512-byte hdwr sectors (0 MB)
Apr 9 12:06:09 hw20 kernel: sdb: Write Protect is off
Apr 9 12:06:09 hw20 kernel: SCSI device sdb: drive cache: write back[/code]

Solution is posted [url=http://www.spinics.net/lists/linux-stgt/msg02247.html]here[/url]

After manual patching and rebuilding scsi-target-utils-1.0.8-0.el5_6.1.src.rpm tgtd seems to work again:

[code]Apr 9 12:40:22 hw20 kernel: scsi10 : iSCSI Initiator over TCP/IP
Apr 9 12:40:22 hw20 kernel: Vendor: IET Model: Controller Rev: 0001
Apr 9 12:40:22 hw20 kernel: Type: RAID ANSI SCSI revision: 05
Apr 9 12:40:22 hw20 kernel: scsi 10:0:0:0: Attached scsi generic sg1 type 12
Apr 9 12:40:22 hw20 kernel: Vendor: IET Model: VIRTUAL-DISK Rev: 0001
Apr 9 12:40:22 hw20 kernel: Type: Direct-Access ANSI SCSI revision: 05
Apr 9 12:40:22 hw20 kernel: sdb : very big device. try to use READ CAPACITY(16).
Apr 9 12:40:22 hw20 kernel: SCSI device sdb: 8589934592 512-byte hdwr sectors (4398047 MB)
Apr 9 12:40:22 hw20 kernel: sdb: Write Protect is off
Apr 9 12:40:22 hw20 kernel: SCSI device sdb: drive cache: write back
Apr 9 12:40:22 hw20 kernel: sdb : very big device. try to use READ CAPACITY(16).
Apr 9 12:40:22 hw20 kernel: SCSI device sdb: 8589934592 512-byte hdwr sectors (4398047 MB)
Apr 9 12:40:22 hw20 kernel: sdb: Write Protect is off
Apr 9 12:40:22 hw20 kernel: SCSI device sdb: drive cache: write back
Apr 9 12:40:22 hw20 kernel: sdb: sdb1
Apr 9 12:40:22 hw20 kernel: sd 10:0:0:1: Attached scsi disk sdb
Apr 9 12:40:22 hw20 kernel: sd 10:0:0:1: Attached scsi generic sg2 type 0[/code]

[milosb]

I'd definitely first try with a more conventional approach, like downgrading the [b]scsi-target-utils[/b] package. (as a quickfix)

Even if this works, filing a bug report with [b]bugs.centos.org[/b] is recommended.


Regards,

[teleport]

That downgrade will need also older version of libibverbs and librdmacm. This bug can be specific to x86_64 arch only, I do not have 32bit machine with 4+ Tb storage to check that.

[pschaff]

Looks like [url=https://bugzilla.redhat.com/show_bug.cgi?id=674394]BZ# 674394[/url]. Added a note to the [url=http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.6]release notes[/url].

[milosb]

[quote]
teleport wrote:
That downgrade will need also older version of libibverbs and librdmacm. This bug can be specific to x86_64 arch only,
I do not have 32bit machine with 4+ Tb storage to check that.[/quote]

Funny, I did a downgrade on x86_64 - and the only thing that was downgraded was [b]scsi-target-utils[/b]. Although, this was done on my desktop machine, 'cause I don't have the need for this particular package.

[quote]
pschaff worte:
Looks like BZ# 674394.
[/quote]
Indeed... but, just a thought: the bug has been around since February, so why wait for TUV to fix it?
It wouldn't be the first time - although, this was with [b]module-init-tools[/b], and the fixed package was published in the ELRepo repository.

[teleport]

[quote]
milosb wrote:
Funny, I did a downgrade on x86_64 - and the only thing that was downgraded was [b]scsi-target-utils[/b].
Although, this was done on my desktop machine, 'cause I don't have the need for this particular package.
[/quote]

Did You try to start it?

In my case I get:

[code][root@c5-64-devel ~]# cat /etc/issue
CentOS release 5.6 (Final)
Kernel \r on an \m

[root@c5-64-devel ~]# uname -a
Linux c5-64-devel 2.6.18-238.5.1.el5 #1 SMP Fri Apr 1 18:41:58 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@c5-64-devel ~]# rpm -q scsi-target-utils
scsi-target-utils-0.0-6.20091205snap.el5_5.3

[root@c5-64-devel ~]# tail -f /var/log/messages &
...
[root@c5-64-devel ~]# service tgtd start
Starting SCSI target daemon: Starting target framework daemon

[root@c5-64-devel ~]# Apr 9 21:14:44 c5-64-devel tgtd: iser transport register
Apr 9 21:14:49 c5-64-devel [b]tgtd: librdmacm.so: cannot open shared object file: No such file or directory - iser transport not used[/b]
Apr 9 21:14:49 c5-64-devel tgtd: Target daemon logger with pid=2049 started![/code]

Patching and rebuilding because of known bug is safest way I think. Fighting dependencies on production server is more dangerous.

[milosb]

[quote]
teleport wrote:

Did You try to start it?
[/quote]
TBH, not until you brought it up... but, here it goes:

[code]
[root@host ~]# yum install scsi-target-utils
Loaded plugins: fastestmirror, kernel-module, kmod, priorities, security
Loading mirror speeds from cached hostfile
* addons: mirror.nsc.liu.se
* base: centosr4-msync-dvd.centos.org
* elrepo: elrepo.reloumirrors.net
* extras: mirror.nsc.liu.se
* rpmforge: fr2.rpmfind.net
* updates: centosn4.centos.org
addons | 951 B 00:00
base | 2.1 kB 00:00
elrepo | 1.9 kB 00:00
extras | 2.1 kB 00:00
rpmforge | 1.1 kB 00:00
updates | 1.9 kB 00:00
Excluding Packages in global exclude list
Finished
Excluding Packages from Red Hat Enterprise 5 - RPMforge.net - dag
Finished
136 packages excluded due to repository priority protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package scsi-target-utils.x86_64 0:1.0.8-0.el5_6.1 set to be updated
--> Processing Dependency: perl(Config::General) for package: scsi-target-utils
--> Processing Dependency: libibverbs.so.1()(64bit) for package: scsi-target-utils
--> Processing Dependency: librdmacm.so.1()(64bit) for package: scsi-target-utils
--> Running transaction check
---> Package libibverbs.x86_64 0:1.1.3-2.el5 set to be updated
--> Processing Dependency: openib >= 1.0 for package: libibverbs
---> Package librdmacm.x86_64 0:1.0.10-1.el5 set to be updated
---> Package perl-Config-General.noarch 0:2.40-1.el5 set to be updated
--> Running transaction check
---> Package openib.noarch 0:1.4.1-5.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================
Package Arch Version Repository Size
======================================================================================================
Installing:
scsi-target-utils x86_64 1.0.8-0.el5_6.1 updates 152 k
Installing for dependencies:
libibverbs x86_64 1.1.3-2.el5 base 44 k
librdmacm x86_64 1.0.10-1.el5 base 23 k
openib noarch 1.4.1-5.el5 base 20 k
perl-Config-General noarch 2.40-1.el5 base 68 k

Transaction Summary
======================================================================================================
Install 5 Package(s)
Upgrade 0 Package(s)

Total download size: 307 k
Is this ok [y/N]: y
Downloading Packages:
(1/5): openib-1.4.1-5.el5.noarch.rpm | 20 kB 00:00
(2/5): librdmacm-1.0.10-1.el5.x86_64.rpm | 23 kB 00:00
(3/5): libibverbs-1.1.3-2.el5.x86_64.rpm | 44 kB 00:00
(4/5): perl-Config-General-2.40-1.el5.noarch.rpm | 68 kB 00:00
(5/5): scsi-target-utils-1.0.8-0.el5_6.1.x86_64.rpm | 152 kB 00:00
----------------------------------------------------------------------------------
Total 160 kB/s | 307 kB 00:01
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : perl-Config-General 1/5
Installing : openib 2/5
Installing : libibverbs 3/5
Installing : librdmacm 4/5
Installing : scsi-target-utils 5/5

Installed:
scsi-target-utils.x86_64 0:1.0.8-0.el5_6.1

Dependency Installed:
libibverbs.x86_64 0:1.1.3-2.el5 librdmacm.x86_64 0:1.0.10-1.el5 openib.noarch 0:1.4.1-5.el5 perl-Config-General.noarch 0:2.40-1.el5

Complete!
[root@host ~]# yum downgrade scsi-target-utils
Loaded plugins: fastestmirror, kernel-module, kmod, priorities, security
Setting up Downgrade Process
Loading mirror speeds from cached hostfile
* addons: mirror.nsc.liu.se
* base: centosr4-msync-dvd.centos.org
* elrepo: elrepo.reloumirrors.net
* extras: mirror.nsc.liu.se
* rpmforge: fr2.rpmfind.net
* updates: centosn4.centos.org
Excluding Packages in global exclude list
Finished
Excluding Packages from Red Hat Enterprise 5 - RPMforge.net - dag
Finished
136 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package scsi-target-utils.x86_64 0:1.0.8-0.el5 set to be updated
---> Package scsi-target-utils.x86_64 0:1.0.8-0.el5_6.1 set to be erased
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================
Package Arch Version Repository Size
======================================================================================================
Downgrading:
scsi-target-utils x86_64 1.0.8-0.el5 base 151 k

Transaction Summary
======================================================================================================
Remove 0 Package(s)
Reinstall 0 Package(s)
Downgrade 1 Package(s)

Total download size: 151 k
Is this ok [y/N]: y
Downloading Packages:
scsi-target-utils-1.0.8-0.el5.x86_64.rpm | 151 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : scsi-target-utils 1/2
Cleanup : scsi-target-utils 2/2

Removed:
scsi-target-utils.x86_64 0:1.0.8-0.el5_6.1

Installed:
scsi-target-utils.x86_64 0:1.0.8-0.el5

Complete!
[root@host ~]# service tgtd start
Starting SCSI target daemon: Starting target framework daemon

[root@host ~]# service tgtd status
tgtd (pid 8677 8676) is running...
[/code]

And here are the relevant lines from [b]/var/log/messages[/b] when starting [b]tgtd[/b]:
[code]
Apr 9 17:46:06 host tgtd: iser transport register
Apr 9 17:46:06 host tgtd: tgtd daemon started, pid:8676
Apr 9 17:46:06 host tgtd: tgtd logger started, pid:8677 debug:0
Apr 9 17:46:07 host tgtd: bs_init(318) use pthread notification
[/code]

[quote]
teleport wrote:
Patching and rebuilding because of known bug is safest way I think. Fighting dependencies on production server is more dangerous.[/quote]
Aaaah, fighting dependencies... shouldn't be an issue if the base packages haven't been overwritten by packages from 3rd party repo - as you can see, the proposed solution works fine for me.
But then again, I don't really have iSCSI storage at home to test it with ;-) so I guess any solution that saves the day is just fine.


Regards,

[teleport]

Downgrading from Centos 5.6 scsi-target-utils.x86_64 0:1.0.8-0.el5_6.1 to Centos 5.6 scsi-target-utils.x86_64 0:1.0.8-0.el5 is useless, bug is present in both of them.
Latest Centos 5.5 version is [b]scsi-target-utils-0.0-6.20091205snap.el5_5.3[/b], which works fine before upgrading to 5.6, but depends from 5.5 version of librdmacm.

So just "yum downgrade" does not help. And I continue to think that mixing 5.5 version of scsi-target-utils with 5.6 versions of librdmacm and libibverbs is dangerous.

[milosb]

[quote]
teleport wrote:
Downgrading from Centos 5.6 scsi-target-utils.x86_64 0:1.0.8-0.el5_6.1 to Centos 5.6 scsi-target-utils.x86_64 0:1.0.8-0.el5 is useless, bug is present in both of them.
Latest Centos 5.5 version is [b]scsi-target-utils-0.0-6.20091205snap.el5_5.3[/b], which works fine before upgrading to 5.6, but depends from 5.5 version of librdmacm.

So just "yum downgrade" does not help. And I continue to think that mixing 5.5 version of scsi-target-utils with 5.6 versions of librdmacm and libibverbs is dangerous.[/quote]
[/quote]
Why didn't you say so in the first place... 'cause this is in fact what I meant by 'downgrade':
[code]
yum downgrade pkg_name
[/code]

I myself also wouldn't downgrade a package if it required downgrading numerous other dependencies.

Besides, no need to get bitchy about it... I was only trying to help and possibly save you the trouble of having to rebuild the RPM, with regard to the fact that you haven't mentioned this as something you've already tried.

No hard feelings, huh? ;-)


Regards,

[teleport]

No hard feelings of course 8-)

This thread was just warning from someone using similar setup: Centos 5.5 as iscsi target with 4+ Tb storage.

I didn't give detail explanation, because everyone using Centos ( and maybe rhel,fedora,scientific and so on ) [b]must[/b] know that yum can upgrade between releases, but cannot downgrade to older release, only to older package from same release.

After upgrading to 5.6 yum knows nothing about 5.5 packages and their dependences. Because of that downloading 5.5 rpm and "yum downgrade scsi-target-utils-0.0-6.20091205snap.el5_5.3.x86_64.rpm" gives no errors.

In this particular case I do not agree with [b]pschaff[/b] recomendation: "The recommended workaround is to downgrade scsi-target-utils to the 5.5 version. "

Better workaround is something like "yum --exclude=scsi-target-utils --exclude=libibverbs --exclude=librdmacm upgrade" for servers used as iscsi target, while they are still on 5.5. This also will keep all packages from 5.5 which depends from 5.5 versions of libibverbs and librdmacm.