samba-vfs-glusterfs missing in CentOS 8

[MaktonTo]

I've gotten to the point there I needed to register onto the site for this one..

I have 2 CentOS 8 GlusterFS nodes and 1 CentOS 8 GlusterFS arbiter. The setup is with GlusteFS, CTDB, and Samba. The current path I'm stuck with is mounting the GlusterFS volume on the storage nodes and having Samba attach to the folders (GlusterFS/FUSE/SAMBA). The performance in this mode is awful and my volume log file (/var/log/gluster/volname.log) is throwing errors when anything is created/added to the share.

Code: Select all

[2020-04-11 00:25:16.656569] W [fuse-bridge.c:949:fuse_entry_cbk] 0-glusterfs-fuse: 2756: LOOKUP() /shares/Documents/SQL Server Management Studio => -1 (Permission denied)
[2020-04-11 00:25:16.656804] W [fuse-bridge.c:949:fuse_entry_cbk] 0-glusterfs-fuse: 2757: LOOKUP() /shares/Documents/SQL Server Management Studio => -1 (Permission denied)
[2020-04-11 00:25:16.656976] W [fuse-bridge.c:949:fuse_entry_cbk] 0-glusterfs-fuse: 2758: LOOKUP() /shares/Documents/SQL Server Management Studio => -1 (Permission denied)

The items are created and the share is stabled, but when I'm dealing with moving over a million files, this is a ton of chatter. I've tried using vfs gluster_fuse, but that doesn't seem to do anything and all the searching on the net has lead me to needing one module from SAMBA known as "vfs glusterfs", which has Samba talking directly to GlusterFS, removing Fuse from the middle. This is in a package on CentOS 7 called "samba-vfs-glusterfs" but is missing in CentOS 8.

I desperately need this module to get the performance corrected on this solution and am inches away from compiling all of Samba from source. I've already tried compiling the 4.10.4 Samba from source, only for the module I need, but when I add the module, the share mounts, but from the windows side, I get "too large for the destination file system". So something is missing there as well.

Any ideas or thoughts on how we can get the "samba-vfs-glusterfs" module into CentOS 8 before I'm compiling the entire Samba system from source?

Best regards,

Mak.

[MaktonTo]

Looks like a lot of my log spamming from the gluster volume log pertains to my workstation requesting attributes, which it would not have access to..

Code: Select all

[2020-04-11 17:32:52.094028] I [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: e22acd2d-1370-4d8a-b15e-e468fd5be57e, req(uid:2004626,gid:2000515,perm:4,ngrps:5), ctx(uid:2001103,gid:2000513,in-groups:0,perm:770,updated-fop:LOOKUP, acl:(tag:1,perm:7,id:4294967295)(tag:2,perm:7,id:0)(tag:2,perm:7,id:2000512)(tag:2,perm:7,id:2000513)(tag:4,perm:7,id:4294967295)(tag:8,perm:7,id:991,in-groups:0)(tag:8,perm:7,id:2000512,in-groups:0)(tag:8,perm:7,id:2000513,in-groups:0)(tag:8,perm:7,id:2001103,in-groups:0)(tag:16,perm:7,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied]
The message "I [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 147e851d-793e-4ef5-b862-cc545d66f9cd, req(uid:2004626,gid:2000515,perm:1,ngrps:5), ctx(uid:2001103,gid:2000513,in-groups:0,perm:770,updated-fop:READDIRP, acl:(tag:1,perm:7,id:4294967295)(tag:2,perm:7,id:0)(tag:2,perm:7,id:2000512)(tag:2,perm:7,id:2000513)(tag:4,perm:7,id:4294967295)(tag:8,perm:7,id:991,in-groups:0)(tag:8,perm:7,id:2000512,in-groups:0)(tag:8,perm:7,id:2000513,in-groups:0)(tag:8,perm:7,id:2001103,in-groups:0)(tag:16,perm:7,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied]" repeated 3 times between [2020-04-11 17:32:52.101157] and [2020-04-11 17:32:52.101718]

Come to find.. the UID 2004626, doing the request is actually the workstation ADS ID. So, to stop having this area spammed, I would have to include the workstation ADS UID (using winbind to find it) to the xattrs of the share with inheritance. That doesn't work for 100+ workstations. This happens, with and without using the vfs glusterfs_fuse.

So, still needing the vfs glusterfs module to help with this..

best regards,

Mak

[MaktonTo]

I downgraded to CentOS7 to get the needed performance differences and there really isn't any.. I'm dealing with millions of small files and was information, using the glusterfs vfs over the fuse mount would help.. The time it takes to copy my test data is the same (maybe couple seconds off)..

Also, instead of the previous messages spamming my logs, I have new ones!

Code: Select all

[2020-04-12 03:17:32.597879] W [MSGID: 114031] [client-rpc-fops_v2.c:920:client4_0_getxattr_cbk] 0-gvol-client-0: remote operation failed. Path: /ShareTest/source/CallRecordingViewer/packages/System.Runtime.Extensions.4.1.0/lib/xamarinios10 (40c7c963-d9ee-471a-be89-81fe7887d675). Key: user.SAMBA_PAI [Permission denied]
[2020-04-12 03:17:32.599646] W [MSGID: 114031] [client-rpc-fops_v2.c:920:client4_0_getxattr_cbk] 0-gvol-client-1: remote operation failed. Path: /ShareTest/source/CallRecordingViewer/packages/System.Runtime.Extensions.4.1.0/lib/xamarinios10 (40c7c963-d9ee-471a-be89-81fe7887d675). Key: user.SAMBA_PAI [Permission denied]
[2020-04-12 03:17:32.599776] W [dict.c:612:dict_del] (-->/usr/lib64/glusterfs/7.4/xlator/cluster/replicate.so(+0x16970) [0x7fed0cbff970] -->/usr/lib64/glusterfs/7.4/xlator/cluster/distribute.so(+0x38542) [0x7fed0c757542] -->/lib64/libglusterfs.so.0(dict_del+0x86) [0x7fed1375bda6] ) 0-dict: !this || key=trusted.glusterfs.dht [Invalid argument]
[2020-04-12 03:17:32.599866] W [dict.c:612:dict_del] (-->/usr/lib64/glusterfs/7.4/xlator/cluster/replicate.so(+0x16970) [0x7fed0cbff970] -->/usr/lib64/glusterfs/7.4/xlator/cluster/distribute.so(+0x38551) [0x7fed0c757551] -->/lib64/libglusterfs.so.0(dict_del+0x86) [0x7fed1375bda6] ) 0-dict: !this || key=trusted.glusterfs.dht.mds [Invalid argument]
[2020-04-12 03:17:32.599956] W [dict.c:612:dict_del] (-->/usr/lib64/glusterfs/7.4/xlator/cluster/replicate.so(+0x16970) [0x7fed0cbff970] -->/usr/lib64/glusterfs/7.4/xlator/cluster/distribute.so(+0x38560) [0x7fed0c757560] -->/lib64/libglusterfs.so.0(dict_del+0x86) [0x7fed1375bda6] ) 0-dict: !this || key=trusted.glusterfs.dht.commithash [Invalid argument]
[2020-04-12 03:17:32.600009] E [MSGID: 101046] [dht-common.c:11237:dht_pt_getxattr_cbk] 0-gvol-dht: dict is null
[2020-04-12 03:17:32.600084] E [MSGID: 101046] [dht-common.c:11238:dht_pt_getxattr_cbk] 0-gvol-dht: dict is null

They are effectively the same messages, just in a different formation as it's a different driver..
my test data is 8,524 files being 771MB in size.

Copying to Clustered CIFS Share -> 8 Minutes 45 Seconds
On a Ubuntu (haven't moved it to CentOS 8 yet) CIFS only system -> 1 minute 41 seconds

I have performance tuning on the cluster and no tuning on the single share server yet sitting at a 8/1 ratio in upload performance. Also, my logs for Gluster are being spammed with "Permission Denied" when dealing with attributes upon uploading documents. At this point, thinking this is a conversation with the Gluster community.. Any thoughts? otherwise I'm not even sure if the vfs module is even worth the headache.

NOTE: I'm doing all this to support a company who has file shares on CentOS 8 Gluster/CTDB/Samba and are looking for better performance with that platform. I would not be able to downgrade them like I did my own systems.

Best regards,

Mak

[fatmoustache]

Hi MaktonTo,

Did you ever find resolution for this? I feel like I'm in the same situation (with CentOS 7) and I'm having a hard time making any progress on it. I'm almost at a point of believing that I missed something obvious like a firewall setting or something along those lines, but I so far haven't found anything.

Any means of accessing my Gluster volumes outside of the gluster vfs module works just fine. It's just this particular configuration that's problematic.

Thanks.

[hunter86_bg]

Recently AnoopCS from the CentOS Storage SIG has managed to build the latest version of samba-vfs-glusterfs.

I had the chance to test gluster 8.0 with 'samba-vfs-glusterfs-4.11.11-1.el8.x86_64.rpm' and the relevant 4.11 samba which worked fine for my tests.

For anyone interested I think that you just need to :

Code: Select all

yum install centos-release-gluster8.noarch centos-release-storage-common.noarch centos-release-samba411.noarch

Note: It seems that there is a 'centos-release-samba412.noarch' , so you can also give it a try.