NFS mounted but cd gives permission denied

General support questions
kdpatil
Posts: 42
Joined: 2020/10/20 07:19:31

NFS mounted but cd gives permission denied

Post by kdpatil » 2021/02/27 03:32:06

Hi


I have centos7 box

NFS Qtree used for home dir from Netapp is able to mount just fine

Netapp Exoport policy shows :

any any any ...ie super user access as well is any

When i am root user on client box , can cd to user home dir of any user

We have SSSD setup & use can login with AD id

when the normal user login The home dir of the user is not able to mount & error is


##
su - userxxxx
Last login: Fri Feb 26 19:17:03 EST 2021 from s...
su: warning: cannot change directory to .../..: Permission denied
-bash: .../.bash_profile: Permission denied
-bash-4.2$


Here is tcpdump shows

###

tcpdump -s 192 port nfs -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 192 bytes




Code: Select all

19:13:34.696861 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [S], seq 4251450372, win 29200, options [mss 1460,sackOK,TS val 1258809 ecr 0,nop,wscale 7], length 0
19:13:34.697147 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [S.], seq 3336189244, ack 4251450373, win 65535, options [mss 8960,nop,wscale 8,sackOK,TS val 1699851922 ecr 1258809], length 0
19:13:34.697169 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 1, win 229, options [nop,nop,TS val 1258809 ecr 1699851922], length 0
19:13:34.697184 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [P.], seq 1:137, ack 1, win 229, options [nop,nop,TS val 1258809 ecr 1699851922], length 136: NFS request xid 2803191295 132 access [|nfs]
19:13:34.707092 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [P.], seq 1:125, ack 137, win 257, options [nop,nop,TS val 1699851932 ecr 1258809], length 124: NFS reply xid 2803191295 reply ok 120 access c 0003
19:13:34.707101 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 125, win 229, options [nop,nop,TS val 1258819 ecr 1699851932], length 0
19:13:34.707147 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [P.], seq 137:281, ack 125, win 229, options [nop,nop,TS val 1258819 ecr 1699851932], length 144: NFS request xid 2819968511 140 lookup [|nfs]
19:13:34.707621 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [P.], seq 125:389, ack 281, win 257, options [nop,nop,TS val 1699851932 ecr 1258819], length 264: NFS reply xid 2819968511 reply ok 260 lookup fh Unknown/01000000A916668000000000F0F93B00CEC24854A91666800000000061000000
19:13:34.747554 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 389, win 237, options [nop,nop,TS val 1258860 ecr 1699851932], length 0
19:14:34.799582 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 389, win 237, options [nop,nop,TS val 1318912 ecr 1699851932], length 0
19:14:34.799848 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [.], ack 281, win 257, options [nop,nop,TS val 1699912024 ecr 1258860], length 0

#######################################

fstab entry :-
xxx:/vol_home/home xxx nfs vers=3,bg,soft,retrans=4 0 0
#######################################

mount options from client end
mount -v |grep -i nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
xx:/vol_home/home on xx type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,soft,proto=tcp,timeo=600,retrans=4,sec=sys,mountaddr=xxx,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=xxx)
#######################################


systemctl |grep -i rpc
var-lib-nfs-rpc_pipefs.mount loaded active mounted RPC Pipe File System
rpc-gssd.service loaded active running RPC security service for NFS client and server
rpc-statd.service loaded active running NFS status monitor for NFSv2/3 locking.
rpcbind.service loaded active running RPC bind service
rpcbind.socket loaded active running RPCbind Server Activation Socket
rpc_pipefs.target loaded active active rpc_pipefs.target
rpcbind.target loaded active active RPC Port Mapper


#######################################

systemctl |grep -i nfs
var-lib-nfs-rpc_pipefs.mount loaded active mounted RPC Pipe File System
rpc-gssd.service loaded active running RPC security service for NFS client and server
rpc-statd.service loaded active running NFS status monitor for NFSv2/3 locking.
nfs-client.target loaded active active NFS client services

#######################################




nothing in message file which will give any indications ...

Any suggestions Please

Thanks

kdpatil
Posts: 42
Joined: 2020/10/20 07:19:31

Re: NFS mounted but cd gives permission denied

Post by kdpatil » 2021/02/27 07:38:26

I enabled NFS & RPC in debug mode .. here is what i see
Feb 26 23:28:29 Server1 su: (to user1) on pts/1
Feb 26 23:28:29 Server1 kernel: NFS: permission(0:40/380174433), mask=0x81, res=-10
Feb 26 23:28:29 Server1 kernel: NFS call access
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_start nfs3 proc ACCESS (sync)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_reserve (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_reserveresult (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_refresh (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_refreshresult (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_allocate (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_bind (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_connect xprt ffff9eb4f4640000 is connected
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_transmit (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 rpc_xdr_encode (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_status (status 120)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_decode (status 120)
Feb 26 23:28:29 Server1 kernel: RPC: 32 call_decode result 0
Feb 26 23:28:29 Server1 kernel: RPC: rpc_release_client(ffff9eb4f58eb200)
Feb 26 23:28:29 Server1 kernel: NFS: nfs_update_inode(0:40/380174433 fh_crc=0x243db35a ct=2 info=0x27e7f)
Feb 26 23:28:29 Server1 kernel: NFS reply access: 0
Feb 26 23:28:29 Server1 kernel: NFS: permission(0:40/380174433), mask=0x1, res=0
Feb 26 23:28:29 Server1 kernel: NFS: nfs_lookup_revalidate(/user1) is valid
Feb 26 23:28:29 Server1 kernel: NFS: revalidating (0:40/378730992)
Feb 26 23:28:29 Server1 kernel: NFS call getattr
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_start nfs3 proc GETATTR (sync)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_reserve (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_reserveresult (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_refresh (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_refreshresult (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_allocate (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_bind (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_connect xprt ffff9eb4f4640000 is connected
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_transmit (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 rpc_xdr_encode (status 0)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_status (status 112)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_decode (status 112)
Feb 26 23:28:29 Server1 kernel: RPC: 33 call_decode result 0
Feb 26 23:28:29 Server1 kernel: RPC: rpc_release_client(ffff9eb4f58eb200)
Feb 26 23:28:29 Server1 kernel: NFS reply getattr: 0
Feb 26 23:28:29 Server1 kernel: NFS: nfs_update_inode(0:40/378730992 fh_crc=0xe0390100 ct=1 info=0x27e7f)
Feb 26 23:28:29 Server1 kernel: NFS: (0:40/378730992) revalidation complete
Feb 26 23:28:29 Server1 kernel: NFS: dentry_delete(/user1, 10808cc)

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: NFS mounted but cd gives permission denied

Post by Whoever » 2021/02/28 19:26:56

What you have not shown is what are the ownership and permissions of the directory on the NetApp device as seen by the CentOS 6 box.

A piece of advice: don't mount network filesystems from fstab. Use the automounter instead.

Am I correct in thinking that you are trying to make the home directories effectively have "777" permissions? That's going to lead to some problems. SSH won't use public/private key pairs if the permissions are not restricted appropriately.

MartinR
Posts: 714
Joined: 2015/05/11 07:53:27
Location: UK

Re: NFS mounted but cd gives permission denied

Post by MartinR » 2021/02/28 23:15:57

Just out of interest, why that advice? If I want a remote filesystem to mount at boot and stay for the duration of the run, why not use fstab? To be fair, I've been doing it since the last century and it works fine though something more recent (systemd?) might have changed things.

kdpatil
Posts: 42
Joined: 2020/10/20 07:19:31

Re: NFS mounted but cd gives permission denied

Post by kdpatil » 2021/03/01 02:22:08

Thanks for the reply folks

Few things i have found :

This is not home dir permission issue ... why so .. ?
I did plain centos 7 box & did SSSD mounted works fine
Now i did centos 7 + sssd + CIS level 2 handing ( my case ) ... this box it is not working

CIS level 2 does 300 + changes .. i did try to find specific no luck.. i was suspecting - TCP Wrappers , but no luck

Nevertheless the sad part is log does not give any indication .. best way should be log to give some clue :(

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: NFS mounted but cd gives permission denied

Post by Whoever » 2021/03/01 04:49:18

MartinR wrote:
2021/02/28 23:15:57
Just out of interest, why that advice? If I want a remote filesystem to mount at boot and stay for the duration of the run, why not use fstab? To be fair, I've been doing it since the last century and it works fine though something more recent (systemd?) might have changed things.
Do you need it mounted for the system to boot? If not, it's just something else that can go wrong when booting. Using the automounter allows you to mount without using the "bg,soft" options, since it's no longer needed on boot.

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: NFS mounted but cd gives permission denied

Post by Whoever » 2021/03/01 04:53:08

Is there anything of interest in the SELinux audit log?

kdpatil
Posts: 42
Joined: 2020/10/20 07:19:31

Re: NFS mounted but cd gives permission denied

Post by kdpatil » 2021/03/01 19:44:42

Selinux i disabled along with Firewall .. so that is ruled out

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: NFS mounted but cd gives permission denied

Post by Whoever » 2021/03/02 16:20:03

You still haven't shown the ownership and permissions of the remote directory as seen by the CentOS box.

kdpatil
Posts: 42
Joined: 2020/10/20 07:19:31

Re: NFS mounted but cd gives permission denied

Post by kdpatil » 2021/03/03 19:21:34

here is eg of working node

-sh-4.2$ pwd
/mnt/xxx/home/xxx
-sh-4.2$ ls -ld .
drwx------ 17 xxx xxxx 4096 Mar 2 15:21 .
-sh-4.2$

Post Reply