XFS: possible memory allocation deadlock in kmem_alloc (mode
Posted: 2015/05/06 01:06:15
At about 1:17am this morning, I started seeing "XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)" and it just kept spewing that out until I rebooted at 9:30am. The machine is a dedicated file server with nfs and samba exports. The time would probably correspond with when the other servers are backing up to this one. The servers that import file systems from this one via nfs just sat there stuck until the reboot.
Does anyone have any suggestions to resolve it?
Possibly relevant information gleaned from a bit of googling
- I'm using the default xfs parameters i.e. not messing with the block sizes. http://tracker.ceph.com/issues/6301 talks about an already applied patch to centos7 but seems to boil down to messing with the block sizes.
- single xeon E5-2609v2, 32gb ram, 7x4g sas raid 5 on lsi controller with hot spare (getinfo for disks below)
- not running ceph
- drbd is replicating and status was ok
Information for driver problems.
Does anyone have any suggestions to resolve it?
Possibly relevant information gleaned from a bit of googling
- I'm using the default xfs parameters i.e. not messing with the block sizes. http://tracker.ceph.com/issues/6301 talks about an already applied patch to centos7 but seems to boil down to messing with the block sizes.
- single xeon E5-2609v2, 32gb ram, 7x4g sas raid 5 on lsi controller with hot spare (getinfo for disks below)
- not running ceph
- drbd is replicating and status was ok
Information for driver problems.
Code: Select all
== BEGIN uname -rmi ==
3.10.0-123.6.3.el7.x86_64 x86_64 x86_64
== END uname -rmi ==
== BEGIN rpm -qa \*-release\* ==
centos-release-7-0.1406.el7.centos.2.5.x86_64
elrepo-release-7.0-2.el7.elrepo.noarch
epel-release-7-2.noarch
== END rpm -qa \*-release\* ==
== BEGIN cat /etc/redhat-release ==
CentOS Linux release 7.0.1406 (Core)
== END cat /etc/redhat-release ==
== BEGIN getenforce ==
Enforcing
== END getenforce ==
== BEGIN free -m ==
total used free shared buffers cached
Mem: 31980 18651 13328 8 5 16246
-/+ buffers/cache: 2399 29580
Swap: 31999 0 31999
== END free -m ==
== BEGIN rpm -qa kernel\* | sort ==
kernel-3.10.0-123.6.3.el7.x86_64
kernel-3.10.0-123.el7.x86_64
kernel-tools-3.10.0-123.6.3.el7.x86_64
kernel-tools-libs-3.10.0-123.6.3.el7.x86_64
== END rpm -qa kernel\* | sort ==
== Warning: lspci is not installed ==
== Warning: lsusb is not installed ==
== BEGIN rpm -qa kmod\* kmdl\* ==
kmod-libs-14-9.el7.x86_64
kmod-drbd84-8.4.5-1.el7.elrepo.x86_64
kmod-14-9.el7.x86_64
== END rpm -qa kmod\* kmdl\* ==
Code: Select all
== BEGIN uname -rmi ==
3.10.0-123.6.3.el7.x86_64 x86_64 x86_64
== END uname -rmi ==
== BEGIN rpm -qa \*-release\* ==
centos-release-7-0.1406.el7.centos.2.5.x86_64
elrepo-release-7.0-2.el7.elrepo.noarch
epel-release-7-2.noarch
== END rpm -qa \*-release\* ==
== BEGIN cat /etc/redhat-release ==
CentOS Linux release 7.0.1406 (Core)
== END cat /etc/redhat-release ==
== BEGIN getenforce ==
Enforcing
== END getenforce ==
== BEGIN free -m ==
total used free shared buffers cached
Mem: 31980 18156 13824 8 1 15758
-/+ buffers/cache: 2397 29583
Swap: 31999 0 31999
== END free -m ==
== BEGIN cat /etc/fstab ==
#
# /etc/fstab
# Created by anaconda on Tue Sep 9 19:33:28 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 1 1
UUID=b8c11e24-72ed-4f0a-9fbb-efebc3d67956 /boot xfs defaults 1 2
/dev/mapper/centos-home /home xfs defaults 1 2
/dev/mapper/centos-swap swap swap defaults 0 0
== END cat /etc/fstab ==
== BEGIN df -h ==
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 2.1G 48G 5% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 8.7M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-home 49G 33M 49G 1% /home
/dev/sda2 497M 126M 372M 26% /boot
/dev/drbd1 6.0T 2.9T 3.2T 47% /mnt/homeagbu
/dev/drbd2 4.0T 2.2T 1.9T 55% /mnt/backdesk
/dev/drbd3 2.0T 322G 1.7T 16% /mnt/backserv
/dev/drbd4 2.0T 1.6T 430G 80% /mnt/genomics
/dev/drbd5 2.0T 224G 1.8T 11% /mnt/agbudata
== END df -h ==
== BEGIN fdisk -lu ==
Disk /dev/sda: 24000.0 GB, 23999998328832 bytes, 46874996736 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 * 1 4294967295 2147483647+ ee GPT
Disk /dev/mapper/centos-swap: 33.6 GB, 33554432000 bytes, 65536000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-homeagbu: 6597.1 GB, 6597069766656 bytes, 12884901888 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-backdesk: 4398.0 GB, 4398046511104 bytes, 8589934592 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-genomics: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-backserv: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-agbudata: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-home: 52.4 GB, 52428800000 bytes, 102400000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
== END fdisk -lu ==
== BEGIN parted -l ==
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 24.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB bios_grub
2 2097kB 526MB 524MB xfs
3 526MB 140GB 140GB lvm
4 140GB 24.0TB 23.9TB
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-home: 52.4GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 52.4GB 52.4GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-agbudata: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-backserv: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-genomics: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-backdesk: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 4398GB 4398GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-homeagbu: 6597GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 6597GB 6597GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-root: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 53.7GB 53.7GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-swap: 33.6GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 33.6GB 33.6GB linux-swap(v1)
Model: Unknown (unknown)
Disk /dev/drbd1: 6597GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 6597GB 6597GB xfs
Model: Unknown (unknown)
Disk /dev/drbd2: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 4398GB 4398GB xfs
Model: Unknown (unknown)
Disk /dev/drbd3: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Unknown (unknown)
Disk /dev/drbd4: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Unknown (unknown)
Disk /dev/drbd5: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
== END parted -l ==
== BEGIN blkid ==
/dev/sda2: UUID="b8c11e24-72ed-4f0a-9fbb-efebc3d67956" TYPE="xfs" PARTUUID="f46afefa-4ee1-43ef-95b9-649cff868e97"
/dev/sda3: UUID="94nxvp-AmhR-DeqH-8kTh-SRY1-pkgD-DNvQhj" TYPE="LVM2_member" PARTUUID="c747b996-7855-41e1-8307-5ac7953f37c7"
/dev/sda4: UUID="z7rrwJ-Wm53-npPk-gNo0-E7qw-ecFl-oOfP3k" TYPE="LVM2_member" PARTUUID="2df9b8e8-9f75-4985-9c57-d2951dccbc3e"
/dev/mapper/centos-swap: UUID="14c07bd7-57d7-42f3-b637-ddee56f4cfeb" TYPE="swap"
/dev/mapper/centos-root: UUID="0899b7b0-7a42-44ab-8e60-193b58907736" TYPE="xfs"
/dev/mapper/DRBD-homeagbu: UUID="d8e829dd339e83a9" TYPE="drbd"
/dev/mapper/DRBD-backdesk: UUID="e5cd491c115d7582" TYPE="drbd"
/dev/mapper/DRBD-genomics: UUID="971797fbff3e2b56" TYPE="drbd"
/dev/mapper/DRBD-backserv: UUID="769898b29d0f3c39" TYPE="drbd"
/dev/mapper/DRBD-agbudata: UUID="22c76dc63c54f5f7" TYPE="drbd"
/dev/mapper/centos-home: UUID="140d6eeb-bf2c-4513-bd74-2e3784b7a6ef" TYPE="xfs"
/dev/drbd5: UUID="a663d717-908b-4cb3-879d-2dd3112fe80e" TYPE="xfs"
/dev/drbd2: UUID="2b490af1-1e8f-4758-9921-ef30fa42b82e" TYPE="xfs"
/dev/drbd4: UUID="5012e580-a9c2-4a25-9a9c-0cb44c69bd73" TYPE="xfs"
/dev/drbd3: UUID="7c92996f-fde8-4df9-9da1-256420f62797" TYPE="xfs"
/dev/drbd1: UUID="5ab7a86c-c8d4-4094-9c1f-ede32c266586" TYPE="xfs"
== END blkid ==
== BEGIN cat /proc/mdstat ==
Personalities :
unused devices: <none>
== END cat /proc/mdstat ==
== BEGIN pvs ==
PV VG Fmt Attr PSize PFree
/dev/sda3 centos lvm2 a-- 130.08g 0
/dev/sda4 DRBD lvm2 a-- 21.70t 5.70t
== END pvs ==
== BEGIN vgs ==
VG #PV #LV #SN Attr VSize VFree
DRBD 1 5 0 wz--n- 21.70t 5.70t
centos 1 3 0 wz--n- 130.08g 0
== END vgs ==
== BEGIN lvs ==
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
agbudata DRBD -wi-ao---- 2.00t
backdesk DRBD -wi-ao---- 4.00t
backserv DRBD -wi-ao---- 2.00t
genomics DRBD -wi-ao---- 2.00t
homeagbu DRBD -wi-ao---- 6.00t
home centos -wi-ao---- 48.83g
root centos -wi-ao---- 50.00g
swap centos -wi-ao---- 31.25g
== END lvs ==
Code: Select all
May 6 00:01:01 hans0 systemd: Created slice user-0.slice.
May 6 00:01:01 hans0 systemd: Starting Session 629 of user root.
May 6 00:01:01 hans0 systemd: Started Session 629 of user root.
May 6 01:01:01 hans0 systemd: Created slice user-0.slice.
May 6 01:01:01 hans0 systemd: Starting Session 630 of user root.
May 6 01:01:01 hans0 systemd: Started Session 630 of user root.
May 6 01:17:36 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:37 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:39 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:41 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:43 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:45 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
....
May 6 02:07:51 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 02:07:53 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 02:07:55 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
....
May 6 04:07:47 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 04:07:49 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 04:07:49 hans0 kernel: net_ratelimit: 55 callbacks suppressed
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket
May 6 04:08:05 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)