XFS: possible memory allocation deadlock in kmem_alloc (mode

General support questions
Post Reply
kgore4
Posts: 9
Joined: 2015/04/22 23:33:01

XFS: possible memory allocation deadlock in kmem_alloc (mode

Post by kgore4 » 2015/05/06 01:06:15

At about 1:17am this morning, I started seeing "XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)" and it just kept spewing that out until I rebooted at 9:30am. The machine is a dedicated file server with nfs and samba exports. The time would probably correspond with when the other servers are backing up to this one. The servers that import file systems from this one via nfs just sat there stuck until the reboot.

Does anyone have any suggestions to resolve it?

Possibly relevant information gleaned from a bit of googling
- I'm using the default xfs parameters i.e. not messing with the block sizes. http://tracker.ceph.com/issues/6301 talks about an already applied patch to centos7 but seems to boil down to messing with the block sizes.
- single xeon E5-2609v2, 32gb ram, 7x4g sas raid 5 on lsi controller with hot spare (getinfo for disks below)
- not running ceph
- drbd is replicating and status was ok


Information for driver problems.

Code: Select all

== BEGIN uname -rmi ==
3.10.0-123.6.3.el7.x86_64 x86_64 x86_64
== END   uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
centos-release-7-0.1406.el7.centos.2.5.x86_64
elrepo-release-7.0-2.el7.elrepo.noarch
epel-release-7-2.noarch
== END   rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
CentOS Linux release 7.0.1406 (Core)
== END   cat /etc/redhat-release ==

== BEGIN getenforce ==
Enforcing
== END   getenforce ==

== BEGIN free -m ==
             total       used       free     shared    buffers     cached
Mem:         31980      18651      13328          8          5      16246
-/+ buffers/cache:       2399      29580
Swap:        31999          0      31999
== END   free -m ==

== BEGIN rpm -qa kernel\* | sort ==
kernel-3.10.0-123.6.3.el7.x86_64
kernel-3.10.0-123.el7.x86_64
kernel-tools-3.10.0-123.6.3.el7.x86_64
kernel-tools-libs-3.10.0-123.6.3.el7.x86_64
== END   rpm -qa kernel\* | sort ==

== Warning: lspci is not installed ==

== Warning: lsusb is not installed ==

== BEGIN rpm -qa kmod\* kmdl\* ==
kmod-libs-14-9.el7.x86_64
kmod-drbd84-8.4.5-1.el7.elrepo.x86_64
kmod-14-9.el7.x86_64
== END   rpm -qa kmod\* kmdl\* ==

Code: Select all

== BEGIN uname -rmi ==
3.10.0-123.6.3.el7.x86_64 x86_64 x86_64
== END   uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
centos-release-7-0.1406.el7.centos.2.5.x86_64
elrepo-release-7.0-2.el7.elrepo.noarch
epel-release-7-2.noarch
== END   rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
CentOS Linux release 7.0.1406 (Core)
== END   cat /etc/redhat-release ==

== BEGIN getenforce ==
Enforcing
== END   getenforce ==

== BEGIN free -m ==
             total       used       free     shared    buffers     cached
Mem:         31980      18156      13824          8          1      15758
-/+ buffers/cache:       2397      29583
Swap:        31999          0      31999
== END   free -m ==

== BEGIN cat /etc/fstab ==

#
# /etc/fstab
# Created by anaconda on Tue Sep  9 19:33:28 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        1 1
UUID=b8c11e24-72ed-4f0a-9fbb-efebc3d67956 /boot                   xfs     defaults        1 2
/dev/mapper/centos-home /home                   xfs     defaults        1 2
/dev/mapper/centos-swap swap                    swap    defaults        0 0


== END   cat /etc/fstab ==

== BEGIN df -h ==
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  2.1G   48G   5% /
devtmpfs                  16G     0   16G   0% /dev
tmpfs                     16G     0   16G   0% /dev/shm
tmpfs                     16G  8.7M   16G   1% /run
tmpfs                     16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/centos-home   49G   33M   49G   1% /home
/dev/sda2                497M  126M  372M  26% /boot
/dev/drbd1               6.0T  2.9T  3.2T  47% /mnt/homeagbu
/dev/drbd2               4.0T  2.2T  1.9T  55% /mnt/backdesk
/dev/drbd3               2.0T  322G  1.7T  16% /mnt/backserv
/dev/drbd4               2.0T  1.6T  430G  80% /mnt/genomics
/dev/drbd5               2.0T  224G  1.8T  11% /mnt/agbudata
== END   df -h ==

== BEGIN fdisk -lu ==

Disk /dev/sda: 24000.0 GB, 23999998328832 bytes, 46874996736 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1  4294967295  2147483647+  ee  GPT

Disk /dev/mapper/centos-swap: 33.6 GB, 33554432000 bytes, 65536000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/DRBD-homeagbu: 6597.1 GB, 6597069766656 bytes, 12884901888 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/DRBD-backdesk: 4398.0 GB, 4398046511104 bytes, 8589934592 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/DRBD-genomics: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/DRBD-backserv: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/DRBD-agbudata: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-home: 52.4 GB, 52428800000 bytes, 102400000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

== END   fdisk -lu ==

== BEGIN parted -l ==
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 24.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2097kB  1049kB                     bios_grub
 2      2097kB  526MB   524MB   xfs
 3      526MB   140GB   140GB                      lvm
 4      140GB   24.0TB  23.9TB


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-home: 52.4GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  52.4GB  52.4GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-agbudata: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  2199GB  2199GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-backserv: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  2199GB  2199GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-genomics: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  2199GB  2199GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-backdesk: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  4398GB  4398GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-homeagbu: 6597GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  6597GB  6597GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-root: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  53.7GB  53.7GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-swap: 33.6GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system     Flags
 1      0.00B  33.6GB  33.6GB  linux-swap(v1)


Model: Unknown (unknown)
Disk /dev/drbd1: 6597GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  6597GB  6597GB  xfs


Model: Unknown (unknown)
Disk /dev/drbd2: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  4398GB  4398GB  xfs


Model: Unknown (unknown)
Disk /dev/drbd3: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  2199GB  2199GB  xfs


Model: Unknown (unknown)
Disk /dev/drbd4: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  2199GB  2199GB  xfs


Model: Unknown (unknown)
Disk /dev/drbd5: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  2199GB  2199GB  xfs


== END   parted -l ==

== BEGIN blkid ==
/dev/sda2: UUID="b8c11e24-72ed-4f0a-9fbb-efebc3d67956" TYPE="xfs" PARTUUID="f46afefa-4ee1-43ef-95b9-649cff868e97"
/dev/sda3: UUID="94nxvp-AmhR-DeqH-8kTh-SRY1-pkgD-DNvQhj" TYPE="LVM2_member" PARTUUID="c747b996-7855-41e1-8307-5ac7953f37c7"
/dev/sda4: UUID="z7rrwJ-Wm53-npPk-gNo0-E7qw-ecFl-oOfP3k" TYPE="LVM2_member" PARTUUID="2df9b8e8-9f75-4985-9c57-d2951dccbc3e"
/dev/mapper/centos-swap: UUID="14c07bd7-57d7-42f3-b637-ddee56f4cfeb" TYPE="swap"
/dev/mapper/centos-root: UUID="0899b7b0-7a42-44ab-8e60-193b58907736" TYPE="xfs"
/dev/mapper/DRBD-homeagbu: UUID="d8e829dd339e83a9" TYPE="drbd"
/dev/mapper/DRBD-backdesk: UUID="e5cd491c115d7582" TYPE="drbd"
/dev/mapper/DRBD-genomics: UUID="971797fbff3e2b56" TYPE="drbd"
/dev/mapper/DRBD-backserv: UUID="769898b29d0f3c39" TYPE="drbd"
/dev/mapper/DRBD-agbudata: UUID="22c76dc63c54f5f7" TYPE="drbd"
/dev/mapper/centos-home: UUID="140d6eeb-bf2c-4513-bd74-2e3784b7a6ef" TYPE="xfs"
/dev/drbd5: UUID="a663d717-908b-4cb3-879d-2dd3112fe80e" TYPE="xfs"
/dev/drbd2: UUID="2b490af1-1e8f-4758-9921-ef30fa42b82e" TYPE="xfs"
/dev/drbd4: UUID="5012e580-a9c2-4a25-9a9c-0cb44c69bd73" TYPE="xfs"
/dev/drbd3: UUID="7c92996f-fde8-4df9-9da1-256420f62797" TYPE="xfs"
/dev/drbd1: UUID="5ab7a86c-c8d4-4094-9c1f-ede32c266586" TYPE="xfs"
== END   blkid ==

== BEGIN cat /proc/mdstat ==
Personalities :
unused devices: <none>
== END   cat /proc/mdstat ==

== BEGIN pvs ==
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda3  centos lvm2 a--  130.08g    0
  /dev/sda4  DRBD   lvm2 a--   21.70t 5.70t
== END   pvs ==

== BEGIN vgs ==
  VG     #PV #LV #SN Attr   VSize   VFree
  DRBD     1   5   0 wz--n-  21.70t 5.70t
  centos   1   3   0 wz--n- 130.08g    0
== END   vgs ==

== BEGIN lvs ==
  LV       VG     Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  agbudata DRBD   -wi-ao----  2.00t
  backdesk DRBD   -wi-ao----  4.00t
  backserv DRBD   -wi-ao----  2.00t
  genomics DRBD   -wi-ao----  2.00t
  homeagbu DRBD   -wi-ao----  6.00t
  home     centos -wi-ao---- 48.83g
  root     centos -wi-ao---- 50.00g
  swap     centos -wi-ao---- 31.25g
== END   lvs ==

Code: Select all

May  6 00:01:01 hans0 systemd: Created slice user-0.slice.
May  6 00:01:01 hans0 systemd: Starting Session 629 of user root.
May  6 00:01:01 hans0 systemd: Started Session 629 of user root.
May  6 01:01:01 hans0 systemd: Created slice user-0.slice.
May  6 01:01:01 hans0 systemd: Starting Session 630 of user root.
May  6 01:01:01 hans0 systemd: Started Session 630 of user root.
May  6 01:17:36 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 01:17:37 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 01:17:39 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 01:17:41 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 01:17:43 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 01:17:45 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
....
May  6 02:07:51 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 02:07:53 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 02:07:55 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May  6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May  6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May  6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
....
May  6 04:07:47 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 04:07:49 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May  6 04:07:49 hans0 kernel: net_ratelimit: 55 callbacks suppressed
May  6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May  6 04:07:50 hans0 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket
May  6 04:08:05 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
Top
User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:
Contact avij

Re: XFS: possible memory allocation deadlock in kmem_alloc (

Post by avij » 2015/05/06 03:57:18

If I were you, I would try upgrading to 7.1.1503 first to see if it helps.
Top
aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: XFS: possible memory allocation deadlock in kmem_alloc (

Post by aks » 2015/05/06 16:42:20

If you have a RH subscription see https://access.redhat.com/solutions/532663

Otherwise, quite a few people have said:
1) Happens on heavily fragmented filesystems (use xfs_db -c frag or xfs_info I think). You should defrag ASAP.
2) As a temporary work around drop the caches (echo 1 > /proc/sys/vm/drop_caches), which'll free some RAM.
Source: https://bugzilla.kernel.org/show_bug.cgi?id=73831
Top
User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:
Contact toracat

Re: XFS: possible memory allocation deadlock in kmem_alloc (

Post by toracat » 2019/08/22 15:31:46

aks wrote: ↑
2015/05/06 16:42:20
 If you have a RH subscription see https://access.redhat.com/solutions/532663
Four years later, at long last, the bug has been fixed in kernel-3.10.0-1062.el7. This is the GA kernel in EL7.7. CentOS will release this version in the CR repository very soon.
CentOS Forum FAQ
Top
Post Reply

Return to “CentOS 7 - General Support”