Out of memory, smb core dump

General support questions
Post Reply
ttyz
Posts: 9
Joined: 2020/05/19 00:42:59

Out of memory, smb core dump

Post by ttyz » 2021/02/22 03:40:05

Hi all,

Recently I've got a SMB server (v4.6.2) set up on CentOS 7.3, kernel 3.10.0-957.21.3.el7.x86_64. In this set up, I have a windows server permanently mapped to the the my SMB server and in the windows server I have a program to constantly delete files after the files are downloaded to the windows server. In my CentOS 7.3 (Same server running the SMB server), I have a program to constantly generate and move files to the SMB server share folder for the windows server to pick up the file.
Upon this set up, I realized that I have a huge file (100GB or more) named "core.<PID>" (not a file name I would generate) appearing every 2-4 days on the SMB server share directory in CentOS 7.3. With this, I proceeded to look the logs in both log.smbd and var/log/messages and this is what was recorded:

In log.smbd:

Code: Select all

 smbd_cleanupd_unlock: Cleaning up brl and lock database after unclean shutdown
[2020/09/27 04:15:37.675693,  0] ../source3/lib/popt_common.c:67(popt_s3_talloc_log_fn)
[2020/09/22 04:15:37.676301,  0] ../source3/lib/util.c:791(smb_panic_s3)
[2020/09/22 04:15:37.676640,  0] ../source3/lib/util.c:902(log_stack_trace)
[2020/09/22 04:15:37.676721,  0] ../source3/lib/util_nttoken.c:44(dup_nt_token)
[2020/09/22 04:15:37.676788,  0] ../source3/smbd/sec_ctx.c:226(push_sec_ctx)
[2020/09/22 04:15:37.676845,  0] ../source3/lib/util.c:791(smb_panic_s3)
[2020/09/22 04:15:37.676904,  0] ../source3/lib/util.c:902(log_stack_trace)
[2020/09/22 04:15:37.676963,  0] ../source3/lib/dumpcore.c:273(dump_core)
[2020/09/22 04:15:37.678843,  0] ../source3/lib/popt_common.c:67(popt_s3_talloc_log_fn)
  Bad talloc magic value - unknown value
[2020/09/22 04:15:37.678988,  0] ../source3/lib/util.c:791(smb_panic_s3)
  PANIC (pid 13296): Bad talloc magic value - unknown value
[2020/09/22 04:15:37.679051,  0] ../source3/lib/util.c:902(log_stack_trace)
  BACKTRACE: 0 stack frames:
[2020/09/22 04:15:37.793060,  1] ../source3/smbd/server.c:868(remove_child_pid)
  Scheduled cleanup of brl and lock database after unclean shutdown
In /var/log/message:

Code: Select all

Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost journal: out of memory [13296]
Sep 22 04:15:37 localhost smbd[13296]: [2020/09/22 04:15:37.478563,  0] ../source3/lib/popt_common.c:67(popt_s3_talloc_log_fn)
Sep 22 04:15:37 localhost smbd[13296]:  Bad talloc magic value - unknown value
Sep 22 04:15:37 localhost smbd[13296]: [2020/09/22 04:15:37.478722,  0] ../source3/lib/util.c:791(smb_panic_s3)
Sep 22 04:15:37 localhost smbd[13296]:  PANIC (pid 13296): Bad talloc magic value - unknown value
Sep 22 04:15:37 localhost smbd[13296]: [2020/09/22 04:15:37.478770,  0] ../source3/lib/util.c:902(log_stack_trace)
Sep 22 04:15:37 localhost smbd[13296]:  BACKTRACE: 0 stack frames:
Sep 22 04:15:37 localhost smbd[13296]: [2020/09/22 04:15:37.478819,  0] ../source3/lib/dumpcore.c:273(dump_core)
Sep 22 04:15:37 localhost smbd[13296]:  dump_core() called recursive
I am not too sure from the logs if these errors are related, could it be that my system went out of memory that caused the smb core dump and the creation of the large core.pid file? Pardon my lack of experience and knowledge, but is there any other troubleshooting steps I can try to find out the cause of the large core.pid file created?

pjsr2
Posts: 614
Joined: 2014/03/27 20:11:07

Re: Out of memory, smb core dump

Post by pjsr2 » 2021/02/22 08:17:14

Yes, the two are related.

Your system is running out of memory. This causes a failure of smbd or the smbd gets killed by the operating system. The operating system will kill processes when the system runs out of memory, in a desperate attempt to limit damage.

When a system runs out of memory, any program that runs at that moment my start to show warnings or errors. This happens when a program does something very normal, but as it this time happens not to get memory when it asks for it, it fails. So in a situation like this you may see error messages from multiple processes showing up in your log file.

The core.<pid> file is essentially a memory dump of a process. The core files are created when a program crashes or when the process receives the SIGABRT signal. These files are useful for debugging a program. (By default, programs running as root do not generate core dumps for security reasons.)

The command

Code: Select all

file core
will show you which program generated the core file.

Your core file is exceptionally large. Looks like something is going wrong in smdb.

CentOS 7.3 is very outdated. First thing to do is update to the latest CentOS version. There are many security fixes, some of which are in the field of the SMB protocol. If your problem persists in the latest CentOS 7.x version, you should follow the smbd process and try to discover what actions make that the memory usage grows. Is it growing gradually or step wise? Check the smbd log files for clues.

ttyz
Posts: 9
Joined: 2020/05/19 00:42:59

Re: Out of memory, smb core dump

Post by ttyz » 2021/02/23 07:11:47

Thanks for the suggestion/reply.

Understand that this version is CentOS is old, but it's being used for some time now, and since the current set up has yet to have any issues, I'm afraid that when an update is performed, it might Impact the system as a whole. Is there any other way to fix this issue without performing an update?
Meanwhile, I will try the proposed set up from scratch to see if the issue will be resolved.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Out of memory, smb core dump

Post by TrevorH » 2021/02/23 09:34:27

Only the current version of CentOS is supported. For 7 that is 7.9. You should not run 7.3 as it dates from 2016 and has numerous high severity security problems.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

BShT
Posts: 584
Joined: 2019/10/09 12:31:40

Re: Out of memory, smb core dump

Post by BShT » 2021/02/23 13:52:05

it is uncommon to update a Centos and break it

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Out of memory, smb core dump

Post by jlehtone » 2021/02/23 15:43:05

ttyz wrote:
2021/02/23 07:11:47
Understand that this version is CentOS is old, but it's being used for some time now, and since the current set up has yet to have any issues,
All issues are not easy to observe.

Programmer's mantra: Crash is Good. Silent corruption is Bad.

poky
Posts: 108
Joined: 2013/03/27 12:18:03

Re: Out of memory, smb core dump

Post by poky » 2021/02/27 12:58:15

yum clean all
yum update samba

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Out of memory, smb core dump

Post by TrevorH » 2021/02/27 13:53:25

Just updating samba is probably not good enough. From 7.3 to 7.9 is about 4 years and several thousand bug fixes apart and it's very possible that the problem lies in e.g. the kernel or glibc.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply