368

While running a C program, It says "(core dumped)" but I can't see any files under the current path.

I have set and verified the ulimit:

ulimit -c unlimited 
ulimit -a 

I also tried to find a file named "core", but didn't get the core dumped file?
Any help, where is my core file?

8
  • 1
    Does the program invoke chdir at some point? If so, look there. Jan 14, 2010 at 17:03
  • 2
    Does the program change its working directory? Look there. Jan 14, 2010 at 17:03
  • 1
    oops no its not there... I checked it ..program chdir to /mnt and / i checked both directories but could not find the file. I even did find / -name "*core." even this didn't show me the file. The program uses C + sqlite ,while inserting values it core dumps It said assertion error==0 for the first time and error=101 for the second time.. Jan 14, 2010 at 17:25
  • 10
    Yes, if you override /proc/sys/kernel/core_pattern with a string starting with /tmp then that's where your core dumps will go.
    – ephemient
    Jan 14, 2010 at 20:43
  • 1

16 Answers 16

292

Read /usr/src/linux/Documentation/sysctl/kernel.txt.

core_pattern is used to specify a core dumpfile pattern name.

  • If the first character of the pattern is a '|', the kernel will treat the rest of the pattern as a command to run. The core dump will be written to the standard input of that program instead of to a file.

Instead of writing the core dump to disk, your system is configured to send it to the abrt (meaning: Automated Bug Reporting Tool, not "abort") program instead. Automated Bug Reporting Tool is possibly not as documented as it should be...

In any case, the quick answer is that you should be able to find your core file in /var/cache/abrt, where abrt stores it after being invoked. Similarly, other systems using Apport may squirrel away cores in /var/crash, and so on.

6
  • 41
    yes,I have edited core_pattern with following content echo "core.%e.%p" > /proc/sys/kernel/core_pattern ..now it creates the core dump file in current directory itself. with name "core.giis.12344" etc. Thank you all for your answers/comments/hints . Jan 15, 2010 at 6:57
  • 24
    Just to note that fedora 18 abrt program is storing core dumps in /var/spool/abrt/ instead of /var/cache/abrt
    – Nelson
    Mar 6, 2013 at 9:31
  • 4
    Is there a way to allow users to configure this for themselves, rather than everyone having to use the system configuration? Oct 17, 2013 at 19:57
  • When this command is run and process is invoked, are stdout and stderr not opened by default? I see very strange things happenning. when i use dup2 of stderr and stdout into my custom file(applog.txt), Data which i am writing into other file(mycore.BIN) is being redirected to file which i have used to catch stdout & stderr(applog.txt). Is there a good read about this one? Please suggest. Thank you
    – Sandeep
    Jun 2, 2014 at 7:53
  • 33
    systemd in archlinux store coredumps in /var/lib/systemd/coredump/
    – Francois
    Jun 21, 2016 at 8:46
277
+300

On recent Ubuntu (12.04 in my case), it's possible for "Segmentation fault (core dumped)" to be printed, but no core file produced where you might expect one (for instance for a locally compiled program).

This can happen if you have a core file size ulimit of 0 (you haven't done ulimit -c unlimited) -- this is the default on Ubuntu. Normally that would suppress the "(core dumped)", cluing you into your mistake, but on Ubuntu, corefiles are piped to Apport (Ubuntu's crash reporting system) via /proc/sys/kernel/core_pattern, and this seems to cause the misleading message.

If Apport discovers that the program in question is not one it should be reporting crashes for (which you can see happening in /var/log/apport.log), it falls back to simulating the default kernel behaviour of putting a core file in the cwd (this is done in the script /usr/share/apport/apport). This includes honouring ulimit, in which case it does nothing. But (I assume) as far as the kernel is concerned, a corefile was generated (and piped to apport), hence the message "Segmentation fault (core dumped)".

Ultimately PEBKAC for forgetting to set ulimit, but the misleading message had me thinking I was going mad for a while, wondering what was eating my corefiles.

(Also, in general, the core(5) manual page -- man 5 core -- is a good reference for where your core file ends up and reasons it might not be written.)

10
  • 4
    Thank you so much -- I ran into the very same problem. Btw, Ubuntu 14.04 exhibits exactly the same behavior as the one you described. Jul 22, 2015 at 10:19
  • 9
    On my Ubuntu install (modified 14.04), there's an easy temporary workaround for this by running sudo service apport stop --- after I ran that, it changed /proc/sys/kernel/core_pattern from the apport pipe to just core. Apport is smart enough to fix up the core_pattern temporarily, I suppose. Feb 22, 2016 at 18:50
  • 9
    "ulimit -c unlimited" is exactly what I needed - Thank you!
    – Dave C
    Mar 20, 2016 at 18:07
  • 7
    I have tried every applicable answer, and every location in every comment here after doing the ulimit command, but still can't find a core file anywhere on my Ubuntu 16.04 LTS...
    – Nagev
    Apr 5, 2017 at 13:39
  • 7
    @ElectricGoat No I haven't. It's poor UX design, IMO. The system should just print the path, or not print a message at all if it's not being created. I'll add my own answer, when or if I get a chance to investigate this further.
    – Nagev
    Aug 17, 2017 at 7:29
103

With the launch of systemd, there's another scenario as well. By default systemd will store core dumps in its journal, being accessible with the systemd-coredumpctl command. Defined in the core_pattern-file:

$ cat /proc/sys/kernel/core_pattern 
|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e

Easiest way to check for stored core dumps is via coredumpctl list (older core dumps may have been removed automatically). This behaviour can be disabled with a simple "hack":

$ ln -s /dev/null /etc/sysctl.d/50-coredump.conf
$ sysctl -w kernel.core_pattern=core      # or just reboot

As always, the size of core dumps has to be equal or higher than the size of the core that is being dumped, as done by for example ulimit -c unlimited.

5
  • That "hack" didn't work for me (even after a restart). I'm running Arch Linux and I've already run ulimit -c unlimited.
    – gsgx
    Oct 16, 2013 at 5:15
  • @gsingh2011 It might be outdated. I don't run Arch anymore so I can't say if it should work these days. If you figure it out feel free to update me/us with a new comment.
    – timss
    Oct 16, 2013 at 10:58
  • 6
    @gsingh2011 Try 50-coredump.conf instead of coredump.conf. This should override /lib/sysctl.d/50-coredump.conf. The default can be restored with sysctl -w kernel.core_pattern=core
    – Lekensteyn
    Nov 15, 2013 at 10:07
  • I had to turn off appport for this to work sudo service apport stop
    – rahul003
    Feb 2, 2018 at 0:06
  • On Ubuntu 21.10 I had to install systemd-coredump first, before I could get the coredumpctl command to work. After that it was easy ;)
    – oligofren
    Apr 27, 2022 at 10:48
95

Writing instructions to get a core dump under Ubuntu 16.04 LTS:

  1. As @jtn has mentioned in his answer, Ubuntu delegates the display of crashes to apport, which in turn refuses to write the dump because the program is not an installed package. Before making changes

  2. To remedy the problem, we need to make sure apport writes core dump files for non-package programs as well. To do so, create a file named ~/.config/apport/settings with the following contents:
    [main] unpackaged=true

  3. Now crash your program again, and see your crash files being generated within folder: /var/crash with names like *.1000.crash. Note that these files cannot be read by gdb directly. After making changes
  4. [Optional] To make the dumps readble by gdb, run the following command:

    apport-unpack <location_of_report> <target_directory>

References: Core_dump – Oracle VM VirtualBox

5
  • 7
    This is the only solution that worked for me in Ubuntu 18.04
    – greuze
    Oct 31, 2018 at 8:58
  • Which user does ~/ pertain to? If process is run by root does that mean /root/.config/apport/settings ?
    – Nicholi
    Jun 25, 2019 at 0:39
  • @Nicholi I believe it should be the user that executed the program. I'm not sure though.
    – ankurrc
    Jun 27, 2019 at 21:36
  • This still works as of Ubuntu 21.04. Dec 2, 2021 at 3:57
  • Worked for Ubuntu 20.04.4
    – Not Saying
    Jun 13, 2022 at 15:32
13

I could think of two following possibilities:

  1. As others have already pointed out, the program might chdir(). Is the user running the program allowed to write into the directory it chdir()'ed to? If not, it cannot create the core dump.

  2. For some weird reason the core dump isn't named core.* You can check /proc/sys/kernel/core_pattern for that. Also, the find command you named wouldn't find a typical core dump. You should use find / -name "*core.*", as the typical name of the coredump is core.$PID

3
  • here is my pattern - does this mean core file is named something like"PID.signal.userid" instead of core.pid ??? $cat /proc/sys/kernel/core_pattern /usr/lib/hookCCpp /var/char/abrt %p %s %u Jan 14, 2010 at 17:55
  • 1
    If contents of virtual file /proc/sys/kernel/core_pattern starts with a pipe symbol | the file will not be written to file defined by core_pattern but the command defined after the the pipe symbol is launched and the core file will be written to stdin of that program. It's 100% up to the program to handle the core file contents after that. Sep 20, 2021 at 16:53
  • On item 1: No, the program won't core dump; it would see an EPERM. Only if the program fails to handle the error correctly it may be forced to terminate (to prevent even more nonsense to happen).
    – U. Windl
    Jan 5, 2022 at 10:28
12

In Ubuntu18.04, the most easist way to get a core file is inputing the command below to stop the apport service.

sudo service apport stop

Then rerun the application, you will get dump file in current directory.

2
  • I tried this and got /etc/init.d/apport: 68: /etc/init.d/apport: cannot create /proc/sys/fs/suid_dumpable: Operation not permitted still cannot find the crash files. I have already set ulimit -c unlimited. Somebody mentioned the /proc/sys/kernel/core_pattern, I can even not see this file. What else should be done?
    – doraemon
    Nov 7, 2020 at 3:15
  • sorry, since I have not encountered the problem, I cannot give a good answer for you. However, I think you can learn the knowledge of apport service, then you can know where the core dump file. Here is the document of apport: https://wiki.ubuntu.com/Apport. Nov 16, 2020 at 3:43
12

I found core files of my Ubuntu 20.04 system at;

/var/lib/apport/coredump 
9

If you're missing core dumps for binaries on RHEL and when using abrt, make sure that /etc/abrt/abrt-action-save-package-data.conf

contains

ProcessUnpackaged = yes

This enables the creation of crash reports (including core dumps) for binaries which are not part of installed packages (e.g. locally built).

1
  • You miss the fact that when "(core dumped)" is output, a core was dumped; if no core was dumped, you do not see that message.
    – U. Windl
    Jan 5, 2022 at 10:29
8

For fedora25, I could find core file at

/var/spool/abrt/ccpp-2017-02-16-16:36:51-2974/coredump

where ccpp-2017-02-16-16:36:51-2974" is pattern "%s %c %p %u %g %t %P % as per `/proc/sys/kernel/core_pattern'

7

I'm on Linux Mint 19 (Ubuntu 18 based). I wanted to have coredump files in current folder. I had to do two things:

  1. Change /proc/sys/kernel/core_pattern (by # echo "core.%p.%s.%c.%d.%P > /proc/sys/kernel/core_pattern or by # sysctl -w kernel.core_pattern=core.%p.%s.%c.%d.%P)
  2. Raising limit for core file size by $ ulimit -c unlimited

That was written already in the answers, but I wrote to summarize succinctly. Interestingly changing limit did not require root privileges (as per https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user non-root can only lower the limit, so that was unexpected - comments about it are welcome).

1
  • sudo sysctl -w kernel.core_pattern=core command helped me to create the core file in the current dir. Thanks.
    – irsis
    Feb 18 at 10:29
6

My efforts in WSL have been unsuccessful.

For those running on Windows Subsystem for Linux (WSL) there seems to be an open issue at this time for missing core dump files.

The comments indicate that

This is a known issue that we are aware of, it is something we are investigating.

Github issue

Windows Developer Feedback

6

In my case, the reason is that the ulimit command only effect the current terminal.

If I set ulimit -c unlimited on the first terminal. Then I start a new terminal to run the program. It will not generate the core file when core dumped.

You have to confirm the core size of the terminal which runs your program.

The following steps work on ubuntu 20.04 and ubuntu 21.04:

  1. stop apport service
sudo service apport stop
  1. set core size of the terminal which ready to run your program
ulimit -c unlimited
3

ulimit -c unlimited made the core file correctly appear in the current directory after a "core dumped".

3

If you use Fedora, in order to generate core dump file in the same directory of binary file:

echo "core.%e.%p" > /proc/sys/kernel/core_pattern

And

ulimit -c unlimited
1
  • This is fine in itself, but if you want to work with the existing system setup, instead of against it, try to figure out how it is setup and inspect the docs for whatever command is capturing the dumps to get to them. See stackoverflow.com/a/14082174/200987 for details.
    – oligofren
    Apr 27, 2022 at 10:51
0

A one-liner to get the latest core dump path:

ls -t $(cat /proc/sys/kernel/core_pattern | awk -F% '{print $1"*"}') 2>/dev/null | head -1

You can of course modify the last -1 on that line to e.g. -4 to get the last 4 core dumps.

Note: That's not expected to work e.g. in case the path pattern uses variables before the last / or when non core dump files are on that dir.

1
  • 2
    or you can just use coredumpctl -1 info Oct 22, 2023 at 11:03
0

The most easy way to start debugging a core dump I found is using coredumpctl (man coredumpctl for more info).

To start a debug session using the latest segmentation fault, just type

$ coredumpctl debug

It is also useful to find and extract core dump files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.