-
Notifications
You must be signed in to change notification settings - Fork 74.7k
Illegal instruction (core dumped) in a CPU with AVX support #45744
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Here I attach the output of running an interactive python shell with |
@cserpell, |
Sure, I already did it in a new virtualenv, but here it goes again in a brand new one. |
I am having the same problem (also on Ubuntu 20.10, also installed through pip). Whenever I import tensorflow and python crashes and I look in journalctl I see something like this:
|
I also see the same issue on an older server after upgrading to 2.4.0. Crashes consistently at same place while importing tensorflow according to
Here's one of the cores from
This is running Ubuntu 18.04.4 LTS. |
@necromuralist did you check AVX support on your CPU? Otherwise, please check #19584 |
Same error for me in
|
Just to pile on, same problem here, 2.4.0 doesn't work, 2.3.0 and earlier do. CPU supports AVX:
|
Couple of things to add though:
|
If I do (asterixes added) I get |
I think I also have the same problem as @alex-orange. 2.4 results in illegal instruction error, whereas 2.3 works. Well, the GPU doesn't work since my CUDA version is incompatible with 2.3 but at least tensorflow works.
|
I just tried the nightly build of tensorflow and it doesn't crash. I used the nvidia-container to get the Nvidia toolkit instead of using Ubuntu's packange and tried to install the latest release of tensorflow via both pip and the tensorflow-container and both still crashed the python interpreter, but when I did
And imported tensorflow into a python intepreter with no errors. Unfortunately in experimenting to get to that state I crashed my server and had to re-install Ubuntu so I don't have anything on it to test whether tensorflow actually runs, but it seems promising. |
@necromuralist Sadly I don't think that's progress. #44668 seems to suggest this problem was introduced when nightly was split off to make the release candidate (or whatever the exact process is). Also worth mentioning #45872 and #45866 seem to be the exact same bug. Adding more info: #45866 seems to suggest that this might be a problem with processors supporting AVX but not AVX2. Is it possible that this requirement got bumped (perhaps by accident)? |
I've just tried building from source with:
and the resulting binary works fine. Is it possible someone is building the packages with --config=opt or --config=avx2_linux by accident? |
@alex-orange Is it somehow possible that you share your custom-built package or does everyone need to build for themselves at the moment? |
@moorugi98 Sure, here's my setup: https://gitlab.flux.utah.edu/alex_orange/tensorflow-2.4.0-build . That includes the Dockerfile I use for my build environment and the command history for building and testing. Not exactly perfect but you should be able to figure out how to make your own build (and the maintainers can hopefully use this to get an idea of what's different from the official build). |
Didn't say it very well, but that repo also includes the binary. |
We found the culprit, working on a fix and then we will do a patch release on |
We should have the patched 2.4.1 release done by the end of the week |
And it has been released. Please test it to see that the issue has been fixed |
It is working now. Many thanks for your hard work, I really appreciate it. |
I'm using Tensorflow 2.4.1, Python 3.6 and I'm still getting |
@InnocentCow does your CPU support AVX? This issue is for AVX2. |
Try |
Tried this, still same error |
I have an Intel Celeron N3550, and from what I've found it does support AVX |
Nevermind, I get Seems like my CPU doesn't support AVX after all, I'll downgrade to TensorFlow 1.5 TensorFlow 1.5 and keras 2.1.4 did the trick for me, sorry for bothering you guys |
Never mind, happy hacking! |
Since previous issues relating to tensorflow/tensorflow#45744 should now be fixed, this should be OK. modified: requirements.txt
have same issue, installed 2.4.1 version, but it didn't help. |
I'm trying to install 2.4.1 onto my Jetson Nano, and I'm having the illegal instructions error, can you help me with compiling it from sources? I am a complete noob when it comes to Linux and python and stuff! |
|
update tensorflow version to 2.4.1. in some env, 2.4.0 will cause `illegal instruction (core dumped)`: tensorflow/tensorflow#45744
CPUs with no AVX support are no longer supported by the default package. You have to compile from source or use a cloud version of TF (via Google Colab for example). |
System information
Describe the current behavior
I was using tensorflow 2 without any issue until a few days ago. I installed it using pip in a new virtualenv environment and now it throws illegal instruction error. My CPU is an i5-3230M, which, according to /proc/cpuinfo supports AVX, so it seems not related to #19584 .
My CPU (according to
/proc/cpuinfo
):model name : Intel(R) Core(TM) i5-3230M CPU @ 2.60GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d
System:
Describe the expected behavior
The import should work and not give the
Illegal instruction (core dumped)
error.Standalone code to reproduce the issue
Other info / logs
The text was updated successfully, but these errors were encountered: