-
Notifications
You must be signed in to change notification settings - Fork 74.7k
Illegal instruction (core dumped) after running import tensorflow #17411
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks. |
I'm having the same (or similar) "illegal instruction" problem when I run
I'm only using the CPU 1.6 version on 64 bit Ubuntu Linux. After downgrading to the CPU 1.5 version, it doesn't have this problem. |
How i can downgrade to the CPU 1.5 version? |
Try running EDIT |
Thanks konnerthg, even i was having the same problem. Your command helped me to sort this issue. Thanks again. |
Same here. |
downgrade to 1.5 worked for me, too |
@konnerthg Downgrading to 1.5 is just work around, this issue is not solved yet. |
I am also getting this error in python 3.6 |
Hey ! |
THANKS for solution.It worked on my Ubuntu 16.04, 64 bit, python3.5 . |
Thanks for the solution! Downgrading to version 1.5 fixed the issue. Tested on a Ubuntu 16.04 server with python 2.7 |
Same issue, downgrading from Tensorflow 1.6 to 1.5 solved it. Running Xubuntu 16.04 64-bit, Python 3.5. |
Thanks for all this solve my issue on Python 3.6
now works without any problem ... |
This is really weird. Does anyone know what causes the issue? I'm surprised that TensorFlow 1.6 would have a bug this big. |
I am encountering this issue as well with tensorflow-gpu 1.6.0, on linux, using python 3.6.4. I have installed tensorflow using pip itself. Simply running this produces a SIGILL:
I get stack traces similar to what is mentioned in this ticket's description. This seems to be occurring due to the use of AVX instructions in the latest Tensorflow packages uploaded to pip. Running python3 through GDB and disassembling the crashing function points to this instruction:
Which is an AVX instruction not supported on older or less-featureful CPUs that do not have AVX support. The tensorflow(-gpu) 1.5.0 pip packages do not use AVX instructions, and thus there are no problems using it with these CPUs. The solution would be for a build of tensorflow(-gpu) that is not compiled with AVX instructions to be published (or to build a copy locally). The provided installation instructions do not mention any specific CPU requirements nor how to determine compatibility with the provided binaries. In the meantime, reverting to tensorflow(-gpu) 1.5.0 using something like what @NinemillaKA mentioned above is an effective workaround. |
I have the same issue, and, as many have commented, downgrade from For the record, I tried running tensorflow (CPU-only version) on 2 different computers: Computer 1:
Computer 2:
I agree with @nacl that we should have those requirements about the instruction set more clear, and if possible, a separated, updated build for processors that doesn't support AVX instructions. To be honest, I find a bit discouraging have to work with outdated version of any technology, I think many feel the same. |
The alternative to having a different build for each architecture type is to use dynamic dispatch. IE, PyTorch has one binary for all architectures and selects most efficient ops during runtime @caisq |
Thanks |
I also encounter the same issue. I tried it on two machines, and it works on one of them. First, I installed it on my MacBook Pro. And I did not have any issues.
So I upgraded my MacPro. But this time, I am getting
(Update on 3/30/2018) |
This is still an issue in 1.6 and potentially in 1.7. Why is this closed? @yaroslavvb 's solution seems reasonable. I have downgraded to 1.5 for now. |
Not sure but from this link, since ver1.6.0, intel CPU instruction optimizer had been introduced to tensorflow. I think that probably this is the cause. |
@captainst that's Intel-specific release, different from the official release that you get by doing |
I have the same issue. |
Related: #19584 |
I have this issue with tensorflow-gpu 2.0
|
I'm getting this crash on an i7-3520M which does support AVX. EDIT: Nevermind, the crash happens on a |
Still have the problem with tensorflow 1.14.0 and 2.0.0b1. |
Same error on Linux Mint 19 with 2.0.0b1. |
tf1.5 isn't available in the Debian 8.8 repos. Time to try avx. |
This was ridiculously hard to find on managed cluster nodes, since the OS kills the related python processes before they can even write and flush this "Illegal instructions" line to a output log file, and the exit code of the python process seems to be 0. I was also using 2.0.0-beta1, currently finding out if replacing it with 2.0.0 fixes this. |
I also got this problem. I'm using |
I'm having this problem with Tensorflow 2 runing in a virtual environment in Ubuntu 18.04. It just blows my mind that the Tensorflow developers would put TF 2 as ready and available with this crap happening. NOT Impressed, you TF developers . |
dmesg output (from bash): This is a BIG CPU-RELATED issue. |
After reading this thread and having the same experience, my problem is that my linux conputer is older and has a CPU which does not support the AVX instruction set. I have tensorflow 1.5 on another virtual environment, bu to use tensorflow 2, I amd going to have to run my scripts on Google Colab |
I don't have the knowledge to say if the requirement of AVX makes sense or not. What I know is that the problem presents itself not only with older CPUs, but also with fairly recent ones, like mine Intel N5000. I get that doing deep learning on a N5000 is a bit of a stretch, but if the tensorflow is supported also by the RaspberryPi, I don't see the problem. Anyway, I installed the last version of TensorFlow (2.0) on my Intel N5000 by compiling it from source. It took 14 hours because I had to run the compilation on a single core, since it needs a lot of RAM and I have only 4Gb invited to the party. I took inspiration from this guide here but the experience was far from smooth, there were constantly dependencies missing that I need to install and re-launch the compilation. And some other stuff too that I solved when the compilation crashed. Have fun and thanks for the hassle. Providing through pip a binary already compiled for non-AVX was clearly too much to add in your continuous integration workflow |
Thanks for your response, Luca. I have a really old Ubuntu 18.04 desktop
which works really well except in running the TF2 and as a matter of fact,
the latest PyTorch (1.3.0) I think.
Torch 1.1.0 works fine for me, and regarding TF2, I use Google Colab which
works fine. Am a retired data analyst and maintaining my hobbyist profile
Thans for your post
…On Sat, Dec 7, 2019 at 6:52 PM Luca Olivieri ***@***.***> wrote:
I don't have the knowledge to say if the requirement of AVX makes sense or
not. What I know is that the problem presents itself not only with older
CPUs, but also with fairly recent ones, like mine Intel N5000. I get that
doing deep learning on a N5000 is a bit of a stretch, but if the tensorflow
is supported also by the RaspberryPi, I don't see the problem.
Anyway, I installed the last version of TensorFlow (2.0) on my Intel N5000
by compiling it from source. It took 14 hours because I had to run the
compilation on a single core, since it needs a lot of RAM and I have only
4Gb invited to the party.
I took inspiration from this guide here
<https://tech.amikelive.com/node-882/how-to-build-and-install-the-latest-tensorflow-without-cuda-gpu-and-with-optimized-cpu-performance-on-ubuntu/>
but the experience was far from smooth, there were constantly dependencies
missing that I need to install and re-launch the compilation. And some
other stuff too that I solved when the compilation crashed.
Have fun and thanks for the hassle. Providing through pip already binary
compiled for non-AVX was clearly too much to add in your continuous
integration worklow
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#17411?email_source=notifications&email_token=ACYHH362LGI5EZY3G74CMVLQXQZKDA5CNFSM4ETNEGXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGSCPQ#issuecomment-562897214>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACYHH3YLNAER6NGNLKYNMPDQXQZKDANCNFSM4ETNEGXA>
.
--
Clive DaSilva CPA,CMA
Home: 416-421-2480|Mobile: 416-560-8820
Email: clive.dasilva@gmail.com
LinkedIN: http://ca.linkedin.com/pub/clive-dasilva/3/197/b89
|
I had the same problem when running CI pipelines on a Gitlab server. The (emulated) CPU of the runners did not provide AVX instructions. Installing Tensorflow with Conda instead of using PyPI's wheels fixed the problem. 👍 |
I have the same issue with Tensorflow 2.1.0. What to do?
|
I managed to fix my problem by building from source using bazel. It created a whl file. Then I did pip install whl file path |
Yes, if your CPU does not support AVX (the likely cause for Furthermore, this guarantees that the pip is built with the highest optimization level available to your platform, so you might actually see some speedup compared to using a pip built on a different platform. Focus on might. |
System information
1.6.0-cp27-cp27mu-manylinux1_x86_64 (can only guess since
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
gives me an error already)import tensorflow
I created a fresh virtual environment:
virtualenv -p python2 test_venv/
And installed tensorflow:
pip install --upgrade --no-cache-dir tensorflow
import tensorflow
gives meIllegal instruction (core dumped)
Please help me understand what's going on and how I can fix it. Thank you.
CPU information:
EDIT
Stacktrace obtained with gdb:
EDIT 2
Bazel version: N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A
After downgrading to an older version of tensorflow the error goes away. I've been advised that my CPU (see information above) might not work with some improvements in the new API. If this is the case, I suppose there's no solution for my problem. Therefore, I will close this thread. Feel free to correct me though. Thank you for your support
The text was updated successfully, but these errors were encountered: