Skip to content

Illegal instruction (core dumped) in a CPU with AVX support #45744

New issue
New issue
Closed
Closed
Illegal instruction (core dumped) in a CPU with AVX support#45744
Assignees
ymodak
Labels
TF 2.4for issues related to TF 2.4stat:awaiting tensorflowerStatus - Awaiting response from tensorflowersubtype: ubuntu/linuxUbuntu/Linux Build/Installation Issuestype:build/installBuild and install issues
@cserpell

Description

@cserpell
cserpell
opened on Dec 16, 2020 · edited by cserpell

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution: Linux Ubuntu 20.10
  • TensorFlow installed from (source or binary): pip, tensorflow-2.4.0-cp38-cp38-manylinux2010_x86_64.whl
  • TensorFlow version (use command below): tensorflow-2.4.0-cp38-cp38-manylinux2010_x86_64.whl
  • Python version: 3.8.6
  • CUDA/cuDNN version: N/A
  • GPU model and memory: N/A

Describe the current behavior

I was using tensorflow 2 without any issue until a few days ago. I installed it using pip in a new virtualenv environment and now it throws illegal instruction error. My CPU is an i5-3230M, which, according to /proc/cpuinfo supports AVX, so it seems not related to #19584 .

My CPU (according to /proc/cpuinfo):
model name : Intel(R) Core(TM) i5-3230M CPU @ 2.60GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d

System:

$ uname -a
Linux e431 5.8.0-33-generic #36-Ubuntu SMP Wed Dec 9 09:14:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Describe the expected behavior
The import should work and not give the Illegal instruction (core dumped) error.

Standalone code to reproduce the issue

$ python3 --version
Python 3.8.6
$ python3 -m venv test
$ . test/bin/activate
$ pip install --upgrade pip
Collecting pip
  Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed pip-20.3.3
$ pip install tensorflow
Collecting tensorflow
  Using cached tensorflow-2.4.0-cp38-cp38-manylinux2010_x86_64.whl (394.8 MB)
Collecting gast==0.3.3
  Using cached gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Collecting absl-py~=0.10
  Using cached absl_py-0.11.0-py3-none-any.whl (127 kB)
Collecting astunparse~=1.6.3
  Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting flatbuffers~=1.12.0
  Using cached flatbuffers-1.12-py2.py3-none-any.whl (15 kB)
Collecting google-pasta~=0.2
  Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting grpcio~=1.32.0
  Using cached grpcio-1.32.0-cp38-cp38-manylinux2014_x86_64.whl (3.8 MB)
Collecting h5py~=2.10.0
  Using cached h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB)
Collecting keras-preprocessing~=1.1.2
  Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Collecting numpy~=1.19.2
  Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
Collecting opt-einsum~=3.3.0
  Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting protobuf>=3.9.2
  Using cached protobuf-3.14.0-cp38-cp38-manylinux1_x86_64.whl (1.0 MB)
Collecting six~=1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting tensorboard~=2.4
  Using cached tensorboard-2.4.0-py3-none-any.whl (10.6 MB)
Requirement already satisfied: setuptools>=41.0.0 in ./test/lib/python3.8/site-packages (from tensorboard~=2.4->tensorflow) (44.0.0)
Collecting google-auth<2,>=1.6.3
  Using cached google_auth-1.24.0-py2.py3-none-any.whl (114 kB)
Collecting cachetools<5.0,>=2.0.0
  Using cached cachetools-4.2.0-py3-none-any.whl (12 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Using cached google_auth_oauthlib-0.4.2-py2.py3-none-any.whl (18 kB)
Collecting markdown>=2.6.8
  Using cached Markdown-3.3.3-py3-none-any.whl (96 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting requests<3,>=2.21.0
  Using cached requests-2.25.0-py2.py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting requests-oauthlib>=0.7.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting tensorboard-plugin-wit>=1.6.0
  Using cached tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779 kB)
Collecting tensorflow-estimator<2.5.0,>=2.4.0rc0
  Using cached tensorflow_estimator-2.4.0-py2.py3-none-any.whl (462 kB)
Collecting termcolor~=1.1.0
  Using cached termcolor-1.1.0.tar.gz (3.9 kB)
Collecting typing-extensions~=3.7.4
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Collecting werkzeug>=0.11.15
  Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting wheel~=0.35
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting wrapt~=1.12.1
  Using cached wrapt-1.12.1.tar.gz (27 kB)
Using legacy 'setup.py install' for termcolor, since package 'wheel' is not installed.
Using legacy 'setup.py install' for wrapt, since package 'wheel' is not installed.
Installing collected packages: urllib3, pyasn1, idna, chardet, certifi, six, rsa, requests, pyasn1-modules, oauthlib, cachetools, requests-oauthlib, google-auth, wheel, werkzeug, tensorboard-plugin-wit, protobuf, numpy, markdown, grpcio, google-auth-oauthlib, absl-py, wrapt, typing-extensions, termcolor, tensorflow-estimator, tensorboard, opt-einsum, keras-preprocessing, h5py, google-pasta, gast, flatbuffers, astunparse, tensorflow
    Running setup.py install for wrapt ... done
    Running setup.py install for termcolor ... done
Successfully installed absl-py-0.11.0 astunparse-1.6.3 cachetools-4.2.0 certifi-2020.12.5 chardet-3.0.4 flatbuffers-1.12 gast-0.3.3 google-auth-1.24.0 google-auth-oauthlib-0.4.2 google-pasta-0.2.0 grpcio-1.32.0 h5py-2.10.0 idna-2.10 keras-preprocessing-1.1.2 markdown-3.3.3 numpy-1.19.4 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.25.0 requests-oauthlib-1.3.0 rsa-4.6 six-1.15.0 tensorboard-2.4.0 tensorboard-plugin-wit-1.7.0 tensorflow-2.4.0 tensorflow-estimator-2.4.0 termcolor-1.1.0 typing-extensions-3.7.4.3 urllib3-1.26.2 werkzeug-1.0.1 wheel-0.36.2 wrapt-1.12.1
$ python --version
Python 3.8.6
$ python
Python 3.8.6 (default, Sep 25 2020, 09:36:53) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
Illegal instruction (core dumped)

Other info / logs

Activity

cserpell
added
type:bugBug
on Dec 16, 2020
google-ml-butler
assigned
Saduf2019
on Dec 16, 2020
cserpell

cserpell commented on Dec 16, 2020

@cserpell
cserpell
on Dec 16, 2020
Author

tf_env.txt

cserpell

cserpell commented on Dec 16, 2020

@cserpell
cserpell
on Dec 16, 2020
Author

Here I attach the output of running an interactive python shell with python -W all -d -v -X dev just to import tensorflow.
py_output.txt

amahendrakar
assigned
amahendrakar
and unassigned
Saduf2019
on Dec 17, 2020
amahendrakar
added
subtype: ubuntu/linuxUbuntu/Linux Build/Installation Issues
TF 2.4for issues related to TF 2.4
type:build/installBuild and install issues
and removed
type:bugBug
on Dec 17, 2020
amahendrakar

amahendrakar commented on Dec 17, 2020

@amahendrakar
amahendrakar
on Dec 17, 2020
Contributor

@cserpell,
Could you please create a new virtual environment and check if you are facing the same issue in that as well? Thanks!

amahendrakar
added
stat:awaiting responseStatus - Awaiting response from author
on Dec 17, 2020
cserpell

cserpell commented on Dec 17, 2020

@cserpell
cserpell
on Dec 17, 2020
Author

Sure, I already did it in a new virtualenv, but here it goes again in a brand new one.

new_output.txt

necromuralist

necromuralist commented on Dec 18, 2020

@necromuralist
necromuralist
on Dec 18, 2020

I am having the same problem (also on Ubuntu 20.10, also installed through pip). Whenever I import tensorflow and python crashes and I look in journalctl I see something like this:

Dec 17 19:30:06 erebus kernel: traps: python[193520] trap invalid opcode ip:7fd16f267c48 sp:7ffd11713f50 error:0 in libtensorflow_framework.so.2[7fd16e4e0000+1c55000]

wingman-jr-addon

wingman-jr-addon commented on Dec 18, 2020

@wingman-jr-addon
wingman-jr-addon
on Dec 18, 2020

I also see the same issue on an older server after upgrading to 2.4.0. Crashes consistently at same place while importing tensorflow according to dmesg.

 8996.519179] traps: python3[2587] trap invalid opcode ip:7f19106a8c48 sp:7ffc8d486220 error:0 in libtensorflow_framework.so.2[7f190f921000+1c55000]

Here's one of the cores from /proc/cpuinfo, should have AVX:

processor       : 0                                                                                                                                                                             
vendor_id       : GenuineIntel                                                                                                                                                                  
cpu family      : 6                                                                                                                                                                             
model           : 45                                                                                                                                                                            
model name      : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz                                                                                                                                      
stepping        : 7                                                                                                                                                                             
microcode       : 0x71a                                                                                                                                                                         
cpu MHz         : 1199.954                                                                                                                                                                      
cache size      : 20480 KB                                                                                                                                                                      
physical id     : 0                                                                                                                                                                             
siblings        : 16                                                                                                                                                                            
core id         : 0                                                                                                                                                                             
cpu cores       : 8                                                                                                                                                                             
apicid          : 0                                                                                                                                                                             
initial apicid  : 0                                                                                                                                                                             
fpu             : yes                                                                                                                                                                           
fpu_exception   : yes                                                                                                                                                                           
cpuid level     : 13                                                                                                                                                                            
wp              : yes                                                                                                                                                                           
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon
 pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer 
aes xsave avx lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d                                                    
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit                                                                                            
bogomips        : 5199.91                                                                                                                                                                       
clflush size    : 64                                                                                                                                                                            
cache_alignment : 64                                                                                                                                                                            
address sizes   : 46 bits physical, 48 bits virtual                                                                                                                                             
power management:

This is running Ubuntu 18.04.4 LTS.

64 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

TF 2.4for issues related to TF 2.4stat:awaiting tensorflowerStatus - Awaiting response from tensorflowersubtype: ubuntu/linuxUbuntu/Linux Build/Installation Issuestype:build/installBuild and install issues

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @mihaimaruseac@foxik@cserpell@SaschaHeyer@byronyi

      Issue actions
        Illegal instruction (core dumped) in a CPU with AVX support · Issue #45744 · tensorflow/tensorflow