In the meanwhile I have tried with Cudnn versions : 7.1,7.0.5,7.3,7.4 , gcc6,still no luck, however I dont get any of these issues when i installed it from conda using conda install tensorflow-gpu.
However I want to build from source hence I would prefer if this issue is resolved

I had the same issue with TensorFlow 1.12 on an almost identical system as yours. Solution is to downgrade TensorFlow to 1.8.0 using:
pip install --upgrade tensorflowgpu==1.8.0

https://devtalk.nvidia.com/default/topic/1043867/failed-to-get-convolution-algorithm-this-is-probably-because-cudnn-failed-to-initialize

I also have the same error with TF 1.12,1.11, and I have Cuda 9.0, and cuDnn 7.3.1, 7.4.2. Sometimes it works but sometimes not, what is causing this error to happen. Did anyone solve this error?

@gunan Can you please take a look or suggest someone? Apparently there is an incompatibility between the cuda 9.0 and cuDNN version above 7.0. Thanks!

I cannot help much on this one. Maybe TF GPU team can help?

This error may be related to installation TF with conda.

The possible solution is like this:
In the command line issue this command:
conda list cudnn
It will print:
Name Version Build Channel

If the result is not empty as the above, so it means you used conda installed TF, when using conda for installing TF, then it will install all the dependencies even CUDA and cuDNN, but the cuDNN version is very low for TF, so it will bring compatibility problem. So you can uninstall the cuDNN and the CUDA which was installed by conda, and then run TF, then it will work.

@deepakrai9185720 Is this still an issue for you? Can you please try @Bahramudin 's suggestion and confirm if it solves the problem for you?

maybe same problem..

I think it will be a version problem. say if @ifssk1991 solution works

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

I'm also facing this issue also. Is any workaround except downgrading ?

Same issue here!

Hi, I had the same error.
Download the cudnn package for my system and replace it with the old ones.
This solved my problem.

I had the same issue with TensorFlow 1.12 on an almost identical system as yours. Solution is to downgrade TensorFlow to 1.8.0 using:
pip uninstall tensorflow-gpu
pip install --upgrade tensorflow-gpu==1.8.0

https://devtalk.nvidia.com/default/topic/1043867/failed-to-get-convolution-algorithm-this-is-probably-because-cudnn-failed-to-initialize

It is just a problem with cuDNN version incompatibility. I also downgrade to 1.8, although solving the problem, but no need when there is a much higher version which is definitely better than the old version. So I found that I was using conda installing TF, which conda was also installed everything even Cuda and cudnn, so the python was not detecting my installed Cuda and cudnn, it was using which was installed by conda which was very old version of cudnn, so I deleted the conda installed Cuda and cudnn, and then installed TF with pip, and it was OK.

i also has same problem
try it delete the old cuDNN SDK (i remember it's no for 9.0)
download the cudnn-9.0-windows10-x64-v7.4.1.5
new cudnn-9.0-windows10-x64-v7.4.2.24.zip also work well
for 9.0 9.0 9.0
it's very important
then it work well

system win 10
tensorflow 1.12
CUDA 9.0
cuDNN SDK 7.4.1.5
GPU GTX1060

Had same problem with cuda 9.0 and cuDNN 7.0.5.15-1 on Ubuntu 16.04 with Tensorflow 1.12. Updating to cuDNN 7.4.2.24 fixed this for me!

Did you use NCCL, if so which version?

Same issue here. I have an RTX 2070, cuda 10, cudnn 7.4.1 and tensorflow 2.0 running on ubuntu 18.04. Downgraded cudnn to 7.3.0 but still same error. I see it helped for some people downgrading tensorflow but I guess that's not an option for me. Any help is much appreciated.

OK, I was able to execute my CNN. I'm using tensorflow tf-nightly-gpu-2.0-preview, and running on a ipython notebook. I had to add this to my notebook:

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

Here are some more details

Also, this issue is associated with [24496] (#24496)

I'm having the same issue with Cuda 9.0, Cudnn 7.4.2 and 7.0.5.
Also, I installed tf using pip, not conda. I downloaded cudnn on my own from the Nvidia website and linked to it.
In my case, downgrading to tf 1.8 did not help. Is there any other fix for this?

Did you try setting up allow_growth = True? That resolved the problem for me.

Did you try setting up allow_growth = True? That resolved the problem for me.

Yes, it helps!
Thanks.
@aishwaryap
You can try setting up allow_growth:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

OK, I was able to execute my CNN. I'm using tensorflow tf-nightly-gpu-2.0-preview, and running on a ipython notebook. I had to add this to my notebook:

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

Here are some more details

Also, this issue is associated with [24496] (#24496)

This solution worked for me. Just to add on - if you set allow_growth = True during training, you have to configure the gpu in the same way when restoring the model otherwise you will have issues.

OK, I was able to execute my CNN. I'm using tensorflow tf-nightly-gpu-2.0-preview, and running on a ipython notebook. I had to add this to my notebook:
from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)
Here are some more details
Also, this issue is associated with [24496] (#24496)

This solution worked for me. Just to add on - if you set allow_growth = True during training, you have to configure the gpu in the same way when restoring the model otherwise you will have issues.

Yea it works! Actually what is the concept behind by enabling allow_growth = True?

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

I tried this solution but doesn't work, my system specifications are:
TF version: 2.2
OS: Windows Server 2019
cuda version: 10.1
cuDNN: 7.6.4
GPU: GTeslaV100
can you help please?

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

I tried this solution but doesn't work, my system specifications are:
TF version: 2.2
OS: Windows Server 2019
cuda version: 10.1
cuDNN: 7.6.4
GPU: GTeslaV100
can you help please?

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

Try with this.

See, I made a solution.
First install graphics respective of your graphics drivers
Install Anaconda

conda update conda
conda update anaconda

then 

conda create -n py36 python=3.6

# first thing you do
conda install tensorflow-gpu=1.15
# this install cudatoolkit=10.0 cudnn=7.6.5 and of course tensorflow-gpu=1.15

Now if you want to want to run tensorflow with eager execution then

import tensorflow as tf
tf.enable_eager_execution()

This makes tensorflow>=2.0.0 codes to run and even you can make them

else if you want to stick with older version tensorflow<2.0.0 then run as usual with tf.Session(), tf.placeholders, tf.Variables, ... and so on.

Confirming that I hit this error while training a model on an RTX 2060, and setting TF_FORCE_ALLOW_GROWTH to True resolved the error.

Had a similar issue on a machine with 2 A100s. There was some device ambiguity and so looping through the devices and setting the memory growth manually worked in a tensorflow 2.x environment.

gpu_devices = tf.config.experimental.list_physical_devices('GPU')
for device in gpu_devices: tf.config.experimental.set_memory_growth(device, True)

Taken from a different issue

#25446 (comment)

What worked for my Win10 using Anaconda (Python 3.5.x and NVIDIA GTX1650 was:
-Downgrade to CUDA 9.0 (with Matching CUDNN 7.x)
-Downgrade to Tensorflow 1.8.0 (check by 'pip show tensorflow')
-Downgrade to Tensorflow-GPU 1.8.0 (check by 'pip show tensorflow-gpu)
-Ensure overwrite the CUDNN files (For some reason I had to overwrite the existing cudnn.lib file -C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\lib\x64 with the CUDNN.zip download. The version downgrade left the more recent version, which had to be replaced to remove the "failed convolution" error to go away

--Hope this helps--

It probably because of the memory growth under tensorflow framework, so try to add
import tensorflow as tf
from keras import backend as K
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4## max GPU occupation
K.set_session(tf.Session(config=config))
K.get_session().run(tf.global_variables_initializer())
before the whole code... It works for me.

try:

import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)

except RuntimeError as e:
    print(e)

Hy I hope that you all are doing good. I need to train my mrcnn model on gtx 3070. Model loads onto the gpu but stuck while starting training no error appears but it stuck. When I list tensorflow device it show GPU exists but training not starts.

Versions I am using:

1. Tensorflow 2.4
2. cudnn 8
3. cuda 11.0
4. nvidia-drivers 460

I will really be thankful to you for helping me out. Thank you

If the result is not empty as the above, so it means you used conda installed TF, when using conda for installing TF, then it will install all the dependencies even CUDA and cuDNN, but the cuDNN version is very low for TF, so it will bring compatibility problem. So you can uninstall the cuDNN and the CUDA which was installed by conda, and then run TF, then it will work.

So one should reinstall using the nvidia installer?

This solve my problem here. Try to match the verison.

@sauravsolanki

This solve my problem here. Try to match the verison.

So you managed to make it work with these following versions ?
tensorflow-2.4.0 | python 3.6-3.8 | GCC 7.3.1 | Bazel 3.1.0 | cuDNN 8.0 | CUDA 11.0

@q-55555 Yes.

This solve my problem here. Try to match the verison.

especially cuDNN version and tensorflow version, I downgraded tf version to match cuDNN version, then it's ok.

I had the same problem and I solved it with this code
import tensorflow as tf
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.compat.v1.InteractiveSession(config=config)

i also has same problem
try it delete the old cuDNN SDK (i remember it's no for 9.0)
download the cudnn-9.0-windows10-x64-v7.4.1.5
new cudnn-9.0-windows10-x64-v7.4.2.24.zip also work well
for 9.0 9.0 9.0
it's very important
then it work well

system win 10
tensorflow 1.12
CUDA 9.0
cuDNN SDK 7.4.1.5
GPU GTX1060

is works !

I got the same issue.
pip install --upgrade tensorflow-gpu==1.10.0 solves it.

Had a similar issue on a machine with 2 A100s. There was some device ambiguity and so looping through the devices and setting the memory growth manually worked in a tensorflow 2.x environment.

gpu_devices = tf.config.experimental.list_physical_devices('GPU') for device in gpu_devices: tf.config.experimental.set_memory_growth(device, True)

Taken from a different issue

#25446 (comment)

Thanks, this also worked for me in a RTX 2070 Super, TF 2.2, CUDA 10.1 on Ubuntu 18.04

In the meanwhile I have tried with Cudnn versions : 7.1,7.0.5,7.3,7.4 , gcc6,still no luck, however I dont get any of these issues when i installed it from conda using conda install tensorflow-gpu. However I want to build from source hence I would prefer if this issue is resolved

Thanx!!!

Add the following code

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)

I had the same error with cudnn=7.6.5 tensorflow-gpu=2.3.0.
Then I downgrade cudnn=7.6.0, then no error reported!

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

I tried this solution but doesn't work, my system specifications are:
TF version: 2.2
OS: Windows Server 2019
cuda version: 10.1
cuDNN: 7.6.4
GPU: GTeslaV100
can you help please?

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

Try with this.

many years later, this worked for me

Im using RTX 4090 on Ubuntu 20.04
cuda_version: 11.2
cudnn_version: