RuntimeError: apex.optimizers.FusedAdam requires cuda extensions #1193

life97 · 2021-10-16T08:53:21Z

My environment is configured as Windows server2016, torch 1.8.1, torchvision 0.9.1, cuda10.2, apex is successfully installed, but when running the project code (NVlabs/imagenaire), an error is reported:

Initialize net_G and net_D weights using type: orthogonal gain: 1
net_G parameter count: 30,258,966
net_D parameter count: 32,322,498
Traceback (most recent call last):
File "H:\19xyy\project\imaginaire-master\train.py", line 100, in
main()
File "H:\19xyy\project\imaginaire-master\train.py", line 60, in main
get_model_optimizer_and_scheduler(cfg, seed=args.seed)
File "H:\19xyy\project\imaginaire-master\imaginaire\utils\trainer.py", line 115, in get_model_optimizer_and_scheduler
opt_G = get_optimizer(cfg.gen_opt, net_G)
File "H:\19xyy\project\imaginaire-master\imaginaire\utils\trainer.py", line 257, in get_optimizer
return get_optimizer_for_params(cfg_opt, params)
File "H:\19xyy\project\imaginaire-master\imaginaire\utils\trainer.py", line 274, in get_optimizer_for_params
opt = FusedAdam(params,
File "G:\Anaconda3\envs\xyy_imagenaire\lib\site-packages\apex\optimizers\fused_adam.py", line 80, in init
raise RuntimeError('apex.optimizers.FusedAdam requires cuda extensions')
RuntimeError: apex.optimizers.FusedAdam requires cuda extensions

The versions of nvcc -V and print(torch.version.cuda) are the same. I don’t know why this error is reported. Are there any good suggestions to make the code run correctly? Looking forward to your reply, thank you very much！

Dawn-bin · 2021-10-24T08:08:33Z

hi, i meet the same problem, has it happen before? it's my first time use this optimizer. my env this ubuntu 18.0.4 torch 1.8.0 cuda 11.1

life97 · 2021-10-26T01:04:12Z

hi, i meet the same problem, has it happen before? it's my first time use this optimizer. my env this ubuntu 18.0.4 torch 1.8.0 cuda 11.1

Sorry, I haven't solved this problem yet. I use imagenaire mainly because I want to run the MUNIT model, and I can run through the official code before, so I didn't continue to solve this problem.

suxin1412 · 2021-12-28T15:22:40Z

hi, i meet the same problem, has it happen before? it's my first time use this optimizer. my env this ubuntu 18.0.4 torch 1.8.0 cuda 11.1

hi, i meet the same problem. Have you solved this problem?

Dawn-bin · 2021-12-28T15:32:34Z

hi, i meet the same problem, has it happen before? it's my first time use this optimizer. my env this ubuntu 18.0.4 torch 1.8.0 cuda 11.1

hi, i meet the same problem. Have you solved this problem?

yeah, it sames like that apex is installed on only cpu, you can solve this trying to reinstall apex CUDA contained follow the readme. hope it works.

kongyuzhuo · 2022-10-26T05:58:08Z

hi, i have meet the same problem, have u solved the problem?

Chiang97912 · 2022-11-28T07:43:12Z

This is because of apex cannot import amp_C，you can check the file "G:\Anaconda3\envs\xyy_imagenaire\lib\site-packages\apex\optimizers\fused_adam.py", also you can use your python shell to verify this:

import torch
import amp_C  # must import torch before import amp_C

Maybe you can get error like: libstdc++.so.6: version 'GLIBCXX_3.4.20' not found', If so, you can try the following commands:

conda install libgcc
export LD_LIBRARY_PATH=/path/to/anaconda/envs/myenv/lib:$LD_LIBRARY_PATH
cd /path/to/anaconda/envs/myenv/lib
ln -s libstdc++.so.6.0.30 libstdc++.so.6

And you can add export LD_LIBRARY_PATH=/path/to/anaconda/envs/myenv/lib:$LD_LIBRARY_PATH to ~/.bashrc file.

huang-zeyu · 2023-02-01T09:17:30Z

Some error. Not solved yet.
Ubuntu-20.04(WSL2) python3.9 cuda116 cudnn850 torch1.12.1
following the readme installation.
Btw, import torch then import amp_C also failed.
Hope someone can fix it or provide a solution.

GuangmingChan · 2023-03-12T13:04:46Z

I have also experienced this error:
I had successfully installed Apex in a certain environment before, but when I switched to a different environment and tried to reinstall Apex, it appeared to install successfully, but when running the code, it always gave the error "RuntimeError: apex.optimizers.FusedAdam requires cuda extensions". Later, I deleted the Apex folder downloaded from GitHub, downloaded it again, and reinstalled Apex. In the end, it was successfully executed.

ShoufaChen · 2023-06-25T13:20:26Z

I solved this problem by building with

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./

rather than

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./

My pip version is 22.3.1.

nshah-sfoundation · 2023-08-25T16:25:40Z

I have installed the apex with the below command. but still getting the error
RuntimeError: apex.optimizers.FusedAdam requires cuda extensions
Linux 5.15.120.2
cuda 11.8
pip 23
torch cu118

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./

filipesmg · 2023-09-05T12:09:34Z

I get the same issue using pip 22.0.4, and the command pointed on the README:

# otherwise
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./

I noticed that even with the command above, --cpp_ext and --cuda_ext are not in sys.argv that reaches setup.py (and that is what seems to be checked):

['(...)/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py', 'dist_info', '--egg-base', '/tmp/pip-modern-metadata-pvaz06q9']

Tolga-Karahan · 2023-10-12T07:19:14Z

Anyone who solved this issue?

frankielp · 2023-10-16T11:49:01Z

this solution works in my case
#1204 (comment)

Tolga-Karahan · 2023-10-17T11:27:33Z

@frankielp thanks. I tried but got another error: ninja: error: '/app/csrc/amp_C_frontend.cpp', needed by '/app/build/temp.linux-x86_64-cpython-310/csrc/amp_C_frontend.o', missing and no known rule to make it. I'll create an issue for that.

WuHongyuQXWX · 2023-10-23T02:38:34Z

This is because of apex cannot import amp_C，you can check the file "G:\Anaconda3\envs\xyy_imagenaire\lib\site-packages\apex\optimizers\fused_adam.py", also you can use your python shell to verify this:
import torch
import amp_C  # must import torch before import amp_C
Maybe you can get error like: libstdc++.so.6: version 'GLIBCXX_3.4.20' not found', If so, you can try the following commands:
conda install libgcc
export LD_LIBRARY_PATH=/path/to/anaconda/envs/myenv/lib:$LD_LIBRARY_PATH
cd /path/to/anaconda/envs/myenv/lib
ln -s libstdc++.so.6.0.30 libstdc++.so.6
And you can add export LD_LIBRARY_PATH=/path/to/anaconda/envs/myenv/lib:$LD_LIBRARY_PATH to ~/.bashrc file.

Finally solve my problem, you are so fucking bralliant Bro!
老哥真nb

Flame-circle · 2024-03-18T07:29:14Z

I solved this problem by building with

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./

rather than

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./

My pip version is 22.3.1.

THANK YOU VERY MUCH,IT IS HELPFUL

mkerin · 2024-05-20T11:52:35Z

I solved this problem by building with

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./

rather than

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./

My pip version is 22.3.1.

This solution didn't work for me (on pip 24.0), and instead I had to use #1204 (comment)

JINNMnm · 2024-11-03T13:54:16Z

This is because of apex cannot import amp_C，you can check the file "G:\Anaconda3\envs\xyy_imagenaire\lib\site-packages\apex\optimizers\fused_adam.py", also you can use your python shell to verify this:
import torch
import amp_C  # must import torch before import amp_C
Maybe you can get error like: libstdc++.so.6: version 'GLIBCXX_3.4.20' not found', If so, you can try the following commands:
conda install libgcc
export LD_LIBRARY_PATH=/path/to/anaconda/envs/myenv/lib:$LD_LIBRARY_PATH
cd /path/to/anaconda/envs/myenv/lib
ln -s libstdc++.so.6.0.30 libstdc++.so.6
And you can add export LD_LIBRARY_PATH=/path/to/anaconda/envs/myenv/lib:$LD_LIBRARY_PATH to ~/.bashrc file.

this works for me, thanks a lot.

dujl · 2025-04-22T12:29:34Z

I solved this problem by building with
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
rather than
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
My pip version is 22.3.1.
This solution didn't work for me (on pip 24.0), and instead I had to use #1204 (comment)

The build failed due to the error: "RuntimeError: Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries."
To resolve this, I modified the setup.py file by removing the check_cuda_version function, which allowed the build to succeed.

Tolga-Karahan mentioned this issue Oct 17, 2023

ninja: error: '/app/csrc/amp_C_frontend.cpp', needed by '/app/build/temp.linux-x86_64-cpython-310/csrc/amp_C_frontend.o', missing and no known rule to make it #1741

Closed

RuntimeError: apex.optimizers.FusedAdam requires cuda extensions #1193

RuntimeError: apex.optimizers.FusedAdam requires cuda extensions #1193

Comments

life97 commented Oct 16, 2021

Dawn-bin commented Oct 24, 2021

Uh oh!

life97 commented Oct 26, 2021

Uh oh!

suxin1412 commented Dec 28, 2021

Uh oh!

Dawn-bin commented Dec 28, 2021

Uh oh!

kongyuzhuo commented Oct 26, 2022

Uh oh!

Chiang97912 commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huang-zeyu commented Feb 1, 2023

Uh oh!

GuangmingChan commented Mar 12, 2023

Uh oh!

ShoufaChen commented Jun 25, 2023

Uh oh!

nshah-sfoundation commented Aug 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filipesmg commented Sep 5, 2023

Uh oh!

Tolga-Karahan commented Oct 12, 2023

Uh oh!

frankielp commented Oct 16, 2023

Uh oh!

Tolga-Karahan commented Oct 17, 2023

Uh oh!

WuHongyuQXWX commented Oct 23, 2023

Uh oh!

Flame-circle commented Mar 18, 2024

Uh oh!

mkerin commented May 20, 2024

Uh oh!

JINNMnm commented Nov 3, 2024

Uh oh!

dujl commented Apr 22, 2025

Uh oh!

Chiang97912 commented Nov 28, 2022 •

edited

Loading

nshah-sfoundation commented Aug 25, 2023 •

edited

Loading