Skip to content

[TensorRT] ERROR: Network must have at least one output #183

New issue
New issue
Closed
Closed
[TensorRT] ERROR: Network must have at least one output#183
Labels
Module:ONNXIssues relating to ONNX usage and import
@marooncn

Description

@marooncn
marooncn
opened on Oct 30, 2019 · edited by rmccorm4

Hi, I try to convert my .onnx model to .trt model using tensorrt, my script is:

import tensorrt as trt

def ONNX_build_engine(onnx_file_path, engine_file_path):
    G_LOGGER = trt.Logger(trt.Logger.WARNING)
    with trt.Builder(G_LOGGER) as builder, builder.create_network() as network, trt.OnnxParser(network, G_LOGGER) as parser:
        builder.max_batch_size = 16
        builder.max_workspace_size = 1 << 20

        print('Loading ONNX file from path {}...'.format(onnx_file_path))
        with open(onnx_file_path, 'rb') as model:
            print('Beginning ONNX file parsing')
            parser.parse(model.read())
        print('Completed parsing of ONNX file')

        print('Building an engine from file {}; this may take a while...'.format(onnx_file_path))
        engine = builder.build_cuda_engine(network)
        print("Completed creating Engine")

        with open(engine_file_path, "wb") as f:
            f.write(engine.serialize())
        return engine

ONNX_build_engine('./BiSeNet.onnx', './BiSeNet.trt')

I have tested in my two desktops, one configue:

cuda 10.0
pytorch1.3
torch 5.0.6.6

the other configure:

cuda 10.1
pytorch1.3
torch6.0.1.5

Both have the same error. there is an error:

Loading ONNX file from path ./BiSeNet_simplifier.onnx...
Beginning ONNX file parsing
Completed parsing of ONNX file
Building an engine from file ./BiSeNet_simplifier.onnx; this may take a while...
[TensorRT] ERROR: Network must have at least one output
Completed creating Engine
Traceback (most recent call last):
File "onnx2trt.py", line 31, in
ONNX_build_engine('./BiSeNet_simplifier.onnx', './BiSeNet.trt')
File "onnx2trt.py", line 28, in ONNX_build_engine
f.write(engine.serialize())
AttributeError: 'NoneType' object has no attribute 'serialize'

The .onnx file is converted by torch.onnx.export fcn, it seems right.

import torch
from model.build_BiSeNet import BiSeNet

model = BiSeNet(13, 'resnet18').eval()  # .cuda().half().eval()
model.load_state_dict(torch.load('./checkpoints_18_sgd/best_dice_loss_13_class.pth'))
x = torch.ones((1, 3, 720, 960))  # .cuda().half()
output_names = ["output"]
onnx_model = torch.onnx._export(model, x, "BiSeNet.onnx", verbose=False, output_names=output_names, export_params=True) #,  keep_initializers_as_inputs=True)

Activity

rmccorm4

rmccorm4 commented on Oct 30, 2019

@rmccorm4
rmccorm4
on Oct 30, 2019 · edited by rmccorm4
Collaborator

Hi @marooncn,

There's probably a way to fix this in the PyTorch code so that the ONNX parser recognizes the output, but I'm not sure how to do that off the top of my head - maybe someone else can chime in on this.

However, I think you can fix this on the TensorRT side with something like this:

...
last_layer = network.get_layer(network.num_layers - 1)
# Check if last layer recognizes it's output
if not last_layer.get_output(0):
    # If not, then mark the output using TensorRT API
    network.mark_output(last_layer.get_output(0))

You can see similar code being used in this issue: triton-inference-server/server#76

Let me know if this or something similar fixes your issue.

marooncn

marooncn commented on Oct 30, 2019

@marooncn
marooncn
on Oct 30, 2019
Author

Hi @rmccorm4 ,
It fails, the error is

python3: ../builder/Network.cpp:863: virtual nvinfer1::ILayer* nvinfer1::Network::getLayer(int) const: Assertion `layerIndex >= 0' failed.
Aborted

marooncn

marooncn commented on Oct 30, 2019

@marooncn
marooncn
on Oct 30, 2019
Author

I also test on pytorch1.1.0, the error is the same as the original however.

AttributeError: 'NoneType' object has no attribute 'serialize'

It's so weird.

rmccorm4

rmccorm4 commented on Oct 30, 2019

@rmccorm4
rmccorm4
on Oct 30, 2019 · edited by rmccorm4
Collaborator

Assertion `layerIndex >= 0' failed.

This means you called network.get_layer(num) where num < 0. Which probably means that network.num_layers returned 0, so network.get_layer(network.num_layers - 1) == network.get_layer(-1).

@marooncn Did you do these steps after parsing?

# parser.parse() populates the network object with information from the model file
with open(onnx_file_path, 'rb') as model:
    parser.parse(model.read())

last_layer = network.get_layer(network.num_layers - 1)
# Check if last layer recognizes it's output
if not last_layer.get_output(0):
    # If not, then mark the output using TensorRT API
    network.mark_output(last_layer.get_output(0))
marooncn

marooncn commented on Oct 30, 2019

@marooncn
marooncn
on Oct 30, 2019
Author

Yes, I added it after parsing. Maybe tensorrt is not installed successfully on Pytorch1.3 desktop. But for pytorch1.1, the error is still as the original.

rmccorm4

rmccorm4 commented on Oct 30, 2019

@rmccorm4
rmccorm4
on Oct 30, 2019
Collaborator

Can you post the full error for pytorch 1.1?

The NoneType error on calling engine.deserialize() is usually because something went wrong before/during creating the engine, it isnt specific to any particular issue.

marooncn

marooncn commented on Oct 30, 2019

@marooncn
marooncn
on Oct 30, 2019
Author

Yes, all of the output are

Loading ONNX file from path ./BiSeNet/BiSeNet.onnx...
Beginning ONNX file parsing
Completed parsing of ONNX file
Building an engine from file ./BiSeNet/BiSeNet.onnx; this may take a while...
[TensorRT] ERROR: Network must have at least one output
Completed creating Engine
Traceback (most recent call last):
File "test.py", line 35, in
ONNX_build_engine('./BiSeNet/BiSeNet.onnx', './BiSeNet.trt')
File "test.py", line 32, in ONNX_build_engine
f.write(engine.serialize())
AttributeError: 'NoneType' object has no attribute 'serialize'

marooncn

marooncn commented on Oct 30, 2019

@marooncn
marooncn
on Oct 30, 2019
Author

just add

last_layer = network.get_layer(network.num_layers - 1)
network.mark_output(last_layer.get_output(0))

it will work. It's a necessary procedure to convert successfully, I'm curious why most tutorials don't mention it.

rmccorm4

rmccorm4 commented on Oct 30, 2019

@rmccorm4
rmccorm4
on Oct 30, 2019 · edited by rmccorm4
Collaborator

Yeah, I just added the if statement so you don't get an error for a model that already recognizes it's output.

Also, the originial issue here was likely due to using PyTorch 1.3 - downgrading to PyTorch<=1.2 was likely the fix, on top of marking the output layer.

Closing.

rmccorm4
closed this as completedon Oct 30, 2019
rmccorm4
mentioned this on Nov 2, 2019
rmccorm4
added
Module:ONNXIssues relating to ONNX usage and import
on Nov 2, 2019
rmccorm4
added
Conversion: PyTorchExportONNX
Framework: PyTorch
on Nov 19, 2019
rmccorm4
mentioned this on Dec 9, 2019
RizhaoCai

RizhaoCai commented on Jun 15, 2020

@RizhaoCai
RizhaoCai
on Jun 15, 2020 · edited by RizhaoCai

just add

last_layer = network.get_layer(network.num_layers - 1)
network.mark_output(last_layer.get_output(0))

it will work. It's a necessary procedure to convert successfully, I'm curious why most tutorials don't mention it.

Hi @marooncn,

There's probably a way to fix this in the PyTorch code so that the ONNX parser recognizes the output, but I'm not sure how to do that off the top of my head - maybe someone else can chime in on this.

However, I think you can fix this on the TensorRT side with something like this:

...
last_layer = network.get_layer(network.num_layers - 1)
# Check if last layer recognizes it's output
if not last_layer.get_output(0):
    # If not, then mark the output using TensorRT API
    network.mark_output(last_layer.get_output(0))

You can see similar code being used in this issue: NVIDIA/triton-inference-server#76

Let me know if this or something similar fixes your issue.

Hi, @rmccorm4

Thanks for your solution. It works for many cases.
However, when my network has two outputs, this seems not to work.

For examples

class Net(torch.nn.Module):
    ....
    def forward(x):
        ....
        return output1, output2

In this case, using last_layer.get_output(0) is OK. But if I put last_layer.get_output(1), and OutOfIndex error will occur.
And I find that the using
second_last_layer = network.get_layer(network.num_layers - 2)
second_last_layer.get_output(0)
do not give me what I want.

giussepi

giussepi commented on Jul 13, 2020

@giussepi
giussepi
on Jul 13, 2020

Hi @RizhaoCai

I just had the same issue with TensorRT 7.0. The solution, as mentioned in the documentaiton, was creating the network using the EXPLICIT_BATCH flag:

explicit_batch = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
network = builder.create_network(explicit_batch)
Captain1986

Captain1986 commented on Aug 19, 2020

@Captain1986
Captain1986
on Aug 19, 2020

just add

last_layer = network.get_layer(network.num_layers - 1)
network.mark_output(last_layer.get_output(0))

it will work. It's a necessary procedure to convert successfully, I'm curious why most tutorials don't mention it.

but I meet network.num_layers equals to 0 bug : virtual nvinfer1::ILayer* nvinfer1::Network::getLayer(int) const: Assertion `layerIndex >= 0' failed
any suggestion is welcome

oleksii-udod

oleksii-udod commented on Aug 19, 2020

@oleksii-udod
oleksii-udod
on Aug 19, 2020

just add

last_layer = network.get_layer(network.num_layers - 1)
network.mark_output(last_layer.get_output(0))

it will work. It's a necessary procedure to convert successfully, I'm curious why most tutorials don't mention it.

but I meet network.num_layers equals to 0 bug : virtual nvinfer1::ILayer* nvinfer1::Network::getLayer(int) const: Assertion `layerIndex >= 0' failed
any suggestion is welcome

#183 (comment) helped me with the similar issue.

Captain1986

Captain1986 commented on Aug 19, 2020

@Captain1986
Captain1986
on Aug 19, 2020

just add

last_layer = network.get_layer(network.num_layers - 1)
network.mark_output(last_layer.get_output(0))

it will work. It's a necessary procedure to convert successfully, I'm curious why most tutorials don't mention it.

but I meet network.num_layers equals to 0 bug : virtual nvinfer1::ILayer* nvinfer1::Network::getLayer(int) const: Assertion `layerIndex >= 0' failed
any suggestion is welcome

#183 (comment) helped me with the similar issue.

So, I need to upgrade my TensorRT 6.0.1.5 to TensoeRT 7?

16 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Module:ONNXIssues relating to ONNX usage and import

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @giussepi@stardust2602@theharshwardhan@rmccorm4@HaohaoNJU

        Issue actions
          [TensorRT] ERROR: Network must have at least one output · Issue #183 · NVIDIA/TensorRT