Skip to content

gRPC streaming keepAlive doesn't work with docker swarm #2549

Closed
@npuichigo

Description

@npuichigo

Recently, I used the grpc-gateway to transcode HTTP/JSON to grpc. The gRPC-Gateway uses a gRPC client to talk to the backend, but in a docker swarm setup using overlay networks, idle connections between grpc client and backend service will end up in a broken state after 15 minutes. (moby/moby#31208)

So I tried the following two ways. Firstly, I set net.ipv4.tcp_keepalive_time to less than 900 seconds, to make sure the TCP connection between grpc-gateway and IPVS doesn't expire. Secondly, I configured grpc.WithKeepaliveParams to set the ping time interval to 30 seconds. However, neither method works.

Here is the code snippet of my client:

ctx := context.Background()
        ctx, cancel := context.WithCancel(ctx)
        defer cancel()

        mux := runtime.NewServeMux()
        opts := []grpc.DialOption{
                grpc.WithInsecure(),
                grpc.WithKeepaliveParams(keepalive.ClientParameters{
                        Time:                30 * time.Second,
                        Timeout:             20 * time.Second,
                        PermitWithoutStream: true,
                }),
        }
        err := gw.RegisterMyHandlerFromEndpoint(ctx, mux, *echoEndpoint, opts)

Is there anyway to configure grpc-go client to support docker swarm?

Activity

mastersingh24

mastersingh24 commented on Jan 6, 2019

@mastersingh24

Did you also set https://godoc.org/google.golang.org/grpc/keepalive#EnforcementPolicy on the server? The default permitted value for time between client pings is 5 minutes ... if you set your client to ping every 30 seconds you'll need to set a matching policy on the gRPC server as well.
We've been using the gRPC keepalives on both the client and server in production for quite some time now. The main thing is making sure you get the settings correct on both sides.

npuichigo

npuichigo commented on Jan 6, 2019

@npuichigo
Author

Thanks for reminding, but my grpc server is based on c++. Is there any counterpart of EnforcementPolicy in grpc c++?

mastersingh24

mastersingh24 commented on Jan 6, 2019

@mastersingh24

I believe the C++ server still wraps gRPC core. You can find the settings here:
https://github.com/grpc/grpc/blob/master/doc/keepalive.md

Make sure to pay attention to the GRPC_ARG_HTTP2_MIN_RECV_PING_INTERVAL_WITHOUT_DATA_MS,GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA, GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS settings.

npuichigo

npuichigo commented on Jan 6, 2019

@npuichigo
Author

I tried the following configurations on c++ server and golang client to make sure that the permitted ping interval of c++(5 minutes by default) is smaller than the golang client, but it still doesn't work.

grpc::ServerBuilder builder;
// Listen on the given address without any authentication mechanism.
builder.AddListeningPort(FLAGS_address, grpc::InsecureServerCredentials());
// Enable to send HTTP2 keepalive pings over the transport.
builder.AddChannelArgument(GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS, 1);
builder.AddChannelArgument(GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA, 0);
ctx := context.Background()
        ctx, cancel := context.WithCancel(ctx)
        defer cancel()

        mux := runtime.NewServeMux()
        opts := []grpc.DialOption{
                grpc.WithInsecure(),
                grpc.WithKeepaliveParams(keepalive.ClientParameters{
                        Time:                10 * time.Minute,
                        Timeout:             20 * time.Second,
                        PermitWithoutStream: true,
                }),
        }
        err := gw.RegisterMyHandlerFromEndpoint(ctx, mux, *echoEndpoint, opts)
mastersingh24

mastersingh24 commented on Jan 6, 2019

@mastersingh24

You definitely need to ping more often than every 10 min when using Docker / Docker Swarm. The Docker proxies timeout after 10 minutes ... we use 5 * time.Minute for the client keepalive time ... you can also try 6 as well ... anything less than 10 should keep idle connections open with Docker Swarm (or just Docker in general)

mastersingh24

mastersingh24 commented on Jan 9, 2019

@mastersingh24

@npuichigo - hopefully you were able to resolve this?

npuichigo

npuichigo commented on Jan 9, 2019

@npuichigo
Author

@mastersingh24 - Yes, the problem has been solved. Thank you for your concern.

locked as resolved and limited conversation to collaborators on Jul 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @mastersingh24@npuichigo

        Issue actions

          gRPC streaming keepAlive doesn't work with docker swarm · Issue #2549 · grpc/grpc-go