gRPC-GoLang-Stackdriver tracer

Question

I am trying to get the stackdriver tracer to work with gRPC and I need some help. I have been looking at these two links for reference and I still can't get it to work:

• https://medium.com/@harlow/tracing-grpc-calls-in-golang-with-google-stackdriver-b22495763a06#.81oa9q21v
• https://rakyll.org/grpc-trace/

For simplicity, I am just working with the hello world gRPC example. Here's my client:

func main() {

    // Set up a connection to the server.
    conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithUnaryInterceptor(grpc.UnaryClientInterceptor(clientInterceptor)))
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()
    c := pb.NewGreeterClient(conn)

    ctx := context.Background()

    tc, err := trace.NewClient(ctx, "{PROJECT-ID}")
    if err != nil {
        log.Fatal(err)
    }

    span := tc.NewSpan("/greeter/SayHello")
    defer span.Finish()

    ctx = trace.NewContext(ctx, span)

    r, err := c.SayHello(ctx, &pb.HelloRequest{Name: "world"})
    if err != nil {
        log.Fatalf("could not greet: %v", err)
    }

    println("Response:", r.Message)
}

func clientInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
    // trace current request w/ child span
    span := trace.FromContext(ctx).NewChild(method)
    defer span.Finish()

    // new metadata, or copy of existing
    md, ok := metadata.FromContext(ctx)
    if !ok {
        md = metadata.New(nil)
    } else {
        md = md.Copy()
    }

    // append trace header to context metadata
    // header specification: https://cloud.google.com/trace/docs/faq
    md["X-Cloud-Trace-Context"] = append(
        md["X-Cloud-Trace-Context"], fmt.Sprintf("%s/%d;o=1", span.TraceID(), 0),
    )
    ctx = metadata.NewContext(ctx, md)

    return invoker(ctx, method, req, reply, cc, opts...)
}

.. and my gRPC server:

// server is used to implement helloworld.GreeterServer.
type server struct{}

// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
    println("HERE")
    return &pb.HelloReply{Message: "Hello " + in.Name}, nil
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }

    ctx := context.Background()
    tc, err := trace.NewClient(ctx, "{PROJECT-ID}")
    if err != nil {
        log.Fatal(err)
    }

    s := grpc.NewServer(EnableGRPCTracingServerOption(tc))
    pb.RegisterGreeterServer(s, &server{})

    println("listening on :50051")
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

// EnableGRPCTracingServerOption enables parsing google trace header from metadata
// and adds a new child span to the incoming request context.
func EnableGRPCTracingServerOption(traceClient *trace.Client) grpc.ServerOption {
    return grpc.UnaryInterceptor(serverInterceptor(traceClient))
}

func serverInterceptor(traceClient *trace.Client) grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
        // fetch metadata from request context
        md, ok := metadata.FromContext(ctx)
        if !ok {
            md = metadata.New(nil)
        }
        header := strings.Join(md["X-Cloud-Trace-Context"], "")

        // create new child span from google trace header, add to
        // current request context
        span := traceClient.SpanFromHeader(info.FullMethod, header)
        defer span.Finish()
        ctx = trace.NewContext(ctx, span)

        return handler(ctx, req)
    }
}

I when I run the client to initiate the trace, I get the error:

rpc error: code = 13 desc = stream terminated by RST_STREAM with error code: 1

I'm confused because I don't see anything else about authentication; only providing the project ID which can't be enough to initiate tracing for a specific project. What am I missing?

Answer 1

The issue was with:

defer span.Finish()

That call does not block so because I was just doing preliminary testing with one call my program was exiting before the traces could be uploaded. I contacted the author of https://rakyll.org/grpc-trace/ and she actually updated her post with the option of using:

defer span.FinishWait()

which blocks and that fixed it by allowing the traces to be successfully uploaded before the program exited.

Also, with a long running webserver this wouldn't have been an issue because the process wouldn't have been terminated.