如何在monorepo中设置protobuf，grpc和Go模块？

Question

I'm trying to create a microservice-based API in Go, very basic, just to learn some stuff. I have three main goals I have to achieve.

1. Monorepo.
2. Be able to use different back-end languages as need arises (ie. one microservice in Go, two in Node.js, one in Python).
3. Use gRPC.

Currently my structure looks like this (and is stored in ~/Projects/tkg, outside of GOPATH).

Each "service" should be a self-contained application written in a "whatever". As you can see I have a Go service and a React front-end application. Additionally there is a Makefile there that I want to use for building stuff, but I might move to shell scripts, Docker, whatever. Doesn't matter.

So now the question. How can I make generated proto files play well with this setup? I think I don't understand something about Go modules and packages because I cannot set it so articles.go (from cmd) can access the generated api/article.pb.go. How to do it?

// services/articles/go.mod
module tkg/services/articles

go 1.12

require (
    github.com/golang/protobuf v1.3.2
    google.golang.org/grpc v1.22.1
)

// services/articles/cmd/article.go
package main

import (
    pb "tkg/services/articles/api/article"
)

type repository interface {
    Create(*pb.Article) (*pb.Article, error)
}

func main() {

}

// services/articles/api/article.proto
syntax = "proto3";

package article;
option go_package = "tkg/services/articles/api/article";
...

// Makefile
build:
    protoc services/articles/api/article.proto --go_out=.

I have tried various different package names in go.mod, different go_packages in the proto file, I had tried different protoc commands and paths. I bet this is silly and it's very obvious to someone who is well-versed in Go, but for someone from Node.js backgroud like me the inability to do import "../api/article.pb.go" is infuriating. :(

The error I am getting is: could not import tkg/services/articles/api/article (no parsed files for package tkg/services/articles/api/article). Of course with different values for package names. I've been trying to solve it for two days now.

How would you approach this problem?

Answer 1

If you are generating the .pb.go file in the same directory as the .proto file (recommended), then your import path should read:

import (
    pb "tkg/services/articles/api"
)

(not tkg/services/articles/api/article.)

Go packages are collections of .go files in a single directory - each with the same package XYZ first line - where XYZ is the package name. So when importing a package, one uses the package base directory - not including any .go filenames.

---

Edit: (this was too long to fit into a comment):

I would step back and think about the base-directory of your entire project.

Standard go packages are usually single words like time, sync etc. because they are part of go's standard library. All other packages should have a full internet path. These typically match a git repo address (e.g. "github.com/boltdb/bolt") - but not always e.g. ("gopkg.in/yaml.v2"). This is actually superior to the centrally-hosted NPM package model - as it easily allows for pulling packages from any repo host. Also since repos are cloned to local disk - those same paths can exist on your local disk first (before they've been hosted on the internet, say for development purposes).

So I would suggest naming your local base-directory something like:

github.com/myname/myproj/tkg/services/...

and then ensure you import packages based on this directory structure e.g.

import "github.com/myname/myproj/tkg/services/api"

If a go build does not pick up the generated (.pb.go) code, then there is something up with your GOPATH or if using the new go-modules your go.mod setup.