duanniu4106
2019-06-17 15:34
浏览 251

从Ubuntu amd64到arm7l进行交叉编译:exec用户进程导致“ exec格式错误”

I am having an headache with cross compilation from amd64 to arm7l

I could finally do it with Gitlab CI, so now, I compile my binary in a docker image, here is the dockerfile:

FROM golang

WORKDIR /go/src/gitlab.com/company/edge_to_bc

COPY . .
RUN dpkg --add-architecture armhf && apt update && apt-get install -y gcc-arm-linux-gnueabihf libltdl-dev:armhf

I build it as Then I will build the the new container "cross-compil" ready with the name ubuntu:cross-compil

Now, I can compile my binary with:

docker run -it -v ${EDGE_TO_BC_PATH}/release:/go/src/gitlab.com/company/edge_to_bc/release ubuntu:cross-compil  bash -c 'CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=7 go build -v -o ./release/edge_to_bc '

I can see my executable generated in ./release/edge_to_bc

Then I build my docker image:

docker build -t registry.gitlab.com/company/edge_to_bc:armhf .

And I push it.

In the Dockerfile, I just copy the executable from host:

FROM alpine:3.7

RUN apk --no-cache add ca-certificates libtool

WORKDIR /sunclient/

COPY ./release/edge_to_bc ./
EXPOSE 5555

CMD [ "./edge_to_bc" ]

But when I run it in my arm board with:

docker run --rm registry.gitlab.com/company/edge_to_bc:armhf

I get:

standard_init_linux.go:207: exec user process caused "no such file or directory"

When debugging, if I want to get list of files with

docker run --rm registry.gitlab.com/company/edge_to_bc:armhf

I get:

standard_init_linux.go:207: exec user process caused "exec format error"

Which indicate binary still not have correct format...

What did I miss ? I spent a lot of time on this topic, and don't have much more ideas.

When I check the architecture of binary, this is what I get:

 edge_to_bc git:(master) ✗ readelf -h ./release/edge_to_bc
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x19209
  Start of program headers:          52 (bytes into file)
  Start of section headers:          23993360 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         10
  Size of section headers:           40 (bytes)
  Number of section headers:         49
  Section header string table index: 48

On the target OS, this is what I get:

[root@gw-sol1 ~]# uname -a
Linux gw-sol-1 4.4.113-UNRELEASED #1 SMP PREEMPT Thu Mar 7 16:46:40 CET 2019 armv7l armv7l armv7l GNU/Linux

EDIT:

When I build the app directly on ARM device, it will work:

go build -o ./release/edge_to_bc -v -ldflags '-w -s -extldflags "-static"' ./...

the ELF:

ELF Header:
  Magic:   7f 45 4c 46 01 01 01 03 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x125f1
  Start of program headers:          52 (bytes into file)
  Start of section headers:          16594072 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         7
  Size of section headers:           40 (bytes)
  Number of section headers:         38
  Section header string table index: 37

It seems quite similar to the other one, at least in architecture.

Then build the docker image:

docker build . -t image/peer-test:armhf
  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • dongqiang8683 2019-06-18 13:40
    已采纳

    When you run the build on the amd64 system of the arm7 image with this command:

    docker build -t registry.gitlab.com/company/edge_to_bc:armhf .
    

    It will use base images and run commands in that image for amd64. So even if your single edge_to_bc binary may be cross compiled, the rest of the image is not. Next, the binary compile command itself looks like it is linking to libraries, which quite likely are not in your final image. You can run ldd edge_to_bc to see these links, and if they are missing, you'll get the file not found error.


    In my own cross compile test, I'm using BuildKit, Buildx, and some experimental features on 19.03.0-rc2, so there will be parts of this that are not backwards compatible, but hopefully you find helpful. I'm using a multi-stage build to avoid the compile outside of docker and then a second build. I'm also specifying the platform for the build host and using the target arch and OS variables to configure go to cross compile. In this scenario, I went with a completely statically linked binary so there were no libraries to include, and with only copy commands in my final release image, I avoid issues running the build on a different platform.

    # syntax=docker/dockerfile:experimental
    # ^ above line must be at the beginning of the Dockerfile for BuildKit
    
    # --platform pulls an image for the build host rather than target OS/Arch
    FROM --platform=$BUILDPLATFORM golang:1.12-alpine as dev
    RUN apk add --no-cache git ca-certificates
    RUN adduser -D appuser
    WORKDIR /src
    COPY . /src/
    CMD CGO_ENABLED=0 go build -o app . && ./app
    
    # my dev stage is separate from build to allow mounting source and rebuilding
    # on developer machines    
    FROM --platform=$BUILDPLATFORM dev as build
    ARG TARGETPLATFORM
    ARG TARGETOS
    ARG TARGETARCH
    # --mount is an experimental BuildKit feature
    RUN --mount=type=cache,id=gomod,target=/go/pkg/mod/cache \
        --mount=type=cache,id=goroot,target=/root/.cache/go-build \
        CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
        go build -ldflags '-w -extldflags -static' -o app .
    USER appuser
    CMD [ "./app" ]
    
    # this stage will have the target OS/Arch and cannot have any RUN commands
    FROM scratch as release
    COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
    COPY --from=build /etc/passwd /etc/group /etc/
    COPY --from=build /src/app /app
    USER appuser
    CMD [ "/app" ]
    
    FROM scratch as artifact
    COPY --from=build /src/app /app
    
    FROM release
    

    To build this, I can run one-off build commands with BuildKit:

    DOCKER_BUILDKIT=1 docker build --platform=linux/amd64 -f Dockerfile -t golang-app:amd64 .
    DOCKER_BUILDKIT=1 docker build --platform=linux/arm64 -f Dockerfile -t golang-app:arm64 .
    

    But even better is the Buildx version that creates a multi-arch image that will run on multiple platforms. This does require that you push to a registry server:

    docker buildx build -f Dockerfile --platform linux/amd64,linux/arm64 \
      -t ${docker_hub_id}/golang-app:multi-arch --output type=registry .
    

    For your scenario, you would swap out the references from arm64 to your own architectures. The --platform option was listed as experimental in many commands I ran, so you may need to configure the following in the /etc/docker/daemon.json file to enable:

    { "experimental": true }
    

    I believe this required a full restart of the docker daemon (systemctl restart docker), not just a reload, to take effect.

    To extract the artifact from the build (the compiled binary for the specific architecture), I use:

    docker build -f Dockerfile --target artifact -o type=local,dest=. .
    

    That will output the contents of the artifact stage (a single binary) to the local directory.


    The above is option 3 in Docker's list of ways to build multi-arch images.

    Option 1 is to configure qemu with binfmt_misc to build and run images for different platforms. I have not yet been able to get this to work on Linux with Buildx, but Docker has done this for Mac and you may be able to find more details on what they did in the LinuxKit project. Using qemu for your situation may be ideal if you need to run commands and install other tools as part of your build, not just cross compile a single statically linked binary.

    Option 2 is to run the build on the target platform, which as you've seen works well. With Buildx, you can add multiple build nodes, up to one per platform, allowing you to run the build from a single location (CI server).

    点赞 打赏 评论

相关推荐 更多相似问题