Mp4 is not the one universal standard of video data. It's dozens of standards called MPEG-4 now. But if you mean .Mp4 most times you mean H.264 in some container.
Proper implementation of H.264 (that will work as fast and as stable that ready-made solutions) is very hard work that requires advanced computer science skills, expert programming skills, expert knowledge of the H.264-related standards, and a lot of time. The draft that you can find on ITU site contains 671 pages and it's referenced to few (or dozens?) other standards.
And another bad news: draft of standard is not the current standard. You need to buy it.
Some of aspects of realization of H.264: frame data compression, entropy coding, code correction, video filters, color profiles. It's really complicated things, all of them.
So why not have try to use the ffmpeg?
You can execute ffmpeg executable with "os/exec" package:
out, err := exec.Command("ffmpeg", "-f", "gif", "-i", gifFileName, mp4FileName).Output()
Or even try use ffmpeg bindings for go https://github.com/giorgisio/goav
The use of ready-made solutions will save years of your work ;)