使用Go解析巨大的XML文件

Question

We need to parse a huge XML file using Go. We'd like to use a SAX-like event based algorithm using xml.NewDecoder() and decoder.Token() library calls. We've created the appropriate struct types with XML annotations. Everything easy peasy so far.

Now, we go through the file and detect the xml.StartElement tokens. And here comes the problem. We need to decode ONLY the attributes of this starting token and continue into its content. If we call token.DecodeElement() the whole content is "decoded" or skipped in our scenario.

How to decode only the attributes of a specific StartElement and continue to the element's body?

Answer 1

I parse wikipedia xml dumps (~50GB xml files) in go-wikiparse using plain struct/reflect decoding. It's super simple.

The strategy is basically this:

First, read the envelope token:

d := xml.NewDecoder(r)
_, err := d.Token()
if err != nil {
    return nil, err
}

e.g., for <someDocument><billions-of-other-things/></someDocument> that will give you someDocument.

Then, you can just struct decode the next things in a loop:

var i item
d.Decode(&i)

Not much RAM, and it's super easy to parse.