I have such document and I want to read each files in loop which is 5GB of size, I tried some way such as (file, err := ioutil.ReadFile(filename)) but it's loads the entire file into memory.I used this func for load files:
func visit(files *[]string) filepath.WalkFunc {
return func(path string, info os.FileInfo, err error) error {
if err != nil {
log.Fatal(err)
}
*files = append(*files, path)
return nil
}
}
and for read files I used:
file, err := os.Open("file")
if err != nil {
log.Fatal(err)
}
defer file.Close()
buf := make([]byte,10*1024)
for {
n, err := file.Read(buf)
if n > 0 {
fmt.Print(buf[:n])
}
if err == io.EOF {
break
}
I want to parse data from buf
err = xml.Unmarshal(buf, &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
m is:
type M struct {
Mc []struct {
Id string `xml:"id"`
NeId string `xml:"neid"`}`xml:"mc"`
Mr struct {
Mh []string `xml:"mh"`}`xml:"mr"`
}
and in func main:
func main() {
var files []string
root := "/folder/files"
err := filepath.Walk(root, visit(&files))
if err != nil {
panic(err)
}
for _, file := range files {
but it takes too long time to execute, what should I do to fast this process? I get an error XML syntax error on line 496: unexpected EOF. concurrency may be useful in this case?