使用Golang进行mmap syscall读取

I am not an Go unsafe package expert - Neither am I a seasoned C programmer. I am trying to read a huge file > 1G using mmap syscall in go. There are a number of reasons I do mmap and munmap as opposed to read, write I/O. That is beside the point - I can write to the file in a test, when I read from the file, I can ascertain that bytes length matches up, but I cannot read contents of this string file :( Can someone suggest some reading? I need to do to go a little further, here's some code I cooked up for sample test:

filename := "/tmp/dd_file.db"
f, err := os.OpenFile(filename, os.O_RDWR, 0666)
defer f.Close()
if err != nil {
    fmt.Printf("error opening file: %v", err)
}
stat, _ := f.Stat()
size := stat.Size()
fmt.Printf("[READ-ONLY] : size was : %+v
", size)
got := make([]byte, size)
if _, err := f.ReadAt(got, 0); err != nil && err != io.EOF {
    panic(err)
}
want, err := ioutil.ReadFile(filename)
if err != nil {
    fmt.Printf("[READ-ONLY] : ioutil.ReadFile: %v", err)
}
// going to change the file size now, punch in a few things
t := unsafe.Sizeof("")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
_, err = f.Seek(int64(t-1), 0)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
_, err = f.Write([]byte(" "))
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
mmap, err := syscall.Mmap(int(f.Fd()), 0, int(t), syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
 // not too sure on reading data on string - doesnt work as expected.
map_array := (*[10000]string)(unsafe.Pointer(&mmap[0]))
map_array[0] = "yellow!"
err = syscall.Munmap(mmap)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
newStat, _ := f.Stat()
newSize := newStat.Size()
fmt.Printf("[mmap( ) RW] : size was : %+v
", newSize)
got = make([]byte, newSize)
if _, err := f.ReadAt(got, 0); err != nil && err != io.EOF {
    panic(err)
}
if len(got) == len(want) {
    fmt.Println("well the lengths are equal atleast??!")
}
if !bytes.Equal(got, want) {
    fmt.Printf("
 [mmap( ) RW] : works! got  %d 
 want %d", len(got), len(want))
}

This obviously works as expected - but what if I wanted to read via mmap( ) on an mmapped file, how do I read string out of these bytes (I have a sense there is an encoding package somewhere that I might have to put to use perhaps but then StringHeader on unsafe documentation confused me).

Suggestions.

douzhoulei8959
douzhoulei8959 我做mmap和munmap的原因有很多,而不是读写I/O。并且,如果这些原因与性能相关,那么您是否真的对mmap()与read()/write()进行基准测试?因为如果出于性能原因使用此类mmap(),则几乎可以肯定是错误的。阅读此答案,尤其是指向一个LinusTorvalds的两个链接,说明mmap()的速度如何缓慢。并注意,尽管答案很好,但作者最后还是将mmap()与fread()进行了比较-缓冲,基于stdio且速度较慢。
大约 3 年之前 回复
dtla92562
dtla92562 看看https://github.com/riobard/go-mmap该软件包可以用作mmap使用的参考。要将字符串写入mmap,可以执行copy(mmap,[]byte(“Yourstring”))。
大约 3 年之前 回复
douzhuang2016
douzhuang2016 我想避免阅读器-但我的问题是我怀疑通过mmap将字符串写入字节时,不是仅刷新一些控制字符就刷新了合法字符-我怀疑我的编码可能搞砸了-有什么建议吗?
大约 3 年之前 回复
dsdtumf776629385
dsdtumf776629385 我可能听不懂你的意思。如果您使用的是mmap,文件的内容不是以[]byte可用吗?要获取字符串,可以执行string(mmap[:100])<-将前100个字节转换为字符串。或者您可以使用bytes.Buffer(但可能不是您想要的,因为避免了io.reader/io.writer模式)
大约 3 年之前 回复

1个回答



正如@putu在注释中所指出的,可以通过简单的类型转换将字节片转换为字符串:</ p> \ n

  asStr = string(byteSlice)//整个切片作为字符串
partStr = string(byteSlice [:100])//前100个字节作为字符串
</ code> </ pre >
</ div>

展开原文

原文

As @putu noted in a comment, a byte slice can be converted to a string by a simple type conversion:

asStr = string(byteSlice) // entire slice as a string
partStr = string(byteSlice[:100]) // first 100 bytes as a string

dtpyvb1873
dtpyvb1873 可能控制字符位于数据中,可能出现的编码与输入的编码不匹配,并且可能是在从多字节UTF-8代码中间开始的字节片上进行字符串转换 点。
大约 3 年之前 回复
douxin5953
douxin5953 那是行不通的-我正在不安全地将指针从mmap转换为字节数组,以获取字符串数组指针-但是当我阅读时,我看到一些控制字符,也许我不是将字符串正确编码为字节 通过mmap将它们刷新为文件?
大约 3 年之前 回复
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问
相关内容推荐