dongzhang6544 2017-02-02 03:03
浏览 12

使用Go-VCR测试Go Web抓取工具

I'm newer to the Go language and it's resources, but have been looking around for quite some time without any luck of finding what I'm looking for. So if there is a resource out there for it, I apologize for the duplicate question and would appreciate being directed that way.

My goal is simply to build a web scraper. I'm using chromedp, which has features to focus on elements, fill in text, etc. I want to create a test environment/server to test it with during development. The main reason being that I do not want to constantly create GET requests from a website (out of common courtesy), but also be able to work offline and in addition it should also make testing a little faster. I stumbled across the go-vcr library and have been trying to get that to work, but to no avail. I can get it to record and create a .yaml, but I can't figure out how to test beyond the raw html that bounces back and gets stored in the .yaml file. My understanding is that it's possible to replicate the website and functionality of it using the library, but I'm unable to piece together how to do that.

Is what I'm trying to do possible, or is the go-vcr library (or any test/fake server for that matter) only capable of returning static data, therefore rendering anything I want to test with the web scraper not possible?

I haven't posted any code simply because I haven't pieced together much more than the examples given from the repository for the go-vcr.

I hope I was able to explain that in a way that made sense. If not I'd be happy to answer questions to clarify.

Update: Adding the example code for sake of ease. I understand how this part of it works (I think) and I can use it for testing whether or not I grabbed the proper elements of a static page, but ideally (as an example) I want to be able to fill in a text box with my program and test whether or not I successfully found the text box and filled it in without hitting the live webpage.

package vcr_test

import (
    "io/ioutil"
    "net/http"
    "strings"
    "testing"

    "github.com/dnaeon/go-vcr/recorder"
)

func TestSimple(t *testing.T) {
    // Start our recorder
    r, err := recorder.New("fixtures/golang-org")
    if err != nil {
        t.Fatal(err)
    }
    defer r.Stop() // Make sure recorder is stopped once done with it

    // Create an HTTP client and inject our transport
    client := &http.Client{
        Transport: r, // Inject as transport!
    }

    url := "http://golang.org/"
    resp, err := client.Get(url)
    if err != nil {
        t.Fatalf("Failed to get url %s: %s", url, err)
    }

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        t.Fatalf("Failed to read response body: %s", err)
    }

    wantTitle := "<title>The Go Programming Language</title>"
    bodyContent := string(body)

    if !strings.Contains(bodyContent, wantTitle) {
        t.Errorf("Title %s not found in response", wantTitle)
    }
}
  • 写回答

0条回答 默认 最新

    报告相同问题?

    悬赏问题

    • ¥20 双层网络上信息-疾病传播
    • ¥50 paddlepaddle pinn
    • ¥20 idea运行测试代码报错问题
    • ¥15 网络监控:网络故障告警通知
    • ¥15 django项目运行报编码错误
    • ¥15 请问这个是什么意思?
    • ¥15 STM32驱动继电器
    • ¥15 Windows server update services
    • ¥15 关于#c语言#的问题:我现在在做一个墨水屏设计,2.9英寸的小屏怎么换4.2英寸大屏
    • ¥15 模糊pid与pid仿真结果几乎一样