I'm newer to the Go language and it's resources, but have been looking around for quite some time without any luck of finding what I'm looking for. So if there is a resource out there for it, I apologize for the duplicate question and would appreciate being directed that way.
My goal is simply to build a web scraper. I'm using chromedp, which has features to focus on elements, fill in text, etc. I want to create a test environment/server to test it with during development. The main reason being that I do not want to constantly create GET requests from a website (out of common courtesy), but also be able to work offline and in addition it should also make testing a little faster. I stumbled across the go-vcr library and have been trying to get that to work, but to no avail. I can get it to record and create a .yaml
, but I can't figure out how to test beyond the raw html that bounces back and gets stored in the .yaml
file. My understanding is that it's possible to replicate the website and functionality of it using the library, but I'm unable to piece together how to do that.
Is what I'm trying to do possible, or is the go-vcr library (or any test/fake server for that matter) only capable of returning static data, therefore rendering anything I want to test with the web scraper not possible?
I haven't posted any code simply because I haven't pieced together much more than the examples given from the repository for the go-vcr.
I hope I was able to explain that in a way that made sense. If not I'd be happy to answer questions to clarify.
Update: Adding the example code for sake of ease. I understand how this part of it works (I think) and I can use it for testing whether or not I grabbed the proper elements of a static page, but ideally (as an example) I want to be able to fill in a text box with my program and test whether or not I successfully found the text box and filled it in without hitting the live webpage.
package vcr_test
import (
"io/ioutil"
"net/http"
"strings"
"testing"
"github.com/dnaeon/go-vcr/recorder"
)
func TestSimple(t *testing.T) {
// Start our recorder
r, err := recorder.New("fixtures/golang-org")
if err != nil {
t.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it
// Create an HTTP client and inject our transport
client := &http.Client{
Transport: r, // Inject as transport!
}
url := "http://golang.org/"
resp, err := client.Get(url)
if err != nil {
t.Fatalf("Failed to get url %s: %s", url, err)
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("Failed to read response body: %s", err)
}
wantTitle := "<title>The Go Programming Language</title>"
bodyContent := string(body)
if !strings.Contains(bodyContent, wantTitle) {
t.Errorf("Title %s not found in response", wantTitle)
}
}