将请求发送到Https站点

Question

Essentially, through goLang I'm trying to send a request on a https site to check if an item is on the site. I have tried to attempt a request to the main site, but keep getting access denied and need a way to tackle this, I'm trying to get the info from the body to separate it and find the correct ids to check if something is on the site.

package main

import (
    "fmt"
    "net/http"
    "io/ioutil"
)

func main() {

   url := "https://www.jdsports.co.uk/"

   req, _ := http.NewRequest("GET", url, nil)

   res, _ := http.DefaultClient.Do(req)

   defer res.Body.Close()

   body, _ := ioutil.ReadAll(res.Body)

   fmt.Println(string(body)) 
}

Answer 1

After testing a little bit, it looks like that specific website is using Akamai Ghost and has been configured to block the default go http package user agent.

The default user agent appears to be Go-http-client/1.1

If you change your user agent

req.Header.Set("User-Agent", "my-client-app")

The request will work. However, the website in question appears to not want to be programatically crawled. Perhaps you should respect their wishes.