I need to fetch the contents from many several URLs and store it in AWS S3. I've written a function to do that which works. But I am looking to make it faster and more efficient by re-using http client connection and re-using the AWS session. Furthermore I'm looking to get them to run concurrently, say 5 at a time.
func fetchPut(fromURL string, toS3 string) error {
start := time.Now()
resp, err := http.Get(fromURL)
if err != nil {
return err
}
defer resp.Body.Close()
sess := session.Must(session.Must(session.NewSession()))
s3svc := s3.New(sess)
s3URL, _ := url.Parse(toS3)
byteArray, _ := ioutil.ReadAll(resp.Body)
fetchElapsed := time.Since(start).Seconds()
start = time.Now()
input := &s3.PutObjectInput{
Body: bytes.NewReader(byteArray),
Bucket: aws.String(s3URL.Host),
Key: aws.String(s3URL.Path),
}
_, err = s3svc.PutObject(input)
putElapsed := time.Since(start).Seconds()
return err
}
What I don't understand is how I can re-use the session (both http & AWS). Can I have it in some global variable? Or do I have to create some sort of context?
Are there any good examples of this sort of use case to study?