As part of scaling pods in kubernetes I want to ensure I gracefully serve my http connections before shutting down. To that extent I have implemented this code in go:
package main
import (
"fmt"
"io"
"net/http"
"os"
"os/signal"
"syscall"
"github.com/braintree/manners"
)
func main() {
shutdown := make(chan int)
//create a notification channel to shutdown
sigChan := make(chan os.Signal, 1)
//start the http server
http.HandleFunc("/", hello)
server := manners.NewWithServer(&http.Server{Addr: ":80", Handler: nil})
go func() {
server.ListenAndServe()
shutdown <- 1
}()
//register for interupt (Ctrl+C) and SIGTERM (docker)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
go func() {
<-sigChan
fmt.Println("Shutting down...")
server.Close()
}()
<-shutdown
}
func hello(w http.ResponseWriter, r *http.Request) {
// time.Sleep(3000 * time.Millisecond)
io.WriteString(w, "Hello world!")
}
This looks out for the docker SIGTERM and gracefully shuts down after existing requests have been served. When I run this container in kubernetes with 10 instances I can scale up and down without incident, as long as I don't scale down to a single instance. When I scale to a single instance I see a short set of http errors, then all looks fine again.
I find it strange as in scaling I would assume the proxy is updated first, then containers are shut down and the code above would allow requests to be served out.
In my current setup I am running 2 nodes, maybe the issue is when scaling drops below the number of nodes and there is some sort of timing issue with etcd updates? Any insight into what is going on here would be really useful