I'm writing a fairly complex Go webapp that I want to make highly available. I'm planning on having multiple VMs running the app, with a load-balancer directing traffic between them for what I want.
Where it gets complex is that the webapp has a sort of database book-keeping routine running, which I only want (at most) one instance of at any time. So if I have three webapp VMs, only one of them should be doing the book-keeping.
(Yes, I'm aware that I could split the book-keeping off into a separate VM instance entirely, but the code has been rather heavily integrated with the rest of the webapp.)
I've spent several hours looking at things like etcd, raft, bully, memberlist, Pacemaker, and so on and so forth. These all seem like quite a lot of information to absorb to accomplish what I'm after, or I can't see a clear way of using them.
What I would like, in this specific use case, is a system whereby Go webapp nodes automatically detect each other and elect a "leader" to do the book-keeping. Ideally this would scale anywhere from 2 to 10 nodes, and not require manually adding IP addresses to config files (but possible, if necessary).
I was thinking in the case of a network partition or something, where one node cannot see the others, I wouldn't want it to elect itself as a leader, because it would be possible to have two nodes attempting to do book-keeping at the same time. That also means that if I stripped down the cluster to being just a single VM, no book-keeping would occur, but that could be tolerated for a brief period during maintenance, or I could set some sort of flag somewhere.
I'm wondering if someone could point me in the right direction, and hopefully how I can accomplish this with low complexity, while leveraging existing code libraries as much as possible.