I'm working on a web application written in Golang that needs to call a Python program/module to do some heavy work. Since that is very memory/CPU intensive, it may be on a separate machine. Since Golang and Python can't talk directly, there are 3 ways to achieve this:
- Just execute the python program as an OS process from Go (if on same machine) (or RPC?)
- Wrap Python process in a service and expose it for it to be called from Go (may be a simple CRUD like service - A Bottle/flask restful service)
- Have a simple pub-sub system in place to achieve this (Redis or some MQ system) - Adding Redis based caching is on the radar so maybe a good reason to go this way. Not sure.
The main thing is that the python process that takes really long to finish must "inform" the web application that it has finished. The data could either be in a file/DB or 'returned' by the process.
What could be the simplest way to achieve this in a pub/sub like environment?
UPDATE REST seems like one way but would incur the cost of implementing server side push which may or may not be easily doable with existing micro web frameworks. The pub/sub would add an additional layer of complexity for maintainability and a learning curve nevertheless. I'm not sure if an RPC like invocation could be achieved across machines. What would be a good choice in this regard?