将文件发送给齿轮工人

Question

I'm currently separating our video conversion part of the web page (kinda like youtube where users upload videos and we convert them to flv/mp4) to a different server. I already have the system running with gearman on the same machine. So when a user uploads a video file to server A in gets picked by a gearman worker on the same server A.

Now I moved the worker to server B. So worker on server B needs to access the uploaded file on server A. Currently I use SCP to copy the file from A to B and then process it. This method works but I feel like there should be a more clean way of doing it but I haven't found any information about sending files (or large files) to gearman workers. How would you approach this problem?

Preferably the client would send the video file as part of the command to start a background job, so I don't have to worry where the file actually is from within the worker. That way I can add more conversion servers without to much hassle.

I'm using PHP (with Gearman extension) for both my webpage and the worker.

Answer 1

As was suggested in the comments, having a shared FS is the (usual) way to implement this, and simply pass the path around in the job request from gearman. Gearman is not well-suited for passing around large blobs of data, as it has to keep all of the information for a job in memory. It was never designed for handling the transfer and distribution of large files. Since MogileFS was also initially developed at Danga, there simply was no need to also incorporate file transfer and handling in Gearman (and that's a good thing, there's quite a few technologies that solve that problem better than Gearman would ever do).

We're using NFS for handling distributed workers when videos arrive, and the encoder puts the encoded video back onto the NFS share that's available to the public when it's done. Haven't had a serious issue yet, NFS is stable and it's problems are well known and already solved for the kind of loads you'll see.