在Golang应用服务器中重新加载Tensorflow模型

Question

I have a Golang app server wherein I keep reloading a saved tensorflow model every 15 minutes. Every api call that uses the tensorflow model, takes a read mutex lock and whenever I reload the model, I take a write lock. Functionality wise, this works fine but during the model load, my API response time increases as the request threads keep waiting for the write lock to be released. Could you please suggest a better approach to keep the loaded model up to date?

Edit, Code updated

Model Load Code:

    tags := []string{"serve"}

    // load from updated saved model
    var m *tensorflow.SavedModel
    var err error
    m, err = tensorflow.LoadSavedModel("/path/to/model", tags, nil)
    if err != nil {
        log.Errorf("Exception caught while reloading saved model %v", err)
        destroyTFModel(m)
    }

    if err == nil {
        ModelLoadMutex.Lock()
        defer ModelLoadMutex.Unlock()

        // destroy existing model
        destroyTFModel(TensorModel)
        TensorModel = m
    }

Model Use Code(Part of the API request):

    config.ModelLoadMutex.RLock()
    defer config.ModelLoadMutex.RUnlock()

    scoreTensorList, err = TensorModel.Session.Run(map[tensorflow.Output]*tensorflow.Tensor{
        UserOp.Output(0): uT,
        DataOp.Output(0): nT},
        []tensorflow.Output{config.SumOp.Output(0)},
        nil,
    )

Answer 1

Presumably destroyTFModel takes a long time. You could try this:

old := TensorModel

ModelLoadMutex.Lock()
TensorModel = new
ModelLoadMutex.Unlock()

go destroyTFModel(old)

So destroy after assign and/or try destroying on another goroutine if it needs to clean up resources and somehow takes a long time blocking this response. I'd look into what you're doing in destroyTFModel and why it is slow though, does it make network requests to the db or involve the file system? Are you sure there isn't another lock external to your app you're not aware of (for example if it had to open a file and locked it for reads while destroying this model?).

Instead of using if err == nil { around it, consider returning on error.