weixin_39970166
weixin_39970166
2020-12-26 12:55

Make the initial mailbox sync interruptable and resumable

Fixes #2578

With large mailboxes the initial sync is slow. Therefore it's prone to run into timeouts. I was hoping cron to be my last resort, but it turns out we can't rely on it as the instance might not use system cron but web cron. Then the same timeout rules apply as for any other AJAX requests. Thus we need an alternative.

I've changed the sync logic to limit the cache population to x (currently 1000 but I'll increase) message per run. Then it just stops without setting the syn tokens. The next sync attempt will detect the missing cache and start over. However, it will also check for the highest known UID in the database, and only fetch the ones that are newer than this message. It relies on the fact that the previous inital sync attempt did not leave any holes. But that shouldn't happen usually. Messages are also inserted in ascending order and both messages and recipients are inserted in the same transaction. Thus this should be a save way to resume the operation with the previous progress. Eventually one of the attempts will finish and no more messages can be fetched. Then the tokens will be set. From then on, partial sync can be used.

This also fixes the /sync request to be a POST. It was okay-ish to do it as GET for the old Horde sync as it was indeed stateless. But now it's changing server state, thus POST seems more appropriate. Moreover we send the list of known UIDs as parameter and AFAIK there are limits in URL length while with POST body we can send more data.

Steps to test

  • Close the app
  • Run update oc_mail_mailboxes set sync_new_token=null, sync_new_lock=null, sync_changed_token=null, sync_changed_lock=null, sync_vanished_token=null, sync_vanished_lock=null;, delete from oc_mail_messages; and delete from oc_mail_recipients;
  • Open the app and browse to a large inbox

You should then see the "indexing … might take longer … " message. and in the browser network tab you see /sync requests. lots of 202s and then finally a 200. then the mailbox content should show.

Todo

  • [x] A bit more testing
  • [x] Find sweet spot of messages per request to optimize performance and still avoid timeouts
  • [x] Ask for a test run on his large mailbox(es)

该提问来源于开源项目:nextcloud/mail

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

5条回答

  • weixin_39970166 weixin_39970166 4月前
    点赞 评论 复制链接分享
  • weixin_39605521 weixin_39605521 4月前

    Didn't review the code really, but I ran it on my 80k folders and they seem to work fine

    点赞 评论 复制链接分享
  • weixin_39846898 weixin_39846898 4月前

    Didn't review the code really, but I ran it on my 80k folders and they seem to work fine

    Bold move, my friend 🙇

    点赞 评论 复制链接分享
  • weixin_39605521 weixin_39605521 4月前

    Bold move, my friend 🙇

    Since I don't depend on Nextcloud Mail it's not as bold as it sounds, and someone has to try it (we ran into timeouts before that is why this follow up is now needed

    点赞 评论 复制链接分享
  • weixin_39970166 weixin_39970166 4月前

    Testing in production > unit testing :grinning:

    点赞 评论 复制链接分享

相关推荐