duandi2853 2018-08-02 21:16
浏览 300

Golang缓存HTTP GET结果在内存中

I am working on a CLI in Go that scrapes a webpage to collect the href attributes of all the links on the page into a slice. I want to store this slice in memory for some time so that the scraper is not being called on every execution of the CLI command. Ideally, the scraper would only be called after the cache expires or the user provides some sort of --update flag.

I came across the library go-cache and other similar libraries, but from what I could tell they only work for something that is continuously running, like a server.

I thought about writing the links to a file, but then how would I expire the results after a specific duration? Would it make sense to create a small server in the background that shuts down after a while in order to use a library like go-cache? Any help is appreciated.

  • 写回答

2条回答 默认 最新

  • drlhsfqoa350437979 2018-08-02 22:33

    There are two main approaches in these scenarios:

    1. Create a daemon, service or background application that acts as your data repository. You can run it as an HTTP server / RPC server depending on your requirements. Your CLI application then interacts with this daemon as required;

    2. Implement a persistence mechanism that will allow data to be written and read across multiple CLI application executions. You may use normal text files, databases or even an implementation of golang's encoding/gob to write and read your slice (a map would probably be better) to and from a binary file.

    You can timestamp entries and simply remove them after their ttl expires by explicitly deleting them, or by simply not rewriting them during subsequent executions, according to the strategy / approach selected above.

    The scope and number of examples for such an open ended question is too myriad to post in a single answer and will most likely require multiple specific questions.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?



  • ¥15 docker部署Mongodb后输入命令报错?
  • ¥15 将下列流程图转变成python程序代码
  • ¥15 我需要全国每个城市的最新小区名字等数据。
  • ¥15 开发一个小区生态的小程序
  • ¥15 如何解决Excel中dependent dropdown list 的问题
  • ¥15 MddBootstrapInitialize2失败
  • ¥15 LCD Flicker
  • ¥15 esp32在micropython环境下使用ssl/tls连接mqtt服务器出现以下报错Connected on发生意外错误: 5无法连接到 MQTT 代理,如何解决?
  • ¥15 关于#genesiscsheel#的问题,如何解决?
  • ¥15 Android aidl for hal