确保我的Go页面浏览量计数器没有被滥用

Question

I believe I have a found a very good and fast solution for efficiently counting page views:

Working example in go playground here: https://play.golang.org/p/q_mYEYLa1h

My idea is to push this to the database every X minutes, and after pushing a key then delete it from the page map.

My question now is, what would be the optimal way to ensure that this isn't abused? Ideally, I would only want to increase page count from the same person if there was a time interval of 2 hours since last visiting the page. As far as I know, it would be ideal to store and compare both IP and user agent (I don't want to rely on cookie/localstorage), but I'm not quite sure how to efficiently store and compare this information.

I'd likely get both the IP (req.Header.Get("x-forwarded-for")) and UserAgent (req.UserAgent()) from http.Request.

I was thinking making a visitor struct similar to my page struct that would look like this:

type visitor struct {
    mutex          sync.Mutex
    urlIPUAAndTime map[string]time
}

This way should make it possible to do something similar to before. However, imagine if the website had so many requests that there would be hundreds of millions of unique visitor maps being stored, and each of these could only be deleted after 2 (or more) hours. I therefore think this is not a good solution.

I guess it would be ideal/necessary to write to and read from some file, but not sure how this should be done efficiently. Help would be greatly appreciated

Answer 1

One of optimization ways is to add a Bloom filter before this map. Bloom filter is a probabilistic structure which can say one of these:

• this user is definitely new

• and this user possibly was here

This is a way to cut off computation on early stage. If many of your users are new then you save requests to database to check all of them. What if structure says "user is possibly non-unique"? Then you go the database and check it. Here's one more optimization: if you do not need very accurate information and can agree with mistake about several percent, you may use the sole bloom filter. I guess many large sites use this technique for estimation.