duan19911992 2019-06-04 15:16
浏览 98

了解响应时间成本

I'm new in Golang and trying to write simple web-server with json response. One of endpoints is for database access. With static-like response (such as any static string or simple data conversion) average response time is about 0s (<1ms). If I add some database request, average time increases up to 250ms and not depending on function content. So I'm trying to understand the timing of this function.

What I've already know:

  1. Query to db costs about 40-50ms
  2. Datetime transformation costs >1ms
  3. json.Marshal costs <1ms
  4. string(json.Marshal) <1ms

All my problem is in rows.Next():

  1. all inside rows.Next() costs <1ms separately
  2. Each row.Next() procedure cost depends on row length and it is about 10-50ms.

If I understand db.Query() gets all rows in one request and keeps it in memory providing virtual cursor as iterator ( .Next()). Why does it cost so much?

  1. All row.Next() data processing costs about 180ms. With 40ms db.Query together it approx equals to total response time (~240ms)
  2. If I comment all row.Next() cycle and just make db.Query total response still costs 200-240ms

Packages I use:

import (
    "database/sql"
    "encoding/json"
    "fmt"
    "log"
    "os"
    "strconv"
    "time"
    "github.com/kshvakov/clickhouse"
    routing "github.com/qiangxue/fasthttp-routing"
    "github.com/valyala/fasthttp"
)
func getRecs(client string, dt string) string {
    dtT := transformDate(&dt)
    rows, err := db.Query(getRecsRequest(&client, &dtT))

    if err != nil {
        log.Fatal(err)
    }
    defer rows.Close()

    start := time.Now()

    got := []databaseRows{}
    elapsed := time.Since(start)
    for rows.Next() {
        elapsed = time.Since(start)
        log.Printf("Next took %s", elapsed)
        var r databaseRows
        err := rows.Scan(&r.ItemIds, &r.DtRecommendation, &r.Number)
        if err != nil {
            log.Fatal(err)
        }
        got = append(got, r)
        start = time.Now()
    }

    jsonData, err := json.Marshal(&got)
    return string(jsonData)
}
func getRecs(client string, dt string) string {
    dtT := transformDate(&dt)
    rows, err := db.Query(getRecsRequest(&client, &dtT))

    if err != nil {
        log.Fatal(err)
    }
    defer rows.Close()

    // start := time.Now()

    got := "[]databaseRows{}"
    // elapsed := time.Since(start)
    // for rows.Next() {
    //  elapsed = time.Since(start)
    //  log.Printf("Next took %s", elapsed)
    //  var r databaseRows
    //  err := rows.Scan(&r.ItemIds, &r.DtRecommendation, &r.Number)
    //  if err != nil {
    //      log.Fatal(err)
    //  }
    //  got = append(got, r)
    //  start = time.Now()
    // }

    jsonData, err := json.Marshal(&got)
    return string(jsonData)
}

Can I expect web-server response time is about database request time?

UPDATE.

I've tried "github.com/jmoiron/sqlx" with "github.com/kshvakov/clickhouse" and result was the same; only syntax changed.

I also tried "github.com/mailru/go-clickhouse" with "database/sql". It provides access to HTTP interface to clickhouse. I got all rows in one requets as expected, but it tooks about 240ms for the http request and about 0ms for data processing.. So I am looking for opportunity to keep tcp connection opened to be able get all nessasery rows in one request and work with my program side memory buffer (with some buffer limit in combnation with tcp/socket buffer).

UPDATE 2.

Ok, I've tried native clickhouse client (and other such DBeaver) and approx 250ms is the limit for this request. But it may be only clickhouse native tcp interface limitation. But what is the 40-50ms db.Query() operation, which returns no rows. Is the time for query on server side? Or just query string check?

  • 写回答

0条回答 默认 最新

    报告相同问题?

    悬赏问题

    • ¥15 C++使用Gunplot
    • ¥15 这个电路是如何实现路灯控制器的,原理是什么,怎么求解灯亮起后熄灭的时间如图?
    • ¥15 matlab数字图像处理频率域滤波
    • ¥15 在abaqus做了二维正交切削模型,给刀具添加了超声振动条件后输出切削力为什么比普通切削增大这么多
    • ¥15 ELGamal和paillier计算效率谁快?
    • ¥15 file converter 转换格式失败 报错 Error marking filters as finished,如何解决?
    • ¥15 Arcgis相交分析无法绘制一个或多个图形
    • ¥15 关于#r语言#的问题:差异分析前数据准备,报错Error in data[, sampleName1] : subscript out of bounds请问怎么解决呀以下是全部代码:
    • ¥15 seatunnel-web使用SQL组件时候后台报错,无法找到表格
    • ¥15 fpga自动售货机数码管(相关搜索:数字时钟)