Create @shopify/logger

Question

Problem Summary

Logging is an important tool in a developer’s arsenal. It can be used to debug applications both in development and production to better diagnose problems. It can be used, along with statistical analysis, to gain insight into the activity of even the most complex distributed systems.

As Shopify becomes increasingly distributed and complex, it will become imperative that logging is something that we do right.

We eventually want to build a great experience with a nice API, filtering, and a slick UI. By getting the API right and creating a nice separation of concerns, we allow for ease of continuous improvement for things like filtering, etc.

In development, for web application, there are concerns around simultaneous, asynchronous requests. We want immediate feedback, but, when many requests are happening at the same time, the UI is hard to get right.

In production web applications, stdout is our interface into splunk (via fluentd). We want our logs to have context about our application, and perhaps we want to aggregate these logs, so that we send only a single JSON payload to splunk per request.

For CLI applications, we’d like to avoid duplicating work in creating a nice UI for logging. It would be nice if our logger was not specifically tied to server applications.

Proposed Solution

• Logging is split into 2 stages:
  • Logging (API emits intermediate data)
  • Formatting (converts intermediate data into desired format)
• We use JSON as the intermediate logging format (pino / bunyan / bole)
• The core logging API will stay as close as possible to https://console.spec.whatwg.org
  • If API exists, we’ll use it:
    • .time, .timeEnd
  • If API doesn’t exist, we’ll use idiomatic naming:
    • .tag, .tagEnd
• Buffering for production is handled by /koa-logger

Why?

• By splitting logging into 2 stages, we conceptually decouple the concept of an API from the concept of a logging UI. This allows us to focus on perfecting the 2 independently of each other.
• By logging to a JSON format, we can easily create aggregation or filtering logic that gets applied before the logs get formatted. Again, we can do all of this independently of the logger API mechanism.
• By staying close to the console specification for our API, we reduce friction for new developers into our code base. We also defer to the W3C arguments or differences of opinion for how things like tagging, grouping, or timing should be done. Sometimes it’s nice to have a decision made for you in a consistent way, even if it’s not your favorite outcome.

References

Inspiration was taken from many sources, but the primary influences were: http://getpino.io/#/ https://github.com/klauscfhq/signale https://console.spec.whatwg.org/

该提问来源于开源项目：Shopify/quilt

Answer 1

On separation of logging and formatting

There are several possible solutions here: * piping the output of a script to a formatting script * outputting the logs to a custom stream * the logger exposes an observable stream, and the formatter can listen and attach itself to this stream * same as above, but using an event-driven approach, rather than observables * the formatter exposes a function to receive input, and the logger gets passed a formatter on initialization

Using I/O streams seems overly complex for this scenario., whether in the code or via an OS pipe. Since we'll need our production logging in koa to accumulate logs into a single JSON object per request, I think it makes sense for a Formatter to be an object that can maintain state. Furthermore, I feel that the formatter should be passed into the Logger in order to reduce glue code.