如何将复杂的JSON映射到其他JSON

Question

I am trying to build aggregation services, to all third party APIs that's I used, this aggregation services taking json values coming from my main system and it will put this value to key equivalent to third party api key then, aggregation services it will send request to third party api with new json format.

example-1:

package main

import (
    "encoding/json"
    "fmt"
    "log"

    "github.com/tidwall/gjson"
)

func main() {
    // mapping JSON
    mapB := []byte(`
    {
        "date": "createdAt",
        "clientName": "data.user.name"
    }
    `)

    // from my main system
    dataB := []byte(`
    {
        "createdAt": "2017-05-17T08:52:36.024Z",
        "data": {
            "user": {
                "name": "xxx"
            }
        }
    }
    `)

    mapJSON := make(map[string]interface{})
    dataJSON := make(map[string]interface{})
    newJSON := make(map[string]interface{})

    err := json.Unmarshal(mapB, &mapJSON)
    if err != nil {
        log.Panic(err)
    }

    err = json.Unmarshal(dataB, &dataJSON)
    if err != nil {
        log.Panic(err)
    }

    for i := range mapJSON {
        r := gjson.GetBytes(dataB, mapJSON[i].(string))
        newJSON[i] = r.Value()
    }

    newB, err := json.MarshalIndent(newJSON, "", "  ")
    if err != nil {
        log.Println(err)
    }

    fmt.Println(string(newB))
}

output:

{
  "clientName": "xxx",
  "date": "2017-05-17T08:52:36.024Z"
}

I use gjson package to get values form my main system request in simple way from a json document.

example-2:

import (
    "encoding/json"
    "fmt"
    "log"

    "github.com/tidwall/gjson"
)

func main() {
    // mapping JSON
    mapB := []byte(`
    {
        "date": "createdAt",
        "clientName": "data.user.name",
        "server":{
            "google":{
                "date" :"createdAt"
            }
        }
    }
    `)

    // from my main system
    dataB := []byte(`
    {
        "createdAt": "2017-05-17T08:52:36.024Z",
        "data": {
            "user": {
                "name": "xxx"
            }
        }
    }
    `)

    mapJSON := make(map[string]interface{})
    dataJSON := make(map[string]interface{})
    newJSON := make(map[string]interface{})

    err := json.Unmarshal(mapB, &mapJSON)
    if err != nil {
        log.Panic(err)
    }

    err = json.Unmarshal(dataB, &dataJSON)
    if err != nil {
        log.Panic(err)
    }

    for i := range mapJSON {
        r := gjson.GetBytes(dataB, mapJSON[i].(string))
        newJSON[i] = r.Value()
    }

    newB, err := json.MarshalIndent(newJSON, "", "  ")
    if err != nil {
        log.Println(err)
    }

    fmt.Println(string(newB))
}

output:

panic: interface conversion: interface {} is map[string]interface {}, not string

I can handle this error by using https://golang.org/ref/spec#Type_assertions, but what if this json object have array and inside this array have json object ....

my problem is I have different apis, every api have own json schema, and my way for mapping json only work if third party api have json key value only, without nested json or array inside this array json object.

is there a way to mapping complex json schema, or golang package to help me to do that?

Answer 1

EDIT:

After comment interaction and with updated question. Before we move forward, I would like to mention.

I just looked at your example-2 Remember one thing. Mapping is from one form to another form. Basically one known format to targeted format. Each data type have to handled. You cannot do generic to generic mapping logically (technically feasible though, would take more time & efforts, you can play around on this).

I have created sample working program of one approach; it does a mapping of source to targeted format. Refer this program as a start point and use your creativity to implement yours.

Playground link: https://play.golang.org/p/MEk_nGcPjZ

Explanation: Sample program achieves two different source format to one target format. The program consist of -

• Targeted Mapping definition of Provider 1
• Targeted Mapping definition of Provider 2
• Provider 1 JSON
• Provider 2 JSON
• Mapping function
• Targeted JSON marshal

Key elements from program: refer play link for complete program.

type MappingInfo struct {
    TargetKey     string
    SourceKeyPath string
    DataType      string
}

Map function:

func mapIt(mapping []*MappingInfo, parsedResult gjson.Result) map[string]interface{} {
    mappedData := make(map[string]interface{})
    for _, m := range mapping {
        switch m.DataType {
        case "time":
            mappedData[m.TargetKey] = parsedResult.Get(m.SourceKeyPath).Time()
        case "string":
            mappedData[m.TargetKey] = parsedResult.Get(m.SourceKeyPath).String()
        }
    }
    return mappedData
}

Output:

Provider 1 Result: map[date:2017-05-17 08:52:36.024 +0000 UTC clientName:provider1 username]
Provider 1 JSON: {
  "clientName": "provider1 username",
  "date": "2017-05-17T08:52:36.024Z"
}

Provider 2 Result: map[date:2017-05-12 06:32:46.014 +0000 UTC clientName:provider2 username]
Provider 2 JSON: {
  "clientName": "provider2 username",
  "date": "2017-05-12T06:32:46.014Z"
}

Good luck, happy coding!

---

Typically Converting/Transforming one structure to another structure, you will have to handle this with application logic.

As you mentioned in the question:

my problem is I have different apis, every api have own json schema

This is true for every aggregation system.

---

One approach to handle this requirement effectively; is to keep mapping of keys for each provider JSON structure and targeted JSON structure.

For example: This is an approach, please go with your design as you see fit.

JSON structures from various provider:

// Provider 1 : JSON structrure
{
  "createdAt": "2017-05-17T08:52:36.024Z",
  "data": {
    "user": {
      "name": "xxx"
    }
  }
}

// Provider 2 : JSON structrure
{
  "username": "yyy"
  "since": "2017-05-17T08:52:36.024Z",
}

Mapping for target JSON structure:

jsonMappingByProvider := make(map[string]string)

// Targeted Mapping for Provider 1
jsonMappingByProvider["provider1"] = `
{
    "date": "createdAt",
    "clientName": "data.user.name"
}
`

// Targeted Mapping for Provider 2
jsonMappingByProvider["provider2"] = `
{
    "date": "since",
    "clientName": "username"
}
`

Now, based the on the provider you're handling, get the mapping and map the response JSON into targeted structure.

// get the mapping info by provider
mapping := jsonMappingByProvider["provider1"]

// Parse the response JSON 
// Do the mapping

This way you can control each provider and it's mapping effectively.