doupang9614
2019-04-14 11:40
浏览 738

修改现有的Yaml文件并添加新数据和注释

I recently saw that the go yaml lib has new version (V3)

with the nodes capabilities (which in my opinion is a killer feature :) ) which can helps a lots with modifying yamls without changing the structure of the file

But since it is fairly new (from last week ) I didn't find some helpful docs and example for the context which I need (add new object/node and to keep the file structure the same without removing the comments etc)

what I need is to manipulate yaml file

for example

lets say I’ve this yaml file

version: 1
type: verbose
kind : bfr

# my list of applications
applications:
  - name: app1
    kind: nodejs
    path: app1
    exec:
      platforms: k8s
      builder: test

Now I got an json object (e.g. with app2) which I need to insert to the existing file

[

    {
        "comment: "Second app",
        "name": "app2",
        "kind": "golang",
        "path": "app2",
        "exec": {
            "platforms": "dockerh",
            "builder": "test"
        }
    }
]

and I need to add it to the yml file after the first application, (applications is array of application)

version: 1
type: verbose
kind : bfr

# my list of applications
applications:

#  First app
  - name: app1
    kind: nodejs
    path: app1
    exec:
      platforms: k8s
      builder: test

# Second app
  - name: app2
    kind: golang
    path: app2
    exec:
      platforms: dockerh
      builder: test

is it possible to add from the yaml file the new json object ? also remove existing

I also found this blog https://blog.ubuntu.com/2019/04/05/api-v3-of-the-yaml-package-for-go-is-available

This is the types which represent the object

type VTS struct {
    version string       `yaml:"version"`
    types   string       `yaml:"type"`
    kind    string       `yaml:"kind,omitempty"`
    apps    Applications `yaml:"applications,omitempty"`
}

type Applications []struct {
    Name string `yaml:"name,omitempty"`
    Kind string `yaml:"kind,omitempty"`
    Path string `yaml:"path,omitempty"`
    Exec struct {
        Platforms string `yaml:"platforms,omitempty"`
        Builder   string `yaml:"builder,omitempty"`
    } `yaml:"exec,omitempty"`
}

update

after testing the solution which is provided by wiil7200 I found 2 issues

I use at the end write it to file err = ioutil.WriteFile("output.yaml", b, 0644)

And the yaml output have 2 issue.

  1. The array of the application is starting from the comments, it should start from the name

  2. After the name entry the kind property and all others after are not aligned to the name

any idea how to solve those issue ? regard the comments issue, lets say I got it from other property and not from the json (if it make it more simpler)

version: 1
type: verbose
kind: bfr


# my list of applications
applications:
-   #  First app
name: app1
    kind: nodejs
    path: app1
    exec:
        platforms: k8s
        builder: test
-   # test 1
name: app2
    kind: golang
    path: app2
    exec:
        platform: dockerh
        builder: test
  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • douzhi7070 2019-04-17 18:28
    已采纳

    First, let Me Start off by saying using yaml.Node does not produce valid yaml when marshalled from a valid yaml, given by the following example. Probably should file an issue.

    package main
    
    import (
        "fmt"
        "log"
    
        "gopkg.in/yaml.v3"
    )
    
    var (
        sourceYaml = `version: 1
    type: verbose
    kind : bfr
    
    # my list of applications
    applications:
    
    #  First app
      - name: app1
        kind: nodejs
        path: app1
        exec:
          platforms: k8s
          builder: test
    `
    )
    
    func main() {
        t := yaml.Node{}
    
        err := yaml.Unmarshal([]byte(sourceYaml), &t)
        if err != nil {
            log.Fatalf("error: %v", err)
        }
    
        b, err := yaml.Marshal(&t)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(string(b))
    }
    

    Produces the following invalid yaml in go version go1.12.3 windows/amd64

    version: 1
    type: verbose
    kind: bfr
    
    
    # my list of applications
    applications:
    -   #  First app
    name: app1
        kind: nodejs
        path: app1
        exec:
            platforms: k8s
            builder: test
    

    Secondly, using a struct such as

    type VTS struct {
        Version string       `yaml:"version" json:"version"`
        Types   string       `yaml:"type" json:"type"`
        Kind    string       `yaml:"kind,omitempty" json:"kind,omitempty"`
        Apps    yaml.Node `yaml:"applications,omitempty" json:"applications,omitempty"`
    }
    

    From ubuntu's blog and the source documentation it made it seem that it would correctly identify fields within the struct that are nodes and build that tree separately, but that is not the case. When unmarshalled, it will give a correct node tree, but when remarshalled it will produce the following yaml with all of the fields that yaml.Node exposes. Sadly we cannot go this route, must find another way.

    version: "1"
    type: verbose
    kind: bfr
    applications:
        kind: 2
        style: 0
        tag: '!!seq'
        value: ""
        anchor: ""
        alias: null
        content:
        -   #  First app
    name: app1
            kind: nodejs
            path: app1
            exec:
                platforms: k8s
                builder: test
        headcomment: ""
        linecomment: ""
        footcomment: ""
        line: 9
        column: 3
    

    Overlooking the first issue and the marshal bug for yaml.Nodes in a struct (on gopkg.in/yaml.v3 v3.0.0-20190409140830-cdc409dda467) we can now go about manipulating the Nodes that the package exposes. Unfortunately, there is no abstraction that will add Nodes with ease, so uses might vary and identifying nodes can be a pain. Reflection might help here a bit, so I leave that as an exercise for you.

    You will find comment spew.Dumps that dump the entire node Tree in a nice format, this helped with debugging when adding Nodes to the source tree.

    You can certainly remove nodes as well, you will just need to identify which particular nodes that need to be removed. You just have to ensure that you remove the parent nodes if it were a map or sequence.

    package main
    
    import (
        "encoding/json"
        "fmt"
        "log"
    
        "gopkg.in/yaml.v3"
    )
    
    var (
        sourceYaml = `version: 1
    type: verbose
    kind : bfr
    
    # my list of applications
    applications:
    
    #  First app
      - name: app1
        kind: nodejs
        path: app1
        exec:
          platforms: k8s
          builder: test
    `
        modifyJsonSource = `
    [
    
        {
            "comment": "Second app",
            "name": "app2",
            "kind": "golang",
            "path": "app2",
            "exec": {
                "platforms": "dockerh",
                "builder": "test"
            }
        }
    ]
    `
    )
    
    // VTS Need to Make Fields Public otherwise unmarshalling will not fill in the unexported fields.
    type VTS struct {
        Version string       `yaml:"version" json:"version"`
        Types   string       `yaml:"type" json:"type"`
        Kind    string       `yaml:"kind,omitempty" json:"kind,omitempty"`
        Apps    Applications `yaml:"applications,omitempty" json:"applications,omitempty"`
    }
    
    type Applications []struct {
        Name string `yaml:"name,omitempty" json:"name,omitempty"`
        Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
        Path string `yaml:"path,omitempty" json:"path,omitempty"`
        Exec struct {
            Platforms string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
            Builder   string `yaml:"builder,omitempty" json:"builder,omitempty"`
        } `yaml:"exec,omitempty" json:"exec,omitempty"`
        Comment string `yaml:"comment,omitempty" json:"comment,omitempty"`
    }
    
    func main() {
        t := yaml.Node{}
    
        err := yaml.Unmarshal([]byte(sourceYaml), &t)
        if err != nil {
            log.Fatalf("error: %v", err)
        }
    
        // Look for the Map Node with the seq array of items
        applicationNode := iterateNode(&t, "applications")
    
        // spew.Dump(iterateNode(&t, "applications"))
    
        var addFromJson Applications
        err = json.Unmarshal([]byte(modifyJsonSource), &addFromJson)
        if err != nil {
            log.Fatalf("error: %v", err)
        }
    
        // Delete the Original Applications the following options:
        // applicationNode.Content = []*yaml.Node{}
        // deleteAllContents(applicationNode)
        deleteApplication(applicationNode, "name", "app1")
    
    
        for _, app := range addFromJson {
            // Build New Map Node for new sequences coming in from json
            mapNode := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
    
            // Build Name, Kind, and Path Nodes
            mapNode.Content = append(mapNode.Content, buildStringNodes("name", app.Name, app.Comment)...)
            mapNode.Content = append(mapNode.Content, buildStringNodes("kind", app.Kind, "")...)
            mapNode.Content = append(mapNode.Content, buildStringNodes("path", app.Path, "")...)
    
            // Build the Exec Nodes and the Platform and Builder Nodes within it
            keyMapNode, keyMapValuesNode := buildMapNodes("exec")
            keyMapValuesNode.Content = append(keyMapValuesNode.Content, buildStringNodes("platform", app.Exec.Platforms, "")...)
            keyMapValuesNode.Content = append(keyMapValuesNode.Content, buildStringNodes("builder", app.Exec.Builder, "")...)
    
            // Add to parent map Node
            mapNode.Content = append(mapNode.Content, keyMapNode, keyMapValuesNode)
    
            // Add to applications Node
            applicationNode.Content = append(applicationNode.Content, mapNode)
        }
        // spew.Dump(t)
        b, err := yaml.Marshal(&t)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(string(b))
    }
    
    // iterateNode will recursive look for the node following the identifier Node,
    // as go-yaml has a node for the key and the value itself
    // we want to manipulate the value Node
    func iterateNode(node *yaml.Node, identifier string) *yaml.Node {
        returnNode := false
        for _, n := range node.Content {
            if n.Value == identifier {
                returnNode = true
                continue
            }
            if returnNode {
                return n
            }
            if len(n.Content) > 0 {
                ac_node := iterateNode(n, identifier)
                if ac_node != nil {
                    return ac_node
                }
            }
        }
        return nil
    }
    
    // deleteAllContents will remove all the contents of a node
    // Mark sure to pass the correct node in otherwise bad things will happen
    func deleteAllContents(node *yaml.Node) {
        node.Content = []*yaml.Node{}
    }
    
    // deleteApplication expects that a sequence Node with all the applications are present
    // if the key value are not found it will not log any errors, and return silently
    // this is expecting a map like structure for the applications
    func deleteApplication(node *yaml.Node, key, value string) {
        state := -1
        indexRemove := -1
        for index, parentNode := range node.Content {
            for _, childNode := range parentNode.Content {
                if key == childNode.Value && state == -1 {
                    state += 1
                    continue // found expected move onto next
                }
                if value == childNode.Value && state == 0 {
                    state += 1
                    indexRemove = index
                    break // found the target exit out of the loop
                } else if state == 0 {
                    state = -1
                }
            }
        }
        if state == 1 {
            // Remove node from contents
            // node.Content = append(node.Content[:indexRemove], node.Content[indexRemove+1:]...)
            // Don't Do this you might have a potential memory leak source: https://github.com/golang/go/wiki/SliceTricks
            // Since the underlying nodes are pointers
            length := len(node.Content)
            copy(node.Content[indexRemove:], node.Content[indexRemove+1:])
            node.Content[length-1] = nil
            node.Content = node.Content[:length-1]
        }
    }
    
    
    // buildStringNodes builds Nodes for a single key: value instance
    func buildStringNodes(key, value, comment string) []*yaml.Node {
        keyNode := &yaml.Node{
            Kind:        yaml.ScalarNode,
            Tag:         "!!str",
            Value:       key,
            HeadComment: comment,
        }
        valueNode := &yaml.Node{
            Kind:  yaml.ScalarNode,
            Tag:   "!!str",
            Value: value,
        }
        return []*yaml.Node{keyNode, valueNode}
    }
    
    // buildMapNodes builds Nodes for a key: map instance
    func buildMapNodes(key string) (*yaml.Node, *yaml.Node) {
        n1, n2 := &yaml.Node{
            Kind:  yaml.ScalarNode,
            Tag:   "!!str",
            Value: key,
        }, &yaml.Node{Kind: yaml.MappingNode,
            Tag: "!!map",
        }
        return n1, n2
    }
    
    

    Produces yaml

    version: 1
    type: verbose
    kind: bfr
    
    
    # my list of applications
    applications:
    -   #  First app
    name: app1
        kind: nodejs
        path: app1
        exec:
            platforms: k8s
            builder: test
    -   # Second app
    name: app2
        kind: golang
        path: app2
        exec:
            platform: dockerh
            builder: test
    
    点赞 评论

相关推荐 更多相似问题