weixin_39925959
weixin_39925959
2021-01-12 06:11

CompatibleFieldSerializer deserialize field problem where default value not null

the kryo version i used was 4.0.1 now, i found a problem where i use CompatibleFieldSerializer. i serialize an object A which has 4 fields fa="a", fb="", fc="c", fd="" and save the content in a temp file, then i comment the field fb of A and i begin to deserialize the object from the temp file, but some of the field value is wrong, the deserialize result is fa="a", fc="c", fd="c", code example:

public class TestNum implements Serializable {

private static final long serialVersionUID = -5733172112236368310L;

private String fa;

private String fb;

private String fc;

private String fd;

public String getFa() {
    return fa;
}

public void setFa(String fa) {
    this.fa = fa;
}

public String getFb() {
    return fb;
}

public void setFb(String fb) {
    this.fb = fb;
}

public String getFc() {
    return fc;
}

public void setFc(String fc) {
    this.fc = fc;
}

public String getFd() {
    return fd;
}

public void setFd(String fd) {
    this.fd = fd;
}

}

public class CompatableTest { static String file = "D://test123.bin";

public static void serialize() throws IOException {
    TestNum tn = new TestNum();
    tn.setFa("中文测试");
    tn.setFb("");
    tn.setFc("c");
    tn.setFd("");
    System.out.println("before :: " + JsonKit.object2Json(tn));
    ByteBuf byteBuf = Unpooled.buffer();
    KryoSerializer.serialize(tn, byteBuf);
    ByteBuffer nioBuffer = byteBuf.nioBuffer();
    nioBuffer.flip();
    byte[] array = nioBuffer.array();
    File f = new File(file);
    FileUtils.writeByteArrayToFile(f, array);
}

public static void desialize() throws IOException {
    File f = new File(file);
    byte[] array = FileUtils.readFileToByteArray(f);
    ByteBuf buf = Unpooled.wrappedBuffer(array, 4, array.length - 4);
    System.out.println("desialize::::" + buf.nioBuffer());
    Object result = KryoSerializer.deserialize(buf);
    System.out.println("after :: " + JsonKit.object2Json(result));
}

public static void main(String[] args) throws IOException {
    serialize();
    desialize();
}

}

when run the code once, comment the field fb of TestNum and just run desialize(), the problem would come! how can i resolve the problem

该提问来源于开源项目:EsotericSoftware/kryo

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

6条回答

  • weixin_39531178 weixin_39531178 3月前

    Your example does not compile. Eg, Unpooled, JsonKit, etc are not part of Kryo or the JRE. If you submit example code that shows the problem, we will reopen this issue. Please do not create a new issue for the same problem.

    Have you tried it in Kryo 5?

    点赞 评论 复制链接分享
  • weixin_39925959 weixin_39925959 3月前

    sorry, i tried kryo 5, it is ok

    点赞 评论 复制链接分享
  • weixin_39925959 weixin_39925959 3月前

    i feel so bad because kryo 5 and kryo 4 is not compatible unless i add the config chuncked = true and readUnknownTagData = false, but in the situation the problem came again(5.0.0-RC1),the first time i run the code is:

    public class CompatableTest {
        static String file = "D://testbean.bin";
    
        private static final ThreadLocal<Kryo> kryoLocal = new ThreadLocal<Kryo>() {
            ("rawtypes")
            protected Kryo initialValue() {
                Kryo kryo = new Kryo();
                kryo.setRegistrationRequired(false);
                kryo.setDefaultSerializer(new SerializerFactory<CompatibleFieldSerializer>() {
    
    
                    public CompatibleFieldSerializer<?> newSerializer(Kryo kryo, Class type) {
                        CompatibleFieldSerializerConfig config = new CompatibleFieldSerializerConfig();
                        config.setChunkedEncoding(true);
                        config.setReadUnknownTagData(false);
                        CompatibleFieldSerializer<?> serializer = new CompatibleFieldSerializer(kryo, type, config);
                        return serializer;
                    }
    
    
                    public boolean isSupported(Class type) {
                        return true;
                    }
    
                });
                kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new SerializingInstantiatorStrategy()));
                return kryo;
            };
        };
    
        public static void serialize() throws IOException {
            TestBean tn = new TestBean();
            tn.setFa("中文测试");
            tn.setFb("");
            tn.setFc("c");
            tn.setFd("");
            System.out.println("serialize :: " + tn);
            Output out = new Output(new FileOutputStream(file));
            kryoLocal.get().writeClassAndObject(out, tn);
            out.flush();
            out.close();
        }
    
        public static void desialize() throws IOException {
            Input input = new Input(new FileInputStream(file));
            Object obj = kryoLocal.get().readClassAndObject(input);
            input.close();
            System.out.println("desialize :: " + obj);
        }
    
        public static void main(String[] args) throws IOException {
            serialize();
            desialize();
        }
    }
    
    public class TestBean implements Serializable {
    
        private static final long serialVersionUID = -5733172112236368310L;
    
        private String fa;
    
        private String fb;
    
        private String fc;
    
        private String fd;
    
        public String getFa() {
            return fa;
        }
    
        public void setFa(String fa) {
            this.fa = fa;
        }
    
        public String getFb() {
            return fb;
        }
    
        public void setFb(String fb) {
            this.fb = fb;
        }
    
        public String getFc() {
            return fc;
        }
    
        public void setFc(String fc) {
            this.fc = fc;
        }
    
        public String getFd() {
            return fd;
        }
    
        public void setFd(String fd) {
            this.fd = fd;
        }
    
        public String toString() {
            return "TestBean[fa=" + fa + ",fb=" + fb + ",fc=" + fc + ",fd=" + fd + "]";
        }
    
    }
    

    it can serilaize and deserialize correcttly and it prints: serialize :: TestBean[fa=中文测试,fb=,fc=c,fd=] desialize :: TestBean[fa=中文测试,fb=,fc=c,fd=]

    then i removed the method serialize() and the fb field of TestBean,the code i run second time is:

    public class CompatableTest {
        static String file = "D://testbean.bin";
    
        private static final ThreadLocal<Kryo> kryoLocal = new ThreadLocal<Kryo>() {
            ("rawtypes")
            protected Kryo initialValue() {
                Kryo kryo = new Kryo();
                kryo.setRegistrationRequired(false);
                kryo.setDefaultSerializer(new SerializerFactory<CompatibleFieldSerializer>() {
    
    
                    public CompatibleFieldSerializer<?> newSerializer(Kryo kryo, Class type) {
                        CompatibleFieldSerializerConfig config = new CompatibleFieldSerializerConfig();
                        config.setChunkedEncoding(true);
                        config.setReadUnknownTagData(false);
                        CompatibleFieldSerializer<?> serializer = new CompatibleFieldSerializer(kryo, type, config);
                        return serializer;
                    }
    
    
                    public boolean isSupported(Class type) {
                        return true;
                    }
    
                });
                kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new SerializingInstantiatorStrategy()));
                return kryo;
            };
        };
    
        public static void serialize() throws IOException {
            TestBean tn = new TestBean();
            tn.setFa("中文测试");
            tn.setFc("c");
            tn.setFd("");
            System.out.println("serialize :: " + tn);
            Output out = new Output(new FileOutputStream(file));
            kryoLocal.get().writeClassAndObject(out, tn);
            out.flush();
            out.close();
        }
    
        public static void desialize() throws IOException {
            Input input = new Input(new FileInputStream(file));
            Object obj = kryoLocal.get().readClassAndObject(input);
            input.close();
            System.out.println("desialize :: " + obj);
        }
    
        public static void main(String[] args) throws IOException {
            desialize();
        }
    }
    
    public class TestBean implements Serializable {
    
        private static final long serialVersionUID = -5733172112236368310L;
    
        private String fa;
    
        private String fc;
    
        private String fd;
    
        public String getFa() {
            return fa;
        }
    
        public void setFa(String fa) {
            this.fa = fa;
        }
    
        public String getFc() {
            return fc;
        }
    
        public void setFc(String fc) {
            this.fc = fc;
        }
    
        public String getFd() {
            return fd;
        }
    
        public void setFd(String fd) {
            this.fd = fd;
        }
    
        public String toString() {
             return "TestBean[fa=" + fa + ",fc=" + fc + ",fd=" + fd + "]";
        }
    
    }
    

    the problem happened, it prints: desialize :: TestBean[fa=中文测试,fc=c,fd=c] the value of fd field was wrong assigned, could you tell me how can i resolve the problem?

    点赞 评论 复制链接分享
  • weixin_39531178 weixin_39531178 3月前

    I simplified your code slightly: http://n4te.com/x/4633-EQod.txt

    Look at the trace log (using registration to reduce noise, and I added line breaks between fields):

    
    00:00 TRACE: [kryo] Write class 9: com.esotericsoftware.kryo.CompatableTest$TestBean [0]
    00:00 TRACE: [kryo] Write: <not null> [1]
    00:00 TRACE: [kryo] Write initial reference 0: TestBean[a=a, c=c, d=] (com.esotericsoftware.kryo.CompatableTest$TestBean) [2]
    00:00 DEBUG: [kryo] Write: TestBean[a=a, c=c, d=] (com.esotericsoftware.kryo.CompatableTest$TestBean) [2]
    00:00 TRACE: [kryo] Write fields for class: com.esotericsoftware.kryo.CompatableTest$TestBean
    00:00 TRACE: [kryo] Write field name: a [3]
    00:00 TRACE: [kryo] Write field name: b [5]
    00:00 TRACE: [kryo] Write field name: c [7]
    00:00 TRACE: [kryo] Write field name: d [9]
    
    00:00 TRACE: [kryo] Write field String: a (com.esotericsoftware.kryo.CompatableTest$TestBean) [11]
    00:00 TRACE: [kryo] Write class 1: String [0]
    00:00 TRACE: [kryo] Write: <not null> [1]
    00:00 TRACE: [kryo] Write initial reference 1: a (String) [2]
    00:00 TRACE: [kryo] Write: a (String) [2]
    00:00 TRACE: [kryo] Write chunk: 4 [4]
    00:00 TRACE: [kryo] End chunk.
    
    00:00 TRACE: [kryo] Write field String: b (com.esotericsoftware.kryo.CompatableTest$TestBean) [1]
    00:00 TRACE: [kryo] Write class 1: String [0]
    00:00 TRACE: [kryo] Write: <not null> [1]
    00:00 TRACE: [kryo] Write initial reference 2:  (String) [2]
    00:00 TRACE: [kryo] Write:  (String) [2]
    00:00 TRACE: [kryo] Write chunk: 3 [3]
    00:00 TRACE: [kryo] End chunk.
    
    00:00 TRACE: [kryo] Write field String: c (com.esotericsoftware.kryo.CompatableTest$TestBean) [1]
    00:00 TRACE: [kryo] Write class 1: String [0]
    00:00 TRACE: [kryo] Write: <not null> [1]
    00:00 TRACE: [kryo] Write initial reference 3: c (String) [2]
    00:00 TRACE: [kryo] Write: c (String) [2]
    00:00 TRACE: [kryo] Write chunk: 4 [4]
    00:00 TRACE: [kryo] End chunk.
    
    00:00 TRACE: [kryo] Write field String: d (com.esotericsoftware.kryo.CompatableTest$TestBean) [1]
    00:00 TRACE: [kryo] Write class 1: String [0]
    00:00 DEBUG: [kryo] Write reference 2:  (String) [1]
    00:00 TRACE: [kryo] Write chunk: 2 [2]
    00:00 TRACE: [kryo] End chunk.
    </not></not></not></not>

    The value (an empty string) for both fields b and d is written as reference 2. Now the trace log for deserializing after removing field b using setReadUnknownTagData(true):

    
    00:00 TRACE: [kryo] Read class 9: com.esotericsoftware.kryo.CompatableTest$TestBean [1]
    00:00 TRACE: [kryo] Read: <not null> [2]
    00:00 TRACE: [kryo] Read initial reference 0: com.esotericsoftware.kryo.CompatableTest$TestBean [2]
    00:00 TRACE: [kryo] Read fields for class: com.esotericsoftware.kryo.CompatableTest$TestBean
    00:00 TRACE: [kryo] Read field name: a
    00:00 TRACE: [kryo] Read field name: b
    00:00 TRACE: [kryo] Read field name: c
    00:00 TRACE: [kryo] Read field name: d
    00:00 TRACE: [kryo] Unknown field will be skipped: b
    
    00:00 TRACE: [kryo] Read chunk: 4
    00:00 TRACE: [kryo] Read class 1: String [1]
    00:00 TRACE: [kryo] Read field String: a (com.esotericsoftware.kryo.CompatableTest$TestBean) [17]
    00:00 TRACE: [kryo] Read: <not null> [2]
    00:00 TRACE: [kryo] Read initial reference 1: String [2]
    00:00 TRACE: [kryo] Read: a (String) [4]
    00:00 TRACE: [kryo] Next chunk.
    
    00:00 TRACE: [kryo] Read chunk: 3
    00:00 TRACE: [kryo] Read class 1: String [1]
    00:00 TRACE: [kryo] Read unknown data, type: String [22]
    00:00 TRACE: [kryo] Read: <not null> [2]
    00:00 TRACE: [kryo] Read initial reference 2: String [2]
    00:00 TRACE: [kryo] Read:  (String) [3]
    00:00 TRACE: [kryo] Next chunk.
    
    00:00 TRACE: [kryo] Read chunk: 4
    00:00 TRACE: [kryo] Read class 1: String [1]
    00:00 TRACE: [kryo] Read field String: c (com.esotericsoftware.kryo.CompatableTest$TestBean) [28]
    00:00 TRACE: [kryo] Read: <not null> [2]
    00:00 TRACE: [kryo] Read initial reference 3: String [2]
    00:00 TRACE: [kryo] Read: c (String) [4]
    00:00 TRACE: [kryo] Next chunk.
    
    00:00 TRACE: [kryo] Read chunk: 2
    00:00 TRACE: [kryo] Read class 1: String [1]
    00:00 TRACE: [kryo] Read field String: d (com.esotericsoftware.kryo.CompatableTest$TestBean) [32]
    00:00 DEBUG: [kryo] Read reference 2:  (String) [2]
    00:00 TRACE: [kryo] Next chunk.
    
    00:00 DEBUG: [kryo] Read: TestBean[a=a, c=c, d=] (com.esotericsoftware.kryo.CompatableTest$TestBean) [32]
    </not></not></not></not>

    Where you find Read unknown data is where field b is being read. When reading the data for b it reads the empty string as reference 2 and stores it for later. Later, field d is read as reference 2 and gets the empty string.

    You need to use setReadUnknownTagData(true) if you have references enabled. If it is false then field b is skipped and the empty string is not read as reference 2. Instead, the value for field c will be read as reference 2, which is wrong.

    点赞 评论 复制链接分享
  • weixin_39925959 weixin_39925959 3月前

    thank you very much ,but the problem i encountered now is kryo 5 is not compatible with kryo 4.x, it means kryo5 can not deserialize the byte stream which was serialized by kryo4.x if setReadUnknownTagData(true),can i keep the compatiblity between kryo5 and kry4.x through special config or what others?

    点赞 评论 复制链接分享
  • weixin_39531178 weixin_39531178 3月前

    Sorry, there is no compatibility between Kryo 4 and 5. There are some ideas here, but updating Kryo is not terribly easy.

    点赞 评论 复制链接分享