Deserialization fails with "Unable to read unknown data" after removing a field with Collection type from the class (Issue with backward compatibility)

[Malobika8]

Got KryoException: ClassNotFoundException(with Maps) /Encountered unregistered class ID(with Lists)
when I try to deserialize data after removing a field with Collection type from the class.

To Reproduce

With Map:

First, serialize the original object.

public static class TestClass {

   String value;
   Map<Integer, String> map; //Comment during deserialization
 }

public static void serialize() throws IOException {

    String someString = "some string";
    Map<Integer, String> map = new LinkedHashMap<>();
    map.put(1111, someString);
    map.put(2222, someString);

    TestClass testClass = new TestClass();
    testClass.value = someString;
    testClass.map = map;
    
    final File classSerialized =
        new File(TEST_RESOURCES_PATH + "/serializedObj.exj");

    //This internally calls [writeObject(Output output, Object object)](https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/Kryo.java#L620) and puts serialized data in a file
    writeObject(testClass, classSerialized, serializer);
}

Then, comment field map, and run

@Test
public void test_deserialize() {

    final File classSerialized =
        new File(TEST_RESOURCES_PATH + "/serializedObj.exj");

     //This reads from the serialized file and internally calls [readObject (Input input, Class<T> type)](https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/Kryo.java#L765)
    TestClass testClass = readObject(classSerialized, serializer);
    assertNotNull(testClass);
    assertEquals("some string", testClass.value);
}

Got

com.esotericsoftware.kryo.kryo5.KryoException: Unable to read unknown data, type: java.util.LinkedHashMap (TestClass#null)
at com.esotericsoftware.kryo.kryo5.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:153)
at com.esotericsoftware.kryo.kryo5.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:153) 
... 75 more
Caused by: java.lang.ClassNotFoundException: ��\ at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

With List:

First, serialize the original object.

public static class TestClass {

   String value;
   List<Long> id; //Comment during deserialization
 }

public static void serialize() throws IOException {
    String someString = "some string";
    List<Long> id = new ArrayList<>();
    list.add(1111L);
    list.add(2222L);

    TestClass testClass = new TestClass();
    testClass.value = someString;
    testClass.id = id;
    
    final File classSerialized =
        new File(TEST_RESOURCES_PATH + "/serializedObj.exj");

    //This internally calls [writeObject(Output output, Object object)](https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/Kryo.java#L620) and puts serialized data in a file
    writeObject(testClass, classSerialized, serializer);
  }

Then, comment field id, and run

@Test
public void test_deserialize() {

    final File classSerialized =
        new File(TEST_RESOURCES_PATH + "/serializedObj.exj");

    //This reads from the serialized file and internally calls [readObject (Input input, Class<T> type)](https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/Kryo.java#L765)
    TestClass testClass = readObject(classSerialized, serializer);
    assertNotNull(testClass);
    assertEquals("some string", testClass.value);
}

Got

com.esotericsoftware.kryo.kryo5.KryoException: Unable to read unknown data, type: java.util.ArrayList (TestClass#null)
 ... 70 more
Caused by: com.esotericsoftware.kryo.kryo5.KryoException: Encountered unregistered class ID: 2466

Additional context

When we try out the same with Kryo version 2.24.0, it works fine and deserializes properly. However, the same fails with Kryo version 5.3.0.

FYI, the config looks like this -

kryo.register(KryoWrapper.class);
kryo.setDefaultSerializer(CompatibleFieldSerializer.class);
kryo.setRegistrationRequired(false);
kryo.setReferences(true);
kryo.setOptimizedGenerics(true); 
kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new StdInstantiatorStrategy()));

I learned that the problem can be solved with config.setChunkedEncoding(true);
However, I noticed it doesn't throw any error even with non-passive changes like changing data type of a field. We want to avoid that.

Is there any other thing that can be done to solve the problem? Thank you in advance!

[theigl]

A similar problem was described in #834. I debugged the issue at the time, but was unable to come up with a solution.

I'd recommend that you use chunked encoding. Alternatively, you can debug the problem and create a PR if you find a solution/workaround. Unfortunately, I don't have the capacity to spend time on this issue in the near future.