Gson deserialization: How to distinguish between fields that are missing and fields that are explicitly set to null?

Question

I'm trying to implement JSON Merge Patch for a Java (JAX-RS) webservice I'm building.

The gist is that partial updates of a record are done by sending a JSON document to the server that only contains the fields that should be changed.

Given this record

{
  "a": "b",
  "c": {
    "d": "e",
    "f": "g"
  }
}

, the following JSON update document

{
  "a":"z",
  "c": {
    "f": null
  }
}

should set a new value for "a" and delete "f" inside "c".

The latter is the problem. I don't know how I can distinguish between an input where f is missing and an input where f is null. Both, as far as I can tell, would be deserialized to null in the target Java Object.

What do?

no good way, xml world has xsi:nil="true", in java if something is null, you dont know if it was initialized to null or someone set it as null, you could define enum if possible and assign empty value — Kalpesh Soni, Nov 16 '15 at 14:15
I was considering writing my own deserializer for Java 8's [Optional](http://www.oracle.com/technetwork/articles/java/java8-optional-2175753.html). — Antares42, Nov 16 '15 at 14:26
Hm. I tried that, but got stuck fairly quickly due to Java's type erasure. I can't implement a ``Serializer`` (or deserializer) for ``Optional`` because the signature doesn't match, and I can't implement a ``TypeAdapterFactory`` because I can't get hold of the type parameter of ``Optional`` at runtime. Very, very frustrating. — Antares42, Nov 16 '15 at 19:56
I also tried wrapping ``Optional`` in a class that explicitly remembers its original type by requiring it as a constructor parameter. This might work, but the code to initialize (or retrieve) a simple string becomes horribly long and twisted. — Antares42, Nov 16 '15 at 20:00

score 4 · Accepted Answer · edited May 23 '17 at 12:00

I acknowledge mlk's answer, but given that I already have (and would nonetheless need) a POJO representation of the JSON object, I feel mapping automatically is still better than looking up manually.

The challenge with that is that, as I said, both missing and explicit null values are set to null in the corresponding POJO that gson.fromJson(...) would populate. (Unlike e.g. R's NULL and NA, Java only has one representation for "not there".)

However, by modelling my data structure using Java 8's Optionals I can do just that: Distinguish between something that is not set, and something that is set to null. Here's what I ended up with:

1) I replaced all fields in my data objects with Optional<T>.

public class BasicObjectOptional {

    private Optional<String> someKey;
    private Optional<Integer> someNumber;
    private Optional<String> mayBeNull;

    public BasicObjectOptional() {
    }

    public BasicObjectOptional(boolean initialize) {
        if (initialize) {
            someKey = Optional.ofNullable("someValue");
            someNumber = Optional.ofNullable(42);
            mayBeNull = Optional.ofNullable(null);
        }
    }

    @Override
    public String toString() {
        return String.format("someKey = %s, someNumber = %s, mayBeNull = %s",
                                            someKey, someNumber, mayBeNull);
    }

}

Or a nested one:

public class ComplexObjectOptional {

    Optional<String> theTitle;  
    Optional<List<Optional<String>>> stringArray;
    Optional<BasicObjectOptional> theObject;

    public ComplexObjectOptional() {
    }

    public ComplexObjectOptional(boolean initialize) {
        if (initialize) {
            theTitle = Optional.ofNullable("Complex Object");   
            stringArray =    Optional.ofNullable(Arrays.asList(Optional.ofNullable("Hello"),Optional.ofNullable("World")));
            theObject = Optional.ofNullable(new BasicObjectOptional(true));
        }
    }

    @Override
    public String toString() {
        return String.format("theTitle = %s, stringArray = %s, theObject = (%s)", theTitle, stringArray, theObject);
    }   
}

2) Implemented a serializer and deserializer based on this useful SO answer.

public class OptionalTypeAdapter<E> extends TypeAdapter<Optional<E>> {

    public static final TypeAdapterFactory FACTORY = new TypeAdapterFactory() {

        //@Override
        public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
            Class<T> rawType = (Class<T>) type.getRawType();
            if (rawType != Optional.class) {
                return null;
            }
            final ParameterizedType parameterizedType = (ParameterizedType) type.getType();
            final Type actualType = parameterizedType.getActualTypeArguments()[0];
            final TypeAdapter<?> adapter = gson.getAdapter(TypeToken.get(actualType));
            return new OptionalTypeAdapter(adapter);
        }
    };
    private final TypeAdapter<E> adapter;

    public OptionalTypeAdapter(TypeAdapter<E> adapter) {
        this.adapter = adapter;
    }

    @Override
    public void write(JsonWriter out, Optional<E> value) throws IOException {
        if(value == null || !value.isPresent()){
            out.nullValue();
        } else {
            adapter.write(out, value.get());
        }
    }

    @Override
    public Optional<E> read(JsonReader in) throws IOException {
        final JsonToken peek = in.peek();
        if(peek != JsonToken.NULL){
            return Optional.ofNullable(adapter.read(in));
        }
        in.nextNull();
        return Optional.empty();
    }

}

3) Registered this adapter when initializing Gson.

Gson gsonOptFact = new GsonBuilder()
    .serializeNulls() // matter of taste, just for output anyway
    .registerTypeAdapterFactory(OptionalTypeAdapter.FACTORY)
    .create();

This allows me to write JSON such that both null and empty Optional are serialized as null (or simply removed from the output), while at the same time reading JSON into Optional fields such that if the field is null I know it was missing from the JSON input, and if the field is Optional.empty I know it was set to null in the input.

Example:

System.out.println(gsonOptFact.toJson(new BasicObjectOptional(true)));
// {"someKey":"someValue","someNumber":42,"mayBeNull":null}

System.out.println(gsonOptFact.toJson(new ComplexObjectOptional(true)));
// {"theTitle":"Complex Object","stringArray":["Hello","World"],"theObject":{"someKey":"someValue","someNumber":42,"mayBeNull":null}}

// Now read back in:
String basic = "{\"someKey\":\"someValue\",\"someNumber\":42,\"mayBeNull\":null}";
String complex = "{\"theTitle\":\"Complex Object\",\"stringArray\":[\"Hello\",\"world\"],\"theObject\":{\"someKey\":\"someValue\",\"someNumber\":42,\"mayBeNull\":null}}";
String complexMissing = "{\"theTitle\":\"Complex Object\",\"theObject\":{\"someKey\":\"someValue\",\"mayBeNull\":null}}";

BasicObjectOptional boo = gsonOptFact.fromJson(basic, BasicObjectOptional.class);
System.out.println(boo);
// someKey = Optional[someValue], someNumber = Optional[42], mayBeNull = Optional.empty

ComplexObjectOptional coo = gsonOptFact.fromJson(complex, ComplexObjectOptional.class);
System.out.println(coo);
// theTitle = Optional[Complex Object], stringArray = Optional[[Optional[Hello], Optional[world]]], theObject = (Optional[someKey = Optional[someValue], someNumber = Optional[42], mayBeNull = Optional.empty])

ComplexObjectOptional coom = gsonOptFact.fromJson(complexMissing, ComplexObjectOptional.class);
System.out.println(coom);
// theTitle = Optional[Complex Object], stringArray = null, theObject = (Optional[someKey = Optional[someValue], someNumber = null, mayBeNull = Optional.empty])

I think this will allow me to integrate JSON Merge Patch with my existing data objects quite well.

Some perspective after a bit of time has passed: We have used this solution quite successfully for roughly half a year now, but we're going to migrate to Jackson instead of Gson for unrelated reasons. Upside: Jackson does this out of the box, distinguishing between ``null`` and "not sent". — Antares42, Aug 22 '16 at 11:29

Michael Lloyd Lee mlk · Answer 2 · 2015-11-16T14:34:34.880

0

I think you will have to use JsonObject and look at the Json object returned.

You can using JsonParser.parse(java.io.Reader) to get the JsonObject.

edited Nov 16 '15 at 14:34

answered Nov 16 '15 at 14:16

Michael Lloyd Lee mlk

14,561
3
44
81

1

Where, or how, would I do that? Currently I'm parsing input directly to domain objects using something like ``MyObject bla = gson.fromJson(someStream, MyObject.class);`` – Antares42 Nov 16 '15 at 14:23
OK, so you're saying instead of mapping the JSON to a domain object, I could just "open" it in the parser and traverse it and look up the elements I'm interested in. – Antares42 Nov 17 '15 at 09:00
Aye. I think you could do something with a custom deserializer. I don't think you can use `Optional` as you need to have three states - Sent and `null`, sent and value and not sent. – Michael Lloyd Lee mlk Nov 17 '15 at 09:26
Have a look at the answer I just submitted. I think I can use ``Optional.empty`` along with ``null`` (either inside the Optional or instead of it, the latter being Gson's default behavior) to model three states. I have to test this a bit more, but I think it can work. – Antares42 Nov 17 '15 at 09:42

score 0 · Answer 3 · answered Jan 24 '19 at 00:11

0

For primitive types, use the nullable version.

Example, use Long where you would normally use long. Integer replaced int, etc.

You can also use String type where you want to detect missing values but then you would also need to parse the String into the type you need.

answered Jan 24 '19 at 00:11

Kenneth Argo

1,697
12
19

I'm not sure this would solve the problem. I needed to distinguish, even only for primitives, three cases: 1: The client sent a value, which may be "0" (asking me to replace the value that was there before). 2: The client actively sent "null" (i.e. asking me to delete what was there before). 3: The client didn't send a new value (i.e. I'm not supposed to touch what I already have). – Antares42 Jan 24 '19 at 23:20
Ah, in my case I only needed to detect missing values. For primitives the above worked and for custom types. they are null when not provided. You are correct; this will not be able to detect a value that is missing vs one that was included in the JSON as null. – Kenneth Argo Jan 26 '19 at 02:48

Gson deserialization: How to distinguish between fields that are missing and fields that are explicitly set to null?

3 Answers3