DateSerializer is not thread-safe

[cdegroc]

Setup

• Version: 0.5.3
• Storage Backend: cql
• Mixed Index Backend: elasticsearch

Under production load, our JanusGraph cluster failed to process from 1 to 30 requests / hour due to ElasticSearch receiving invalid dates (+0000-01-31T08:49:54.000+0000):

org.janusgraph.core.JanusGraphException: Could not process individual retrieval call
[...]
    Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent failure in storage backend
    at org.janusgraph.diskstorage.es.ElasticSearchIndex.query(ElasticSearchIndex.java:1117)
    at org.janusgraph.diskstorage.indexing.IndexTransaction.queryStream(IndexTransaction.java:108)
    at org.janusgraph.diskstorage.BackendTransaction$6.call(BackendTransaction.java:416)
    at org.janusgraph.diskstorage.BackendTransaction$6.call(BackendTransaction.java:413)
    at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
    at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
    ... 49 common frames omitted
    Caused by: org.elasticsearch.client.ResponseException: method [POST], host [...], URI [/.../_search?scroll=60s], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse date field [+0000-01-31T08:49:54.000+0000] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [+0000-01-31T08:49:54.000+0000] with format [strict_date_optional_time||epoch_millis]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"...","node":"...","reason":{"type":"parse_exception","reason":"failed to parse date field [+0000-01-31T08:49:54.000+0000] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [+0000-01-31T08:49:54.000+0000] with format [strict_date_optional_time||epoch_millis]]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [+0000-01-31T08:49:54.000+0000] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}]},"status":400}
    at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
    at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
    at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.search(RestElasticSearchClient.java:467)
    at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.search(RestElasticSearchClient.java:437)
    at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.search(RestElasticSearchClient.java:60)
    at org.janusgraph.diskstorage.es.ElasticSearchIndex.query(ElasticSearchIndex.java:1108)
    ... 54 common frames omitted

I tracked down the issue and was able to reproduce it with the following unit test:

public class DateSerializerTest {
    private final StandardSerializer serializer = new StandardSerializer();

    @Test
    public void dateSerializerConvertStringThreadSafe() throws Exception {
        ExecutorService pool = Executors.newFixedThreadPool(64);
        String input = "2021-01-30T17:30:31.000";
        Date reference = StdDateFormat.instance.parse(input);

        List<Future<Date>> futures = new ArrayList<>();
        try {
            // Have serializer parse the same date 100x in parallel
            for (int i = 0; i < 100; ++i) {
                futures.add(pool.submit(() -> serializer.convert(Date.class, input)));
            }
            for (Future<Date> future : futures) {
                assertEquals(reference, future.get());
            }
        } finally {
            pool.shutdown();
        }
    }
}

The unit test fails due to partial date matches such as:

org.opentest4j.AssertionFailedError: 
Expected :Sat Jan 30 18:30:31 CET 2021
Actual   :Fri Jan 01 18:30:31 CET 2021

org.opentest4j.AssertionFailedError: 
Expected :Sat Jan 30 18:30:31 CET 2021
Actual   :Thu Jan 01 01:00:00 CET 1970

I think DateSerializer is not thread-safe because StdDateFormat.instance#parse reuses a Calendar instance that is not thread-safe.

There are multiple ways to solve this (confirmed by the above unit test passing):

• Wrap dateFormat.parse in a synchronized block/method
• Store StdDateFormat in a ThreadLocal or create new local instances
• Use Java 8+ DateTimeFormatter (if so, do we want to remain compatible with the RFC-1123 format that StdDateFormat handled?)

[cdegroc]

Created an initial PR. I'll be happy to get feedback and switch to a different approach.