LC-416 added json logging for elk and setting mdc values #100
Conversation
| @@ -183,7 +183,7 @@ private void checkForDoc() { | |||
| sendToIndexWithAccounting(batch.flushIfExpired()); | |||
| return; | |||
| } | |||
|
|
|||
| log.debug("Indexing Doc {}", doc.getId()); | |||
There was a problem hiding this comment.
the call to sendToIndexWithAccounting is not necessarily the time when the doc is sent to the search engine
batch.add(doc) will typically add the doc to a batch being built up, and return an empty list so the sendToIndexWithAccounting call is effectively a no-op
batch.add(doc) returns a non-empty list only in certain situations, like when the batch is full and ready to be flushed
| @@ -122,6 +125,7 @@ private void publishInternal(Document document) throws Exception { | |||
| } | |||
|
|
|||
| if (previousDoc.getId().equals(document.getId())) { | |||
| log.debug("Sending document."); | |||
There was a problem hiding this comment.
this clause isn't when we're sending the document
this clause is when we're "collapsing" the current document into the previous one because they had the same IDs
| @@ -93,6 +94,7 @@ public PublisherImpl(Config config, PublisherMessenger messenger, String runId, | |||
|
|
|||
| @Override | |||
| public void publish(Document document) throws Exception { | |||
| MDC.put("docId", document.getId()); | |||
There was a problem hiding this comment.
in local mode, the publisher will be running in one thread, some number of workers will be running in their own threads, and an indexer will be running in yet another thread. I don't think setting the docId in the MDC here will propagate the change to other threads where docId is needed, but even if it did, the propogated docId would be "right" for those other threads because everything is happening concurrently -- one doc3 might be published at the same time worker 1 is processing doc2 and the indexer is indexing doc1...
We are adding more logging to be able to be ingested by Elasticsearch. We decided that what would make the most sense is to create a separate json log file so it can easily be indexed and would not require any GROK. In addition, we want to know some values for everything single log so we are setting mdc values for run id, doc id, and stage name at the moment.