Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use log timestamps from docker logs. #401

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

localghost
Copy link

The rationale for this PR is that not preserving original timestamps by logspout makes investigation of issues across multiple services hard.

@gbolo
Copy link
Collaborator

gbolo commented Sep 16, 2018

Hi @localghost
This is an interesting idea. I would imagine that the timestamps would be close since logspout retrieves the logs fairly quickly. I guess you would like to preserve the timestamp returned by the docker API. I would consider this if you can make it optional/configurable. Maybe you can introduce a bool flag, and by default we would use time.Now()

Copy link
Collaborator

@gbolo gbolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please re-work this PR. Also add any new feature descriptions to README.md. Also add your change to CHANGELOG.md.

Thanks for your contribution!

router/pump.go Outdated
@@ -235,6 +235,7 @@ func (p *LogsPump) pumpLogs(event *docker.APIEvents, backlog bool, inactivityTim
Since: sinceTime.Unix(),
InactivityTimeout: inactivityTimeout,
RawTerminal: rawTerminal,
Timestamps: true,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set this to true only if this feature is enabled

router/pump.go Outdated
@@ -361,10 +362,15 @@ func newContainerPump(container *docker.Container, stdout, stderr io.Reader) *co
}
return
}
logMessage, logTime, err := parseLogLine(line)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseLogLine should only be called if this feature is enabled. Also, if we get an error, perhaps we should fall back to default behaviour of using Time.Now()

router/pump.go Show resolved Hide resolved
router/pump.go Outdated
if err != nil {
return "", time.Time{}, err
}
return logEntry[1], logTime, nil
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no guarantee that logEntry[1] exists. You should rework this function to check the length of logEntry

@localghost
Copy link
Author

@gbolo
Thanks for review. I will try to apply necessary fixes in a few days.

Copy link
Collaborator

@gbolo gbolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

almost good for for merge. I like that the default behaviour remains during errors. Just review my comment.

@michaelshobbs how does the PR look to you?

if len(logEntry) == 2 {
return logEntry[1], logTime
}
return "", logTime
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we want to return an empty log line? Shouldn't this also be: return line, time.Now()

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if line has no white spaces - if it had the function would finish at len(logEntry) == 2 - and logEntry[0] is a valid time string - otherwise the function would finish at err != nil then it means that an empty message was logged

Copy link
Member

@michaelshobbs michaelshobbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks reasonable. needs tests, however

@gbolo
Copy link
Collaborator

gbolo commented Oct 4, 2018

Hi @localghost

I agree with @michaelshobbs. This shouldn't be too hard. Just make a test case that focuses on testing the function parseLogLine

@localghost
Copy link
Author

@gbolo, I know, I just don't have time currently. I will update this pr in a few days. Thanks.

@localghost
Copy link
Author

@gbolo @michaelshobbs
I have added tests but the CI failed, however, the failure does not look to be connected to my changes - maybe caused by recent github problems.

@michaelshobbs
Copy link
Member

something changed with the upstream golint repo. i've had to modify other projects to get around this. please fix this in logspout by changing this line:

test -x $(GOPATH)/bin/golint || go get github.com/golang/lint/golint

to

test -x $(GOPATH)/bin/golint || go get golang.org/x/lint/golint

@localghost
Copy link
Author

@gbolo @michaelshobbs
CI is green

@yunusemrecatalcam
Copy link

yunusemrecatalcam commented Oct 22, 2019

It looks like you changed back it to time.Now() . I am searching for a way to getting miliseconds of the dates. Since they are all 0, observing exceptions is hard. Is there a way to do it in updated version?
Screenshot_2019-10-22 Graylog - Search

@arseniybanayev
Copy link

I have the same problem as @yunusemrecatalcam. What is the workaround for seeing real timestamps with at least millisecond resolution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants