Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[log] Refactor and measure performance using benchmark #90

Merged
merged 37 commits into from
Dec 9, 2018
Merged

Commits on Sep 6, 2018

  1. [log] Move survey of log lib into survey folder

    - will have a design folder next to it, so the doc folder is no longer flat
    at15 committed Sep 6, 2018
    Configuration menu
    Copy the full SHA
    2a6b47c View commit details
    Browse the repository at this point in the history
  2. [log] Add design doc (blog)

    - basically covered background, what was the goals, good thing about
    current implementation, bad thing etc.
    at15 committed Sep 6, 2018
    Configuration menu
    Copy the full SHA
    fde590d View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2018

  1. [httputil] Add unix domain sock transport #93

    - get `{"message":"page not found"}`
    at15 committed Sep 22, 2018
    Configuration menu
    Copy the full SHA
    2c06217 View commit details
    Browse the repository at this point in the history
  2. [httputil] Unix Transport can proxy docker sock

    - just trim the path and keep the first /
    at15 committed Sep 22, 2018
    Configuration menu
    Copy the full SHA
    10e5c92 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2205cab View commit details
    Browse the repository at this point in the history
  4. [util] Fix #92 add dump val as (pretty) json yaml

    - disable server handler check in unix server listen test, it failed CI
    because when close the server/remove socket file it seems the server
    didn't shut down gracefully ... https://travis-ci.org/dyweb/gommon/jobs/431772857
    at15 committed Sep 22, 2018
    Configuration menu
    Copy the full SHA
    b360fe0 View commit details
    Browse the repository at this point in the history
  5. [log][doc] Add implementation order

    - first do benchmark to get a basic understanding of other logging
    libraries performance and what feature is lacking is gommon so we can
    modify the interface
    - then handle fields, structured logging makes log parsing much more
    easier, and is more generic since parser don't need to know the exact
    context, i.e. http access log etc.
    - finally handle the tree of logger problem, along with the skip runtime
    caller, it's essential for clean logging in testing and util functions
    at15 committed Sep 22, 2018
    Configuration menu
    Copy the full SHA
    33da251 View commit details
    Browse the repository at this point in the history
  6. [log] Init benchmark, json format no fields #88

    go test -bench=.
    goos: linux
    goarch: amd64
    pkg: github.com/dyweb/gommon/log/_benchmarks
    BenchmarkWithoutFieldsJSON/gommon-8         	20000000	        96.9 ns/op
    BenchmarkWithoutFieldsJSON/Zap-8            	10000000	       176 ns/op
    BenchmarkWithoutFieldsJSON/Zap.Sugar-8      	 5000000	       257 ns/op
    BenchmarkWithoutFieldsJSON/zerorlog-8       	20000000	       118 ns/op
    PASS
    ok  	github.com/dyweb/gommon/log/_benchmarks	8.031s
    at15 committed Sep 22, 2018
    Configuration menu
    Copy the full SHA
    c0d47a3 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2018

  1. [log] Add mem allocation to benchmark #88

    go test -bench=. -benchmem
    goos: linux
    goarch: amd64
    pkg: github.com/dyweb/gommon/log/_benchmarks
    BenchmarkWithoutFieldsJSON/gommon-8         	10000000	       126 ns/op	     496 B/op	       4 allocs/op
    BenchmarkWithoutFieldsJSON/Zap-8            	10000000	       205 ns/op	       0 B/op	       0 allocs/op
    BenchmarkWithoutFieldsJSON/Zap.Sugar-8      	 3000000	       353 ns/op	     128 B/op	       2 allocs/op
    BenchmarkWithoutFieldsJSON/zerorlog-8       	10000000	       131 ns/op	       0 B/op	       0 allocs/op
    PASS
    ok  	github.com/dyweb/gommon/log/_benchmarks	6.617s
    at15 committed Nov 18, 2018
    Configuration menu
    Copy the full SHA
    7fc0d70 View commit details
    Browse the repository at this point in the history
  2. [log] Accept []byte in formatHead

    - previously make is called inside formatHead so it will allocate on
    heap for sure, after moving the make to handler and increase the hard
    coded prefix ... some allocation can be avoided, at least we know the
    direction for optimization now (either use a pool or use a fixed length
    make, if it's large enough it can handle most log messages on stack)
    at15 committed Nov 18, 2018
    Configuration menu
    Copy the full SHA
    58e9dc6 View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2018

  1. [log][bench] Add logrus and apex/bench to benchmark

    BenchmarkWithoutFieldsText/gommon-8         	30000000	       129 ns/op	     272 B/op	       3 allocs/op
    BenchmarkWithoutFieldsText/std-8            	10000000	       401 ns/op	     128 B/op	       2 allocs/op
    BenchmarkWithoutFieldsText/zap-8            	20000000	       273 ns/op	      72 B/op	       3 allocs/op
    BenchmarkWithoutFieldsText/zap.sugar-8      	20000000	       309 ns/op	     200 B/op	       5 allocs/op
    BenchmarkWithoutFieldsText/zerolog-8        	 3000000	      1395 ns/op	    2011 B/op	      36 allocs/op
    BenchmarkWithoutFieldsText/apex-8           	 3000000	      1551 ns/op	     320 B/op	      15 allocs/op
    BenchmarkWithoutFieldsText/logrus-8         	 1000000	      3209 ns/op	     769 B/op	      15 allocs/op
    BenchmarkWithoutFieldsJSON/gommon-8         	50000000	        79.6 ns/op	     288 B/op	       3 allocs/op
    BenchmarkWithoutFieldsJSON/zap-8            	20000000	       179 ns/op	       0 B/op	       0 allocs/op
    BenchmarkWithoutFieldsJSON/zap.sugar-8      	20000000	       259 ns/op	     128 B/op	       2 allocs/op
    BenchmarkWithoutFieldsJSON/zerolog-8        	50000000	       113 ns/op	       0 B/op	       0 allocs/op
    BenchmarkWithoutFieldsJSON/apex-8           	 2000000	      1920 ns/op	     512 B/op	      10 allocs/op
    BenchmarkWithoutFieldsJSON/logrus-8         	 2000000	      2990 ns/op	    1218 B/op	      22 allocs/op
    at15 committed Nov 19, 2018
    Configuration menu
    Copy the full SHA
    806e8e9 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2018

  1. [log][doc] Zap is encoding context rightaway when add

    - originally I though it merge the fields when output, but actually when
    you call logger.With(f1, f2), it encode f1, f2 right away into the
    underlying, it makes adding context cheap but couple context with the
    core (similar to handler in gommon/log) so that each core actually has
    state (handler in gommon/log are stateless, for now), exchange space for
    time
    - a drawback of this if a key is used two times, you end up having two
    fields instead of 1, zerolog has the same problem, and they mentioned in
    README that json allow duplicated key and json encoding/decoding library
    will handle it
    at15 committed Nov 21, 2018
    Configuration menu
    Copy the full SHA
    27f6107 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2018

  1. [log][bench] Add disabled level and klog (glog)

    - when use `interface{}` as parameter, it will cause allocation, I know
    it is caued by `interface{}` based on the `16B` size but don't quite get
    why it escape ...
    at15 committed Nov 24, 2018
    Configuration menu
    Copy the full SHA
    1e01659 View commit details
    Browse the repository at this point in the history
  2. [log][doc] Add example of other log libraries

    - json format with context (logger w/ fields)
      - logrus and apex are actually entry w/ context where entry hold a
    pointer to the logger it should use ...
    at15 committed Nov 24, 2018
    Configuration menu
    Copy the full SHA
    773794c View commit details
    Browse the repository at this point in the history
  3. [log] Return struct instead of pointer in factory

    - was planning to change identity to a struct, but since it could be nil
    and it is not used most of the time, decided to change it back to
    pointer
    - [ ] TODO: updated gommon generator template, so all code that use
    struct logger factory need update
    at15 committed Nov 24, 2018
    Configuration menu
    Copy the full SHA
    a4fa6b1 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2018

  1. [log][handler] Support contex in handler interface

    - the 'context` is actually just fields attached to logger
    - there aren't many tests so we didn't break any test ....
    at15 committed Nov 25, 2018
    Configuration menu
    Copy the full SHA
    04bd691 View commit details
    Browse the repository at this point in the history
  2. [log][handler] Add multi handler Fix #87

    - fan out log in serial
    at15 committed Nov 25, 2018
    Configuration menu
    Copy the full SHA
    6496dca View commit details
    Browse the repository at this point in the history
  3. [log][handler] Reduce interface methods 6 -> 1

    - previously was not using empty value for optional parameters and endup
    having 6 functions, this made handler implementation error prone due to
    copy and paste also made handler func impossible
    - now there is just one log method in Handler call HandleLog
    at15 committed Nov 25, 2018
    Configuration menu
    Copy the full SHA
    19539b3 View commit details
    Browse the repository at this point in the history
  4. [log][bench] Add InfoF to benchmark

    - it has less allocation than `Info` because it accepts string instead of
    `interface{}`
    at15 committed Nov 25, 2018
    Configuration menu
    Copy the full SHA
    beff00e View commit details
    Browse the repository at this point in the history
  5. [log][level] Add Print as alias for Info level

    - this should makes a gommon/log drop in replacement for some library
    that requires a standard library like interface `Print` and `Printf`
    - [ ] TODO: I don't want to add `Println` because `\n` does not make
    sense for most handlers, json, text w/ fields etc. there will be a `\n`
    for most most handlers, not after the message but after the fields
    at15 committed Nov 25, 2018
    Configuration menu
    Copy the full SHA
    ba8d224 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2018

  1. [log][json] Support escape in json string

    - just copy and paste from standard library and remove check on html and
    jsonp, looked at zap and zerolog, they are basically the same ...
    at15 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    affaa35 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b9725a1 View commit details
    Browse the repository at this point in the history
  3. [log][bench] io.Writer cause byte slice to escape

    - at first I thought allocate a fixed size on stack would remove the
    allocation, but I was wrong, because the slice is passed to io.Writer
    and a call on interface is marked as `parameter to indirect call` and
    so even a fixed size slice can escape ...
    at15 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    53a65fe View commit details
    Browse the repository at this point in the history
  4. [log][handler] Use magic number to reduce alloc

    - since we can't avoid the alloc due to call on io.Writer, we just need
    to make sure to allocate a slice large enough for message and fields so
    we don't need to allocate during the middle of append
    - [ ] TODO: these are just magic numbers, which will waste
    resource/won't work when the size of fields key/value are too small/too
    big ...
    - [ ] TODO: no context with fields json still have two alloc
    ````
    BenchmarkWithContextFieldsJSON/gommon-8         	30000000	       117 ns/op	     352 B/op	       3 allocs/op
    BenchmarkWithContextFieldsJSON/gommon.F-8       	50000000	        74.0 ns/op	     224 B/op	       1 allocs/op
    BenchmarkNoContextWithFieldsJSON/gommon.F-8     	50000000	        92.2 ns/op	     352 B/op	       2 allocs/op
    ````
    at15 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    276d177 View commit details
    Browse the repository at this point in the history
  5. [log][bench] fields seems to be causing alloc

    - at first I suspect it's because varadic arg but it seems not ...
      - could be the `Fields` type
    at15 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    7fc9d48 View commit details
    Browse the repository at this point in the history
  6. [log][bench] Interface caused alloc on fields

    - this happens to zap as well but didn't happen to zerolog because
    latter don't have any usage of interface methods when attaching fields
      - this only works because zerolog only support json, so it can just
    encode things right away and don't need handler interface, the drawback
    is for all the other format, it has to decode the bytes into
    map[string]interface{} and then print them out ...
    at15 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    d2217b8 View commit details
    Browse the repository at this point in the history
  7. [util] Add dump json to testutil #92

    - actually they are alredy there in pretty.go under same package ...
    at15 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    a51fd3f View commit details
    Browse the repository at this point in the history
  8. [log] Add Flush to Logger Fix #8

    at15 committed Nov 26, 2018
    Configuration menu
    Copy the full SHA
    e880246 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2018

  1. [log][doc] Add design for tree of logger

    - in v1 we followed logrus and add a filter for filtering log
    - in v2 we added parent and children but it will cause gc problem
    at15 committed Dec 6, 2018
    Configuration menu
    Copy the full SHA
    701080d View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2018

  1. Configuration menu
    Copy the full SHA
    13979fc View commit details
    Browse the repository at this point in the history
  2. [log] Add log registry to keep tree #33 #78

    - registry will make the tree explicit, logger has to manually register
    themselves to registry and registry have to manually register to other
    registry, short lived logger should not register them self
    - registry also have identity and is like folder while logger is like
    file (log structured file system XD)
    - the map for children logger will be removed in following commit, and
    generator logic will change as well
    at15 committed Dec 8, 2018
    Configuration menu
    Copy the full SHA
    9f39ed1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5e27c56 View commit details
    Browse the repository at this point in the history
  4. [log] Remove map of child loggers #33 #78

    - now only registry is used to keep the relationship between loggers
    across package and projects
    at15 committed Dec 8, 2018
    Configuration menu
    Copy the full SHA
    33a3df1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b8957fa View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2018

  1. [log] Add Copy for create func & method logger

    - add WalkRegistry
    - remove deprecated logger_tree logic
    at15 committed Dec 9, 2018
    Configuration menu
    Copy the full SHA
    e7d7305 View commit details
    Browse the repository at this point in the history
  2. [log][bench] Add caller increased 3 times

    - for single message w/ 1 allocation, increased from 60ns -> 236
    - remove some outdated doc on log package, most of them are still very
    outdated though ...
    at15 committed Dec 9, 2018
    Configuration menu
    Copy the full SHA
    6d2de9b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f1a6678 View commit details
    Browse the repository at this point in the history