Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up path processing #319

Open
wants to merge 37 commits into
base: develop
Choose a base branch
from

Commits on Oct 14, 2018

  1. Use explicit types for some integer constants

    The following constant expression:
    
        1024 * 1024
    
    should be evaluated by the compiler at runtime as though it were:
    
        (int)1024 * (int)1024
    
    and one would expect the result to be:
    
        (int)1048576
    
    However, strictly speaking, an `int'  is only required to be able
    to represent  at least  the number 32767.  However, a  `long int'
    must be able  to represent at least  `2147483647'. Thus, strictly
    speaking, proper C  code should specify that at least  one of the
    integer constants is to be represented as a `long int':
    
        1024L * 1024
    
    Consider, also, the following program:
    
        #include <limits.h> // UINT_MAX
        #include <stdio.h>  // printf
    
        #define PRINT_SIZE_OF(type) \
            printf("sizeof(" #type ") = %u\n", (unsigned int)sizeof(type))
    
        #define PRINT_UNSIGNED_LONG_LONG_INT(expr) \
            printf(#expr " = %llu\n", (unsigned long long int)(expr))
    
        int main()
        {
            PRINT_SIZE_OF(int);
            PRINT_SIZE_OF(long long int);
    
            PRINT_UNSIGNED_LONG_LONG_INT(UINT_MAX + 1);
            PRINT_UNSIGNED_LONG_LONG_INT(UINT_MAX + 1ULL);
    
            PRINT_UNSIGNED_LONG_LONG_INT(65536U * 65536);
            PRINT_UNSIGNED_LONG_LONG_INT(65536ULL * 65536);
        }
    
    That program might produce the following output:
    
        sizeof(int) = 4
        sizeof(long long int) = 8
        UINT_MAX + 1 = 0
        UINT_MAX + 1ULL = 4294967296
        65536U * 65536 = 0
        65536ULL * 65536 = 4294967296
    
    As you can see, specifying the type of an integer constant can be
    important. Take, for example, one of the above calculations:
    
        65536U * 65536
    
    Without the "U"  suffix to specify that the  integer constant has
    an unsigned type, the compiler would compute:
    
        (int)65536 * (int)65536
    
    On the  same machine, this would  result in overflow of  a signed
    integer type,  which invokes undefined behavior;  a good compiler
    would probably warn about it, or even produce an error.
    mfwitten committed Oct 14, 2018
    Configuration menu
    Copy the full SHA
    ff9a53d View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2018

  1. lib/cmdline.c: Fix error message

    The main  problem was that  the error message used  `Negativ' rather
    than `Negative'. However, the message was also a little too playful;
    this commit  changes the entire  message to something a  little more
    dour.
    mfwitten committed Oct 15, 2018
    Configuration menu
    Copy the full SHA
    d085572 View commit details
    Browse the repository at this point in the history
  2. Miscellaneous improvements

    mfwitten committed Oct 15, 2018
    Configuration menu
    Copy the full SHA
    63bc4be View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2019

  1. lib/cfg.*: Clean up `rm_cfg_free_paths()'

    This simply  introduces `const'  discipline.
    Also, `g_assert(cfg);' has been inserted.
    mfwitten committed Jan 22, 2019
    Configuration menu
    Copy the full SHA
    a3e9d5c View commit details
    Browse the repository at this point in the history
  2. lib/cfg.*: rm_cfg_add_path()' now returns bool'

    It was already returning values as though it did.
    mfwitten committed Jan 22, 2019
    Configuration menu
    Copy the full SHA
    93541fc View commit details
    Browse the repository at this point in the history
  3. lib/cmdline.c: return the status of `rm_cfg_add_path()'

    Before, `rm_cmd_parse_replay()' simply ignored whether there was a
    failure; now, it both returns a status and sets an error message.
    mfwitten committed Jan 22, 2019
    Configuration menu
    Copy the full SHA
    9c51e9d View commit details
    Browse the repository at this point in the history
  4. lib/cfg.*: Refactor `rm_cfg_add_path()'

    This commit introduces  a number of finer-grained  functions related to
    resolving and verifying  paths, and then uses those  functions not only
    to refactor `rm_cfg_add_path()', but also to introduce  a new function,
    `rm_cfg_prepend_json()'. The  finer-grained functions are defined  in a
    new source file:
    
        lib/path-funcs.h
    
    Similarly, `RmPath' has been moved to its own header:
    
        lib/path.h
    
    Additionally, some of the `#include' directives have been cleaned up as
    part of this  refactoring; if source file `X' uses  an identifier whose
    declaration comes from source file  `Y', then `X' *always* includes `Y'
    explicitly  (even if  `Y'  is included  indirectly  through some  other
    source  file),  and each  `#include'  directive  is associated  with  a
    comment that lists each such identifier that it provides.
    mfwitten committed Jan 22, 2019
    Configuration menu
    Copy the full SHA
    f8629f1 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2019

  1. lib/cfg.*: Refactor `rm_cfg_add_path()'

    This commit introduces  a number of finer-grained  functions related to
    resolving and verifying  paths, and then uses those  functions not only
    to refactor `rm_cfg_add_path()', but also to introduce  a new function,
    `rm_cfg_prepend_json()'. The  finer-grained functions are defined  in a
    new source file:
    
        lib/path-funcs.h
    
    Similarly, `RmPath' has been moved to its own header:
    
        lib/path.h
    
    Additionally, some of the `#include' directives have been cleaned up as
    part of this  refactoring; if source file `X' uses  an identifier whose
    declaration comes from source file  `Y', then `X' *always* includes `Y'
    explicitly  (even if  `Y'  is included  indirectly  through some  other
    source  file),  and each  `#include'  directive  is associated  with  a
    comment that lists each such identifier that it provides.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    3539636 View commit details
    Browse the repository at this point in the history
  2. lib/cmdline.c: Move `rm_cmd_read_paths_from_stdin()'

    Move it closer to `rm_cmd_set_paths()'.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    fb1ba2d View commit details
    Browse the repository at this point in the history
  3. lib/cmdline.c: Simplify size in call to `malloc()'

    By definition, `sizeof(char)' evaluates to 1.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    ee9b9a0 View commit details
    Browse the repository at this point in the history
  4. lib/cmdline.c: Replace RmSession' with RmCfg' in `rm_cmd_set_paths*()'

    A pointer to an `RmSession' object was being passed around, but only a
    pointer to its associated `RmCfg' object is needed.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    f8cea9f View commit details
    Browse the repository at this point in the history
  5. lib/cmdline.c: Abstract out struct `rm_cmd_set_paths_vars'

    Variables that need to be shared across `rm_cmd_set_paths*()' may
    be placed in an instance of this struct.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    25db8f6 View commit details
    Browse the repository at this point in the history
  6. lib/cmdline.c: Add error handling to `rm_cmd_set_paths_from_stdin()'

    Before, it was just ignoring failures in `malloc()' and `getdelim()'.
    Also, one of the error messages in `lib/treemerge.c' has been set to
    match the error message introduced here.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    1e81e41 View commit details
    Browse the repository at this point in the history
  7. lib/cfg.h: Remove member `read_stdin'

    It's only used locally in one function, so it can just be a local
    variable in that one function.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    fb31ce0 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    d4f7805 View commit details
    Browse the repository at this point in the history
  9. lib/cmdline.c: Move constant loop condition (`paths') out of the loop

    The compiler  probably already does  this, but I find  it clearer
    and more  satisfying to do  this explicitly, especially  since it
    allows avoiding a call to  `g_strfreev()', rather than relying on
    that function to check `paths' for us again.
    
    For the  sake of clarity in  the diff, the existing  code has not
    yet been indented to reflect the  fact that it's part of the `if'
    statement's consequent.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    13304f7 View commit details
    Browse the repository at this point in the history
  10. lib/cmdline.c: Indent code to reflect the fact that it's in a block

    You can check that only white space changed:
    
        $ git diff HEAD^ --ignore-space-change && echo Nothing\ changed
        Nothing changed
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    a9f4de0 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    05191ee View commit details
    Browse the repository at this point in the history
  12. lib/cmdline.c: Free each command-line path as it is processed.

    On each iteration of the loop, a path string is freed when
    processing is finished, and then the whole list of strings
    is simply freed without having to iterate through the list
    again to free each string.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    4f14e8a View commit details
    Browse the repository at this point in the history
  13. lib/cfg-funcs.h: Move cfg->replay' and cfg->path_count++'

    These variables were handled directly in `rm_cfg_prepend_path()';
    now, they're handled directly in `rm_cmd_set_paths*()'.
    
    This has allowed for optimizing the call to `rm_cfg_prepend_path()'
    that is used for adding the current working directory when no other
    path has been supplied by the user; there's no reason to check
    whether the current working directory is a JSON file.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    5dafe3a View commit details
    Browse the repository at this point in the history
  14. lib/path.h: s/idx/index/ in `RmPath'

    The field `idx' of struct `RmPath' is now `index'.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    a5698fd View commit details
    Browse the repository at this point in the history
  15. lib/cmdline.c: Move cfg->replay' and cfg->path_count++' out of loops

    Though an optimizing compiler might do this already, one might as
    well just do it explicitly, so as not to depend on the whims of
    any particular compiler.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    322d7cc View commit details
    Browse the repository at this point in the history
  16. lib/path.h: s/treat_as_single_vol/single_volume/ in `RmPath'

    The field `treat_as_single_vol' of struct `RmPath' is now `single_volume'.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    67e0738 View commit details
    Browse the repository at this point in the history
  17. lib/cfg.h: Use `RmCfg::path_count' more purposefully

    Now, member `path_count' of a struct `RmCfg' has a more guaranteed
    and expected  purpose: During and after  the user-input processing
    of non-option path arguments, it represents the number of items in
    the associated list `paths'.
    
    This has allowed for the removal of a call to `g_slist_length()'.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    e39fa29 View commit details
    Browse the repository at this point in the history
  18. lib/cfg.c: git mv lib/cfg.c lib/cfg-funcs.h

    This breaks the project, and is thus meant to be merged into
    a working commit.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    2f72e34 View commit details
    Browse the repository at this point in the history
  19. lib/cfg.*: Rename rm_cfg_add_path' to rm_cfg_prepend_path()'

    This makes it consistent with `rm_cfg_prepend_json()', and it also
    implies that the  list of files is being constructed  such that it
    will be  reversed with  respect to  the order  in which  paths are
    provided by the user.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    eb09ea9 View commit details
    Browse the repository at this point in the history
  20. lib/cmdline.c: Abstract out `rm_cmd_set_paths_from_cmdline()'

    Code from `rm_cmd_set_paths()' has been moved to its own function:
    
        rm_cmd_set_paths_from_cmdline()
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    bc825f3 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    74daf3b View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    6dcbfde View commit details
    Browse the repository at this point in the history
  23. lib/cmdline.c: Replace `strcmp()' with direct character comparisons

    The way `rmlint' currently works is that it produces a shell script
    that sometimes  performs its  work by  running `rmlint'  many times
    with  varying arguments;  `rmlint' can  be  run a  large number  of
    times,  and each  invocation involves  argument processing  whereby
    each path  argument undergoes multiple char-by-char  comparisons in
    search of  a match for  a special,  well-known value (e.g.,  "-" or
    "//").
    
    Therefore,  this commit  optimizes the  process by  using tailored,
    idiomatic  C-string calculations  in  lieu of  invoking a  generic,
    open-ended,  arbitrary comparison  function;  this commit  replaces
    the  Standard C  Library's  `strcmp()' with  boolean operators  and
    character comparisons.
    
    The calculations are  idiomatic in that they are a  familiar way of
    processing strings, but  are somewhat particular to C  code. To aid
    reading  this code,  comments  have been  embedded  to clarify  the
    purpose, as in the following:
    
      if(/* path is "-" */ path[0] == '-' && path[1] == 0)
    
    It's  true   that  an   optimizing  compiler  might   perform  this
    substitution  automatically,  or  it  might  even  make  a  better,
    platform-specific optimization. However,  this seems unlikely based
    on  the following  rudimentary benchmarks;  first define  some Bash
    functions:
    
      # Copy this script with the following command line, where "$THIS"
      # expands to the revision of this commit log message:
      #
      #   git log -1 --format=%b "$THIS" |
      #     sed -n '/^  #-/,/^  #-/p' | cut -c3- >/tmp/script.sh
      #
      #----------------------------------------------------------------
      make_test()
      {
      	# Note that the body of this function is
      	# explicitly indented with tab characters
      	# for the benefit of the "<<-" operator.
    
      	local name="$1" expression="$2"
    
      	cat >"$name".c <<-EOF
      	#include <stdbool.h> // bool
      	#include <string.h>  // strcmp
    
      	#define ITERATIONS 10000000 // 10 million
    
      	int main()
      	{
      	    int result = 0;
      	    char string[3] = {0};
      	    const char *const volatile pointer = string;
      	    bool toggle = 0;
      	    for (long i = 0; i < ITERATIONS; ++i) {
      	        string[0] = string[1] = '/' + toggle;
      	        toggle = !toggle;
      	        const char *const s = pointer;
      	        result += $expression;
      	    }
      	    return result;
      	}
      	EOF
      }
    
      build()
      {
        local name="$1"; shift
        gcc -std=c99 -pedantic -Wall -static -fno-plt -fno-pie "$@" \
          "$name".c -o "$name"
      }
    
      build_all() { build strcmp "$@" && build equals "$@"; }
    
      run() { time -p ./"$1"; }
    
      benchmark()
      {
        echo =========== "$@"
        run strcmp
        echo -----------
        run equals
      }
    
      build_and_benchmark()
      {
        build_all "$@" || return 1
        benchmark "$@"; return 0
      }
      #----------------------------------------------------------------
    
    Then, create and run some benchmarks:
    
      make_test strcmp 'strcmp(s,"//")'
      make_test equals "(s[0] == '/' && s[1] == '/' && s[2] == 0)"
    
      gcc --version | head -n 1
      build_and_benchmark -O0
      build_and_benchmark -Og
      build_and_benchmark -O1
      build_and_benchmark -O2
    
    All together,  this produces the  following output on my  [very old
    and slow] system; at each optimization level, timing information is
    provided first for `strcmp()' and  then for the manual calculation.
    Perversely,  optimization  seems  to make  `strcmp()'  slower,  not
    faster;   in  contrast,   optimization  drastically   improves  the
    performance of manual calculation, which is also always faster than
    using `strcmp()'.
    
      gcc (GCC) 7.3.1 20180312
      =========== -O0
      real 0.35
      user 0.35
      sys 0.00
      -----------
      real 0.24
      user 0.24
      sys 0.00
      =========== -Og
      real 0.58
      user 0.57
      sys 0.00
      -----------
      real 0.15
      user 0.15
      sys 0.00
      =========== -O1
      real 0.60
      user 0.59
      sys 0.00
      -----------
      real 0.16
      user 0.16
      sys 0.00
      =========== -O2
      real 0.58
      user 0.57
      sys 0.00
      -----------
      real 0.07
      user 0.07
      sys 0.00
    
    Assembler code  for the unoptimized  case can be produced  with the
    following:
    
      build_all -S -O0
      cat strcmp # review
      cat equals
    
    Here is the meat of the comparison:
    
               strcmp                    equals
      ------------------------- | -------------------------
         .LC0:                  |  movl    -20(%ebp), %eax
         .string "//"           |  movzbl  (%eax), %eax
         ...                    |  cmpb    $47, %al           // s[0] == '/'
         pushl   $.LC0          |  jne     .L3
         pushl   -20(%ebp)      |  movl    -20(%ebp), %eax
         call    *strcmp@GOT    |  addl    $1, %eax
                                |  movzbl  (%eax), %eax
                                |  cmpb    $47, %al           // s[1] == '/'
                                |  jne     .L3
                                |  movl    -20(%ebp), %eax
                                |  addl    $2, %eax
                                |  movzbl  (%eax), %eax
                                |  testb   %al, %al           // s[2] == 0
                                |  jne     .L3
                                |  movl    $1, %eax
                                |  jmp     .L4
                                |  .L3:
                                |  movl    $0, %eax
    
    As  you can  see, one  is calling  a function,  while the  other is
    performing  the manual  calculations. Now,  consider the  optimized
    version:
    
      build_all -S -O2
      cat strcmp
      cat equals
    
    This yields:
    
               strcmp                    equals
      ------------------------- | -------------------------
       .LC0:                    |  movl    -20(%ebp), %ebx
       .string "//"             |  cmpb    $47, (%ebx)
       ...                      |  jne     .L2
       movl    -36(%ebp), %esi  |  cmpb    $47, 1(%ebx)
       movl    $.LC0, %edi      |  jne     .L2
       ...                      |  cmpb    $1, 2(%ebx)
       movl    $3, %ecx         |
       repz cmpsb               |
       seta    %cl              |
       setb    %bl              |
       subl    %ebx, %ecx       |
       movsbl  %cl, %ecx        |
    
    There  is no  longer a  call to  `strcmp()', but  the compiler  has
    substituted what is  essentially an exact replica  of `strcmp()' in
    terms of  x86 assember  code; the `repz cmpsb'  iterates  through 2
    strings in  memory, comparing the  bytes, stopping when there  is a
    mismatch. Strangely, one would think  that a native x86 instruction
    sequence might be inherently faster, but the benchmarks reveal that
    the naive integer comparisons are much faster. I suspect the reason
    for this is 2-fold:
    
      * Perhaps, `repz cmpsb'  is performing a lot  more memory access;
        in  contrast, each  naive  integer  comparison instruction  can
        encode  a comparand  directly (e.g.,  the number  47, which  is
        ASCII for '/').
    
      * Maybe, `repz cmpsb' cannot  benefit from instruction pipelining
        as readily as separate integer comparisons.
    
    Perhaps  the GNU  GCC  team should  re-consider  how `strcmp()'  is
    optimized.
    
    Anyway,  the results  are  clear:  It's better  to  do these  small
    calculations  by hand,  rather  than  rely on  the  compiler to  do
    something magical.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    4182c59 View commit details
    Browse the repository at this point in the history
  24. lib/cfg.c -> lib/cfg-funcs.h: Make functions inlinable

    Each of these functions is pretty much single-purpose, and should
    therefore be inlined at the few call sites.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    605b5a9 View commit details
    Browse the repository at this point in the history
  25. lib/cmdline.c: Tell compiler to optimize for the value of `replay'

    When `cfg->replay' is false, there's no reason to check whether a
    path is a `json' file.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    a4e4b88 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    30e7156 View commit details
    Browse the repository at this point in the history
  27. lib/cmdline.c: INLINE `rm_cmd_set_paths()'

    It's a static function that is used in only place.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    2c24f77 View commit details
    Browse the repository at this point in the history
  28. lib/cfg.c -> lib/cfg-funcs.h: Refactor functions

    The functions in question are now finer-grained and inlinable, which
    is worthwhile because these functions are only used in basically one
    part  of  the  program,  and  thus  exist  mainly  as  a  matter  of
    organization (abstraction, documentation, etc.).
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    3b1098c View commit details
    Browse the repository at this point in the history
  29. lib/cmdline.c: Refactor `rm_cmd_set_paths*()'

    The code has been organized into finer-grained functions/macros,
    and has thereby been easily optimized for certain cases.
    mfwitten committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    c0ba1b5 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    0d332c9 View commit details
    Browse the repository at this point in the history