Skip to content

Commit

Permalink
Use cgroups aware processor count by default
Browse files Browse the repository at this point in the history
In containerized environments, a number of CPU cores isn't the
same as the available CPUs. In this case, we need to consider cgroups.

`concurrent-ruby` now has the method to get that since v1.3.1. I think it's better
to use this for setting more container environment friendly default value.
Ref: ruby-concurrency/concurrent-ruby#1038

I keep `Parallel.processor_count` as is to keep supporting setting
processor count via env(`PARALLEL_PROCESSOR_COUNT`).
  • Loading branch information
y-yagi committed Aug 8, 2024
1 parent 9644d13 commit 2e13dfd
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 9 deletions.
1 change: 1 addition & 0 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ Tips
- [Isolation] Do not reuse previous worker processes: `isolation: true`
- [Stop all processes with an alternate interrupt signal] `'INT'` (from `ctrl+c`) is caught by default. Catch `'TERM'` (from `kill`) with `interrupt_signal: 'TERM'`
- [Process count via ENV] `PARALLEL_PROCESSOR_COUNT=16` will use `16` instead of the number of processors detected. This is used to reconfigure a tool using `parallel` without inserting custom logic.
- [Process count] `parallel` uses a number of processors seen by the OS for process count by default. If you want to use a value considering CPU quota, please add `concurrent-ruby` to your `Gemfile`.

TODO
====
Expand Down
18 changes: 14 additions & 4 deletions lib/parallel.rb
Original file line number Diff line number Diff line change
Expand Up @@ -337,10 +337,10 @@ def physical_processor_count
end
end

# Number of processors seen by the OS, used for process scheduling
# Number of processors seen by the OS or value considering CPU quota if the process is inside a cgroup,
# used for process scheduling
def processor_count
require 'etc'
@processor_count ||= Integer(ENV['PARALLEL_PROCESSOR_COUNT'] || Etc.nprocessors)
@processor_count ||= Integer(ENV['PARALLEL_PROCESSOR_COUNT'] || available_processor_count)
end

def worker_number
Expand Down Expand Up @@ -695,5 +695,15 @@ def instrument_start(item, index, options)
return unless (on_start = options[:start])
options[:mutex].synchronize { on_start.call(item, index) }
end

def available_processor_count
begin
require 'concurrent-ruby'
Concurrent.available_processor_count.floor
rescue LoadError, NoMethodError
require 'etc'
Etc.nprocessors
end
end
end
end
end
5 changes: 0 additions & 5 deletions spec/parallel_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,6 @@ def without_ractor_warning(out)
(1..999).should include(Parallel.processor_count)
end
end

it 'uses Etc.nprocessors in Ruby 2.2+' do
defined?(Etc).should == "constant"
Etc.respond_to?(:nprocessors).should == true
end
end

describe ".physical_processor_count" do
Expand Down

0 comments on commit 2e13dfd

Please sign in to comment.