Processes are leaking, elevated memory usage

[pmusaraj]

Hi there,

We run sass-embedded on multiple servers and are noticing that the processes are leaking, causing containers to use a lot of RAM.

ELAPSED   RSS COMMAND         ARGS
 239640 29156 dart:sass.snaps /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sass-embedded-1.66.1-x86_64-linux-gnu/ext/sass/dart-sass/src/dart /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/
 239662 23176 dart:sass.snaps /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sass-embedded-1.66.1-x86_64-linux-gnu/ext/sass/dart-sass/src/dart /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/
 239663 22308 dart:sass.snaps /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sass-embedded-1.66.1-x86_64-linux-gnu/ext/sass/dart-sass/src/dart /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/
 223323 21092 dart:sass.snaps /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sass-embedded-1.66.1-x86_64-linux-gnu/ext/sass/dart-sass/src/dart /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/
 237514 20944 dart:sass.snaps /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sass-embedded-1.66.1-x86_64-linux-gnu/ext/sass/dart-sass/src/dart /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/
 239663 13060 dart:sass.snaps /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sass-embedded-1.66.1-x86_64-linux-gnu/ext/sass/dart-sass/src/dart /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/

On our servers, we've observed this behaviour with versions 1.66.1 and 1.69.2. I can also reproduce the issue locally with version 1.69.5:

  PID TTY           TIME CMD
81862 ttys000    0:00.63 /Users/pmusaraj/.gem/ruby/3.2.1/gems/sass-embedded-1.69.5-arm64-darwin/ext/sass/dart-sass/src/dart /Users/pmusaraj/.gem/ruby/3.2.1/gems/sass
81863 ttys000    0:00.64 /Users/pmusaraj/.gem/ruby/3.2.1/gems/sass-embedded-1.69.5-arm64-darwin/ext/sass/dart-sass/src/dart /Users/pmusaraj/.gem/ruby/3.2.1/gems/sass
81914 ttys000    0:00.13 /Users/pmusaraj/.gem/ruby/3.2.1/gems/sass-embedded-1.69.5-arm64-darwin/ext/sass/dart-sass/src/dart /Users/pmusaraj/.gem/ruby/3.2.1/gems/sass

In theory, should there be just one dart-sass process running?

[ntkme]

I don’t control how sass-embedded is used in discourse. Can you reproduce this standalone?

Note, there are two ways to run:

1. Use the singleton compiler Sass. This would start a single dart-sass compiler per Ruby process, and the compiler shuts down when Ruby process shuts down.
2. Use Sass::Embedded.new. This require user to explicitly close the compiler.

[ntkme]

Also, can you please run ps --forest and share the output? I would like to see the PPID (parent PID) of each compiler process.

[pmusaraj]

Thanks for the quick reply.

Not sure if this is helpful, but Discourse uses the recommended method in the https://github.com/ntkme/sassc-embedded-shim-ruby repo, i.e. SassC::Engine.new(sass, style: :compressed).render

https://github.com/discourse/discourse/blob/main/lib/stylesheet/compiler.rb#L51-L63

Note that in the public repo, we're using the dartsass-ruby fork, but I can repro locally with sassc-embedded as well.

[ntkme]

the recommended method

It's not recommended. The recommended method is to directly use Sass.compile or Sass.compile_string without the SassC shim. However, it's probably unrelated to this issue here.

[pmusaraj]

Ok, I'll try to repro this directly without the shim and report back.

[ntkme]

@pmusaraj Can you please provide a ps output with either with forest output or with PPID and parent process information so that we can see if they are leaked by a single ruby process or something else is happening?

One suspicion I have is that I see discourse runs puma, so it might be that puma is spinning multiple ruby processes and each ruby process is running a dart-sass compiler.

[ntkme]

This should have been address in main branch:

1. Previously, the global compiler is kept alive until ruby process shuts down. Now, global compiler process is automatically closed after idle for 10s and restarted when needed.
2. Previously, Process.fork was not handled properly, that the global compiler's stdio file descriptors were duplicated and leaked, which prevents the compiler process to close until the forked process shuts down. Now, these duplicated file descriptors are cleaned up immediately after fork.

Overall using a persisted compiler would perform better than restarting the compiler for every single compilation. Therefore I would like to revert sass-contrib/sassc-embedded-shim-ruby#68 once these changes are out.

[pmusaraj]

Just wanted to follow up and say that we just switched Discourse to using sassc-embedded and the updated version of this repo, in initial testing we no longer see leaks.

Appreciate the quick and excellent work here @ntkme! Thanks!

[ntkme]

@pmusaraj Can you please have a try with sassc-embedded ~> 1.69 and sass-embedded >=1.69.6? This combination has the enhancements described in #173 (comment).

[pmusaraj]

@ntkme, we've rolled out 1.69 and are not seeing any issues. Thanks again for your work here!