Error 500 on OpenShift/okd.io with MariaDB related to passphrase field

[Grommingen]

Checklist

• I've searched the project's issues.
• I've searched the project's discussions.

❓ Question

I filed this as a question as i don't know yet if its a bug or not.

I have deployed pwpush(i want to run on v.1.37.10) in our okd4 stage environment. When i goto the URL exposed by my ingress i get a 500 error.
The pod logs say:

[cb6c7539-f9e4-4a87-822a-53dd5c70575b] method=GET path=/ format=*/* controller=PasswordsController action=new status=500 allocations=1457 duration=5.57 view=0.00 db=0.00 ip=myip forwarded_for=myip
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]   
[cb6c7539-f9e4-4a87-822a-53dd5c70575b] ActionView::Template::Error (Decryption failed):
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]     125:                 <div class='col'>
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]     126:                     <div class="input-group">
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]     127:                         <span class="input-group-text"><%= _('Passphrase Lockdown') %></span>
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]     128:                         <%= f.text_field(:passphrase, { class: "form-control",
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]     129:                                                         autocomplete: "off",
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]     130:                                                         placeholder: _('Optional: Require recipients to enter a passphrase to view this push') }) %>
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]     131:                     </div>
[cb6c7539-f9e4-4a87-822a-53dd5c70575b]   
[cb6c7539-f9e4-4a87-822a-53dd5c70575b] app/views/passwords/new.html.erb:128
[cb6c7539-f9e4-4a87-822a-53dd5c70575b] app/views/passwords/new.html.erb:32
[cb6c7539-f9e4-4a87-822a-53dd5c70575b] app/controllers/application_controller.rb:18:in `custom_set_locale_from_url'

I've generated PWPUSH_MASTER_KEY and SECRET_KEY_BASE and they are in encoded format as a secret. I've also tried it without these secrets but that did not help either. What does this custom_set_locale_from_url mean? Does it have anything to do with the master and secret keys?

I've tried to bump the versions we use but it does not help.
Bumped versions to 1.40.0 and 1.42.0.

br Bjørn-Ivar Høibye - Redpill Linpro

[github-actions]

Hello @Grommingen, thanks for contributing to the Password Pusher community! We will respond as soon as possible.

[pglombardo]

Hi @Grommingen,

This is most likely a problem in the encryption area. I see two possibilities:

1. Are you using a pre-existing database and changed the encryption key?
2. Is the format of the encryption key correct? Did you use this page to generate a new key?

Also - which DB backend are you using?

What does this custom_set_locale_from_url mean? Does it have anything to do with the master and secret keys?

Unrelated - this is just a wrapper that sets the language for the page to be displayed.

Also v1.40.x is old. There have been some CVE's filed against older releases. Could you try latest v1.49.0?

[Grommingen]

Thanks for the reply!

1. No, but i've tried changing the keys several times. I just recreated the database after i turned of the openshift deployment and fed it the same PWPUSH_MASTER_KEY. I can Also see it in the pod when i do Lockbox.master_key command.
2. Yes. I've followed the guide on that page.

The database backend is mysql. Our production pwpush is running on the same backend, albeit a different database.
I've tried to remove characters as described here in the URL to no avail.

My version is now running 1.49.0 and the error shows:

D, [2024-11-27T11:43:26.127739 #24] DEBUG -- : [34326619-8e88-42ff-b2a9-9406132445d3]   Rendering layout layouts/application.html.erb
D, [2024-11-27T11:43:26.127802 #24] DEBUG -- : [34326619-8e88-42ff-b2a9-9406132445d3]   Rendering passwords/new.html.erb within layouts/application
I, [2024-11-27T11:43:26.130684 #24]  INFO -- : [34326619-8e88-42ff-b2a9-9406132445d3] method=GET path=/ format=html controller=PasswordsController action=new status=500 allocations=2007 duration=4.12 view=0.00 db=0.00 user_id= ip=<myip> forwarded_for=<myip>
E, [2024-11-27T11:43:26.136895 #24] ERROR -- : [34326619-8e88-42ff-b2a9-9406132445d3]   
[34326619-8e88-42ff-b2a9-9406132445d3] ActionView::Template::Error (Decryption failed):
[34326619-8e88-42ff-b2a9-9406132445d3] 
Causes:
[34326619-8e88-42ff-b2a9-9406132445d3] Lockbox::DecryptionError (Decryption failed)
[34326619-8e88-42ff-b2a9-9406132445d3] Lockbox::DecryptionError (Decryption failed)
[34326619-8e88-42ff-b2a9-9406132445d3]     119:                 <div class='col'>
[34326619-8e88-42ff-b2a9-9406132445d3]     120:                     <div class="input-group">
[34326619-8e88-42ff-b2a9-9406132445d3]     121:                         <span class="input-group-text"><%= _('Passphrase Lockdown') %></span>
[34326619-8e88-42ff-b2a9-9406132445d3]     122:                         <%= f.text_field(:passphrase, { class: "form-control",
[34326619-8e88-42ff-b2a9-9406132445d3]     123:                                                         autocomplete: "off",
[34326619-8e88-42ff-b2a9-9406132445d3]     124:                                                         placeholder: _('Optional: Require recipients to enter a passphrase to view this push') }) %>
[34326619-8e88-42ff-b2a9-9406132445d3]     125:                     </div>
[34326619-8e88-42ff-b2a9-9406132445d3]   
[34326619-8e88-42ff-b2a9-9406132445d3] app/views/passwords/new.html.erb:122
[34326619-8e88-42ff-b2a9-9406132445d3] app/views/passwords/new.html.erb:26
[34326619-8e88-42ff-b2a9-9406132445d3] app/controllers/concerns/set_locale.rb:9:in `set_locale'

So the new version now also mentions lockbox as a cause. The keys, both PWPUSH_MASTER_KEY and SECRET_KEY_BASE are created and base64 encoded before put into a secret in the namespace. I can also clearly see it in the pod when i do the env command.

[pglombardo]

Hrm that's an odd one. Open an app console and paste in this:

Lockbox.master_key

That will tell us if the booted application has the right master key.

[Grommingen]

oc rsh passwordpusher-856947767-6zldn /bin/bash
passwordpusher-856947767-6zldn:/opt/PasswordPusher$ ./bin/rails c
Password Pusher Version: 1.49.0


Type 'jobs_help' to see how to connect to the available job servers to manage jobs

Loading production environment (Rails 7.2.2)
password-pusher(prod)> Lockbox.master_key
=> "<mycorrectmasterkey>"
password-pusher(prod)>

Verified its the key that i've used in my secrets.

I should also mention that the API works. i can send a POST to the endpoint /p.json "password[payload]=mypassword"
This gives me a url_token which i can create a url to retrieve the password en/p/<url_token>

[pglombardo]

I should also mention that the API works. i can send a POST to the endpoint /p.json "password[payload]=mypassword"
This gives me a url_token which i can create a url to retrieve the password en/p/<url_token>

And you retrieve the payload without issue?

Then it's something with the passphrase. Give me some time to give it some thought.

Could you try a POST with this payload?

/p.json "password[payload]=mypassword,password[passphrase]=asdf"

[Grommingen]

curl -X POST --data "password[payload]=mypassword&password[passphrase]=asdf" \
      https://<myurl>/p.json

Generates a password and the passphrase to unlock it = asdf works.

[Grommingen]

Logs from the api POST incase its relevant:

[lockbox] Decrypting previous value failed
D, [2024-11-27T12:42:46.044570 #25] DEBUG -- : [b8ae2800-b8ea-4889-9a01-45357dc439d7]   TRANSACTION (1.7ms)  BEGIN
D, [2024-11-27T12:42:46.046289 #25] DEBUG -- : [b8ae2800-b8ea-4889-9a01-45357dc439d7]   Password Create (3.4ms)  INSERT INTO `passwords` (`expire_after_days`, `expire_after_views`, `expired`, `url_token`, `created_at`, `updated_at`, `user_id`, `deleted`, `deletable_by_viewer`, `retrieval_step`, `expired_on`, `payload_ciphertext`, `note_ciphertext`, `passphrase_ciphertext`) VALUES (1, 1, FALSE, '50rtrb89bjbfga', '2024-11-27 12:42:46', '2024-11-27 12:42:46', NULL, FALSE, TRUE, TRUE, NULL, '1eOb6ctUvcHSuH/a434RtemYK0iE95XPBEcPLSVW0/sVoHKgU+I=', '', '') RETURNING `id`
D, [2024-11-27T12:42:46.056188 #25] DEBUG -- : [b8ae2800-b8ea-4889-9a01-45357dc439d7]   TRANSACTION (8.8ms)  COMMIT
D, [2024-11-27T12:42:46.060924 #25] DEBUG -- : [b8ae2800-b8ea-4889-9a01-45357dc439d7]   View Count (2.1ms)  SELECT COUNT(*) FROM `views` WHERE `views`.`password_id` = 9 AND `views`.`kind` = 0
I, [2024-11-27T12:42:46.061358 #25]  INFO -- : [b8ae2800-b8ea-4889-9a01-45357dc439d7] method=POST path=/p.json format=json controller=PasswordsController action=create status=201 allocations=1727 duration=33.70 view=2.26 db=17.80 user_id= ip=<myip> forwarded_for=<myip>

[pglombardo]

Honestly I have no idea at the moment why it would throw that error only on UI page rendering - and only with the passphrase field. Higher up in the page is the payload field and it rendered without error.

Give me a bit to think about it - I'll post back soon.

[Grommingen]

I've tried spinning up an ephemeral deployment and that works. Just for the record.

[pglombardo]

So then I would suspect it has something to do with the MySQL database but then the JSON API works...which doesn't make sense.

Have you enabled file and URL pushes? Could you try these paths to see if you get the same error?

• /r/new
• /f/new

[Grommingen]

After enabling:

      containers:
      - env:
        - name: PWP__ENABLE_FILE_PUSHES
          value: "true"
        - name: PWP__ENABLE_URL_PUSHES
          value: "true"
        - name: PWP__ENABLE_LOGINS
          value: "true"

/r/new and /f/new works. When going to Passwords: /p/new on the same window, a 500 error appears.

[pglombardo]

😂 Twilight zone! Those other paths do the exact same thing... Thanks for the confirmation I'll keep digging.

[pglombardo]

I'll throw this out there because I don't have many ideas right now. It's a long shot but you could drop the db and recreate the database.

Open an app console again and run:

rails db:drop db:create db:migrate

You might have to use ./bin/rails instead of the above doesn't work.

This of course will delete all data (and logins) that you created. Again this is a hail mary until I figure out what's happening.

I've never seen this and never had another report of this. Thanks for being patient!

[Grommingen]

I have done the ./bin/rails commands which deleted and created the database again however it did not help.

I've also created another mariadb instance with the newest stable version(11.4) and that did not help either.

Thanks for looking into this @pglombardo

br

[Grommingen]

Hi again!

I've tried with a postgres backend now and it works. So the issue may only affect mysql/mariadb.

br Grommingen

[pglombardo]

Great find! It crossed my mind that this might be a MySQL related issue as I've had other MySQL oddities a couple years back.

I'll launch a fresh MySQL deploy hopefully today to figure out what's different.

[pglombardo]

In fact, this was it. I put in a special handler for passphrase for MySQL.

This is the code. Not sure why it wouldn't affect file based pushes though.

What version of MySQL are you running?

[Grommingen]

mysql and mariadb are interchangeable? I've tried with both mariadb 10.7 and 11.4(our latest stable version).

I can spin up a mysql instance and see if there is any difference but i want it to run on mariadb because thats what our current prod pwpush instance is using.

[Grommingen]

Spun up a mysql db instance and it works, got the message from the code you pasted as well.

[pglombardo]

Apologies - I was AFK. So that means it's only a MariaDB thing? I'll test with that and see if I can find the cause. It has to be something with that passphrase column.

[Grommingen]

Yeah, my assumption is that this only affects MariaDB. When i did a migrate just now the messages about passphrase says:

Migrating to FixPassphraseDefault (20230420175410)
== 20230420175410 FixPassphraseDefault: migrating =============================
-- change_column_null(:passwords, :passphrase_ciphertext, true, nil)
   -> 0.0547s
-- change_column_null(:file_pushes, :passphrase_ciphertext, true, nil)
   -> 0.0557s
-- change_column_null(:urls, :passphrase_ciphertext, true, nil)
   -> 0.0678s
-- change_column_default(:passwords, :passphrase_ciphertext, nil)
   -> 0.0286s
-- change_column_default(:file_pushes, :passphrase_ciphertext, nil)
   -> 0.0288s
-- change_column_default(:urls, :passphrase_ciphertext, nil)
   -> 0.0275s
== 20230420175410 FixPassphraseDefault: migrated (0.2635s) ====================

[pglombardo]

I just booted docker-compose-mariadb.yml but first I updated the image to mariadb:11.4

And it works - including pushing a payload with a passphrase:

I had a pre-existing MySQL/MariaDB database so I opened the container console and ran rails db:drop db:create db:migrate:

and it still worked.

You could try and re-fetch the latest hashes for your images but I think you already did this...

docker pull pglombardo/pwpush:latest
docker pull mariadb:11.4

Not sure what could be different locally for you. Just wanted to post this for now until I get an idea.

[Grommingen]

I am not doing this locally, its all on openshift(okd4).

I can try it locally as well, i only tested ephemeral locally before.

But i've tested latest docker image and mariadb 11.4 instance on openshift.

I can't test anymore today but i'l look into it when i have spare time.

[pglombardo]

I can't test anymore today but i'l look into it when i have spare time.

Understood - when you have more time here's one more brute force test.

Get a shell into the container and add the following line of code to app/controllers/passwords_controller.rb on line 116:

@push.passphrase = ""

Here's a screenshot of the modified file:

That may fix it and if it works I'll put out a special fix just for this issue.

[Grommingen]

I've added the line but it does not seem to help. I did a ./bin/update and ./bin/setup as well to see if it helped. same message.

passwordpusher-58574b8b9b-7ggfn:/opt/PasswordPusher$ cat app/controllers/passwords_controller.rb |grep -i 'def new' -C 10
    else
      # Passphrase is invalid
      # Redirect to the passphrase page
      flash[:alert] =
        _('That passphrase is incorrect.  Please try again or contact the person or organization that sent you this link.')
      redirect_to passphrase_password_path(@push.url_token)
    end
  end

  # GET /passwords/new
  def new
    # Require authentication if allow_anonymous is false
    # See config/settings.yml
    authenticate_user! if Settings.enable_logins && !Settings.allow_anonymous

    @push = Password.new
    @push.passphrase = ""

    respond_to(&:html)
  end

[Grommingen]

Good monday @pglombardo !

I am sorry to say that my previous statement is not correct.

I implemented this fix on version 1.37.0 first, this did not work.
Then i changed the container image to :latest (1.49.1) and voila, the page loads.

So your proposed solution/fix will probably work in my case.

br Grommingen

[Grommingen]

The latest version 1.49.3 implemented the wanted solution. The fix is a success and our production instance with mariadb as a backend is now running. Thanks for your assistance @pglombardo and have a nice Christmas Holiday!

[Grommingen]

reopening :(

More to come.

[pglombardo]

Hi @Grommingen - try v1.49.4 which is building now. It should be done in about 45 minutes.

[Grommingen]

thanks, il try that one too. It says

E, [2024-12-13T10:53:24.498556 #22] ERROR -- : [272bdfce-5e4a-4618-a8eb-9bb1348b1bd1]
[272bdfce-5e4a-4618-a8eb-9bb1348b1bd1] Errno::EROFS (Read-only file system @ dir_s_mkdir - /opt/PasswordPusher/tmp/cache/104):
[272bdfce-5e4a-4618-a8eb-9bb1348b1bd1]

now but i dont know why yet.

[Grommingen]

@pglombardo False alarm. I've mounted /tmp as a volume in okd. I saw a permission denied on /opt/passwordpusher/tmp and changed the mount to that instead and then it worked. I am guessing this is a change from long ago that we've yet to correct on our installation.

Its all running now again.

[MindTooth]

I saw from the code that you can have a /up endpoint for running a health check? Is that something that must be manually activated?

The issue was with the livenessProbe (touching /tmp), I want to see if we can improve it simultaneous?

[pglombardo]

No I haven't added that to the OSS version yet but it a good feature request. Filed in #2886.