Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError since 2.5 when using MIDI controller, caused by deepMerge() in applyLayer() #14197

Closed
git-developer opened this issue Jan 19, 2025 · 32 comments

Comments

@git-developer
Copy link
Contributor

Bug Description

Problem

The following error is raised since Mixxx 2.5 when using a certain controller mapping:

Uncaught exception: file:///home/christian/.mixxx/controllers/common-controller-scripts.js:187:
TypeError: Cannot assign to read-only property "id"

Backtrace: @file:///home/christian/.mixxx/controllers/common-controller-scripts.js:187
deepMerge@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:180
@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:185
deepMerge@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:180
deepMerge@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:176
@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:185
deepMerge@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:180
@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:185
deepMerge@file:///home/christian/.mixxx/controllers/common-controller-scripts.js:180
applyLayer@file:///home/christian/.mixxx/controllers/midi-components-0.0.js:664
shift@file:///home/christian/.mixxx/controllers/Behringer-Extension-scripts.js:1115
@file:///home/christian/.mixxx/controllers/midi-components-0.0.js:614
applyOperationTo@file:///home/christian/.mixxx/controllers/midi-components-0.0.js:536
forEachComponent@file:///home/christian/.mixxx/controllers/midi-components-0.0.js:548
shift@file:///home/christian/.mixxx/controllers/midi-components-0.0.js:593
inSetValue@file:///home/christian/.mixxx/controllers/Behringer-Extension-scripts.js:154
input@file:///home/christian/.mixxx/controllers/midi-components-0.0.js:184
input@file:///home/christian/.mixxx/controllers/Behringer-Extension-scripts.js:1143
input@file:///home/christian/.mixxx/controllers/Behringer-Extension-scripts.js:1310
@:1

Analysis

The problem

  • does not occur in 2.4.
  • does no longer occur in 2.5 if midi-components-0.0.js and common-controller-scripts.js are replaced by the revisions of 2.4.
  • also occurs in 2.4 if midi-components-0.0.js and common-controller-scripts.js are replaced by the revisions of 2.5.
  • can be narrowed down to single a change in applyLayer() of midi-components-0.0.js: 2.4 uses _.merge() (from lodash), 2.5 uses the newly introduces deepMerge() function. Reverting this change solves the problem.

How to reproduce

  1. Unzip the files Behringer DDM4000.midi.xml and Behringer-DDM4000-scripts.js from the attached test case in ~/.mixxx/controllers/
  2. Configure a MIDI controller to use this mapping.
  3. Send MIDI command [0x90, 0x01] twice.

The attached file deepmerge-issue.log contains a log from this test case.

Additional Details

  • The test case was created from a custom mapping for Behringer DDM4000 mixer. It uses a Shift Button to switch between layers. For example, button P1 is usually configured to roll a beatloop , but as long as the Shift Button P2 is held down, P1 triggers the first quick effect. The error occurs when the Shift Button is pressed, i.e. when the layer is changed.
  • The DDM4000 mapping is working as expected when the deepMerge() change is reverted.
  • The reported read-only property "id" belongs to a connection object:
    {
    "group": "[QuickEffectRack1_[Channel1]_Effect1]",
    "key": "button_parameter1",
    "type": 2,
    "isShifted": false,
    "connections": [
        {
        "objectName": "",
        "id": "{7fcd247a-e284-4eef-b512-0796dcd5f827}",
        "isConnected": false
        }
    ],
    "inKey": "enabled",
    "outKey": "enabled",
    "midi": [
        144,
        2
    ]
    }

@christophehenry might be interested.

Version

2.5.0-1~noble

OS

Ubuntu 24.04

@Swiftb0y
Copy link
Member

@christophehenry are you interested in providing a quick fix? I'm afraid I don't have the time to do it myself.

@christophehenry
Copy link
Contributor

christophehenry commented Jan 19, 2025

I'm not sure I am able to provide a fix quickly but I will fix my mess I soon as I can for sure.

@git-developer
Copy link
Contributor Author

I prepared a quick fix in #14203, just for the case that a simple revert would be acceptable. I cannot evaluate whether this is appropriate, though, and thus marked it as draft.

It might also be possible to use Object.assign() in favor of _.merge(), but I don't know the exact syntax and cannot overlook the effects, might be too risky. Maybe as part of removing lodash, towards 2.6?

Just for the reference:

@christophehenry
Copy link
Contributor

christophehenry commented Jan 20, 2025

I'm OK with reverting until a better fix is written. This change was here to deprecate Lodash. Removal is not happening any time soon so revert and a fix-release seems to be the best solution.

Object.assign was discussed but _.merge does a deep merging which may be necessary but we weren't sure.

@Swiftb0y
Copy link
Member

reverting is not an option because using lodash creates loads of warnings. Object.assign won't work either because it has different semantics, probably breaking other mappings. Object.getOwnPropertyDescriptor().writeable is likely what we need.

@christophehenry
Copy link
Contributor

I will dig into Lodash's code when I have time to understand what I missed.

@git-developer
Copy link
Contributor Author

reverting is not an option because using lodash creates loads of warnings

I understand and agree that removing lodash is a worthwile goal. As far as I know, is was deprecated right after release of 2.5 (in current main). deepMerge() appears to be deprecated since the same time, so I don't understand why deepMerge() should be preferred over lodash.

I'm worried about the fact that any quick fix which is not a revert adds a risk of introducing unexpected behavior.

@christophehenry
Copy link
Contributor

deepMerge() appears to be deprecated since the same time, so I don't understand why deepMerge() should be preferred over lodash.

As I said, Object.assign should be used here but _.merge does a deep merging. We weren't sure that a controller wasn't relying on deep merging so deepMerge() was written to replace _.merge but immediately marked as deprecated so that it wouldn't be used elsewhere. In time I think deepMerge() should be removed altogether and replaced by Object.assign. I don't really see the use case for deep merging here.

@git-developer
Copy link
Contributor Author

Thanks for the explanation, I think I understand now.

This issue might be an indicator that such a use case for deep merging exists. We will know for sure when the root cause is found and fixed.

Hypothesis: If deep merging is not required, deepMerge() can safely replaced with Object.assign() (shallow merge) in applyLayer(). Do you agree?

@christophehenry
Copy link
Contributor

If deep merging is not required, deepMerge() can safely replaced with Object.assign() (shallow merge) in applyLayer(). Do you agree?

Yes but we don't know for sure whether someone relies on that deep merging behavior of not.

@christophehenry
Copy link
Contributor

christophehenry commented Jan 22, 2025

@git-developer Can you try after applying the following diff?

diff --git a/res/controllers/common-controller-scripts.js b/res/controllers/common-controller-scripts.js
index fcdf516a58..7954ce15f7 100644
--- a/res/controllers/common-controller-scripts.js
+++ b/res/controllers/common-controller-scripts.js
@@ -184,7 +184,16 @@ script.deepMerge = function(target, source) {
               isSimpleObject(target[key]) && isSimpleObject(source[key])
             ) {
                 deepMerge(target[key], source[key]);
-            } else if (source[key] !== undefined && source[key] !== null) {
+            } else if (
+                (
+                    source[key] !== undefined &&
+                    !(
+                        source[key] === source[key] ||
+                        (source[key] !== source[key] && source[key] !== source[key])
+                    )
+                ) ||
+                (source[key] === undefined && !(key in target))
+            ) {
                 Object.assign(target, {[key]: source[key]});
             }
         });

@Swiftb0y I think the problem comes from that Lodash prevents doing useless assignation while deepMerge always does it for simplicity. I adapted the code from assignMergeValue. Can you double check the code, since it is difficult to test.

@git-developer
Copy link
Contributor Author

Thanks for your time and willingness to help.

With your patch, the TypeError no longer occurs. That's good news. Unfortunately, the behavior still differs from the previous implementation: applyLayer() seems to have no effect now.

I need some time to investigate. I'l check the difference of the result of applyLayer() between 2.4 and your patched implementation.

@Swiftb0y
Copy link
Member

Thank you both for looking into the issue. Looking forward to a fix.

@git-developer, alternatively to relying on setLayer() (and in turn deepMerge()), you could also instance both versions of your container (the shifted and the unshifted one) and then use connect() and disconnect() along with assigning the appropriate instance. Does that make sense?

@git-developer
Copy link
Contributor Author

From a first test:

  • container.applyLayer(newLayer) is called
  • container is an empty object ({})
  • newLayer is listed further down.

When using the patch above, the result is empty. The expected result is newLayer, obviously. Maybe the JSON helps for a test setup.

newLayer:
{
  "[0x90, 0x03]": {
    "buttons": [
      {
        "group": "[EffectRack1_EffectUnit1]",
        "midi": [
          176,
          56
        ],
        "effectUnit": 1,
        "key": "group_[Channel1]_enable",
        "isShifted": false,
        "connections": [
          {
            "objectName": "",
            "id": "{ddb6d292-527d-44c8-8f88-cc1caa188858}",
            "isConnected": true
          }
        ],
        "inKey": "group_[Channel1]_enable",
        "outKey": "group_[Channel1]_enable"
      },
      {
        "group": "[EffectRack1_EffectUnit2]",
        "midi": [
          176,
          55
        ],
        "effectUnit": 2,
        "key": "group_[Channel1]_enable",
        "isShifted": false,
        "connections": [
          {
            "objectName": "",
            "id": "{1169d423-cc69-4447-98d5-1fe26f2f4d06}",
            "isConnected": true
          }
        ],
        "inKey": "group_[Channel1]_enable",
        "outKey": "group_[Channel1]_enable"
      }
    ],
    "midi": [
      144,
      3
    ],
    "midiAddresses": [
      [
        176,
        56
      ],
      [
        176,
        55
      ]
    ],
    "group": "[Channel1]",
    "longPressTimeout": 500,
    "isShifted": false,
    "connections": [],
    "longPressTimer": {
      "timeout": 500,
      "oneShot": true,
      "owner": "[object Object]",
      "id": 0
    },
    "isLongPressed": false,
    "types": {
      "push": 0,
      "toggle": 1,
      "powerWindow": 2
    },
    "type": 0,
    "on": 127,
    "off": 0,
    "shiftChannel": true,
    "shiftOffset": 32,
    "outConnect": true,
    "outTrigger": true,
    "max": 127,
    "sendShifted": false,
    "shiftControl": false
  },
  "[0x90, 0x00]": {
    "midi": [
      144,
      0
    ],
    "inKey": "beatlooproll_activate",
    "group": "[Channel1]",
    "isShifted": false,
    "connections": []
  },
  "[0x90, 0x01]": {
    "midi": [
      144,
      1
    ],
    "inKey": "reverseroll",
    "group": "[Channel1]",
    "isShifted": false,
    "connections": []
  },
  "[0x90, 0x02]": {
    "group": "[QuickEffectRack1_[Channel1]_Effect1]",
    "key": "enabled",
    "type": 2,
    "isShifted": false,
    "connections": [
      {
        "objectName": "",
        "id": "{7f4a7529-f8cc-476a-b3bd-ee09eb51ee64}",
        "isConnected": true
      }
    ],
    "inKey": "enabled",
    "outKey": "enabled",
    "midi": [
      144,
      2
    ]
  },
  "[0xB0, 0x05]": {
    "midi": [
      176,
      5
    ],
    "inKey": "beatloop_size",
    "group": "[Channel1]",
    "values": [
      0.03125,
      0.0625,
      0.125,
      0.25,
      0.5,
      1,
      2,
      4,
      8,
      16,
      32,
      64,
      128,
      256,
      512
    ],
    "softTakeover": true,
    "maxIndex": 14,
    "isShifted": false,
    "connections": []
  },
  "[0xB0, 0x06]": {
    "group": "[QuickEffectRack1_[Channel1]]",
    "key": "super1",
    "isShifted": false,
    "connections": [],
    "inKey": "super1",
    "outKey": "super1",
    "firstValueReceived": false,
    "midi": [
      176,
      6
    ]
  }
}

@git-developer
Copy link
Contributor Author

instance both versions of your container (the shifted and the unshifted one) and then use connect() and disconnect() along with assigning the appropriate instance. Does that make sense?

Thanks for your idea. I have to think about that. The LayerManager is already structured like that, but it's calling applyLayer() instead of manual disconnect/connect. The docs make me think this is the purpose of applyLayer().

A ComponentContainer is prepared to contain Components (obviously) and ComponentContainers. This might be the reason why deep merge is required. applyLayer() performs an optional disconnect, merge, and optional connect, and all 3 tasks are set up to work recursively (because forEachComponent handles both Components and ComponentContainers).

@christophehenry
Copy link
Contributor

christophehenry commented Jan 23, 2025

@git-developer Dammit I messed up my code replacement. Damn ADHD… I was more careful this time so I hope this next diff works. Can you confirm?

diff --git a/res/controllers/common-controller-scripts.js b/res/controllers/common-controller-scripts.js
index fcdf516a58..2a0cdf6493 100644
--- a/res/controllers/common-controller-scripts.js
+++ b/res/controllers/common-controller-scripts.js
@@ -184,7 +184,16 @@ script.deepMerge = function(target, source) {
               isSimpleObject(target[key]) && isSimpleObject(source[key])
             ) {
                 deepMerge(target[key], source[key]);
-            } else if (source[key] !== undefined && source[key] !== null) {
+            } else if (
+                (
+                    source[key] !== undefined &&
+                    !(
+                        target[key] === source[key] ||
+                        (target[key] !== target[key] && source[key] !== source[key])
+                    )
+                ) ||
+                (source[key] === undefined && !(key in target))
+            ) {
                 Object.assign(target, {[key]: source[key]});
             }
         });

@git-developer
Copy link
Contributor Author

This patch reintroduces the TypeError.

@christophehenry
Copy link
Contributor

Ok. So I'm lost, here… In a normal situation I'd run the code step by step in the browser, but QJSEngine doesn't offer any way to plug a debugger, AFAIK. @Swiftb0y I'd say, since the code is broken anyway, I'd suggest we replace it with Object.assign immediately. Can't be worse.

@git-developer
Copy link
Contributor Author

No panic. Instead of changing the recently introduced code with something untested and new, I suggest to revert to lodash for the quick fix. It is known to work properly and thus reduces the risk of defects. We have then more time to fix / refactor the hand-writtten merge algorithm, targeting 2.6.

@Swiftb0y
Copy link
Member

I'd suggest we replace it with Object.assign immediately. Can't be worse.

In the case of ComponentContainer.applyLayer()? Sure... I agree, can't be worse and the scope is likely limited. @git-developer have you tried doing that in your mapping? What are the results?

I suggest to revert to lodash for the quick fix. It is known to work properly and thus reduces the risk of defects. We have then more time to fix / refactor the hand-writtten merge algorithm, targeting 2.6.

That assumes all the midi-components mappings still import lodash in 2.5. I'm sure whether thats the case.

git-developer added a commit to git-developer/mixxx that referenced this issue Jan 24, 2025
@git-developer
Copy link
Contributor Author

have you tried doing that in your mapping? What are the results?

Thanks for this suggestion! Just tried it, and the results of a first test look very good. The TypeError has gone, and the behavior is as expected. I updated #14203 accordingly. I plan to do some more tests, just to be safe.

I still can't tell for sure if a recursive merge might be required in applyLayer(), but to be honest I'm focused on this issue. I think Object.assign() would be a perfect fix for everyone here: no lodash, no custom deep merge and no regression.

@christophehenry
Copy link
Contributor

christophehenry commented Jan 24, 2025

Hmm… Wait, re-reading the code, I notice that we deprecated applyLayer in favor of setLayer exactly for this reason: setLayer uses Object.assign. So no need to change the code: just use setLayer.

In the meantime, what I'd do is add a try/catch to script.deepMerge. This is inefficient I agree but since this code shouldn't be used anyway…

Edit: or I could just copy and unfold the code from Lodash so that it fits in a single function. It will be a spaghetti mess but since we want to remove this code eventually, I don't think it's a big deal if it's unreadable and unmaintaintable.

@git-developer
Copy link
Contributor Author

git-developer commented Jan 24, 2025

setLayer() is available only in main, not in maintenance branch 2.5.

I see no advantage in keeping deepMerge() when Object.assign() works (restricting to applyLayer()). try-catch did not work (see #14206).

@christophehenry
Copy link
Contributor

Wait… There's a problem in this code anyway… This and this should be script.deepMerge, not just deepMerge. How come this code hasn't exploded before?

@git-developer
Copy link
Contributor Author

Maybe not detected due to missing tests?

@git-developer
Copy link
Contributor Author

Looks like I'm the first and only one who ever called applyLayer() 🤷

@Swiftb0y
Copy link
Member

Looks like I'm the first and only one who ever called applyLayer() 🤷

almost, yeah 😅

$ grep -Hirn applyLayer
Behringer-Extension-scripts.js:1114:            this.activeLayer.applyLayer(this.shiftLayer());
Behringer-Extension-scripts.js:1124:            this.activeLayer.applyLayer(this.defaultLayer());
midi-components-0.0.js:522:            this.applyLayer(initialLayer);
midi-components-0.0.js:653:        applyLayer: function(newLayer, reconnectComponents) {

@christophehenry
Copy link
Contributor

The one usage in midi-components-0.0.js:522 is called before any connection is made so it should not trigger this issue.

@Swiftb0y
Copy link
Member

yeah, I'm just worried about the Behringer stuff. I think those "extension scripts" are used by numerous downstream mappings.

@git-developer
Copy link
Contributor Author

I don't think so. I'm the author of the Behringer-Extension-scripts.js and these two mappings built on top:

$ grep -Hirn Behringer-Extension-scripts
res/controllers/Behringer BCR2000.midi.xml:12:      <file filename="Behringer-Extension-scripts.js" />
res/controllers/Behringer DDM4000.midi.xml:12:      <file filename="Behringer-Extension-scripts.js" />

The other Behringer mappings are independent as far as I know.

I just made another test. Everything works as expected with Object.assign() instead of script.deepMerge(). I thus marked #14203 as ready for review.

@git-developer
Copy link
Contributor Author

I restructured the PRs to keep things sorted:

Swiftb0y added a commit that referenced this issue Jan 28, 2025
Quick fix for #14197 (`TypeError` in `midi-components-0.0.js`)
Swiftb0y added a commit that referenced this issue Jan 28, 2025
Quick fix for #14197 (`TypeError` in `midi-components-0.0.js`)
@git-developer
Copy link
Contributor Author

Thanks for merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants