avm1: Fix Object < Object comparison

[Herschel]

{} < {} should return false, but was incorrectly returning undefined in Ruffle. Stumbled on this while trying to add a test for #6544.

If the parameters to the LessThan2 action are a non-movieclip object, Object.valueOf is called. If the result of valueOf is a non-movieclip Object, the comparison immediately returns false. This is the usual case because Object.prototype.valueOf returns this object itself.

In the other case where valueOf returns a primitive value or a display object, the comparison continues as normal, coercing the value to number, where any NaN results in the comparison returning undefined.

• Change Value::to_primitive_num to not call valueOf on display objects.
• Change abstract_lt to return false immediately if the result of to_primitive_num is an object (excluding display objects).
• Unignore string_coercion test as this now works properly.
• Add various checks for object < object to the lessthan2 tests.
• Adds a test for avm1: String(function) returns [type Function] #6544

[Toad06]

Tested the PR and it looks good to me.

Though this is not really in the scope of this PR, I've found another issue with valueOf when used with the Add2 action:

var o = new Object();
o.valueOf = function()
{
   return "ABC";
};
trace(123 + o); // NaN (expected: "123ABC")

[relrelb]

Seems like if both operands are plain objects, then both valueOf callbacks should be invoked:

var a = { valueOf: function() { trace("in a.valueOf"); } };
var b = { valueOf: function() { trace("in b.valueOf"); } };
trace(a < b); // traces "a.valueOf", "b.valueOf"
trace(b < a); // traces "b.valueOf", "a.valueOf"

If I get it right, Ruffle with this PR would bail-out early and invoke only the first valueOf.
So I think the correct order is something like this:

1. a.to_primitive_num()
2. b.to_primitive_num()
3. if prim_self is an object, return false
4. if prim_other is an object, return false

[Herschel]

The PR runs this example correctly -- the valueOfs in this case return undefined and not an Object, so the early-exit doesn't happen.

The early-exit does happen if you add return {} to the valueOf calls.

[relrelb]

Not directly related to this PR, but I think StageObject should be a separate enum member of Value rather than a "subclass" of Value::Object, since I have seen this special case exclusion too many times. This probably means that StageObject won't be able to implement TObject anymore, so it's not trivial at all, but I still have a feeling that StageObject and Object are more different than they have in common. Maybe this worth re-thinking as part of #5492.

[Herschel]

I had the same thought while writing this PR. This would fit nicely in with #5492.