[BUG] ExoPlayer uses SurfaceView by default, needs way to optionally use TextureView to fix glitchy behavior and extend usage to transparent background

[jonmdev]

Is there an existing issue for this?

• I have searched the existing issues

Did you read the "Reporting a bug" section on Contributing file?

• I have read the "Reporting a bug" section on Contributing file: https://github.com/CommunityToolkit/Maui/blob/main/CONTRIBUTING.md#reporting-a-bug

Current Behavior

ISSUE

Regarding this bug here: #1730

I suspect the problem is that ExoPlayer uses SurfaceView by default. This is explained here:

https://proandroiddev.com/building-a-video-chat-app-webrtc-in-jetpack-compose-part2-69b4045b7cd3

Typically, you can use SurfaceViewRenderer to display real-time video streams on a layout which is plane structures or simple.

However, if you want to implement complicated layouts, such as one video track overlays another, you should figure out different ways. Let’s suppose you should implement a complex video call screen, such as one video call layout should overlay another video call layout like the image below:

In that case, the SurfaceViewRenderer doesn’t work as expected. There are two reasons that the SurfaceViewRenderer is not working correctly:

SurfaceViewRenderer is embedded inside a view hierarchy: SurfaceView lives on its plane, which means it essentially punches a hole in its window to display the content directly on the screen. Also, the surface is Z-ordered, so when you overlay multiple SurfaceViews, they can destroy each other, and Z-ordering may not work as expected.

This is also further explained here: https://medium.com/androiddevelopers/android-hdr-migrating-from-textureview-to-surfaceview-part-1-how-to-migrate-6bfd7f4b970e

TEXTURE VIEW

Based on this document, we can programmatically switch the ExoPlayer to use a TextureView instead which would then fix this:

https://stackoverflow.com/questions/68629114/use-textureview-programmatically-with-exoplayer

1) Create XML file

<com.xxx.xxx.PlayerView
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    app:surface_type="texture_view"
/>

2) Create attribute set

val xmlAttributes = context.resources.getXml(R.xml.player_view).let {
    try {
        it.next()
        it.nextTag()
    } catch (e: Exception) {
        // do something, log, whatever
    }
    Xml.asAttributeSet(it)
}

3) Set that in on construction:

PlayerView(context, xmlAttributes)

MAUI

Having the option to use TextureView would also let us perform special tasks like this: https://medium.com/go-electra/unlock-transparency-in-videos-on-android-5dc43776cc72

I can get the MauiMediaElement like: MauiMediaElement mauiMediElement = mediaElement.ToPlatform(mediaElement.Handler.MauiContext) as MauiMediaElement; but that is as deep as I can get.

This doesn't help either. As far as I can tell, we can't access the ExoPlayer directly by any means at all can we? Reflection?

Also, as far as I can tell, the only way to set it to TextureView is on construction which occurs here and we have no control over:

public (PlatformMediaElement platformView, StyledPlayerView PlayerView) CreatePlatformView()
	{
		ArgumentNullException.ThrowIfNull(MauiContext.Context);
		Player = new IExoPlayer.Builder(MauiContext.Context).Build() ?? throw new NullReferenceException();
		Player.AddListener(this);

		PlayerView = new StyledPlayerView(MauiContext.Context)
		{
			Player = Player,
			UseController = false,
			ControllerAutoShow = false,
			LayoutParameters = new FrameLayout.LayoutParams(ViewGroup.LayoutParams.MatchParent, ViewGroup.LayoutParams.MatchParent, GravityFlags.CenterHorizontal)
		};

https://github.com/CommunityToolkit/Maui/blob/ee03326ab43f35f90d27d92a37b9df06d6663f27/src/CommunityToolkit.Maui.MediaElement/Views/MediaManager.android.cs

I presume that would have to be where we would switch it to TextureView.

IDEAS?

Any way to get a TextureView ExoPlayer? Should I just copy/paste all the MediaElement code out into a new folder/namespace and edit that directly to try? I think that would be an absolute mess. I can't imagine that is the best way. Or can I create a new Handler or something else to override something?

How would the XML approach work in Android/MAUI in the context of the constructor above? After it is created, is there any way to access the ExoPlayer directly any way through this code? Or re-create it manually with a new constructor externally?

Thanks.

Expected Behavior

Should have a way to set as TextureView to prevent bugs and extend usage. Should have some way to access the ExoPlayer directly to extend platform specific fixes like transparent videos.

Steps To Reproduce

Play repo project linked in other issue report, see that it does not work correctly, likely due to SurfaceView as noted.

Link to public reproduction project repository

https://github.com/jonmdev/MediaElementOverlapBug

Environment

- .NET MAUI CommunityToolkit:
- OS:
- .NET MAUI:

Anything else?

Thanks for any ideas or suggestions. I don't mind doing work but I don't know where to start. I would like to get a reference to an ExoPlayer within MediaElement that is using a TextureView so I can fix the bug and try getting transparent video as well.

[pictos]

@jonmdev, you can take a look on the CameraView implementation for XCT, we used the TextureView there. The code part is here and here and xml part is here.

The easiest way will be to create your own mediaElement control and reuse our handler. For me this isn't a bug, since the MediaElement was designed to show videos and not provide features like call videos, as you mentioned on your report. So I'm changing this to be an enhancement.

[jonmdev]

Thanks for the info. I will try to do that. But also as noted, @pictos, it is the only way to fix the bug here:

#1730

Without a way to make the MediaElement work as a TextureView, we can't get it to crop/overlap correctly in Android like it does in iOS/Windows.

[ne0rrmatrix]

Wow. This looks like something I want to get involved in. :)
I will be looking into this and seeing if I can figure out a way to implement this. I will start in a few days. I am currently writing tests for another PR and as soon as that is done and PR is updated I will switch and to implementing texture view. It would be great to be able to dim the view and do other fancy stuff! Using transparencies and stuff is a wonderful idea.

[jonmdev]

That would be amazing @ne0rrmatrix! Thanks for any help. Otherwise I must recreate my own video player (which I already started work on but it is tedious). It seems unnecessary for me to reproduce all the work of your team just to have one toggle essentially to switch over the ExoPlayer construction to TextureView. I was also stuck on how to use XML (or how to set the attribute) to make it a TextureView in .NET.

Current Method

The ExoPlayer is constructed here:

PlayerView = new StyledPlayerView(MauiContext.Context)
		{
			Player = Player,
			UseController = false,
			ControllerAutoShow = false,
			LayoutParameters = new FrameLayout.LayoutParams(ViewGroup.LayoutParams.MatchParent, ViewGroup.LayoutParams.MatchParent, GravityFlags.CenterHorizontal)
		};

		return (Player, PlayerView);

Android states this is obsolete (we are supposed to use Media3) but in any case, this StyledPlayerView constructor is supposed to take an extra argument as per: public unsafe StyledPlayerView (global::Android.Content.Context? context, global::Android.Util.IAttributeSet? attrs)

It is through the attributes we are supposed to set the type of view:

https://exoplayer.dev/doc/reference/com/google/android/exoplayer2/ui/StyledPlayerView.html

The following attributes can be set on a StyledPlayerView when used in a layout XML file:

surface_type - The type of surface view used for video playbacks. Valid values are surface_view, texture_view, spherical_gl_surface_view, video_decoder_gl_surface_view and none. Using none is recommended for audio only applications, since creating the surface can be expensive. Default: surface_view

But I was stuck on how to create an attributes object that would accomplish this. Presumably we can make an attributes file then based on the constructor just in C# to pass in, right? I am not familiar with Android .NET sufficiently to know how to create this. Do you have any ideas? I am happy to continue my experiment on my end as well if I can get past this point but I couldn't figure it out.

Or if we need to make an XML file and convert it like I posted in the OP, can we make the XML file programmatically (ie. just from a text string of the necessary contents) to convert to attributes on the fly in the code right there (rather than making an actual XML file to load from disk or the object pointlessly)?

If you are not sure as well, let me know. I can post on StackOverflow or other forums and see if anyone can help. That was my next step I was planning in any case.

Media3

Perhaps also of interest, I noted in my research Media3 is being integrated into AndroidX: dotnet/android-libraries#779

I don't know enough about GitHub or how these updates work to know when we will have that available. Is it available now based on comments here? #1511

I'm not sure if that will make the job any easier or if it makes no difference. Google states the designation of TextureView vs. SurfaceView should be very easy:

For video apps that implement their own UI, the target SurfaceView, TextureView, SurfaceHolder or Surface can be set using ExoPlayer's setVideoSurfaceView, setVideoTextureView, setVideoSurfaceHolder, and setVideoSurface methods respectively.

https://developer.android.com/media/media3/exoplayer/hello-world

TextureView vs. SurfaceView

Once created, this article describes the differences between TextureView and SurfaceView in terms of events, etc and how to switch from one to the other in much more detail:

https://medium.com/androiddevelopers/android-hdr-migrating-from-textureview-to-surfaceview-part-1-how-to-migrate-6bfd7f4b970e

Again, thanks for any help with this. If you are able it will save me weeks of learning the nitty gritty of these video players to make my own from scratch just for this change. Please let me know if anything is giving you any special difficulty and I am happy to look at it also or help in any way.

If you can help me figure out the XML/attributes issue I may still just finish up my video player I have started in any case so I'd appreciate that as well.

[jonmdev]

Hey @ne0rrmatrix , I managed to figure out a working method to construct the ExoPlayer as a TextureView in .NET. It is actually quite simple but a bit silly. One must save a short snippet of an XML file into the Android Resources folder like say: Resources/layout/textureview.xml

<?xml version="1.0" encoding="utf-8" ?> 
<com.google.android.exoplayer2.ui.StyledPlayerView 
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    app:surface_type="texture_view" />

Then one can use it in Android code as:

XmlReader xmlResource = this.Resources.GetXml(Resource.Layout.textureview);
xmlResource.Read();
Android.Util.IAttributeSet attributes = Android.Util.Xml.AsAttributeSet(xmlResource);

Com.Google.Android.Exoplayer2.UI.StyledPlayerView styledPlayerView = new(this, attributes);
xmlResource.Dispose();

System.Diagnostics.Debug.WriteLine("SURFACE TYPE " + styledPlayerView.VideoSurfaceView.GetType()); //default is SurfaceView, need to make TextureView

This therefore seems like it should hopefully be a relatively straight forward feature to add. Hopefully it can be. I made a proposal here explaining further if it is any help:

#1891

[ne0rrmatrix]

This looks like a great idea. I fully support it and would like to help if you want to do it yourself. If you do not I would be happy to collaborate with you and we could do it as a team. Otherwise I fully support you trying this yourself. It is up to you.

[jonmdev]

This looks like a great idea. I fully support it and would like to help if you want to do it yourself. If you do not I would be happy to collaborate with you and we could do it as a team. Otherwise I fully support you trying this yourself. It is up to you.

Hey @ne0rrmatrix - Thanks! I appreciate it. Unfortunately, I have never modified or even built anything like a library or nuget in C#. So while I can figure out how it should work in my own project experiments and dig through the library code, I am not sure of how to do anything else like actually integrating it into the code and then building the library so it can be tested, etc.

I am only a self taught programmer. For example, with MAUI I can identify bugs and fill reports, but I must leave it to the MAUI team to fix as that is as far as I can go. Similarly for this, I can think it through, but I don't think I can implement it.

I am not sure if you'd be willing to try implementing it. It would be great if so and you can.

Either way, for the best I can provide, here are what I think the steps would be:

#1891 (comment)

What do you think? I believe I have thought this through to the maximum I can without actually doing it and testing the outcome, which as I said is a bit above my capacity or knowledge.

Is it something you could try to implement? If you try and hit any walls I'm happy to investigate them or figure out solutions. But the actual doing I think needs to be saved for someone more expert than me. Thanks either way.