enableDecoderFallback does not work as expected

[pmendozav]

Hello everyone,

I’m having trouble understanding the behavior of enableDecoderFallback and the specific level at which it operates. Essentially, I want to confirm that enableDecoderFallback is functioning as expected.

Here’s what I did:

I customized demos/main with the following classes:

• VideoRendererEventListener (from VideoRendererEventListener): to add logs and monitor behavior when exceptions are thrown.
• CustomMediaCodecRenderer (from MediaCodecVideoRenderer): to generate exceptions and examine available decoders.
• CustomRendererFactory (from DefaultRenderersFactory): to manage both VideoRendererEventListener and CustomMediaCodecRenderer.

With these helper classes, I performed the following tests:

1. When I add two CustomMediaCodecRenderer instances at indices 0 and 1, and trigger an exception in index 0 (for example, by having getDecoderInfos() return an empty list), playback fails, and there’s no indication that the player attempts to use index 1.
2. If I use only one CustomMediaCodecRenderer at index 0 while keeping the hardware decoder selected by the player, playback behaves differently depending on where I place the exception. If I place it in onCodecInitialized, playback continues; however, if I place it in onQueueInputBuffer, playback doesn’t recover (see comments error_1, error_2 in the code).

In each case, logs capture the error but don’t indicate any fallback from hardware to software within the same CustomMediaCodecRenderer. This raises some questions:

• How can I confirm that CustomMediaCodecRenderer has indeed switched to a fallback decoder?
• Am I triggering the exception incorrectly? Where should it be placed, and why do some exceptions (like those in onQueueInputBuffer) remain unhandled, causing playback to fail?

Thank you for any insights!

Here is my fork with the changes I made

utilities classes:

package androidx.media3.demo.main;

import android.annotation.SuppressLint;
import android.content.Context;
import android.os.Handler;
import android.util.Log;
import androidx.annotation.Nullable;
import androidx.media3.common.C;
import androidx.media3.common.Format;
import androidx.media3.common.VideoSize;
import androidx.media3.common.util.UnstableApi;
import androidx.media3.decoder.DecoderInputBuffer;
import androidx.media3.exoplayer.DecoderCounters;
import androidx.media3.exoplayer.DecoderReuseEvaluation;
import androidx.media3.exoplayer.DefaultRenderersFactory;
import androidx.media3.exoplayer.ExoPlaybackException;
import androidx.media3.exoplayer.Renderer;
import androidx.media3.exoplayer.audio.AudioRendererEventListener;
import androidx.media3.exoplayer.audio.AudioSink;
import androidx.media3.exoplayer.mediacodec.MediaCodecAdapter;
import androidx.media3.exoplayer.mediacodec.MediaCodecInfo;
import androidx.media3.exoplayer.mediacodec.MediaCodecSelector;
import androidx.media3.exoplayer.mediacodec.MediaCodecUtil;
import androidx.media3.exoplayer.video.MediaCodecVideoRenderer;
import androidx.media3.exoplayer.video.VideoRendererEventListener;
import java.util.ArrayList;
import java.util.List;




/**
 * helper class used to combine 2 VideoRendererEventListener instances
 */
@UnstableApi
@SuppressLint("UnsafeOptInUsageError")
class CompositeVideoRendererEventListener implements VideoRendererEventListener {

  private final VideoRendererEventListener listener1;
  private final VideoRendererEventListener listener2;

  public CompositeVideoRendererEventListener(VideoRendererEventListener listener1, VideoRendererEventListener listener2) {
    this.listener1 = listener1;
    this.listener2 = listener2;
  }

  @Override
  public void onVideoEnabled(DecoderCounters counters) {
    listener1.onVideoEnabled(counters);
    listener2.onVideoEnabled(counters);
  }

  @Override
  public void onVideoDecoderInitialized(String decoderName, long initializedTimestampMs,
      long initializationDurationMs) {
    listener1.onVideoDecoderInitialized(decoderName, initializedTimestampMs,
        initializationDurationMs);
    listener2.onVideoDecoderInitialized(decoderName, initializedTimestampMs,
        initializationDurationMs);
  }

  @Override
  public void onVideoInputFormatChanged(Format format,
      @Nullable DecoderReuseEvaluation decoderReuseEvaluation) {

    listener1.onVideoInputFormatChanged(format, decoderReuseEvaluation);
    listener2.onVideoInputFormatChanged(format, decoderReuseEvaluation);
  }

  @Override
  public void onDroppedFrames(int count, long elapsedMs) {
    listener1.onDroppedFrames(count, elapsedMs);
    listener2.onDroppedFrames(count, elapsedMs);
  }

  @Override
  public void onVideoFrameProcessingOffset(long totalProcessingOffsetUs, int frameCount) {
    listener1.onVideoFrameProcessingOffset(totalProcessingOffsetUs, frameCount);
    listener2.onVideoFrameProcessingOffset(totalProcessingOffsetUs, frameCount);
  }

  @Override
  public void onVideoSizeChanged(VideoSize videoSize) {
    listener1.onVideoSizeChanged(videoSize);
    listener2.onVideoSizeChanged(videoSize);
  }

  @Override
  public void onRenderedFirstFrame(Object output, long renderTimeMs) {
    listener1.onRenderedFirstFrame(output, renderTimeMs);
    listener2.onRenderedFirstFrame(output, renderTimeMs);
  }

  @Override
  public void onVideoDecoderReleased(String decoderName) {
    listener1.onVideoDecoderReleased(decoderName);
    listener2.onVideoDecoderReleased(decoderName);
  }

  @Override
  public void onVideoDisabled(DecoderCounters counters) {
    listener1.onVideoDisabled(counters);
    listener2.onVideoDisabled(counters);
  }

  @Override
  public void onVideoCodecError(Exception videoCodecError) {
    listener1.onVideoCodecError(videoCodecError);
    listener2.onVideoCodecError(videoCodecError);
  }
}

@SuppressLint("UnsafeOptInUsageError")
class CustomVideoRendererEventListener implements
    VideoRendererEventListener {

  public CustomVideoRendererEventListener() {
    Log.d("CVideoRendererEListener", "new instance");
  }

  @Override
  public void onVideoEnabled(DecoderCounters counters) {
    Log.d("CVideoRendererEListener", "onVideoEnabled: " + counters);
  }

  @Override
  public void onVideoDecoderInitialized(String decoderName, long initializedTimestampMs,
      long initializationDurationMs) {
    Log.d("CVideoRendererEListener","onVideoDecoderInitialized: decoderName=" + decoderName + ". initializedTimestampMs=" + initializedTimestampMs + " . initializationDurationMs=" + initializationDurationMs);
  }

  @Override
  public void onVideoInputFormatChanged(Format format,
      @Nullable DecoderReuseEvaluation decoderReuseEvaluation) {
    Log.d("CVideoRendererEListener", "onVideoInputFormatChanged: format=" + format + ". decoderReuseEvaluation=" + decoderReuseEvaluation);
  }

  @Override
  public void onDroppedFrames(int count, long elapsedMs) {
    Log.d("CVideoRendererEListener", "onDroppedFrames: count=" + count + ". elapsedMs=" + elapsedMs);
  }

  @Override
  public void onVideoFrameProcessingOffset(long totalProcessingOffsetUs, int frameCount) {
    Log.d("CVideoRendererEListener", "onVideoFrameProcessingOffset: totalProcessingOffsetUs=" + totalProcessingOffsetUs + ". frameCount=" + frameCount);
  }

  @Override
  public void onVideoSizeChanged(VideoSize videoSize) {
    Log.d("CVideoRendererEListener", "onVideoSizeChanged: videoSize=" + videoSize);
  }

  @Override
  public void onRenderedFirstFrame(Object output, long renderTimeMs) {
    Log.d("CVideoRendererEListener", "onRenderedFirstFrame: output=" + output + ". renderTimeMs=" + renderTimeMs);
  }

  @Override
  public void onVideoDecoderReleased(String decoderName) {
    Log.d("CVideoRendererEListener", "onVideoDecoderReleased: decoderName=" + decoderName);
  }

  @Override
  public void onVideoDisabled(DecoderCounters counters) {
    Log.d("CVideoRendererEListener", "onVideoDisabled: counters=" + counters);
  }

  @Override
  public void onVideoCodecError(Exception videoCodecError) {
    Log.d("CVideoRendererEListener", "onVideoCodecError: videoCodecError=" + videoCodecError);
  }
}

@SuppressLint("UnsafeOptInUsageError")
class CustomMediaCodecRenderer extends MediaCodecVideoRenderer {
  final String codec_name_filtered = "OMX.amlogic.avc.decoder.awesome2";
  private String currentDecoderName;
  private boolean forceException;

  private boolean checkFilterCodec(String decoderName) {
    return forceException && decoderName != null && decoderName.contains(codec_name_filtered);
  }

  public CustomMediaCodecRenderer(
      Context context,
      MediaCodecSelector mediaCodecSelector,
      boolean enableDecoderFallback,
      Handler eventHandler,
      VideoRendererEventListener eventListener,
      long allowedVideoJoiningTimeMs,
      int maxDroppedFrameToNotify,
      boolean useForceException
  ) {
    super(context, mediaCodecSelector, allowedVideoJoiningTimeMs, enableDecoderFallback,
        eventHandler, eventListener, maxDroppedFrameToNotify);
    forceException = useForceException;
    Log.d("CMediaCodecRenderer", "new instance. enableDecoderFallback=" + enableDecoderFallback);
  }

  @Override
  protected void onCodecError(Exception codecError) {
    Log.d("CMediaCodecRenderer", "onCodecError: " + codecError);
    super.onCodecError(codecError);
  }

  @Override
  protected List<MediaCodecInfo> getDecoderInfos(
      MediaCodecSelector mediaCodecSelector, Format format, boolean requiresSecureDecoder)
      throws MediaCodecUtil.DecoderQueryException {

    Log.d("CMediaCodecRenderer", "getDecoderInfos: mediaCodecSelector=" + mediaCodecSelector + ". format=" + format + ". requiresSecureDecoder" + requiresSecureDecoder);
    List<MediaCodecInfo> decoderInfos = super.getDecoderInfos(mediaCodecSelector, format,
        requiresSecureDecoder);

    Log.d("CMediaCodecRenderer", "decoderInfo names:");
    for (MediaCodecInfo info : decoderInfos) {
      Log.d("CMediaCodecRenderer", "decoderInfos[i].name: " + info.name);
    }

    return decoderInfos;
  }

  @Override
  protected void onCodecInitialized(
      String name,
      MediaCodecAdapter.Configuration configuration,
      long initializedTimestampMs,
      long initializationDurationMs
  ) throws ExoPlaybackException {
    currentDecoderName = name;
    Log.d("CMediaCodecRenderer", "onCodecInitialized: name=" + name);

    super.onCodecInitialized(currentDecoderName, configuration, initializedTimestampMs, initializationDurationMs);

    if (checkFilterCodec(currentDecoderName)) {
        Log.d("CMediaCodecRenderer", "trying to force error for hardware decoder (onCodecInitialized)");

//      // error_1: recoverable error
     throw ExoPlaybackException.createForRenderer(new IllegalStateException("Simulated failure in hardware decoder"), codec_name_filtered, 0, null, C.FORMAT_UNSUPPORTED_SUBTYPE, true, 5001);
    }
  }

  @Override
  protected void onQueueInputBuffer(DecoderInputBuffer buffer) throws ExoPlaybackException {
    super.onQueueInputBuffer(buffer);

    if (checkFilterCodec(currentDecoderName)) {
//      Log.d("CMediaCodecRenderer", "trying to force hardware decoder error (onQueueInputBuffer)");
//      // error_2: unrecoverable error
//      throw ExoPlaybackException.createForRenderer(new IllegalStateException("Simulated failure in hardware decoder"), codec_name_filtered, 0, null, C.FORMAT_UNSUPPORTED_SUBTYPE, true, 5001);
    }
  }
}

@SuppressLint("UnsafeOptInUsageError")
class CustomRendererFactory extends DefaultRenderersFactory {
  public CustomRendererFactory(Context context) {
    super(context);

    Log.d("CustomRendererFactory", "new instance");
  }

  @Override
  protected void buildAudioRenderers(
      Context context,
      @ExtensionRendererMode int extensionRendererMode,
      MediaCodecSelector mediaCodecSelector,
      boolean enableDecoderFallback,
      AudioSink audioSink,
      Handler eventHandler,
      AudioRendererEventListener eventListener,
      ArrayList<Renderer> out
  ) {
    Log.d("CustomRendererFactory", "buildAudioRenderers. enableDecoderFallback=" + enableDecoderFallback);

    super.buildAudioRenderers(
        context,
        extensionRendererMode,
        mediaCodecSelector,
        enableDecoderFallback,
        audioSink,
        eventHandler,
        eventListener,
        out
    );
  }

  @Override
  protected void buildVideoRenderers(
      Context context,
      @ExtensionRendererMode int extensionRendererMode,
      MediaCodecSelector mediaCodecSelector,
      boolean enableDecoderFallback,
      Handler eventHandler,
      VideoRendererEventListener eventListener,
      long allowedVideoJoiningTimeMs,
      ArrayList<Renderer> out
  ) {
    Log.d("CustomRendererFactory", "buildVideoRenderers. enableDecoderFallback=" + enableDecoderFallback);

    VideoRendererEventListener customVideoRendererEventListener = new CompositeVideoRendererEventListener(eventListener, new CustomVideoRendererEventListener());

    super.buildVideoRenderers(
        context,
        extensionRendererMode,
        mediaCodecSelector,
        enableDecoderFallback,
        eventHandler,
        customVideoRendererEventListener,
        allowedVideoJoiningTimeMs,
        out
    );

    out.add(0, new CustomMediaCodecRenderer(
        context,
        mediaCodecSelector,
        true,
        eventHandler,
        customVideoRendererEventListener,
        allowedVideoJoiningTimeMs,
        DefaultRenderersFactory.MAX_DROPPED_VIDEO_FRAME_COUNT_TO_NOTIFY,
        true
    ));

    out.add(1, new CustomMediaCodecRenderer(
        context,
        mediaCodecSelector,
        true,
        eventHandler,
        customVideoRendererEventListener,
        allowedVideoJoiningTimeMs,
        DefaultRenderersFactory.MAX_DROPPED_VIDEO_FRAME_COUNT_TO_NOTIFY,
        false
    ));

    Log.d("CustomRendererFactory", "video_rendererss_list_size = " + out.size());
  }
}

Change in PlayerActivity.java

  private void setRenderersFactory(
      ExoPlayer.Builder playerBuilder, boolean preferExtensionDecoders) {
    RenderersFactory renderersFactory = new CustomRendererFactory(this);
    playerBuilder.setRenderersFactory(renderersFactory);
  }

[icbaker]

1. When I add two CustomMediaCodecRenderer instances at indices 0 and 1, and trigger an exception in index 0 (for example, by having getDecoderInfos() return an empty list), playback fails, and there’s no indication that the player attempts to use index 1.

This sounds like you're expecting to see fallback between Renderer instances, but the boolean you're toggling is about decoder fallback. These are different abstractions. I think it's expected that you don't see fallback from one Renderer to another.

---

2. If I place it in onCodecInitialized, playback continues; however, if I place it in onQueueInputBuffer, playback doesn’t recover (see comments error_1, error_2 in the code).

This matches my reading of the documentation on the enableDecoderFallback parameter of the MediaCodecRenderer constructor (emphasis mine):

Whether to enable fallback to lower-priority decoders if decoder initialization fails

---

• How can I confirm that CustomMediaCodecRenderer has indeed switched to a fallback decoder?

The code looks like it should emit the event to VideoRendererEventListener (and log it to logcat) via MediaCodecVideoRenderer.onCodecError:

media/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/mediacodec/MediaCodecRenderer.java

Lines 1140 to 1147 in c35a9d6

	// This codec failed to initialize, so fall back to the next codec in the list (if any). We
	// won't try to use this codec again unless there's a format change or the renderer is
	// disabled and re-enabled.
	availableCodecInfos.removeFirst();
	DecoderInitializationException exception =
	new DecoderInitializationException(
	inputFormat, e, mediaCryptoRequiresSecureDecoder, codecInfo);
	onCodecError(exception);

media/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/video/MediaCodecVideoRenderer.java

Lines 1173 to 1176 in c35a9d6

	protected void onCodecError(Exception codecError) {
	Log.e(TAG, "Video codec error", codecError);
	eventDispatcher.videoCodecError(codecError);
	}

When I try this in the demo app by enabling decoder fallback on the DefaultRenderersFactory instantiated in DemoUtil and throwing an exception for the first code (based on name) at the bottom MediaCodecVideoRenderer.onCodecInitialized, i see the exception logged from MediaCodecVideoRenderer. If i then implement AnalyticsListener.onVideoCodecError in EventLogger I also see it logged there (this is wired via VideoRendererEventListener).

Aside: MediaCodecRenderer.onCodecInitialized is documented as (emphasis mine):

Called when a MediaCodec has been created and configured.

i.e. this is only called after initialization is complete, so throwing an exception in here to simulate an initialization failure is technically too late. It happens to work at the moment, due to the structure of the code, but I wouldn't rely on this remaining true forever.

[pmendozav]

Thanks for the feedback, @icbaker!

As you mentioned, I can indeed see the forced error I generated, and I'm able to recover the playback. However, my issue is that I cannot confirm whether, after logging the error, the player attempted to retry the playback using the next codec (within the same MediaCodecRenderer instance). In other words, the only way to verify that the codec fallback mechanism worked would be to detect an event indicating successful playback, right? Additionally, if I encounter a codec initialization error and the fallback fails, how can I determine if the player even attempted to use the fallback?

Additionally, I'm wondering about the practicality of having multiple MediaCodecRenderer instances for video. From what I understand, the player will always use the first renderer in the list that supports the given format, so is there a scenario where using multiple MediaCodecRenderer instances would be beneficial?