東川印記

一本東川,笑看爭龍斗虎;寰茫兦者,度橫佰昧人生。

Exoplayer学习07 音频处理器 AudioProcessor

2021年6月3日星期四



继承关系

AudioProcessor.png

接口定义

/** * Interface for audio processors, which take audio data as input and transform it, potentially * modifying its channel count, encoding and/or sample rate. * * <p>In addition to being able to modify the format of audio, implementations may allow parameters * to be set that affect the output audio and whether the processor is active/inactive. * 音频处理器的接口,它将音频数据作为输入并进行转换,从而有可能修改其通道数,编码和/或采样率。 * <p>除了能够修改音频的格式外,实现还可以设置影响输出音频以及处理器是否处于活动状态的参数。 */ public interface AudioProcessor { /** PCM audio format that may be handled by an audio processor.音频处理器可以处理的PCM音频格式。 */ final class AudioFormat { public static final AudioFormat NOT_SET = //没有设置,就都是-1 new AudioFormat( /* sampleRate= */ Format.NO_VALUE, /* channelCount= */ Format.NO_VALUE, /* encoding= */ Format.NO_VALUE); /** The sample rate in Hertz赫兹采样率. */ public final int sampleRate; /** The number of interleaved channels. */ public final int channelCount; /** The type of linear PCM encoding. */ @C.PcmEncoding public final int encoding; /** The number of bytes used to represent one audio frame用来表示一个音频帧的字节数. */ public final int bytesPerFrame; public AudioFormat(int sampleRate, int channelCount, @C.PcmEncoding int encoding) { this.sampleRate = sampleRate; this.channelCount = channelCount; this.encoding = encoding; bytesPerFrame = Util.isEncodingLinearPcm(encoding) ? Util.getPcmFrameSize(encoding, channelCount) : Format.NO_VALUE; Logger.w("AudioProcessor",sampleRate,channelCount,encoding,bytesPerFrame); } @Override public String toString() { return "AudioProcessor 音频处理 AudioFormat[" + "sampleRate=" + sampleRate + ", channelCount=" + channelCount + ", encoding=" + encoding + ']'; } } /** Exception thrown when a processor can't be configured for a given input audio format * 无法为给定的输入音频格式配置处理器时抛出异常. */ final class UnhandledAudioFormatException extends Exception { public UnhandledAudioFormatException(AudioFormat inputAudioFormat) { super("Unhandled format: " + inputAudioFormat); } } /** An empty, direct {@link ByteBuffer}. */ ByteBuffer EMPTY_BUFFER = ByteBuffer.allocateDirect(0).order(ByteOrder.nativeOrder()); /** * Configures the processor to process input audio with the specified format. After calling this * method, call {@link #isActive()} to determine whether the audio processor is active. Returns * the configured output audio format if this instance is active. * * <p>After calling this method, it is necessary to {@link #flush()} the processor to apply the * new configuration. Before applying the new configuration, it is safe to queue input and get * output in the old input/output formats. Call {@link #queueEndOfStream()} when no more input * will be supplied in the old input format. * 将处理器配置为处理指定格式的输入音频。 调用此方法后,调用isActive()以确定音频处理器是否处于活动状态。 如果此实例处于活动状态,则返回配置的输出音频格式。 * 调用此方法后,有必要flush()处理器以应用新配置。 在应用新配置之前,可以安全地将输入排队并以旧的输入/输出格式获取输出。 当不再以旧的输入格式提供输入时,请调用queueEndOfStream()。 * * @param inputAudioFormat The format of audio that will be queued after the next call to {@link * #flush()}. * @return The configured output audio format if this instance is {@link #isActive() active}. * @throws UnhandledAudioFormatException Thrown if the specified format can't be handled as input. */ AudioFormat configure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException; /** Returns whether the processor is configured and will process input buffers返回是否配置了处理器并将处理输入缓冲区. */ boolean isActive(); /** * Queues audio data between the position and limit of the input {@code buffer} for processing. * {@code buffer} must be a direct byte buffer with native byte order. Its contents are treated as * read-only. Its position will be advanced by the number of bytes consumed (which may be zero). * The caller retains ownership of the provided buffer. Calling this method invalidates any * previous buffer returned by {@link #getOutput()}. * 在输入缓冲区的位置和限制之间排队音频数据以进行处理。 * 缓冲区必须是具有本地字节顺序的直接字节缓冲区。 其内容被视为只读。 它的位置将增加所消耗的字节数(可能为零)。 * 调用方保留提供的缓冲区的所有权。 调用此方法会使getOutput()返回的所有先前缓冲区无效。 * * @param buffer The input buffer to process. */ void queueInput(ByteBuffer buffer); /** * Queues an end of stream signal. After this method has been called, * {@link #queueInput(ByteBuffer)} may not be called until after the next call to * {@link #flush()}. Calling {@link #getOutput()} will return any remaining output data. Multiple * calls may be required to read all of the remaining output data. {@link #isEnded()} will return * {@code true} once all remaining output data has been read. * 将流信号的末尾排队。 * 调用此方法后,直到下一次调用flush()之后,才能调用queueInput(ByteBuffer)。 * 调用getOutput()将返回所有剩余的输出数据。 可能需要多次调用才能读取所有剩余的输出数据。 一旦读取了所有剩余的输出数据,isEnded()将返回true。 */ void queueEndOfStream(); /** * Returns a buffer containing processed output data between its position and limit. The buffer * will always be a direct byte buffer with native byte order. Calling this method invalidates any * previously returned buffer. The buffer will be empty if no output is available. * 返回一个缓冲区,该缓冲区包含在其位置和限制之间的已处理输出数据。 缓冲区将始终是具有本地字节顺序的直接字节缓冲区。 * 调用此方法会使以前返回的所有缓冲区无效。 如果没有可用的输出,缓冲区将为空。 * * 返回值: * 包含在其位置和极限之间的已处理输出数据的缓冲区。 * * @return A buffer containing processed output data between its position and limit. */ ByteBuffer getOutput(); /** * Returns whether this processor will return no more output from {@link #getOutput()} until it * has been {@link #flush()}ed and more input has been queued. */ boolean isEnded(); /** * Clears any buffered data and pending output. If the audio processor is active, also prepares * the audio processor to receive a new stream of input in the last configured (pending) format. * 清除所有缓冲的数据和挂起的输出。 如果音频处理器处于活动状态,则还要准备音频处理器以接收最后配置的(待定)格式的新输入流。 */ void flush(); /** Resets the processor to its unconfigured state, releasing any resources. */ void reset(); }

在demo中,实际使用在DefaultAudioSink

入口在DefaultRenderFactory

/** * Builds an {@link AudioSink} to which the audio renderers will output. * 构建一个{@link AudioSink},音频渲染器将输出到该音频。 * * @param context The {@link Context} associated with the player. * @param enableFloatOutput Whether to enable use of floating point audio output, if available. * @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link * android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported. * @param enableOffload Whether to enable use of audio offload for supported formats, if * available. * @return The {@link AudioSink} to which the audio renderers will output. May be {@code null} if * no audio renderers are required. If {@code null} is returned then {@link * #buildAudioRenderers} will not be called. */ @Nullable protected AudioSink buildAudioSink( Context context, boolean enableFloatOutput, boolean enableAudioTrackPlaybackParams, boolean enableOffload) { return new DefaultAudioSink( AudioCapabilities.getCapabilities(context), new DefaultAudioProcessorChain(), enableFloatOutput, enableAudioTrackPlaybackParams, enableOffload); }

对于AudioProcess的调用,就是循环各个processor。。。。

/** * Creates a new default audio sink, optionally using float output for high resolution PCM and * with the specified {@code audioProcessorChain}. * 创建一个新的默认音频接收器,还可以选择将浮点输出用于高分辨率PCM,并使用指定的{@code audioProcessorChain}。 * * @param audioCapabilities The audio capabilities for playback on this device在此设备上播放的音频功能. May be null if the * default capabilities (no encoded audio passthrough support) should be assumed. * @param audioProcessorChain An {@link AudioProcessorChain} which is used to apply playback * parameters adjustments. The instance passed in must not be reused in other sinks. * @param enableFloatOutput Whether to enable 32-bit float output. Where possible, 32-bit float * output will be used if the input is 32-bit float, and also if the input is high resolution * (24-bit or 32-bit) integer PCM. Float output is supported from API level 21. Audio * processing (for example, speed adjustment) will not be available when float output is in * use. * @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link * android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported. * @param enableOffload Whether to enable audio offload. If an audio format can be both played * with offload and encoded audio passthrough, it will be played in offload. Audio offload is * supported from API level 29. Most Android devices can only support one offload {@link * android.media.AudioTrack} at a time and can invalidate it at any time. Thus an app can * never be guaranteed that it will be able to play in offload. Audio processing (for example, * speed adjustment) will not be available when offload is in use. */ public DefaultAudioSink( @Nullable AudioCapabilities audioCapabilities, AudioProcessorChain audioProcessorChain, boolean enableFloatOutput, boolean enableAudioTrackPlaybackParams, boolean enableOffload) { this.audioCapabilities = audioCapabilities; this.audioProcessorChain = Assertions.checkNotNull(audioProcessorChain); this.enableFloatOutput = Util.SDK_INT >= 21 && enableFloatOutput; this.enableAudioTrackPlaybackParams = Util.SDK_INT >= 23 && enableAudioTrackPlaybackParams; this.enableOffload = Util.SDK_INT >= 29 && enableOffload; releasingConditionVariable = new ConditionVariable(true); audioTrackPositionTracker = new AudioTrackPositionTracker(new PositionTrackerListener()); channelMappingAudioProcessor = new ChannelMappingAudioProcessor(); trimmingAudioProcessor = new TrimmingAudioProcessor(); ArrayList<AudioProcessor> toIntPcmAudioProcessors = new ArrayList<>(); Collections.addAll( toIntPcmAudioProcessors, new ResamplingAudioProcessor(), channelMappingAudioProcessor, trimmingAudioProcessor); Collections.addAll(toIntPcmAudioProcessors, audioProcessorChain.getAudioProcessors()); toIntPcmAvailableAudioProcessors = toIntPcmAudioProcessors.toArray(new AudioProcessor[0]); toFloatPcmAvailableAudioProcessors = new AudioProcessor[] {new FloatResamplingAudioProcessor()}; volume = 1f; audioAttributes = AudioAttributes.DEFAULT; audioSessionId = C.AUDIO_SESSION_ID_UNSET; auxEffectInfo = new AuxEffectInfo(AuxEffectInfo.NO_AUX_EFFECT_ID, 0f); mediaPositionParameters = new MediaPositionParameters( PlaybackParameters.DEFAULT, DEFAULT_SKIP_SILENCE, /* mediaTimeUs= */ 0, /* audioTrackPositionUs= */ 0); audioTrackPlaybackParameters = PlaybackParameters.DEFAULT; drainingAudioProcessorIndex = C.INDEX_UNSET; activeAudioProcessors = new AudioProcessor[0]; outputBuffers = new ByteBuffer[0]; mediaPositionParametersCheckpoints = new ArrayDeque<>(); initializationExceptionPendingExceptionHolder = new PendingExceptionHolder<>(AUDIO_TRACK_RETRY_DURATION_MS); writeExceptionPendingExceptionHolder = new PendingExceptionHolder<>(AUDIO_TRACK_RETRY_DURATION_MS); }

搞了两个AudioProcessor数组,一个是toIntPcm Avaliable AudioProcessor,一个是toFloatPcm Avaliable AudioProcess。

分别对应解码后的输出类型,因为好多设备不支持浮点输出

然后去config

@Override public void configure(Format inputFormat, int specifiedBufferSize, @Nullable int[] outputChannels) throws ConfigurationException { int inputPcmFrameSize; @Nullable AudioProcessor[] availableAudioProcessors; 。。。 Logger.w(TAG,"configure方法",inputFormat.toString(),specifiedBufferSize,outputChannels);//Format(2, null, null, audio/ac3, null, -1, en, [-1, -1, -1.0], [6, 48000]),0,null if (MimeTypes.AUDIO_RAW.equals(inputFormat.sampleMimeType)) { Assertions.checkArgument(Util.isEncodingLinearPcm(inputFormat.pcmEncoding)); inputPcmFrameSize = Util.getPcmFrameSize(inputFormat.pcmEncoding, inputFormat.channelCount); availableAudioProcessors = shouldUseFloatOutput(inputFormat.pcmEncoding) ? toFloatPcmAvailableAudioProcessors : toIntPcmAvailableAudioProcessors; 。。。 AudioProcessor.AudioFormat outputFormat = new AudioProcessor.AudioFormat( inputFormat.sampleRate, inputFormat.channelCount, inputFormat.pcmEncoding); for (AudioProcessor audioProcessor : availableAudioProcessors) { try { AudioProcessor.AudioFormat nextFormat = audioProcessor.configure(outputFormat); if (audioProcessor.isActive()) { outputFormat = nextFormat; Logger.w(TAG,"configure",outputFormat); } } catch (UnhandledAudioFormatException e) { throw new ConfigurationException(e, inputFormat); } } outputMode = OUTPUT_MODE_PCM; outputEncoding = outputFormat.encoding; outputSampleRate = outputFormat.sampleRate; outputChannelConfig = Util.getAudioTrackChannelConfig(outputFormat.channelCount); outputPcmFrameSize = Util.getPcmFrameSize(outputEncoding, outputFormat.channelCount); } else { inputPcmFrameSize = C.LENGTH_UNSET; availableAudioProcessors = new AudioProcessor[0]; outputSampleRate = inputFormat.sampleRate; outputPcmFrameSize = C.LENGTH_UNSET; Logger.w(TAG,"configure方法x2",enableOffload,isOffloadedPlaybackSupported(inputFormat, audioAttributes));//false,false if (enableOffload && isOffloadedPlaybackSupported(inputFormat, audioAttributes)) { outputMode = OUTPUT_MODE_OFFLOAD; outputEncoding = MimeTypes.getEncoding( Assertions.checkNotNull(inputFormat.sampleMimeType), inputFormat.codecs); outputChannelConfig = Util.getAudioTrackChannelConfig(inputFormat.channelCount); } else { outputMode = OUTPUT_MODE_PASSTHROUGH;//直通输出 @Nullable Pair<Integer, Integer> encodingAndChannelConfig = getEncodingAndChannelConfigForPassthrough(inputFormat, audioCapabilities); Logger.w("pass though x2",encodingAndChannelConfig);//Pair{5 252} if (encodingAndChannelConfig == null) { throw new ConfigurationException( "Unable to configure passthrough for: " + inputFormat, inputFormat); } outputEncoding = encodingAndChannelConfig.first;//5 outputChannelConfig = encodingAndChannelConfig.second;//252 } } 。。。 }

也就是说,需要满足inputFormat.sampleMImeType是 /raw才会去配置 AudioProcessor

也就是会走解码输出

否则会自动的去走音频分载或者passthrough

然后实际的干活方法里

AudioSink接口

/** * Attempts to process data from a {@link ByteBuffer}, starting from its current position and * ending at its limit (exclusive). The position of the {@link ByteBuffer} is advanced by the * number of bytes that were handled. {@link Listener#onPositionDiscontinuity()} will be called if * {@code presentationTimeUs} is discontinuous with the last buffer handled since the last reset. * 尝试处理ByteBuffer的数据,从其当前位置开始,直到其限制(不包括限制)。 * ByteBuffer的位置提前处理的字节数。 如果presentationTimeUs与自上次重置以来处理的最后一个缓冲区不连续,则将调用AudioSink.Listener.onPositionDiscontinuity()。 * 返回数据是否已全部处理。 如果未对数据进行完整处理,则必须将相同的ByteBuffer提供给后续调用,直到完全消耗完为止, * 除非是对flush()(或configure(Format,int,int [])的中间调用) 导致水槽被冲洗)。 * * <p>Returns whether the data was handled in full. If the data was not handled in full then the * same {@link ByteBuffer} must be provided to subsequent calls until it has been fully consumed, * except in the case of an intervening call to {@link #flush()} (or to {@link #configure(Format, * int, int[])} that causes the sink to be flushed). * * @param buffer The buffer containing audio data. * @param presentationTimeUs The presentation timestamp of the buffer in microseconds. * @param encodedAccessUnitCount The number of encoded access units in the buffer, or 1 if the * buffer contains PCM audio. This allows batching multiple encoded access units in one * buffer. * @return Whether the buffer was handled fully. * @throws InitializationException If an error occurs initializing the sink. * @throws WriteException If an error occurs writing the audio data. */ boolean handleBuffer(ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount) throws InitializationException, WriteException;

默认实现

@Override @SuppressWarnings("ReferenceEquality") public boolean handleBuffer( ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount) throws InitializationException, WriteException { Assertions.checkArgument(inputBuffer == null || buffer == inputBuffer); if (pendingConfiguration != null) { 。。。。 // Re-apply playback parameters. applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); } if (!isAudioTrackInitialized()) { try { initializeAudioTrack(); //初始化 audioTrack } catch (InitializationException e) { 。。。。 return false; } } initializationExceptionPendingExceptionHolder.clear(); if (startMediaTimeUsNeedsInit) { 。。。。 applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); if (playing) { play();//audioTrack开始播放 } } if (!audioTrackPositionTracker.mayHandleBuffer(getWrittenFrames())) { return false; } if (inputBuffer == null) { // We are seeing this buffer for the first time. Assertions.checkArgument(buffer.order() == ByteOrder.LITTLE_ENDIAN); if (!buffer.hasRemaining()) { // The buffer is empty. return true; } 。。。。 if (afterDrainParameters != null) { if (!drainToEndOfStream()) { // Don't process any more input until draining completes. return false; } applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); afterDrainParameters = null; } // Check that presentationTimeUs is consistent with the expected value. long expectedPresentationTimeUs = startMediaTimeUs + configuration.inputFramesToDurationUs( getSubmittedFrames() - trimmingAudioProcessor.getTrimmedFrameCount()); if (!startMediaTimeUsNeedsSync && Math.abs(expectedPresentationTimeUs - presentationTimeUs) > 200000) { Log.e( TAG, "Discontinuity detected [expected " + expectedPresentationTimeUs + ", got " + presentationTimeUs + "]"); startMediaTimeUsNeedsSync = true; } if (startMediaTimeUsNeedsSync) { if (!drainToEndOfStream()) { // Don't update timing until pending AudioProcessor buffers are completely drained. return false; } // Adjust startMediaTimeUs to be consistent with the current buffer's start time and the // number of bytes submitted. long adjustmentUs = presentationTimeUs - expectedPresentationTimeUs; startMediaTimeUs += adjustmentUs; startMediaTimeUsNeedsSync = false; // Re-apply playback parameters because the startMediaTimeUs changed. applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); if (listener != null && adjustmentUs != 0) { listener.onPositionDiscontinuity(); } } if (configuration.outputMode == OUTPUT_MODE_PCM) { submittedPcmBytes += buffer.remaining(); } else { submittedEncodedFrames += framesPerEncodedSample * encodedAccessUnitCount; } inputBuffer = buffer; inputBufferAccessUnitCount = encodedAccessUnitCount; } processBuffers(presentationTimeUs);//调用02篇的build时创建的音频处理器处理 if (!inputBuffer.hasRemaining()) { inputBuffer = null; inputBufferAccessUnitCount = 0; return true; } if (audioTrackPositionTracker.isStalled(getWrittenFrames())) { Log.w(TAG, "Resetting stalled audio track"); flush(); return true; } return false; }

两处对audio processor调用,一个是设置audioProcessor,设置完成后用来处理

设置

private void setupAudioProcessors() { AudioProcessor[] audioProcessors = configuration.availableAudioProcessors; ArrayList<AudioProcessor> newAudioProcessors = new ArrayList<>(); for (AudioProcessor audioProcessor : audioProcessors) { if (audioProcessor.isActive()) { newAudioProcessors.add(audioProcessor); } else { audioProcessor.flush(); } } int count = newAudioProcessors.size(); activeAudioProcessors = newAudioProcessors.toArray(new AudioProcessor[count]); outputBuffers = new ByteBuffer[count]; flushAudioProcessors(); } private void flushAudioProcessors() { for (int i = 0; i < activeAudioProcessors.length; i++) { AudioProcessor audioProcessor = activeAudioProcessors[i]; audioProcessor.flush(); outputBuffers[i] = audioProcessor.getOutput(); } }

用来处理

private void processBuffers(long avSyncPresentationTimeUs) throws WriteException { int count = activeAudioProcessors.length; int index = count; Logger.w(TAG,"processBuffers",avSyncPresentationTimeUs,count);//0,0 | 32000,0 | 64000,0|96000,0 while (index >= 0) { ByteBuffer input = index > 0 ? outputBuffers[index - 1] : (inputBuffer != null ? inputBuffer : AudioProcessor.EMPTY_BUFFER); if (index == count) { writeBuffer(input, avSyncPresentationTimeUs); } else { AudioProcessor audioProcessor = activeAudioProcessors[index]; if (index > drainingAudioProcessorIndex) { audioProcessor.queueInput(input); //输入 } ByteBuffer output = audioProcessor.getOutput();//输出 outputBuffers[index] = output; if (output.hasRemaining()) { // Handle the output as input to the next audio processor or the AudioTrack. index++; continue; } } if (input.hasRemaining()) { // The input wasn't consumed and no output was produced, so give up for now. return; } // Get more input from upstream. index--; } }

这逻辑写的。。。。

index == count就输出音频到audioTrack,index--

音频processor处理完成后就index++


也就是说,DefaultAudioSink对AudioProcessor接口的调用是这样的

狄仁杰上线

狄仁杰:我想事情的真相应该是这样的.jpg

new DefaultAudioSink()的时候,增加了增加了几个AudioProcessor的实现,也就是增加了几个音频处理器的不同实现到toIntPcmAudioProcessors数组;

然后在DefaultAudioSink.configure()的时候,判断如果是解码输出,就先判断是不是浮点输出,如果是toIntPcm而不是toFloatPcm,就把avaliableAudioProcessors指定为toIntPcmAudioProcessor数组,然后循环avaliableProcessor数组,挨个调用AudioProcessor实现类的configure()方法,返回AudioProcess.AudioFormat,判断AudioProcessor.isActive(),如果is active,就会使用返回的AudioFormat去new Configuration(),这个后面的大配置。

接下来,就到了处理数据流的时候,首先设置 setupAudioProcess(),依然是循环那几个实现,取得就是大配置configuration.avaliableAudioProcessor数组,挨个判断是否启用isActive,如果启用就增加到 activeAudioProcessor数组,干完后循环 调用activeAudioProcessor.flush(),flush后配置全类私有数组outputBuffers[i] = activieAudioProcess[i].getOutputj(); 也就是保留了对应的输出通道。

接下来,终于要实际处理流了,依然是要循环子处理器实现通过上面的outBuffer[index]取对应子处理器的返回,然后调用audioProcessor.queueInput(),audioProcess.getOutput(),写入到audioTrack....

对于AudioProcessor的调用顺序,简单理解为  configure() -> isActive() -> flush() -> queueInput() -> getOutput()。

回到configuration方法

Configuration pendingConfiguration = new Configuration( inputFormat, inputPcmFrameSize, outputMode, outputPcmFrameSize, outputSampleRate, outputChannelConfig, outputEncoding, specifiedBufferSize, enableAudioTrackPlaybackParams, availableAudioProcessors);

解码输出时,avaliableAudioProcessors为 toIntPcmAvaliableAudioProcessors或toIntPcmXxxxxx

不解码输出时,为empty

有了 avaliableAudioProcessors才有了后面的 audioProcessor.process()

所以如果要直通输出,可以在这里增加audio processor....

解码输出,构造函数初始化时就配置了四个默认的处理

channelMappingAudioProcessor = new ChannelMappingAudioProcessor(); trimmingAudioProcessor = new TrimmingAudioProcessor(); ArrayList<AudioProcessor> toIntPcmAudioProcessors = new ArrayList<>(); Collections.addAll( toIntPcmAudioProcessors, new ResamplingAudioProcessor(), channelMappingAudioProcessor, trimmingAudioProcessor); Collections.addAll(toIntPcmAudioProcessors, audioProcessorChain.getAudioProcessors()); toIntPcmAvailableAudioProcessors = toIntPcmAudioProcessors.toArray(new AudioProcessor[0]); toFloatPcmAvailableAudioProcessors = new AudioProcessor[] {new FloatResamplingAudioProcessor()};

分别为

ResamplingAudioProcessor  重采样

ChannelMappingAudioProcessor 声道

TrimmingAudioProcessor 修剪?

还有传过来的 audioProcessorChain.getAudioProcessors()

/** * Creates a new default chain of audio processors, with the user-defined {@code * audioProcessors} applied before silence skipping and speed adjustment processors. */ public DefaultAudioProcessorChain(AudioProcessor... audioProcessors) { this(audioProcessors, new SilenceSkippingAudioProcessor(), new SonicAudioProcessor()); }

按照添加顺序,分别为

SilenceSkippingAudioProcessor  静音跳过

SonicAudioProcessor uses the Sonic library to modify audio speed/pitch/sample rate

demo中对解码输出 总共应用了这五个音频处理器。

1)ResamplingAudioProcessor

/** * An {@link AudioProcessor} that converts different PCM audio encodings to 16-bit integer PCM. The * following encodings are supported as input: * 将不同的PCM音频编码转换为16位整数PCM的AudioProcessor。 支持以下编码作为输入: * * <ul> * <li>{@link C#ENCODING_PCM_8BIT} * <li>{@link C#ENCODING_PCM_16BIT} ({@link #isActive()} will return {@code false}) * <li>{@link C#ENCODING_PCM_16BIT_BIG_ENDIAN} * <li>{@link C#ENCODING_PCM_24BIT} * <li>{@link C#ENCODING_PCM_32BIT} * <li>{@link C#ENCODING_PCM_FLOAT} * </ul> */ /* package */ final class ResamplingAudioProcessor extends BaseAudioProcessor {}

实现也是简单粗暴,只重写了configure() 和 queueInput()方法

@Override public AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { @C.PcmEncoding int encoding = inputAudioFormat.encoding; if (encoding != C.ENCODING_PCM_8BIT && encoding != C.ENCODING_PCM_16BIT && encoding != C.ENCODING_PCM_16BIT_BIG_ENDIAN && encoding != C.ENCODING_PCM_24BIT && encoding != C.ENCODING_PCM_32BIT && encoding != C.ENCODING_PCM_FLOAT) { throw new UnhandledAudioFormatException(inputAudioFormat); } return encoding != C.ENCODING_PCM_16BIT ? new AudioFormat( inputAudioFormat.sampleRate, inputAudioFormat.channelCount, C.ENCODING_PCM_16BIT) : AudioFormat.NOT_SET; }

onConfigure时,不是支持的格式直接抛异常,不是16bit 返回16bit....

onConfigure定义与BaseAudioProcessor抽象类,是AudioProcessor接口中configure方法的延伸

BaseAudioProcessor抽象类:

@Override public final AudioFormat configure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { pendingInputAudioFormat = inputAudioFormat; pendingOutputAudioFormat = onConfigure(inputAudioFormat); return isActive() ? pendingOutputAudioFormat : AudioFormat.NOT_SET; }

/** Called when the processor is configured for a new input format. */ protected AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { return AudioFormat.NOT_SET; }

数据输入时进行重采样

@Override public void queueInput(ByteBuffer inputBuffer) { // Prepare the output buffer. int position = inputBuffer.position(); int limit = inputBuffer.limit(); int size = limit - position; int resampledSize; switch (inputAudioFormat.encoding) { case C.ENCODING_PCM_8BIT: resampledSize = size * 2; break; case C.ENCODING_PCM_16BIT_BIG_ENDIAN: resampledSize = size; break; 。。。 } // Resample the little endian input and update the input/output buffers. ByteBuffer buffer = replaceOutputBuffer(resampledSize); switch (inputAudioFormat.encoding) { case C.ENCODING_PCM_8BIT: // 8 -> 16 bit resampling. Shift each byte from [0, 256) to [-128, 128) and scale up. for (int i = position; i < limit; i++) { buffer.put((byte) 0); buffer.put((byte) ((inputBuffer.get(i) & 0xFF) - 128)); } break; case C.ENCODING_PCM_16BIT_BIG_ENDIAN: // Big endian to little endian resampling. Swap the byte order. for (int i = position; i < limit; i += 2) { buffer.put(inputBuffer.get(i + 1)); buffer.put(inputBuffer.get(i)); } break; 。。。 } inputBuffer.position(inputBuffer.limit()); buffer.flip(); }

简单粗暴的转换


2ChannelMappingAudioProcessor

/** * An {@link AudioProcessor} that applies a mapping from input channels onto specified output * channels. This can be used to reorder, duplicate or discard channels. * 一个AudioProcessor,将输入通道的映射应用于指定的输出通道。 这可用于重新排序,复制或放弃频道。 */ /* package */ final class ChannelMappingAudioProcessor extends BaseAudioProcessor {}

这个是重新映射输出通道,demo中把8声道改成了6声道,把双声道改成了六声道。。。。

configure()的时候,把之前的inputAudioFormat中的outputChannel变更到了设置的pendingOutChannel。

/** * Resets the channel mapping. After calling this method, call {@link #configure(AudioFormat)} to * start using the new channel map. * * @param outputChannels The mapping from input to output channel indices, or {@code null} to * leave the input unchanged. * @see AudioSink#configure(com.google.android.exoplayer2.Format, int, int[]) */ public void setChannelMap(@Nullable int[] outputChannels) { pendingOutputChannels = outputChannels; } @Override public AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { @Nullable int[] outputChannels = pendingOutputChannels; 。。。 return active ? new AudioFormat(inputAudioFormat.sampleRate, outputChannels.length, C.ENCODING_PCM_16BIT) : AudioFormat.NOT_SET; }

数据流输入的时候,对新的目标channel循环做了变更

@Override public void queueInput(ByteBuffer inputBuffer) { int[] outputChannels = Assertions.checkNotNull(this.outputChannels); int position = inputBuffer.position(); int limit = inputBuffer.limit(); int frameCount = (limit - position) / inputAudioFormat.bytesPerFrame; int outputSize = frameCount * outputAudioFormat.bytesPerFrame; ByteBuffer buffer = replaceOutputBuffer(outputSize); while (position < limit) { for (int channelIndex : outputChannels) { buffer.putShort(inputBuffer.getShort(position + 2 * channelIndex)); } position += inputAudioFormat.bytesPerFrame; } inputBuffer.position(limit); buffer.flip(); }


3TrimmingAudioProcessor

/** Audio processor for trimming samples from the start/end of data音频处理器,用于从数据的开始/结束处修剪样本. */ /* package */ final class TrimmingAudioProcessor extends BaseAudioProcessor {}

设置起止帧

/** * Sets the number of audio frames to trim from the start and end of audio passed to this * processor. After calling this method, call {@link #configure(AudioFormat)} to apply the new * trimming frame counts. * 设置要从传递给此处理器的音频的开头和结尾修剪的音频帧数。 调用此方法后,调用 configure(AudioProcessor.AudioFormat) 以应用新的修剪帧计数。 * * 参数: * trimStartFrames – 从音频开始要修剪的音频帧数。 * trimEndFrames – 要从音频末尾修剪的音频帧数。 * * @param trimStartFrames The number of audio frames to trim from the start of audio. * @param trimEndFrames The number of audio frames to trim from the end of audio. * @see AudioSink#configure(com.google.android.exoplayer2.Format, int, int[]) */ public void setTrimFrameCount(int trimStartFrames, int trimEndFrames) { this.trimStartFrames = trimStartFrames; this.trimEndFrames = trimEndFrames; }

在DefaultAudioSink.configure方法进行了设置

trimmingAudioProcessor.setTrimFrameCount( inputFormat.encoderDelay, inputFormat.encoderPadding);

这俩变量的定义

/** * The number of frames to trim from the start of the decoded audio stream, or 0 if not * applicable. */ public final int encoderDelay; /** * The number of frames to trim from the end of the decoded audio stream, or 0 if not applicable. */ public final int encoderPadding;


配置

@Override public AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { if (inputAudioFormat.encoding != OUTPUT_ENCODING) { throw new UnhandledAudioFormatException(inputAudioFormat); } reconfigurationPending = true; return trimStartFrames != 0 || trimEndFrames != 0 ? inputAudioFormat : AudioFormat.NOT_SET; }

在DefaultAudioSink.handleBuffer方法处理流前,减去了剪去帧的时间

// Check that presentationTimeUs is consistent with the expected value. long expectedPresentationTimeUs = startMediaTimeUs + configuration.inputFramesToDurationUs( getSubmittedFrames() - trimmingAudioProcessor.getTrimmedFrameCount());

剪帧

@Override public void queueInput(ByteBuffer inputBuffer) { int position = inputBuffer.position(); int limit = inputBuffer.limit(); int remaining = limit - position; if (remaining == 0) { return; } // Trim any pending start bytes from the input buffer. int trimBytes = min(remaining, pendingTrimStartBytes);//取小 trimmedFrameCount += trimBytes / inputAudioFormat.bytesPerFrame; pendingTrimStartBytes -= trimBytes; inputBuffer.position(position + trimBytes); if (pendingTrimStartBytes > 0) { // Nothing to output yet. return; } remaining -= trimBytes; // endBuffer must be kept as full as possible, so that we trim the right amount of media if we // don't receive any more input. After taking into account the number of bytes needed to keep // endBuffer as full as possible, the output should be any surplus bytes currently in endBuffer // followed by any surplus bytes in the new inputBuffer.endBuffer 必须尽可能地保持满,这样如果我们不再接收到任何输入, // 我们就可以修剪适量的媒体。 考虑到保持 endBuffer 尽可能满所需的字节数后,输出应该是当前 endBuffer 中的任何剩余字节,然后是新 inputBuffer 中的任何剩余字节。 int remainingBytesToOutput = endBufferSize + remaining - endBuffer.length; ByteBuffer buffer = replaceOutputBuffer(remainingBytesToOutput); // Output from endBuffer. int endBufferBytesToOutput = Util.constrainValue(remainingBytesToOutput, 0, endBufferSize); buffer.put(endBuffer, 0, endBufferBytesToOutput); remainingBytesToOutput -= endBufferBytesToOutput; // Output from inputBuffer, restoring its limit afterwards. int inputBufferBytesToOutput = Util.constrainValue(remainingBytesToOutput, 0, remaining); inputBuffer.limit(inputBuffer.position() + inputBufferBytesToOutput); buffer.put(inputBuffer); inputBuffer.limit(limit); remaining -= inputBufferBytesToOutput; // Compact endBuffer, then repopulate it using the new input. endBufferSize -= endBufferBytesToOutput; System.arraycopy(endBuffer, endBufferBytesToOutput, endBuffer, 0, endBufferSize); inputBuffer.get(endBuffer, endBufferSize, remaining); endBufferSize += remaining; buffer.flip(); }


4)SilenceSkippingAudioProcessor

/** * An {@link AudioProcessor} that skips silence in the input stream. Input and output are 16-bit * PCM. * {@link AudioProcessor},可跳过输入流中的静音。 输入和输出是16位PCM。 */ public final class SilenceSkippingAudioProcessor extends BaseAudioProcessor {}

定义了三个状态

private @interface State {} /** State when the input is not silent 非静音时. */ private static final int STATE_NOISY = 0; /** State when the input may be silent but we haven't read enough yet to know说明何时输入可能是无声的,但我们还没有阅读足够的内容. */ private static final int STATE_MAYBE_SILENT = 1; /** State when the input is silent输入静音时的状态. */ private static final int STATE_SILENT = 2;

默认配置

/** Creates a new silence skipping audio processor.创建一个新的静音跳过音频处理器。 */ public SilenceSkippingAudioProcessor() { this( DEFAULT_MINIMUM_SILENCE_DURATION_US, //150s 音频的最小持续时间必须低于{@code silenceThresholdLevel}才能将音频的该部分分类为无声,以微秒为单位。 DEFAULT_PADDING_SILENCE_US, //20s 延长非沉默部分的沉默持续时间(以微秒为单位)。 该值不能超过{@code minimumSilenceDurationUs}。 DEFAULT_SILENCE_THRESHOLD_LEVEL); //1024 绝对水平(低于该绝对水平时,单个PCM样本被分类为无声)。 }

onFlush会默认配置成noisy状态,然后在实际处理状态时变更状态

@Override protected void onFlush() { if (enabled) { bytesPerFrame = inputAudioFormat.bytesPerFrame; int maybeSilenceBufferSize = durationUsToFrames(minimumSilenceDurationUs) * bytesPerFrame; if (maybeSilenceBuffer.length != maybeSilenceBufferSize) { maybeSilenceBuffer = new byte[maybeSilenceBufferSize]; } paddingSize = durationUsToFrames(paddingSilenceUs) * bytesPerFrame; if (paddingBuffer.length != paddingSize) { paddingBuffer = new byte[paddingSize]; } } state = STATE_NOISY; skippedFrames = 0; maybeSilenceBufferSize = 0; hasOutputNoise = false; }

输入处理

@Override public void queueInput(ByteBuffer inputBuffer) { while (inputBuffer.hasRemaining() && !hasPendingOutput()) { switch (state) { case STATE_NOISY: processNoisy(inputBuffer); break; case STATE_MAYBE_SILENT: processMaybeSilence(inputBuffer); break; case STATE_SILENT: processSilence(inputBuffer); break; default: throw new IllegalStateException(); } } }


处理非静音

// Internal methods. /** * Incrementally processes new input from {@code inputBuffer} while in {@link #STATE_NOISY}, * updating the state if needed. * 在 STATE_NOISY 中增量处理来自 inputBuffer 的新输入,如果需要更新状态。 */ private void processNoisy(ByteBuffer inputBuffer) { int limit = inputBuffer.limit(); // Check if there's any noise within the maybe silence buffer duration. inputBuffer.limit(min(limit, inputBuffer.position() + maybeSilenceBuffer.length)); int noiseLimit = findNoiseLimit(inputBuffer); if (noiseLimit == inputBuffer.position()) { // The buffer contains the start of possible silence. state = STATE_MAYBE_SILENT; } else { inputBuffer.limit(noiseLimit); output(inputBuffer); } // Restore the limit. inputBuffer.limit(limit); }


5)SonicAudioProcessor

/** * An {@link AudioProcessor} that uses the Sonic library to modify audio speed/pitch/sample rate. * 一个{@link AudioProcessor},它使用Sonic库来修改音频速度/音高/采样率。 */ public final class SonicAudioProcessor implements AudioProcessor {}

一样的调用套路,不同的是 用了一个Sonic库

输入

@Override public void queueInput(ByteBuffer inputBuffer) { if (!inputBuffer.hasRemaining()) { return; } Sonic sonic = checkNotNull(this.sonic); ShortBuffer shortBuffer = inputBuffer.asShortBuffer(); int inputSize = inputBuffer.remaining(); inputBytes += inputSize; sonic.queueInput(shortBuffer); inputBuffer.position(inputBuffer.position() + inputSize); }

输出

@Override public ByteBuffer getOutput() { @Nullable Sonic sonic = this.sonic; if (sonic != null) { int outputSize = sonic.getOutputSize(); if (outputSize > 0) { if (buffer.capacity() < outputSize) { buffer = ByteBuffer.allocateDirect(outputSize).order(ByteOrder.nativeOrder()); shortBuffer = buffer.asShortBuffer(); } else { buffer.clear(); shortBuffer.clear(); } sonic.getOutput(shortBuffer); outputBytes += outputSize; buffer.limit(outputSize); outputBuffer = buffer; } } ByteBuffer outputBuffer = this.outputBuffer; this.outputBuffer = EMPTY_BUFFER; return outputBuffer; }


6)用来调试的TeeAudioProcessor

这个demo里没调用,用在了test里。。。。

/** * Audio processor that outputs its input unmodified and also outputs its input to a given sink. * This is intended to be used for diagnostics and debugging. * * <p>This audio processor can be inserted into the audio processor chain to access audio data * before/after particular processing steps have been applied. For example, to get audio output * after playback speed adjustment and silence skipping have been applied it is necessary to pass a * custom {@link com.google.android.exoplayer2.audio.DefaultAudioSink.AudioProcessorChain} when * creating the audio sink, and include this audio processor after all other audio processors. * 音频处理器,其输出保持不变,并且将其输入输出到给定的接收器。 这旨在用于诊断和调试。 * 该音频处理器可以插入音频处理器链中,以在应用特定处理步骤之前/之后访问音频数据。 * 例如,要在应用了播放速度调整和静音跳过之后才能获得音频输出,则在创建音频接收器时必须传递自定义的DefaultAudioSink.AudioProcessorChain,并将此音频处理器包括在所有其他音频处理器之后。 */ public final class TeeAudioProcessor extends BaseAudioProcessor { /** A sink for audio buffers handled by the audio processor. */ public interface AudioBufferSink {}

/** * A sink for audio buffers that writes output audio as .wav files with a given path prefix. When * new audio data is handled after flushing the audio processor, a counter is incremented and its * value is appended to the output file name. * * <p>Note: if writing to external storage it's necessary to grant the {@code * WRITE_EXTERNAL_STORAGE} permission. * 音频缓冲区的接收器,用于将输出音频作为具有给定路径前缀的.wav文件写入。 在刷新音频处理器后处理新的音频数据时,计数器会增加,并将其值附加到输出文件名。 * 注意:如果要写入外部存储,则必须授予WRITE_EXTERNAL_STORAGE权限。 */ public static final class WavFileAudioBufferSink implements AudioBufferSink {}

}

直接在Chain里增加就可以调用

/** * Builds an {@link AudioSink} to which the audio renderers will output. * 构建一个{@link AudioSink},音频渲染器将输出到该音频。 * * @param context The {@link Context} associated with the player. * @param enableFloatOutput Whether to enable use of floating point audio output, if available. * @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link * android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported. * @param enableOffload Whether to enable use of audio offload for supported formats, if * available. * @return The {@link AudioSink} to which the audio renderers will output. May be {@code null} if * no audio renderers are required. If {@code null} is returned then {@link * #buildAudioRenderers} will not be called. */ @Nullable protected AudioSink buildAudioSink( Context context, boolean enableFloatOutput, boolean enableAudioTrackPlaybackParams, boolean enableOffload) { DefaultAudioSink.AudioProcessorChain chain = new DefaultAudioProcessorChain( new TeeAudioProcessor(new TeeAudioProcessor.WavFileAudioBufferSink("/mnt/sdcard/SENRSL/dc")) ); AudioSink audioSink = new DefaultAudioSink( AudioCapabilities.getCapabilities(context), chain, enableFloatOutput, enableAudioTrackPlaybackParams, enableOffload); return audioSink; }

捕获的文件也可以抓取到多声道

概览
完整名称                           : /Users/senrsl/Downloads/dc-06021912-0000.wav
格式                             : Wave
文件大小                           : 25.4 MiB
时长                             : 46 秒 272 毫秒
总体码率模式                         : 恒定码率 (CBR)
总体码率                           : 4 608 kb/s

音频
格式                             : PCM
格式设置                           : Little / Signed
编解码器 ID                        : 1
时长                             : 46 秒 272 毫秒
码率模式                           : 恒定码率 (CBR)
码率                             : 4 608 kb/s
声道数                            : 6 声道
采样率                            : 48.0 kHz
位深                             : 16 位
流大小                            : 25.4 MiB (100%)

适当调整,也就能捕获音频输出了,可以替代AudioPort之类的 隐藏API来用了。。。。

回家做饭,明天再发。。。。

2021年06月02日19:22:35

--
senRsl
2021年05月26日14:08:17

没有评论 :

发表评论