继承关系
AudioProcessor.png
接口定义
/** * Interface for audio processors, which take audio data as input and transform it, potentially * modifying its channel count, encoding and/or sample rate. * * <p>In addition to being able to modify the format of audio, implementations may allow parameters * to be set that affect the output audio and whether the processor is active/inactive. * 音频处理器的接口，它将音频数据作为输入并进行转换，从而有可能修改其通道数，编码和/或采样率。 * <p>除了能够修改音频的格式外，实现还可以设置影响输出音频以及处理器是否处于活动状态的参数。 */ public interface AudioProcessor { /** PCM audio format that may be handled by an audio processor.音频处理器可以处理的PCM音频格式。 */ final class AudioFormat { public static final AudioFormat NOT_SET = //没有设置，就都是-1 new AudioFormat( /* sampleRate= */ Format.NO_VALUE, /* channelCount= */ Format.NO_VALUE, /* encoding= */ Format.NO_VALUE); /** The sample rate in Hertz赫兹采样率. */ public final int sampleRate; /** The number of interleaved channels. */ public final int channelCount; /** The type of linear PCM encoding. */ @C.PcmEncoding public final int encoding; /** The number of bytes used to represent one audio frame用来表示一个音频帧的字节数. */ public final int bytesPerFrame; public AudioFormat(int sampleRate, int channelCount, @C.PcmEncoding int encoding) { this.sampleRate = sampleRate; this.channelCount = channelCount; this.encoding = encoding; bytesPerFrame = Util.isEncodingLinearPcm(encoding) ? Util.getPcmFrameSize(encoding, channelCount) : Format.NO_VALUE; Logger.w("AudioProcessor",sampleRate,channelCount,encoding,bytesPerFrame); } @Override public String toString() { return "AudioProcessor 音频处理 AudioFormat[" + "sampleRate=" + sampleRate + ", channelCount=" + channelCount + ", encoding=" + encoding + ']'; } } /** Exception thrown when a processor can't be configured for a given input audio format * 无法为给定的输入音频格式配置处理器时抛出异常. */ final class UnhandledAudioFormatException extends Exception { public UnhandledAudioFormatException(AudioFormat inputAudioFormat) { super("Unhandled format: " + inputAudioFormat); } } /** An empty, direct {@link ByteBuffer}. */ ByteBuffer EMPTY_BUFFER = ByteBuffer.allocateDirect(0).order(ByteOrder.nativeOrder()); /** * Configures the processor to process input audio with the specified format. After calling this * method, call {@link #isActive()} to determine whether the audio processor is active. Returns * the configured output audio format if this instance is active. * * <p>After calling this method, it is necessary to {@link #flush()} the processor to apply the * new configuration. Before applying the new configuration, it is safe to queue input and get * output in the old input/output formats. Call {@link #queueEndOfStream()} when no more input * will be supplied in the old input format. * 将处理器配置为处理指定格式的输入音频。 调用此方法后，调用isActive（）以确定音频处理器是否处于活动状态。 如果此实例处于活动状态，则返回配置的输出音频格式。 * 调用此方法后，有必要flush（）处理器以应用新配置。 在应用新配置之前，可以安全地将输入排队并以旧的输入/输出格式获取输出。 当不再以旧的输入格式提供输入时，请调用queueEndOfStream（）。 * * @param inputAudioFormat The format of audio that will be queued after the next call to {@link * #flush()}. * @return The configured output audio format if this instance is {@link #isActive() active}. * @throws UnhandledAudioFormatException Thrown if the specified format can't be handled as input. */ AudioFormat configure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException; /** Returns whether the processor is configured and will process input buffers返回是否配置了处理器并将处理输入缓冲区. */ boolean isActive(); /** * Queues audio data between the position and limit of the input {@code buffer} for processing. * {@code buffer} must be a direct byte buffer with native byte order. Its contents are treated as * read-only. Its position will be advanced by the number of bytes consumed (which may be zero). * The caller retains ownership of the provided buffer. Calling this method invalidates any * previous buffer returned by {@link #getOutput()}. * 在输入缓冲区的位置和限制之间排队音频数据以进行处理。 * 缓冲区必须是具有本地字节顺序的直接字节缓冲区。 其内容被视为只读。 它的位置将增加所消耗的字节数（可能为零）。 * 调用方保留提供的缓冲区的所有权。 调用此方法会使getOutput（）返回的所有先前缓冲区无效。 * * @param buffer The input buffer to process. */ void queueInput(ByteBuffer buffer); /** * Queues an end of stream signal. After this method has been called, * {@link #queueInput(ByteBuffer)} may not be called until after the next call to * {@link #flush()}. Calling {@link #getOutput()} will return any remaining output data. Multiple * calls may be required to read all of the remaining output data. {@link #isEnded()} will return * {@code true} once all remaining output data has been read. * 将流信号的末尾排队。 * 调用此方法后，直到下一次调用flush（）之后，才能调用queueInput（ByteBuffer）。 * 调用getOutput（）将返回所有剩余的输出数据。 可能需要多次调用才能读取所有剩余的输出数据。 一旦读取了所有剩余的输出数据，isEnded（）将返回true。 */ void queueEndOfStream(); /** * Returns a buffer containing processed output data between its position and limit. The buffer * will always be a direct byte buffer with native byte order. Calling this method invalidates any * previously returned buffer. The buffer will be empty if no output is available. * 返回一个缓冲区，该缓冲区包含在其位置和限制之间的已处理输出数据。 缓冲区将始终是具有本地字节顺序的直接字节缓冲区。 * 调用此方法会使以前返回的所有缓冲区无效。 如果没有可用的输出，缓冲区将为空。 * * 返回值： * 包含在其位置和极限之间的已处理输出数据的缓冲区。 * * @return A buffer containing processed output data between its position and limit. */ ByteBuffer getOutput(); /** * Returns whether this processor will return no more output from {@link #getOutput()} until it * has been {@link #flush()}ed and more input has been queued. */ boolean isEnded(); /** * Clears any buffered data and pending output. If the audio processor is active, also prepares * the audio processor to receive a new stream of input in the last configured (pending) format. * 清除所有缓冲的数据和挂起的输出。 如果音频处理器处于活动状态，则还要准备音频处理器以接收最后配置的（待定）格式的新输入流。 */ void flush(); /** Resets the processor to its unconfigured state, releasing any resources. */ void reset(); }
在demo中，实际使用在DefaultAudioSink
入口在DefaultRenderFactory
/** * Builds an {@link AudioSink} to which the audio renderers will output. * 构建一个{@link AudioSink}，音频渲染器将输出到该音频。 * * @param context The {@link Context} associated with the player. * @param enableFloatOutput Whether to enable use of floating point audio output, if available. * @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link * android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported. * @param enableOffload Whether to enable use of audio offload for supported formats, if * available. * @return The {@link AudioSink} to which the audio renderers will output. May be {@code null} if * no audio renderers are required. If {@code null} is returned then {@link * #buildAudioRenderers} will not be called. */ @Nullable protected AudioSink buildAudioSink( Context context, boolean enableFloatOutput, boolean enableAudioTrackPlaybackParams, boolean enableOffload) { return new DefaultAudioSink( AudioCapabilities.getCapabilities(context), new DefaultAudioProcessorChain(), enableFloatOutput, enableAudioTrackPlaybackParams, enableOffload); }
对于AudioProcess的调用，就是循环各个processor。。。。
/** * Creates a new default audio sink, optionally using float output for high resolution PCM and * with the specified {@code audioProcessorChain}. * 创建一个新的默认音频接收器，还可以选择将浮点输出用于高分辨率PCM，并使用指定的{@code audioProcessorChain}。 * * @param audioCapabilities The audio capabilities for playback on this device在此设备上播放的音频功能. May be null if the * default capabilities (no encoded audio passthrough support) should be assumed. * @param audioProcessorChain An {@link AudioProcessorChain} which is used to apply playback * parameters adjustments. The instance passed in must not be reused in other sinks. * @param enableFloatOutput Whether to enable 32-bit float output. Where possible, 32-bit float * output will be used if the input is 32-bit float, and also if the input is high resolution * (24-bit or 32-bit) integer PCM. Float output is supported from API level 21. Audio * processing (for example, speed adjustment) will not be available when float output is in * use. * @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link * android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported. * @param enableOffload Whether to enable audio offload. If an audio format can be both played * with offload and encoded audio passthrough, it will be played in offload. Audio offload is * supported from API level 29. Most Android devices can only support one offload {@link * android.media.AudioTrack} at a time and can invalidate it at any time. Thus an app can * never be guaranteed that it will be able to play in offload. Audio processing (for example, * speed adjustment) will not be available when offload is in use. */ public DefaultAudioSink( @Nullable AudioCapabilities audioCapabilities, AudioProcessorChain audioProcessorChain, boolean enableFloatOutput, boolean enableAudioTrackPlaybackParams, boolean enableOffload) { this.audioCapabilities = audioCapabilities; this.audioProcessorChain = Assertions.checkNotNull(audioProcessorChain); this.enableFloatOutput = Util.SDK_INT >= 21 && enableFloatOutput; this.enableAudioTrackPlaybackParams = Util.SDK_INT >= 23 && enableAudioTrackPlaybackParams; this.enableOffload = Util.SDK_INT >= 29 && enableOffload; releasingConditionVariable = new ConditionVariable(true); audioTrackPositionTracker = new AudioTrackPositionTracker(new PositionTrackerListener()); channelMappingAudioProcessor = new ChannelMappingAudioProcessor(); trimmingAudioProcessor = new TrimmingAudioProcessor(); ArrayList<AudioProcessor> toIntPcmAudioProcessors = new ArrayList<>(); Collections.addAll( toIntPcmAudioProcessors, new ResamplingAudioProcessor(), channelMappingAudioProcessor, trimmingAudioProcessor); Collections.addAll(toIntPcmAudioProcessors, audioProcessorChain.getAudioProcessors()); toIntPcmAvailableAudioProcessors = toIntPcmAudioProcessors.toArray(new AudioProcessor[0]); toFloatPcmAvailableAudioProcessors = new AudioProcessor[] {new FloatResamplingAudioProcessor()}; volume = 1f; audioAttributes = AudioAttributes.DEFAULT; audioSessionId = C.AUDIO_SESSION_ID_UNSET; auxEffectInfo = new AuxEffectInfo(AuxEffectInfo.NO_AUX_EFFECT_ID, 0f); mediaPositionParameters = new MediaPositionParameters( PlaybackParameters.DEFAULT, DEFAULT_SKIP_SILENCE, /* mediaTimeUs= */ 0, /* audioTrackPositionUs= */ 0); audioTrackPlaybackParameters = PlaybackParameters.DEFAULT; drainingAudioProcessorIndex = C.INDEX_UNSET; activeAudioProcessors = new AudioProcessor[0]; outputBuffers = new ByteBuffer[0]; mediaPositionParametersCheckpoints = new ArrayDeque<>(); initializationExceptionPendingExceptionHolder = new PendingExceptionHolder<>(AUDIO_TRACK_RETRY_DURATION_MS); writeExceptionPendingExceptionHolder = new PendingExceptionHolder<>(AUDIO_TRACK_RETRY_DURATION_MS); }
搞了两个AudioProcessor数组，一个是toIntPcm Avaliable AudioProcessor，一个是toFloatPcm Avaliable AudioProcess。
分别对应解码后的输出类型，因为好多设备不支持浮点输出
然后去config
@Override public void configure(Format inputFormat, int specifiedBufferSize, @Nullable int[] outputChannels) throws ConfigurationException { int inputPcmFrameSize; @Nullable AudioProcessor[] availableAudioProcessors; 。。。 Logger.w(TAG,"configure方法",inputFormat.toString(),specifiedBufferSize,outputChannels);//Format(2, null, null, audio/ac3, null, -1, en, [-1, -1, -1.0], [6, 48000]),0,null if (MimeTypes.AUDIO_RAW.equals(inputFormat.sampleMimeType)) { Assertions.checkArgument(Util.isEncodingLinearPcm(inputFormat.pcmEncoding)); inputPcmFrameSize = Util.getPcmFrameSize(inputFormat.pcmEncoding, inputFormat.channelCount); availableAudioProcessors = shouldUseFloatOutput(inputFormat.pcmEncoding) ? toFloatPcmAvailableAudioProcessors : toIntPcmAvailableAudioProcessors; 。。。 AudioProcessor.AudioFormat outputFormat = new AudioProcessor.AudioFormat( inputFormat.sampleRate, inputFormat.channelCount, inputFormat.pcmEncoding); for (AudioProcessor audioProcessor : availableAudioProcessors) { try { AudioProcessor.AudioFormat nextFormat = audioProcessor.configure(outputFormat); if (audioProcessor.isActive()) { outputFormat = nextFormat; Logger.w(TAG,"configure",outputFormat); } } catch (UnhandledAudioFormatException e) { throw new ConfigurationException(e, inputFormat); } } outputMode = OUTPUT_MODE_PCM; outputEncoding = outputFormat.encoding; outputSampleRate = outputFormat.sampleRate; outputChannelConfig = Util.getAudioTrackChannelConfig(outputFormat.channelCount); outputPcmFrameSize = Util.getPcmFrameSize(outputEncoding, outputFormat.channelCount); } else { inputPcmFrameSize = C.LENGTH_UNSET; availableAudioProcessors = new AudioProcessor[0]; outputSampleRate = inputFormat.sampleRate; outputPcmFrameSize = C.LENGTH_UNSET; Logger.w(TAG,"configure方法x2",enableOffload,isOffloadedPlaybackSupported(inputFormat, audioAttributes));//false,false if (enableOffload && isOffloadedPlaybackSupported(inputFormat, audioAttributes)) { outputMode = OUTPUT_MODE_OFFLOAD; outputEncoding = MimeTypes.getEncoding( Assertions.checkNotNull(inputFormat.sampleMimeType), inputFormat.codecs); outputChannelConfig = Util.getAudioTrackChannelConfig(inputFormat.channelCount); } else { outputMode = OUTPUT_MODE_PASSTHROUGH;//直通输出 @Nullable Pair<Integer, Integer> encodingAndChannelConfig = getEncodingAndChannelConfigForPassthrough(inputFormat, audioCapabilities); Logger.w("pass though x2",encodingAndChannelConfig);//Pair{5 252} if (encodingAndChannelConfig == null) { throw new ConfigurationException( "Unable to configure passthrough for: " + inputFormat, inputFormat); } outputEncoding = encodingAndChannelConfig.first;//5 outputChannelConfig = encodingAndChannelConfig.second;//252 } } 。。。 }
也就是说，需要满足inputFormat.sampleMImeType是 /raw才会去配置 AudioProcessor
也就是会走解码输出
否则会自动的去走音频分载或者passthrough
然后实际的干活方法里
AudioSink接口
/** * Attempts to process data from a {@link ByteBuffer}, starting from its current position and * ending at its limit (exclusive). The position of the {@link ByteBuffer} is advanced by the * number of bytes that were handled. {@link Listener#onPositionDiscontinuity()} will be called if * {@code presentationTimeUs} is discontinuous with the last buffer handled since the last reset. * 尝试处理ByteBuffer的数据，从其当前位置开始，直到其限制（不包括限制）。 * ByteBuffer的位置提前处理的字节数。 如果presentationTimeUs与自上次重置以来处理的最后一个缓冲区不连续，则将调用AudioSink.Listener.onPositionDiscontinuity（）。 * 返回数据是否已全部处理。 如果未对数据进行完整处理，则必须将相同的ByteBuffer提供给后续调用，直到完全消耗完为止， * 除非是对flush（）（或configure（Format，int，int []）的中间调用） 导致水槽被冲洗）。 * * <p>Returns whether the data was handled in full. If the data was not handled in full then the * same {@link ByteBuffer} must be provided to subsequent calls until it has been fully consumed, * except in the case of an intervening call to {@link #flush()} (or to {@link #configure(Format, * int, int[])} that causes the sink to be flushed). * * @param buffer The buffer containing audio data. * @param presentationTimeUs The presentation timestamp of the buffer in microseconds. * @param encodedAccessUnitCount The number of encoded access units in the buffer, or 1 if the * buffer contains PCM audio. This allows batching multiple encoded access units in one * buffer. * @return Whether the buffer was handled fully. * @throws InitializationException If an error occurs initializing the sink. * @throws WriteException If an error occurs writing the audio data. */ boolean handleBuffer(ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount) throws InitializationException, WriteException;
默认实现
@Override @SuppressWarnings("ReferenceEquality") public boolean handleBuffer( ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount) throws InitializationException, WriteException { Assertions.checkArgument(inputBuffer == null || buffer == inputBuffer); if (pendingConfiguration != null) { 。。。。 // Re-apply playback parameters. applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); } if (!isAudioTrackInitialized()) { try { initializeAudioTrack(); //初始化 audioTrack } catch (InitializationException e) { 。。。。 return false; } } initializationExceptionPendingExceptionHolder.clear(); if (startMediaTimeUsNeedsInit) { 。。。。 applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); if (playing) { play();//audioTrack开始播放 } } if (!audioTrackPositionTracker.mayHandleBuffer(getWrittenFrames())) { return false; } if (inputBuffer == null) { // We are seeing this buffer for the first time. Assertions.checkArgument(buffer.order() == ByteOrder.LITTLE_ENDIAN); if (!buffer.hasRemaining()) { // The buffer is empty. return true; } 。。。。 if (afterDrainParameters != null) { if (!drainToEndOfStream()) { // Don't process any more input until draining completes. return false; } applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); afterDrainParameters = null; } // Check that presentationTimeUs is consistent with the expected value. long expectedPresentationTimeUs = startMediaTimeUs + configuration.inputFramesToDurationUs( getSubmittedFrames() - trimmingAudioProcessor.getTrimmedFrameCount()); if (!startMediaTimeUsNeedsSync && Math.abs(expectedPresentationTimeUs - presentationTimeUs) > 200000) { Log.e( TAG, "Discontinuity detected [expected " + expectedPresentationTimeUs + ", got " + presentationTimeUs + "]"); startMediaTimeUsNeedsSync = true; } if (startMediaTimeUsNeedsSync) { if (!drainToEndOfStream()) { // Don't update timing until pending AudioProcessor buffers are completely drained. return false; } // Adjust startMediaTimeUs to be consistent with the current buffer's start time and the // number of bytes submitted. long adjustmentUs = presentationTimeUs - expectedPresentationTimeUs; startMediaTimeUs += adjustmentUs; startMediaTimeUsNeedsSync = false; // Re-apply playback parameters because the startMediaTimeUs changed. applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs); if (listener != null && adjustmentUs != 0) { listener.onPositionDiscontinuity(); } } if (configuration.outputMode == OUTPUT_MODE_PCM) { submittedPcmBytes += buffer.remaining(); } else { submittedEncodedFrames += framesPerEncodedSample * encodedAccessUnitCount; } inputBuffer = buffer; inputBufferAccessUnitCount = encodedAccessUnitCount; } processBuffers(presentationTimeUs);//调用02篇的build时创建的音频处理器处理 if (!inputBuffer.hasRemaining()) { inputBuffer = null; inputBufferAccessUnitCount = 0; return true; } if (audioTrackPositionTracker.isStalled(getWrittenFrames())) { Log.w(TAG, "Resetting stalled audio track"); flush(); return true; } return false; }
两处对audio processor调用，一个是设置audioProcessor，设置完成后用来处理
设置
private void setupAudioProcessors() { AudioProcessor[] audioProcessors = configuration.availableAudioProcessors; ArrayList<AudioProcessor> newAudioProcessors = new ArrayList<>(); for (AudioProcessor audioProcessor : audioProcessors) { if (audioProcessor.isActive()) { newAudioProcessors.add(audioProcessor); } else { audioProcessor.flush(); } } int count = newAudioProcessors.size(); activeAudioProcessors = newAudioProcessors.toArray(new AudioProcessor[count]); outputBuffers = new ByteBuffer[count]; flushAudioProcessors(); } private void flushAudioProcessors() { for (int i = 0; i < activeAudioProcessors.length; i++) { AudioProcessor audioProcessor = activeAudioProcessors[i]; audioProcessor.flush(); outputBuffers[i] = audioProcessor.getOutput(); } }
用来处理
private void processBuffers(long avSyncPresentationTimeUs) throws WriteException { int count = activeAudioProcessors.length; int index = count; Logger.w(TAG,"processBuffers",avSyncPresentationTimeUs,count);//0,0 | 32000,0 | 64000，0|96000，0 while (index >= 0) { ByteBuffer input = index > 0 ? outputBuffers[index - 1] : (inputBuffer != null ? inputBuffer : AudioProcessor.EMPTY_BUFFER); if (index == count) { writeBuffer(input, avSyncPresentationTimeUs); } else { AudioProcessor audioProcessor = activeAudioProcessors[index]; if (index > drainingAudioProcessorIndex) { audioProcessor.queueInput(input); //输入 } ByteBuffer output = audioProcessor.getOutput();//输出 outputBuffers[index] = output; if (output.hasRemaining()) { // Handle the output as input to the next audio processor or the AudioTrack. index++; continue; } } if (input.hasRemaining()) { // The input wasn't consumed and no output was produced, so give up for now. return; } // Get more input from upstream. index--; } }
这逻辑写的。。。。
index == count就输出音频到audioTrack,index--
音频processor处理完成后就index++
也就是说，DefaultAudioSink对AudioProcessor接口的调用是这样的
狄仁杰上线
狄仁杰：我想事情的真相应该是这样的.jpg
new DefaultAudioSink()的时候，增加了增加了几个AudioProcessor的实现，也就是增加了几个音频处理器的不同实现到toIntPcmAudioProcessors数组；
然后在DefaultAudioSink.configure()的时候，判断如果是解码输出，就先判断是不是浮点输出，如果是toIntPcm而不是toFloatPcm，就把avaliableAudioProcessors指定为toIntPcmAudioProcessor数组，然后循环avaliableProcessor数组，挨个调用AudioProcessor实现类的configure()方法，返回AudioProcess.AudioFormat，判断AudioProcessor.isActive()，如果is active，就会使用返回的AudioFormat去new Configuration()，这个后面的大配置。
接下来，就到了处理数据流的时候，首先设置 setupAudioProcess()，依然是循环那几个实现，取得就是大配置configuration.avaliableAudioProcessor数组，挨个判断是否启用isActive，如果启用就增加到 activeAudioProcessor数组，干完后循环 调用activeAudioProcessor.flush()，flush后配置全类私有数组outputBuffers[i] = activieAudioProcess[i].getOutputj(); 也就是保留了对应的输出通道。
接下来，终于要实际处理流了，依然是要循环子处理器实现通过上面的outBuffer[index]取对应子处理器的返回,然后调用audioProcessor.queueInput()，audioProcess.getOutput()，写入到audioTrack....
对于AudioProcessor的调用顺序，简单理解为 configure() -> isActive() ->
flush() -> queueInput() -> getOutput()。
回到configuration方法
Configuration pendingConfiguration = new Configuration( inputFormat, inputPcmFrameSize, outputMode, outputPcmFrameSize, outputSampleRate, outputChannelConfig, outputEncoding, specifiedBufferSize, enableAudioTrackPlaybackParams, availableAudioProcessors);
解码输出时，avaliableAudioProcessors为 toIntPcmAvaliableAudioProcessors或toIntPcmXxxxxx
不解码输出时，为empty
有了 avaliableAudioProcessors才有了后面的 audioProcessor.process()
所以如果要直通输出，可以在这里增加audio processor....
解码输出，构造函数初始化时就配置了四个默认的处理
channelMappingAudioProcessor = new ChannelMappingAudioProcessor(); trimmingAudioProcessor = new TrimmingAudioProcessor(); ArrayList<AudioProcessor> toIntPcmAudioProcessors = new ArrayList<>(); Collections.addAll( toIntPcmAudioProcessors, new ResamplingAudioProcessor(), channelMappingAudioProcessor, trimmingAudioProcessor); Collections.addAll(toIntPcmAudioProcessors, audioProcessorChain.getAudioProcessors()); toIntPcmAvailableAudioProcessors = toIntPcmAudioProcessors.toArray(new AudioProcessor[0]); toFloatPcmAvailableAudioProcessors = new AudioProcessor[] {new FloatResamplingAudioProcessor()};
分别为
ResamplingAudioProcessor 重采样
ChannelMappingAudioProcessor 声道
TrimmingAudioProcessor 修剪？
还有传过来的 audioProcessorChain.getAudioProcessors()
/** * Creates a new default chain of audio processors, with the user-defined {@code * audioProcessors} applied before silence skipping and speed adjustment processors. */ public DefaultAudioProcessorChain(AudioProcessor... audioProcessors) { this(audioProcessors, new SilenceSkippingAudioProcessor(), new SonicAudioProcessor()); }
按照添加顺序，分别为
SilenceSkippingAudioProcessor 静音跳过
SonicAudioProcessor uses the Sonic library to modify audio
speed/pitch/sample rate
demo中对解码输出 总共应用了这五个音频处理器。
1）ResamplingAudioProcessor
/** * An {@link AudioProcessor} that converts different PCM audio encodings to 16-bit integer PCM. The * following encodings are supported as input: * 将不同的PCM音频编码转换为16位整数PCM的AudioProcessor。 支持以下编码作为输入： * * <ul> * <li>{@link C#ENCODING_PCM_8BIT} * <li>{@link C#ENCODING_PCM_16BIT} ({@link #isActive()} will return {@code false}) * <li>{@link C#ENCODING_PCM_16BIT_BIG_ENDIAN} * <li>{@link C#ENCODING_PCM_24BIT} * <li>{@link C#ENCODING_PCM_32BIT} * <li>{@link C#ENCODING_PCM_FLOAT} * </ul> */ /* package */ final class ResamplingAudioProcessor extends BaseAudioProcessor {}
实现也是简单粗暴，只重写了configure() 和 queueInput()方法
@Override public AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { @C.PcmEncoding int encoding = inputAudioFormat.encoding; if (encoding != C.ENCODING_PCM_8BIT && encoding != C.ENCODING_PCM_16BIT && encoding != C.ENCODING_PCM_16BIT_BIG_ENDIAN && encoding != C.ENCODING_PCM_24BIT && encoding != C.ENCODING_PCM_32BIT && encoding != C.ENCODING_PCM_FLOAT) { throw new UnhandledAudioFormatException(inputAudioFormat); } return encoding != C.ENCODING_PCM_16BIT ? new AudioFormat( inputAudioFormat.sampleRate, inputAudioFormat.channelCount, C.ENCODING_PCM_16BIT) : AudioFormat.NOT_SET; }
onConfigure时，不是支持的格式直接抛异常，不是16bit 返回16bit....
onConfigure定义与BaseAudioProcessor抽象类，是AudioProcessor接口中configure方法的延伸
BaseAudioProcessor抽象类：
@Override public final AudioFormat configure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { pendingInputAudioFormat = inputAudioFormat; pendingOutputAudioFormat = onConfigure(inputAudioFormat); return isActive() ? pendingOutputAudioFormat : AudioFormat.NOT_SET; }
/** Called when the processor is configured for a new input format. */ protected AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { return AudioFormat.NOT_SET; }
数据输入时进行重采样
@Override public void queueInput(ByteBuffer inputBuffer) { // Prepare the output buffer. int position = inputBuffer.position(); int limit = inputBuffer.limit(); int size = limit - position; int resampledSize; switch (inputAudioFormat.encoding) { case C.ENCODING_PCM_8BIT: resampledSize = size * 2; break; case C.ENCODING_PCM_16BIT_BIG_ENDIAN: resampledSize = size; break; 。。。 } // Resample the little endian input and update the input/output buffers. ByteBuffer buffer = replaceOutputBuffer(resampledSize); switch (inputAudioFormat.encoding) { case C.ENCODING_PCM_8BIT: // 8 -> 16 bit resampling. Shift each byte from [0, 256) to [-128, 128) and scale up. for (int i = position; i < limit; i++) { buffer.put((byte) 0); buffer.put((byte) ((inputBuffer.get(i) & 0xFF) - 128)); } break; case C.ENCODING_PCM_16BIT_BIG_ENDIAN: // Big endian to little endian resampling. Swap the byte order. for (int i = position; i < limit; i += 2) { buffer.put(inputBuffer.get(i + 1)); buffer.put(inputBuffer.get(i)); } break; 。。。 } inputBuffer.position(inputBuffer.limit()); buffer.flip(); }
简单粗暴的转换
2）ChannelMappingAudioProcessor
/** * An {@link AudioProcessor} that applies a mapping from input channels onto specified output * channels. This can be used to reorder, duplicate or discard channels. * 一个AudioProcessor，将输入通道的映射应用于指定的输出通道。 这可用于重新排序，复制或放弃频道。 */ /* package */ final class ChannelMappingAudioProcessor extends BaseAudioProcessor {}
这个是重新映射输出通道，demo中把8声道改成了6声道，把双声道改成了六声道。。。。
configure()的时候，把之前的inputAudioFormat中的outputChannel变更到了设置的pendingOutChannel。
/** * Resets the channel mapping. After calling this method, call {@link #configure(AudioFormat)} to * start using the new channel map. * * @param outputChannels The mapping from input to output channel indices, or {@code null} to * leave the input unchanged. * @see AudioSink#configure(com.google.android.exoplayer2.Format, int, int[]) */ public void setChannelMap(@Nullable int[] outputChannels) { pendingOutputChannels = outputChannels; } @Override public AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { @Nullable int[] outputChannels = pendingOutputChannels; 。。。 return active ? new AudioFormat(inputAudioFormat.sampleRate, outputChannels.length, C.ENCODING_PCM_16BIT) : AudioFormat.NOT_SET; }
数据流输入的时候，对新的目标channel循环做了变更
@Override public void queueInput(ByteBuffer inputBuffer) { int[] outputChannels = Assertions.checkNotNull(this.outputChannels); int position = inputBuffer.position(); int limit = inputBuffer.limit(); int frameCount = (limit - position) / inputAudioFormat.bytesPerFrame; int outputSize = frameCount * outputAudioFormat.bytesPerFrame; ByteBuffer buffer = replaceOutputBuffer(outputSize); while (position < limit) { for (int channelIndex : outputChannels) { buffer.putShort(inputBuffer.getShort(position + 2 * channelIndex)); } position += inputAudioFormat.bytesPerFrame; } inputBuffer.position(limit); buffer.flip(); }
3）TrimmingAudioProcessor
/** Audio processor for trimming samples from the start/end of data音频处理器，用于从数据的开始/结束处修剪样本. */ /* package */ final class TrimmingAudioProcessor extends BaseAudioProcessor {}
设置起止帧
/** * Sets the number of audio frames to trim from the start and end of audio passed to this * processor. After calling this method, call {@link #configure(AudioFormat)} to apply the new * trimming frame counts. * 设置要从传递给此处理器的音频的开头和结尾修剪的音频帧数。 调用此方法后，调用 configure(AudioProcessor.AudioFormat) 以应用新的修剪帧计数。 * * 参数： * trimStartFrames – 从音频开始要修剪的音频帧数。 * trimEndFrames – 要从音频末尾修剪的音频帧数。 * * @param trimStartFrames The number of audio frames to trim from the start of audio. * @param trimEndFrames The number of audio frames to trim from the end of audio. * @see AudioSink#configure(com.google.android.exoplayer2.Format, int, int[]) */ public void setTrimFrameCount(int trimStartFrames, int trimEndFrames) { this.trimStartFrames = trimStartFrames; this.trimEndFrames = trimEndFrames; }
在DefaultAudioSink.configure方法进行了设置
trimmingAudioProcessor.setTrimFrameCount( inputFormat.encoderDelay, inputFormat.encoderPadding);
这俩变量的定义
/** * The number of frames to trim from the start of the decoded audio stream, or 0 if not * applicable. */ public final int encoderDelay; /** * The number of frames to trim from the end of the decoded audio stream, or 0 if not applicable. */ public final int encoderPadding;
配置
@Override public AudioFormat onConfigure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException { if (inputAudioFormat.encoding != OUTPUT_ENCODING) { throw new UnhandledAudioFormatException(inputAudioFormat); } reconfigurationPending = true; return trimStartFrames != 0 || trimEndFrames != 0 ? inputAudioFormat : AudioFormat.NOT_SET; }
在DefaultAudioSink.handleBuffer方法处理流前，减去了剪去帧的时间
// Check that presentationTimeUs is consistent with the expected value. long expectedPresentationTimeUs = startMediaTimeUs + configuration.inputFramesToDurationUs( getSubmittedFrames() - trimmingAudioProcessor.getTrimmedFrameCount());
剪帧
@Override public void queueInput(ByteBuffer inputBuffer) { int position = inputBuffer.position(); int limit = inputBuffer.limit(); int remaining = limit - position; if (remaining == 0) { return; } // Trim any pending start bytes from the input buffer. int trimBytes = min(remaining, pendingTrimStartBytes);//取小 trimmedFrameCount += trimBytes / inputAudioFormat.bytesPerFrame; pendingTrimStartBytes -= trimBytes; inputBuffer.position(position + trimBytes); if (pendingTrimStartBytes > 0) { // Nothing to output yet. return; } remaining -= trimBytes; // endBuffer must be kept as full as possible, so that we trim the right amount of media if we // don't receive any more input. After taking into account the number of bytes needed to keep // endBuffer as full as possible, the output should be any surplus bytes currently in endBuffer // followed by any surplus bytes in the new inputBuffer.endBuffer 必须尽可能地保持满，这样如果我们不再接收到任何输入， // 我们就可以修剪适量的媒体。 考虑到保持 endBuffer 尽可能满所需的字节数后，输出应该是当前 endBuffer 中的任何剩余字节，然后是新 inputBuffer 中的任何剩余字节。 int remainingBytesToOutput = endBufferSize + remaining - endBuffer.length; ByteBuffer buffer = replaceOutputBuffer(remainingBytesToOutput); // Output from endBuffer. int endBufferBytesToOutput = Util.constrainValue(remainingBytesToOutput, 0, endBufferSize); buffer.put(endBuffer, 0, endBufferBytesToOutput); remainingBytesToOutput -= endBufferBytesToOutput; // Output from inputBuffer, restoring its limit afterwards. int inputBufferBytesToOutput = Util.constrainValue(remainingBytesToOutput, 0, remaining); inputBuffer.limit(inputBuffer.position() + inputBufferBytesToOutput); buffer.put(inputBuffer); inputBuffer.limit(limit); remaining -= inputBufferBytesToOutput; // Compact endBuffer, then repopulate it using the new input. endBufferSize -= endBufferBytesToOutput; System.arraycopy(endBuffer, endBufferBytesToOutput, endBuffer, 0, endBufferSize); inputBuffer.get(endBuffer, endBufferSize, remaining); endBufferSize += remaining; buffer.flip(); }
4）SilenceSkippingAudioProcessor
/** * An {@link AudioProcessor} that skips silence in the input stream. Input and output are 16-bit * PCM. * {@link AudioProcessor}，可跳过输入流中的静音。 输入和输出是16位PCM。 */ public final class SilenceSkippingAudioProcessor extends BaseAudioProcessor {}
定义了三个状态
private @interface State {} /** State when the input is not silent 非静音时. */ private static final int STATE_NOISY = 0; /** State when the input may be silent but we haven't read enough yet to know说明何时输入可能是无声的，但我们还没有阅读足够的内容. */ private static final int STATE_MAYBE_SILENT = 1; /** State when the input is silent输入静音时的状态. */ private static final int STATE_SILENT = 2;
默认配置
/** Creates a new silence skipping audio processor.创建一个新的静音跳过音频处理器。 */ public SilenceSkippingAudioProcessor() { this( DEFAULT_MINIMUM_SILENCE_DURATION_US, //150s 音频的最小持续时间必须低于{@code silenceThresholdLevel}才能将音频的该部分分类为无声，以微秒为单位。 DEFAULT_PADDING_SILENCE_US, //20s 延长非沉默部分的沉默持续时间（以微秒为单位）。 该值不能超过{@code minimumSilenceDurationUs}。 DEFAULT_SILENCE_THRESHOLD_LEVEL); //1024 绝对水平（低于该绝对水平时，单个PCM样本被分类为无声）。 }
onFlush会默认配置成noisy状态，然后在实际处理状态时变更状态
@Override protected void onFlush() { if (enabled) { bytesPerFrame = inputAudioFormat.bytesPerFrame; int maybeSilenceBufferSize = durationUsToFrames(minimumSilenceDurationUs) * bytesPerFrame; if (maybeSilenceBuffer.length != maybeSilenceBufferSize) { maybeSilenceBuffer = new byte[maybeSilenceBufferSize]; } paddingSize = durationUsToFrames(paddingSilenceUs) * bytesPerFrame; if (paddingBuffer.length != paddingSize) { paddingBuffer = new byte[paddingSize]; } } state = STATE_NOISY; skippedFrames = 0; maybeSilenceBufferSize = 0; hasOutputNoise = false; }
输入处理
@Override public void queueInput(ByteBuffer inputBuffer) { while (inputBuffer.hasRemaining() && !hasPendingOutput()) { switch (state) { case STATE_NOISY: processNoisy(inputBuffer); break; case STATE_MAYBE_SILENT: processMaybeSilence(inputBuffer); break; case STATE_SILENT: processSilence(inputBuffer); break; default: throw new IllegalStateException(); } } }
处理非静音
// Internal methods. /** * Incrementally processes new input from {@code inputBuffer} while in {@link #STATE_NOISY}, * updating the state if needed. * 在 STATE_NOISY 中增量处理来自 inputBuffer 的新输入，如果需要更新状态。 */ private void processNoisy(ByteBuffer inputBuffer) { int limit = inputBuffer.limit(); // Check if there's any noise within the maybe silence buffer duration. inputBuffer.limit(min(limit, inputBuffer.position() + maybeSilenceBuffer.length)); int noiseLimit = findNoiseLimit(inputBuffer); if (noiseLimit == inputBuffer.position()) { // The buffer contains the start of possible silence. state = STATE_MAYBE_SILENT; } else { inputBuffer.limit(noiseLimit); output(inputBuffer); } // Restore the limit. inputBuffer.limit(limit); }
5）SonicAudioProcessor
/** * An {@link AudioProcessor} that uses the Sonic library to modify audio speed/pitch/sample rate. * 一个{@link AudioProcessor}，它使用Sonic库来修改音频速度/音高/采样率。 */ public final class SonicAudioProcessor implements AudioProcessor {}
一样的调用套路，不同的是 用了一个Sonic库
输入
@Override public void queueInput(ByteBuffer inputBuffer) { if (!inputBuffer.hasRemaining()) { return; } Sonic sonic = checkNotNull(this.sonic); ShortBuffer shortBuffer = inputBuffer.asShortBuffer(); int inputSize = inputBuffer.remaining(); inputBytes += inputSize; sonic.queueInput(shortBuffer); inputBuffer.position(inputBuffer.position() + inputSize); }
输出
@Override public ByteBuffer getOutput() { @Nullable Sonic sonic = this.sonic; if (sonic != null) { int outputSize = sonic.getOutputSize(); if (outputSize > 0) { if (buffer.capacity() < outputSize) { buffer = ByteBuffer.allocateDirect(outputSize).order(ByteOrder.nativeOrder()); shortBuffer = buffer.asShortBuffer(); } else { buffer.clear(); shortBuffer.clear(); } sonic.getOutput(shortBuffer); outputBytes += outputSize; buffer.limit(outputSize); outputBuffer = buffer; } } ByteBuffer outputBuffer = this.outputBuffer; this.outputBuffer = EMPTY_BUFFER; return outputBuffer; }
6）用来调试的TeeAudioProcessor
这个demo里没调用，用在了test里。。。。
/** * Audio processor that outputs its input unmodified and also outputs its input to a given sink. * This is intended to be used for diagnostics and debugging. * * <p>This audio processor can be inserted into the audio processor chain to access audio data * before/after particular processing steps have been applied. For example, to get audio output * after playback speed adjustment and silence skipping have been applied it is necessary to pass a * custom {@link com.google.android.exoplayer2.audio.DefaultAudioSink.AudioProcessorChain} when * creating the audio sink, and include this audio processor after all other audio processors. * 音频处理器，其输出保持不变，并且将其输入输出到给定的接收器。 这旨在用于诊断和调试。 * 该音频处理器可以插入音频处理器链中，以在应用特定处理步骤之前/之后访问音频数据。 * 例如，要在应用了播放速度调整和静音跳过之后才能获得音频输出，则在创建音频接收器时必须传递自定义的DefaultAudioSink.AudioProcessorChain，并将此音频处理器包括在所有其他音频处理器之后。 */ public final class TeeAudioProcessor extends BaseAudioProcessor { /** A sink for audio buffers handled by the audio processor. */ public interface AudioBufferSink {}
/** * A sink for audio buffers that writes output audio as .wav files with a given path prefix. When * new audio data is handled after flushing the audio processor, a counter is incremented and its * value is appended to the output file name. * * <p>Note: if writing to external storage it's necessary to grant the {@code * WRITE_EXTERNAL_STORAGE} permission. * 音频缓冲区的接收器，用于将输出音频作为具有给定路径前缀的.wav文件写入。 在刷新音频处理器后处理新的音频数据时，计数器会增加，并将其值附加到输出文件名。 * 注意：如果要写入外部存储，则必须授予WRITE_EXTERNAL_STORAGE权限。 */ public static final class WavFileAudioBufferSink implements AudioBufferSink {}
}
直接在Chain里增加就可以调用
/** * Builds an {@link AudioSink} to which the audio renderers will output. * 构建一个{@link AudioSink}，音频渲染器将输出到该音频。 * * @param context The {@link Context} associated with the player. * @param enableFloatOutput Whether to enable use of floating point audio output, if available. * @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link * android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported. * @param enableOffload Whether to enable use of audio offload for supported formats, if * available. * @return The {@link AudioSink} to which the audio renderers will output. May be {@code null} if * no audio renderers are required. If {@code null} is returned then {@link * #buildAudioRenderers} will not be called. */ @Nullable protected AudioSink buildAudioSink( Context context, boolean enableFloatOutput, boolean enableAudioTrackPlaybackParams, boolean enableOffload) { DefaultAudioSink.AudioProcessorChain chain = new DefaultAudioProcessorChain( new TeeAudioProcessor(new TeeAudioProcessor.WavFileAudioBufferSink("/mnt/sdcard/SENRSL/dc")) ); AudioSink audioSink = new DefaultAudioSink( AudioCapabilities.getCapabilities(context), chain, enableFloatOutput, enableAudioTrackPlaybackParams, enableOffload); return audioSink; }
捕获的文件也可以抓取到多声道
概览
完整名称 : /Users/senrsl/Downloads/dc-06021912-0000.wav
格式 : Wave
文件大小 : 25.4 MiB
时长 : 46 秒 272 毫秒
总体码率模式 : 恒定码率 (CBR)
总体码率 : 4 608 kb/s
音频
格式 : PCM
格式设置 : Little / Signed
编解码器 ID : 1
时长 : 46 秒 272 毫秒
码率模式 : 恒定码率 (CBR)
码率 : 4 608 kb/s
声道数 : 6 声道
采样率 : 48.0 kHz
位深 : 16 位
流大小 : 25.4 MiB (100%)
适当调整，也就能捕获音频输出了，可以替代AudioPort之类的 隐藏API来用了。。。。
回家做饭，明天再发。。。。
2021年06月02日19:22:35
senRsl
2021年05月26日14:08:17
没有评论 :
发表评论