继承关系
AudioProcessor.png
接口定义
/**
* Interface for audio processors, which take audio data as input and transform it, potentially
* modifying its channel count, encoding and/or sample rate.
*
* <p>In addition to being able to modify the format of audio, implementations may allow parameters
* to be set that affect the output audio and whether the processor is active/inactive.
* 音频处理器的接口，它将音频数据作为输入并进行转换，从而有可能修改其通道数，编码和/或采样率。
* <p>除了能够修改音频的格式外，实现还可以设置影响输出音频以及处理器是否处于活动状态的参数。
*/
public interface AudioProcessor {
/** PCM audio format that may be handled by an audio processor.音频处理器可以处理的PCM音频格式。 */
final class AudioFormat {
public static final AudioFormat NOT_SET = //没有设置，就都是-1
new AudioFormat(
/* sampleRate= */ Format.NO_VALUE,
/* channelCount= */ Format.NO_VALUE,
/* encoding= */ Format.NO_VALUE);
/** The sample rate in Hertz赫兹采样率. */
public final int sampleRate;
/** The number of interleaved channels. */
public final int channelCount;
/** The type of linear PCM encoding. */
@C.PcmEncoding public final int encoding;
/** The number of bytes used to represent one audio frame用来表示一个音频帧的字节数. */
public final int bytesPerFrame;
public AudioFormat(int sampleRate, int channelCount, @C.PcmEncoding int encoding) {
this.sampleRate = sampleRate;
this.channelCount = channelCount;
this.encoding = encoding;
bytesPerFrame =
Util.isEncodingLinearPcm(encoding)
? Util.getPcmFrameSize(encoding, channelCount)
: Format.NO_VALUE;
Logger.w("AudioProcessor",sampleRate,channelCount,encoding,bytesPerFrame);
}
@Override
public String toString() {
return "AudioProcessor 音频处理 AudioFormat["
+ "sampleRate="
+ sampleRate
+ ", channelCount="
+ channelCount
+ ", encoding="
+ encoding
+ ']';
}
}
/** Exception thrown when a processor can't be configured for a given input audio format
* 无法为给定的输入音频格式配置处理器时抛出异常. */
final class UnhandledAudioFormatException extends Exception {
public UnhandledAudioFormatException(AudioFormat inputAudioFormat) {
super("Unhandled format: " + inputAudioFormat);
}
}
/** An empty, direct {@link ByteBuffer}. */
ByteBuffer EMPTY_BUFFER = ByteBuffer.allocateDirect(0).order(ByteOrder.nativeOrder());
/**
* Configures the processor to process input audio with the specified format. After calling this
* method, call {@link #isActive()} to determine whether the audio processor is active. Returns
* the configured output audio format if this instance is active.
*
* <p>After calling this method, it is necessary to {@link #flush()} the processor to apply the
* new configuration. Before applying the new configuration, it is safe to queue input and get
* output in the old input/output formats. Call {@link #queueEndOfStream()} when no more input
* will be supplied in the old input format.
* 将处理器配置为处理指定格式的输入音频。 调用此方法后，调用isActive（）以确定音频处理器是否处于活动状态。 如果此实例处于活动状态，则返回配置的输出音频格式。
* 调用此方法后，有必要flush（）处理器以应用新配置。 在应用新配置之前，可以安全地将输入排队并以旧的输入/输出格式获取输出。 当不再以旧的输入格式提供输入时，请调用queueEndOfStream（）。
*
* @param inputAudioFormat The format of audio that will be queued after the next call to {@link
* #flush()}.
* @return The configured output audio format if this instance is {@link #isActive() active}.
* @throws UnhandledAudioFormatException Thrown if the specified format can't be handled as input.
*/
AudioFormat configure(AudioFormat inputAudioFormat) throws UnhandledAudioFormatException;
/** Returns whether the processor is configured and will process input buffers返回是否配置了处理器并将处理输入缓冲区. */
boolean isActive();
/**
* Queues audio data between the position and limit of the input {@code buffer} for processing.
* {@code buffer} must be a direct byte buffer with native byte order. Its contents are treated as
* read-only. Its position will be advanced by the number of bytes consumed (which may be zero).
* The caller retains ownership of the provided buffer. Calling this method invalidates any
* previous buffer returned by {@link #getOutput()}.
* 在输入缓冲区的位置和限制之间排队音频数据以进行处理。
* 缓冲区必须是具有本地字节顺序的直接字节缓冲区。 其内容被视为只读。 它的位置将增加所消耗的字节数（可能为零）。
* 调用方保留提供的缓冲区的所有权。 调用此方法会使getOutput（）返回的所有先前缓冲区无效。
*
* @param buffer The input buffer to process.
*/
void queueInput(ByteBuffer buffer);
/**
* Queues an end of stream signal. After this method has been called,
* {@link #queueInput(ByteBuffer)} may not be called until after the next call to
* {@link #flush()}. Calling {@link #getOutput()} will return any remaining output data. Multiple
* calls may be required to read all of the remaining output data. {@link #isEnded()} will return
* {@code true} once all remaining output data has been read.
* 将流信号的末尾排队。
* 调用此方法后，直到下一次调用flush（）之后，才能调用queueInput（ByteBuffer）。
* 调用getOutput（）将返回所有剩余的输出数据。 可能需要多次调用才能读取所有剩余的输出数据。 一旦读取了所有剩余的输出数据，isEnded（）将返回true。
*/
void queueEndOfStream();
/**
* Returns a buffer containing processed output data between its position and limit. The buffer
* will always be a direct byte buffer with native byte order. Calling this method invalidates any
* previously returned buffer. The buffer will be empty if no output is available.
* 返回一个缓冲区，该缓冲区包含在其位置和限制之间的已处理输出数据。 缓冲区将始终是具有本地字节顺序的直接字节缓冲区。
* 调用此方法会使以前返回的所有缓冲区无效。 如果没有可用的输出，缓冲区将为空。
*
* 返回值：
* 包含在其位置和极限之间的已处理输出数据的缓冲区。
*
* @return A buffer containing processed output data between its position and limit.
*/
ByteBuffer getOutput();
/**
* Returns whether this processor will return no more output from {@link #getOutput()} until it
* has been {@link #flush()}ed and more input has been queued.
*/
boolean isEnded();
/**
* Clears any buffered data and pending output. If the audio processor is active, also prepares
* the audio processor to receive a new stream of input in the last configured (pending) format.
* 清除所有缓冲的数据和挂起的输出。 如果音频处理器处于活动状态，则还要准备音频处理器以接收最后配置的（待定）格式的新输入流。
*/
void flush();
/** Resets the processor to its unconfigured state, releasing any resources. */
void reset();
}
在demo中，实际使用在DefaultAudioSink
入口在DefaultRenderFactory
/**
* Builds an {@link AudioSink} to which the audio renderers will output.
* 构建一个{@link AudioSink}，音频渲染器将输出到该音频。
*
* @param context The {@link Context} associated with the player.
* @param enableFloatOutput Whether to enable use of floating point audio output, if available.
* @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link
* android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported.
* @param enableOffload Whether to enable use of audio offload for supported formats, if
* available.
* @return The {@link AudioSink} to which the audio renderers will output. May be {@code null} if
* no audio renderers are required. If {@code null} is returned then {@link
* #buildAudioRenderers} will not be called.
*/
@Nullable
protected AudioSink buildAudioSink(
Context context,
boolean enableFloatOutput,
boolean enableAudioTrackPlaybackParams,
boolean enableOffload) {
return new DefaultAudioSink(
AudioCapabilities.getCapabilities(context),
new DefaultAudioProcessorChain(),
enableFloatOutput,
enableAudioTrackPlaybackParams,
enableOffload);
}
对于AudioProcess的调用，就是循环各个processor。。。。
/**
* Creates a new default audio sink, optionally using float output for high resolution PCM and
* with the specified {@code audioProcessorChain}.
* 创建一个新的默认音频接收器，还可以选择将浮点输出用于高分辨率PCM，并使用指定的{@code audioProcessorChain}。
*
* @param audioCapabilities The audio capabilities for playback on this device在此设备上播放的音频功能. May be null if the
* default capabilities (no encoded audio passthrough support) should be assumed.
* @param audioProcessorChain An {@link AudioProcessorChain} which is used to apply playback
* parameters adjustments. The instance passed in must not be reused in other sinks.
* @param enableFloatOutput Whether to enable 32-bit float output. Where possible, 32-bit float
* output will be used if the input is 32-bit float, and also if the input is high resolution
* (24-bit or 32-bit) integer PCM. Float output is supported from API level 21. Audio
* processing (for example, speed adjustment) will not be available when float output is in
* use.
* @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link
* android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported.
* @param enableOffload Whether to enable audio offload. If an audio format can be both played
* with offload and encoded audio passthrough, it will be played in offload. Audio offload is
* supported from API level 29. Most Android devices can only support one offload {@link
* android.media.AudioTrack} at a time and can invalidate it at any time. Thus an app can
* never be guaranteed that it will be able to play in offload. Audio processing (for example,
* speed adjustment) will not be available when offload is in use.
*/
public DefaultAudioSink(
@Nullable AudioCapabilities audioCapabilities,
AudioProcessorChain audioProcessorChain,
boolean enableFloatOutput,
boolean enableAudioTrackPlaybackParams,
boolean enableOffload) {
this.audioCapabilities = audioCapabilities;
this.audioProcessorChain = Assertions.checkNotNull(audioProcessorChain);
this.enableFloatOutput = Util.SDK_INT >= 21 && enableFloatOutput;
this.enableAudioTrackPlaybackParams = Util.SDK_INT >= 23 && enableAudioTrackPlaybackParams;
this.enableOffload = Util.SDK_INT >= 29 && enableOffload;
releasingConditionVariable = new ConditionVariable(true);
audioTrackPositionTracker = new AudioTrackPositionTracker(new PositionTrackerListener());
channelMappingAudioProcessor = new ChannelMappingAudioProcessor();
trimmingAudioProcessor = new TrimmingAudioProcessor();
ArrayList<AudioProcessor> toIntPcmAudioProcessors = new ArrayList<>();
Collections.addAll(
toIntPcmAudioProcessors,
new ResamplingAudioProcessor(),
channelMappingAudioProcessor,
trimmingAudioProcessor);
Collections.addAll(toIntPcmAudioProcessors, audioProcessorChain.getAudioProcessors());
toIntPcmAvailableAudioProcessors = toIntPcmAudioProcessors.toArray(new AudioProcessor[0]);
toFloatPcmAvailableAudioProcessors = new AudioProcessor[] {new FloatResamplingAudioProcessor()};
volume = 1f;
audioAttributes = AudioAttributes.DEFAULT;
audioSessionId = C.AUDIO_SESSION_ID_UNSET;
auxEffectInfo = new AuxEffectInfo(AuxEffectInfo.NO_AUX_EFFECT_ID, 0f);
mediaPositionParameters =
new MediaPositionParameters(
PlaybackParameters.DEFAULT,
DEFAULT_SKIP_SILENCE,
/* mediaTimeUs= */ 0,
/* audioTrackPositionUs= */ 0);
audioTrackPlaybackParameters = PlaybackParameters.DEFAULT;
drainingAudioProcessorIndex = C.INDEX_UNSET;
activeAudioProcessors = new AudioProcessor[0];
outputBuffers = new ByteBuffer[0];
mediaPositionParametersCheckpoints = new ArrayDeque<>();
initializationExceptionPendingExceptionHolder =
new PendingExceptionHolder<>(AUDIO_TRACK_RETRY_DURATION_MS);
writeExceptionPendingExceptionHolder =
new PendingExceptionHolder<>(AUDIO_TRACK_RETRY_DURATION_MS);
}
搞了两个AudioProcessor数组，一个是toIntPcm Avaliable
AudioProcessor，一个是toFloatPcm Avaliable AudioProcess。
分别对应解码后的输出类型，因为好多设备不支持浮点输出
然后去config
@Override
public void configure(Format inputFormat, int specifiedBufferSize, @Nullable int[] outputChannels)
throws ConfigurationException {
int inputPcmFrameSize;
@Nullable AudioProcessor[] availableAudioProcessors;
。。。
Logger.w(TAG,"configure方法",inputFormat.toString(),specifiedBufferSize,outputChannels);//Format(2, null, null, audio/ac3, null, -1, en, [-1, -1, -1.0], [6, 48000]),0,null
if (MimeTypes.AUDIO_RAW.equals(inputFormat.sampleMimeType)) {
Assertions.checkArgument(Util.isEncodingLinearPcm(inputFormat.pcmEncoding));
inputPcmFrameSize = Util.getPcmFrameSize(inputFormat.pcmEncoding, inputFormat.channelCount);
availableAudioProcessors =
shouldUseFloatOutput(inputFormat.pcmEncoding)
? toFloatPcmAvailableAudioProcessors
: toIntPcmAvailableAudioProcessors;
。。。
AudioProcessor.AudioFormat outputFormat =
new AudioProcessor.AudioFormat(
inputFormat.sampleRate, inputFormat.channelCount, inputFormat.pcmEncoding);
for (AudioProcessor audioProcessor : availableAudioProcessors) {
try {
AudioProcessor.AudioFormat nextFormat = audioProcessor.configure(outputFormat);
if (audioProcessor.isActive()) {
outputFormat = nextFormat;
Logger.w(TAG,"configure",outputFormat);
}
} catch (UnhandledAudioFormatException e) {
throw new ConfigurationException(e, inputFormat);
}
}
outputMode = OUTPUT_MODE_PCM;
outputEncoding = outputFormat.encoding;
outputSampleRate = outputFormat.sampleRate;
outputChannelConfig = Util.getAudioTrackChannelConfig(outputFormat.channelCount);
outputPcmFrameSize = Util.getPcmFrameSize(outputEncoding, outputFormat.channelCount);
} else {
inputPcmFrameSize = C.LENGTH_UNSET;
availableAudioProcessors = new AudioProcessor[0];
outputSampleRate = inputFormat.sampleRate;
outputPcmFrameSize = C.LENGTH_UNSET;
Logger.w(TAG,"configure方法x2",enableOffload,isOffloadedPlaybackSupported(inputFormat, audioAttributes));//false,false
if (enableOffload && isOffloadedPlaybackSupported(inputFormat, audioAttributes)) {
outputMode = OUTPUT_MODE_OFFLOAD;
outputEncoding =
MimeTypes.getEncoding(
Assertions.checkNotNull(inputFormat.sampleMimeType), inputFormat.codecs);
outputChannelConfig = Util.getAudioTrackChannelConfig(inputFormat.channelCount);
} else {
outputMode = OUTPUT_MODE_PASSTHROUGH;//直通输出
@Nullable
Pair<Integer, Integer> encodingAndChannelConfig =
getEncodingAndChannelConfigForPassthrough(inputFormat, audioCapabilities);
Logger.w("pass though x2",encodingAndChannelConfig);//Pair{5 252}
if (encodingAndChannelConfig == null) {
throw new ConfigurationException(
"Unable to configure passthrough for: " + inputFormat, inputFormat);
}
outputEncoding = encodingAndChannelConfig.first;//5
outputChannelConfig = encodingAndChannelConfig.second;//252
}
}
。。。
}
也就是说，需要满足inputFormat.sampleMImeType是 /raw才会去配置 AudioProcessor
也就是会走解码输出
否则会自动的去走音频分载或者passthrough
然后实际的干活方法里
AudioSink接口
/**
* Attempts to process data from a {@link ByteBuffer}, starting from its current position and
* ending at its limit (exclusive). The position of the {@link ByteBuffer} is advanced by the
* number of bytes that were handled. {@link Listener#onPositionDiscontinuity()} will be called if
* {@code presentationTimeUs} is discontinuous with the last buffer handled since the last reset.
* 尝试处理ByteBuffer的数据，从其当前位置开始，直到其限制（不包括限制）。
* ByteBuffer的位置提前处理的字节数。 如果presentationTimeUs与自上次重置以来处理的最后一个缓冲区不连续，则将调用AudioSink.Listener.onPositionDiscontinuity（）。
* 返回数据是否已全部处理。 如果未对数据进行完整处理，则必须将相同的ByteBuffer提供给后续调用，直到完全消耗完为止，
* 除非是对flush（）（或configure（Format，int，int []）的中间调用） 导致水槽被冲洗）。
*
* <p>Returns whether the data was handled in full. If the data was not handled in full then the
* same {@link ByteBuffer} must be provided to subsequent calls until it has been fully consumed,
* except in the case of an intervening call to {@link #flush()} (or to {@link #configure(Format,
* int, int[])} that causes the sink to be flushed).
*
* @param buffer The buffer containing audio data.
* @param presentationTimeUs The presentation timestamp of the buffer in microseconds.
* @param encodedAccessUnitCount The number of encoded access units in the buffer, or 1 if the
* buffer contains PCM audio. This allows batching multiple encoded access units in one
* buffer.
* @return Whether the buffer was handled fully.
* @throws InitializationException If an error occurs initializing the sink.
* @throws WriteException If an error occurs writing the audio data.
*/
boolean handleBuffer(ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount)
throws InitializationException, WriteException;
默认实现
@Override
@SuppressWarnings("ReferenceEquality")
public boolean handleBuffer(
ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount)
throws InitializationException, WriteException {
Assertions.checkArgument(inputBuffer == null || buffer == inputBuffer);
if (pendingConfiguration != null) {
。。。。
// Re-apply playback parameters.
applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs);
}
if (!isAudioTrackInitialized()) {
try {
initializeAudioTrack(); //初始化 audioTrack
} catch (InitializationException e) {
。。。。
return false;
}
}
initializationExceptionPendingExceptionHolder.clear();
if (startMediaTimeUsNeedsInit) {
。。。。
applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs);
if (playing) {
play();//audioTrack开始播放
}
}
if (!audioTrackPositionTracker.mayHandleBuffer(getWrittenFrames())) {
return false;
}
if (inputBuffer == null) {
// We are seeing this buffer for the first time.
Assertions.checkArgument(buffer.order() == ByteOrder.LITTLE_ENDIAN);
if (!buffer.hasRemaining()) {
// The buffer is empty.
return true;
}
。。。。
if (afterDrainParameters != null) {
if (!drainToEndOfStream()) {
// Don't process any more input until draining completes.
return false;
}
applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs);
afterDrainParameters = null;
}
// Check that presentationTimeUs is consistent with the expected value.
long expectedPresentationTimeUs =
startMediaTimeUs
+ configuration.inputFramesToDurationUs(
getSubmittedFrames() - trimmingAudioProcessor.getTrimmedFrameCount());
if (!startMediaTimeUsNeedsSync
&& Math.abs(expectedPresentationTimeUs - presentationTimeUs) > 200000) {
Log.e(
TAG,
"Discontinuity detected [expected "
+ expectedPresentationTimeUs
+ ", got "
+ presentationTimeUs
+ "]");
startMediaTimeUsNeedsSync = true;
}
if (startMediaTimeUsNeedsSync) {
if (!drainToEndOfStream()) {
// Don't update timing until pending AudioProcessor buffers are completely drained.
return false;
}
// Adjust startMediaTimeUs to be consistent with the current buffer's start time and the
// number of bytes submitted.
long adjustmentUs = presentationTimeUs - expectedPresentationTimeUs;
startMediaTimeUs += adjustmentUs;
startMediaTimeUsNeedsSync = false;
// Re-apply playback parameters because the startMediaTimeUs changed.
applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs);
if (listener != null && adjustmentUs != 0) {
listener.onPositionDiscontinuity();
}
}
if (configuration.outputMode == OUTPUT_MODE_PCM) {
submittedPcmBytes += buffer.remaining();
} else {
submittedEncodedFrames += framesPerEncodedSample * encodedAccessUnitCount;
}
inputBuffer = buffer;
inputBufferAccessUnitCount = encodedAccessUnitCount;
}
processBuffers(presentationTimeUs);//调用02篇的build时创建的音频处理器处理
if (!inputBuffer.hasRemaining()) {
inputBuffer = null;
inputBufferAccessUnitCount = 0;
return true;
}
if (audioTrackPositionTracker.isStalled(getWrittenFrames())) {
Log.w(TAG, "Resetting stalled audio track");
flush();
return true;
}
return false;
}
两处对audio processor调用，一个是设置audioProcessor，设置完成后用来处理
设置
private void setupAudioProcessors() {
AudioProcessor[] audioProcessors = configuration.availableAudioProcessors;
ArrayList<AudioProcessor> newAudioProcessors = new ArrayList<>();
for (AudioProcessor audioProcessor : audioProcessors) {
if (audioProcessor.isActive()) {
newAudioProcessors.add(audioProcessor);
} else {
audioProcessor.flush();
}
}
int count = newAudioProcessors.size();
activeAudioProcessors = newAudioProcessors.toArray(new AudioProcessor[count]);
outputBuffers = new ByteBuffer[count];
flushAudioProcessors();
}
private void flushAudioProcessors() {
for (int i = 0; i < activeAudioProcessors.length; i++) {
AudioProcessor audioProcessor = activeAudioProcessors[i];
audioProcessor.flush();
outputBuffers[i] = audioProcessor.getOutput();
}
}
用来处理
private void processBuffers(long avSyncPresentationTimeUs) throws WriteException {
int count = activeAudioProcessors.length;
int index = count;
Logger.w(TAG,"processBuffers",avSyncPresentationTimeUs,count);//0,0 | 32000,0 | 64000，0|96000，0
while (index >= 0) {
ByteBuffer input = index > 0 ? outputBuffers[index - 1] : (inputBuffer != null ? inputBuffer : AudioProcessor.EMPTY_BUFFER);
if (index == count) {
writeBuffer(input, avSyncPresentationTimeUs);
} else {
AudioProcessor audioProcessor = activeAudioProcessors[index];
if (index > drainingAudioProcessorIndex) {
audioProcessor.queueInput(input); //输入
}
ByteBuffer output = audioProcessor.getOutput();//输出
outputBuffers[index] = output;
if (output.hasRemaining()) {
// Handle the output as input to the next audio processor or the AudioTrack.
index++;
continue;
}
}
if (input.hasRemaining()) {
// The input wasn't consumed and no output was produced, so give up for now.
return;
}
// Get more input from upstream.
index--;
}
}
这逻辑写的。。。。
index == count就输出音频到audioTrack,index--
音频processor处理完成后就index++
也就是说，DefaultAudioSink对AudioProcessor接口的调用是这样的
狄仁杰上线
狄仁杰：我想事情的真相应该是这样的.jpg
new
DefaultAudioSink()的时候，增加了增加了几个AudioProcessor的实现，也就是增加了几个音频处理器的不同实现到toIntPcmAudioProcessors数组；
然后在DefaultAudioSink.configure()的时候，判断如果是解码输出，就先判断是不是浮点输出，如果是toIntPcm而不是toFloatPcm，就把avaliableAudioProcessors指定为toIntPcmAudioProcessor数组，然后循环avaliableProcessor数组，挨个调用AudioProcessor实现类的configure()方法，返回AudioProcess.AudioFormat，判断AudioProcessor.isActive()，如果is
active，就会使用返回的AudioFormat去new Configuration()，这个后面的大配置。
接下来，就到了处理数据流的时候，首先设置
setupAudioProcess()，依然是循环那几个实现，取得就是大配置configuration.avaliableAudioProcessor数组，挨个判断是否启用isActive，如果启用就增加到
activeAudioProcessor数组，干完后循环
调用activeAudioProcessor.flush()，flush后配置全类私有数组outputBuffers[i] =
activieAudioProcess[i].getOutputj(); 也就是保留了对应的输出通道。
接下来，终于要实际处理流了，依然是要循环子处理器实现通过上面的outBuffer[index]取对应子处理器的返回,然后调用audioProcessor.queueInput()，audioProcess.getOutput()，写入到audioTrack....
对于AudioProcessor的调用顺序，简单理解为 configure() -> isActive() ->
flush() -> queueInput() -> getOutput()。
回到configuration方法
Configuration pendingConfiguration =
new Configuration(
inputFormat,
inputPcmFrameSize,
outputMode,
outputPcmFrameSize,
outputSampleRate,
outputChannelConfig,
outputEncoding,
specifiedBufferSize,
enableAudioTrackPlaybackParams,
availableAudioProcessors);
解码输出时，avaliableAudioProcessors为
toIntPcmAvaliableAudioProcessors或toIntPcmXxxxxx
不解码输出时，为empty
有了 avaliableAudioProcessors才有了后面的 audioProcessor.process()
所以如果要直通输出，可以在这里增加audio processor....
解码输出，构造函数初始化时就配置了四个默认的处理
channelMappingAudioProcessor = new ChannelMappingAudioProcessor();
trimmingAudioProcessor = new TrimmingAudioProcessor();
ArrayList<AudioProcessor> toIntPcmAudioProcessors = new ArrayList<>();
Collections.addAll(
toIntPcmAudioProcessors,
new ResamplingAudioProcessor(),
channelMappingAudioProcessor,
trimmingAudioProcessor);
Collections.addAll(toIntPcmAudioProcessors, audioProcessorChain.getAudioProcessors());
toIntPcmAvailableAudioProcessors = toIntPcmAudioProcessors.toArray(new AudioProcessor[0]);
toFloatPcmAvailableAudioProcessors = new AudioProcessor[] {new FloatResamplingAudioProcessor()};
分别为
ResamplingAudioProcessor 重采样
ChannelMappingAudioProcessor 声道
TrimmingAudioProcessor 修剪？
还有传过来的 audioProcessorChain.getAudioProcessors()
/**
* Creates a new default chain of audio processors, with the user-defined {@code
* audioProcessors} applied before silence skipping and speed adjustment processors.
*/
public DefaultAudioProcessorChain(AudioProcessor... audioProcessors) {
this(audioProcessors, new SilenceSkippingAudioProcessor(), new SonicAudioProcessor());
}
按照添加顺序，分别为
SilenceSkippingAudioProcessor 静音跳过
SonicAudioProcessor uses the Sonic library to modify audio
speed/pitch/sample rate
demo中对解码输出 总共应用了这五个音频处理器。
1）ResamplingAudioProcessor
/**
* An {@link AudioProcessor} that converts different PCM audio encodings to 16-bit integer PCM. The
* following encodings are supported as input:
* 将不同的PCM音频编码转换为16位整数PCM的AudioProcessor。 支持以下编码作为输入：
*
* <ul>
* <li>{@link C#ENCODING_PCM_8BIT}
* <li>{@link C#ENCODING_PCM_16BIT} ({@link #isActive()} will return {@code false})
* <li>{@link C#ENCODING_PCM_16BIT_BIG_ENDIAN}
* <li>{@link C#ENCODING_PCM_24BIT}
* <li>{@link C#ENCODING_PCM_32BIT}
* <li>{@link C#ENCODING_PCM_FLOAT}
* </ul>
*/
/* package */ final class ResamplingAudioProcessor extends BaseAudioProcessor {}
实现也是简单粗暴，只重写了configure() 和 queueInput()方法
@Override
public AudioFormat onConfigure(AudioFormat inputAudioFormat)
throws UnhandledAudioFormatException {
@C.PcmEncoding int encoding = inputAudioFormat.encoding;
if (encoding != C.ENCODING_PCM_8BIT
&& encoding != C.ENCODING_PCM_16BIT
&& encoding != C.ENCODING_PCM_16BIT_BIG_ENDIAN
&& encoding != C.ENCODING_PCM_24BIT
&& encoding != C.ENCODING_PCM_32BIT
&& encoding != C.ENCODING_PCM_FLOAT) {
throw new UnhandledAudioFormatException(inputAudioFormat);
}
return encoding != C.ENCODING_PCM_16BIT
? new AudioFormat(
inputAudioFormat.sampleRate, inputAudioFormat.channelCount, C.ENCODING_PCM_16BIT)
: AudioFormat.NOT_SET;
}
onConfigure时，不是支持的格式直接抛异常，不是16bit 返回16bit....
onConfigure定义与BaseAudioProcessor抽象类，是AudioProcessor接口中configure方法的延伸
BaseAudioProcessor抽象类：
@Override
public final AudioFormat configure(AudioFormat inputAudioFormat)
throws UnhandledAudioFormatException {
pendingInputAudioFormat = inputAudioFormat;
pendingOutputAudioFormat = onConfigure(inputAudioFormat);
return isActive() ? pendingOutputAudioFormat : AudioFormat.NOT_SET;
}
/** Called when the processor is configured for a new input format. */
protected AudioFormat onConfigure(AudioFormat inputAudioFormat)
throws UnhandledAudioFormatException {
return AudioFormat.NOT_SET;
}
数据输入时进行重采样
@Override
public void queueInput(ByteBuffer inputBuffer) {
// Prepare the output buffer.
int position = inputBuffer.position();
int limit = inputBuffer.limit();
int size = limit - position;
int resampledSize;
switch (inputAudioFormat.encoding) {
case C.ENCODING_PCM_8BIT:
resampledSize = size * 2;
break;
case C.ENCODING_PCM_16BIT_BIG_ENDIAN:
resampledSize = size;
break;
。。。
}
// Resample the little endian input and update the input/output buffers.
ByteBuffer buffer = replaceOutputBuffer(resampledSize);
switch (inputAudioFormat.encoding) {
case C.ENCODING_PCM_8BIT:
// 8 -> 16 bit resampling. Shift each byte from [0, 256) to [-128, 128) and scale up.
for (int i = position; i < limit; i++) {
buffer.put((byte) 0);
buffer.put((byte) ((inputBuffer.get(i) & 0xFF) - 128));
}
break;
case C.ENCODING_PCM_16BIT_BIG_ENDIAN:
// Big endian to little endian resampling. Swap the byte order.
for (int i = position; i < limit; i += 2) {
buffer.put(inputBuffer.get(i + 1));
buffer.put(inputBuffer.get(i));
}
break;
。。。
}
inputBuffer.position(inputBuffer.limit());
buffer.flip();
}
简单粗暴的转换
2）ChannelMappingAudioProcessor
/**
* An {@link AudioProcessor} that applies a mapping from input channels onto specified output
* channels. This can be used to reorder, duplicate or discard channels.
* 一个AudioProcessor，将输入通道的映射应用于指定的输出通道。 这可用于重新排序，复制或放弃频道。
*/
/* package */ final class ChannelMappingAudioProcessor extends BaseAudioProcessor {}
这个是重新映射输出通道，demo中把8声道改成了6声道，把双声道改成了六声道。。。。
configure()的时候，把之前的inputAudioFormat中的outputChannel变更到了设置的pendingOutChannel。
/**
* Resets the channel mapping. After calling this method, call {@link #configure(AudioFormat)} to
* start using the new channel map.
*
* @param outputChannels The mapping from input to output channel indices, or {@code null} to
* leave the input unchanged.
* @see AudioSink#configure(com.google.android.exoplayer2.Format, int, int[])
*/
public void setChannelMap(@Nullable int[] outputChannels) {
pendingOutputChannels = outputChannels;
}
@Override
public AudioFormat onConfigure(AudioFormat inputAudioFormat)
throws UnhandledAudioFormatException {
@Nullable int[] outputChannels = pendingOutputChannels;
。。。
return active
? new AudioFormat(inputAudioFormat.sampleRate, outputChannels.length, C.ENCODING_PCM_16BIT)
: AudioFormat.NOT_SET;
}
数据流输入的时候，对新的目标channel循环做了变更
@Override
public void queueInput(ByteBuffer inputBuffer) {
int[] outputChannels = Assertions.checkNotNull(this.outputChannels);
int position = inputBuffer.position();
int limit = inputBuffer.limit();
int frameCount = (limit - position) / inputAudioFormat.bytesPerFrame;
int outputSize = frameCount * outputAudioFormat.bytesPerFrame;
ByteBuffer buffer = replaceOutputBuffer(outputSize);
while (position < limit) {
for (int channelIndex : outputChannels) {
buffer.putShort(inputBuffer.getShort(position + 2 * channelIndex));
}
position += inputAudioFormat.bytesPerFrame;
}
inputBuffer.position(limit);
buffer.flip();
}
3）TrimmingAudioProcessor
/** Audio processor for trimming samples from the start/end of data音频处理器，用于从数据的开始/结束处修剪样本. */
/* package */ final class TrimmingAudioProcessor extends BaseAudioProcessor {}
设置起止帧
/**
* Sets the number of audio frames to trim from the start and end of audio passed to this
* processor. After calling this method, call {@link #configure(AudioFormat)} to apply the new
* trimming frame counts.
* 设置要从传递给此处理器的音频的开头和结尾修剪的音频帧数。 调用此方法后，调用 configure(AudioProcessor.AudioFormat) 以应用新的修剪帧计数。
*
* 参数：
* trimStartFrames – 从音频开始要修剪的音频帧数。
* trimEndFrames – 要从音频末尾修剪的音频帧数。
*
* @param trimStartFrames The number of audio frames to trim from the start of audio.
* @param trimEndFrames The number of audio frames to trim from the end of audio.
* @see AudioSink#configure(com.google.android.exoplayer2.Format, int, int[])
*/
public void setTrimFrameCount(int trimStartFrames, int trimEndFrames) {
this.trimStartFrames = trimStartFrames;
this.trimEndFrames = trimEndFrames;
}
在DefaultAudioSink.configure方法进行了设置
trimmingAudioProcessor.setTrimFrameCount(
inputFormat.encoderDelay, inputFormat.encoderPadding);
这俩变量的定义
/**
* The number of frames to trim from the start of the decoded audio stream, or 0 if not
* applicable.
*/
public final int encoderDelay;
/**
* The number of frames to trim from the end of the decoded audio stream, or 0 if not applicable.
*/
public final int encoderPadding;
配置
@Override
public AudioFormat onConfigure(AudioFormat inputAudioFormat)
throws UnhandledAudioFormatException {
if (inputAudioFormat.encoding != OUTPUT_ENCODING) {
throw new UnhandledAudioFormatException(inputAudioFormat);
}
reconfigurationPending = true;
return trimStartFrames != 0 || trimEndFrames != 0 ? inputAudioFormat : AudioFormat.NOT_SET;
}
在DefaultAudioSink.handleBuffer方法处理流前，减去了剪去帧的时间
// Check that presentationTimeUs is consistent with the expected value.
long expectedPresentationTimeUs =
startMediaTimeUs
+ configuration.inputFramesToDurationUs(
getSubmittedFrames() - trimmingAudioProcessor.getTrimmedFrameCount());
剪帧
@Override
public void queueInput(ByteBuffer inputBuffer) {
int position = inputBuffer.position();
int limit = inputBuffer.limit();
int remaining = limit - position;
if (remaining == 0) {
return;
}
// Trim any pending start bytes from the input buffer.
int trimBytes = min(remaining, pendingTrimStartBytes);//取小
trimmedFrameCount += trimBytes / inputAudioFormat.bytesPerFrame;
pendingTrimStartBytes -= trimBytes;
inputBuffer.position(position + trimBytes);
if (pendingTrimStartBytes > 0) {
// Nothing to output yet.
return;
}
remaining -= trimBytes;
// endBuffer must be kept as full as possible, so that we trim the right amount of media if we
// don't receive any more input. After taking into account the number of bytes needed to keep
// endBuffer as full as possible, the output should be any surplus bytes currently in endBuffer
// followed by any surplus bytes in the new inputBuffer.endBuffer 必须尽可能地保持满，这样如果我们不再接收到任何输入，
// 我们就可以修剪适量的媒体。 考虑到保持 endBuffer 尽可能满所需的字节数后，输出应该是当前 endBuffer 中的任何剩余字节，然后是新 inputBuffer 中的任何剩余字节。
int remainingBytesToOutput = endBufferSize + remaining - endBuffer.length;
ByteBuffer buffer = replaceOutputBuffer(remainingBytesToOutput);
// Output from endBuffer.
int endBufferBytesToOutput = Util.constrainValue(remainingBytesToOutput, 0, endBufferSize);
buffer.put(endBuffer, 0, endBufferBytesToOutput);
remainingBytesToOutput -= endBufferBytesToOutput;
// Output from inputBuffer, restoring its limit afterwards.
int inputBufferBytesToOutput = Util.constrainValue(remainingBytesToOutput, 0, remaining);
inputBuffer.limit(inputBuffer.position() + inputBufferBytesToOutput);
buffer.put(inputBuffer);
inputBuffer.limit(limit);
remaining -= inputBufferBytesToOutput;
// Compact endBuffer, then repopulate it using the new input.
endBufferSize -= endBufferBytesToOutput;
System.arraycopy(endBuffer, endBufferBytesToOutput, endBuffer, 0, endBufferSize);
inputBuffer.get(endBuffer, endBufferSize, remaining);
endBufferSize += remaining;
buffer.flip();
}
4）SilenceSkippingAudioProcessor
/**
* An {@link AudioProcessor} that skips silence in the input stream. Input and output are 16-bit
* PCM.
* {@link AudioProcessor}，可跳过输入流中的静音。 输入和输出是16位PCM。
*/
public final class SilenceSkippingAudioProcessor extends BaseAudioProcessor {}
定义了三个状态
private @interface State {}
/** State when the input is not silent 非静音时. */
private static final int STATE_NOISY = 0;
/** State when the input may be silent but we haven't read enough yet to know说明何时输入可能是无声的，但我们还没有阅读足够的内容. */
private static final int STATE_MAYBE_SILENT = 1;
/** State when the input is silent输入静音时的状态. */
private static final int STATE_SILENT = 2;
默认配置
/** Creates a new silence skipping audio processor.创建一个新的静音跳过音频处理器。 */
public SilenceSkippingAudioProcessor() {
this(
DEFAULT_MINIMUM_SILENCE_DURATION_US, //150s 音频的最小持续时间必须低于{@code silenceThresholdLevel}才能将音频的该部分分类为无声，以微秒为单位。
DEFAULT_PADDING_SILENCE_US, //20s 延长非沉默部分的沉默持续时间（以微秒为单位）。 该值不能超过{@code minimumSilenceDurationUs}。
DEFAULT_SILENCE_THRESHOLD_LEVEL); //1024 绝对水平（低于该绝对水平时，单个PCM样本被分类为无声）。
}
onFlush会默认配置成noisy状态，然后在实际处理状态时变更状态
@Override
protected void onFlush() {
if (enabled) {
bytesPerFrame = inputAudioFormat.bytesPerFrame;
int maybeSilenceBufferSize = durationUsToFrames(minimumSilenceDurationUs) * bytesPerFrame;
if (maybeSilenceBuffer.length != maybeSilenceBufferSize) {
maybeSilenceBuffer = new byte[maybeSilenceBufferSize];
}
paddingSize = durationUsToFrames(paddingSilenceUs) * bytesPerFrame;
if (paddingBuffer.length != paddingSize) {
paddingBuffer = new byte[paddingSize];
}
}
state = STATE_NOISY;
skippedFrames = 0;
maybeSilenceBufferSize = 0;
hasOutputNoise = false;
}
输入处理
@Override
public void queueInput(ByteBuffer inputBuffer) {
while (inputBuffer.hasRemaining() && !hasPendingOutput()) {
switch (state) {
case STATE_NOISY:
processNoisy(inputBuffer);
break;
case STATE_MAYBE_SILENT:
processMaybeSilence(inputBuffer);
break;
case STATE_SILENT:
processSilence(inputBuffer);
break;
default:
throw new IllegalStateException();
}
}
}
处理非静音
// Internal methods.
/**
* Incrementally processes new input from {@code inputBuffer} while in {@link #STATE_NOISY},
* updating the state if needed.
* 在 STATE_NOISY 中增量处理来自 inputBuffer 的新输入，如果需要更新状态。
*/
private void processNoisy(ByteBuffer inputBuffer) {
int limit = inputBuffer.limit();
// Check if there's any noise within the maybe silence buffer duration.
inputBuffer.limit(min(limit, inputBuffer.position() + maybeSilenceBuffer.length));
int noiseLimit = findNoiseLimit(inputBuffer);
if (noiseLimit == inputBuffer.position()) {
// The buffer contains the start of possible silence.
state = STATE_MAYBE_SILENT;
} else {
inputBuffer.limit(noiseLimit);
output(inputBuffer);
}
// Restore the limit.
inputBuffer.limit(limit);
}
5）SonicAudioProcessor
/**
* An {@link AudioProcessor} that uses the Sonic library to modify audio speed/pitch/sample rate.
* 一个{@link AudioProcessor}，它使用Sonic库来修改音频速度/音高/采样率。
*/
public final class SonicAudioProcessor implements AudioProcessor {}
一样的调用套路，不同的是 用了一个Sonic库
输入
@Override
public void queueInput(ByteBuffer inputBuffer) {
if (!inputBuffer.hasRemaining()) {
return;
}
Sonic sonic = checkNotNull(this.sonic);
ShortBuffer shortBuffer = inputBuffer.asShortBuffer();
int inputSize = inputBuffer.remaining();
inputBytes += inputSize;
sonic.queueInput(shortBuffer);
inputBuffer.position(inputBuffer.position() + inputSize);
}
输出
@Override
public ByteBuffer getOutput() {
@Nullable Sonic sonic = this.sonic;
if (sonic != null) {
int outputSize = sonic.getOutputSize();
if (outputSize > 0) {
if (buffer.capacity() < outputSize) {
buffer = ByteBuffer.allocateDirect(outputSize).order(ByteOrder.nativeOrder());
shortBuffer = buffer.asShortBuffer();
} else {
buffer.clear();
shortBuffer.clear();
}
sonic.getOutput(shortBuffer);
outputBytes += outputSize;
buffer.limit(outputSize);
outputBuffer = buffer;
}
}
ByteBuffer outputBuffer = this.outputBuffer;
this.outputBuffer = EMPTY_BUFFER;
return outputBuffer;
}
6）用来调试的TeeAudioProcessor
这个demo里没调用，用在了test里。。。。
/**
* Audio processor that outputs its input unmodified and also outputs its input to a given sink.
* This is intended to be used for diagnostics and debugging.
*
* <p>This audio processor can be inserted into the audio processor chain to access audio data
* before/after particular processing steps have been applied. For example, to get audio output
* after playback speed adjustment and silence skipping have been applied it is necessary to pass a
* custom {@link com.google.android.exoplayer2.audio.DefaultAudioSink.AudioProcessorChain} when
* creating the audio sink, and include this audio processor after all other audio processors.
* 音频处理器，其输出保持不变，并且将其输入输出到给定的接收器。 这旨在用于诊断和调试。
* 该音频处理器可以插入音频处理器链中，以在应用特定处理步骤之前/之后访问音频数据。
* 例如，要在应用了播放速度调整和静音跳过之后才能获得音频输出，则在创建音频接收器时必须传递自定义的DefaultAudioSink.AudioProcessorChain，并将此音频处理器包括在所有其他音频处理器之后。
*/
public final class TeeAudioProcessor extends BaseAudioProcessor {
/** A sink for audio buffers handled by the audio processor. */
public interface AudioBufferSink {}
/**
* A sink for audio buffers that writes output audio as .wav files with a given path prefix. When
* new audio data is handled after flushing the audio processor, a counter is incremented and its
* value is appended to the output file name.
*
* <p>Note: if writing to external storage it's necessary to grant the {@code
* WRITE_EXTERNAL_STORAGE} permission.
* 音频缓冲区的接收器，用于将输出音频作为具有给定路径前缀的.wav文件写入。 在刷新音频处理器后处理新的音频数据时，计数器会增加，并将其值附加到输出文件名。
* 注意：如果要写入外部存储，则必须授予WRITE_EXTERNAL_STORAGE权限。
*/
public static final class WavFileAudioBufferSink implements AudioBufferSink {}
}
直接在Chain里增加就可以调用
/**
* Builds an {@link AudioSink} to which the audio renderers will output.
* 构建一个{@link AudioSink}，音频渲染器将输出到该音频。
*
* @param context The {@link Context} associated with the player.
* @param enableFloatOutput Whether to enable use of floating point audio output, if available.
* @param enableAudioTrackPlaybackParams Whether to enable setting playback speed using {@link
* android.media.AudioTrack#setPlaybackParams(PlaybackParams)}, if supported.
* @param enableOffload Whether to enable use of audio offload for supported formats, if
* available.
* @return The {@link AudioSink} to which the audio renderers will output. May be {@code null} if
* no audio renderers are required. If {@code null} is returned then {@link
* #buildAudioRenderers} will not be called.
*/
@Nullable
protected AudioSink buildAudioSink(
Context context,
boolean enableFloatOutput,
boolean enableAudioTrackPlaybackParams,
boolean enableOffload) {
DefaultAudioSink.AudioProcessorChain chain = new DefaultAudioProcessorChain(
new TeeAudioProcessor(new TeeAudioProcessor.WavFileAudioBufferSink("/mnt/sdcard/SENRSL/dc"))
);
AudioSink audioSink = new DefaultAudioSink(
AudioCapabilities.getCapabilities(context),
chain,
enableFloatOutput,
enableAudioTrackPlaybackParams,
enableOffload);
return audioSink;
}
捕获的文件也可以抓取到多声道
概览
完整名称 :
/Users/senrsl/Downloads/dc-06021912-0000.wav
格式 : Wave
文件大小 : 25.4 MiB
时长 : 46 秒 272 毫秒
总体码率模式 : 恒定码率 (CBR)
总体码率 : 4 608 kb/s
音频
格式 : PCM
格式设置 : Little / Signed
编解码器 ID : 1
时长 : 46 秒 272 毫秒
码率模式 : 恒定码率 (CBR)
码率 : 4 608 kb/s
声道数 : 6 声道
采样率 : 48.0 kHz
位深 : 16 位
流大小 : 25.4 MiB (100%)
适当调整，也就能捕获音频输出了，可以替代AudioPort之类的 隐藏API来用了。。。。
回家做饭，明天再发。。。。
2021年06月02日19:22:35
--
senRsl
2021年05月26日14:08:17
