Chasing ghosts: Diagnosing crackling audio in modern vehicles

By: Ji Yang, Senior Software Developer, Audio Platform Services

When audio artifacts become more than annoyances

Imagine cruising down the highway when a sudden crackle interrupts your playlist. For most drivers, it's a nuisance. But beneath the surface, audio artifacts like crackling and dropouts signal deeper vehicle system-level issues, with implications for engineering teams, service centers, and brand reputation. At GM, we're chasing these "intermittent glitches," to tackle complex, system-level challenges that can impact customer satisfaction and operational efficiency.

Why these "ghost" bugs are so hard to catch

Modern vehicles function as distributed audio systems. Sound can originate from sources like a paired phone, a streaming app, or navigation prompts. These signals then pass through multiple Electronic Control Units (ECUs), digital buses, operating systems, hypervisors, schedulers, and amplifiers before reaching a driver's ears. Each transition - whether within a single compute unit or across an ECU boundary - can introduce glitches.

The hardest part is that these issues are often intermittent. They appear without warning and disappear just as quickly, sometimes after a simple ignition cycle, leaving behind no clear error code or crash log. "It sometimes crackles" is less a bug report than the start of a research effort, at least until the issue can be reproduced.

What readers will take away

This article presents a repeatable approach for turning a vague customer complaint into a scoped engineering problem. The process is simple: Reproduce the issue, map the system architecture, instrument key boundaries, identify the first bad boundary, validate the root cause, and enforce the contracts that help prevent recurrence. The goal is to give readers a practical framework they can apply to their own investigations, particularly when intermittent issues span multiple systems and resist conventional debugging. Real case studies show how this approach uncovered root causes that were not obvious at the outset.

How the methodology works

The methodology turns a complaint like "it sometimes crackles" into a verified fix by breaking the audio path into observable stages and testing each boundary in sequence. Figure 1 shows the investigation flow. Figure 2 maps the end-to-end audio path and its key boundaries. Read together, the diagrams show both how the team investigates the problem and where issues can emerge across the system.

Step 1 - Reproduce or stop

No boundary capture, trace, or waveform is meaningful until the failure can be reproduced on demand. The first task is to establish a stable reproduction scenario by repeating routes, conditions, and usage patterns until the complaint occurs reliably. Until the issue is controlled, every subsequent step is guesswork.

Step 2 - Map the architecture

Before placing a single probe, build a concrete model of the audio path, as shown in Figure 2. In a representative infotainment compute ECU (CCU), audio starts in application threads in the guest virtual machine (GVM), crosses into the privileged virtual machine (PVM), moves through the CCU CPU/DSP pipeline, and then exits over the digital audio bus to the amplifier ECU and speakers. The amplifier loopback path runs in the opposite direction, back toward the CCU.

The boundaries themselves don't process data. They are observation points where probes can be attached. The segment between adjacent boundaries is the potential fault domain. Step 3 explains how we instrument these points (and what each probe captures).

Figure 2 marks six probe points (P1 through P6) at the boundaries that matter most. This map does not assume where the problem is. It assumes only that the problem is somewhere on this path, and that the path can be observed at each of its seams.

Step 3 - Instrument the boundaries

Instrumentation follows the path in Figure 2 and uses two modes in parallel. Most probe points (P2-P6) capture the audio stream at boundaries, so adjacent segments can be compared. P1 is the exception: An in-segment probe used after the fault domain has been narrowed to capture stage- or thread-level behavior.

Physical boundary capture targets ECU-to-ECU interfaces, specifically P3 and P6. Bus-powered monitors record bus frames, PCM data, faults, and timestamps across sleep/wake cycles without a laptop in the vehicle.

Internal boundary capture targets seams inside the CCU, specifically P2, P4, and P5, where the goal is to capture the stream as it crosses a boundary such as a virtual machined (VM) crossing, CPU-to-DSP handoff, endpoint loopback. P1 complements these boundary captures by recording stage- or thread-level behavior inside a narrowed segment, including PCM data and timing markers, to explain why the segment failed after boundary probes established where the stream first went bad.

At every probe point, the same two questions are applied: Is the waveform still clean here? And do the event timestamps indicate anything abnormal?

Step 4 - Find the first bad boundary

With captures from P2 through P6, the comparison becomes systematic: walk the probes in order, identify the last clean signal, identify the first corrupted one. The suspect lies in the component, stage, or thread between those two points.

For example: If P2 is clean but P4 shows corruption, the Platform VM becomes the primary suspect domain. A deeper probe can then narrow it to the specific stage, thread, or component where the signal first goes bad. Rather than debugging the entire system at once, the team can focus on the narrow slice between the last clean probe and the first corrupted one and then go deeper with traces and automated log mining.

Step 5 - Validate and close

Once a suspect is characterized, boundary captures provide shared, objective evidence across teams.

P1 data is then used to explain how the failure occurs inside that narrowed segment. From there, the team runs controlled experiments against the suspected cause - adjusting factors such as scheduling priorities, policies, or CPU affinities for audio-critical threads - while collecting the same probe data and comparing it with the baseline.

When a change eliminates the artifact, we replay the original reproduction scenario with all probes active. If the waveform stays clean at P2-P6, the fix has end-to-end evidence behind it and can move forward with confidence.

Case studies

Case Study 1: When "broken speakers" weren't speakers at all

One of the most visible crackling issues manifested as loud, persistent noise affecting one or several speakers at once, across multiple sources. It appeared in many vehicles and quickly became a major audio warranty driver, reinforcing the perception that the speakers themselves were defective.

But replacing speakers almost never fixed it. The crackles might disappear after an ignition cycle, only to return later - shifting the problem from an apparent hardware failure to a broader system-level investigation.

Data breaks the myth

The first real breakthrough came when time-correlated data was captured alongside the symptom. Audio dumps showed that the stream crossing from the GVM into the PVM inside the CCU was clean - effectively ruling out the application layer and much of the infotainment stack. However, a loopback capture from the amplifier back toward the CCU already contained distinct crackles before the signal ever reached the speakers.

This was the first piece of data that clearly ruled out the speakers as the source. Whatever was causing corruption occurred somewhere between the CCU output and the amplifier return path.

A humbling lesson: When tools become the bug

When we first installed a digital bus monitor between the CCU and the amplifier, the results were inconsistent. Faults showed up in traces but didn't reliably match what we heard. The surprise was that crackling appeared only when the monitor laptop was powered through a specific in-vehicle adapter. Switching the adapter - or powering the laptop externally - made the "issue" vanish. Our test setup was injecting noise. We weren't just debugging the vehicle; we were debugging our tools.

We replaced the lab setup with a pocket-sized, bus-powered, field-ready monitor that woke up reliably with the vehicle's sleep/wake cycles and required no laptop. Installed in test vehicles for weeks, it delivered unambiguous data when the issue reappeared.

Once the issue reappeared and was captured with the bus-powered monitor, the data localized the fault: the CCU-side stream remained clean, while the amplifier loopback returning toward the CCU already contained crackles. Figure 3 illustrates that location in the signal path. Together, those findings ruled out the speakers as the source and narrowed the fault to the path between the CCU output and the amplifier loopback.

Narrow, verify, resolve

With the fault domain narrowed, collaboration with the amplifier subsystem team yielded two decisive findings:

A workload reduction (including a flat EQ diagnostic setting) eliminated the crackles, confirming the issue was load-sensitive and pointing away from hardware.

Deeper tracing identified an internal synchronization flaw in the amplifier's audio manager that intermittently corrupted the upstream return path under production EQ load.

The amplifier team reworked the audio manager task model and corrected the synchronization behavior. Once deployed, the crackling condition stopped reproducing in test fleets, and repeated speaker-replacement reports dropped out of new field data.

The result was a fix that resolved the actual failure mechanism, rather than treating one of its symptoms.

Case Study 2: Same symptom, different seam

The amplifier firmware fix eliminated crackling in many vehicles, but not all of them. A remaining subset showed the same customer-visible symptom with a different root cause. This time, the fault lay at another seam in the system: CPU scheduling inside the CCU.

Scheduling starvation: Runnable but not scheduled

Inside the CCU, the infotainment stack in the guest virtual machine (GVM) and the real-time platform domain in the privileged virtual machine (PVM) share physical CPU cores under a hypervisor.

• GVM (Guest VM): Runs the infotainment stack and the user-facing portion of the audio pipeline.

• PVM (Platform VM): Runs real-time/platform services and the platform portion of the audio pipeline.

CPU traces showed a clear pattern. Under concurrent system load, GVM vCPUs were runnable but were not scheduled onto physical cores in time. The resulting gaps were typically 8 to 13 ms, and one capture reached about 22 ms (Figure 4). During those windows, the audio path missed its deadline and produced no output samples, resulting in an audible dropout.

Fragile one-off tuning

The immediate mitigation combined core-affinity tuning with higher priority for audio-critical threads. In practice, that meant pinning GVM vCPUs away from the highest-priority real-time cores and giving audio work a better chance to run on time. In instrumented builds, those changes eliminated the observed starvation. But the fix was still a static configuration - useful as a mitigation, yet vulnerable to erosion as workloads and system behavior changed over time.

The durable fix: A scheduling contract

What the system needed was not another tuning guess, but a scheduling contract: a kernel-enforced policy that guarantees CPU budget for the relevant workload within defined limits, regardless of competing activity elsewhere in the system. Sporadic scheduling provides that mechanism (Figure 5).

Under sporadic scheduling, a thread or VM is assigned a replenishing CPU budget over a defined period. While budget remains, it can run at its assigned priority. Once the budget is exhausted, execution is deferred until it is replenished. This gives the audio path bounded access to CPU time without allowing it to monopolize the system.

With the tuned sporadic scheduling configuration in place, traces showed no instances of GVM starvation even during sustained high-load scenarios. Audio stayed stable while other features ran concurrently, resulting in a durable fix that made audio stability a schedulable property of the system.

Case Study 3: Chasing the remaining ghosts - platform optimization

Even after hypervisor-level protection was in place, a small number of crackles remained, most often when navigation and streaming audio were active at the same time. That pattern pointed away from the real-time OS and toward contention inside the application platform.

Correcting core assignments

Profiling showed the platform's core-group configuration was letting background work compete with latency-sensitive audio on the high-performance cores. On an asymmetric big/little CPU, that meant non-critical services could crowd the same big cores needed for consistent audio timing.

The core-group assignments were corrected, so background work ran primarily on small cores, leaving the big cores available for foreground and audio tasks. Contention dropped and crackles were reduced in mixed navigation-and-audio scenarios but not eliminated.

Audio thread starvation under navigation load

Even with better core isolation, crackles still appeared when navigation downloads spiked CPU loads while streaming audio was playing. Traces showed why: FastMixer itself stayed on time under its real-time policy. The starvation was downstream, where its output is consumed and written to hardware.

AudioOutput and the Audio HAL writer were running under the default fair scheduler. Under heavy navigation load, they were delayed 15-25 ms - well beyond a typical audio service interval - so clean PCM data still turned into audible dropouts.

System-level fix (across the platform stack)

The output-path threads were treated like normal best-effort work under load, so the fix couldn't live in the application alone. It required coordinated changes across the audio stack and the scheduler that determined when those threads could run.

The key step was to make the output path time-safe by moving AudioOutput and the Audio HAL writer to a real-time scheduling policy, so they could meet their deadlines consistently even when the rest of the system was busy.

With sporadic scheduling, corrected core assignments, and real-time output threads in place, validation runs showed zero crackles across the test tracks.

Conclusion: From chasing ghosts to mapping engineering playbooks

What started as "my speakers are broken" ended as a cross-engineering playbook. Along the way, the work reinforced two lessons: guessing is the most expensive path through a system-level problem, and the most useful evidence often comes from a clean capture at the right seam.

The playbook in five lines:

1. Reproduce before you analyze. A vague complaint is not a debugging target. The work begins when failure can be produced reliably.

2. Map the architecture first. Knowing where every boundary is before placing a probe prevents the most expensive mistake in embedded diagnostics: looking in the wrong place with the right tool.

3. Observe at seams, not inside components. Probes at boundary crossings - across VMs, between CPU and DSP, or across ECUs - help localize the fault domain without requiring immediate access to every internal stage.

4. Let the first bad boundary narrow the scope. Walking the probes in order converts a system-wide problem into a component-level one.

5. Contracts beat heroics. Durable behavior under load comes from enforceable scheduling guarantees, not one-off tuning that can erode as software and workloads evolve.

This approach now informs how intermittent system-level artifacts are investigated across domains, whether the symptom appears in audio, video, sensor processing, or networking. The probe points, tools, and automation may differ, but the logic remains the same.

For drivers, the outcome is simple: stable audio even while the vehicle is navigating, streaming, and handling other concurrent workloads. No special setting. No repeated service visits. Just a cabin experience that works as expected, even as the systems behind it grow more complex.

Note: Results here reflect the specific software builds, configurations, and workload mixes used in these investigations. Behavior can vary by vehicle program and runtime conditions.