Plugins crashing

[Monofin]

Just a side note:
The version of Binutils current with Debian buster seems to be 2.31.1-16, rather than 2.31.1-11 (according to the package site).
I'll pull the source packages and see if there's anything interesting - it maybe it makes no difference at all! :-)

[mattjmail]

Chasing down some possible leads, I came across GCC bug #86735 (can't post URL on the forum it seems!), which points at binutils (GNU assembler) bug #23465

Digest: a version of memcpy() that is inlined and uses AVX512F generates incorrect offsets, and its a GNU Assembler bug. That would cause heap scribbling, amongst other things, only on AVX512 capable hardware.
What compiler version is used for Harrison 32C 9.1.172 I wonder? (just to rule it in/out).
----
Mixbus32C9.2.172 (built using 9.2-172-gc80d313a86 and GCC version 8.3-posix 20190406)
----
So this was the version of GCC released in 2019 - and overlaps with the period that the bug was in the wild in binutils.
There are further reports of the bug being associated with  GCC 8.3 and quite a bit later up to about 2020, depending on the build environment.

So it does look quite possible that bug exists in the version of binutils 2.30/2.31 (rather than GCC itself) that may well have used to build Mixbus 32C...

Note  that MIxbus itself does not use gcc to generate AVX instructions. Mixbus itself runs on generic x86 CPUs (there is no AVX512 memcpy).
Mixbus has dedicated DSP code for given CPU types, for realtime processing, yet the AVX512 asm intrinsic code [1] does not call memcpy either.

Still, it would be great if you could try to set the environment variable ARDOUR_FPU_FLAGS=95  to disable AVX512 when starting Mixbus?


[1] https://github.com/Ardour/ardour/blob/ma...avx512f.cc

Setting the environment variable ARDOUR_FPU_FLAGS=95 fixes a problem I've had with certain Waves and UAD plugins: if I put them on a mixbus and pass signal through them I get a staticky crackling sound, which might be the "continual random clicks in the output audio" that Monofin described on his first post (together with random cashes). Before setting the environment variable I could make the crackling sound go away by playing with the plugin latency as Monofin describes (sometimes decrementing by 1 works, sometimes other small increases/decreases cause it to go away).  I'm using Mixbus32C version 9.2.172 on Windows 11, Lenovo Legion 5 Pro 11th Gen Intel Core i7-11800H with an NVIDIA RTX 3050 (I noticed Monofin only observes the problem on an i9). 

The problem also exists on Ardour version 8.4.0: if I insert one of these plugins on an audio channel it's fine, but if I  insert it on an audio buss and rout the signal through the plugin via an aux send I get the nasty crackling sound.

The plugins I've noticed that exhibit this behavior are Waves Abbey Road Chambers, Waves dbx-160, UAD Waterfall, UAD Lexicon 224, UAD Distressor,  Relab LX480 essentials. Most other plugins work fine.  

I'm grateful for the environment variable suggestion.

[Monofin]

Chasing down some possible leads, I came across GCC bug #86735 (can't post URL on the forum it seems!), which points at binutils (GNU assembler) bug #23465

Digest: a version of memcpy() that is inlined and uses AVX512F generates incorrect offsets, and its a GNU Assembler bug. That would cause heap scribbling, amongst other things, only on AVX512 capable hardware.
What compiler version is used for Harrison 32C 9.1.172 I wonder? (just to rule it in/out).
----
Mixbus32C9.2.172 (built using 9.2-172-gc80d313a86 and GCC version 8.3-posix 20190406)
----
So this was the version of GCC released in 2019 - and overlaps with the period that the bug was in the wild in binutils.
There are further reports of the bug being associated with  GCC 8.3 and quite a bit later up to about 2020, depending on the build environment.

So it does look quite possible that bug exists in the version of binutils 2.30/2.31 (rather than GCC itself) that may well have used to build Mixbus 32C...

Note  that MIxbus itself does not use gcc to generate AVX instructions. Mixbus itself runs on generic x86 CPUs (there is no AVX512 memcpy).
Mixbus has dedicated DSP code for given CPU types, for realtime processing, yet the AVX512 asm intrinsic code [1] does not call memcpy either.

Still, it would be great if you could try to set the environment variable ARDOUR_FPU_FLAGS=95  to disable AVX512 when starting Mixbus?


[1] https://github.com/Ardour/ardour/blob/ma...avx512f.cc

Setting the environment variable ARDOUR_FPU_FLAGS=95 fixes a problem I've had with certain Waves and UAD plugins: if I put them on a mixbus and pass signal through them I get a staticky crackling sound, which might be the "continual random clicks in the output audio" that Monofin described on his first post (together with random cashes). Before setting the environment variable I could make the crackling sound go away by playing with the plugin latency as Monofin describes (sometimes decrementing by 1 works, sometimes other small increases/decreases cause it to go away).  I'm using Mixbus32C version 9.2.172 on Windows 11, Lenovo Legion 5 Pro 11th Gen Intel Core i7-11800H with an NVIDIA RTX 3050 (I noticed Monofin only observes the problem on an i9). 

The problem also exists on Ardour version 8.4.0: if I insert one of these plugins on an audio channel it's fine, but if I  insert it on an audio buss and rout the signal through the plugin via an aux send I get the nasty crackling sound.

The plugins I've noticed that exhibit this behavior are Waves Abbey Road Chambers, Waves dbx-160, UAD Waterfall, UAD Lexicon 224, UAD Distressor,  Relab LX480 essentials. Most other plugins work fine.  

I'm grateful for the environment variable suggestion.

Bang on - that sounds exactly like my symptoms too here:

I had neglected to try the same routing on Ardour 8.4.0 - and I have just tested that, and reproduced @mattjmail's symptoms.

My i7s were only 7700K and other 7th gen, and I didn't have anything 11th gen to try on: sounds like the 11th gen also has AVX512F support?

Glad to have some more evidence in the pot! :-)

[x42]

The problem also exists on Ardour version 8.4.0: if I insert one of these plugins on an audio channel it's fine, but if I  insert it on an audio buss and rout the signal through the plugin via an aux send I get the nasty crackling sound.

Could you try today's Ardour https://nightly.ardour.org/ build (demo is fine to test) 8.4.122 or later. Those are now built with more recent binutils (2.35).

[mattjmail]

The problem also exists on Ardour version 8.4.0: if I insert one of these plugins on an audio channel it's fine, but if I  insert it on an audio buss and rout the signal through the plugin via an aux send I get the nasty crackling sound.

Could you try today's Ardour https://nightly.ardour.org/ build (demo is fine to test) 8.4.122 or later. Those are now built with more recent binutils (2.35).

I get the same problem on Ardour 8.4.122 after clearing ARDOUR_FPU_FLAGS, and setting it to 95 still fixes it.  I used the demo and tried both the debug and optimized version.

[x42]

Thanks for checking. I have neither a Windows machine, nor a system that features AVX512f. It would be great if you could help out to get to the bottom of this!

If the issue persists with 8.1.126, please try various values for ARDOUR_AVX_DEBUG (do not set ARDOUR_FPU_FLAGS), and see if any of those helps.

• ARDOUR_AVX_DEBUG=32  (use default memcpy, not custom AVX to copy audio buffers)
  
• ARDOUR_AVX_DEBUG=28  (do not use AVX512f to mix buffers)
  
• ARDOUR_AVX_DEBUG=60  (only use AVX512f for peak calculation)
  

That will allow us to narrow down the issue.

(those are bitwise flags https://github.com/Ardour/ardour/commit/...db3784dae7 )

[johne53]

I do have a Xeon CPU here (running Windows 10) and I've tried Mixbus32C 9.2.32 here as well as 9.2.172. I don't have any Abbey Road plugins or CLA etc but I do have the XT/ACE plugins that came with Mixbus and I don't see any crashing or hear any crackling when I use them on mix busses. Is there something more to this? e.g do the problems only happen with certain buffer sizes (or maybe when used in conjunction with other plugins)?

[Edit...] Solved - see post #31

[Monofin]

Hi, @x42 - thanks for pointing out the bitwise flags:
Can you confirm that the version of Binutils has been updated on the Ardour 8.4.122 nightly build machine(s)?

If not, with the other flags there is a risk of misidentification of the specific issue by reducing the likelihood of encountering the issue - and therefore a chasing a blind alley.
(If it is the Binutils bug, it is likely to affect all of the above - and changing the flags may shift the probabilities of crash or clicks to the point of them not occurring significantly regularly - the bug is still there, it is now hidden - a 'signal to noise' issue).

[mattjmail]

Thanks for checking. I have neither a Windows machine, nor a system that features AVX512f. It would be great if you could help out to get to the bottom of this!

If the issue persists with 8.1.126, please try various values for ARDOUR_AVX_DEBUG (do not set ARDOUR_FPU_FLAGS), and see if any of those helps.

• ARDOUR_AVX_DEBUG=32  (use default memcpy, not custom AVX to copy audio buffers)
  
• ARDOUR_AVX_DEBUG=28  (do not use AVX512f to mix buffers)
  
• ARDOUR_AVX_DEBUG=60  (only use AVX512f for peak calculation)
  

That will allow us to narrow down the issue.

(those are bitwise flags https://github.com/Ardour/ardour/commit/...db3784dae7 )

Ardour build 8.4.126 fixed the staticky/crackling problem on my particular machine on the plugins that were giving me trouble when placed on aux sends (Waves Abbey Road chambers and dbx-160, UAD distressor, waterfall, Lexicon224, and Relab LX480 essentials).

[x42]

Hi, @x42 - thanks for pointing out the bitwise flags:
Can you confirm that the version of Binutils has been updated on the Ardour 8.4.122 nightly build machine(s)?

Yes, running builds on Debian GNU/Linux 11.9 (bullseye) since Ardour 8.4.122.

Code:

# x86_64-w64-mingw32-ld --version
GNU ld (GNU Binutils) 2.35.2

If not, with the other flags there is a risk of misidentification of the specific issue by reducing the likelihood of encountering the issue - and therefore a chasing a blind alley.

I have found a fixed a bug in the AVX512f code [1] (fixed in 8.4.126), and then thought that there could more than one bug which I missed last last night.
In that case it would be handy to have some method to debug and isolate the specific AVX function to track down the issue, so I have added those debug flags.

It could well be that both changes were required (binutil update, and the memory corruption fix) though I don't see how the binutils issue [2] would have any effect on the AVX512f code in Ardour/Mixbus. It's still good to rule it out by using a more recent toolchain.

--

[1] https://github.com/Ardour/ardour/commit/...d2e7565598
[2] https://sourceware.org/bugzilla/show_bug.cgi?id=23465