Plugins crashing

[x42]

Chasing down some possible leads, I came across GCC bug #86735 (can't post URL on the forum it seems!), which points at binutils (GNU assembler) bug #23465

Digest: a version of memcpy() that is inlined and uses AVX512F generates incorrect offsets, and its a GNU Assembler bug. That would cause heap scribbling, amongst other things, only on AVX512 capable hardware.
What compiler version is used for Harrison 32C 9.1.172 I wonder? (just to rule it in/out).
----
Mixbus32C9.2.172 (built using 9.2-172-gc80d313a86 and GCC version 8.3-posix 20190406)
----
So this was the version of GCC released in 2019 - and overlaps with the period that the bug was in the wild in binutils.
There are further reports of the bug being associated with  GCC 8.3 and quite a bit later up to about 2020, depending on the build environment.

So it does look quite possible that bug exists in the version of binutils 2.30/2.31 (rather than GCC itself) that may well have used to build Mixbus 32C...

Note that MIxbus itself does not use gcc to generate AVX instructions. Mixbus itself runs on generic x86 CPUs (there is no AVX512 memcpy).
Mixbus has dedicated DSP code for given CPU types, for realtime processing, yet the AVX512 asm intrinsic code [1] does not call memcpy either.

Still, it would be great if you could try to set the environment variable ARDOUR_FPU_FLAGS=95 to disable AVX512 when starting Mixbus?


[1] https://github.com/Ardour/ardour/blob/ma...avx512f.cc

[x42]

PS. If the issue was related to Mixbus AVX/binutils then it would manifest directly when using MIxbus.

This rather smells like some different issue with plugins.

[Monofin]

Great stuff :-)
I’ve not noticed a drop on DSP performance (but I run buffers at 256 on an RME HDSPe) when running 32C with the optimisations off. Perhaps that’s buffer constrained rather than CPU ;-)

I have noticed a bit of increased lag on the UI, and much slower export though.
Not sure if that is directly related.

Glad to be of help: I’ve got some C/C++ chops if that’s any help on the Ardour side of things, and very happy to test fixes/builds.

I’m going to take a look at the Linux build+ WINE for the plugins too.

I realise I’m probably pushing the forum areas now, so please DM me if there are other better paths to help out!

[Monofin]

PS. If the issue was related to Mixbus AVX/binutils then it would manifest directly when using MIxbus.

This rather smells like some different issue with plugins.

I disagree with this analysis:
I have been very careful to separate a number of issues relating to memory overwrites - including how VST3 plugins are executed in the Harrison/Ardour environment.
I have used the appropriate tools and debuggers to exclude plugin problems as a direct cause in the cases I have described.
I have also described the possibility of the bug in binutils creating any number of Heisenbugs - due to the nature of the fault/crash, it is not reasonable to assert that it would manifest directly - it will not happen every time, it will be non-deterministic and it will depend heavily on the heap allocation order.

That said: from the OP, I observe the possibility that there may be several issues present
However, I would suggest that the clearly identified likely problems, including the binutils bug need to be  *excluded* by facts before asserting the contrary.

[Monofin]

Chasing down some possible leads, I came across GCC bug #86735 (can't post URL on the forum it seems!), which points at binutils (GNU assembler) bug #23465

Digest: a version of memcpy() that is inlined and uses AVX512F generates incorrect offsets, and its a GNU Assembler bug. That would cause heap scribbling, amongst other things, only on AVX512 capable hardware.
What compiler version is used for Harrison 32C 9.1.172 I wonder? (just to rule it in/out).
----
Mixbus32C9.2.172 (built using 9.2-172-gc80d313a86 and GCC version 8.3-posix 20190406)
----
So this was the version of GCC released in 2019 - and overlaps with the period that the bug was in the wild in binutils.
There are further reports of the bug being associated with  GCC 8.3 and quite a bit later up to about 2020, depending on the build environment.

So it does look quite possible that bug exists in the version of binutils 2.30/2.31 (rather than GCC itself) that may well have used to build Mixbus 32C...

Note  that MIxbus itself does not use gcc to generate AVX instructions. Mixbus itself runs on generic x86 CPUs (there is no AVX512 memcpy).
Mixbus has dedicated DSP code for given CPU types, for realtime processing, yet the AVX512 asm intrinsic code [1] does not call memcpy either.

Still, it would be great if you could try to set the environment variable ARDOUR_FPU_FLAGS=95  to disable AVX512 when starting Mixbus?


[1] https://github.com/Ardour/ardour/blob/ma...avx512f.cc

However, this is a bug in Binutils, not GCC: To bisect the issue, it would be very helpful to have the specific revision (git hash) that Harrison 32C V9.2.172 is based upon.
I could then rebuild and test a hypothesis against the various binutils versions?

I note that the current release of Ardour does not manifest the issues as described in the report here, but yet it is specific to a given release version of 32C.

Calling an intrinsic, or other related inline, or 'library' function may trigger this in any number of the related libraries to Harrison/Ardour libraries that perform feature detection as well.
A full trace of the paths that the AVX512F optimisations in Harrison/Ardour take, including any compiler or assembler auto-vectorisation (Or wbo/tco paths, including BOLT) would be key to nailing this down.

As such, whether the Harrison/Ardour code directly invokes the 'faulty_memcpy()' is moot - until it can be shown that there are no other paths to a possible underlying binutils AVX512F bug exist.

[x42]

However, this is a bug in Binutils, not GCC: To bisect the issue, it would be very helpful to have the specific revision (git hash) that Harrison 32C V9.2.172 is based upon.
I could then rebuild and test a hypothesis against the various binutils versions?

I note that the current release of Ardour does not manifest the issues as described in the report here, but yet it is specific to a given release version of 32C.

Ardour and MIxbus use the exact same build environment (windows binaries are cross compiled on a debian 10.13 aka buster cowbuilder).

As for the version of binutils used:


Package: binutils-mingw-w64-x86-64
Source: binutils-mingw-w64 (8.3)
Version: 2.31.1-11+8.3
Homepage: https://www.gnu.org/software/binutils/
Built-Using: binutils (= 2.31.1-11)
Filename: pool/main/b/binutils-mingw-w64/binutils-mingw-w64-x86-64_2.31.1-11+8.3_amd64.deb
MD5sum: 49bc16bda4a3dde124d1977f56516f03
SHA256: dde3fa0a3e0da70d638ed2e5eb856e4f46033b4824a435fbe8d8bac0b67acb


https://packages.debian.org/buster/binut...w64-x86-64

Mixbus runs fine on systems without AVX. There is also no auto-vectorization for a given arch since that would prevent the binary from running on CPUs that do not support a given feature. -- The functions that use SSE, AVX, FMA, or AVX512 are optional.

[johne53]

@Monofin - the evidence so far suggests that supporting AVX512F has introduced this problem but your current solution (--no-hw-optimizations) will disable several optimizations so we can't say definitively that AVX512F is the troublesome one.

@x42's suggestion is for you to launch from a Command Prompt but without using --no-hw-optimizations and instead, to set an environment variable prior to launching Mixbus - i.e. type set ARDOUR_FPU_FLAGS=95 and then launch Mixbus from your Command Line. This will isolate the optimizations and tells us if AVX512F is causing the problem or if imaybe it's one of the others. Do please give this a try.

[Monofin]

The environment variable is very helpful, thanks - I will try that next: it will hopefully confirm the assertion that it is specifically AVX512F optimisations causing the issue.

However, I must pick up that I did not assert that —no-hw-optimizations was a solution, only a debugging data point, and also a means for me to continue using 32C v9 (as per my first post): now we have a specific flag to work with, it narrows the window further :-)

(The AVX512F assertion come from wide trials on a reasonable range of machines, trying to find one with enough power but also stable enough to complete a large-ish project of mine.)

With the link to the .cc file, I can get a lot further - I appreciate that, thank you:-)
What would be very useful is the commit hash or tag that it was built from, and the tooling environment, including the binutils:
However, only of limited use - the current version of Ardour does not exhibit the same crashes- that may be because it is built against an updated binutils (I will see if I can check), or it could be that the Harrison specific DLLs for the channel processing are contributing in some way (obviously we don’t have the source for those).

Either way, a version of 32C and it’s built in plugins guaranteed not built against the identified buggy binutils, (or even just a more recent version of GCC;-) might be a more generally useful result - if not a little less specific in the bugfix :-)

[Monofin]

Cheer to @x42 for the binutils version - much appreciated. It's version is definitely in the region where the bug was identified and fixed, just need to work backwards towards it.

Also - ok, so no auto vectorisation or other library based voodoo (symbol multi-versioning, etc.). All good stuff, and really appreciated information

[Monofin]

Trying the suggested set ARDOUR_FPU_FLAGS=95: Under Windows 11 power shell, 'set' is insufficient (getenv() doesn't return the correct value), so I used the Environment Variables (per user) dialog - and then it worked.

With ARDOUR_APU_FLAGS=95 verified as set in the environment, clicking is gone, crashing I experienced is gone too.
(Unsetting this and the clicking is back and the crashes are too.)

I'd suggest that its an issue with the AVX512F code - and looking at the source code, it seems that the intrinsics being used (if I'm reading it correctly) are some of the ones that are affected by the Binutils bug listed earlier. (Though I've not picked apart the actual routines to check them).

----
Console Output
-----
PS C:\Program Files\Mixbus32C-9\bin> echo $env:ARDOUR_FPU_FLAGS
95
PS C:\Program Files\Mixbus32C-9\bin> .\Mixbus32C.exe
PS C:\Program Files\Mixbus32C-9\bin> ARDOUR_DATA_PATH not set in environment
Mixbus32C9.2.172 (built using 9.2-172-gc80d313a86 and GCC version 8.3-posix 20190406)
Crash Log: C:\Users\ebeas\AppData\Local\Mixbus9\CrashLog\Mixbus32C-9.2.172-crash-1712488959.txt
Mixbus32C: [INFO]: MMCSS Initialized
Mixbus32C: [INFO]: Your system is configured to limit Mixbus32C to 2048 open files
ARDOUR_CONFIG_PATH not set in environment
Mixbus32C: [INFO]: Loading system configuration file C:\Program Files\Mixbus32C-9\share\ardour9\system_config
Mixbus32C: [INFO]: Loading user configuration file C:\Users\ebeas\AppData\Local\Mixbus9\config
Mixbus32C: [INFO]: Using AVX and FMA optimized routines
Mixbus32C: [INFO]: Loading plugin meta data file C:\Program Files\Mixbus32C-9\share\ardour9\plugin_metadata\plugin_tags
Mixbus32C: [INFO]: Loading plugin statistics file C:\Users\ebeas\AppData\Local\Mixbus9\plugin_metadata\plugin_stats
Mixbus32C: [INFO]: add_lrdf_data 'C:\Users\ebeas\AppData\Local\Mixbus9\rdf;C:\Program Files\Mixbus32C-9\share\ardour9\rdf'
Mixbus32C: [INFO]: read rdf_file 'file:///C:/Program%20Files/Mixbus32C-9/share/ardour9/rdf/ardour-presets.n3'
Mixbus32C: [INFO]: Loading default ui configuration file C:\Program Files\Mixbus32C-9\share\ardour9\default_ui_config
Mixbus32C: [INFO]: Loading user ui configuration file C:\Users\ebeas\AppData\Local\Mixbus9\ui_config
Mixbus32C: [INFO]: Loading 457 MIDI patches from C:\Program Files\Mixbus32C-9\share\ardour9\patchfiles
Mixbus32C: [INFO]: Loading color file C:\Program Files\Mixbus32C-9\share\ardour9\themes\dark-mixbus32c.colors
Mixbus32C: [INFO]: Loading ui configuration file C:\Program Files\Mixbus32C-9\share\ardour9\clearlooks.rc
Mixbus32C: [INFO]: Loading bindings from C:\Program Files\Mixbus32C-9\share\ardour9\ardour.keys
Loading ui configuration file C:\Program Files\Mixbus32C-9\share\ardour9\clearlooks.rc
Announcement string is too long (probably behind a proxy).
Set cursor set to default
Setting time domain
Mixbus: Screen height is 1340; Font scale is 1; User-scale is 1; Suggested scale: 1.21818