For anyone stumbling onto this having a similar issue, I have found a solution:

Either: fully disable (or never enable) Google assistant on the headset (select skip / later when prompted, or go into bluetooth settings and remove the Google assistant)

Or: enable Low latency mode. 

I picked the first one, as hardly talking to Google when on my phone with headset on, and also the audio sounds better with ldac enabled Vs low energy Bluetooth mode (you can't have ldac on low latency LE MODE)

Hope this helps someone else