I haven't timed it but it's of the order of 30-40 minutes on 8-core i7 so I assume there is an issue with your build.
I was actually very pleasantly surprised by the speed and simplicity of the build. These things are usually a nightmare.

It turns out that a clean build is required and then it all works

I have successfully built the buildroot firmware and it works well. I then used the kernel config tool to add modules for usb serial devices. The build produces the following output:

make: Entering directory '/home/andy/Lyra-sdk/kernel-6.1'
SYNC include/config/auto.conf.cmd
CALL scripts ...