I recently spoke with support about this, and it is behaving as designed:
The fan speed is controlled by external temperature sensors outside the main APU. So it will take a longer time for the fan to spin up. The fan start temperatures are actually 40/46C. But the sensors are located further away from the main APU. You can see the configuration here: EmbeddedController/zephyr/program/lotus/azalea/overlay.dtsi at lotus-zephyr · FrameworkComputer/EmbeddedController · GitHub
Perhaps future ec firmware versions will tweak the fan curve?