Damaging LED drivers by applying power to the output

Let's say I have a powerful LED that I want to direct-drive. Let's also say I want to run another circuit through a driver with a switch selecting which circuit is powered up, so I can drive the same LED at reduced output as well.
This implies that, running at full power in direct-drive, the driver's input has no energy, but its output sees whatever voltage the LED is currently running at.

Would this damage the driver?

Thanks

Use a DPDT switch, to move both the cathode and anode of the LED from the driver to direct drive and avoid the issue all together

You'd run into issues that depend on the driver. You run into stuff like the LED- not being connected to batt- because of stuff like current sense resistors, etc. Or on buck-boost drivers, the LED + could be the one connected to ground, and the driver actually generates a negative voltage for the LED-.

The only driver I think you could get away with something similar would be with one of the AMC7135 based boards. Since the LED is just connected to batt+ anyway, a switch to connect the output- of the board to batt- would bypass the regulator and go into direct drive.