It's all about context.
- Middle of the day daylight at >5000K is overwhelming to the tune of some 1000 watts per square meter. It snaps those pupils nearly shut and is everywhere, forcing your visual perception to adapt. But this is what our eyes are adapted for so it looks right.
- Step indoors into a well-lit space 5-10 meters away from any windows and suddenly 5000K overhead lights would look cold by compare since there's a good 2 orders of magnitude less light energy involved. In fact 5000K will look sufficiently cold that it's not the color temperature of choice in workplaces and retail - CCTs around 4000K are preferred in general lighting for a neutral cast (at least in workplaces where promoting alertness is important).
- Step into a home lit more comfortably than a public space more relaxing than a workplace - likely an order of magnitude less light energy per unit area - and the preferred CCTs will tend to drop again because the home is about relaxation and leisure.
- Step into the dark and even relatively low CCTs like 3500K can appear to be an almost stark white because you're experiencing something close to the maximum possible contrast - a pool of bright light dropping off to darkness with your color vision and night vision in mild conflict.
At each step your pupils dilate to collect more light and your eyes have to work with ranges off by both orders of magnitude and wildly varying min/max values for each situation. Something that's perceived to be dark in daylight could still be beyond the top of the scale in an office setting.
On a related note, outside of bright sunlight-like conditions, cooler CCTs are subjectively perceived to be brighter than lower CCTs even when the lumens are the same.