I never thought art class would ever come in handy but here we go (please bear with me, I'll try to be succinct) We'll blame any inaccuracies on my textbooks and horrible note-taking skills.
Warm, cold, dry and moist were terms for physical and animal qualities that were used from the middle ages to around the 18th century.
Warm equaled animated, exertion, glowing with artent feelings or skin bright with fever.
Cold equaled a dark spirit, lacking in enthusiasm life and energy.
The earliest (or so my professor said) artistic use of the term warm was the 1890 Oxford English Dictionary that quoted a descriptive passage (that I've long forgotten) about a warm painting.
In 1813 Charles Hayter, (ironic last name that accurately described my feelings for my history professor) in his book 'Introduction to Perspective', diagrammed the first recorded color wheel.
The colors yellow, orange and red were under warm, and green, blue and purple were under cold. Later color wheels used warm and cool (instead of cold).
I guess we just continue to call them that out of habit (and changing all those books would cost a fortune).
Hope this helped, and I agree; the English language is awkward and way too complicated.