Big time when I was a kid. mainly crystals and fossils. Also sea shells. Rock crystals, amber, ammonites and trilobites were my favorite. Spend way too much of my pocket money on this stuff back in the day. Still have like 2 cabinets full of it.
I think halved and polished ammonites make great display pieces and quartz still fascinates me. Only gems I work with now are synthetic and in watch & clock movements. Pieces from the early 20th century and before still used natural stones, distinguishable by their paler color.