My experience hammering only makes the issue worse. Press in style rivets all will loosen after awhile in use with pocket clips. They are made with a twisted shank designed to hold down non moveable objects like name plates. Not pocket clips. Imo, this is another one of SF poor R&D decisions.
Anyway, if you pry, you need to apply even pressure (not cocked to one side) Plus, rivet head needs to be sitting up high enough to get underneath with makeshift tools..This is way too much messing around..
Since head is flattened, I'd center punch and drill the rivet. Then remove it with (1/16-3/32) ez-out extractor. The appropriate size drill to extractor size stamped onto ez-out. Only need tip of ez-out to engage the rivet head. Extractor bites-in when left hand rotated. The rivet should pop right out Ez PeEZzy..
As long as hole in Al is not made oblong prior to unnecessary hammering, prying and jabbing at it..
I'd then replace rivet with SS miniature machine screw, hole tapping with 6/40 thread, or 8/32 thread might be size that fits hole diameter. You will need to find best screw with right size head dia.. Allen or Torx driver slot..
Edit: Loctite red "stud-lock" forced into hole with compressed air might work temporarily. Stick body into warm oven to cure the stuff afterwards. Try this first, see how long it holds?