Mine seemed to have been glued in somehow. I had the same issue installing my McE2s, but it only needed a little shot of the heat and it broke free. I'm not a weak guy, and I have some dexterity with tools, but this thing wouldn't budge!
What I ended up doing was heating it up enough till it was a little soft, then I pulled the button out. After it cooled down, I used my dremel to widen the hole, round it out again and sand down the rough edges from my earlier attempts. Now it works great! Just a little stiff. I'll play with it more another time.
Ted