Rifle's are shaped as they are so the user can hold aim and manipulate it. If it's built into a mechanism that isn't handled by humans, you have much more leeway as to where stuff is placed. The control electronics need not even be in the sentry gun, just the cameras and the motors. It could communicate over encrypted wi-fi to a base unit kept in a secure bunker elsewhere. Hell, maybe even via satelite all the way back to the Pentagon.
It shouldn't be too hard to build the whole thing into an armored box where nothing but the sensors and the muzzle are projecting. Cover the sensors with wheels of Lexan that can take a hit, then rotate until a clear section is exposed, kind of like the covers on race car cameras that keep turning to get the dirt off...
Think of a unit like the coaxial machineguns on tank turrets, especialy in WWII, where there was usualy just a armored "ball" with a few inches of muzzle sticking out. Since no operator needs to sit in the thing, packing it all inside an armored box should be even easier. Slope and angle the sides like an M1 Abrams tank, so bullets are more prone to glancing off...
Also, maybe just don't bother armoring it. It's a GOOD thing if it forces enemy snipers to shoot at it, instead of our troops. Maybe for every sentry gun unit, three or four of them are cheap non-functional decoys with a motor that makes them twitch occasionaly?
There are high-speed video cameras, triangulating microphones, and things like mili-wave radar that can see, track, and balisticaly plot the origin of something as small as .22 and .30 caliber bullets. A larger anti-material round like a .50 is even easier. The engineering and targeting is already well understood since it's been applied to counter artillery and anti-mortar systems already. Hook that info the networked sentry guns, and they could return fire accurately no matter how well concealed the sniper was. The balistic path of the snipers bullet can't lie.
1 second after he fires his rifle, he's pinned down by machinegun fire that's accurate to within one foot of his position, and if he's baracaded, indirect fire weapons like the auto-detonating micro-shells from the OICW project, or just plain mortar rounds are dropping on him.