Whether or not removal of the closet would hurt the value of your home depends on some variables (in my
opinion only).
Are you eliminating a
bedroom? If you're going from three bedrooms to two, for example, then yes I think market value of your home would be adversely affected. There are other factors to consider, including your overall house square footage, floor plan, your neighborhood/community, etc., etc., etc.
Could the existing walls be incorporated into the closet conversion, so that if/when it's time to sell, the buyer can
easily convert the giant closet back to its original purpose (or you can do it prior to the sale)? Don't know if it would work for you, but could you simply remove the closet doors, and install removable/adjustable, wall-to-wall shelving inside the closet space?
Such space could be earmarked for the wifey's shoe/handbag collection, along with any other stuff that would go on a shelf, rather than on hangers, in drawers, etc.
My spouse sells real estate and had a listing for a luxury townhome with two bedrooms, plus a den without a closet (nor a door to the room itself). It would have been fairly simple to add a door and a closet, but a surprising number of people just weren't interested because they either couldn't envision the conversion, or just didn't want to be bothered making the change themselves.
Just one,
uninformed
opinion that is worth every penny you're paying for it.