Well, if I'm remembering my "EMP-ology" right, you would need a weapon of around 20 megatons (quite a lot larger - on the order of a thousand times - than the Hiroshima/Nagasaki blasts) and, yes, needs to be very high indeed.
Could an electromagnetic pulse event damage quite a lot of things? Yes, yes it could - but I've never considered it to be a huge threat to my daily life. This may be because I live in Alaska, where power is an iffy thing at best anyway.
There are non-nuclear ways to induce high-intensity EM fields, most of which cannot be used (or are impractical to) to launch far-ranging events.
Furthermore, you can protect EM-sensitive equipment with a Faraday cage, proper grounding, and other techniques.
I guess my point is that a "real" EM attack would likely be a significant event - but one that would be survivable by society and a reasonably well-prepared nation. There are ways of both seeing it coming and defending against it. Moreover, on my personal heirarchy of "things to worry about," it's not very high. But your mileage may, of course, vary.