Apple recently dropped support for IE6 users for MobileMe (I think).
As long as you can get your pages to degrade gracefully and allow the user to be aware of why it looks bad or has less/little normal functionality you shouldn't loose too many people.
Sadly though, I still ask people what their favorite browser is, and they respond, "Internet Explorer?" And I ask what version they use, "I don't know."