Having been to many a Speyside festival, and being a local (I grew up in Rothes) i would say that your best bet would be to stay in, or around Dufftown. There are obviously events in Elgin, Rothes, and Aberlour, but there are also buses that run periodically through the day that will take you to the other "hubs".
I agree that having a car is not the best of ideas, unless you take along someone who doesn't like whisky!
If you're not that bothered about a wee bit of travelling to, and from the different areas, i would recommend:
The Highlander Inn, Craigellachie...great pub, great ales, geat whiskies!!
Alternatively, if you go on to http://www.dufftown.co.uk
and have a look at their festival website, there will be contact details for Fiona at The Whisky Shop. She is an absolute legend when it comes to Speyside Festivals. Speak to her, tell her that Susan from Douglas Laing said you should talk to her and she'll try and do her best for you!
....and after she's done that, she'll shout at me for giving her more work to do!!