I went when I was younger (around 8 or 9) and didn't get a chance to fully appreciate it. Christmas is typically the time of the year where the estate is decorated really well, but also a time of pretty heavy tourism. Mid-to-late fall is also supposed to be really nice to see the fall colors of the landscape.
Anyways, the house tour usually takes about an hour and a half to two hours: be sure to go and walk around the grounds of the estate to get the full experience. You can get to downtown Asheville fairly easily from the Biltmore, but a fair bit of advice: depending on when you go in May, you might encounter a lot of college students (either just before their exams or just after; Asheville is one of the "hippy" areas of western NC), which I could see as being a turn-off for some people.