I would be hurt if my DH went sightseeing with another woman, but not with me. I'm sure he would understand that.
Your job is important to you and he should want you to stay at a nice hotel that is close to your conference. Orlando is known for the traffic so the closer you are to the conference the easier it will be for you. Maybe you should go to your work conference alone and maybe your family could go on a vacation at another time?
It sounds like you two need to compromise. Suppose you book a more inexpensive vacation this year then maybe next year you two can create a "vacation fund" and go somewhere a little more expensive like Hawaii. I hope you two can figure it out
