"Back then, we would have about 30 people at the royal table," he shared. "That's because the Gloucesters, Kents, and ...