Let the agent handle it and remove yourself if you possibly can. Rember the agent has worked many hours with the potential buyers. The agent knows what they are looking for.
You may think an agent isn't showing the important features of your house, but the agent knows buyers aren't sold by the details until they've become emotionally involved with the big picture of your home. The presence of any member of the sellers family can't help. Always unnerves possible buyers, and often prevents a sale. Don't put this obstacle in your path. Leave when buyers are coming.