How do we know that the house in "The Listeners" was very old and no one lived there?

I'd like to start by saying that readers can't know that the house is old and/or deserted.  There are lines that suggest that the house is old and empty, but those lines don't guarantee that fact.  

Let's start with old.  I believe that the best indication that the house is old is the mentioning of the turret.  A turret is reminiscent of old castles, but the design and use of turrets gained popularity again...

I'd like to start by saying that readers can't know that the house is old and/or deserted.  There are lines that suggest that the house is old and empty, but those lines don't guarantee that fact.  

Let's start with old.  I believe that the best indication that the house is old is the mentioning of the turret.  A turret is reminiscent of old castles, but the design and use of turrets gained popularity again during the Victorian era.  That period of history covers the second half of the 1800's.  The problem is that readers don't know a specific date for the poem's setting.  It's possible that the home was built in 1860, and the traveler in the story is visiting the home in 1880.  I wouldn't call a 20 year old home old.  

The poem does tell readers that birds flew out of the turret, and that the windows are "leaf-fringed." Both of those details make it seem like nature is taking back the home.  That does take some time, but even homes that have sat empty for even as little as a year are subject to ecological succession taking back the house.  I do believe that the house is old because it makes it creepier and more haunted feeling; however, I could defend the idea that the house is newer and has been recently deserted.  

As for the home being empty, I believe that is easier to support.  The narrator tells readers that the halls are empty.  Silence is a repeated motif in the poem.  Reader attention is drawn to the fact that the house is unusually quiet and still.  That suggests to me that the house is devoid of inhabitants.  The use of the word "phantom" also alerts readers to the lack of living souls in the house.  

But only a host of phantom listeners   

   That dwelt in the lone house then 

Whether you believe in ghosts or not doesn't really matter.  Ghosts would not be considered living, which supports the idea that no humans live in the house anymore. No humans in the house would also help explain why nature seems to be taking back the house.  There are no people around to stop nature's slow assault.