Where did the signalman say he had seen the narrator before?

At the beginning of the story, the signalman is certain that he has seen the narrator before:

"I was doubtful," he returned, "whether I had seen you before."

"There?" I said.

Intently watchful of me, he replied (but without sound), "Yes."

When asked, the signalman says that he has seen the narrator at the red light, just a short distance from his signal box. The narrator denies having ever been at the red light, but...

At the beginning of the story, the signalman is certain that he has seen the narrator before:

"I was doubtful," he returned, "whether I had seen you before."

"There?" I said.

Intently watchful of me, he replied (but without sound), "Yes."

When asked, the signalman says that he has seen the narrator at the red light, just a short distance from his signal box. The narrator denies having ever been at the red light, but the signalman does not seem convinced, and he has a good reason for such a feeling. As the narrator later learns, the signalman has been haunted by the specter of a man at the red light. In addition, the signalman heard this mysterious figure say the exact same words as the narrator:

Halloa! Below there!

While the signalman wants to believe that he saw the narrator at the red light, he is, in fact, very much mistaken. This mysterious figure is a ghost who, in a tragic turn of events, is warning the signalman of his own demise. Both the signalman and the narrator, however, will learn this lesson too late.