How does Mr.Hooper respond when the parishioners first react to his veil in "The Minister's Black Veil"?

The members of the congregation of Mr. Hooper are filled with astonishment when they see their minister with a black veil covering the lower part of his face as he steps out his door. But, Mr. Hooper continues on his way deliberately, bending somewhat to look at the ground, yet "nodding kindly" to the members of the congregation who remain on the steps of the meeting-house.

After Mr. Hooper dons the black veil and steps out his door, the sexton who watches because the minister's presence is the signal to ring the church bell, cries out in amazement, "But what has good Parson Hooper got upon his face?" The others are so shocked to see that he has covered all but his eyes with a black veil that when he passes them and nods with gentleness toward them, few return his greeting.

This reaction toward the Reverend Mr. Hooper intensifies the longer that he wears the veil because people wonder if he is trying to hide something or if he sees in their faces some secret sin and, lest he reveal to others this sin, he shields his face. At any rate, they are threatened by the wearing of this veil, and sense a growing discomfiture around him. For this reason, Mr. Hooper is not invited to share Sunday dinners or attend weddings any more.

Therefore, rather than causing his congregation to become open about their human sins, the veil serves only to isolate Mr. Hooper himself. In fact, on his deathbed when he is asked by an attending minister to remove his veil so that others may see his "triumphant aspect" as he goes to "his reward," Mr. Hooper adamantly refuses, 

When the friend shows his inmost heart to his friend, the lover to his best-beloved; when man does not vainly shrink from the eye of his Creator, loathsomely treasuring up the secret of his sin,—then deem me a monster for the symbol beneath which I have lived and die. I look around me, and, lo! on every visage a black veil!" 

And, so, Mr. Hooper is buried with the veil upon his corpse, having desired to teach a moral lesson by wearing this veil as a symbol of the veil of falseness that each man and woman wears to conceal secret sins.