A lot of theories are abound that he's God, the Devil, one of the Angels of Death/Judgment, or simply John's conscience. I doubt we'll ever get answers, but I'm not sure his talk about his son growing up to kill as easily as Marston, nor the fact that his response to "Damn You" is "Many have" fit either with God or a ghost of someone from John's past.
I do think someone was either testing or tormenting him, but I'm not sure it was someone from his past.
Also, I think it's kinda cool that the last place you see him is so significant in the end.