Who was Caesar speaking to in the quote "But I am constant as the Northern Star, Of those true fixed and resting quality" in Julius Caesar?

Caesar is talking to Cassius and the conspirators when he says that he is constant and will not change his mind.

By the time Caesar met up with the senators on the Ides of March, they had already planned to kill him.  They were all outfitted with daggers, and they had a handy excuse for surrounding Caesar.  They had Metellus Cimber put forth a petition on behalf of his brother, Publius Cimber, whom Caesar had previously banished.  They knew he would never approve it, but the petition gave them a chance to make him look bad and gave them their opening to stab him.

Metellus Cimber begins the farce, asking Caesar who he would rather listen to in order to plead his brother’s case.  Then Cassius chimes in, making it look good.

Pardon, Caesar; Caesar, pardon:
As low as to thy foot doth Cassius fall,
To beg enfranchisement for Publius Cimber. (Act 3, Scene 1)

This is all just a distraction, of course.  They want any onlookers who are not involved in the plot to think that everything is normal.  Caesar is annoyed, insisting that he will not change his mind.  In his arrogance, he does not question their motives.  He just tells them angrily that once his mind is made up, it is made up!  He is speaking to Cassius, Metellus, and all of the others gathered around him.

Yet in the number I do know but one
That unassailable holds on his rank,
Unshaked of motion: and that I am he,
Let me a little show it, even in this;
That I was constant Cimber should be banish'd,
And constant do remain to keep him so. (Act 3, Scene 1)

Since Caesar does not relent, as they knew he wouldn’t, the men just continue until Casca stabs Caesar.  Then they all follow suit, ending last with Brutus.  Caesar is shocked that the man he regards as a son would be involved, let alone take the final blow.  Shakespeare has him saying, in Latin, what translates to “not you too, Brutus!”

The assassination of Caesar was well-planned and superbly executed.  There were many conspirators holding the knives, and Caesar was stabbed 23 times.  The conspirators let him bleed to death and then, in Shakespeare’s play, bathed their hands in his blood.  In their minds, they had freed Rome.