YOU SAID:

to whom am I speaking

INTO JAPANESE

私は話している人に

BACK INTO ENGLISH

I was talking to

INTO JAPANESE

私に話していた

BACK INTO ENGLISH

I was talking to