If you're talking about using a loopback cable as a timing reference, what you're observing is excess phase due to time of flight for audio to reach the microphone, which could be several milliseconds. Have a look at the impulse, you should notice is no longer resides at t=0.
There's a timing offset option when measuring which you can use to move the impulse closer to T=0 which will make the phase response easier to view, and still maintain relative timing relationships between multiple measurements.