Inconsistent Acoustic Timing Reference using UMIK-1

I'm taking multiple measurements of the same driver and getting inconsistent Estimated IR Delays. I'm using a tweeter as the Acoustic Timing Reference, and then measuring that same tweeter. I would expect any delay to be 0 when using the same driver for both the timing reference and the signal, But what I get instead is a variety of delays from -0.09ms to 0.3 ms. This inconsistency is making my measurements unusable. What do I do?

Background:
I'm using a UMIK-1, have the mic 30" from the tweeter on axis. Using the left channel for Acoustic Timing Reference, and also using left channel for measurement. I plan to use the tweeter as the acoustic timing reference when measuring other drivers on the right channel at later steps where I would observe a delay. I'm building a 3-way speaker, with passive crossover and the base assumption is drivers are time aligned. The speaker is a passive speaker system, I do not have an active system or dsp equiment. Using REW V5.31.3.

UMIK-1 does not have its own clock generator. It gets the time from USB. Therefore there is a spread.

I am using the acoustic timing reference feature John specifically added for USB mics. I am under the impression that it works out the relative timing with the sweep/cricket before and after the sweep, making only the recording and known sample rate the only requirement. Since it knows what signal was sent, it uses cross correlation to find the peaks of the timing marks and then uses cross correlation between what was sent and what was received and then subtract the difference. I believe the instability is not in any hardware at all, but in the error in the cross correlation routine where the highest correlation isn't actually in the correct phase. I am not an expert on this however, so, hence i'm posting here.

So what do speaker designers do?

The USB clock has some variation, which can lead to variations in results. That would be evident in varying figures for the clock correction. Strong reflections may also cause issues. For polar speaker measurements it is common to use an analog mic with a loopback timing reference to avoid issues for the off-axis measurements.