The drive in to work gave me some time to think about what happened last night.
There is nothing wrong with my binary. I remember it staying around 27.85 as I attempted to send it higher. It has to be the VCO varactor voltage railing out. I'd bet that if I adjust the slug out a little, it will go to 10m.
And I think I figured out one of my glitches too. When it detects the first reference pulse, it is not in the table and serial prints invalid code (as expected) to the computer (I have debug code printing every step). but IIRC, I think I have a return; after that which sends it back to loop(). Theres not much time between the reference and vco divider pulses so I think sending it back to loop is causing it to occasionally lose the first digit of the vco divider data. I will get rid of return; and see what happens.
Not sure if I should continue with these updates or wait until I'm done,..