That was the clue I was looking for.
"built and cached based on the "Timezone For Guests."
That is exactly what it is doing.
Because I have guests set to UTC-4 and myself as UTC-4.
That is why only my time offset matches.
I will move this to a non bug report.