Weirdly Long DNS Switchover?

by jefftkjefftk1 min read16th Feb 20211 comment

8

Personal Blog

[EDIT: as Jai points out in the FB comments, the problem was likely forgetting IPv6. The AAAA record is now updated, and I should be able to tell in a few hours whether that fixed it.]

I'm helping my dad migrate some code to a new server. It was at 162.209.99.139 for years, but ten days ago he changed his DNS settings to point to 34.199.143.13. Everything looks good to me:

$ dig notes.billingadvantage.com
notes.billingadvantage.com. 1800
    IN    A    34.199.143.13
The TTL is 1800s, or 30min, which agrees with dns.google. I expected everyone would be moved over within a couple hours, but a week later the old server is still receiving traffic nearly as much as the new:

date old server new server
2021-02-05 943 0
2021-02-06 201 127
2021-02-07 17 108
2021-02-08 364 423
2021-02-09 488 448
2021-02-10 255 503
2021-02-11 281 345
2021-02-12 250 248
2021-02-13 0 88
2021-02-14 0 78
2021-02-15 217 262
2021-02-16 202 287

The old server getting no traffic on the 13th and 14th is probably because that's the weekend, and the users who happen to be still stuck on the old site aren't using it on the weekend. I asked one of the users still getting the old server to try rebooting, to no effect.

I thought maybe something was misconfigured with the name servers, but it looks fine:

$ whois billingadvantage.com \
   | grep 'Name Server'
Name Server: NS1.ZEROLAG.COM
Name Server: NS2.ZEROLAG.COM

$ dig notes.billingadvantage.com \
   @ns1.zerolag.com
notes.billingadvantage.com. 1800
   IN    A    34.199.143.13

$ dig notes.billingadvantage.com \
   @ns2.zerolag.com
notes.billingadvantage.com. 1800
   IN    A    34.199.143.13
I'm not seeing any references to the old IP address anywhere.

Any guesses about why the traffic isn't moving over?

8

1 comments, sorted by Highlighting new comments since Today at 12:20 AM
New Comment

Your A records are fine, but you seem to have changed name servers. Your old NS records are probably cached all over the place; the TTL on those seems to be 48 hours. It looks like the old server (at my-tss.com) is serving the correct data now, but it was probably still serving stale data when you saw the problem. Possibly it took it a while to realize that it wasn't authoritative for the zone, or possibly there was an update problem.

Generally, it's better to do the server change and the data change separately. And you have to make sure that the new and old servers are serving the same thing through the full TTL of the old NS records, or at least have the old server definitively reconfigured not to see itself as authoritative so that it can avoid misleading other systems when it gets a query.

Evidence: If I let my system use my ISP's servers and do dig -t ns bilingadvantage.com., I get the wrong cached data:

   ; <<>> DiG 9.11.27-RedHat-9.11.27-1.fc33 <<>> -t ns billingadvantage.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38501
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: b250042072aca8b7ccde8a21602d31062c20fb70fa6f7d57 (good)
;; QUESTION SECTION:
;billingadvantage.com.          IN      NS

;; ANSWER SECTION:
billingadvantage.com.   3600    IN      NS      dns2.my-tss.com.
billingadvantage.com.   3600    IN      NS      dns1.my-tss.com.

;; Query time: 168 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Feb 17 10:06:46 EST 2021
;; MSG SIZE  rcvd: 122

... but if I go and query the actual GTLD servers, say with dig -t ns @a.gtld-servers.net billingadvantage.com, I get the right data:

; <<>> DiG 9.11.27-RedHat-9.11.27-1.fc33 <<>> -t ns @a.gtld-servers.net billingadvantage.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27939
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;billingadvantage.com.          IN      NS

;; AUTHORITY SECTION:
billingadvantage.com.   172800  IN      NS      ns1.zerolag.com.
billingadvantage.com.   172800  IN      NS      ns2.zerolag.com.