Amazon Linux 2023, DNS, and systemd-resolved — a story of no caching

I’ve learned a few things on an adventure this week, and I figure I should probably write them down.

First off, AWS throttles the number of DNS queries you can perform on a VPC. Apparently you’re limited to 1,024 packets for Elastic Network Interface (ENI). I am a little unclear on if the limit is per instance ENI, or the ENI on the VPC that is the DNS server. I am also unsure if that’s 1,024 request packets, or 1,024 total packets, but either way there is definitely a limit after which you will be throttled.

Secondly, AL2023 disables the systemd-resolved DNS caching behaviour, which means its pretty easy to hit that throttling limit. When you google for solutions you’ll find re:Post posts recommending dnsmasq, which is a perfectly fine piece of software but not really necessary if you already have systemd-resolved installed on your instance (as you do with AL2023).

First off you can verify that you’re not caching DNS with a command like this:

$ sudo resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: uplink

Link 2 (eth0)
Current Scopes: DNS
     Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
   DNS Servers: 192.168.1.3

The “resolv.conf mode: uplink” here means the “stub resolver” in resolved is disabled. You can extra doubly confirm that by asking about caching statistics:

$ sudo systemd-resolve --statistics
DNSSEC supported by current servers: no

Transactions           
Current Transactions: 0
  Total Transactions: 0
                       
Cache                  
  Current Cache Size: 0
          Cache Hits: 0
        Cache Misses: 0
                       
DNSSEC Verdicts        
              Secure: 0
            Insecure: 0
               Bogus: 0
       Indeterminate: 0

Not much caching happening here!

The stub resolver is disabled in /usr/lib/systemd/resolved.conf.d/resolved-disable-stub-listener.conf which is very cunning because its not in /etc/ like you’d expect. To enable resolved with caching, you need to do the following:

sudo rm /usr/lib/systemd/resolved.conf.d/resolved-disable-stub-listener.conf
sudo systemctl restart systemd-resolved

You should probably also check that /etc/systemd/resolved.conf doesn’t have Cache=no or DNSStubListener=no. You can then test like this:

$ dig madebymikal.com

; <<>> DiG 9.18.28 <<>> madebymikal.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31719
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;madebymikal.com.		IN	A

;; ANSWER SECTION:
madebymikal.com.	10	IN	A	192.99.17.214

;; Query time: 410 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Nov 11 07:25:42 UTC 2024
;; MSG SIZE  rcvd: 60

$ sudo systemd-resolve --statistics
DNSSEC supported by current servers: no

Transactions           
Current Transactions: 0
  Total Transactions: 1
                       
Cache                  
  Current Cache Size: 1
          Cache Hits: 0
        Cache Misses: 1
                       
DNSSEC Verdicts        
              Secure: 0
            Insecure: 0
               Bogus: 0
       Indeterminate: 0

A quick final note: I lost more time than I care to admit realising that the /etc/resolv.conf symlink is not managed by systemd-resolved. You need to manually point that at /run/systemd/resolve/stub-resolv.conf if it does not already. You’re welcome.