The Minecraft client does a PTR (reverse DNS) lookup on an IP address when it loads the server list. If you see a 3-5 second delay loading your Minecraft server in the Minecraft client server list, make sure that your server IP address has a proper PTR DNS record. You can check for PTR records on an IP address on this website: https://mxtoolbox.com/ReverseLookup.aspx
If your server IP does not have one, you should contact your hosting provider to fix this as it can only be fixed on their end.