Fixes #6: An oddysee: All resolves to a known Ubuntu bug :( Now a new…

… role resolves that problem and kube-dns picks up the correct DNS nameserver from `/etc/resolve.conf`.
jonashackt · Aug 27, 2018 · 3091bbe · 3091bbe
1 parent dc5e77c
commit 3091bbe
Show file tree

Hide file tree

Showing 6 changed files with 53 additions and 3 deletions.
diff --git a/README.md b/README.md
@@ -137,11 +137,16 @@ See the following links:
 
 ##### Kubernetes DNS (kube-dns)
 
+Debug Service DNS: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#does-the-service-work-by-ip
+
+Debug kube-dns: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
+
 https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/:
 
 > Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names.
 
-####### nslookup for kubernetes not working in kubedns / main.yml
+
+###### nslookup for kubernetes not working in kubedns / main.yml
 
 We set `--ip-masq=false` inside the `docker.service`. The problem is
 
@@ -167,6 +172,36 @@ Name:      kubernetes
 Address 1: 10.32.0.1 kubernetes.default.svc.cluster.local
 ```
 
+###### if nslookup still doesnt work - "If the outer resolv.conf points to 127.0.0.1:53, then you will have a DNS lookup loop"
+
+see https://github.com/kubernetes/kubernetes/issues/49411#issuecomment-318096636
+
+> Kubedns inherits the contents of “/etc/resolv.conf” something the maintainers of these pods should document at the following site (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns ) as it takes a lot of research and googling to find such details hidden under forum comments. My two cents
+
+check your Ubuntu boxes `cat /etc/resolve.conf`: 
+
+```
+nameserver 127.0.0.53
+```
+
+> systemd-resolved on my host listens on 127.0.0.53:53 for dns queries, as soon as I uninstall systemd-resolved and install dnsmasq my node has an entry of 127.0.0.1 in /etc/resolv.conf instead of 127.0.0.53, this gets inherited by the kubedns pods and for some reason its able to forward unresolved queries to my host dnsmasq. Earlier, the kubedns inherited the 127.0.0.53 IP from node “/etc/resolv.conf” and for some reason its not able to talk to the node systemd-resolved at that IP.
+
+
+__Solution:__
+
+See https://askubuntu.com/a/974482/451114 & https://askubuntu.com/questions/952284/dns-system-is-failing-to-resolve-domain-names-occasionally#comment1589832_952284
+
+```
+sudo mv /etc/resolv.conf /etc/resolv.conf_orig
+sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
+```
+
+Now `cat /etc/resolve.conf` should inherit the correct nameserver (VirtualBox DNS server):
+
+```
+nameserver 10.0.2.3
+```
+
 ### Where did we stop? (on 22. Juni 2018)
 
 We´ve reached every step till:
@@ -206,6 +241,7 @@ curl --cacert certificates/ca.pem --key certificates/admin-key.pem --cert certif
   * [kubernetes by example][4] 
   * [Best practice by google][5] 
   * [Tutorialspoint][6] 
+  * Tutorials: https://kubernetes.io/docs/tutorials/kubernetes-basics/explore/explore-intro/
 
 
 [0]: https://kubernetes.io/docs/reference/kubectl/cheatsheet/

diff --git a/k8s-external-access.yml b/k8s-external-access.yml
@@ -1,7 +1,6 @@
 - hosts: master-0
   become: yes
   roles:
-    - { role: kubedns, tags: kubedns }
     - { role: dashboard, tags: dashboard }
 
 - hosts: external

diff --git a/k8s-worker-nodes.yml b/k8s-worker-nodes.yml
@@ -1,9 +1,11 @@
 - hosts: worker
   become: yes
   roles:
+    - { role: fix-ubuntu-dns, tags: fixubuntu }
     - { role: k8s-worker, tags: k8s-worker }
 
 - hosts: master-0
   become: yes
   roles:
     - { role: verify-worker, tags: verify }
+    - { role: kubedns, tags: kubedns }
diff --git a/roles/fix-ubuntu-dns/tasks/main.yml b/roles/fix-ubuntu-dns/tasks/main.yml
@@ -0,0 +1,11 @@
+# see https://stackoverflow.com/a/52036125/4964553
+# this is needed later for a working kube-dns DNS resolution
+- name: Remove original /etc/resolv.conf
+  shell: mv /etc/resolv.conf /etc/resolv.conf_orig
+
+- name: Link /run/systemd/resolve/resolv.conf as new /etc/resolv.conf
+  file:
+    src: /run/systemd/resolve/resolv.conf
+    dest: /etc/resolv.conf
+    state: link
+
diff --git a/roles/k8s-master/templates/kube-controller-manager.service.j2 b/roles/k8s-master/templates/kube-controller-manager.service.j2
@@ -7,7 +7,7 @@ ExecStart=/usr/local/bin/kube-controller-manager \
   --address=0.0.0.0 \
   --cluster-cidr={{ kubernetes_cluster_cidr }} \
   --allocate-node-cidrs=true \
-  --cluster-name=kubernetes \
+  --cluster-name=kubernetes-the-ansible-way \
   --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \
   --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \
   --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \

diff --git a/roles/kubedns/templates/kube-dns.yaml b/roles/kubedns/templates/kube-dns.yaml
@@ -163,6 +163,8 @@ spec:
         - --server=/cluster.local/127.0.0.1#10053
         - --server=/in-addr.arpa/127.0.0.1#10053
         - --server=/ip6.arpa/127.0.0.1#10053
+        # see https://github.com/eBayClassifiedsGroup/PanteraS/issues/166#issuecomment-169779226
+        - --dns-forward-max=300
         ports:
         - containerPort: 53
           name: dns