当前位置：移动技术网 > 网络运营>服务器>nginx > nginx-ingress-controller日志持久化方案的解决

nginx-ingress-controller日志持久化方案的解决

2020年05月13日 | 移动技术网网络运营 | 我要评论

阮桥板鸭,抗日火神,湖南卫视跨年演唱会2018节目单

最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化，刚好工作上遇到该问题，整理一个方案，仅供参考。

nginx-ingress-controller的日志

nginx-ingress-controller的日志包括三个部分：

controller日志：输出到stdout，通过启动参数中的–log_dir可已配置输出到文件，重定向到文件后会自动轮转，但不会自动清理
accesslog：输出到stdout，通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理
errorlog：输出到stderr，配置方式与accesslog类似。

给controller日志落盘

给nginx-ingress-controller挂一个hostpath： /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ，
给nginx-ingress-controller配置log-dir和logtostderr参数，将日志重定向到/var/log/nginx_ingress_controller/中。

controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的，会进行日志滚动，所以我们通过脚本定时清理一定时间之前的日志文件即可。

给nginx日志落盘

修改configmap： nginx-configuration。配置accesslog和errorlog的输出路径，替换默认的stdout和stderr。输出路径我们可以与controller一致，便于查找。

accesslog和errorlog都只有一个日志文件，我们可以使用logrotate进行日志轮转，将输出到宿主机上的日志进行轮转和清理。配置如：

$ cat /etc/logrotate.d/nginx.log
/data/log/nginx_ingress_controller/access.log {
  su root list
  rotate 7
  daily
  maxsize 50m
  copytruncate
  missingok
  create 0644 www-data root
}

官方提供的模板中，nginx-ingress-controller默认都是以33这个用户登录启动容器的，因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -r 33:33 /data/log/nginx_ingress_controller.

自动化ops

nginx日志落盘中，第2、3两点均需要人工运维，有什么解决办法吗？

问题的关键是：有什么办法可以在nginx-ingress-controller容器启动之前加一个hook，将宿主机的指定目录执行chown呢？

可以用initcontainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。利用这一k8s特性，我们开发一个docker image，里面只执行如下脚本：

#!/bin/bash
logdir=$log_dir
userid=$user_id
echo "try to set dir: $logdir 's group as $userid"
chown -r $userid:$userid $logdir

脚本读取一些环境变量，确认需要修改哪个目录，改成怎样的user group。

将脚本打包成dockerimage，放在nginx-ingress-controller的deploy yaml中，作为initcontainers。注意要对该initcontainer配置环境变量和volumemount.

再说第二点，我们注意到nginx-ingress-controller的基础镜像中就自带了logrotate，那么问题就简单了，我们将写好的logrotate配置文件以configmap的形式挂载到容器中就可以了。

一个deploy yaml如下：

---
apiversion: v1
kind: service
metadata:
 name: ingress-nginx
 namespace: kube-system
spec:
 type: clusterip
 ports:
 - name: http
  port: 80
  targetport: 80
  protocol: tcp
 - name: https
  port: 443
  targetport: 443
  protocol: tcp
 selector:
  app: ingress-nginx
---
apiversion: v1
kind: service
metadata:
 name: default-http-backend
 namespace: kube-system
 labels:
  app: default-http-backend
spec:
 ports:
 - port: 80
  targetport: 8080
 selector:
  app: default-http-backend
---
apiversion: extensions/v1beta1
kind: ingress
metadata:
 name: default
 namespace: kube-system
spec:
 backend:
  servicename: default-http-backend
  serviceport: 80
---
kind: configmap
apiversion: v1
metadata:
 name: nginx-configuration
 namespace: kube-system
 labels:
  app: ingress-nginx
data:
 use-forwarded-headers: "true"
 # 此处配置nginx日志的重定向目标
 access-log-path: /var/log/nginx_ingress_controller/access.log
 error-log-path: /var/log/nginx_ingress_controller/error.log

---

# 创建一个configmap，配置nginx日志的轮转策略，对应的是nginx日志在容器内的日志文件
apiversion: v1
data:
 nginx.log: |
  {{ user_nginx_log.host_path }}/access.log {
    rotate {{ user_nginx_log.rotate_count }}
    daily
    maxsize {{ user_nginx_log.rotate_size }}
    minsize 10m
    copytruncate
    missingok
    create 0644 root root
  }
  {{ user_nginx_log.host_path }}/error.log {
    rotate {{ user_nginx_log.rotate_count }}
    daily
    maxsize {{ user_nginx_log.rotate_size }}
    minsize 10m
    copytruncate
    missingok
    create 0644 root root
  }
kind: configmap
metadata:
 name: nginx-ingress-logrotate
 namespace: kube-system
---

kind: configmap
apiversion: v1
metadata:
 name: tcp-services
 namespace: kube-system
---
kind: configmap
apiversion: v1
metadata:
 name: udp-services
 namespace: kube-system
---
apiversion: v1
kind: serviceaccount
metadata:
 name: nginx-ingress-serviceaccount
 namespace: kube-system
---
apiversion: rbac.authorization.k8s.io/v1beta1
kind: clusterrole
metadata:
 name: nginx-ingress-clusterrole
rules:
 - apigroups:
   - ""
  resources:
   - configmaps
   - endpoints
   - nodes
   - pods
   - secrets
  verbs:
   - list
   - watch
 - apigroups:
   - ""
  resources:
   - nodes
  verbs:
   - get
 - apigroups:
   - ""
  resources:
   - services
  verbs:
   - get
   - list
   - watch
 - apigroups:
   - "extensions"
  resources:
   - ingresses
  verbs:
   - get
   - list
   - watch
 - apigroups:
   - ""
  resources:
    - events
  verbs:
    - create
    - patch
 - apigroups:
   - "extensions"
  resources:
   - ingresses/status
  verbs:
   - update
---
apiversion: rbac.authorization.k8s.io/v1beta1
kind: role
metadata:
 name: nginx-ingress-role
 namespace: kube-system
rules:
 - apigroups:
   - ""
  resources:
   - configmaps
   - pods
   - secrets
   - namespaces
  verbs:
   - get
 - apigroups:
   - ""
  resources:
   - configmaps
  resourcenames:
   # defaults to "<election-id>-<ingress-class>"
   # here: "<ingress-controller-leader>-<nginx>"
   # this has to be adapted if you change either parameter
   # when launching the nginx-ingress-controller.
   - "ingress-controller-leader-nginx"
  verbs:
   - get
   - update
 - apigroups:
   - ""
  resources:
   - configmaps
  verbs:
   - create
 - apigroups:
   - ""
  resources:
   - endpoints
  verbs:
   - get
---
apiversion: rbac.authorization.k8s.io/v1beta1
kind: rolebinding
metadata:
 name: nginx-ingress-role-nisa-binding
 namespace: kube-system
roleref:
 apigroup: rbac.authorization.k8s.io
 kind: role
 name: nginx-ingress-role
subjects:
 - kind: serviceaccount
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiversion: rbac.authorization.k8s.io/v1beta1
kind: clusterrolebinding
metadata:
 name: nginx-ingress-clusterrole-nisa-binding
roleref:
 apigroup: rbac.authorization.k8s.io
 kind: clusterrole
 name: nginx-ingress-clusterrole
subjects:
 - kind: serviceaccount
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiversion: apps/v1
kind: daemonset
metadata:
 name: ingress-nginx
 namespace: kube-system
spec:
 selector:
  matchlabels:
   app: ingress-nginx
 template:
  metadata:
   labels:
    app: ingress-nginx
   annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
  spec:
   serviceaccountname: nginx-ingress-serviceaccount
   tolerations:
   - key: dedicated
    value: ingress-nginx
    effect: noschedule
   affinity:
    nodeaffinity:
     requiredduringschedulingignoredduringexecution:
      nodeselectorterms:
      - matchexpressions:
       - key: "system/ingress"
        operator: in
        values:
        - "true"
   dnspolicy: clusterfirstwithhostnet
   hostnetwork: true
   # 配置initcontainer，确保在nginx-ingress-controller容器启动前将日志目录的权限配置好
   initcontainers:
   - name: adddirperm
    image: "{{ image_registry.addr }}/{{ image.adddirperm }}"
    env:
    - name: log_dir
     value: /var/log/nginx_ingress_controller
    - name: user_id
      value: "33"
    volumemounts:
    - name: logdir
     mountpath: /var/log/nginx_ingress_controller
   containers:
   - name: nginx-ingress-controller
    image: "{{ image_registry.addr }}/{{ image.ingress }}"
    imagepullpolicy: ifnotpresent
    args:
    - /nginx-ingress-controller
    - --default-backend-service=$(pod_namespace)/default-http-backend
    - --configmap=$(pod_namespace)/nginx-configuration
    - --tcp-services-configmap=$(pod_namespace)/tcp-services
    - --udp-services-configmap=$(pod_namespace)/udp-services
    - --publish-service=$(pod_namespace)/ingress-nginx
    - --annotations-prefix=nginx.ingress.kubernetes.io
    
    # 设置controller日志的输出路径和方式
    - --log_dir=/var/log/nginx_ingress_controller
    - --logtostderr=false
    securitycontext:
     capabilities:
       drop:
       - all
       add:
       - net_bind_service
     # www-data -> 33
     runasuser: 33
    env:
     - name: pod_name
      valuefrom:
       fieldref:
        fieldpath: metadata.name
     - name: pod_namespace
      valuefrom:
       fieldref:
        fieldpath: metadata.namespace
    ports:
    - name: http
     containerport: 80
    - name: https
     containerport: 443
    resources:
     requests:
      cpu: 100m
      memory: 256mi
    livenessprobe:
     failurethreshold: 3
     httpget:
      path: /healthz
      port: 10254
      scheme: http
     initialdelayseconds: 10
     periodseconds: 10
     successthreshold: 1
     timeoutseconds: 1
    readinessprobe:
     failurethreshold: 3
     httpget:
      path: /healthz
      port: 10254
      scheme: http
     periodseconds: 10
     successthreshold: 1
     timeoutseconds: 1
    volumemounts:
    # 配置挂载容器中控制器组件和nginx的日志输出路径
    - name: logdir
     mountpath: /var/log/nginx_ingress_controller
    # 配置nginx日志的logrotate配置挂载路径
    - name: logrotateconf
     mountpath: /etc/logrotate.d/nginx.log
     subpath: nginx.log
   volumes:
   # 控制器组件和nginx的日志输出路径为宿主机的hostpath
   - name: logdir
    hostpath:
     path: {{ user_nginx_log.host_path }}
     type: ""
   # nginx日志的轮转配置文件来自于configmap
   - name: logrotateconf
    configmap:
     name: nginx-ingress-logrotate
     items:
     - key: nginx.log
      path: nginx.log
---

apiversion: apps/v1
kind: daemonset
metadata:
 name: default-http-backend
 namespace: kube-system
 labels:
  app: default-http-backend
spec:
 selector:
  matchlabels:
   app: default-http-backend
 template:
  metadata:
   labels:
    app: default-http-backend
  spec:
   terminationgraceperiodseconds: 60
   tolerations:
   - key: dedicated
    value: ingress-nginx
    effect: noschedule
   affinity:
    nodeaffinity:
     requiredduringschedulingignoredduringexecution:
      nodeselectorterms:
      - matchexpressions:
       - key: "system/ingress"
        operator: in
        values:
        - "true"
   containers:
   - name: default-http-backend
    # any image is permissible as long as:
    # 1. it serves a 404 page at /
    # 2. it serves 200 on a /healthz endpoint
    image: "{{ image_registry.addr }}/{{ image.http_backend }}"
    imagepullpolicy: ifnotpresent
    livenessprobe:
     httpget:
      path: /healthz
      port: 8080
      scheme: http
     initialdelayseconds: 30
     timeoutseconds: 5
    ports:
    - containerport: 8080
    resources:
     limits:
      cpu: 10m
      memory: 20mi
     requests:
      cpu: 10m
      memory: 20mi
---

最后，有的人建议将initcontainer去掉，改为基于原有的nginx-ingress-controller镜像加一层layer，将配置路径权限的脚本放在该层执行。个人认为这种方法既不美观，也不方便。唯一的好处仅在于deploy yaml仍然简洁（但少不了volumemount之类的配置）。不过还是看个人使用感受吧~

到此这篇关于nginx-ingress-controller日志持久化方案的解决的文章就介绍到这了,更多相关nginx ingress controller日志持久化内容请搜索移动技术网以前的文章或继续浏览下面的相关文章希望大家以后多多支持移动技术网！

您可能感兴趣的文章:

如对本文有疑问，请在下面进行留言讨论，广大热心网友会与你互动！！点击进行留言回复

详解nginx请求头数据读取流程

在上一篇文章中，我们讲解了nginx是如何读取请求行的数据，并且解析请求行的。本文我们则主要讲解nginx是如何读取客户端发送来的请求头的数据，并且解析这些数据... [阅读全文]
Nginx 转发匹配规则的实现

一、正则表达式匹配 ~ 为区分大小写匹配 ~* 为不区分大小写匹配 !~ 和 !~* 分别为区分大小写不匹配及不区分大小写不匹配二、文件及目录匹配 -f... [阅读全文]
Nginx与Lua灰度发布的实现

安装 memcachedyum install -y memcached#启动memcached服务，端口11211，以nobody身份启动，-d后台模式运行m... [阅读全文]
nginx-ingress-controller日志持久化方案的解决

最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化，刚好工作上遇到该问题，整理一个方案，仅供参考。ngi... [阅读全文]
使用Nginx实现301跳转至https的根域名示例代码

基于 seo 和安全性的考量，需要进行 301 跳转，以下使用 nginx 作通用处理实现结果需要将以下地址都统一跳转到 https 的根域名 https://... [阅读全文]
Keepalived实现Nginx负载均衡高可用的示例代码

第一章：keepalived介绍vrrp协议目的就是为了解决静态路由单点故障问题的第二章： keepalived工作原理2.1 作为系统网络服务的高可用功能(f... [阅读全文]
Nginx session丢失问题处理解决方法

在用nginx的反向代理tomcat的路径中，可能会出现session丢失问题。每发送一次请求 jessionid 都会发生改变，说明上一次形成的session... [阅读全文]
Nginx 负载均衡算法及故障转移解析

概述 nginx负载均衡提供上游服务器(真实业务逻辑访问的服务器),负载均衡、故障转移、失败重试、容错、健康检查等。当上游服务器(真实... [阅读全文]
Nginx实现负载均衡的方法总结

要理解负载均衡，必须先搞清楚正向代理和反向代理。注：正向代理，代理的是用户。反向代理，代理的是服务器什么是负载均... [阅读全文]
详解nginx中location、rewrite用法总结

一、location用法总结 location可以把不同方式的请求，定位到不同的处理方式上. 1.location的用法 location... [阅读全文]

网友评论


验证码：

nginx-ingress-controller日志持久化方案的解决

2020年05月13日 | 移动技术网网络运营 | 我要评论

您可能感兴趣的文章:

相关文章:

网友评论