• 1. confd的部署
  • 2. confd的配置
    • 2.1. confd.toml
    • 2.2. 创建confdir
      • 2.2.1. Template Resources
      • 2.2.2. Template
  • 3. 创建后端存储的配置数据
  • 4. 启动confd的服务
  • 5. 查看生成的配置文件
  • 6. confd动态更新twemproxy
    • 6.1. twemproxy.toml
    • 6.2. twemproxy.tmpl
    • 6.3. etcd中的配置格式
    • 6.4. 生成twemproxy配置文件
  • 7. confd的命令

    confd的源码参考：https://github.com/kelseyhightower/confd

    1. confd的部署

    以下Linux系统为例。

    下载confd的二进制文件，下载地址为：https://github.com/kelseyhightower/confd/releases。例如：

    1. # Download the binary
    2. wget https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64
    4. # 重命名二进制文件，并移动到PATH的目录下
    5. mv confd-0.16.0-linux-amd64 /usr/local/bin/confd
    6. chmod +x /usr/local/bin/confd
    8. # 验证是否安装成功
    9. confd --help

    2. confd的配置

    Confd通过读取后端存储的配置信息来动态更新对应的配置文件，对应的后端存储可以是etcd，redis等，其中etcd的v3版本对应的存储后端为etcdv3。

    2.1. confd.toml

    confd.toml为confd服务本身的配置文件，主要记录了使用的存储后端、协议、confdir等参数。

    示例：

    • 存储后端etcdv3：
    1. backend = "etcdv3"
    2. confdir = "/etc/confd"
    3. log-level = "debug"
    4. interval = 5
    5. nodes = [
    6.   "http://192.168.10.4:12379",
    7. ]
    8. scheme = "http"
    9. watch = true

    其中watch参数表示实时监听后端存储的变化，如有变化则更新confd管理的配置。

    • 存储后端为redis
    1. backend = "redis"
    2. confdir = "/etc/confd"
    3. log-level = "debug"
    4. interval = 1  # 间隔 1 秒同步一次配置文件
    5. nodes = [
    6.   "127.0.0.1:6379",
    7. ]
    8. scheme = "http"
    9. client_key = "123456"  # redis的密码，不是 password 参数
    10. #watch = true

    如果没有启动watch参数，则会依据interval参数定期去redis存储后端拿取数据，并比较与当前配置数据是否有变化（主要比较md5值），如果有变化则更新配置，没有变化则定期再去拿取数据，以此循环。

    如果启动了watch参数，则修改redis存储数据的同时，还要执行publish的操作，促使confd去触发比较配置并更新配置的操作。

    publish的命令格式如下:

    1. publish __keyspace@0__:{prefix}/{key} set(or del)

    2.2. 创建confdir

    confdir底下包含两个目录:

    • conf.d:confd的配置文件，主要包含配置的生成逻辑，例如模板源，后端存储对应的keys，命令执行等。
    • templates:配置模板Template，即基于不同组件的配置，修改为符合 Golang text templates的模板文件。
    1. sudo mkdir -p /etc/confd/{conf.d,templates}

    2.2.1. Template Resources

    模板源配置文件是TOML格式的文件，主要包含配置的生成逻辑，例如模板源，后端存储对应的keys，命令执行等。默认目录在/etc/confd/conf.d。

    参数说明：

    必要参数

    • dest (string) - The target file.
    • keys (array of strings) - An array of keys.
    • src (string) - The relative path of a configuration template.

    可选参数

    • gid (int) - The gid that should own the file. Defaults to the effective gid.
    • mode (string) - The permission mode of the file.
    • uid (int) - The uid that should own the file. Defaults to the effective uid.
    • reload_cmd (string) - The command to reload config.
    • check_cmd (string) - The command to check config. Use {{src}} to reference the rendered source template.
    • prefix (string) - The string to prefix to keys.

    例子

    例如：/etc/confd/conf.d/myapp-nginx.toml

    1. [template]
    2. prefix = "/myapp"
    3. src = "nginx.tmpl"
    4. dest = "/tmp/myapp.conf"
    5. owner = "nginx"
    6. mode = "0644"
    7. keys = [
    8.   "/services/web"
    9. ]
    10. check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
    11. reload_cmd = "/usr/sbin/service nginx reload"

    2.2.2. Template

    Template定义了单一应用配置的模板，默认存储在/etc/confd/templates目录下，模板文件符合Go的text/template格式。

    模板文件常用函数有base，get，gets，lsdir，json等。具体可参考https://github.com/kelseyhightower/confd/blob/master/docs/templates.md。

    例子：

    /etc/confd/templates/nginx.tmpl

    1. {{range $dir := lsdir "/services/web"}}
    2. upstream {{base $dir}} {
    3.     {{$custdir := printf "/services/web/%s/*" $dir}}{{range gets $custdir}}
    4.     server {{$data := json .Value}}{{$data.IP}}:80;
    5.     {{end}}
    6. }
    8. server {
    9.     server_name {{base $dir}}.example.com;
    10.     location / {
    11.         proxy_pass {{base $dir}};
    12.     }
    13. }
    14. {{end}}

    3. 创建后端存储的配置数据

    以etcdv3存储为例，在etcd中创建以下数据。

    1. etcdctl --endpoints=$endpoints put /services/web/cust1/2 '{"IP": "10.0.0.2"}'
    2. etcdctl --endpoints=$endpoints put /services/web/cust2/2 '{"IP": "10.0.0.4"}'
    3. etcdctl --endpoints=$endpoints put /services/web/cust2/1 '{"IP": "10.0.0.3"}'
    4. etcdctl --endpoints=$endpoints put /services/web/cust1/1 '{"IP": "10.0.0.1"}'

    4. 启动confd的服务

    confd支持以daemon或者onetime两种模式运行，当以daemon模式运行时，confd会监听后端存储的配置变化，并根据配置模板动态生成目标配置文件。

    confd可以使用-config-file参数来指定confd的配置文件，而将其他参数写在配置文件中。

    1. /usr/local/bin/confd -config-file /etc/confd/conf/confd.toml

    如果以daemon模式运行，在命令后面添加&符号，例如：

    1. confd -watch -backend etcdv3 -node http://172.16.5.4:12379 &

    以下以onetime模式运行为例。其中对应的后端存储类型是etcdv3。

    1. # 执行命令
    2. confd -onetime -backend etcdv3 -node http://172.16.5.4:12379
    4. # output
    5. 2018-05-11T18:04:59+08:00 k8s-dbg-master-1 confd[35808]: INFO Backend set to etcdv3
    6. 2018-05-11T18:04:59+08:00 k8s-dbg-master-1 confd[35808]: INFO Starting confd
    7. 2018-05-11T18:04:59+08:00 k8s-dbg-master-1 confd[35808]: INFO Backend source(s) set to http://172.16.5.4:12379
    8. 2018-05-11T18:04:59+08:00 k8s-dbg-master-1 confd[35808]: INFO /root/myapp/twemproxy/conf/twemproxy.conf has md5sum 6f0f43abede612c75cb840a4840fbea3 should be 32f48664266e3fd6b56ee73a314ee272
    9. 2018-05-11T18:04:59+08:00 k8s-dbg-master-1 confd[35808]: INFO Target config /root/myapp/twemproxy/conf/twemproxy.conf out of sync
    10. 2018-05-11T18:04:59+08:00 k8s-dbg-master-1 confd[35808]: INFO Target config /root/myapp/twemproxy/conf/twemproxy.conf has been updated

    5. 查看生成的配置文件

    在/etc/confd/conf.d/myapp-nginx.toml中定义的配置文件的生成路径为/tmp/myapp.conf。

    1. [root@k8s-dbg-master-1 dest]# cat myapp.conf
    2. upstream cust1 {
    3.     server 10.0.0.1:80;
    4.     server 10.0.0.2:80;
    5. }
    7. server {
    8.     server_name cust1.example.com;
    9.     location / {
    10.         proxy_pass cust1;
    11.     }
    12. }
    14. upstream cust2 {
    15.     server 10.0.0.3:80;
    16.     server 10.0.0.4:80;
    17. }
    19. server {
    20.     server_name cust2.example.com;
    21.     location / {
    22.         proxy_pass cust2;
    23.     }
    24. }

    6. confd动态更新twemproxy

    6.1. twemproxy.toml

    confd的模板源文件配置：/etc/confd/conf.d/twemproxy.toml

    1. [template]
    2. src = "twemproxy.tmpl"
    3. dest = "/root/myapp/twemproxy/conf/twemproxy.conf"
    4. keys = [
    5.   "/twemproxy/pool"
    6. ]
    7. check_cmd = "/usr/local/bin/nutcracker -t -c /root/myapp/twemproxy/conf/twemproxy.conf"
    8. reload_cmd = "bash /root/myapp/twemproxy/reload.sh"

    6.2. twemproxy.tmpl

    模板文件：/etc/confd/templates/twemproxy.tmpl

    1. global:
    2.   worker_processes: 4         # 并发进程数, 如果为0, 这 fallback 回原来的单进程模型(不支持 config reload!)
    3.   user: nobody                # worker 进程的用户, 默认 nobody. 只要主进程是 root 用户启动才生效.
    4.   group: nobody               # worker 进程的用户组
    5.   worker_shutdown_timeout: 30 # 单位为秒. 用于 reload 过程中在改时间段之后强制退出旧的 worker 进程.
    7. pools: {{range gets "/twemproxy/pool/*"}}
    8.   {{base .Key}}: {{$pool := json .Value}}
    9.     listen: {{$pool.ListenAddr.IP}}:{{$pool.ListenAddr.Port}}
    10.     hash: fnv1a_64 # 选择实例的 hash 规则
    11.     distribution: ketama
    12.     auto_eject_hosts: true # server 有问题是否剔除
    13.     redis: true # 是否为 Redis 协议
    14.     {{if $pool.Password}}redis_auth: {{$pool.Password}}{{end}}
    15.     server_retry_timeout: 5000 # 被剔除多长时间后会重试
    16.     server_connections: 25 # NOTE: server 连接池的大小, 默认为 1, 建议调整
    17.     server_failure_limit: 5 # 失败多少次后暂时剔除
    18.     timeout: 1000 # Server 超时时间, 1 sec
    19.     backlog: 1024 # 连接队列大小
    20.     preconnect: true # 预连接大小
    21.     servers:{{range $server := $pool.Servers}}
    22.      - {{$server.IP}}:{{$server.Port}}:1 {{if $server.Master}}master{{end}}
    23.     {{end}}
    24. {{end}}

    6.3. etcd中的配置格式

    etcd中的配置通过一个map来定义为完整的配置内容。其中key是twemproxy中pool的名称，value是pool的所有内容。

    配置对应go结构体如下：

    1. type Pool struct{
    2.     ListenAddr  ListenAddr `json:"ListenAddr,omitempty"`
    3.     Servers []Server `json:"Servers,omitempty"`
    4.     Password string `json:"Password,omitempty"`
    5. }
    7. type ListenAddr struct {
    8.     IP string `json:"IP,omitempty"`
    9.     Port string `json:"Port,omitempty"`
    10. }
    12. type Server struct {
    13.     IP string `json:"IP,omitempty"`
    14.     Port string `json:"Port,omitempty"`
    15.     Master bool `json:"Master,omitempty"`
    16. }

    配置对应JSON格式如下：

    1. {
    2.     "ListenAddr": {
    3.         "IP": "192.168.5.7",
    4.         "Port": "22225"
    5.     },
    6.     "Servers": [
    7.         {
    8.             "IP": "10.233.116.168",
    9.             "Port": "6379",
    10.             "Master": true
    11.         },
    12.         {
    13.             "IP": "10.233.110.207",
    14.             "Port": "6379",
    15.             "Master": false
    16.         }
    17.     ],
    18.     "Password": "987654"
    19. }

    6.4. 生成twemproxy配置文件

    1. global:
    2.   worker_processes: 4         # 并发进程数, 如果为0, 这 fallback 回原来的单进程模型(不支持 config reload!)
    3.   user: nobody                # worker 进程的用户, 默认 nobody. 只要主进程是 root 用户启动才生效.
    4.   group: nobody               # worker 进程的用户组
    5.   worker_shutdown_timeout: 30 # 单位为秒. 用于 reload 过程中在改时间段之后强制退出旧的 worker 进程.
    7. pools:
    8.   redis1:
    9.     listen: 192.168.5.7:22223
    10.     hash: fnv1a_64 # 选择实例的 hash 规则
    11.     distribution: ketama
    12.     auto_eject_hosts: true # server 有问题是否剔除
    13.     redis: true # 是否为 Redis 协议
    14.     redis_auth: 987654
    15.     server_retry_timeout: 5000 # 被剔除多长时间后会重试
    16.     server_connections: 25 # NOTE: server 连接池的大小, 默认为 1, 建议调整
    17.     server_failure_limit: 5 # 失败多少次后暂时剔除
    18.     timeout: 1000 # Server 超时时间, 1 sec
    19.     backlog: 1024 # 连接队列大小
    20.     preconnect: true # 预连接大小
    21.     servers:
    22.      - 10.233.116.169:6379:1
    25.   redis2:
    26.     listen: 192.168.5.7:22224
    27.     hash: fnv1a_64 # 选择实例的 hash 规则
    28.     distribution: ketama
    29.     auto_eject_hosts: true # server 有问题是否剔除
    30.     redis: true # 是否为 Redis 协议
    31.     redis_auth: 987654
    32.     server_retry_timeout: 5000 # 被剔除多长时间后会重试
    33.     server_connections: 25 # NOTE: server 连接池的大小, 默认为 1, 建议调整
    34.     server_failure_limit: 5 # 失败多少次后暂时剔除
    35.     timeout: 1000 # Server 超时时间, 1 sec
    36.     backlog: 1024 # 连接队列大小
    37.     preconnect: true # 预连接大小
    38.     servers:
    39.      - 10.233.110.223:6379:1 master
    41.      - 10.233.111.21:6379:1

    7. confd的命令

    1. $ confd --help
    2. Usage of confd:
    3.   -app-id string
    4.         Vault app-id to use with the app-id backend (only used with -backend=vault and auth-type=app-id)
    5.   -auth-token string
    6.         Auth bearer token to use
    7.   -auth-type string
    8.         Vault auth backend type to use (only used with -backend=vault)
    9.   -backend string
    10.         backend to use (default "etcd")
    11.   -basic-auth
    12.         Use Basic Auth to authenticate (only used with -backend=consul and -backend=etcd)
    13.   -client-ca-keys string
    14.         client ca keys
    15.   -client-cert string
    16.         the client cert
    17.   -client-key string
    18.         the client key
    19.   -confdir string
    20.         confd conf directory (default "/etc/confd")
    21.   -config-file string
    22.         the confd config file (default "/etc/confd/confd.toml")
    23.   -file value
    24.         the YAML file to watch for changes (only used with -backend=file)
    25.   -filter string
    26.         files filter (only used with -backend=file) (default "*")
    27.   -interval int
    28.         backend polling interval (default 600)
    29.   -keep-stage-file
    30.         keep staged files
    31.   -log-level string
    32.         level which confd should log messages
    33.   -node value
    34.         list of backend nodes
    35.   -noop
    36.         only show pending changes
    37.   -onetime
    38.         run once and exit
    39.   -password string
    40.         the password to authenticate with (only used with vault and etcd backends)
    41.   -path string
    42.         Vault mount path of the auth method (only used with -backend=vault)
    43.   -prefix string
    44.         key path prefix
    45.   -role-id string
    46.         Vault role-id to use with the AppRole, Kubernetes backends (only used with -backend=vault and either auth-type=app-role or auth-type=kubernetes)
    47.   -scheme string
    48.         the backend URI scheme for nodes retrieved from DNS SRV records (http or https) (default "http")
    49.   -secret-id string
    50.         Vault secret-id to use with the AppRole backend (only used with -backend=vault and auth-type=app-role)
    51.   -secret-keyring string
    52.         path to armored PGP secret keyring (for use with crypt functions)
    53.   -separator string
    54.         the separator to replace '/' with when looking up keys in the backend, prefixed '/' will also be removed (only used with -backend=redis)
    55.   -srv-domain string
    56.         the name of the resource record
    57.   -srv-record string
    58.         the SRV record to search for backends nodes. Example: _etcd-client._tcp.example.com
    59.   -sync-only
    60.         sync without check_cmd and reload_cmd
    61.   -table string
    62.         the name of the DynamoDB table (only used with -backend=dynamodb)
    63.   -user-id string
    64.         Vault user-id to use with the app-id backend (only used with -backend=value and auth-type=app-id)
    65.   -username string
    66.         the username to authenticate as (only used with vault and etcd backends)
    67.   -version
    68.         print version and exit
    69.   -watch
    70.         enable watch support

    参考文章：

    https://github.com/kelseyhightower/confd/blob/master/docs/installation.md

    https://github.com/kelseyhightower/confd/blob/master/docs/quick-start-guide.md

    https://github.com/kelseyhightower/confd/blob/master/docs/template-resources.md

    https://github.com/kelseyhightower/confd/blob/master/docs/templates.md