Nginx和Haproxy端口复用配置和容器镜像构建

梅钟园

背景：
  最近网站打算使用SSL证书，曾经为 RH124、RH134、RH294、RH358、CL260、DO374 等环境准备的联网更新脚本仅支持 HTTP 协议，重构全部环境不太现实（当初未考虑SSL）。
  现在为兼容 HTTP 和 HTTPS，服务提供端决定使用端口复用以同时支持 HTTP 和 HTTPS，故而有此处内容分享。
  此处仅分享核心配置，是经过验证能够使用的配置，具体细节不作展示，有需要可以随时与我交流。

一、端口复用
  简单来说，同一端口根据请求协议（四层或七层）转发至不同后端服务，以实现根据不同协议访问同一端口时，能够到达不同后端服务。例如 18080/tcp 端口同时支持 http 和 https 访问。
  使用典型的 Nginx 和 Haproxy 均可以实现。

二、构建镜像
  此处使用Docker将所有业务组件均构建为容器，便于后续升级。此处默认读者已经熟悉容器技术、Dockerfile、docker-compose、Compose Files，下列仅列出核心配置。

base images:
  从官方镜像构建自定义基础镜像，必要时可以从其他地方编译安装以植入更多模块（如Nginx模块）。
  nginx-base:1.27.2 from nginx:1.27.2
  haproxy-base:3.0.6 from haproxy:3.0.6
示例：nginx-base:1.27.2 Dockerfile 如下；根据自己需要，改时区，安装小工具等。
可以再创建一个

1. FROM nginx:1.27.2
   
2. ENV TZ="Asia/Shanghai"
   
3. RUN mkdir -p /etc/nginx/certs.d ; \
   
4.   rm -rf /etc/localtime ; \
   
5.   ln -sv /usr/share/zoneinfo/Asia/shanghai /etc/localtime ; \
   
6.   apt update && apt install -y iproute2 lsof telnet traceroute tree curl vim && \
   
7.   apt-get clean && rm -rf /var/lib/apt/lists/*
   
8. CMD ["nginx","-g","daemon off;"]

复制代码

None 镜像清理

1. docker rmi $(docker images -f 'dangling=true' -q)

复制代码

apps images:
  从自定义的基础镜像再次构建应用镜像。
  nginx-webapp:n1.27.2-v1.0 from nginx-base:1.27.2
  haproxy-balances:h3.0.6-v1.0 from haproxy-balances:3.0.6
  nginx-balances:n1.27.2-v1.0 from nginx-base:1.27.2

nginx-webapp:n1.27.2-v1.0 from nginx-base:1.27.2
nginx-webapp:n1.27.2-v1.0 镜像构建材料（建议配置文件和证书通过挂载方式使用，应用镜像只构建网站源代码即可，不要包含太多其他信息，更不要包含机密信息）
  说明：主要构建网站源代码，nginx-webapp 将同时支持 80/tcp HTTP 访问，443/tcp HTTPS 访问，也可以分成两个镜像分别支持 HTTP 和 HTTPS。
  重要配置：nginx-webapp 启用 proxy_protocol ，并且配置允许从上游代理获取源IP。这一配置的上游代理服务器必须也同样启用这一功能，如上游代理为 nginx，则必须也启用 proxy_protocol，若为 Haproxy，必须 send_proxy。
                  否则整体网站无法访问。

1. nginx-webapp/
   
2. ├── Dockerfile
   
3. ├── config-nginx
   
4. │   ├── conf.d
   
5. │   │   └── vhost.conf
   
6. │   └── nginx.conf
   
7. └── html

复制代码

nginx.conf -> 容器 /etc/nginx/nginx.conf

1. user  nginx;
   
2. worker_processes  auto;
   
3. 
   
4. error_log  /var/log/nginx/error.log notice;
   
5. pid        /var/run/nginx.pid;
   
6. 
   
7. 
   
8. events {
   
9.     worker_connections  1024;
   
10. }
    
11. 
    
12. http {
    
13.     include       /etc/nginx/mime.types;
    
14.     default_type  application/octet-stream;
    
15. 
    
16.     log_format  main  '$proxy_protocol_addr - $remote_addr - $remote_user [$time_local] "$request" '
    
17.                       '$status $body_bytes_sent "$http_referer" '
    
18.                       '"$http_user_agent" "$http_x_forwarded_for"';
    
19. 
    
20.     access_log  /var/log/nginx/access.log  main;
    
21. 
    
22.     sendfile        on;
    
23.     #tcp_nopush     on;
    
24. 
    
25.     keepalive_timeout  65;
    
26. 
    
27.     # 压缩设置
    
28.     gzip on;
    
29.     gzip_disable "msie6";
    
30.     gzip_vary on;
    
31.     gzip_proxied any;
    
32.     gzip_comp_level 6;
    
33.     gzip_buffers 16 8k;
    
34.     gzip_types text/plain text/css application/javascript application/json application/xml application/rss+xml image/svg+xml;
    
35.     map $status $status_name {
    
36.         100 "Continue";
    
37.         101 "Switching Protocols";
    
38.         102 "Processing";
    
39.         103 "Early Hints";
    
40.         200 "OK";
    
41.         201 "Created";
    
42.         202 "Accepted";
    
43.         203 "Non-Authoritative Information";
    
44.         204 "No Content";
    
45.         205 "Reset Content";
    
46.         206 "Partial Content";
    
47.         300 "Multiple Choices";
    
48.         301 "Moved Permanently";
    
49.         302 "Found";
    
50.         303 "See Other";
    
51.         304 "Not Modified";
    
52.         305 "Use Proxy";
    
53.         307 "Temporary Redirect";
    
54.         308 "Permanent Redirect";
    
55.         400 "Bad Request";
    
56.         401 "Unauthorized";
    
57.         403 "Forbidden";
    
58.         404 "Not Found";
    
59.         405 "Method Not Allowed";
    
60.         406 "Not Acceptable";
    
61.         407 "Proxy Authentication Required";
    
62.         408 "Request Timeout";
    
63.         409 "Conflict";
    
64.         410 "Gone";
    
65.         411 "Length Required";
    
66.         412 "Precondition Failed";
    
67.         413 "Payload Too Large";
    
68.         414 "URI Too Long";
    
69.         415 "Unsupported Media Type";
    
70.         416 "Range Not Satisfiable";
    
71.         417 "Expectation Failed";
    
72.         418 "I'm a teapot";
    
73.         429 "Too Many Requests";
    
74.         500 "Internal Server Error";
    
75.         501 "Not Implemented";
    
76.         502 "Bad Gateway";
    
77.         503 "Service Unavailable";
    
78.         504 "Gateway Timeout";
    
79.         505 "HTTP Version Not Supported";
    
80.         default "Unknown";
    
81.     }
    
82. 
    
83.     include /etc/nginx/conf.d/*.conf;
    
84. }

复制代码

conf.d/vhost.conf -> 容器 /etc/nginx/conf.d/vhost.conf

1. server {
   
2.     listen       80 <font color="#ff0000">proxy_protocol</font>;
   
3.     server_name nginx-webapp;
   
4.     root /usr/share/nginx/html;
   
5.     access_log /var/log/nginx/access_backend_http.log main;
   
6.     error_log /var/log/nginx/error_backend.log warn;
   
7.     # proxy_protocol docs: https://nginx.org/en/docs/stream/ngx_stream_realip_module.html
   
8. 
   
9.     # 重写策略 特殊链接同时支持 http 和 https
   
10.     location / {
    
11.         root /usr/share/nginx/html;
    
12.         index  index.html index.htm;
    
13.     }
    
14.     # 获取源 IP 配置 适用于代理负载均衡器 CDN 环境
    
15.     set_real_ip_from 172.16.0.0/16;
    
16.     set_real_ip_from 172.17.0.0/16;
    
17.     set_real_ip_from 172.18.0.0/16;
    
18.     set_real_ip_from 172.19.0.0/16;
    
19.     set_real_ip_from 127.0.0.1/32;
    
20.     #real_ip_header X-Forwarded-For;
    
21.     #上游还有代理 端口复用专用配置
    
22.     real_ip_header proxy_protocol;
    
23.     proxy_set_header X-Real-IP       $proxy_protocol_addr;
    
24.     proxy_set_header X-Forwarded-For $proxy_protocol_addr;
    
25.     real_ip_recursive on;
    
26.     #常规配置
    
27.     #real_ip_header X-Forwarded-For;
    
28.     #real_ip_recursive on;
    
29.     #proxy_set_header X-Real-IP $remote_addr;
    
30.     #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    
31. 
    
32.     # 安全头部设置
    
33.     add_header X-Frame-Options "SAMEORIGIN";
    
34.     #add_header X-XSS-Protection "1; mode=block";
    
35.     #add_header X-Content-Type-Options "nosniff";
    
36.     #add_header Referrer-Policy "strict-origin-when-cross-origin";
    
37.     #add_header Content-Security-Policy "default-src 'self'; style-src 'self';" always;
    
38.     #add_header Content-Security-Policy "default-src 'self'";
    
39.     # XSS
    
40.     add_header X-XSS-Protection "1; mode=block";
    
41.     # 点击劫持
    
42.     add_header X-Frame-Options "DENY";
    
43.     # 文件上传限制
    
44.     client_max_body_size 400M;
    
45. 
    
46.     # 请求超时配置
    
47.     client_body_timeout 10s;
    
48.     client_header_timeout 10s;
    
49.     send_timeout 10s;
    
50. 
    
51.     # 基本网络优化
    
52.     sendfile on;
    
53.     tcp_nopush on;
    
54.     tcp_nodelay on;
    
55.     keepalive_timeout 65;
    
56. 
    
57.     # 压缩设置
    
58.     gzip on;
    
59.     gzip_disable "msie6";
    
60.     gzip_vary on;
    
61.     gzip_proxied any;
    
62.     gzip_comp_level 6;
    
63.     gzip_buffers 16 8k;
    
64.     gzip_types text/plain text/css application/javascript application/json application/xml application/rss+xml image/svg+xml;
    
65. 
    
66.     # 静态文件缓存优化
    
67.     location ~* \.(jpg|jpeg|png|gif|ico|css|js|svg|woff|woff2|ttf|eot|otf|ttc|map)$ {
    
68.         root /usr/share/nginx/html;
    
69.         expires 6M;
    
70.         add_header Cache-Control "public";
    
71.         access_log off;
    
72.     }
    
73. 
    
74.     # 禁止访问 .git 目录
    
75.     location ~ /\.git {
    
76.         root /usr/share/nginx/html;
    
77.         deny all;
    
78.         return 404;
    
79.     }
    
80. 
    
81.     # 禁止访问特定文件
    
82.     location ~* \.(htaccess|git|env)$ {
    
83.         root /usr/share/nginx/html;
    
84.         deny all;
    
85.         return 404;
    
86.     }
    
87.     # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    
88.     #
    
89.     #location ~ \.php$ {
    
90.     #    proxy_pass   http://127.0.0.1;
    
91.     #}
    
92. 
    
93.     # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    
94.     #
    
95.     #location ~ \.php$ {
    
96.     #    root           html;
    
97.     #    fastcgi_pass   127.0.0.1:9000;
    
98.     #    fastcgi_index  index.php;
    
99.     #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    
100.     #    include        fastcgi_params;
     
101.     #}
     
102. 
     
103.     # deny access to .htaccess files, if Apache's document root
     
104.     # concurs with nginx's one
     
105.     #
     
106.     location ~ /\.ht {
     
107.         root /usr/share/nginx/html;
     
108.         deny  all;
     
109.     }
     
110. 
     
111.     # ERROR 404
     
112.     error_page 404 /404.html;
     
113.     location = /404.html {
     
114.         root /usr/share/nginx/html;
     
115.         internal;
     
116.         sub_filter '{{ status_code }}' $status;
     
117.         sub_filter "{{ status_name }}" $status_name;
     
118.         sub_filter_once off;
     
119.     }
     
120. 
     
121.     # ERROR 403
     
122.     error_page 403 /403.html;
     
123.     location = /403.html {
     
124.         root /usr/share/nginx/html;
     
125.         internal;
     
126.         sub_filter '{{ status_code }}' $status;
     
127.         sub_filter "{{ status_name }}" $status_name;
     
128.         sub_filter_once off;
     
129.     }
     
130. 
     
131.     # ERROR 40x
     
132.     error_page 401 402 400 405 406 407 408 409 410 411 412 413 414 415 416 417 418 429 /40x.html;
     
133.     location = /40x.html {
     
134.         root /usr/share/nginx/html;
     
135.         internal;
     
136.         sub_filter '{{ status_code }}' $status;
     
137.         sub_filter "{{ status_name }}" $status_name;
     
138.         sub_filter_once off;
     
139.     }
     
140. 
     
141.     # ERROR 50x
     
142.     error_page 500 501 502 503 504 505 /50x.html;
     
143.     location = /50x.html {
     
144.         root /usr/share/nginx/html;
     
145.         internal;
     
146.         sub_filter '{{ status_code }}' $status;
     
147.         sub_filter "{{ status_name }}" $status_name;
     
148.         sub_filter_once off;
     
149.     }
     
150. }
     
151. 
     
152. server {
     
153.     listen 443 ssl proxy_protocol;
     
154.     http2 on;
     
155.     server_name nginx-webapp;
     
156.     root /usr/share/nginx/html;
     
157.     access_log /var/log/nginx/access_backend_https.log main;
     
158.     error_log /var/log/nginx/error_backend_https.log warn;
     
159.     # proxy_protocol docs: https://nginx.org/en/docs/stream/ngx_stream_realip_module.html
     
160. 
     
161.     location / {
     
162.         root /usr/share/nginx/html;
     
163.         index  index.html index.htm;
     
164.     }
     
165. 
     
166.     # SSL 证书配置
     
167.     ssl_certificate /etc/nginx/certs.d/server.pem;
     
168.     ssl_certificate_key /etc/nginx/certs.d/server.key;
     
169. 
     
170.     # SSL 其他配置
     
171.     #ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
     
172.     #ssl_ciphers  HIGH:!aNULL:!MD5;
     
173.     ssl_protocols TLSv1.2 TLSv1.3;
     
174.     ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
     
175.     ssl_prefer_server_ciphers on;
     
176.     ssl_session_cache shared:SSL:10m;
     
177.     ssl_session_timeout 1h;
     
178.     ssl_stapling off;
     
179.     ssl_stapling_verify off;
     
180.     # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
     
181. 
     
182.     # 获取源 IP 配置 适用于代理负载均衡器 CDN 环境
     
183.     set_real_ip_from 172.16.0.0/16;
     
184.     set_real_ip_from 172.17.0.0/16;
     
185.     set_real_ip_from 172.18.0.0/16;
     
186.     set_real_ip_from 172.19.0.0/16;
     
187.     set_real_ip_from 127.0.0.1/32;
     
188.     #real_ip_header X-Forwarded-For;
     
189.     #上游还有代理 端口复用专用配置
     
190.     real_ip_header proxy_protocol;
     
191.     proxy_set_header X-Real-IP       $proxy_protocol_addr;
     
192.     proxy_set_header X-Forwarded-For $proxy_protocol_addr;
     
193.     real_ip_recursive on;
     
194.     #常规配置
     
195.     #real_ip_header X-Forwarded-For;
     
196.     #real_ip_recursive on;
     
197.     #proxy_set_header X-Real-IP $remote_addr;
     
198.     #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     
199. 
     
200.     # 安全头部设置
     
201.     add_header X-Frame-Options "SAMEORIGIN";
     
202.     #add_header X-XSS-Protection "1; mode=block";
     
203.     #add_header X-Content-Type-Options "nosniff";
     
204.     #add_header Referrer-Policy "strict-origin-when-cross-origin";
     
205.     #add_header Content-Security-Policy "default-src 'self'";
     
206.     # XSS
     
207.     add_header X-XSS-Protection "1; mode=block";
     
208.     # 点击劫持
     
209.     add_header X-Frame-Options "DENY";
     
210.     # 文件上传限制
     
211.     client_max_body_size 400M;
     
212. 
     
213.     # 请求超时配置
     
214.     client_body_timeout 10s;
     
215.     client_header_timeout 10s;
     
216.     send_timeout 10s;
     
217. 
     
218.     # 基本网络优化
     
219.     sendfile on;
     
220.     tcp_nopush on;
     
221.     tcp_nodelay on;
     
222.     keepalive_timeout 65;
     
223. 
     
224.     # 压缩设置
     
225.     gzip on;
     
226.     gzip_disable "msie6";
     
227.     gzip_vary on;
     
228.     gzip_proxied any;
     
229.     gzip_comp_level 6;
     
230.     gzip_buffers 16 8k;
     
231.     gzip_types text/plain text/css application/javascript application/json application/xml application/rss+xml image/svg+xml;
     
232. 
     
233.     # 静态文件缓存优化
     
234.     location ~* \.(jpg|jpeg|png|gif|ico|css|js|svg|woff|woff2|ttf|eot|otf|ttc|map)$ {
     
235.         root /usr/share/nginx/html;
     
236.         expires 6M;
     
237.         add_header Cache-Control "public";
     
238.         access_log off;
     
239.     }
     
240. 
     
241.     # 特殊文件访问控制
     
242.     location ~* \.(htaccess|git|env)$ {
     
243.         root /usr/share/nginx/html;
     
244.         deny all;
     
245.         return 404;
     
246.     }
     
247. 
     
248.     location ~ /\.ht {
     
249.         root /usr/share/nginx/html;
     
250.         deny  all;
     
251.         return 404;
     
252.     }
     
253. 
     
254.     # ERROR 404
     
255.     error_page 404 /404.html;
     
256.     location = /404.html {
     
257.         root /usr/share/nginx/html;
     
258.         internal;
     
259.         sub_filter '{{ status_code }}' $status;
     
260.         sub_filter "{{ status_name }}" $status_name;
     
261.         sub_filter_once off;
     
262.     }
     
263. 
     
264.     # ERROR 403
     
265.     error_page 403 /403.html;
     
266.     location = /403.html {
     
267.         root /usr/share/nginx/html;
     
268.         internal;
     
269.         sub_filter '{{ status_code }}' $status;
     
270.         sub_filter "{{ status_name }}" $status_name;
     
271.         sub_filter_once off;
     
272.     }
     
273. 
     
274.     # ERROR 40x
     
275.     error_page 401 402 400 405 406 407 408 409 410 411 412 413 414 415 416 417 418 429 /40x.html;
     
276.     location = /40x.html {
     
277.         root /usr/share/nginx/html;
     
278.         internal;
     
279.         sub_filter '{{ status_code }}' $status;
     
280.         sub_filter "{{ status_name }}" $status_name;
     
281.         sub_filter_once off;
     
282.     }
     
283. 
     
284.     # ERROR 50x
     
285.     error_page 500 501 502 503 504 505 /50x.html;
     
286.     location = /50x.html {
     
287.         root /usr/share/nginx/html;
     
288.         internal;
     
289.         sub_filter '{{ status_code }}' $status;
     
290.         sub_filter "{{ status_name }}" $status_name;
     
291.         sub_filter_once off;
     
292.     }
     
293. }

复制代码

三、核心配置
3.1 Nginx 端口复用配置
  nginx-balances:n1.27.2-v1.0 from nginx-base:1.27.2
  重要配置：nginx-balances 必须启用 proxy_protocol: on ，以适应 nginx-webapp
nginx.conf -> /etc/nginx/nginx.conf

1. user  nginx;
   
2. worker_processes  auto;
   
3. 
   
4. error_log  /var/log/nginx/error.log notice;
   
5. pid        /var/run/nginx.pid;
   
6. 
   
7. 
   
8. events {
   
9.     worker_connections  1024;
   
10. }
    
11. 
    
12. 
    
13. # docs: https://nginx.org/en/docs/stream/ngx_stream_core_module.html
    
14. stream {
    
15.      # docs: https://nginx.org/en/docs/stream/ngx_stream_log_module.html
    
16.      # log_format stream '$remote_addr [$time_local] '
    
17.      #            '$protocol $status $bytes_sent $bytes_received '
    
18.      #            '$session_time "$upstream_addr" '
    
19.      #            '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';
    
20.      # custom
    
21.     log_format stream '{"@access_time":"$time_iso8601",'
    
22.         '"clientip":"$remote_addr",'
    
23.         '"pid":$pid,'
    
24.         '"pro":"$protocol",'
    
25.         '"ssl_pro": "$ssl_preread_protocol",'
    
26.         '"pro":"$protocol",'
    
27.         '"stus":$status,'
    
28.         '"sent":$bytes_sent,'
    
29.         '"recv":$bytes_received,'
    
30.         '"sess_time":$session_time,'
    
31.         '"up_addr":"$upstream_addr",'
    
32.         '"up_sent":$upstream_bytes_sent,'
    
33.         '"up_recv":$upstream_bytes_received,'
    
34.         '"up_conn_time":$upstream_connect_time,'
    
35.         '"up_resp_time":"$upstream_first_byte_time",'
    
36.         '"up_sess_time":$upstream_session_time}';
    
37. 
    
38.     upstream http {
    
39.         server nginx-webapp:80;
    
40.     }
    
41. 
    
42.     upstream https {
    
43.         server nginx-webapp:443;
    
44.     }
    
45. 
    
46.     map $ssl_preread_protocol $upstream {
    
47.         default http;
    
48.         "TLSv1.0" https;
    
49.         "TLSv1.1" https;
    
50.         "TLSv1.2" https;
    
51.         "TLSv1.3" https;
    
52.     }
    
53. 
    
54.     server {
    
55.         listen 18080;
    
56.         proxy_pass $upstream;
    
57.         proxy_connect_timeout 3s;
    
58.         proxy_timeout 5s;
    
59.         ssl_preread on;
    
60.         proxy_protocol on;
    
61.         access_log /var/log/nginx/stream_access.log stream;
    
62.         error_log /var/log/nginx/stream_error.log notice;
    
63.     }
    
64. }
    
65. 
    
66. http {
    
67.     include       /etc/nginx/mime.types;
    
68.     default_type  application/octet-stream;
    
69. 
    
70.     log_format  main  '$proxy_protocol_addr - $remote_addr - $remote_user [$time_local] "$request" '
    
71.                       '$status $body_bytes_sent "$http_referer" '
    
72.                       '"$http_user_agent" "$http_x_forwarded_for"';
    
73. 
    
74.     access_log  /var/log/nginx/access.log  main;
    
75. 
    
76.     sendfile        on;
    
77.     #tcp_nopush     on;
    
78. 
    
79.     keepalive_timeout  65;
    
80. 
    
81.     # 压缩设置
    
82.     gzip on;
    
83.     gzip_disable "msie6";
    
84.     gzip_vary on;
    
85.     gzip_proxied any;
    
86.     gzip_comp_level 6;
    
87.     gzip_buffers 16 8k;
    
88.     gzip_types text/plain text/css application/javascript application/json application/xml application/rss+xml image/svg+xml;
    
89.     map $status $status_name {
    
90.         100 "Continue";
    
91.         101 "Switching Protocols";
    
92.         102 "Processing";
    
93.         103 "Early Hints";
    
94.         200 "OK";
    
95.         201 "Created";
    
96.         202 "Accepted";
    
97.         203 "Non-Authoritative Information";
    
98.         204 "No Content";
    
99.         205 "Reset Content";
    
100.         206 "Partial Content";
     
101.         300 "Multiple Choices";
     
102.         301 "Moved Permanently";
     
103.         302 "Found";
     
104.         303 "See Other";
     
105.         304 "Not Modified";
     
106.         305 "Use Proxy";
     
107.         307 "Temporary Redirect";
     
108.         308 "Permanent Redirect";
     
109.         400 "Bad Request";
     
110.         401 "Unauthorized";
     
111.         403 "Forbidden";
     
112.         404 "Not Found";
     
113.         405 "Method Not Allowed";
     
114.         406 "Not Acceptable";
     
115.         407 "Proxy Authentication Required";
     
116.         408 "Request Timeout";
     
117.         409 "Conflict";
     
118.         410 "Gone";
     
119.         411 "Length Required";
     
120.         412 "Precondition Failed";
     
121.         413 "Payload Too Large";
     
122.         414 "URI Too Long";
     
123.         415 "Unsupported Media Type";
     
124.         416 "Range Not Satisfiable";
     
125.         417 "Expectation Failed";
     
126.         418 "I'm a teapot";
     
127.         429 "Too Many Requests";
     
128.         500 "Internal Server Error";
     
129.         501 "Not Implemented";
     
130.         502 "Bad Gateway";
     
131.         503 "Service Unavailable";
     
132.         504 "Gateway Timeout";
     
133.         505 "HTTP Version Not Supported";
     
134.         default "Unknown";
     
135.     }
     
136. 
     
137.     include /etc/nginx/conf.d/*.conf;
     
138. }

复制代码

3.1 Haproxy 端口复用配置
  haproxy-balances:h3.0.6-v1.0 from haproxy-balances:3.0.6
  重要配置：Haproxy-balances 必须启用 send-proxy ，以适应 nginx-webapp

1. global
   
2.     log /dev/stdout local0
   
3.     maxconn 2000
   
4.     user haproxy
   
5.     group haproxy
   
6.     daemon
   
7. 
   
8. defaults
   
9.     log     global
   
10.     option  httplog
    
11.     option  dontlognull
    
12.     timeout http-request    30s
    
13.     timeout queue           10s
    
14.     timeout connect         10s
    
15.     timeout client          1m
    
16.     timeout server          1m
    
17.     timeout http-keep-alive 10s
    
18.     timeout check           10s
    
19. #frontend stats
    
20. #    bind *:18081 ssl crt /var/lib/haproxy/certs.d/server.pem
    
21. #    bind *:18081
    
22. #    bind *:18081 ssl crt /var/lib/haproxy/certs.d/server.pem
    
23. #    mode http
    
24. #    stats enable
    
25. #    stats uri /haproxy_stats
    
26. #    stats realm Haproxy\ Statistics
    
27. #    stats auth admin:password
    
28. #    stats show-node
    
29. #    stats show-legends
    
30. #    stats hide-version
    
31. 
    
32. frontend main
    
33.     bind *:18080
    
34.     mode tcp
    
35.     option tcplog
    
36.     log-format %ft\ %b/%s
    
37.     tcp-request inspect-delay 5s
    
38.     # acl is_ssh req.payload(0,3) -m bin 535348
    
39.     # acl is_rdp req.payload(0,3) -m bin 030000
    
40.     # acl is_telnet req.payload(0,4) -m bin 54656c6e
    
41.     # acl is_smb req.payload(0,4) -m bin 4d5a9000
    
42.     # acl is_ftp_user req.payload(0,4) -m bin 55534552
    
43.     # acl is_ftp_pass req.payload(0,4) -m bin 50415353
    
44.     # acl is_smtp_helo req.payload(0,4) -m bin 48454c4f
    
45.     # acl is_pop3_user req.payload(0,4) -m bin 55534552
    
46.     # acl is_pop3_pass req.payload(0,4) -m bin 50415353
    
47.     # acl is_imap_login req.payload(0,5) -m bin 4c4f4749
    
48.     # acl is_dns_query req.payload(0,2) -m bin 0000
    
49.     # acl is_ldap req.payload(0,4) -m bin 30303030
    
50. 
    
51.    acl is_https req.payload(0,3) -m bin 160301
    
52.                                        # HTTPS
    
53.     acl is_http req.payload(0,3) -m bin 474554 504f53 505554 44454c 4f5054 484541 434f4e 545241
    
54.                                        #  GET   POST   PUT   DELETE OPTIONS HEAD  CONNECT TRACE
    
55.     #acl is_https req.ssl_hello_type 1
    
56.     tcp-request content accept if is_https
    
57.     # tcp-request content accept if { req.ssl_hello_type 1 }
    
58.     tcp-request content accept if is_http
    
59.     use_backend backend_http if is_http
    
60.     use_backend backend_https if is_https
    
61.     default_backend backend_http
    
62. 
    
63. backend backend_http
    
64.     mode tcp
    
65.     balance roundrobin
    
66.     server backend1 nginx-webapp:80 maxconn 10 check inter 3s send-proxy
    
67. 
    
68. backend backend_https
    
69.     mode tcp
    
70.     balance roundrobin
    
71.     server backend1 nginx-webapp:443 maxconn 10 check inter 3s send-proxy

复制代码

四、docker-compose compose file
compose file 示例

1. name: nginx-scripts-app
   
2. # docker-compose v20.3
   
3. services:
   
4.   nginx-webapp:
   
5.     image: nginx-webapp:n1.27.2-v1.0
   
6.     expose:
   
7.       - "80"
   
8.       - "443"
   
9.     volumes:
   
10.       - ./certs.d:/etc/nginx/certs.d:ro
    
11.       - ./nginx-webapp/conf.d:/etc/nginx/conf.d
    
12.       - ./nginx-webapp/nginx.conf:/etc/nginx/nginx.conf
    
13.       - ./nginx-webapp/html/:/usr/share/nginx/html
    
14.       - ./nginx-webapp/logs/:/var/log/nginx
    
15.     networks:
    
16.       - nginx-scripts
    
17. 
    
18.   haproxy-balances:
    
19.     image: haproxy-balances:h3.0.6-v1.0
    
20.     expose:
    
21.       - "18080"
    
22.     ports:
    
23.       - "18081:18080"
    
24.     volumes:
    
25.       - ./haproxy-balances/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    
26.     networks:
    
27.       - nginx-scripts
    
28. 
    
29.   nginx-balances:
    
30.     image: nginx-webapps:1.27.2
    
31.     expose:
    
32.       - "80"
    
33.       - "443"
    
34.     ports:
    
35.       - "18080:18080"
    
36.     volumes:
    
37.       #- ./certs.d:/etc/nginx/certs.d:ro
    
38.       - ./nginx-balances/conf.d:/etc/nginx/conf.d
    
39.       - ./nginx-balances/nginx.conf:/etc/nginx/nginx.conf
    
40.       - ./nginx-balances/html/:/usr/share/nginx/html
    
41.       - ./nginx-balances/logs/:/var/log/nginx
    
42.     networks:
    
43.       - nginx-scripts
    
44. 
    
45. networks:
    
46.   nginx-scripts:
    
47.     driver: bridge
    
48.     ipam:
    
49.       driver: default
    
50.       config:
    
51.         - subnet: 172.18.0.0/16
    
52.           ip_range: 172.18.18.0/24
    
53.           gateway: 172.18.18.254

复制代码

                               
登录/注册后可看大图