熔断

此任务说明如何为连接,请求和异常检测配置断路。

断路是创建弹性微服务应用程序的重要模式。电路中断功能使您可以编写应用程序来限制故障,延迟尖峰和网络特性的其他不良影响的影响。

在此任务中,您将配置断路规则,然后通过有意地"跳闸"断路器来测试配置。

在你开始之前 按照安装指南中的说明安装Istio 。 启动httpbin示例。

如果启用了自动边车注入,请部署httpbin服务:

$ kubectl apply -f samples/httpbin/httpbin.yaml

否则,您必须在部署httpbin应用程序之前手动注入sidecar :

$ kubectl apply -f <(istioctl kube-inject -f samples/httpbin/httpbin.yaml)

该httpbin应用程序充当此任务的后端服务。

配置断路器 创建目标规则以在调用httpbin服务时应用断路设置:

如果您安装/配置了启用了双向TLS身份验证的Istio,则必须在应用之前将TLS流量策略添加mode: ISTIO_MUTUAL到DestinationRule。否则,请求描述将产生503错误这里。 $ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: httpbin spec: host: httpbin trafficPolicy: connectionPool: tcp: maxConnections: 1 http: http1MaxPendingRequests: 1 maxRequestsPerConnection: 1 outlierDetection: consecutiveErrors: 1 interval: 1s baseEjectionTime: 3m maxEjectionPercent: 100 EOF

验证目标规则已正确创建:

$ kubectl get destinationrule httpbin -o yaml apiVersion: networking.istio.io/v1beta1 kind: DestinationRule ... spec: host: httpbin trafficPolicy: connectionPool: http: http1MaxPendingRequests: 1 maxRequestsPerConnection: 1 tcp: maxConnections: 1 outlierDetection: baseEjectionTime: 3m consecutiveErrors: 1 interval: 1s maxEjectionPercent: 100

添加客户端 创建一个客户端以将流量发送到httpbin服务。该客户端是一个简单的负载测试客户端,称为fortio。Fortio使您可以控制连接数,并发性和传出HTTP调用的延迟。您将使用此客户端来"跳闸"您在中设置的断路器策略DestinationRule.

向客户端注入Istio边车代理,以便网络交互受Istio支配。

如果启用了自动边车注入,请部署fortio服务:

$ kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml

否则,您必须在部署fortio应用程序之前手动注入sidecar :

$ kubectl apply -f <(istioctl kube-inject -f samples/httpbin/sample-client/fortio-deploy.yaml)

登录到客户端窗格,然后使用fortio工具进行调用httpbin。传递curl来表示您只想打一个电话:

$ export FORTIO_POD=$(kubectl get pods -lapp=fortio -o 'jsonpath={.items[0].metadata.name}') $ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio curl -quiet http://httpbin:8000/get HTTP/1.1 200 OK server: envoy date: Tue, 25 Feb 2020 20:25:52 GMT content-type: application/json content-length: 586 access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 36

{ "args": {}, "headers": { "Content-Length": "0", "Host": "httpbin:8000", "User-Agent": "fortio.org/fortio-1.3.1", "X-B3-Parentspanid": "8fc453fb1dec2c22", "X-B3-Sampled": "1", "X-B3-Spanid": "071d7f06bc94943c", "X-B3-Traceid": "86a929a0e76cda378fc453fb1dec2c22", "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin;Hash=68bbaedefe01ef4cb99e17358ff63e92d04a4ce831a35ab9a31d3c8e06adb038;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/default" }, "origin": "127.0.0.1", "url": "http://httpbin:8000/get" }

您可以看到请求成功！现在,该打破一些东西了。

断路器跳闸 在DestinationRule设置中,您指定了maxConnections: 1和 http1MaxPendingRequests: 1。这些规则表明,如果您超过一个以上的连接并发请求,则istio-proxy在为其他请求和连接打开电路时应该会看到一些故障 。

通过两个并发连接(-c 2)调用服务,并发送20个请求(-n 20):

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 2 -qps 0 -n 20 -loglevel Warning http://httpbin:8000/get 20:33:46 I logger.go:97> Log level is now 3 Warning (was 2 Info) Fortio 1.3.1 running at 0 queries per second, 6->6 procs, for 20 calls: http://httpbin:8000/get Starting at max qps with 2 thread(s) [gomax 6] for exactly 20 calls (10 per thread + 0) 20:33:46 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:33:47 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:33:47 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) Ended after 59.8524ms : 20 calls. qps=334.16 Aggregated Function Time : count 20 avg 0.0056869 +/- 0.003869 min 0.000499 max 0.0144329 sum 0.113738

range, mid point, percentile, count

= 0.000499 <= 0.001 , 0.0007495 , 10.00, 2 0.001 <= 0.002 , 0.0015 , 15.00, 1 0.003 <= 0.004 , 0.0035 , 45.00, 6 0.004 <= 0.005 , 0.0045 , 55.00, 2 0.005 <= 0.006 , 0.0055 , 60.00, 1 0.006 <= 0.007 , 0.0065 , 70.00, 2 0.007 <= 0.008 , 0.0075 , 80.00, 2 0.008 <= 0.009 , 0.0085 , 85.00, 1 0.011 <= 0.012 , 0.0115 , 90.00, 1 0.012 <= 0.014 , 0.013 , 95.00, 1 0.014 <= 0.0144329 , 0.0142165 , 100.00, 1

target 50% 0.0045

target 75% 0.0075

target 90% 0.012

target 99% 0.0143463

target 99.9% 0.0144242

Sockets used: 4 (for perfect keepalive, would be 2) Code 200 : 17 (85.0 %) Code 503 : 3 (15.0 %) Response Header Sizes : count 20 avg 195.65 +/- 82.19 min 0 max 231 sum 3913 Response Body/Total Sizes : count 20 avg 729.9 +/- 205.4 min 241 max 817 sum 14598 All done 20 calls (plus 0 warmup) 5.687 ms avg, 334.2 qps

有趣的是,几乎所有请求都通过了！该istio-proxy 确实允许一些余地。

Code 200 : 17 (85.0 %) Code 503 : 3 (15.0 %)

使并发连接数达到3:

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 3 -qps 0 -n 30 -loglevel Warning http://httpbin:8000/get 20:32:30 I logger.go:97> Log level is now 3 Warning (was 2 Info) Fortio 1.3.1 running at 0 queries per second, 6->6 procs, for 30 calls: http://httpbin:8000/get Starting at max qps with 3 thread(s) [gomax 6] for exactly 30 calls (10 per thread + 0) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) 20:32:30 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503) Ended after 51.9946ms : 30 calls. qps=576.98 Aggregated Function Time : count 30 avg 0.0040001633 +/- 0.003447 min 0.0004298 max 0.015943 sum 0.1200049

range, mid point, percentile, count

= 0.0004298 <= 0.001 , 0.0007149 , 16.67, 5 0.001 <= 0.002 , 0.0015 , 36.67, 6 0.002 <= 0.003 , 0.0025 , 50.00, 4 0.003 <= 0.004 , 0.0035 , 60.00, 3 0.004 <= 0.005 , 0.0045 , 66.67, 2 0.005 <= 0.006 , 0.0055 , 76.67, 3 0.006 <= 0.007 , 0.0065 , 83.33, 2 0.007 <= 0.008 , 0.0075 , 86.67, 1 0.008 <= 0.009 , 0.0085 , 90.00, 1 0.009 <= 0.01 , 0.0095 , 96.67, 2 0.014 <= 0.015943 , 0.0149715 , 100.00, 1

target 50% 0.003

target 75% 0.00583333

target 90% 0.009

target 99% 0.0153601

target 99.9% 0.0158847

Sockets used: 20 (for perfect keepalive, would be 3) Code 200 : 11 (36.7 %) Code 503 : 19 (63.3 %) Response Header Sizes : count 30 avg 84.366667 +/- 110.9 min 0 max 231 sum 2531 Response Body/Total Sizes : count 30 avg 451.86667 +/- 277.1 min 241 max 817 sum 13556 All done 30 calls (plus 0 warmup) 4.000 ms avg, 577.0 qps

现在,您开始看到预期的断路行为。仅成功的请求的36.7％,其余被电路中断困住了:

Code 200 : 11 (36.7 %) Code 503 : 19 (63.3 %)

查询istio-proxy统计信息以查看更多信息:

$ kubectl exec "$FORTIO_POD" -c istio-proxy -- pilot-agent request GET stats | grep httpbin | grep pending cluster.outbound|8000||httpbin.default.svc.cluster.local.circuit_breakers.default.rq_pending_open: 0 cluster.outbound|8000||httpbin.default.svc.cluster.local.circuit_breakers.high.rq_pending_open: 0 cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_active: 0 cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_failure_eject: 0 cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_overflow: 21 cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_total: 29

You can see 21 for the upstream_rq_pending_overflow value which means 21 calls so far have been flagged for circuit breaking.

Cleaning up Remove the rules:

$ kubectl delete destinationrule httpbin

Shutdown the httpbin service and client:

$ kubectl delete deploy httpbin fortio-deploy $ kubectl delete svc httpbin fortio