kubernetes中基础服务排障记

# 在某台node节点上启动iperf3服务端
$ iperf3 --server

# 在另一台node节点上启动iperf3客户端
$ iperf3 --client ${node_ip}  --length 150 --parallel 100 -t 60

# 在kuberntes中部署iperf3的服务端与客户端
$ kubectl apply -f https://raw.githubusercontent.com/Pharb/kubernetes-iperf3/master/iperf3.yaml

# 查看iperf3相关pod的podIP
$ kubectl get pod -o wide

# 在某个iperf3 client的pod中执行iperf3命令，以测试其到iperf3 server pod的网络状况
$ kubectl exec -ti iperf3-clients-5b5ll -- iperf3 --client ${iperf3_server_pod_ip} --length 150 --parallel 100 -t 60

$ kubectl logs mysql-0 -c mysql

2018-04-22T15:24:36.984054Z 0 [ERROR] [MY-000000] [InnoDB] InnoDB: Assertion failure: log0write.cc:1799:time_elapsed >= 0
InnoDB: thread 139746458191616
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
15:24:36 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=2
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 67841 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
/home/mdcallag/b/orig811/bin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x3d) [0x1b1461d]
/home/mdcallag/b/orig811/bin/mysqld(handle_fatal_signal+0x4c1) [0xd58441]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f1cae617390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f1cacb0a428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f1cacb0c02a]
/home/mdcallag/b/orig811/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0xea) [0xb25e13]
/home/mdcallag/b/orig811/bin/mysqld() [0x1ce5408]
/home/mdcallag/b/orig811/bin/mysqld(log_flusher(log_t*)+0x2fb) [0x1ce5fab]
/home/mdcallag/b/orig811/bin/mysqld(std:🧵:_Impl<std::_Bind_simple<Runnable (void (*)(log_t*), log_t*)> >::_M_run()+0x68) [0x1ccbe18]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f1cad476c80]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f1cae60d6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1cacbdc41d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

apiVersion: mysql.oracle.com/v1alpha1
kind: Cluster
metadata:
  name: mysql
spec:
  members: 3
  version: "8.0.16"

apiVersion: v1
data:
  my.cnf: |
    [mysqld]
    default_authentication_plugin=mysql_native_password
    skip-name-resolve
    binlog_expire_logs_seconds=300    
kind: ConfigMap
metadata:
  name: mycnf
---
apiVersion: mysql.oracle.com/v1alpha1
kind: Cluster
metadata:
  name: mysql
spec:
  members: 3
  version: "8.0.16"
  config:
    name: mycnf
  volumeClaimTemplate:
    metadata:
      name: data
    spec:
      storageClassName: default
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi
  backupVolumeClaimTemplate:
    metadata:
      name: backup-data
    spec:
      storageClassName: default
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi