Error: unhealthy cluster | ETC node is not adding

General support questions
Post Reply
z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

Error: unhealthy cluster | ETC node is not adding

Post by z_haseeb » 2022/09/21 07:09:46

ISSUE FACING
[root@PostgreSQL-HA1 etcd]#
[root@PostgreSQL-HA1 etcd]# ETCDCTL_API=3 etcdctl --endpoints http://192.168.2.174:2379,http://192.168.2.177:2379,http://192.168.2.181 member list
3466554c14b862ba, started, PostgreSQL-HA1, http://192.168.2.174:2380, http://192.168.2.174:2379
c3309c4b9c698f42, started, PostgreSQL-HA2, http://192.168.2.177:2380, http://192.168.2.177:2379

[root@PostgreSQL-HA1 etcd]# ETCDCTL_API=3 etcdctl --endpoints http://192.168.2.174:2379,http://192.168.2.177:2379,http://192.168.2.181 endpoint health
http://192.168.2.174:2379 is healthy: successfully committed proposal: took = 2.724403ms
http://192.168.2.177:2379 is healthy: successfully committed proposal: took = 2.72823ms
http://192.168.2.181 is unhealthy: failed to connect: dial tcp: address 192.168.2.181: missing port in address
Error: unhealthy cluster


HOST FILE FROM ALL THREE NODES
[root@PostgreSQL-HA3 etcd]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.2.174 PostgreSQL-HA1
192.168.2.177 PostgreSQL-HA2
192.168.2.181 PostgreSQL-HA3


ETCD CONFIGURATION FROM ALL THREE NODES
/etc/etcd/etcd.conf #(NODE-3)
ETCD_LISTEN_PEER_URLS="http://192.168.2.181:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.2.181:2379"
ETCD_NAME="PostgreSQL-HA3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.2.181:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.2.181:2379"
ETCD_INITIAL_CLUSTER="PostgreSQL-HA1=http://192.168.2.174:2380,PostgreSQL-HA2=http://192.168.2.177:2380,PostgreSQL-HA3=http://192.168.2.181:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

/etc/etcd/etcd.conf #(NODE-2)
ETCD_LISTEN_PEER_URLS="http://192.168.2.177:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.2.177:2379"
ETCD_NAME="PostgreSQL-HA2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.2.177:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.2.177:2379"
ETCD_INITIAL_CLUSTER="PostgreSQL-HA1=http://192.168.2.174:2380,PostgreSQL-HA2=http://192.168.2.177:2380,PostgreSQL-HA3=http://192.168.2.181:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

/etc/etcd/etcd.conf #(NODE-1)
ETCD_LISTEN_PEER_URLS="http://192.168.2.174:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.2.174:2379"
ETCD_NAME="PostgreSQL-HA1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.2.174:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.2.174:2379"
ETCD_INITIAL_CLUSTER="PostgreSQL-HA1=http://192.168.2.174:2380,PostgreSQL-HA2=http://192.168.2.177:2380,PostgreSQL-HA3=http://192.168.2.181:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"


ETC SERVICE STATUS

etcd service is running on all nodes when I run systemctl status etcd


Firewall & selinux are disabled

z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

Re: Error: unhealthy cluster | ETC node is not adding

Post by z_haseeb » 2022/09/21 09:27:59

I got a bit success. the 3rd node I was trying to add after creating the cluster. Now I am able to add the new node but unable to see the cluster as healthy

[root@PostgreSQL-HA2 ~]# ETCDCTL_API=3 etcdctl --endpoints http://192.168.2.174:2379,http://192.168.2.177:2379,http://192.168.2.181 member list
3466554c14b862ba, started, PostgreSQL-HA1, http://192.168.2.174:2380, http://192.168.2.174:2379
a457597fc069735a, started, PostgreSQL-HA3, http://192.168.2.181:2380, http://192.168.2.181:2379
c3309c4b9c698f42, started, PostgreSQL-HA2, http://192.168.2.177:2380, http://192.168.2.177:2379


[root@PostgreSQL-HA2 ~]# ETCDCTL_API=3 etcdctl --endpoints http://192.168.2.174:2379,http://192.168.2.177:2379,http://192.168.2.181 endpoint health
http://192.168.2.177:2379 is healthy: successfully committed proposal: took = 2.16678ms
http://192.168.2.174:2379 is healthy: successfully committed proposal: took = 1.887412ms
http://192.168.2.181 is unhealthy: failed to connect: dial tcp: address 192.168.2.181: missing port in address
Error: unhealthy cluster

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Error: unhealthy cluster | ETC node is not adding

Post by TrevorH » 2022/09/21 11:41:15

Perhaps it would help if you specified the port on the command you are running. e.g.

not
ETCDCTL_API=3 etcdctl --endpoints http://192.168.2.174:2379,http://192.168.2.177:2379,http://192.168.2.181 endpoint health
but
ETCDCTL_API=3 etcdctl --endpoints http://192.168.2.174:2379,http://192.168.2.177:2379,http://192.168.2.181:2379 endpoint health
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply