{"id":402022,"date":"2024-06-29T16:14:28","date_gmt":"2024-06-29T16:14:28","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=402022"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=402022","title":{"rendered":"Extending and moving a ZooKeeper ensemble<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n

\u00a0\u00a0\u00a0 Once upon a time our DBA team had a task. We had to move a ZooKeeper ensemble which we had been using for Clickhouse <\/a>cluster. Everyone is used to moving an ensemble by moving its data files. It seems easy and obvious but our Clickhouse cluster had more than 400 TB replicated data. All replication information had been collected in ZooKeeper cluster from the very beginning. At the end of the day we couldn\u2019t miss even a row of data. Then we looked for information on the internet. Unfortunately there was a good tutorial<\/a> about 3.4.5 and didn\u2019t fit our version 3.6.2. So we decided to use \u201cthe extending\u201d for moving our ensemble.<\/p>\n

Work Plan<\/h2>\n

\u00a0\u00a0\u00a0 Here we have a ZooKeeper ensemble consisting of 3 instances running on 3 independent servers. Also we had given 3 new servers where we had to move the ensemble in. Additionally we found another 1 temporary server for our quorum. Why? Because a Clickhouse cluster with ReplicatedMergeTree tables provides writing only with a ZooKeeper ensemble quorum else we can only have a reading. For better understanding here is the scheme\u00a0<\/p>\n

3 (id: 1, 2, 3) +1 (id: 4) + 3 (id: 5, 6, 7)<\/p>\n

\u00a0\u00a0\u00a0 The genuine ensemble is located in 3 servers (id: 1, 2, 3). We added 1 temporary server (id: 4) and new servers (id: 5, 6, 7). When the ensemble is synchronised we are ready to remove unnecessary instances (id: 1, 2, 3, 4). Finally we had an ensemble with quorum which is running on new servers (id: 5, 6, 7)<\/p>\n

Extending the genuine ensemble<\/h2>\n

\u00a0\u00a0\u00a0 In the official documentation<\/a> you can find information about an ensemble extending. However it\u2019s not a tutorial.<\/p>\n

The genuine ensemble runs on 3 independent servers with CentOS 7<\/p>\n

server.1=zk-1:2888:3888 server.2=zk-2:2888:3888 server.3=zk-3:2888:3888<\/code><\/pre>\n
Check your configuration file and if reconfigEnabled=true<\/code> is absent you must add it. Also you will face environment issues leading to dynamic reconfiguration problems.\u00a0 The solution is to add\u00a0<\/p>\n
Dzookeeper.skipACL=yes export SERVER_JVMFLAGS=\"$SERVER_JVMFLAGS -Dzookeeper.skipACL=yes\"<\/code><\/pre>\n
inside the $ZK_HOME\/bin\/zkEnv.sh<\/em> file. Now you are ready to restart the ensemble. Make sure you stop a Clickhouse cluster first.<\/p>\n
It\u2019s time to prepare new servers:<\/p>\n
    \n
  1. \n
    Upload the apache-zookeeper tarball to new servers (in our case apache-zookeeper-3.6.2-bin.tar.gz);<\/p>\n<\/li>\n
  2. \n
    Create OS user for zookeeper and extract tarball in home directory of zookeeper;<\/p>\n<\/li>\n
  3. \n
    Create a directory for zookeeper data files and a myid file which is an indicator for instance. For example we already have 3 instances in the genuine ensemble (server.1=zk-1:2888:3888 server.2=zk-2:2888:3888 server.3=zk-3:2888:3888.) myid files of new instances will contain numbers 4, 5, 6, 7;<\/p>\n<\/li>\n
  4. \n
    Add rules in firewall (ports: 2888, 3888, 2181, 7000);<\/p>\n<\/li>\n
  5. \n
    Create a service file<\/p>\n<\/li>\n<\/ol>\n
    \u00a0\u00a0\u00a0 When everything is ready run $ZK_HOME\/bin\/zkCli.sh<\/em> and enter\u00a0<\/p>\n
    reconfig -add server.4=zk-4:2888:3888:participant;2181<\/code><\/pre>\n
    \u00a0\u00a0\u00a0 If you find zoo.conf.dynamic.100000000<\/em> in $ZK_HOME\/conf<\/em> directory on all 3 servers you will be on the right path. Then you can start\u00a0zookeeper server on the 4th server and zoo.conf.dynamic.100000000<\/em> must appear in $ZK_HOME\/conf<\/em>\u00a0 as well. After each other reconfig the number<\/u> after zoo.conf.dynamic (100000000<\/u>)<\/em> will change. Repeat this step for instances 5, 6, 7. At the end the ensemble must contain all 7 instances synchronised.<\/p>\n
    Removing unnecessary instances from the ensemble<\/h2>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0 Removing can be done with the well known $ZK_HOME\/bin\/zkCli.sh<\/em>. Just execute\u00a0<\/p>\n
    reconfig -remove 1 reconfig -remove 2 reconfig -remove 3 reconfig -remove 4<\/code><\/pre>\n
    \u00a0\u00a0\u00a0 After removing stop\/disable zookeeper services on servers 1, 2, 3, 4 and check $ZK_HOME\/ \/conf\/zoo.conf.dynamic.XXXXXXX<\/em>. They must be the same.<\/p>\n
     Summary<\/h2>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 At the end of the day we moved the ensemble from servers zk-1, zk-2, zk-3 to zk-5, zk-6, zk-7. Additionally we recommend to add\u00a0<\/p>\n
    metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider<\/code><\/pre>\n
    in zoo.conf<\/em> and open the 7000 port of the firewall. This is a metrics exporter for Prometheus.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
    <\/div>\n

     \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438  https:\/\/habr.com\/ru\/articles\/586490\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
    <\/div>\n
    \n
    \n
    \n
    \n
    \u00a0\u00a0\u00a0 Once upon a time our DBA team had a task. We had to move a ZooKeeper ensemble which we had been using for Clickhouse <\/a>cluster. Everyone is used to moving an ensemble by moving its data files. It seems easy and obvious but our Clickhouse cluster had more than 400 TB replicated data. All replication information had been collected in ZooKeeper cluster from the very beginning. At the end of the day we couldn\u2019t miss even a row of data. Then we looked for information on the internet. Unfortunately there was a good tutorial<\/a> about 3.4.5 and didn\u2019t fit our version 3.6.2. So we decided to use \u201cthe extending\u201d for moving our ensemble.<\/p>\n
     Work Plan<\/h2>\n
    \u00a0\u00a0\u00a0 Here we have a ZooKeeper ensemble consisting of 3 instances running on 3 independent servers. Also we had given 3 new servers where we had to move the ensemble in. Additionally we found another 1 temporary server for our quorum. Why? Because a Clickhouse cluster with ReplicatedMergeTree tables provides writing only with a ZooKeeper ensemble quorum else we can only have a reading. For better understanding here is the scheme\u00a0<\/p>\n
    3 (id: 1, 2, 3) +1 (id: 4) + 3 (id: 5, 6, 7)<\/p>\n
    \u00a0\u00a0\u00a0 The genuine ensemble is located in 3 servers (id: 1, 2, 3). We added 1 temporary server (id: 4) and new servers (id: 5, 6, 7). When the ensemble is synchronised we are ready to remove unnecessary instances (id: 1, 2, 3, 4). Finally we had an ensemble with quorum which is running on new servers (id: 5, 6, 7)<\/p>\n
    Extending the genuine ensemble<\/h2>\n
    \u00a0\u00a0\u00a0 In the official documentation<\/a> you can find information about an ensemble extending. However it\u2019s not a tutorial.<\/p>\n
    The genuine ensemble runs on 3 independent servers with CentOS 7<\/p>\n
    server.1=zk-1:2888:3888 server.2=zk-2:2888:3888 server.3=zk-3:2888:3888<\/code><\/pre>\n
    Check your configuration file and if reconfigEnabled=true<\/code> is absent you must add it. Also you will face environment issues leading to dynamic reconfiguration problems.\u00a0 The solution is to add\u00a0<\/p>\n
    Dzookeeper.skipACL=yes export SERVER_JVMFLAGS=\"$SERVER_JVMFLAGS -Dzookeeper.skipACL=yes\"<\/code><\/pre>\n
    inside the $ZK_HOME\/bin\/zkEnv.sh<\/em> file. Now you are ready to restart the ensemble. Make sure you stop a Clickhouse cluster first.<\/p>\n
    It\u2019s time to prepare new servers:<\/p>\n
      \n
    1. \n
      Upload the apache-zookeeper tarball to new servers (in our case apache-zookeeper-3.6.2-bin.tar.gz);<\/p>\n<\/li>\n
    2. \n
      Create OS user for zookeeper and extract tarball in home directory of zookeeper;<\/p>\n<\/li>\n
    3. \n
      Create a directory for zookeeper data files and a myid file which is an indicator for instance. For example we already have 3 instances in the genuine ensemble (server.1=zk-1:2888:3888 server.2=zk-2:2888:3888 server.3=zk-3:2888:3888.) myid files of new instances will contain numbers 4, 5, 6, 7;<\/p>\n<\/li>\n
    4. \n
      Add rules in firewall (ports: 2888, 3888, 2181, 7000);<\/p>\n<\/li>\n
    5. \n
      Create a service file<\/p>\n<\/li>\n<\/ol>\n
      \u00a0\u00a0\u00a0 When everything is ready run $ZK_HOME\/bin\/zkCli.sh<\/em> and enter\u00a0<\/p>\n
      reconfig -add server.4=zk-4:2888:3888:participant;2181<\/code><\/pre>\n
      \u00a0\u00a0\u00a0 If you find zoo.conf.dynamic.100000000<\/em> in $ZK_HOME\/conf<\/em> directory on all 3 servers you will be on the right path. Then you can start\u00a0zookeeper server on the 4th server and zoo.conf.dynamic.100000000<\/em> must appear in $ZK_HOME\/conf<\/em>\u00a0 as well. After each other reconfig the number<\/u> after zoo.conf.dynamic (100000000<\/u>)<\/em> will change. Repeat this step for instances 5, 6, 7. At the end the ensemble must contain all 7 instances synchronised.<\/p>\n
      Removing unnecessary instances from the ensemble<\/h2>\n
      \u00a0\u00a0\u00a0\u00a0\u00a0 Removing can be done with the well known $ZK_HOME\/bin\/zkCli.sh<\/em>. Just execute\u00a0<\/p>\n
      reconfig -remove 1 reconfig -remove 2 reconfig -remove 3 reconfig -remove 4<\/code><\/pre>\n
      \u00a0\u00a0\u00a0 After removing stop\/disable zookeeper services on servers 1, 2, 3, 4 and check $ZK_HOME\/ \/conf\/zoo.conf.dynamic.XXXXXXX<\/em>. They must be the same.<\/p>\n
       Summary<\/h2>\n
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 At the end of the day we moved the ensemble from servers zk-1, zk-2, zk-3 to zk-5, zk-6, zk-7. Additionally we recommend to add\u00a0<\/p>\n
      metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider<\/code><\/pre>\n
      in zoo.conf<\/em> and open the 7000 port of the firewall. This is a metrics exporter for Prometheus.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
      <\/div>\n

       \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438  https:\/\/habr.com\/ru\/articles\/586490\/<\/a>
      <\/br><\/br><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-402022","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/402022","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=402022"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/402022\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=402022"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=402022"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=402022"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}