第1章 搭建

前言
该文档为部署CDH6.3.1,其他版本的CDH安装步骤大体一样，如果安装其他版本的cdh遇到报错，请参考我的另一篇文档。
CDH-7.1.4安装报错解决

1.1 说明

这里的ip和主机名视实际情况而定

1.2 环境

操作系统
CentOS 7
服务器5台，1台master，5台slave（master兼master和slave）
主机名解析
免密登录

1.3 时间同步

Master节点操作

yum -y install chrony

配置从阿里云同步时间，注意allow后的ip段，slave是哪个段的就配置成哪个段。

vim /etc/chrony.conf 

systemctl enable chronyd
systemctl start chronyd

slave节点操作

yum -y install chrony

配置，直接从master节点同步时间

vim /etc/chrony.conf

systemctl enable chronyd
systemctl start chronyd

1.4 关闭防火墙和selinux

所有节点操作

sed -i "s#SELINUX=enforcing#SELINUX=disabled#g" /etc/selinux/config && systemctl disable firewalld
systemctl stop firewalld
setenforce 0

1.5 修改主机名

hostnamectl set-hostname cdh-1
hostnamectl set-hostname cdh-2
hostnamectl set-hostname cdh-3
hostnamectl set-hostname cdh-4
hostnamectl set-hostname cdh-5

1.6 节点分配

cdh-1为master节点及slave节点
其他为slave节点

1.7 主机名解析

所有主机都一样

cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.235.120 cdh-1
192.168.235.126 cdh-2
192.168.235.108 cdh-3
192.168.235.123 cdh-4
192.168.235.113 cdh-5

1.8 免密登录

所有主机都执行

ssh-keygen
ssh-copy-id 192.168.235.120
ssh-copy-id 192.168.235.126
ssh-copy-id 192.168.235.108
ssh-copy-id 192.168.235.123
ssh-copy-id 192.168.235.113

1.9 安装jdk

所有节点执行

tar zxvf jdk-8u181-linux-x64.tar.gz -C /opt/
echo "export JAVA_HOME=/opt/jdk1.8.0_181/" >> /etc/profile
echo "export PATH=\$JAVA_HOME/bin:\$JAVA_HOME/jre/bin:\$PATH" >> /etc/profile
echo "export CLASSPATH=.\$CLASSPATH:\$JAVA_HOME/lib:\$JAVA_HOME/jre/lib:\$JAVA_HOME/lib/tools.jar" >> /etc/profile
source /etc/profile

1.10 Cloudera manager安装包地址

https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/

1.11 Parcels安装包地址

https://archive.cloudera.com/cdh6/6.3.2/parcels/

1.12 master端需要的安装包

cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
mysql-connector-java-8.0.15-1.el7.noarch.rpm

master节点操作
传包到master主机的/root/目录
传包到其他主机

cd /root
scp jdk-8u181-linux-x64.tar.gz cloudera-manager-* oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* manifest.json cdh-2:/root/
scp jdk-8u181-linux-x64.tar.gz cloudera-manager-* oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* manifest.json cdh-3:/root/
scp jdk-8u181-linux-x64.tar.gz cloudera-manager-* oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* manifest.json cdh-4:/root/
scp jdk-8u181-linux-x64.tar.gz cloudera-manager-* oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* manifest.json cdh-5:/root/

1.13 slave端需要的安装包

cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

1.14 安装cloudera-server

master节点执行
安装cloudera-server需要的依赖

yum -y install psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap rpcbind fuse-libs redhat-lsb net-tools postgresql-server

安装cloudera相关包

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

1.15 安装配置mysql

Master节点执行

yum -y install mariadb mariadb-server mariadb-devel
systemctl enable mariadb
systemctl start mariadb

安装8.0.15版本的jdbc

rpm -ivh mysql-connector-java-8.0.15-1.el7.noarch.rpm
cd /usr/share/java
cp -p mysql-connector-java-8.0.15.jar mysql-connector-java.jar

或者直接执行

cp -p /usr/share/java/mysql-connector-java-8.0.15.jar /usr/share/java/mysql-connector-java.jar

1.16 slave端安装jdbc

yum -y install java-headless

rpm -ivh /web/mysql-connector-java-8.0.15-1.el7.noarch.rpm
cp -p /usr/share/java/mysql-connector-java-8.0.15.jar /usr/share/java/mysql-connector-java.jar

1.17 数据库建库授权

master节点执行
说明：这里的数据库不一定非得是这个名字，看自己喜好

CREATE DATABASE report_manager DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navigator_audit DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navigator_metadata DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE activity_monitor DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON report_manager.* TO 'cdh6'@'%' IDENTIFIED BY 'cdh@336';
GRANT ALL ON report_manager.* TO 'cdh6'@'cdh-1' IDENTIFIED BY 'cdh@336';
GRANT ALL ON navigator_audit.* TO 'cdh6'@'%' IDENTIFIED BY 'cdh@336';
GRANT ALL ON navigator_audit.* TO 'cdh6'@'cdh-1' IDENTIFIED BY 'cdh@336';
GRANT ALL ON navigator_metadata.* TO 'cdh6'@'%' IDENTIFIED BY 'cdh@336';
GRANT ALL ON navigator_metadata.* TO 'cdh6'@'cdh-1' IDENTIFIED BY 'cdh@336';
GRANT ALL ON activity_monitor.* TO 'cdh6'@'%' IDENTIFIED BY 'cdh@336';
GRANT ALL ON activity_monitor.* TO 'cdh6'@'cdh-1' IDENTIFIED BY 'cdh@336';
GRANT ALL ON hue.* TO 'cdh6'@'%' IDENTIFIED BY 'cdh@336';
GRANT ALL ON hue.* TO 'cdh6'@'cdh-1' IDENTIFIED BY 'cdh@336';
GRANT ALL ON oozie.* TO 'cdh6'@'%' IDENTIFIED BY 'cdh@336';
GRANT ALL ON oozie.* TO 'cdh6'@'cdh-1' IDENTIFIED BY 'cdh@336';
GRANT ALL ON hive.* TO 'cdh6'@'%' IDENTIFIED BY 'cdh@336';
GRANT ALL ON hive.* TO 'cdh6'@'cdh-1' IDENTIFIED BY 'cdh@336';

1.18 填充数据库

master节点执行

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm   -uroot -p  scm scm scm

或者

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm   -uroot -p -hcdh-1 --scm-host cdh-1  scm scm scm

1.19 启动cloudera-scm-server

master节点执行

systemctl start cloudera-scm-server

1.20 安装cloudera-agent

所有节点执行
cloudera-agent需要的依赖

yum -y install bind-utils httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libpq.so.5 psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap /lib/lsb/init-functions fuse fuse-libs

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

1.21 安装oracle-j2sdk

所有节点执行
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

1.22 登录

浏览器输入master节点ip:7180

1.23 web界面配置集群

注意这里，不显示可用的CDH版本的话，就点击更多选项去加一个可用的url地址

这里不选择，因为已经安装了jdk

如下图，在每个节点的/opt/cloudera/parcels目录下，完成安装agents后会多一个./flood文件

这个报错没事，因为我们配置了本地的parcels存储库全部激活后点击继续

点击检查，按照顺序来，检查完毕后查看检查结果。

点击Show All Hosts

echo 10 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

优化之后重新运行

这里省略了步骤

第2章 报错

2.1 parcels下载慢的问题

安装过程中如果parcels下载很慢，那么，去https://archive.cloudera.com/cdh6/6.3.2/parcels/直接下载好parcels包，传到master和slave的/opt/cloudera/parcel-repo/目录下，注意slave没有这个目录，mkdir创建之后再传。
下载的包如下

CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
manifest.json

传包到master节点
master节点执行
在每台主机上重命名CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1包

mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256 manifest.json /opt/cloudera/parcel-repo/

传包到slave节点
slave节点执行

mkdir /opt/cloudera/parcel-repo/ && chown -R cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/
cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256 manifest.json /opt/cloudera/parcel-repo/

2.2 web界面卡在获取安全锁

解决
cdh的web几面点击终止安装
登录失败的主机，杀掉进程

ps aux|grep scm_prepare_node|awk '{print $2}'|xargs kill -9

删除多余的文件

rm -rf /tmp/scm_prepare_node.*

2.3 主机运行状态不良

rm -rf /var/lib/cloudera-scm-agent/cm_guid
systemctl restart cloudera-scm-agent

2.4 spark-shell启动报错

执行spark启动报错

/opt/cloudera/parcels/CDH/bin/spark-shell

解决

echo "export HADOOP_USER_NAME=hdfs" >> /etc/profile && source /etc/profile

2.5 hue集成hbase出现TSocket read 0 bytes

hue连接hbase默认用到hbase Thrift Server的9090端口，而hue.ini关于hbase配置的部分，连接hbase的9090端口是用的localhost

[hbase]
# Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'.
hbase_clusters=(Cluster|localhost:9090)

将其改成

[hbase]
# Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'.
hbase_clusters=(Cluster|10.0.15.186:9090)

10.0.15.186为启动了hbase Thrift Server服务的服务器的ip

解决办法:修改hbase的配置文件 添加以下配置

https://stackoverflow.com/questions/20415493/api-error-tsocket-read-0-bytes-when-using-hue-with-hbase
Add this to your hbase "core-site.conf":
<property>
  <name>hbase.thrift.support.proxyuser</name>
  <value>true</value>
</property>
<property>
  <name>hbase.regionserver.thrift.http</name>
  <value>true</value>
</property>

或者直接在web界面改，推荐在web界面改

重启hbase和hue

2.6 hbase is not allowed to impersonate admin

按F12调出浏览器的调试界面，报错如下

web界面单独看hue的角色日志，找到一条有价值的报错
hbase is not allowed to impersonate admin
解决方法：
1、在hadoop和hbase的配置文件core-site.xml中配置如下内容：

<property>
    <name>hadoop.proxyuser.hbase.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hbase.groups</name>
    <value>*</value>
</property>

通过web界面去配置（貌似只有通过web界面改后才生效）

2、重启hadoop和hbase和hue

2.7 oozie的web ui打开报错

解决：
下载ext-2.2.zip

wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
mv ext-2.2.zip /var/lib/oozie
cd /var/lib/oozie
unzip ext-2.2.zip

2.8 hdfs纠错码报错

重启namenode

2.9 修改集群ip之后导致cloudera-scm-server启动失败的问题

说明：集群个节点的ip保存在初始化数据库是那个库的（这里是scm）的HOSTS表中，改了集群节点的ip后，这个表会自动更新，但是依然会启动不了cloudera-scm-server，是因为在/etc/cloudera-scm-server/db.properties文件中
com.cloudera.cmf.db.host=cdh-1这个配置指定了cmf.db.host的ip，要改掉才可以。

第3章 优化

echo 'vm.swappiness=10'>> /etc/sysctl.conf
echo 10 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

3.1 配置命令的环境变量

因为集群软件的bin目录在/opt/cloudera/很多层以下，每次执行很费时间，故配置下命令的环境变量
将下面这几行内容加到/etc/profile文件的末尾，然后source /etc/profile

export PATH=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:$PATH
export PATH=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin:$PATH

第4章 安装后的目录

cdh集群所有的数据存放地都在hadoop的hdfs上，而hdfs的目录在/dfs/下，如图

所有软件的日志目录在默认的/var/log/下
程序相关目录在/var/lib/下
还有tmp下有一些临时目录，比如pid文件等
集群的配置文件，bin等目录在/opt/cloudera/下，如图

hbase的目录是在hadoop的hdfs文件系统的根目录下，查看方法如下

第5章 重装脚本

5.1 Master节点的重装脚本

cat reconstruction.sh
#!/bin/bash
mysql -uroot -e "drop database db_cdh6_scm;"
rpm -e `rpm -qa|grep cloud`
rm -rf /opt/cloudera/
#rpm -e `rpm -qa|grep cloud`
rm -rf /var/lib/cloud*
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
systemctl enable cloudera-scm-server
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql db_cdh6_scm -h10.0.15.166  -uroot -p --scm-host 10.0.15.166 db_cdh6_scm scm scm
systemctl start cloudera-scm-server
rm -rf /dfs/
rm -rf /yarn
rm -rf /var/lib/hadoop-*
rm -rf /var/lib/zookeeper/
rm -rf /var/lib/hbase/
rm -rf /var/lib/flume-ng/
rm -rf /var/lib/hive/
rm -rf /var/lib/impala/
rm -rf /var/lib/spark/
rm -rf /var/lib/oozie/
cp /root/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
cp /root/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha /opt/cloudera/parcel-repo/
cp /root/manifest.json /opt/cloudera/parcel-repo/
reboot

5.2 slave节点的重装脚本

cat reconstruction.sh
#!/bin/bash
rpm -e `rpm -qa|grep cloud`
rm -rf /opt/cloudera/
rm -rf /var/lib/cloud*
#rpm -e `rpm -qa|grep cloud`
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
rm -rf /dfs/
rm -rf /yarn
rm -rf /var/lib/hadoop-*
rm -rf /var/lib/zookeeper/
rm -rf /var/lib/hbase/
rm -rf /var/lib/flume-ng/
rm -rf /var/lib/hive/
rm -rf /var/lib/impala/
rm -rf /var/lib/spark/
rm -rf /var/lib/oozie/
mkdir /opt/cloudera/parcel-repo/
cp /root/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
cp /root/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha /opt/cloudera/parcel-repo/
cp /root/manifest.json /opt/cloudera/parcel-repo/
chown -R cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/
reboot