搭建hadoop平台时遇到的问题及解决方法

笔者最近在搭建伪分布式hadoop平台的时候,遇到了一些问题,在解决了这些问题之后就将问题及解决方法记录了下来,以供参考。

出现的问题1

hadoop@daixiang-virtual-machine:/usr/local/hadoop/etc/hadoop$ start-dfs.sh
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-daixiang-virtual-machine.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-daixiang-virtual-machine.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 65:d4:b8:4c:c1:c6:eb:b6:f8:02:7b:fd:52:80:f8:84.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-daixiang-virtual-machine.out

解决方法

删除 tmp 目录 ,重新格式化


出现的问题2

使用start-dfs.sh启动HDFS时发现datanode没有启动:

hadoop@daixiang-virtual-machine:/usr/local/hadoop/etc/hadoop$ jps
15158 Jps
15049 SecondaryNameNode
14751 NameNode

解决方法

查看etc/hadoop/hdfs-site.xml配置文件是否写错,一般是该配置文件写错导致的。


出现的问题3

hadoop@daixiang-virtual-machine:/usr/local/hadoop/etc/hadoop$ hadoop fs -mkdir input
mkdir: `input’: No such file or directory

无法stop-dfs.sh,进入到/usr/local/hadoop/sbin 执行./stop-dfs.sh可以关闭

解决方法:

权限问题,未对hadoop用户进行授权,使用命令sudo chmod -R a+w /usr/local/hadoop进行授权。


出现问题4:

修改文件时,出现提示:E45: ‘readonly’ option is set (add ! to override) 。无法报错更改。

解决方法:

权限问题,该错误为当前用户没有权限对文件作修改。进入到/usr/local目录,使用命令修改hadoop文件夹以及子文件夹的所属用户为hadoop用户

$ sudo chown -R hadoop ./hadoop

出现的问题5:

启动hdfs时报错:Error: Cannot find configuration directory: /etc/hadoop
找不到配置目录,并且没有成功启动进程。

hadoop@daixiang-virtual-machine:/usr/local$ start-dfs.sh
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
Error: Cannot find configuration directory: /etc/hadoop
Error: Cannot find configuration directory: /etc/hadoop
Starting secondary namenodes [0.0.0.0]
Error: Cannot find configuration directory: /etc/hadoop
hadoop@daixiang-virtual-machine:/usr/local$ jps
15782 Jps

解决方法:

将文件/usr/local/hadoop/etc/hadoop/hadoop-env.sh中的

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

改为绝对路径:

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
daixiangcn CSDN认证博客专家 后端开发工程师 产品体验官
中国公民,中共预备党员,CSDN博客专家,本科大四在读。江苏省第四届“互联网+”大学生创新创业大赛二等奖获得者,第二届、第三届微信小程序应用开发大赛全国三等奖、华东赛区二等奖项目作者,微信小程序《约车吗》、《庆云慈善会》、《锋云速检》项目工程师。
已标记关键词 清除标记
相关推荐
©️2020 CSDN 皮肤主题: 技术工厂 设计师:CSDN官方博客 返回首页