parse error(parse error near)一看就会

一旦出现该关键字，则认定过程执行失败，同时会把FAILED这一行的内容插入到bca日志表的NOTE字段中。

 

大多数脚本报错会在bca日志表中result字段写入FAIL,note字段写入FAILED的内容，目前确认我们的脚本报错是在日志文件中抓取FAILED关键字，一旦出现该关键字，则认定过程执行失败，同时会把FAILED这一行的内容插入到bca日志表的NOTE字段中。

1 表不存在FAILED: SemanticException [Error 10001]: Line 101:19 Table not found ODS_CDR_PSTN_D报错分析：Table not found 很明显表ODS_CDR_PSTN_D 不存在

解决办法：可以先去检查下对应ODS库下是否存在该表，若不存在，则迁移该表，若存在，那很可能就是过程脚本中未在表名前加上库名2 权限问题FAILED: RuntimeException Cannot create staging directory hdfs://beh/user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d/month_part=201512/day_part=17/.hive-staging_hive_2015-12-25_15-58-34_081_5222500334215480980-1 : Permission denied: user=shdev, access=WRITE, inode="/user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d/month_part=201512":shadmin:supergroup:drwxr-xr-x。

报错分析：显示对应hdfs的目录权限为shadmin用户所属，且权限为755，shdev用户的write操作被拒绝了解决办法：连接shadmin用户，对shdev用户赋予该路径的acl权限可以先查看该hdfs路径的权限：

hadoop fs –getfacl /user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d再赋权：hadoop fs –setfacl –m user:shdev:rwx /user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d

3 通用报错FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTase类似于一句总结的话，基本大多数报错都会用该句收尾，所以这个报错信息不能帮忙解决问题，需要查看日志。

日志存放目录：/shbigdata/dwd/logsdwd对应库名，除src外其它所有库下都有自己过程上一次执行的具体日志信息4 本地模式报错报错分析：本地模式的内存无法满足当前任务的需求解决办法：关闭本地模式。

过程语句前加上：set hive.exec.mode.local.auto=false;set hive.auto.convert.join=false;5 内存溢出报错Execution failed with exit status: 3

Obtaining error informationTask failed!Task ID: Stage-8Logs:/opt/beh/logs/hive/shleader-hive.logFAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTaskContainer [pid=17441,containerID=container_1446899682569_0355_01_000019] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used. Killing container.

报错分析：container执行需要的内存超出了物理内存限制解决办法：给它更大的内存过程语句前加上：set mapreduce.map.memory.mb=4096;set mapreduce.reduce.memory.mb=4096;

6 Group by 后面有两个相同字段报错FAILED: ClassCastException org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo

查看有sql语句中group by后面是否有2个同名的字段7 NoViableAltException(-1@[])报错NoViableAltException(-1@[])at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1087)

at org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)

at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)

at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)

at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)

at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)

at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)

at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)

at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

FAILED: ParseException line 1:14 cannot recognize input near in select clause报错分析：NoViableAltException(-1@[])表示语法问题

检查对应sql段是否有语法错误，常见错误为中文注释引起或者注释中存在分号”；”脚本中—符号可以将一行注释掉，但是如果这一行存在分号”；“,是无法把分号”；“注释掉的8 kryo报错Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 590

报错分析:过程执行时启动的job数目过多导致解决办法：重新执行过程9 表存储格式报错报错信息：Task with the most failures(4): -----Task ID: task_1447035354510_316137_m_000000

URL: http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1447035354510_316137&tipid=task_1447035354510_316137_m_000000

-----Diagnostic Messages for this Task:Error: java.io.IOException: java.lang.reflect.InvocationTargetException

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)

at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:266)

at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:213)

at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:333)

at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:719)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:252)

... 11 moreCaused by: java.io.IOException: hdfs://beh/user/hive/warehouse/ods.db/ods_usr_fx_kuayu_one_m/month_part=201512/ODS_USR_KUAYU_ONE_M.data not a RCFile and has magic of 201

at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1473) at org.apache.hadoop.hive.ql.io.RCFile$Reader.(RCFile.java:1363)

at org.apache.hadoop.hive.ql.io.RCFile$Reader.(RCFile.java:1343) at org.apache.hadoop.hive.ql.io.RCFileRecordReader.(RCFileRecordReader.java:100)

at org.apache.hadoop.hive.ql.io.RCFileInputFormat.getRecordReader(RCFileInputFormat.java:57) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:67)

... 16 moreFAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTaskMapReduce Jobs Launched:

报错分析：提示该表hdfs路径下的数据不是rcfile,根据标红报错可以看出ODS_USR_KUAYU_ONE_M.data是TEXTFILE文件检查表ods_usr_fx_kuayu_one_m的存储格式,发现该表存储格式是RCFILE，联想到该表数据是从oracle同步至hive中，同步方法为hadoop fs –put 文件至hdfs目录的，而RCFILE的表是不能直接-put或者load数据的，所以就算文件放在指定hdfs目录，查询的时候依旧会报错

tu5解决办法：若需要继续使用脚本同步数据，则需要将该表的存储格式修改为TEXTFILE总结： Hive迁移开发过程中会出现各种各样的错误，其中部分为Oracle支持但是Hive不支持导致的报错，多数为开发时编写不小心多敲或少敲造成的。

大部分的错误日志中都可以分析清楚，所以一旦报错，需要先查看日志，定位问题，再解决问题