 HORTONWORKS.

COM Explore  Create  Tracks  Entrar

HCC Hortonworks Community Connection Encontrar publicações, tópicos e utilizadores... 

Inicio / Hadoop Core /

Timeline Service V2.0 Reader not starting

Pergunta por Daniel Zafar 11 de Aug de 2018 às 12:08 Tez yarntimeline timeline

Everything in my HDP is working except Timeline Service V2.0 Reader. When I try to start it I get the following error log:

Traceback (most recent call last):

File "/var/lib/ambari‐agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 108, in <mod
ule>
ApplicationTimelineReader().execute()
File "/usr/lib/ambari‐agent/lib/resource_management/libraries/script/script.py", line 353, in execute
method(env)
File "/var/lib/ambari‐agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 51, in start
hbase(action='start')
File "/var/lib/ambari‐agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 80, in hbase
createTables()

1

9Respostas · Adicionar a sua resposta 86

SEGUIDORES
MELHOR RESPOSTA  SEGUIR

Resposta por Akhil S Naik  · 13 de Aug de 2018 às 05:12

 Responstas
Hi @Daniel Zafar,
 Respostas E
Can you please navigate to the host where Timeline Reader is installed and Install Hbase Client in that host and let me know
Comentários
the result.
Seeing the code base(the missing file is in hadoop-yarn-server-timelineservice-hbase-client) I have strong feeling this can fix
the issue.

Code reference : https://github.com/hortonworks/hadoop‐release/blob/HDP‐3.0.0.0‐1634‐tag/hadoop‐yarn‐project/hadoop‐yarn

/hadoop‐yarn‐server/hadoop‐yarn‐server‐timelineservice‐hbase/hadoop‐yarn‐server‐timelineservice‐hbase‐client/src/main/ja
va/org/apache/hadoop/yarn/server/timelineservice/storage/TimelineSchemaCreator.java

Please log in and accept the answer if you find it helpful :)

 4 thub.questions.view.hide-n-comments · Partilhar

Daniel Zafar · 13 de Aug de 2018 às 05:51 0

@Akhil S Naik That worked! Nice job!

Siva Botcha · 01 de Oct de 2018 às 04:40 0

Hello @Akhil S Naik, @Daniel Zafar

I am running on HDP-3.0.1 and Yarn 3.1.0 and ran into the same issue which says
resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s
/bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-
server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-
agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.1.0-187/hadoop-yarn/timelineservice/*;
/usr/hdp/3.0.1.0-187/hbase/bin/hbase --config /usr/hdp/3.0.1.0-187/hadoop/conf/embedded-yarn-ats-hbase
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create
-s'' was killed due timeout after 300 seconds
Also can you please specify how to install hbase client here? Is it as a part of hbase service through ambari or just the
hbase client using the cli? Any guidance here please.

Resposta por Akhil S Naik  · 13 de Aug de 2018 às 05:06

Hi @Daniel zafar,
I looks like your package has some missing jars.
Can you please confirm whats the content of your folder ,this is how mine looks like.
If my assumption is correct yours will be missing : webapps, timelineservice and yarn-service-example folders.

[root@anaik1 hadoop‐yarn]# pwd

/usr/hdp/3.0.0.0‐1634/hadoop‐yarn
[root@anaik1 hadoop‐yarn]# ls ‐l
total 12544
drwxr‐xr‐x. 2 root root 93 Jul 17 06:00 bin
lrwxrwxrwx. 1 root root 16 Jul 17 06:00 conf ‐> /etc/hadoop/conf
drwxr‐xr‐x. 3 root root 30 Jul 17 06:00 etc
‐rw‐r‐‐r‐‐. 1 root root 3097487 Jul 12 21:02 hadoop‐yarn‐api‐3.1.0.3.0.0.0‐1634.jar
lrwxrwxrwx. 1 root root 38 Jul 17 06:00 hadoop‐yarn‐api.jar ‐> hadoop‐yarn‐api‐3.1.0.3.0.0.0‐1634.jar
‐rw‐r‐‐r‐‐. 1 root root 110407 Jul 12 21:02 hadoop‐yarn‐applications‐distributedshell‐3.1.0.3.0.0.0‐1634.jar

 0 · Partilhar

Resposta por amarnath reddy pappu  · 11 de Aug de 2018 às 12:19

@Daniel Zafar It looks like service restart is taking much longer time and Ambari is timing out - can you timeline service logs
to understand why it is taking that long?

 0 · Partilhar

Resposta por Daniel Zafar · 11 de Aug de 2018 às 12:39

Interestingly I don't have anything in the logs from today at all. I have tried to restart it today....

I am looking in:

/var/log/hadoop‐yarn/yarn/hadoop‐yarn‐timelinereader‐MYNODE01.log

Is that right?

 0 · Partilhar

Resposta por amarnath reddy pappu  · 11 de Aug de 2018 às 01:11

@Daniel Zafar yes - that is correct, its correct path. May be you can attempt to execute manually and see what happens.

1. cd /var/lib/ambari‐agent
2.

./ambari‐sudo.sh su yarn‐ats ‐l ‐s /bin/bash ‐c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari‐server/*:/usr/local

/texlive/2016/bin/x86_64‐linux:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/
lib64/qt‐3.3/bin:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven
/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari‐agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/
hdp/3.0.0.0‐1634/hadoop‐yarn/timelineservice/*; /usr/hdp/3.0.0.0‐1634/hbase/bin/hbase ‐‐config /usr/hdp/3.0.0.0‐1634/h
adoop/conf/embedded‐yarn‐ats‐hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator ‐Dhbase
.client.retries.number=35 ‐create ‐s'

This would print some logging in the UI

  0 · Partilhar

Resposta por Daniel Zafar · 11 de Aug de 2018 às 01:21

@amarnath reddy pappu

Good idea! This is what I got:

[root@DRIVER_NODE ambari‐agent]# ./ambari‐sudo.sh su yarn‐ats ‐l ‐s /bin/bash ‐c 'export PATH='"'"'/usr/sbin:/sbin:/usr/

lib/ambari‐server/:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/local/texlive/
2016/bin/x86_64‐linux:/usr/lib64/qt‐3.3/bin:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/local/sbin:/usr/local/bin:/usr
/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari‐agent'"'"' ; sleep 10;export HBASE
_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0‐1634/hadoop‐yarn/timelineservice/; /usr/hdp/3.0.0.0‐1634/hbase/bin/hbase ‐‐config /us
r/hdp/3.0.0.0‐1634/hadoop/conf/embedded‐yarn‐ats‐hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSch
emaCreator ‐Dhbase.client.retries.number=35 ‐create ‐s'
Error: Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator

 0  Esconder 1 · Partilhar

Cibi Chakaravarthi  · 18 de Oct de 2018 às 10:42 0

After the error "Error:Could not find or load main class

org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator", following the steps in below link fixed
the issue.

Enable System Service Mode On an Upgraded Cluster

Resposta por Daniel Zafar · 13 de Aug de 2018 às 03:33

@Akhil S Naik

Hm, I actually have all of those, any other ideas?

<br>[root@NCIENSPK01 ~]# cd /usr/hdp/3.0.0.0‐1634/hadoop‐yarn

[root@NCIENSPK01 hadoop‐yarn]# ls ‐l
total 12544
drwxr‐xr‐x 2 root root 93 Aug 10 17:51 bin
lrwxrwxrwx 1 root root 16 Aug 10 17:51 conf ‐> /etc/hadoop/conf
drwxr‐xr‐x 3 root root 30 Aug 10 17:51 etc
‐rw‐r‐‐r‐‐ 1 root root 3097487 Jul 12 16:02 hadoop‐yarn‐api‐3.1.0.3.0.0.0‐1634.jar
lrwxrwxrwx 1 root root 38 Aug 10 17:51 hadoop‐yarn‐api.jar ‐> hadoop‐yarn‐api‐3.1.0.3.0.0.0‐1634.jar
‐rw‐r‐‐r‐‐ 1 root root 110407 Jul 12 16:02 hadoop‐yarn‐applications‐distributedshell‐3.1.0.3.0.0.0‐1634.jar
lrwxrwxrwx 1 root root 64 Aug 10 17:51 hadoop‐yarn‐applications‐distributedshell.jar ‐> hadoop‐yarn‐applications‐

Then in timeline service:

[root@NCIENSPK01 hadoop‐yarn]# ls timelineservice
hadoop‐yarn‐server‐timelineservice‐3.1.0.3.0.0.0‐1634.jar hadoop‐yarn‐server‐timelineservice‐hbase‐common‐
3.1.0.3.0.0.0‐1634.jar hadoop‐yarn‐server‐timelineservice‐hbase‐coprocessor.jar
hadoop‐yarn‐server‐timelineservice‐hbase‐client‐3.1.0.3.0.0.0‐1634.jar hadoop‐yarn‐server‐timelineservice‐hbase‐common.
jar hadoop‐yarn‐server‐timelineservice.jar
hadoop‐yarn‐server‐timelineservice‐hbase‐client.jar

 0  Esconder 1 · Partilhar

Cibi Chakaravarthi  · 17 de Oct de 2018 às 12:47 0

@Daniel Zafar

Hi,
I'm also facing the same error, were you able to find anything?

# ./ambari‐sudo.sh su yarn‐ats ‐l ‐s /bin/bash ‐c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari‐server/*:/usr/loca

l/texlive/2016/bin/x86_64‐linux:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/local/texlive/2016/bin/x86_64‐linux:/usr
/lib64/qt‐3.3/bin:/usr/local/texlive/2016/bin/x86_64‐linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/mave
n/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari‐agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr
/hdp/3.0.1.0‐187/hadoop‐yarn/timelineservice/^*; /usr/hdp/3.0.1.0‐187/hbase/bin/hbase ‐‐config /usr/hdp/3.0.1.0‐187//h
adoop/conf/embedded‐yarn‐ats‐hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator ‐Dhbase
.client.retries.number=35 ‐create ‐s'
Error: Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator

Resposta por Marek Martofel  · 20 de Oct de 2018 às 01:52

Myself I do not have any issues with class not found but:
client.ConnectionImplementation: Retrieve cluster id failed

may be any related so posting

018‐10‐19 22:48:33,152 INFO [ReadOnlyZKClient‐emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client

environment:java.library.path=:/usr/hdp/3.0.1.0‐187/hadoop/lib/native/Linux‐amd64‐64:/usr/hdp/3.0.1.0‐187/hadoop/lib/n
ative
2018‐10‐19 22:48:33,152 INFO [ReadOnlyZKClient‐emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2018‐10‐19 22:48:33,152 INFO [ReadOnlyZKClient‐emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2018‐10‐19 22:48:33,152 INFO [ReadOnlyZKClient‐emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2018‐10‐19 22:48:33,152 INFO [ReadOnlyZKClient‐emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client

 0  Esconder 1 · Partilhar

Cibi Chakaravarthi  · 22 de Oct de 2018 às 07:24 1

@Marek Martofel Following the steps in below link fixed this issue:

Enable System Service Mode On an Upgraded Cluster

Resposta por Marek Martofel  · 22 de Oct de 2018 às 01:34

Many thanks Cibi! It works now. Indeed yarn-system queue was with 0% capacity and is_hbase_system_service_launch was
false.

 0 · Partilhar

Sua resposta

Preencha os detalhes...

Dica: Pode notificar um utilizador sobre esta publicação digitando @username

Não sou um robô

reCAPTCHA
Privacidade - Termos

PUBLICAR RESPOSTA
THIS WEBSITE USES COOKIES FOR ANALYTICS, PERSONALISATION AND ADVERTISING. TO LEARN MORE OR CHANGE YOUR COOKIE SETTINGS, PLEASE READ OUR COOKIE POLICY. BY CONTINUING TO BROWSE, YOU
AGREE TO OUR USE OF COOKIES.