HADOOP: Kereberos Setup and configuration using Ambari Wizard

Install

yum install krb5-server krb5-libs krb5-auth-dialog krb5-workstation

Edit realm

# vi /etc/krb5.conf

[realms]
EXAMPLE.COM = {
kdc = kerberos.example.com
admin_server = kerberos.example.com
}

Create database

# kdb5_util create -s
Loading random data
Initializing database ‘/var/kerberos/krb5kdc/principal’ for realm ‘EXAMPLE.COM’,
master key name ‘K/M@EXAMPLE.COM’
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

Start KDC server and kdb admin

#/etc/rc.d/init.d/krb5kdc start && /etc/rc.d/init.d/kadmin start
Starting Kerberos 5 KDC:                                   [  OK  ]
Starting Kerberos 5 Admin Server:                          [  OK  ]

Create KDC Admin

$kadmin.local -q “addprinc admin/admin”
Authenticating as principal root/admin@EXAMPLE.COM with password.
WARNING: no policy specified for admin/admin@EXAMPLE.COM; defaulting to no policy
Enter password for principal “admin/admin@EXAMPLE.COM”:
Re-enter password for principal “admin/admin@EXAMPLE.COM”:
Principal “admin/admin@EXAMPLE.COM” created.

more /var/kerberos/krb5kdc/kadm5.acl
*/admin@EXAMPLE.COM    *

Now create principals and keytabs manually in KDC using kadmin:

–Ambari kerberos enable wizard provides a csv file, which will have principal names  and keytab file names along with ownership deatils.

Created Principals using that csv file:

for PRN in `cut -d “,” -f3 kerberos.csv`;do kadmin.local -q “addprinc -randkey $PRN”; done

Generate the keytab files

Generate genkeytab.txt file using below awk step from kerberos.csv file and pass it to kadmin.local

awk -F”,” ‘{print “xst -k”,$6,” “,$3}’ kerberos.csv > genkeytab.txtkadmin.local < genkeytab.txt genkeytab.txt content:
xst -k /etc/security/keytabs/spnego.service.keytab   HTTP/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/smokeuser.headless.keytab   ambari-qa@EXAMPLE.COM
xst -k /etc/security/keytabs/ams.collector.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-hbase.master.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-hbase.regionserver.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-zk.service.keytab   amszk/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/atlas.service.keytab   atlas/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/dn.service.keytab   dn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/falcon.service.keytab   falcon/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/hbase.service.keytab   hbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/hbase.headless.keytab   hbase@EXAMPLE.COM
xst -k /etc/security/keytabs/hdfs.headless.keytab   hdfs@EXAMPLE.COM
xst -k /etc/security/keytabs/hive.service.keytab   hive/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/jhs.service.keytab   jhs/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/kafka.service.keytab   kafka/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/knox.service.keytab   knox/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nimbus.service.keytab   nimbus/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nm.service.keytab   nm/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nn.service.keytab   nn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/oozie.service.keytab   oozie/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/rm.service.keytab   rm/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/spark.headless.keytab   spark@EXAMPLE.COM
xst -k /etc/security/keytabs/storm.service.keytab   storm@EXAMPLE.COM
xst -k /etc/security/keytabs/yarn.service.keytab   yarn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/zk.service.keytab   zookeeper/sandbox.hortonworks.com@EXAMPLE.COM

Change ownership of keytab file:

chown root:hadoop   /etc/security/keytabs/spnego.service.keytab
chown ambari-qa:hadoop   /etc/security/keytabs/smokeuser.headless.keytab
chown ams:hadoop   /etc/security/keytabs/ams.collector.keytab
chown ams:hadoop   /etc/security/keytabs/ams-hbase.master.keytab
chown ams:hadoop   /etc/security/keytabs/ams-hbase.regionserver.keytab
chown ams:hadoop   /etc/security/keytabs/ams-zk.service.keytab
chown atlas:hadoop   /etc/security/keytabs/atlas.service.keytab
chown hdfs:hadoop   /etc/security/keytabs/dn.service.keytab
chown falcon:hadoop   /etc/security/keytabs/falcon.service.keytab
chown hbase:hadoop   /etc/security/keytabs/hbase.service.keytab
chown hbase:hadoop   /etc/security/keytabs/hbase.headless.keytab
chown hdfs:hadoop   /etc/security/keytabs/hdfs.headless.keytab
chown hive:hadoop   /etc/security/keytabs/hive.service.keytab
chown mapred:hadoop   /etc/security/keytabs/jhs.service.keytab
chown kafka:hadoop   /etc/security/keytabs/kafka.service.keytab
chown knox:hadoop   /etc/security/keytabs/knox.service.keytab
chown storm:hadoop   /etc/security/keytabs/nimbus.service.keytab
chown yarn:hadoop   /etc/security/keytabs/nm.service.keytab
chown hdfs:hadoop   /etc/security/keytabs/nn.service.keytab
chown oozie:hadoop   /etc/security/keytabs/oozie.service.keytab
chown yarn:hadoop   /etc/security/keytabs/rm.service.keytab
chown spark:hadoop   /etc/security/keytabs/spark.headless.keytab
chown storm:hadoop   /etc/security/keytabs/storm.service.keytab
chown yarn:hadoop   /etc/security/keytabs/yarn.service.keytab
chown zookeeper:hadoop   /etc/security/keytabs/zk.service.keytab

Changer permission:

chmod 440 /etc/security/keytabs/spnego.service.keytab
chmod 440 /etc/security/keytabs/smokeuser.headless.keytab
chmod 400 /etc/security/keytabs/ams.collector.keytab
chmod 400 /etc/security/keytabs/ams-hbase.master.keytab
chmod 400 /etc/security/keytabs/ams-hbase.regionserver.keytab
chmod 400 /etc/security/keytabs/ams-zk.service.keytab
chmod 400 /etc/security/keytabs/atlas.service.keytab
chmod 400 /etc/security/keytabs/dn.service.keytab
chmod 400 /etc/security/keytabs/falcon.service.keytab
chmod 400 /etc/security/keytabs/hbase.service.keytab
chmod 440 /etc/security/keytabs/hbase.headless.keytab
chmod 440 /etc/security/keytabs/hdfs.headless.keytab
chmod 400 /etc/security/keytabs/hive.service.keytab
chmod 400 /etc/security/keytabs/jhs.service.keytab
chmod 400 /etc/security/keytabs/kafka.service.keytab
chmod 400 /etc/security/keytabs/knox.service.keytab
chmod 400 /etc/security/keytabs/nimbus.service.keytab
chmod 400 /etc/security/keytabs/nm.service.keytab
chmod 400 /etc/security/keytabs/nn.service.keytab
chmod 400 /etc/security/keytabs/oozie.service.keytab
chmod 400 /etc/security/keytabs/rm.service.keytab
chmod 400 /etc/security/keytabs/spark.headless.keytab
chmod 400 /etc/security/keytabs/storm.service.keytab
chmod 400 /etc/security/keytabs/yarn.service.keytab
chmod 400 /etc/security/keytabs/zk.service.keytab

Continue Kerberos Enable Wizard in AMBARI.

I noticed it performing below checks.

Performing kinit using ambari-qa@EXAMPLE.COM
2015-10-11 05:39:12,400 - Execute['/usr/bin/kinit -c /var/lib/ambari-agent/data/tmp/kerberos_service_check_cc_1f6a760d597577f9618bf539df44a098 -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa@EXAMPLE.COM'] {}

After restart hdfs is working fine (but given some issue with initially, which listed below)

$hdfs dfsadmin -fs hdfs://sandbox.hortonworks.com:8020 -safemode get
Safe mode is OFF

Kerberos Troubleshooting checks:

ERRORS:

5/10/11 06:46:52 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/10/11 06:47:00 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/10/11 06:47:09 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Verify Keytab for hdfs service

$/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs

(Returns no error means, working good)

Check Ticket expiry

$kinit -R
kinit: Ticket expired while renewing credentials

klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs@EXAMPLE.COM

Valid starting     Expires            Service principal
10/11/15 06:50:49  10/12/15 06:50:49  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 10/11/15 06:50:49

klist -e -k -t /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp         Principal
—- —————– ——————————————————–
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des3-cbc-sha1)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (arcfour-hmac)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des-hmac-sha1)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des-cbc-md5)

Kerberos Configuration on HDFS:

$egrep -A1 “security.authentication|security.authorization” /etc/hadoop/conf/core-site.xml
<name>hadoop.security.authentication</name>
<value>kerberos</value>

<name>hadoop.security.authorization</name>
<value>true</value>

$egrep -A1 “kerberos.principal” /etc/hadoop/conf/hdfs-site.xml
<name>dfs.datanode.kerberos.principal</name>
<value>dn/_HOST@EXAMPLE.COM</value>

<name>dfs.namenode.kerberos.principal</name>
<value>nn/_HOST@EXAMPLE.COM</value>

<name>dfs.secondary.namenode.kerberos.principal</name>
<value>nn/_HOST@EXAMPLE.COM</value>

<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_HOST@EXAMPLE.COM</value>

Reference:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.0.0/Ambari_Doc_Suite/ADS_v200.html#ref-de2249ba-7be6-4286-ae72-848b9d327e15

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.3.1/bk_installing_manually_book/content/rpm-chap14.html

http://hortonworks.com/community/forums/topic/kerberos-security-in-hdp-gss-initiate-failed-for-the-hdfs-user/

Advertisements

Author: rajukv

Hadoop(BigData) Architect and Hadoop Security Architect can design and build hadoop system to meet various data science projects.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s