Ambari LDAP setup and Kerberos Setup

 Ambari steps to configure LDAP

Note: These steps performed on Ambari version 2.2.2 with HDP 2.3.2 hortonworks hadoop version.
  1. Configure /etc/ambari-server/conf/ambari.properties

Used ambari-ldap-ad.sh to update ambari.properties file

cat <<-‘EOF’ | sudo tee -a /etc/ambari-server/conf/ambari.properties
authentication.ldap.baseDn=dc=example,dc=com
authentication.ldap.managerDn=cn=ldap-connect,ou=users,ou=hdp,dc=example,dc=com
authentication.ldap.primaryUrl=activedirectory.example.com:389
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=distinguishedName
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=group
authentication.ldap.useSSL=false
authentication.ldap.userObjectClass=user
authentication.ldap.usernameAttribute=sAMAccountName
EOF

Now run the command:

#ambari-server setup-ldap

It will read all required properties from the ambari.properties file which got setup above. Some important properties are:

Primary URL* (host:port): (activedirectory.example.com:389)

Base DN* (dc=example,dc=com)

Manager DN* (cn=ldap-connect,ou=users,ou=hdp,dc=example,dc=com)

Enter Manager Passwrod*: ******

Re-Enter passwrod: ******

ambari-server start
 Ambari LDAP SYNC

create users.csv or groups.csv with required users and groups to be sync with Ambari.

echo “user1,user2,user3” > users.txt

echo “group1,group2,group3” > groups.txt

ambari-server sync-ldap --user users.txt

ambari-server sync-ldap --group groups.txt

Enter Ambari Admin login: admin

Enter Ambari Admin password: *******

AMBARI KERBEROS SETUP

Pre requisite: Get the Service Principal (Ad service account if AD is configured for Kerberos)

Steps to create new service principal, setpassword and create keytab  (AD with centrify configuration)
  1.  Create Ambari Service Principal (Service account in Active directory, typically we take help of AD admin team to create this AD service account)

ambari-adm@example.com

adkeytab --new --upn ambari-adm@example.com --keytab ambari-adm.keytab --container "OU=Hadoop,OU=Application,DC=example,DC=com" -V ambari-adm --user adadmin@example.com --ignore

3. Set passwod for the new principal (Ad service account)

adpasswd  -a adadmin@example.com ambari-adm@example.com

4.Generate Keytab file for this user account (Again AD admin will help)

adkeytab -A --ignore -u adadmin@example.com -K ambari-adm.keytab -e arcfour-hmac-md5 --fource --newpassword P@$$w0rd -S ambari-adm  ambari-adm -V
Now setup ambari with kerberos
 ambari-server setup-security

Select option: 3

Setup Ambari Kerberos JAAS configuration.

Enter Ambari Server’s kerberos Principal Name: amabri-adm@example.com

Enter keytab path: /root/ambari-adm@example.com

Note: keep 600 permissions the keytab file

Once setup is done, need to configure kerberos principal

Hive View configuration:

Hive Authentication=auth=KERBEROS;principal=hive/<hive host fqdn>@EXAMPLE.COM;hive.server2.proxy.user=$(username)

WebHDFS Authentication=auth=KERBEROS;proxyuser=ambari-adm@EXAMPLE.COM

It requires proxy user configuration (personification) in HADOOP configuration: setup_HDFS_proxy_user

 

hadoop.proxyuser.ambari-adm.hosts=*
hadoop.proxyuser.ambari-adm.groups=*
hadoop.proxyuser.ambari-adm.users=*

 

HADOOP: Kereberos Setup and configuration using Ambari Wizard

Install

yum install krb5-server krb5-libs krb5-auth-dialog krb5-workstation

Edit realm

# vi /etc/krb5.conf

[realms]
EXAMPLE.COM = {
kdc = kerberos.example.com
admin_server = kerberos.example.com
}

Create database

# kdb5_util create -s
Loading random data
Initializing database ‘/var/kerberos/krb5kdc/principal’ for realm ‘EXAMPLE.COM’,
master key name ‘K/M@EXAMPLE.COM’
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

Start KDC server and kdb admin

#/etc/rc.d/init.d/krb5kdc start && /etc/rc.d/init.d/kadmin start
Starting Kerberos 5 KDC:                                   [  OK  ]
Starting Kerberos 5 Admin Server:                          [  OK  ]

Create KDC Admin

$kadmin.local -q “addprinc admin/admin”
Authenticating as principal root/admin@EXAMPLE.COM with password.
WARNING: no policy specified for admin/admin@EXAMPLE.COM; defaulting to no policy
Enter password for principal “admin/admin@EXAMPLE.COM”:
Re-enter password for principal “admin/admin@EXAMPLE.COM”:
Principal “admin/admin@EXAMPLE.COM” created.

more /var/kerberos/krb5kdc/kadm5.acl
*/admin@EXAMPLE.COM    *

Now create principals and keytabs manually in KDC using kadmin:

–Ambari kerberos enable wizard provides a csv file, which will have principal names  and keytab file names along with ownership deatils.

Created Principals using that csv file:

for PRN in `cut -d “,” -f3 kerberos.csv`;do kadmin.local -q “addprinc -randkey $PRN”; done

Generate the keytab files

Generate genkeytab.txt file using below awk step from kerberos.csv file and pass it to kadmin.local

awk -F”,” ‘{print “xst -k”,$6,” “,$3}’ kerberos.csv > genkeytab.txtkadmin.local < genkeytab.txt genkeytab.txt content:
xst -k /etc/security/keytabs/spnego.service.keytab   HTTP/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/smokeuser.headless.keytab   ambari-qa@EXAMPLE.COM
xst -k /etc/security/keytabs/ams.collector.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-hbase.master.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-hbase.regionserver.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-zk.service.keytab   amszk/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/atlas.service.keytab   atlas/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/dn.service.keytab   dn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/falcon.service.keytab   falcon/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/hbase.service.keytab   hbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/hbase.headless.keytab   hbase@EXAMPLE.COM
xst -k /etc/security/keytabs/hdfs.headless.keytab   hdfs@EXAMPLE.COM
xst -k /etc/security/keytabs/hive.service.keytab   hive/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/jhs.service.keytab   jhs/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/kafka.service.keytab   kafka/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/knox.service.keytab   knox/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nimbus.service.keytab   nimbus/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nm.service.keytab   nm/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nn.service.keytab   nn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/oozie.service.keytab   oozie/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/rm.service.keytab   rm/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/spark.headless.keytab   spark@EXAMPLE.COM
xst -k /etc/security/keytabs/storm.service.keytab   storm@EXAMPLE.COM
xst -k /etc/security/keytabs/yarn.service.keytab   yarn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/zk.service.keytab   zookeeper/sandbox.hortonworks.com@EXAMPLE.COM

Change ownership of keytab file:

chown root:hadoop   /etc/security/keytabs/spnego.service.keytab
chown ambari-qa:hadoop   /etc/security/keytabs/smokeuser.headless.keytab
chown ams:hadoop   /etc/security/keytabs/ams.collector.keytab
chown ams:hadoop   /etc/security/keytabs/ams-hbase.master.keytab
chown ams:hadoop   /etc/security/keytabs/ams-hbase.regionserver.keytab
chown ams:hadoop   /etc/security/keytabs/ams-zk.service.keytab
chown atlas:hadoop   /etc/security/keytabs/atlas.service.keytab
chown hdfs:hadoop   /etc/security/keytabs/dn.service.keytab
chown falcon:hadoop   /etc/security/keytabs/falcon.service.keytab
chown hbase:hadoop   /etc/security/keytabs/hbase.service.keytab
chown hbase:hadoop   /etc/security/keytabs/hbase.headless.keytab
chown hdfs:hadoop   /etc/security/keytabs/hdfs.headless.keytab
chown hive:hadoop   /etc/security/keytabs/hive.service.keytab
chown mapred:hadoop   /etc/security/keytabs/jhs.service.keytab
chown kafka:hadoop   /etc/security/keytabs/kafka.service.keytab
chown knox:hadoop   /etc/security/keytabs/knox.service.keytab
chown storm:hadoop   /etc/security/keytabs/nimbus.service.keytab
chown yarn:hadoop   /etc/security/keytabs/nm.service.keytab
chown hdfs:hadoop   /etc/security/keytabs/nn.service.keytab
chown oozie:hadoop   /etc/security/keytabs/oozie.service.keytab
chown yarn:hadoop   /etc/security/keytabs/rm.service.keytab
chown spark:hadoop   /etc/security/keytabs/spark.headless.keytab
chown storm:hadoop   /etc/security/keytabs/storm.service.keytab
chown yarn:hadoop   /etc/security/keytabs/yarn.service.keytab
chown zookeeper:hadoop   /etc/security/keytabs/zk.service.keytab

Changer permission:

chmod 440 /etc/security/keytabs/spnego.service.keytab
chmod 440 /etc/security/keytabs/smokeuser.headless.keytab
chmod 400 /etc/security/keytabs/ams.collector.keytab
chmod 400 /etc/security/keytabs/ams-hbase.master.keytab
chmod 400 /etc/security/keytabs/ams-hbase.regionserver.keytab
chmod 400 /etc/security/keytabs/ams-zk.service.keytab
chmod 400 /etc/security/keytabs/atlas.service.keytab
chmod 400 /etc/security/keytabs/dn.service.keytab
chmod 400 /etc/security/keytabs/falcon.service.keytab
chmod 400 /etc/security/keytabs/hbase.service.keytab
chmod 440 /etc/security/keytabs/hbase.headless.keytab
chmod 440 /etc/security/keytabs/hdfs.headless.keytab
chmod 400 /etc/security/keytabs/hive.service.keytab
chmod 400 /etc/security/keytabs/jhs.service.keytab
chmod 400 /etc/security/keytabs/kafka.service.keytab
chmod 400 /etc/security/keytabs/knox.service.keytab
chmod 400 /etc/security/keytabs/nimbus.service.keytab
chmod 400 /etc/security/keytabs/nm.service.keytab
chmod 400 /etc/security/keytabs/nn.service.keytab
chmod 400 /etc/security/keytabs/oozie.service.keytab
chmod 400 /etc/security/keytabs/rm.service.keytab
chmod 400 /etc/security/keytabs/spark.headless.keytab
chmod 400 /etc/security/keytabs/storm.service.keytab
chmod 400 /etc/security/keytabs/yarn.service.keytab
chmod 400 /etc/security/keytabs/zk.service.keytab

Continue Kerberos Enable Wizard in AMBARI.

I noticed it performing below checks.

Performing kinit using ambari-qa@EXAMPLE.COM
2015-10-11 05:39:12,400 - Execute['/usr/bin/kinit -c /var/lib/ambari-agent/data/tmp/kerberos_service_check_cc_1f6a760d597577f9618bf539df44a098 -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa@EXAMPLE.COM'] {}

After restart hdfs is working fine (but given some issue with initially, which listed below)

$hdfs dfsadmin -fs hdfs://sandbox.hortonworks.com:8020 -safemode get
Safe mode is OFF

Kerberos Troubleshooting checks:

ERRORS:

5/10/11 06:46:52 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/10/11 06:47:00 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/10/11 06:47:09 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Verify Keytab for hdfs service

$/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs

(Returns no error means, working good)

Check Ticket expiry

$kinit -R
kinit: Ticket expired while renewing credentials

klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs@EXAMPLE.COM

Valid starting     Expires            Service principal
10/11/15 06:50:49  10/12/15 06:50:49  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 10/11/15 06:50:49

klist -e -k -t /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp         Principal
—- —————– ——————————————————–
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des3-cbc-sha1)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (arcfour-hmac)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des-hmac-sha1)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des-cbc-md5)

Kerberos Configuration on HDFS:

$egrep -A1 “security.authentication|security.authorization” /etc/hadoop/conf/core-site.xml
<name>hadoop.security.authentication</name>
<value>kerberos</value>

<name>hadoop.security.authorization</name>
<value>true</value>

$egrep -A1 “kerberos.principal” /etc/hadoop/conf/hdfs-site.xml
<name>dfs.datanode.kerberos.principal</name>
<value>dn/_HOST@EXAMPLE.COM</value>

<name>dfs.namenode.kerberos.principal</name>
<value>nn/_HOST@EXAMPLE.COM</value>

<name>dfs.secondary.namenode.kerberos.principal</name>
<value>nn/_HOST@EXAMPLE.COM</value>

<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_HOST@EXAMPLE.COM</value>

Reference:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.0.0/Ambari_Doc_Suite/ADS_v200.html#ref-de2249ba-7be6-4286-ae72-848b9d327e15

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.3.1/bk_installing_manually_book/content/rpm-chap14.html

http://hortonworks.com/community/forums/topic/kerberos-security-in-hdp-gss-initiate-failed-for-the-hdfs-user/