APACHE RANGER Version 0.6 has tag based Policies

APACHE RANGER Version 0.6 has tag based Policies for more details read 

Key Features are:

  1. TAG STOREDetails of tags associated with resources are stored in a tag store. Apache Ranger plugins retrieve the tag details from the tag store for use during policy evaluation. To minimize the performance impact during policy evaluation (in finding tags for resources), Apache Ranger plugins cache the tags and periodically poll the tag store for any changes. On detecting change, the plugins update the cache. In addition, the plugins store the tag details in a local cache file – just as the policies are stored in a local cache file. On component restart, the plugins will use the tag data from the local cache file if the tag store is not reachable.In the current release, Apache Ranger plugins download the tag details from the store managed by Ranger Admin. Ranger Admin persists the tag details in its policy store and provides a REST interface for the plugins to download the tag details
  2. TAG SYNCApache Ranger introduces a new module, ranger-tagsync, to populate the tag store from the tag details available in an external system like Apache Atlas.  Tag sync is a daemon process similar to ranger-usersync process.In the current release, ranger-tagsync supports receiving tag details from Apache Atlas via change notifications. As tags are added/updated/deleted to resources in Apache Atlas, ranger-tagsync would receive notifications and update the tag store.
  3. Ranger Admin UI

Apache Ranger provides a new UI page, named ‘Tag Based Policies’, to work with tag based policies. The workflow to create/update tag-based policies is essentially same as with the existing ‘Resource Based Policies’. Start by adding a tag service instance, in which tag-based policies can be created. Multiple tag service instances can be created – like tag-dev/tag-test/tag-prod, to group tag-based policies for different clusters.

Policy UI for tag-based policy looks very similar to existing resource-based policies. The name of the tag should be specified at the top half of the page; the bottom half of the page provides the UI to specify permissions for users and groups. Following are few differences from resource-based policies UI:

  • Permissions UI lists the permissions available in all the service-types. This allows policy authors to restrict type of accesses users/groups can perform on tagged resources
  • Wildcards are not allowed in tag names. Also only one tag can be entered per policy
  • Delegated Admin is not available for tag-based policies. Currently only an administrator can work with tag-based policies

4. TAG ATTRIBUTES

Tags in Apache Ranger can have attributes. Tag attribute values can be used in Ranger tag-based-policies to influence the authorization decision.

For example, to deny access to a resource after a specific date:

  • add EXPIRES_ON tag to the resource
  • add a tag attribute, named expiry_date, with its value set to the expiry date
  • create a Ranger policy for EXPIRES_ON tag
  • add a condition in this policy to deny the access when the date specified in expiry_date tag attribute is later than the current date

In fact, the above detailed EXPIRES_ON tag policy is created as the default policy in tag service instances.

5. TAGS IN POLICY EVALUATION

 

Ranger-Policy-Evaluation-Flow-with-Tags

 

Advertisements

HADOOP: Kereberos Setup and configuration using Ambari Wizard

Install

yum install krb5-server krb5-libs krb5-auth-dialog krb5-workstation

Edit realm

# vi /etc/krb5.conf

[realms]
EXAMPLE.COM = {
kdc = kerberos.example.com
admin_server = kerberos.example.com
}

Create database

# kdb5_util create -s
Loading random data
Initializing database ‘/var/kerberos/krb5kdc/principal’ for realm ‘EXAMPLE.COM’,
master key name ‘K/M@EXAMPLE.COM’
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

Start KDC server and kdb admin

#/etc/rc.d/init.d/krb5kdc start && /etc/rc.d/init.d/kadmin start
Starting Kerberos 5 KDC:                                   [  OK  ]
Starting Kerberos 5 Admin Server:                          [  OK  ]

Create KDC Admin

$kadmin.local -q “addprinc admin/admin”
Authenticating as principal root/admin@EXAMPLE.COM with password.
WARNING: no policy specified for admin/admin@EXAMPLE.COM; defaulting to no policy
Enter password for principal “admin/admin@EXAMPLE.COM”:
Re-enter password for principal “admin/admin@EXAMPLE.COM”:
Principal “admin/admin@EXAMPLE.COM” created.

more /var/kerberos/krb5kdc/kadm5.acl
*/admin@EXAMPLE.COM    *

Now create principals and keytabs manually in KDC using kadmin:

–Ambari kerberos enable wizard provides a csv file, which will have principal names  and keytab file names along with ownership deatils.

Created Principals using that csv file:

for PRN in `cut -d “,” -f3 kerberos.csv`;do kadmin.local -q “addprinc -randkey $PRN”; done

Generate the keytab files

Generate genkeytab.txt file using below awk step from kerberos.csv file and pass it to kadmin.local

awk -F”,” ‘{print “xst -k”,$6,” “,$3}’ kerberos.csv > genkeytab.txtkadmin.local < genkeytab.txt genkeytab.txt content:
xst -k /etc/security/keytabs/spnego.service.keytab   HTTP/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/smokeuser.headless.keytab   ambari-qa@EXAMPLE.COM
xst -k /etc/security/keytabs/ams.collector.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-hbase.master.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-hbase.regionserver.keytab   amshbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/ams-zk.service.keytab   amszk/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/atlas.service.keytab   atlas/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/dn.service.keytab   dn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/falcon.service.keytab   falcon/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/hbase.service.keytab   hbase/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/hbase.headless.keytab   hbase@EXAMPLE.COM
xst -k /etc/security/keytabs/hdfs.headless.keytab   hdfs@EXAMPLE.COM
xst -k /etc/security/keytabs/hive.service.keytab   hive/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/jhs.service.keytab   jhs/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/kafka.service.keytab   kafka/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/knox.service.keytab   knox/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nimbus.service.keytab   nimbus/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nm.service.keytab   nm/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/nn.service.keytab   nn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/oozie.service.keytab   oozie/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/rm.service.keytab   rm/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/spark.headless.keytab   spark@EXAMPLE.COM
xst -k /etc/security/keytabs/storm.service.keytab   storm@EXAMPLE.COM
xst -k /etc/security/keytabs/yarn.service.keytab   yarn/sandbox.hortonworks.com@EXAMPLE.COM
xst -k /etc/security/keytabs/zk.service.keytab   zookeeper/sandbox.hortonworks.com@EXAMPLE.COM

Change ownership of keytab file:

chown root:hadoop   /etc/security/keytabs/spnego.service.keytab
chown ambari-qa:hadoop   /etc/security/keytabs/smokeuser.headless.keytab
chown ams:hadoop   /etc/security/keytabs/ams.collector.keytab
chown ams:hadoop   /etc/security/keytabs/ams-hbase.master.keytab
chown ams:hadoop   /etc/security/keytabs/ams-hbase.regionserver.keytab
chown ams:hadoop   /etc/security/keytabs/ams-zk.service.keytab
chown atlas:hadoop   /etc/security/keytabs/atlas.service.keytab
chown hdfs:hadoop   /etc/security/keytabs/dn.service.keytab
chown falcon:hadoop   /etc/security/keytabs/falcon.service.keytab
chown hbase:hadoop   /etc/security/keytabs/hbase.service.keytab
chown hbase:hadoop   /etc/security/keytabs/hbase.headless.keytab
chown hdfs:hadoop   /etc/security/keytabs/hdfs.headless.keytab
chown hive:hadoop   /etc/security/keytabs/hive.service.keytab
chown mapred:hadoop   /etc/security/keytabs/jhs.service.keytab
chown kafka:hadoop   /etc/security/keytabs/kafka.service.keytab
chown knox:hadoop   /etc/security/keytabs/knox.service.keytab
chown storm:hadoop   /etc/security/keytabs/nimbus.service.keytab
chown yarn:hadoop   /etc/security/keytabs/nm.service.keytab
chown hdfs:hadoop   /etc/security/keytabs/nn.service.keytab
chown oozie:hadoop   /etc/security/keytabs/oozie.service.keytab
chown yarn:hadoop   /etc/security/keytabs/rm.service.keytab
chown spark:hadoop   /etc/security/keytabs/spark.headless.keytab
chown storm:hadoop   /etc/security/keytabs/storm.service.keytab
chown yarn:hadoop   /etc/security/keytabs/yarn.service.keytab
chown zookeeper:hadoop   /etc/security/keytabs/zk.service.keytab

Changer permission:

chmod 440 /etc/security/keytabs/spnego.service.keytab
chmod 440 /etc/security/keytabs/smokeuser.headless.keytab
chmod 400 /etc/security/keytabs/ams.collector.keytab
chmod 400 /etc/security/keytabs/ams-hbase.master.keytab
chmod 400 /etc/security/keytabs/ams-hbase.regionserver.keytab
chmod 400 /etc/security/keytabs/ams-zk.service.keytab
chmod 400 /etc/security/keytabs/atlas.service.keytab
chmod 400 /etc/security/keytabs/dn.service.keytab
chmod 400 /etc/security/keytabs/falcon.service.keytab
chmod 400 /etc/security/keytabs/hbase.service.keytab
chmod 440 /etc/security/keytabs/hbase.headless.keytab
chmod 440 /etc/security/keytabs/hdfs.headless.keytab
chmod 400 /etc/security/keytabs/hive.service.keytab
chmod 400 /etc/security/keytabs/jhs.service.keytab
chmod 400 /etc/security/keytabs/kafka.service.keytab
chmod 400 /etc/security/keytabs/knox.service.keytab
chmod 400 /etc/security/keytabs/nimbus.service.keytab
chmod 400 /etc/security/keytabs/nm.service.keytab
chmod 400 /etc/security/keytabs/nn.service.keytab
chmod 400 /etc/security/keytabs/oozie.service.keytab
chmod 400 /etc/security/keytabs/rm.service.keytab
chmod 400 /etc/security/keytabs/spark.headless.keytab
chmod 400 /etc/security/keytabs/storm.service.keytab
chmod 400 /etc/security/keytabs/yarn.service.keytab
chmod 400 /etc/security/keytabs/zk.service.keytab

Continue Kerberos Enable Wizard in AMBARI.

I noticed it performing below checks.

Performing kinit using ambari-qa@EXAMPLE.COM
2015-10-11 05:39:12,400 - Execute['/usr/bin/kinit -c /var/lib/ambari-agent/data/tmp/kerberos_service_check_cc_1f6a760d597577f9618bf539df44a098 -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa@EXAMPLE.COM'] {}

After restart hdfs is working fine (but given some issue with initially, which listed below)

$hdfs dfsadmin -fs hdfs://sandbox.hortonworks.com:8020 -safemode get
Safe mode is OFF

Kerberos Troubleshooting checks:

ERRORS:

5/10/11 06:46:52 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/10/11 06:47:00 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/10/11 06:47:09 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Verify Keytab for hdfs service

$/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs

(Returns no error means, working good)

Check Ticket expiry

$kinit -R
kinit: Ticket expired while renewing credentials

klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs@EXAMPLE.COM

Valid starting     Expires            Service principal
10/11/15 06:50:49  10/12/15 06:50:49  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 10/11/15 06:50:49

klist -e -k -t /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp         Principal
—- —————– ——————————————————–
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des3-cbc-sha1)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (arcfour-hmac)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des-hmac-sha1)
3 10/11/15 05:26:37 hdfs@EXAMPLE.COM (des-cbc-md5)

Kerberos Configuration on HDFS:

$egrep -A1 “security.authentication|security.authorization” /etc/hadoop/conf/core-site.xml
<name>hadoop.security.authentication</name>
<value>kerberos</value>

<name>hadoop.security.authorization</name>
<value>true</value>

$egrep -A1 “kerberos.principal” /etc/hadoop/conf/hdfs-site.xml
<name>dfs.datanode.kerberos.principal</name>
<value>dn/_HOST@EXAMPLE.COM</value>

<name>dfs.namenode.kerberos.principal</name>
<value>nn/_HOST@EXAMPLE.COM</value>

<name>dfs.secondary.namenode.kerberos.principal</name>
<value>nn/_HOST@EXAMPLE.COM</value>

<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_HOST@EXAMPLE.COM</value>

Reference:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.0.0/Ambari_Doc_Suite/ADS_v200.html#ref-de2249ba-7be6-4286-ae72-848b9d327e15

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.3.1/bk_installing_manually_book/content/rpm-chap14.html

http://hortonworks.com/community/forums/topic/kerberos-security-in-hdp-gss-initiate-failed-for-the-hdfs-user/