Ambari LDAP setup and Kerberos Setup

 Ambari steps to configure LDAP

Note: These steps performed on Ambari version 2.2.2 with HDP 2.3.2 hortonworks hadoop version.
  1. Configure /etc/ambari-server/conf/ambari.properties

Used ambari-ldap-ad.sh to update ambari.properties file

cat <<-‘EOF’ | sudo tee -a /etc/ambari-server/conf/ambari.properties
authentication.ldap.baseDn=dc=example,dc=com
authentication.ldap.managerDn=cn=ldap-connect,ou=users,ou=hdp,dc=example,dc=com
authentication.ldap.primaryUrl=activedirectory.example.com:389
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=distinguishedName
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=group
authentication.ldap.useSSL=false
authentication.ldap.userObjectClass=user
authentication.ldap.usernameAttribute=sAMAccountName
EOF

Now run the command:

#ambari-server setup-ldap

It will read all required properties from the ambari.properties file which got setup above. Some important properties are:

Primary URL* (host:port): (activedirectory.example.com:389)

Base DN* (dc=example,dc=com)

Manager DN* (cn=ldap-connect,ou=users,ou=hdp,dc=example,dc=com)

Enter Manager Passwrod*: ******

Re-Enter passwrod: ******

ambari-server start
 Ambari LDAP SYNC

create users.csv or groups.csv with required users and groups to be sync with Ambari.

echo “user1,user2,user3” > users.txt

echo “group1,group2,group3” > groups.txt

ambari-server sync-ldap --user users.txt

ambari-server sync-ldap --group groups.txt

Enter Ambari Admin login: admin

Enter Ambari Admin password: *******

AMBARI KERBEROS SETUP

Pre requisite: Get the Service Principal (Ad service account if AD is configured for Kerberos)

Steps to create new service principal, setpassword and create keytab  (AD with centrify configuration)
  1.  Create Ambari Service Principal (Service account in Active directory, typically we take help of AD admin team to create this AD service account)

ambari-adm@example.com

adkeytab --new --upn ambari-adm@example.com --keytab ambari-adm.keytab --container "OU=Hadoop,OU=Application,DC=example,DC=com" -V ambari-adm --user adadmin@example.com --ignore

3. Set passwod for the new principal (Ad service account)

adpasswd  -a adadmin@example.com ambari-adm@example.com

4.Generate Keytab file for this user account (Again AD admin will help)

adkeytab -A --ignore -u adadmin@example.com -K ambari-adm.keytab -e arcfour-hmac-md5 --fource --newpassword P@$$w0rd -S ambari-adm  ambari-adm -V
Now setup ambari with kerberos
 ambari-server setup-security

Select option: 3

Setup Ambari Kerberos JAAS configuration.

Enter Ambari Server’s kerberos Principal Name: amabri-adm@example.com

Enter keytab path: /root/ambari-adm@example.com

Note: keep 600 permissions the keytab file

Once setup is done, need to configure kerberos principal

Hive View configuration:

Hive Authentication=auth=KERBEROS;principal=hive/<hive host fqdn>@EXAMPLE.COM;hive.server2.proxy.user=$(username)

WebHDFS Authentication=auth=KERBEROS;proxyuser=ambari-adm@EXAMPLE.COM

It requires proxy user configuration (personification) in HADOOP configuration: setup_HDFS_proxy_user

 

hadoop.proxyuser.ambari-adm.hosts=*
hadoop.proxyuser.ambari-adm.groups=*
hadoop.proxyuser.ambari-adm.users=*

 

Set Up LDAP Authentication – Ambari

https://ambari.apache.org/1.2.1/installing-hadoop-using-ambari/content/ambari-chap2-4.html

vi /etc/ambari-server/conf/ambari.properties

client.security=ldap

ambari-server setup-ldap
Using python  /usr/bin/python
Setting up LDAP properties...
Primary URL* {host:port} (adserver.abc.com:3268):
Secondary URL {host:port} :
Use SSL* [true/false] (false):
User object class* (user):
User name attribute* (sAMAccountName):
Group object class* (group):
Group name attribute* (cn):
Group member attribute* (member):
Distinguished name attribute* (distinguishedName):
Base DN* (dc=abc,dc=com):
Referral method [follow/ignore] :
Bind anonymously* [true/false] (false):
Manager DN* (cn=<AD service account>,OU=Hadoop,OU=Applications,DC=abc,DC=com):
Enter Manager Password* :
Re-enter password:
====================
Review Settings
====================
authentication.ldap.managerDn: cn=<AD service account>,OU=Hadoop,OU=Applications,DC=abc,DC=com
authentication.ldap.managerPassword: *****
Save settings [y/n] (y)? y
Saving...done
Ambari Server 'setup-ldap' completed successfully.

To Sync the groups.
vi groups.csv
<add all the ad groups which need to be sync with ambari>

ambari-server sync-ldap --groups groups.csv

To Sync the users. create users.csv file with list of ad user accounts separated by comma.
ambari-server sync-ldap --user users.csv

ambari-server setup-security

ambari-server setup-security
Using python  /usr/bin/python
Security setup options…
===========================================================================
Choose one of the following options:
[1] Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 3
Setting up Ambari kerberos JAAS configuration to access secured Hadoop daemons…
Enter ambari server’s kerberos principal name (ambari@EXAMPLE.COM): qa-ambari-server@ABC.COM
Enter keytab path for ambari server’s kerberos principal: /root/qa-ambari-server.keytab
Ambari Server ‘setup-security’ completed successfully.

Apache Ambari standalone server and Views

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_standalone_server_setup.html

http://docambari_views_guide_Cluster_Configuration_Custom_Ambari-2.2.0.0

KeyPoints:

  1. Its a second instance not part of hadoop cluster.
  2. In other terms it’s a web server with WebUI specially for views.
  3. Installation procedure similar to Amabri cluster without cluster configuration.

Installation Steps include:

  1. Install ambari -server package
  2. Run ambari -server setup (DB, JDK)
  3. Configure external LDAP authentication
  4. Deploy views
  5. Create + configure view instances
  6. (Optional) Repeat for each Ambari Server instance
  7. (Optional) Setup proxy for Ambari Server instances

1. yum install ambari-server.x86_64

2. ambari-server setup

[3] Custom JDK

Path to JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk.x86_64

[3] – MySQL

Enter choice (1): 3
Hostname (localhost): mysqldbhost
Port (3306):
Database name (ambari):ambari-view

(create the database prior to this setup)

https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_ambari_reference_guide/content/_using_ambari_with_mysql.html

yum install mysql-connector-java

  • # mysql -u root -pCREATE USER '<AMBARIUSER>'@'%' IDENTIFIED BY '<AMBARIPASSWORD>';

    GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'%';

    CREATE USER '<AMBARIUSER>'@'localhost' IDENTIFIED BY '<AMBARIPASSWORD>';

    GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'localhost';

    CREATE USER '<AMBARIUSER>'@'<AMBARISERVERFQDN>' IDENTIFIED BY '<AMBARIPASSWORD>';

    GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'<AMBARISERVERFQDN>';

    FLUSH PRIVILEGES;

  • Where <AMBARIUSER> is the Ambari user name, <AMBARIPASSWORD> is the Ambari user password and <AMBARISERVERFQDN> is the Fully Qualified Domain Name of the Ambari Server host.

 

If Kerberos configured on Main Ambari cluster and Linux systems

 RHEL/CentOS/Oracle Linux

yum install krb5-workstation

SLES

zypper install krb5-client

Ubuntu/Debian

apt-get install krb5-user krb5-config

Use below steps to configure Kerberos.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Ambari_Security_Guide/content/_optional_set_up_kerberos_for_ambari_server.html#header

When a cluster is enabled for Kerberos, the component REST endpoints (such as the YARN ATS component) require SPNEGO authentication.

Depending on the Services in your cluster, Ambari Web needs access to these APIs. As well, views such as the Tez View need access to ATS. Therefore, the Ambari Server requires a Kerberos principal in order to authenticate via SPNEGO against these APIs. This section describes how to configure Ambari Server with a Kerberos principal and keytab to allow views to authenticate via SPNEGO against cluster components.

  1. Create a principal in your KDC for the Ambari Server. For example, using kadmin:
    addprinc -randkey ambari-server@EXAMPLE.COM
  2. Generate a keytab for that principal.
    xst -k ambari.server.keytab ambari-server@EXAMPLE.COM
  3. Place that keytab on the Ambari Server host. Be sure to set the file permissions so the user running the Ambari Server daemon can access the keytab file.
    /etc/security/keytabs/ambari.server.keytab
  4. Stop the ambari server.
    ambari-server stop
  5. Run the setup-security command.
    ambari-server setup-security
  6. Select 3 for Setup Ambari kerberos JAAS configuration.
  7. Enter the Kerberos principal name for the Ambari Server you set up earlier.
  8. Enter the path to the keytab for the Ambari principal.
  9. Restart Ambari Server.
    ambari-server restart

Configure external LDAP authentication

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Ambari_Security_Guide/content/_configure_ambari_to_use_ldap_server.html

cat ambari-ldap-ad.sh
#!/usr/bin/env bash## Simply preloading the ambari config with Active Directory
## compatible settings.
##
## You’ll need to update the 1st 3 settings.
##
## Then execute:
## sudo ambari-server setup-ldap
## sudo ambari-server restart
## sudo ambari-agent restart
## sudo ambari-server sync-ldap –allcat <<-‘EOF’ | sudo tee -a /etc/ambari-server/conf/ambari.properties
authentication.ldap.baseDn=dc=abc,dc=com
authentication.ldap.managerDn=cn=ambari-admin,OU=Hadoop,OU=Applications,DC=jci,DC=com
authentication.ldap.primaryUrl=ldap.abc.com:3268
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=distinguishedName
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=group
authentication.ldap.useSSL=false
authentication.ldap.userObjectClass=user
authentication.ldap.usernameAttribute=sAMAccountName
EOF

chmod u+x ambari-ldap-ad.sh

Take the backup of existing amabri.properties file

cp -p /etc/ambari-server/conf/ambari.properties /etc/ambari-server/conf/ambari.properties.date +%m%d%y

Execute following steps
sudo ambari-server setup-ldap
sudo ambari-server restart
sudo ambari-agent restart
sudo ambari-server sync-ldap –all
For sync-ldap, we have alternate way to do using AMBARI API:
1. TO sync up groups
curl -uadmin:admin -H ‘X-Requested-By: ambari’ -X POST -d ‘[{“Event”: {“specs”: [{“principal_type”: “users”, “sync_type”: “existing”}, {“principal_type”: “groups”, “sync_type”: “existing”}]}}]’ http://127.0.0.1:8080/api/v1/ldap_sync_events
{
“resources” : [
{
“href” : “http://127.0.0.1:8080/api/v1/ldap_sync_events/7&#8221;,
“Event” : {
“id” : 7
}
}
]
}
And to see the statuscurl -uadmin:admin http://127.0.0.1:8080/api/v1/ldap_sync_events/7
[root@jgias063 ~]# curl -uadmin:admin http://127.0.0.1:8080/api/v1/ldap_sync_events/7
{
“href” : “http://127.0.0.1:8080/api/v1/ldap_sync_events/7&#8221;,
“Event” : {
“id” : 7,
“specs” : [
{
“sync_type” : “existing”,
“principal_type” : “users”
},
{
“sync_type” : “existing”,
“principal_type” : “groups”
}
],
“status” : “COMPLETE”,
“status_detail” : “Completed LDAP sync.”,
“summary” : {
“groups” : {
“created” : 0,
“removed” : 0,
“updated” : 0
},
“memberships” : {
“created” : 0,
“removed” : 0
},
“users” : {
“created” : 0,
“removed” : 0,
“updated” : 0
}
},
“sync_time” : {
“end” : 1464020406077,
“start” : 1464020406052
}
}
}
TO sync for specific group:
curl -uadmin:admin -H ‘X-Requested-By: ambari’ -X POST -d ‘[{“Event”: {“specs”: [{“principal_type”:”groups”,”sync_type”:”specific”, “names”: “hdpadmgrp”}]}}]’ http://127.0.0.1:8080/api/v1/ldap_sync_events
Note: Group name here to be given is dependent on ldap-setp parameter “authentication.ldap.groupNamingAttr=cn”. Here we have configure CN as groupNamingAttr. So get the CN of group name from ldapquery -A <group Name> and use that in the above sync API syntax.
Same can be achieved using ambari-server ldap-sync syntax
Add group or groups in a text file
echo “hdpadmgrp” > groups.txt
Run below line to sync up.
ambari-server sync-ldap –groups groups.txt
For specific user sync up
echo “hdpadmin” > users.txt
Run below line to sync up.
ambari-server sync-ldap –users users.txt

 

Using AMBARI API to sync specific users:

curl -uadmin:admin -H ‘X-Requested-By: ambari’ -X POST -d ‘[{“Event”: {“specs”: [{“principal_type”:”users”,”sync_type”:”specific”, “names”: “hdpadmin”}]}}]’ http://127.0.0.1:8080/api/v1/ldap_sync_events

Note: User name depends on ldap-setup attribute “authentication.ldap.usernameAttribute=sAMAccountName”

To create Hive View Instance, need hive cluster properties, here is the details from where to pick them up.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_ambari_views_guide/content/section_get_values_non-standard_cluster_config.html

Property Value
Hive Authentication auth=NONE;user=${username}
Scripts HDFS Directory* /user/${username}/hive/scripts
Jobs HDFS Directory* /user/${username}/hive/jobs
HiveServer2 Host* Click Hive > Summary > HiveServer2 to view the host name. For example, c6401.ambari.apache.org
HiveServer2 Thrift port* Click Hive > Configs > Advanced > General > HiveServer2 Port. For example, 10000
WebHDFS FileSystem URI* Click HDFS > Configs > Advanced > Advanced hdfs-site > dfs.namenode.http-address. When you enter the value in the view definition, pre-pend “webhdfs://” to the value you find in the advanced HDFS configuration settings. For example,webhdfs://c6401.ambari.apache.org:50070
YARN Application Timeline Server URL* Click YARN > Configs > Advanced > Application Timeline Server > yarn.timeline-service.webapp.address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the YARN advanced configuration settings. For example,http://c6401.ambari.apache.org:8188
YARN ResourceManager URL* Click YARN > Configs > Advanced > Advanced yarn-site > yarn.resourcemanager.webapp.address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the YARN advanced configuration settings. For example,http://c6401.ambari.apache.org:8088

For Name Node

Property Value
First NameNode RPC Address orSecond NameNode RPC Address Select the primary or secondary NameNode to view settings from that host in the cluster. See how to get the NameNode RPC address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the advanced hdfs-site settings. For example, http://c6401.ambari.apache.org:8020
First NameNode HTTP (WebHDFS) Address or Second NameNode HTTP (WebHDFS) Address Click HDFS > Configs > Advanced > Advanced hdfs-site > dfs.namenode.http-address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the advanced hdfs-site settings. For example, http://c6401.ambari.apache.org:50070

 

 

Hive Views need proxy user to setup:

https://developer.ibm.com/hadoop/2015/10/28/use-ambari-hive-view-write-execute-debug-hive-queries/

1. Set up an HDFS proxy user for the Ambari daemon user account:

To allow the process user to impersonate the logged in user, setup a proxy user for root.

From the Ambari dashboard, navigate to Advanced tab on the HDFS service Configs tab.
Expand the Custom core-site section and add the following two new properties:
hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=*

2. Create the /user/admin folder on HDFS:

Since the view stores user metadata in HDFS under the /user/<logged-in-user> folder, create this folder if it does not exist. To do this, execute the following commands as the hdfs user (admin is the logged-in user in this case.) :

Now that the cluster configuration is complete, we are ready to create an instance of the view.

Creating a view instance:
1. Navigate to admin->Manage Ambari. This takes you to the Ambari admin view, and to see the pre-deployed views, click on Views link under section Views.

2. Expand HIVE and click on Create Instance to open up the Create Instance UI. In the Details section, specify the instance name, display name and description for the view.

3. Specify settings to match the Hive configuration you have in the Settings section. In the example used in this post, Hive uses the default authentication mode, so we will use the defaults and no changes need to be made to the view settings.

4. You can select the local Ambari managed cluster or specify a Custom url in the Cluster Configuration section. We will use the local cluster for this post.

Click on Save. You will see a popup that notifies you that the view was successfully created.

 

Note: Before executing the Hive view, make sure to grant permissions to the appropriate users and groups. For the example in this post, we will grant user ‘ambari-qa’ permissions to use the view.

To do this, open up the definition of HiveView and specify the user in the Permissions section.

 

hadoop: How to take ambari backup

AMBARI backups can be taken using posgresql backup command pg_dump.

AMBARI BACKUP

ambari-server stop
  • Stop the original Ambari Server.
    ambari-server stop
  • Create a directory to hold the database backups.
    cd /tmp
    mkdir dbdumps
    cd dbdumps/
  • Create the database backups.
    pg_dump -U $AMBARI-SERVER USERNAME ambari > ambari.sql Password: $AMBARI-SERVER PASSWORD
    pg_dump -U $MAPRED USERNAME ambarirca > ambarirca.sql Password: $MAPRED PASSWORD

Postgresql backup commands from its man page:

EXAMPLES
To dump a database called mydb into a SQL-script file:

$ pg_dump mydb > db.sql

To reload such a script into a (freshly created) database named newdb:

$ psql -d newdb -f db.sql

To dump a database into a custom-format archive file:

$ pg_dump -Fc mydb > db.dump

To reload an archive file into a (freshly created) database named newdb:

$ pg_restore -d newdb db.dump

To dump a single table named mytab:

$ pg_dump -t mytab mydb > db.sql

To  dump  all  tables whose names start with emp in the detroit schema, except for the table
named employee_log:

$ pg_dump -t ’detroit.emp*’ -T detroit.employee_log mydb > db.sql

To dump all schemas whose names start with east or  west  and  end  in  gsm,  excluding  any
schemas whose names contain the word test:

$ pg_dump -n ’east*gsm’ -n ’west*gsm’ -N ’*test*’ mydb > db.sql

The same, using regular expression notation to consolidate the switches:

$ pg_dump -n ’(east|west)*gsm’ -N ’*test*’ mydb > db.sql

To dump all database objects except for tables whose names begin with ts_:

$ pg_dump -T ’ts_*’ mydb > db.sql

To  specify an upper-case or mixed-case name in -t and related switches, you need to double-
quote the name; else it will be folded to lower case (see Patterns  [psql(1)]).  But  double
quotes are special to the shell, so in turn they must be quoted.  Thus, to dump a single ta-
ble with a mixed-case name, you need something like

$ pg_dump -t ’”MixedCaseName”’ mydb > mytab.sql