Month: May 2016

Centrify cheat sheet

http://community.centrify.com/t5/Community-Tech-Blog/TIPS-A-Centrify-Server-Suite-Cheat-Sheet/ba-p/22568

Advertisements

Ambari services details using API on hortonworks hdp2.3

 

http://ambari-server.abc.com:8080/api/v1/stacks/HDP/versions/2.3/services

Expand each service

http://ambari-server.abc.com:8080/api/v1/stacks/HDP/versions/2.3/services/ATLAS

 

Apache Ambari standalone server and Views

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_standalone_server_setup.html

http://docambari_views_guide_Cluster_Configuration_Custom_Ambari-2.2.0.0

KeyPoints:

  1. Its a second instance not part of hadoop cluster.
  2. In other terms it’s a web server with WebUI specially for views.
  3. Installation procedure similar to Amabri cluster without cluster configuration.

Installation Steps include:

  1. Install ambari -server package
  2. Run ambari -server setup (DB, JDK)
  3. Configure external LDAP authentication
  4. Deploy views
  5. Create + configure view instances
  6. (Optional) Repeat for each Ambari Server instance
  7. (Optional) Setup proxy for Ambari Server instances

1. yum install ambari-server.x86_64

2. ambari-server setup

[3] Custom JDK

Path to JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk.x86_64

[3] – MySQL

Enter choice (1): 3
Hostname (localhost): mysqldbhost
Port (3306):
Database name (ambari):ambari-view

(create the database prior to this setup)

https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_ambari_reference_guide/content/_using_ambari_with_mysql.html

yum install mysql-connector-java

  • # mysql -u root -pCREATE USER '<AMBARIUSER>'@'%' IDENTIFIED BY '<AMBARIPASSWORD>';

    GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'%';

    CREATE USER '<AMBARIUSER>'@'localhost' IDENTIFIED BY '<AMBARIPASSWORD>';

    GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'localhost';

    CREATE USER '<AMBARIUSER>'@'<AMBARISERVERFQDN>' IDENTIFIED BY '<AMBARIPASSWORD>';

    GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'<AMBARISERVERFQDN>';

    FLUSH PRIVILEGES;

  • Where <AMBARIUSER> is the Ambari user name, <AMBARIPASSWORD> is the Ambari user password and <AMBARISERVERFQDN> is the Fully Qualified Domain Name of the Ambari Server host.

 

If Kerberos configured on Main Ambari cluster and Linux systems

 RHEL/CentOS/Oracle Linux

yum install krb5-workstation

SLES

zypper install krb5-client

Ubuntu/Debian

apt-get install krb5-user krb5-config

Use below steps to configure Kerberos.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Ambari_Security_Guide/content/_optional_set_up_kerberos_for_ambari_server.html#header

When a cluster is enabled for Kerberos, the component REST endpoints (such as the YARN ATS component) require SPNEGO authentication.

Depending on the Services in your cluster, Ambari Web needs access to these APIs. As well, views such as the Tez View need access to ATS. Therefore, the Ambari Server requires a Kerberos principal in order to authenticate via SPNEGO against these APIs. This section describes how to configure Ambari Server with a Kerberos principal and keytab to allow views to authenticate via SPNEGO against cluster components.

  1. Create a principal in your KDC for the Ambari Server. For example, using kadmin: 
    addprinc -randkey ambari-server@EXAMPLE.COM
  2. Generate a keytab for that principal. 
    xst -k ambari.server.keytab ambari-server@EXAMPLE.COM
  3. Place that keytab on the Ambari Server host. Be sure to set the file permissions so the user running the Ambari Server daemon can access the keytab file. 
    /etc/security/keytabs/ambari.server.keytab
  4. Stop the ambari server. 
    ambari-server stop
  5. Run the setup-security command. 
    ambari-server setup-security
  6. Select 3 for Setup Ambari kerberos JAAS configuration.
  7. Enter the Kerberos principal name for the Ambari Server you set up earlier.
  8. Enter the path to the keytab for the Ambari principal.
  9. Restart Ambari Server. 
    ambari-server restart

Configure external LDAP authentication

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Ambari_Security_Guide/content/_configure_ambari_to_use_ldap_server.html

cat ambari-ldap-ad.sh
#!/usr/bin/env bash## Simply preloading the ambari config with Active Directory
## compatible settings.
##
## You’ll need to update the 1st 3 settings.
##
## Then execute:
## sudo ambari-server setup-ldap
## sudo ambari-server restart
## sudo ambari-agent restart
## sudo ambari-server sync-ldap –allcat <<-‘EOF’ | sudo tee -a /etc/ambari-server/conf/ambari.properties
authentication.ldap.baseDn=dc=abc,dc=com
authentication.ldap.managerDn=cn=ambari-admin,OU=Hadoop,OU=Applications,DC=jci,DC=com
authentication.ldap.primaryUrl=ldap.abc.com:3268
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=distinguishedName
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=group
authentication.ldap.useSSL=false
authentication.ldap.userObjectClass=user
authentication.ldap.usernameAttribute=sAMAccountName
EOF

chmod u+x ambari-ldap-ad.sh

Take the backup of existing amabri.properties file

cp -p /etc/ambari-server/conf/ambari.properties /etc/ambari-server/conf/ambari.properties.date +%m%d%y

Execute following steps
sudo ambari-server setup-ldap
sudo ambari-server restart
sudo ambari-agent restart
sudo ambari-server sync-ldap –all
For sync-ldap, we have alternate way to do using AMBARI API:
1. TO sync up groups
curl -uadmin:admin -H ‘X-Requested-By: ambari’ -X POST -d ‘[{“Event”: {“specs”: [{“principal_type”: “users”, “sync_type”: “existing”}, {“principal_type”: “groups”, “sync_type”: “existing”}]}}]’ http://127.0.0.1:8080/api/v1/ldap_sync_events
{
“resources” : [
{
“href” : “http://127.0.0.1:8080/api/v1/ldap_sync_events/7&#8221;,
“Event” : {
“id” : 7
}
}
]
}
And to see the statuscurl -uadmin:admin http://127.0.0.1:8080/api/v1/ldap_sync_events/7
[root@jgias063 ~]# curl -uadmin:admin http://127.0.0.1:8080/api/v1/ldap_sync_events/7
{
“href” : “http://127.0.0.1:8080/api/v1/ldap_sync_events/7&#8221;,
“Event” : {
“id” : 7,
“specs” : [
{
“sync_type” : “existing”,
“principal_type” : “users”
},
{
“sync_type” : “existing”,
“principal_type” : “groups”
}
],
“status” : “COMPLETE”,
“status_detail” : “Completed LDAP sync.”,
“summary” : {
“groups” : {
“created” : 0,
“removed” : 0,
“updated” : 0
},
“memberships” : {
“created” : 0,
“removed” : 0
},
“users” : {
“created” : 0,
“removed” : 0,
“updated” : 0
}
},
“sync_time” : {
“end” : 1464020406077,
“start” : 1464020406052
}
}
}
TO sync for specific group:
curl -uadmin:admin -H ‘X-Requested-By: ambari’ -X POST -d ‘[{“Event”: {“specs”: [{“principal_type”:”groups”,”sync_type”:”specific”, “names”: “hdpadmgrp”}]}}]’ http://127.0.0.1:8080/api/v1/ldap_sync_events
Note: Group name here to be given is dependent on ldap-setp parameter “authentication.ldap.groupNamingAttr=cn”. Here we have configure CN as groupNamingAttr. So get the CN of group name from ldapquery -A <group Name> and use that in the above sync API syntax.
Same can be achieved using ambari-server ldap-sync syntax
Add group or groups in a text file
echo “hdpadmgrp” > groups.txt
Run below line to sync up.
ambari-server sync-ldap –groups groups.txt
For specific user sync up
echo “hdpadmin” > users.txt
Run below line to sync up.
ambari-server sync-ldap –users users.txt

 

Using AMBARI API to sync specific users:

curl -uadmin:admin -H ‘X-Requested-By: ambari’ -X POST -d ‘[{“Event”: {“specs”: [{“principal_type”:”users”,”sync_type”:”specific”, “names”: “hdpadmin”}]}}]’ http://127.0.0.1:8080/api/v1/ldap_sync_events

Note: User name depends on ldap-setup attribute “authentication.ldap.usernameAttribute=sAMAccountName”

To create Hive View Instance, need hive cluster properties, here is the details from where to pick them up.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_ambari_views_guide/content/section_get_values_non-standard_cluster_config.html

Property Value
Hive Authentication auth=NONE;user=${username}
Scripts HDFS Directory* /user/${username}/hive/scripts
Jobs HDFS Directory* /user/${username}/hive/jobs
HiveServer2 Host* Click Hive > Summary > HiveServer2 to view the host name. For example, c6401.ambari.apache.org
HiveServer2 Thrift port* Click Hive > Configs > Advanced > General > HiveServer2 Port. For example, 10000
WebHDFS FileSystem URI* Click HDFS > Configs > Advanced > Advanced hdfs-site > dfs.namenode.http-address. When you enter the value in the view definition, pre-pend “webhdfs://” to the value you find in the advanced HDFS configuration settings. For example,webhdfs://c6401.ambari.apache.org:50070
YARN Application Timeline Server URL* Click YARN > Configs > Advanced > Application Timeline Server > yarn.timeline-service.webapp.address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the YARN advanced configuration settings. For example,http://c6401.ambari.apache.org:8188
YARN ResourceManager URL* Click YARN > Configs > Advanced > Advanced yarn-site > yarn.resourcemanager.webapp.address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the YARN advanced configuration settings. For example,http://c6401.ambari.apache.org:8088

For Name Node

Property Value
First NameNode RPC Address orSecond NameNode RPC Address Select the primary or secondary NameNode to view settings from that host in the cluster. See how to get the NameNode RPC address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the advanced hdfs-site settings. For example, http://c6401.ambari.apache.org:8020
First NameNode HTTP (WebHDFS) Address or Second NameNode HTTP (WebHDFS) Address Click HDFS > Configs > Advanced > Advanced hdfs-site > dfs.namenode.http-address. When you enter the value in the view definition, pre-pend “http://&#8221; to the value you find in the advanced hdfs-site settings. For example, http://c6401.ambari.apache.org:50070

 

 

Hive Views need proxy user to setup:

https://developer.ibm.com/hadoop/2015/10/28/use-ambari-hive-view-write-execute-debug-hive-queries/

1. Set up an HDFS proxy user for the Ambari daemon user account:

To allow the process user to impersonate the logged in user, setup a proxy user for root.

From the Ambari dashboard, navigate to Advanced tab on the HDFS service Configs tab.
Expand the Custom core-site section and add the following two new properties:
hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=*

2. Create the /user/admin folder on HDFS:

Since the view stores user metadata in HDFS under the /user/<logged-in-user> folder, create this folder if it does not exist. To do this, execute the following commands as the hdfs user (admin is the logged-in user in this case.) :

Now that the cluster configuration is complete, we are ready to create an instance of the view.

Creating a view instance:
1. Navigate to admin->Manage Ambari. This takes you to the Ambari admin view, and to see the pre-deployed views, click on Views link under section Views.

2. Expand HIVE and click on Create Instance to open up the Create Instance UI. In the Details section, specify the instance name, display name and description for the view.

3. Specify settings to match the Hive configuration you have in the Settings section. In the example used in this post, Hive uses the default authentication mode, so we will use the defaults and no changes need to be made to the view settings.

4. You can select the local Ambari managed cluster or specify a Custom url in the Cluster Configuration section. We will use the local cluster for this post.

Click on Save. You will see a popup that notifies you that the view was successfully created.

 

Note: Before executing the Hive view, make sure to grant permissions to the appropriate users and groups. For the example in this post, we will grant user ‘ambari-qa’ permissions to use the view.

To do this, open up the definition of HiveView and specify the user in the Permissions section.