Month: May 2016
Ambari services details using API on hortonworks hdp2.3
Apache Ambari standalone server and Views
http://docambari_views_guide_Cluster_Configuration_Custom_Ambari-2.2.0.0
KeyPoints:
- Its a second instance not part of hadoop cluster.
- In other terms it’s a web server with WebUI specially for views.
- Installation procedure similar to Amabri cluster without cluster configuration.
Installation Steps include:
- Install ambari -server package
- Run ambari -server setup (DB, JDK)
- Configure external LDAP authentication
- Deploy views
- Create + configure view instances
- (Optional) Repeat for each Ambari Server instance
- (Optional) Setup proxy for Ambari Server instances
1. yum install ambari-server.x86_64
2. ambari-server setup
[3] Custom JDK
Path to JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk.x86_64
[3] – MySQL
Enter choice (1): 3
Hostname (localhost): mysqldbhost
Port (3306):
Database name (ambari):ambari-view
(create the database prior to this setup)
yum install mysql-connector-java
# mysql -u root -p
CREATE USER '<AMBARIUSER>'@'%' IDENTIFIED BY '<AMBARIPASSWORD>';
GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'%';
CREATE USER '<AMBARIUSER>'@'localhost' IDENTIFIED BY '<AMBARIPASSWORD>';
GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'localhost';
CREATE USER '<AMBARIUSER>'@'<AMBARISERVERFQDN>' IDENTIFIED BY '<AMBARIPASSWORD>';
GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'<AMBARISERVERFQDN>';
FLUSH PRIVILEGES;
- Where
<AMBARIUSER>
is the Ambari user name,<AMBARIPASSWORD>
is the Ambari user password and<AMBARISERVERFQDN>
is the Fully Qualified Domain Name of the Ambari Server host.
If Kerberos configured on Main Ambari cluster and Linux systems
RHEL/CentOS/Oracle Linux
yum install krb5-workstation
SLES
zypper install krb5-client
Ubuntu/Debian
apt-get install krb5-user krb5-config
Use below steps to configure Kerberos.
When a cluster is enabled for Kerberos, the component REST endpoints (such as the YARN ATS component) require SPNEGO authentication.
Depending on the Services in your cluster, Ambari Web needs access to these APIs. As well, views such as the Tez View need access to ATS. Therefore, the Ambari Server requires a Kerberos principal in order to authenticate via SPNEGO against these APIs. This section describes how to configure Ambari Server with a Kerberos principal and keytab to allow views to authenticate via SPNEGO against cluster components.
- Create a principal in your KDC for the Ambari Server. For example, using kadmin:
addprinc -randkey ambari-server@EXAMPLE.COM
- Generate a keytab for that principal.
xst -k ambari.server.keytab ambari-server@EXAMPLE.COM
- Place that keytab on the Ambari Server host. Be sure to set the file permissions so the user running the Ambari Server daemon can access the keytab file.
/etc/security/keytabs/ambari.server.keytab
- Stop the ambari server.
ambari-server stop
- Run the setup-security command.
ambari-server setup-security
- Select
3
for Setup Ambari kerberos JAAS configuration. - Enter the Kerberos principal name for the Ambari Server you set up earlier.
- Enter the path to the keytab for the Ambari principal.
- Restart Ambari Server.
ambari-server restart
Configure external LDAP authentication
Additional links
https://community.hortonworks.com/questions/2652/ambari-ad-lds-ldap-integration.html
https://community.hortonworks.com/questions/2909/how-do-i-automate-the-ambari-ldap-sync.html
https://github.com/seanorama/ambari-bootstrap/blob/master/extras/ambari-ldap-ad.sh
https://gist.github.com/nsabharwal/ed498a4fbcfd75b7dd97
1.2. Configure Ambari to use LDAP Server
#!/usr/bin/env bash## Simply preloading the ambari config with Active Directory
## compatible settings.
##
## You’ll need to update the 1st 3 settings.
##
## Then execute:
## sudo ambari-server setup-ldap
## sudo ambari-server restart
## sudo ambari-agent restart
## sudo ambari-server sync-ldap –allcat <<-‘EOF’ | sudo tee -a /etc/ambari-server/conf/ambari.properties
authentication.ldap.baseDn=dc=abc,dc=com
authentication.ldap.managerDn=cn=ambari-admin,OU=Hadoop,OU=Applications,DC=jci,DC=com
authentication.ldap.primaryUrl=ldap.abc.com:3268
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=distinguishedName
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=group
authentication.ldap.useSSL=false
authentication.ldap.userObjectClass=user
authentication.ldap.usernameAttribute=sAMAccountName
EOF
chmod u+x ambari-ldap-ad.sh
Take the backup of existing amabri.properties file
cp -p /etc/ambari-server/conf/ambari.properties /etc/ambari-server/conf/ambari.properties.date +%m%d%y
sudo ambari-server restart
sudo ambari-agent restart
sudo ambari-server sync-ldap –all
{
“resources” : [
{
“href” : “http://127.0.0.1:8080/api/v1/ldap_sync_events/7”,
“Event” : {
“id” : 7
}
}
]
}
And to see the statuscurl -uadmin:admin http://127.0.0.1:8080/api/v1/ldap_sync_events/7
{
“href” : “http://127.0.0.1:8080/api/v1/ldap_sync_events/7”,
“Event” : {
“id” : 7,
“specs” : [
{
“sync_type” : “existing”,
“principal_type” : “users”
},
{
“sync_type” : “existing”,
“principal_type” : “groups”
}
],
“status” : “COMPLETE”,
“status_detail” : “Completed LDAP sync.”,
“summary” : {
“groups” : {
“created” : 0,
“removed” : 0,
“updated” : 0
},
“memberships” : {
“created” : 0,
“removed” : 0
},
“users” : {
“created” : 0,
“removed” : 0,
“updated” : 0
}
},
“sync_time” : {
“end” : 1464020406077,
“start” : 1464020406052
}
}
}
ambari-server sync-ldap –groups groups.txt
Run below line to sync up.
ambari-server sync-ldap –users users.txt
Using AMBARI API to sync specific users:
curl -uadmin:admin -H ‘X-Requested-By: ambari’ -X POST -d ‘[{“Event”: {“specs”: [{“principal_type”:”users”,”sync_type”:”specific”, “names”: “hdpadmin”}]}}]’ http://127.0.0.1:8080/api/v1/ldap_sync_events
Note: User name depends on ldap-setup attribute “authentication.ldap.usernameAttribute=sAMAccountName”
To create Hive View Instance, need hive cluster properties, here is the details from where to pick them up.
Property | Value |
---|---|
Hive Authentication | auth=NONE;user=${username} |
Scripts HDFS Directory* | /user/${username}/hive/scripts |
Jobs HDFS Directory* | /user/${username}/hive/jobs |
HiveServer2 Host* | Click Hive > Summary > HiveServer2 to view the host name. For example, c6401.ambari.apache.org |
HiveServer2 Thrift port* | Click Hive > Configs > Advanced > General > HiveServer2 Port. For example, 10000 |
WebHDFS FileSystem URI* | Click HDFS > Configs > Advanced > Advanced hdfs-site > dfs.namenode.http-address. When you enter the value in the view definition, pre-pend “webhdfs://” to the value you find in the advanced HDFS configuration settings. For example,webhdfs://c6401.ambari.apache.org:50070 |
YARN Application Timeline Server URL* | Click YARN > Configs > Advanced > Application Timeline Server > yarn.timeline-service.webapp.address. When you enter the value in the view definition, pre-pend “http://” to the value you find in the YARN advanced configuration settings. For example,http://c6401.ambari.apache.org:8188 |
YARN ResourceManager URL* | Click YARN > Configs > Advanced > Advanced yarn-site > yarn.resourcemanager.webapp.address. When you enter the value in the view definition, pre-pend “http://” to the value you find in the YARN advanced configuration settings. For example,http://c6401.ambari.apache.org:8088 |
For Name Node
Property | Value |
---|---|
First NameNode RPC Address orSecond NameNode RPC Address | Select the primary or secondary NameNode to view settings from that host in the cluster. See how to get the NameNode RPC address. When you enter the value in the view definition, pre-pend “http://” to the value you find in the advanced hdfs-site settings. For example, http://c6401.ambari.apache.org:8020 |
First NameNode HTTP (WebHDFS) Address or Second NameNode HTTP (WebHDFS) Address | Click HDFS > Configs > Advanced > Advanced hdfs-site > dfs.namenode.http-address. When you enter the value in the view definition, pre-pend “http://” to the value you find in the advanced hdfs-site settings. For example, http://c6401.ambari.apache.org:50070
|
Hive Views need proxy user to setup:
https://developer.ibm.com/hadoop/2015/10/28/use-ambari-hive-view-write-execute-debug-hive-queries/
1. Set up an HDFS proxy user for the Ambari daemon user account:
To allow the process user to impersonate the logged in user, setup a proxy user for root.
From the Ambari dashboard, navigate to Advanced tab on the HDFS service Configs tab.
Expand the Custom core-site section and add the following two new properties:
hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=*
2. Create the /user/admin folder on HDFS:
Since the view stores user metadata in HDFS under the /user/<logged-in-user> folder, create this folder if it does not exist. To do this, execute the following commands as the hdfs user (admin is the logged-in user in this case.) :
Now that the cluster configuration is complete, we are ready to create an instance of the view.
Creating a view instance:
1. Navigate to admin->Manage Ambari. This takes you to the Ambari admin view, and to see the pre-deployed views, click on Views link under section Views.
2. Expand HIVE and click on Create Instance to open up the Create Instance UI. In the Details section, specify the instance name, display name and description for the view.
3. Specify settings to match the Hive configuration you have in the Settings section. In the example used in this post, Hive uses the default authentication mode, so we will use the defaults and no changes need to be made to the view settings.
4. You can select the local Ambari managed cluster or specify a Custom url in the Cluster Configuration section. We will use the local cluster for this post.
Click on Save. You will see a popup that notifies you that the view was successfully created.
Note: Before executing the Hive view, make sure to grant permissions to the appropriate users and groups. For the example in this post, we will grant user ‘ambari-qa’ permissions to use the view.
To do this, open up the definition of HiveView and specify the user in the Permissions section.
APACHE Ambari Views (Hellow View)
HelloWorld View Example Document on github
https://github.com/apache/ambari/blob/trunk/ambari-views/examples/helloworld-view/docs/index.md