Hadoop Kerberos Guide

A nice guide on hadoop Kerberos guide by Dillon Woods

Hadoop Kerberos Guide

Advertisements

MYSQL Backup using mysqldump

  1. Create a script “mysql_backup.sh” and set in cron to run the jobs.

0 3 * * * /var/lib/mysql/mysql_backup.sh >> /var/log/mysql_backup/mysql_backup.log 2>> /var/log/mysql_backup/mysql_backup.err

2. SCRIPT:

#!/bin/bash
#Script to run on cron for taking mysql db backups
#Author: rajukv
#Date:
Host=slavemysqlserver.mydomain.com
BKPDir=/logs/mysql_backup
USER=mysql_bkpuser
PASS=’*******’
LOG=/var/log/mysql_backup/mysql_backup.log
LOG_ERROR=/var/log/mysql_backup/mysql_backup.log

#Set MYSQL Dump Command
Dump=”/usr/bin/mysqldump -u $USER -p$PASS -h $Host –skip-extended-insert –force”
#Set MYSQL DUMP COMMAND with –skip-lock-tables options
Dump_skip_lock_tables=”/usr/bin/mysqldump -u $USER -p$PASS -h $Host –skip-extended-insert –force –skip-lock-tables”
MySQL=/usr/bin/mysql

Today=$(date “+%a”)
echo ‘>>>Starting mysql backup at `date`’

# Get a list of all databases
Databases=$(echo “SHOW DATABASES” | $MySQL -u $USER -p$PASS -h $Host|grep -v Database)

for db in $Databases; do
date=`date`
file=”$BKPDir/$Host-$db-$Today.sql.gz”
echo “Backing up ‘$db’ from ‘$Host’ on ‘$Today’ to: ”
echo ” $file”
if [ “$db” == “information_schema” ] || [ “$db” == “performance_schema” ]; then
$Dump_skip_lock_tables -h $Host $db | gzip > $file
else
$Dump -h $Host $db | gzip > $file
fi
done
echo ‘>>>End of mysql backup at `date`’

3. Backup file verification

-rw-r–r– 1 mysql mysql    6424169 Sep 18 03:00 slavemysqlserver.mydomain.com-information_schema-Sun.sql.gz

-rw-r–r– 1 mysql mysql 546263 Sep 18 03:02 slavemysqlserver.mydomain.com-performance_schema-Sun.sql.gz

-rw-r–r– 1 mysql mysql      28365 Sep 18 03:04 slavemysqlserver.mydomain.com-sys-Sun.sql.gz

 

APACHE RANGER Version 0.6 has tag based Policies

APACHE RANGER Version 0.6 has tag based Policies for more details read 

Key Features are:

  1. TAG STOREDetails of tags associated with resources are stored in a tag store. Apache Ranger plugins retrieve the tag details from the tag store for use during policy evaluation. To minimize the performance impact during policy evaluation (in finding tags for resources), Apache Ranger plugins cache the tags and periodically poll the tag store for any changes. On detecting change, the plugins update the cache. In addition, the plugins store the tag details in a local cache file – just as the policies are stored in a local cache file. On component restart, the plugins will use the tag data from the local cache file if the tag store is not reachable.In the current release, Apache Ranger plugins download the tag details from the store managed by Ranger Admin. Ranger Admin persists the tag details in its policy store and provides a REST interface for the plugins to download the tag details
  2. TAG SYNCApache Ranger introduces a new module, ranger-tagsync, to populate the tag store from the tag details available in an external system like Apache Atlas.  Tag sync is a daemon process similar to ranger-usersync process.In the current release, ranger-tagsync supports receiving tag details from Apache Atlas via change notifications. As tags are added/updated/deleted to resources in Apache Atlas, ranger-tagsync would receive notifications and update the tag store.
  3. Ranger Admin UI

Apache Ranger provides a new UI page, named ‘Tag Based Policies’, to work with tag based policies. The workflow to create/update tag-based policies is essentially same as with the existing ‘Resource Based Policies’. Start by adding a tag service instance, in which tag-based policies can be created. Multiple tag service instances can be created – like tag-dev/tag-test/tag-prod, to group tag-based policies for different clusters.

Policy UI for tag-based policy looks very similar to existing resource-based policies. The name of the tag should be specified at the top half of the page; the bottom half of the page provides the UI to specify permissions for users and groups. Following are few differences from resource-based policies UI:

  • Permissions UI lists the permissions available in all the service-types. This allows policy authors to restrict type of accesses users/groups can perform on tagged resources
  • Wildcards are not allowed in tag names. Also only one tag can be entered per policy
  • Delegated Admin is not available for tag-based policies. Currently only an administrator can work with tag-based policies

4. TAG ATTRIBUTES

Tags in Apache Ranger can have attributes. Tag attribute values can be used in Ranger tag-based-policies to influence the authorization decision.

For example, to deny access to a resource after a specific date:

  • add EXPIRES_ON tag to the resource
  • add a tag attribute, named expiry_date, with its value set to the expiry date
  • create a Ranger policy for EXPIRES_ON tag
  • add a condition in this policy to deny the access when the date specified in expiry_date tag attribute is later than the current date

In fact, the above detailed EXPIRES_ON tag policy is created as the default policy in tag service instances.

5. TAGS IN POLICY EVALUATION

 

Ranger-Policy-Evaluation-Flow-with-Tags