HADOOP: hdfs quotas

Two Types of Quotas:

  1. Name Quota – limit number of files/directories
  2. Space Quota – limits size(storage) of the directory

hdfs dfsadmin -help|grep quota
-setQuota <quota> <dirname>…<dirname>: Set the quota <quota> for each directory <dirName>.
The directory quota is a long integer that puts a hard limit
For each directory, attempt to set the quota. An error will be reported if
Note: A quota of 1 would force the directory to remain empty.
-clrQuota <dirname>…<dirname>: Clear the quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
It does not fault if the directory has no quota.
-setSpaceQuota <quota> [-storageType <storagetype>] <dirname>…<dirname>: Set the space quota <quota> for each directory <dirName>.
The space quota is a long integer that puts a hard limit
a 1GB file with replication of 3 consumes 3GB of the quota.
For each directory, attempt to set the quota. An error will be reported if
The storage type specific quota is set when -storageType option is specified.
-clrSpaceQuota [-storageType <storagetype>] <dirname>…<dirname>: Clear the space quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
It does not fault if the directory has no quota.
The storage type specific quota is cleared when -storageType option is specified.

Examples:

SPACE QUOTA

 $hdfs dfsadmin -setQuota 10 /user/test
$hadoop fs -count -q /user/test
10               7            none             inf            2            1          209715200 /user/test

for i in 1 2 3 4 5 6 7 8 9;do hdfs dfs -mkdir /user/test/file_$i; done
mkdir: The NameSpace quota (directories and files) of directory /user/test is exceeded: quota=10 file count=11
mkdir: The NameSpace quota (directories and files) of directory /user/test is exceeded: quota=10 file count=11
mkdir: The NameSpace quota (directories and files) of directory /user/test is exceeded: quota=10 file count=11

Let’s clear the name quota:

$hdfs dfsadmin -clrQuota /user/test
$hadoop fs -count -q -h /user/test
none             inf            none             inf            9            1              200 M /user/test

SPACE QUOTA

hdfs dfsadmin -setSpaceQuota 300m /user/test
$ hadoop fs -count -q -h /user/test
none             inf           300 M          -100 M            9            1              200 M /user/test

hdfs dfs -put /tmp/lfile /user/test

put: The DiskSpace quota of /user/test is exceeded: quota = 314572800 B = 300 MB but diskspace consumed = 343932928 B = 328 MB

Let’s clear the space quota

hdfs dfsadmin -clrSpaceQuota /user/test
$ hadoop fs -count -q -h /user/test
none             inf            none             inf            2            0                  0 /user/test

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html

Advertisements

Author: rajukv

Hadoop(BigData) Architect and Hadoop Security Architect can design and build hadoop system to meet various data science projects.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s