HADOOP ISSUES – dncp_block_verification.log.prev too large

dncp_block_verification.log.prev too large

Seen this bug in hdp2.3.2

Using hadoop version : Hadoop 2.0.0-cdh4.7.0

can see on one datanode, dncp_block_verification.log.prev is too large.

Is it safe to delete this file?

-rw-r--r-- 1 hdfs hdfs 1166438426181 Oct 31 09:34 dncp_block_verification.log.prev
-rw-r--r-- 1 hdfs hdfs     138576163 Dec 15 22:16 dncp_block_verification.log.curr

This is similar to HDFS-6114 but that is for dncp_block_verification.log.curr file.

workaround:

Reference:

HDFS-7532

https://community.hortonworks.com/questions/40595/hdfs-datanode-one-volume-full.html

Advertisements

HADOOP: hdfs quotas

Two Types of Quotas:

  1. Name Quota – limit number of files/directories
  2. Space Quota – limits size(storage) of the directory

hdfs dfsadmin -help|grep quota
-setQuota <quota> <dirname>…<dirname>: Set the quota <quota> for each directory <dirName>.
The directory quota is a long integer that puts a hard limit
For each directory, attempt to set the quota. An error will be reported if
Note: A quota of 1 would force the directory to remain empty.
-clrQuota <dirname>…<dirname>: Clear the quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
It does not fault if the directory has no quota.
-setSpaceQuota <quota> [-storageType <storagetype>] <dirname>…<dirname>: Set the space quota <quota> for each directory <dirName>.
The space quota is a long integer that puts a hard limit
a 1GB file with replication of 3 consumes 3GB of the quota.
For each directory, attempt to set the quota. An error will be reported if
The storage type specific quota is set when -storageType option is specified.
-clrSpaceQuota [-storageType <storagetype>] <dirname>…<dirname>: Clear the space quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
It does not fault if the directory has no quota.
The storage type specific quota is cleared when -storageType option is specified.

Examples:

SPACE QUOTA

 $hdfs dfsadmin -setQuota 10 /user/test
$hadoop fs -count -q /user/test
10               7            none             inf            2            1          209715200 /user/test

for i in 1 2 3 4 5 6 7 8 9;do hdfs dfs -mkdir /user/test/file_$i; done
mkdir: The NameSpace quota (directories and files) of directory /user/test is exceeded: quota=10 file count=11
mkdir: The NameSpace quota (directories and files) of directory /user/test is exceeded: quota=10 file count=11
mkdir: The NameSpace quota (directories and files) of directory /user/test is exceeded: quota=10 file count=11

Let’s clear the name quota:

$hdfs dfsadmin -clrQuota /user/test
$hadoop fs -count -q -h /user/test
none             inf            none             inf            9            1              200 M /user/test

SPACE QUOTA

hdfs dfsadmin -setSpaceQuota 300m /user/test
$ hadoop fs -count -q -h /user/test
none             inf           300 M          -100 M            9            1              200 M /user/test

hdfs dfs -put /tmp/lfile /user/test

put: The DiskSpace quota of /user/test is exceeded: quota = 314572800 B = 300 MB but diskspace consumed = 343932928 B = 328 MB

Let’s clear the space quota

hdfs dfsadmin -clrSpaceQuota /user/test
$ hadoop fs -count -q -h /user/test
none             inf            none             inf            2            0                  0 /user/test

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html

Hadoop: FAQ on hdfs snapshot

$ hdfs dfs -mkdir /user/test
$ hdfs dfs -put output23.txt /user/test
$ hdfs dfs -du /user/test
67335  /user/test/output23.txt
$ hdfs dfsadmin -allowSnapshot /user/test
Allowing snaphot on /user/test succeeded
$hdfs dfs -createSnapshot /user/test
Created snapshot /user/test/.snapshot/s20150924-050641.662

How to recover the file from .snapshot directory ?

$hdfs dfs -rm /user/test/output23.txt
15/09/24 05:08:31 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.
Moved: ‘hdfs://sandbox.hortonworks.com:8020/user/test/output23.txt’ to trash at: hdfs://sandbox.hortonworks.com:8020/user/hdfs/.Trash/Current

Confirm File is deleted:
 hdfs dfs -ls /user/test/output23.txt
ls: `/user/test/output23.txt’: No such file or directory

hdfs dfs -cp /user/test/.snapshot/s20150924-050641.662/output23.txt /user/test

15/09/24 05:12:02 WARN hdfs.DFSClient: DFSInputStream has been closed already
$ hdfs dfs -ls /user/test/output23.txt
-rw-r–r–   1 hdfs hdfs      67335 2015-09-24 05:12 /user/test/output23.txt

Can we delete files from .snapshot copy ?

Try to delete file from .snapshot
As .snapshot is a readonly copy it will not allow to delete the files.

hdfs dfs -rm /user/test/.snapshot/s20150924-050641.662/output23.txt
rm: Failed to move to trash: hdfs://sandbox.hortonworks.com:8020/user/test/.snapshot/s20150924-050641.662/output23.txt: “.snapshot” is a reserved name.

How to list snapshottable directories ?

hdfs lsSnapshottableDir
drwxr-xr-x 0 hdfs hdfs 0 2015-09-24 05:12 1 65536 /user/test

How to rename a snapshot ?

hdfs dfs -renameSnapshot /user/test s20150924-050641.662 b4patch.24sep15

$ hdfs dfs -ls /user/test/.snapshot
drwxr-xr-x   – hdfs hdfs          0 2015-09-24 05:06 /user/test/.snapshot/b4patch.24sep15

How to show difference between two snapshots ?

$ touch testfile
$ hdfs dfs -put testfile /user/test
$ hdfs dfs -ls /user/test
Found 2 items
-rw-r–r–   1 hdfs hdfs      67335 2015-09-24 05:12 /user/test/output23.txt
-rw-r–r–   1 hdfs hdfs          0 2015-09-24 05:32 /user/test/testfile

$ hdfs dfs -createSnapshot /user/test
Created snapshot /user/test/.snapshot/s20150924-053313.181
$ hdfs dfs -renameSnapshot /user/test s20150924-053313.181 afterPatch.24sep15

$ hdfs snapshotDiff /user/test b4patch.24sep15 afterPatch.24sep15
Difference between snapshot b4patch.24sep15 and snapshot afterPatch.24sep15 under directory /user/test:
M    .
+    ./output23.txt
+    ./testfile
–    ./output23.txt

How to delete the snapshot ?

hdfs dfs -deleteSnapshot /user/test afterPatch.24sep15
hdfs dfs -deleteSnapshot /user/test b4patch.24sep15

Reference:

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html#Delete_Snapshots