hadoop: distcp

  1. Internal cluster copy between two hdfs folders

hadoop distcp /tmp/output23.txt /demo

2. Internal cluster copy using hdfs url

hadoop distcp hdfs://sandbox:8020/demo hdfs://sandbox:8020/apps

File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=640985
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=953979
HDFS: Number of bytes written=948926
HDFS: Number of read operations=179
HDFS: Number of large read operations=0
HDFS: Number of write operations=41
Job Counters
Launched map tasks=5
Other local map tasks=5
Total time spent by all maps in occupied slots (ms)=283995
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=56799
Total vcore-seconds taken by all map tasks=56799
Total megabyte-seconds taken by all map tasks=58162176
Map-Reduce Framework
Map input records=20
Map output records=0
Input split bytes=575
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=1634
CPU time spent (ms)=9360
Physical memory (bytes) snapshot=683446272
Virtual memory (bytes) snapshot=4112478208
Total committed heap usage (bytes)=659030016
File Input Format Counters
Bytes Read=4478
File Output Format Counters
Bytes Written=0
org.apache.hadoop.tools.mapred.CopyMapper$Counter
BYTESCOPIED=948926
BYTESEXPECTED=948926
COPY=20

Advertisements

Author: rajukv

Hadoop(BigData) Architect and Hadoop Security Architect can design and build hadoop system to meet various data science projects.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s