Using DistCp with Amazon S3
S3 credentials can be provided in a configuration file (for example, core-site.xml):
<property> <name>fs.s3a.access.key</name> <value>...</value> </property> <property> <name>fs.s3a.secret.key</name> <value>...</value> </property>
hadoop distcp -Dfs.s3a.access.key=myAccessKey -Dfs.s3a.secret.key=mySecretKey hdfs://MyNameservice-id/user/hdfs/mydata s3a://myBucket/mydata_backup
Using DistCp with Microsoft Azure (WASB)
Configure connectivity to Azure by setting the following property in core-site.xml.
<property> <name>fs.azure.account.key.youraccount.blob.core.windows.net</name> <value>your_access_key</value> </property>
hadoop distcp wasb://<sample_container>@<sample_account>.blob.core.windows.net/ hdfs://hdfs_destination_path