PIG how to run as Script and store output into a file ?

Open a file in vi editor “count.pig” and provide pig procedure.

vi count.pig

FDCLN = load ‘/user/horton/flightdelays_clean/part*’ using PigStorage(‘,’);
FDCLN_G = GROUP FDCLN ALL;
FDCLN_CNT = foreach FDCLN_G GENERATE COUNT(FDCLN.$0);
store FDCLN_CNT INTO ‘/user/horton/cleaned_total’ using PigStorage();

Now run the count.pig

pig count.pig

If this script runs without error, then it stores the output in “/user/horton/cleaned_total/”

hdfs dfs -ls cleaned_total/
Found 2 items
-rw-r–r–   3 horton hdfs          0 2015-12-05 12:58 cleaned_total/_SUCCESS
-rw-r–r–   3 horton hdfs          6 2015-12-05 12:58 cleaned_total/part-r-00000

 

 

 

Advertisements

Author: rajukv

Hadoop(BigData) Architect and Hadoop Security Architect can design and build hadoop system to meet various data science projects.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s