PIG (transforming data)

FD = load ‘/user/horton/flightdelays/flight_delays?.csv’ using PigStorage(‘,’) as (Year, Month, DayofMonth, DayOfWeek, DepTime, CRSDepTime, ArrTime, CRSArrTime,
UniqueCarrier, FlightNum, TailNum, ActualElapsedTime, CRSElapsedTime, AirTime,
ArrDelay, DepDelay, Origin, Dest, Distance, TaxiIn, TaxiOut, Cancelled,
CancellationCode, Diverted, CarrierDelay, WeatherDelay, NASDelay, SecurityDelay,
LateAircraftDelay);

FDF = filter FD by DepTime != ‘NA’;
FDOUT = foreach FDF generate Year, Month, DayofMonth, DepTime, UniqueCarrier, FlightNum, ArrDelay, Origin, Dest;

store FDOUT INTO ‘/user/horton/flightdelays_clean’ using PigStorage (‘,’);

2015-12-04 15:18:06,911 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl – Timeline service address: http://namenode:8188/ws/v1/timeline/
2015-12-04 15:18:06,911 [main] INFO  org.apache.hadoop.yarn.client.RMProxy – Connecting to ResourceManager at namenode/172.17.0.2:8050
2015-12-04 15:18:06,916 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate – Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-12-04 15:18:06,955 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – Success!

hdfs dfs -tail /user/horton/flightdelays_clean/part-m-00000
2008,1,6,1845,WN,225,90,LAX,RNO
2008,1,6,1644,WN,2360,153,LAX,RNO
2008,1,6,1220,WN,2624,14,LAX,SAT
2008,1,6,900,WN,469,-16,LAX,SFO
2008,1,6,2000,WN,593,6,LAX,SFO
2008,1,6,1624,WN,618,2,LAX,SFO
2008,1,6,1946,WN,646,89,LAX,SFO
2008,1,6,1549,WN,656,76,LAX,SFO

 

Advertisements

Author: rajukv

Hadoop(BigData) Architect and Hadoop Security Architect can design and build hadoop system to meet various data science projects.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s