Chathura Herath 's Blog

My Photo
Name:
Location: Bloomington, Indiana, United States

Tuesday, May 04, 2010

Amazon Elastic Map Reduce: API call to find and terminate job flows without Amazon Console

Recently one of the interns in the lab asked me how can he kill the running job flows in Amazon Elastic Map Reduce so he will not run up a big bill. This is straight forward to do in Amazon Management Console, but in this case the student didn't have the username/password to the amazon account because he was using a community account (although he had access to the accesskeys). Another application would be to write a cron job that checks the left over Job Flows and their EC2 instances every night so being able to do this programmatically helps.

Terminating a Job flow is straight forward using the client API that Amazon has provided and a sample can be found here. Now the issue is how to find the running Job Flows using API calls. The DescribeJobFlow API may give the impression that one need to give the Job Flow id to begin with to query the Job Flow Status, but my testing shows that if we don't specify any Job Flow ids in the request as shown in the sample here, it would return the Job Flows with the most recent activities. If there are Job Flows that are running those would have most recent activities. So once you have these two its easy to write a piece of code that would glue these two together to kill all the existing Job Flows.

Thats it!!!

Labels: , , , , , ,