Prettier Diffing of Terraform policy changes

I love Terraform. It makes managing complex infrastructure and keeping track of what changes are happening infinitely easier........except if you are changing AWS IAM policies.

UPDATE: Someone has made an excellent tool that takes care of this for you!
terraform-landscape

Currently, if you change a single field in policy, you are returned two unformatted, uncolored strings of json. Leaving it up to you to somehow figure out the difference.

For example:

Can you spot the difference? ....because I sure can't.

If I am feeling lazy, I will assume this diff contains only the change I want. However, I quickly noticed that policies are one of the most often changed resources by someone manually. When I would apply these updates without carefully reviewing them, random coworkers would show up at my desk with questions like: "Do you know why I just lost access to view our AWS Lambdas?". Doh!

So to be a more resonsible Terraformer, I started formatting and diffing the polices myself in vim. But after doing that a couple times, I knew I needed quicker way. So I whipped up a quick bash script to help with this problem. Heres how to use it.

First copy the policy diff from the Terraform plan, and save it in a file called policy.txt. It should look like this.

"{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"RDSAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"rds:ListTagsForResource\",\n        \"rds:Describe*\",\n        \"ec2:DescribeVpcs\",\n        \"ec2:DescribeSecurityGroups\",\n        \"ec2:DescribeAvailabilityZones\",\n        \"ec2:DescribeAccountAttributes\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"CloudwatchPutMetricDataAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"xray:PutTraceSegments\",\n        \"xray:PutTelemetryRecords\",\n        \"cloudwatch:PutMetricData\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"MonitoringAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"sns:List*\",\n        \"sns:Get*\",\n        \"logs:TestMetricFilter\",\n        \"logs:GetLogEvents\",\n        \"logs:Get*\",\n        \"logs:FilterLogEvents\",\n        \"logs:DescribeLogStreams\",\n        \"logs:Describe*\",\n        \"cloudwatch:List*\",\n        \"cloudwatch:GetMetricStatistics\",\n        \"cloudwatch:Get*\",\n        \"cloudwatch:Describe*\",\n        \"autoscaling:Describe*\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"EcrAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ecr:GetDownloadUrlForLayer\",\n        \"ecr:GetAuthorizationToken\",\n        \"ecr:BatchGetImage\",\n        \"ecr:BatchCheckLayerAvailability\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"RekognitionAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"rekognition:*\",\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"LambdaInvoke\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"lambda:InvokeFunction\",\n        \"lambda:GetFunction\"\n      ],\n      \"Resource\": \"*\"\n    }\n  ]\n}" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"RDSAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"rds:ListTagsForResource\",\n        \"rds:Describe*\",\n        \"ec2:DescribeSecurityGroups\",\n        \"ec2:DescribeAvailabilityZones\",\n        \"ec2:DescribeAccountAttributes\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"CloudwatchPutMetricDataAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"xray:PutTraceSegments\",\n        \"xray:PutTelemetryRecords\",\n        \"cloudwatch:PutMetricData\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"MonitoringAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"sns:List*\",\n        \"sns:Get*\",\n        \"logs:TestMetricFilter\",\n        \"logs:GetLogEvents\",\n        \"logs:Get*\",\n        \"logs:FilterLogEvents\",\n        \"logs:DescribeLogStreams\",\n        \"logs:Describe*\",\n        \"cloudwatch:List*\",\n        \"cloudwatch:GetMetricStatistics\",\n        \"cloudwatch:Get*\",\n        \"cloudwatch:Describe*\",\n        \"autoscaling:Describe*\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"EcrAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ecr:GetDownloadUrlForLayer\",\n        \"ecr:GetAuthorizationToken\",\n        \"ecr:BatchGetImage\",\n        \"ecr:BatchCheckLayerAvailability\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"RekognitionAccess\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"rekognition:*\",\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"LambdaInvoke\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"lambda:InvokeFunction\",\n        \"lambda:GetFunction\"\n      ],\n      \"Resource\": \"*\"\n    }\n  ]\n}"

Next install color-diff
Note: This assumes you are using OSX

brew install color-diff

Now you can use this little script to print out a nice colorized diff of what changed:

cat policy.txt | sed s/=\>/\\n/ | split -l 1;
cat xaa | sed 's/\s\+$//' | sed s/\"// | sed 's/\"$//' | sed 's/\\\n//g' | sed 's/\\\"/\"/g' | python -m json.tool > initial.json;
cat xab | sed 's/\s\+$//' | sed s/\"// | sed 's/\"$//' | sed 's/\\\n//g' | sed 's/\\\"/\"/g' | python -m json.tool > new.json;
diff initial.json new.json | colordiff

For the above example you will see:

We now see that this terraform apply is going to remove ec2:DescribeVpcs from the policy. Much nicer! And now I can apply in peace.