Insufficient permissions to delete bucket

I was recently playing with the S3 service and I noticed that I had a bucket that I no longer needed. It was created by Elastic Beanstalk when I was playing with the service earlier.

The error is actually quite simple to understand, it's clear that based on what we see below, the issue is with permissions.