After publishing the application, I had quite a few people contact me asking how I did it. So, I figured I’d write up an article on how it works. So here goes.
First, the platform is built on Amazon Web Services (AWS). Here’s a list of the AWS services it uses:
- Simple Queueing Service (SQS)
- Relational Database Service (RDS)
- Elastic Compute Cloud (EC2), and
- Simple Storage Service (S3)
Here’s how it works. Once a day, a Python script grabs the latest CSV schedules from the Town’s Open Data website. The script saves a copy of the schedules to an S3 bucket for processing. I’ve added a Life Cycle Management (LCM) rule to make sure that the bucket doesn’t keep all the CSV files. After the files are saved to S3 the script adds a job to one of the three SQS queues (one for each activity type – swimming, fitness, or skating).
Another Python script watches the SQS queues for jobs. When it sees that there is work to do it parses the CSV files and updates tables in a simple MySQL database running in RDS.
That’s pretty much it. The project was more of an exploration of using application services from AWS and improving my Python scripts.