really x loco

Record Nerd

Friday, September 4, 2020 · 9:02PM

If you didn't know, I collect records. Not "vinyls." No, please, never that. I've been collecting since maybe late 2001 or around 2002, essentially whenever I got into hardcore because that's what hardcore kids do. They collect records. Well, at least some of us still do.
Also, if you didn't know, my profession is in web development. I guess you could call me a software engineer, but that doesn't really sit well with me. I don't know why. Maybe another blog for another time. Sometimes work isn't very fulfilling. So I work on projects. Or try to. It's really hard to come up with ideas. It's really not the idea that I'm after (most times) but the learning experience.
Earlier this year, I came up with a project to mix those two things. Prior to this, I've kinda had this idea for a "collection" app where you could keep track of whatever collection of things you wanted. My initial thought is that it would help me keep track of movies I owned (yes, I have purchased multiple copies of movies I hadn't even opened). I guess it would have been like a non-spreadsheet spreadsheet. I dunno.
I've also been wanting to keep track of my records somehow. For some reason, I was hesistant to use the Discogs. After some time of not doing anything, I finally gave in and added my collection. Mostly. It's actually really fun and satisfying to just dig through my stuff. It's also disappointing to find I don't have things I thought I did (miss you Hope Con Cold Blue and first 7" :cry-emoji:).
On top of this, Discogs has an API, so that led to this:
The main thing I wanted to pick up when I started working on this was CDK, AWS's Cloud Development Kit. I'll try to sum up as best I can. Basically, if you want to orchestrate your AWS resources, you would typically write some Cloudformation templates, which aren't really too difficult to work with. CDK is kind of a wrapper to Cloudformation, letting you create and deploy templates with ~~your~~Amazon's languages of choice, including Javascript, Python, and Java. Besides just being able to build simple CF templates, you can parameterize your deployments, like your DynamoDB table in your staging environment can be PAY_PER_REQUEST, but your production environment should be PROVISIONED. You wouldn't need to write two separate templates for that.
Anyway, I wasn't trying to do anything fancy like that. I just wanted to get familiar with CDK since we were starting to use it more at work. So, my back end is two lambdas and a DynamoDB table.
The first lambda would read in my collection from the Discogs API and save it to the table. Initially, it was just a normal flow—query the API, get the data in the shape I wanted, then write to the table. Recently, just for kicks, I refactored it to use RxJS. Why? Just because. This really isn't the right use-case, I think. It was an interesting way to approach the problem though. The reader also makes use of Step Functions. The Discogs API is borderline unusable because you're limited to 60 requests per second. My collection was taking about 13ish minutes to process 800+ records. Lambdas have a maximum execution time of 15 minutes, so not ideal. So the step function lets me run a chunk at a time and just repeats until it's done.
The second lambda sits behind an API gateway, but it only reads from the table. Really nothing fancy with this.
My CDK setup creates the DynamoDB table, deploys the lambdas, sets up the step function, and sets up the API gateway. I originally also had a Codepipeline going, but now I cut that part out (saves some money). I also got around to adding some unit tests for the lambdas, so I think this is a pretty solid system. I don't have a great way of updating my collection; currently, I'm just destroying the table, doesn't sound too bad, but that stack is coupled with the lambdas, so I essentially have to tear everything down. Less than optimal for sure.
On the front end, it's just a React app. Nothing too fancy going on. I'm using Material UI for the layout. It's pretty basic, but it works. I'm not sure what else I want to do with it. I think this was my first serious React project, and I'm happy with it.
Anyway, that's my show-and-tell.

record collecting