migrate to azure data factory version 2

i recently migrated my clash of clans daily export from the data factory to data factory version 2. here was my experience

Technology
Hobby Gaming Azure Azure Data Factory
November 16, 2019

I've been playing clash of clans (mobile game) for many years now. As I kept playing, I got to know my clan well. I moved up the ranks rather quickly. As soon as I became a co-leader of my clan, I realized I could contribute to the team using some of my skills. Another member of the clan had some extensive stats on each of the clan members, wars results, other clans and just good data. I decided to help him out, I got in touch with him and told him a little about my background and that I could help automate some of his manual processes.

The first thing I needed to do was build a daily ETL (Extract-Transform-Load) process. There are many ways to go about this, but I wanted something cheap and quick. At the same time Azure was new and I thought what better way to get some experience than to build something in Azure. I spent some time looking at what my options were and found the data factory. I was going to be talking to the clash of clans API and store the data into a Azure SQL database. So, it was the perfect product. I signed up for an account and got to work.

I had a SQL database built out in about 5 minutes. I choose a basic database because I'm not planning to have a high amount of use. Once a day it will get hit and then some small hits here or there to view data. The total cost would be $5 a month! Not too bad for me to store some great data. Next step was to build a data factory. This was a bit more complicated, but I was able to build one rather quickly because it used JSON for putting together the inputs and outputs. The best part was it has a built-in scheduler to let it run daily!

After a few years of it running successfully I saw that V2 had come out. It was time to migrate to V2, which I wasn't happy to find out wasn't a simple click of a button. I waited a few months to see if that feature would be built in, but no luck. I decided to just bite the built and rebuild. I'm glad I did, it now had a nice interface that I got to try out. I was able to rebuild the data factory in about 10 minutes. I tested it and it was perfect, very easy to do.

Here is what I like:
  • Easy source control integration to my Azure Dev Ops source control
  • The UI is solid; it is easy to understand and acts just like a standard application.
  • Easy to modify when future updates need to be made
  • Scheduling is easier and more robust.

This data factory now allows me to track stats on each player of our clan daily (consistent time every day). It will make a call to the clash of clans api and pull back all stats on each member of our clan. I'm now able to use that data to track a member's history. We can use the data to better plan wars and track participation.

Scroll through a few of the screen shots I captured of the Azure setup. I highly recommend using Azure for your next ETL project.