The final step in the ETL process is to load the transformed data into the target datastore. For this example, I have created a landing database on the same SQL Server, but this final target could be anywhere data can be stored. Given that much of…

With landed data and the first pass at type analysis completed. It is time to take a better look at the current state of the data. Since this data set is quite small and I was on the edge of my SQL knowledge I performed this manually. Creating a…

As discussed in previous posts (ETL and Extracting with Bulk Insert), the first step in the ETL process is to land the data. Once landed in SQL Server, it is best practice to never assume the landing schema will be the final target schema. Since this…

Much like web developers will create an API endpoint that performs a specific function. DBAs have a similar tool in the toolbox, a stored procedure. Much like a single API endpoint, a stored procedure is a reusable piece of script that can be exposed…

With the CSV dataset landed in SQL server the next step is to understand what type of data is in the dataset. While this can be done manually, we are developers why not perform these tasks programmatically? To be successful in this step of the…

For this ETL journey, I will be using Microsoft tooling. More specifically Microsoft SQL Server. I have already identified the data set for this learning experience. The data can be downloaded from Kaggle as a set of comma-separated value (CSV) files…

No matter what your role is in the IT domain, you will have some task that requires you to do some with a data set. There is an ever growing subset of positions in the IT domain that are solely roles in manipulating or interpreting data. As one might…

I felt the need to add paging to the blog list. As I started down this path, I knew that during the implementation of this feature there would be something that I did not yet know. In past Gatsby implementations and components, I have used a graphql…

Implementation speed is critical in our domain. Beating the competition to market is what will bring you and your organization to the top of the pile. If you have ever written an API using ExpressJS you will understand why the framework is so popular…

There are times when API responses are too large. In my experience, I have seen this in 2 flavors. The most common is a data model that grows and grows to a large number of properties. Increasing the amount of data being provided to the client on…

When everyone was sent home in early 2020 my team, like many other teams, struggled to keep the communication going as it had been in the office. Initially, there was a thought the stay-at-home orders were going to be a short-term thing. The missing…

Raise your hand if you have ever implemented a trigger in some database technology ✋. What did you think of that experience? Did you hit a wall with the ability that database triggers offer? What if you wanted to signal another system that data was…