All posts by Alex Volok
Azure Data Factory & DevOps – Automated Deployment via Azure CLI
In a coming series of posts, I would like to shift a focus to subjects like automation and DevOps. This post will show how the entire Data Factory environment and surrounding services can be deployed by a command line or be more precise, by the Azure CLI. Such an approach may look tedious and overkilling […]
Azure Data Engineer Certification Tips
Recently I passed exams related to such certification and since then there is a flow of related but often-repeated questions. Over time, they are crystallized into a post in QA format, which hopefully can be helpful for someone who wants to pass this Microsoft certification. So, the questions are: What kind of questions can I […]
Azure Data Factory and REST APIs – Managing Pipeline Secrets by a Key Vault
In this post, I will touch a slightly different topic to the other few published in a series. The topic is a security or, to be more precise, the management of secrets like passwords and keys.The pipeline created previously finally works. It ingests data from a third-party REST source and stores it in a data lake. However, […]
Azure Data Factory and REST APIs – Mapping and Pagination
In this blog post I would like to put a light on a mapping and pagination of a Copy activity which are often are requirements for the ingestion of REST data.Mapping is optional for data sources, like relational databases, parquet or csv files, because the incoming dataset contains an inherited structure, so the Data Factory […]
Azure Data Factory and REST APIs – Setting up a Copy activity
In this blog post, I would like to show how to add and configure a Copy Activity which will ingest REST data and store it in a data lake. The REST API used as an example in this post is secured with OAUTH2 authentication. It requires a Bearer token to be obtained and passed as […]