Category Archives: Azure
Azure Data Factory and REST APIs – Managing Pipeline Secrets by a Key Vault
In this post, I will touch a slightly different topic to the other few published in a series. The topic is a security or, to be more precise, the management of secrets like passwords and keys.The pipeline created previously finally works. It ingests data from a third-party REST source and stores it in a data lake. However, […]
Azure Data Factory and REST APIs – Mapping and Pagination
In this blog post I would like to put a light on a mapping and pagination of a Copy activity which are often are requirements for the ingestion of REST data.Mapping is optional for data sources, like relational databases, parquet or csv files, because the incoming dataset contains an inherited structure, so the Data Factory […]
Azure Data Factory and REST APIs – Setting up a Copy activity
In this blog post, I would like to show how to add and configure a Copy Activity which will ingest REST data and store it in a data lake. The REST API used as an example in this post is secured with OAUTH2 authentication. It requires a Bearer token to be obtained and passed as […]
Azure Data Factory and REST APIs – Dealing with oauth2 authentication
In this first post I am going to discuss how to apply oauth2 authentication to ingest REST APIs data. OAUTH2 became a standard de facto in cloud and SaaS services, it used widely by Twitter, Microsoft Azure, Amazon. My example is based on EXACT Online API. The end-target of the blog series is to setup […]
Azure Data Lake Gen2 and SQL Server Integration Services
How it used to be “Azure Data Lake Generation 2” (or ADLS gen2) is the newest cloud data lake offering from a Microsoft. It supposes to bring the best of two worlds together: excelent performance and redundancy of a blob storage and secure filesystem capabilities of a data lake. However, for some time the ADLS […]