The Azure Data Lake Connector will crawl content from an Azure Data Lake Storage Gen2 for either all file systems or an specified file system and paths.

Azure Data Lake Storage is a comprehensive, scalable, and efficient data lake solution designed for big data analysis, and it provides a hierarchical file system. It brings the capabilities of Azure Data Lake Storage Gen1 together with the Azure Blob storage.

For more information about the Azure Data Lake Storage Gen 2, see the official Microsoft Overview of Azure Data Lake Storage Gen2 documentation.

The Azure Data Lake Connector supports crawling the following repositories

Before installing the Azure Data Lake connector, make sure that:

• You have created the necessary service-to-service application account with pertinent access to your data lake.

• The Azure Data Lake is up and running.

• You have admin rights to allow Read and Execute permissions on the folders to crawl.

To access the Azure Data Lake, an application account with sufficient privileges must be supplied. The following fields must be configured to set up a new data lake connection:

• Storage Account Name

• Application ID

• Application Secret (Application Key)

• Tenant ID

Following are the steps on how to get the required credentials:

1. See Microsoft's Use portal to create an Azure Active Directory application and service principal that can access resources for the steps on how to create an application ID properly, its key (client secret), and Tenant ID. Make sure to write down your application key at the time of creation. It will not be shown again after you exit the portal. Important: make sure to grant the necessary access to your application. See the Microsoft page Assign an Azure Role.

2. Make sure to grant Read and Execute access (at least) to files and folders to crawl. See Microsoft's Manage Access Control documentation. To recursively apply the same parent folder permissions to sub-folders, please see the "Apply an ACL recursively" section.

The Azure Data Lake connector has the following features:

• Performs incremental crawling (so that only new/updated documents are indexed)

• Fetches Object ACLs (Access Control Lists) for Azure document-level security

• Runs from any machine with access to the given Data Lake source

• Service-to-Service Authentication via OAuth 2.0 token

The Azure Data Lake connector can crawl the following objects: