Azure Blob Storage

Nextflow includes support for Azure Blob Storage. You can access files stored in a blob container in your pipeline script like any other file in the local file system.

Azure Storage provides several services for storing and accessing data in Azure. Nextflow supports Azure Blob Storage for most workloads. Limited support is available for Azure file shares when using the Azure Batch executor.

Blob Storage paths

To access a Blob Storage file, prefix the file path with the az:// scheme and the name of the blob container where it is stored.

For example, to access the file /data/sequences.fa stored in a container named my-container, use the following fully qualified path:

az://my-container/data/sequences.fa

You can apply the usual file operations to a path handle with this notation. For example, to print the content of a Blob Storage file:

println file('az://my-container/data/sequences.fa').text

Specify the storage account in the Nextflow configuration or with the AZURE_STORAGE_ACCOUNT_NAME environment variable:

azure {
    storage {
        accountName = '<storage-account-name>'
    }
}

See Working with files and the Path reference to learn more about available file operations.

Authentication

Nextflow supports three authentication methods for Azure services, listed in order of security and recommended usage:

• Managed identities: The most secure option, available only within Azure. Uses Azure-managed credentials without storing secrets.

• Service principals: A secure and flexible method that works across environments. Uses Microsoft Entra credentials for authentication.

• Access keys: The most basic and least secure method. Relies on direct access keys for authentication.

Managed identities

Added in version 24.10

Managed identity is the recommended authentication method when running Nextflow within Azure. Azure manages the credentials, and you do not need to store secrets.

When using a managed identity, an Azure resource can authenticate based on what it is rather than using access keys. For example, Nextflow can submit jobs to Azure Batch and access private storage accounts without additional credentials if it is running on Azure Virtual Machines with a managed identity and the necessary permissions.

There are two types of managed identities:

• System-assigned managed identity
• User-assigned managed identity

System-assigned managed identity

A system-assigned identity is tied to a specific Azure resource. To use it:

1. Enable system-assigned managed identity on your Azure resource.

2. Configure the required role assignments. See Required permissions for more information.

3. Add the following configuration:

azure {
    managedIdentity {
        system = true
    }
    storage {
        accountName = '<storage-account-name>'
    }
    batch {
        accountName = '<batch-account-name>'
        location = '<batch-account-location>'
    }
}

Replace the following:

• storage-account-name: your Azure storage account name
• batch-account-name: your Azure Batch account name
• batch-account-location: your Azure Batch account location

tip

Nextflow uses the AZURE_MANAGED_IDENTITY_SYSTEM environment variable if the managed identity is not set in the Nextflow configuration file. Set this to true to enable a system-assigned managed identity.

User-assigned managed identity

You can share a user-assigned identity across multiple Azure resources. Its lifecycle is not tied to any specific resource. See the Azure documentation for more information.

To use a user-assigned managed identity:

1. Create a managed identity.

2. Configure the required role assignments.

3. Assign the identity to your Azure resource.

4. Add the following configuration:

azure {
    managedIdentity {
        clientId = '<user-assigned-managed-identity-client-id>'
    }
    storage {
        accountName = '<storage-account-name>'
    }
    batch {
        accountName = '<batch-account-name>'
        location = '<batch-account-location>'
    }
}

Replace the following:

• user-assigned-managed-identity-client-id: your user-assigned managed identity client ID
• storage-account-name: your Azure Storage account name
• batch-account-name: your Azure Batch account name
• batch-account-location: your Azure Batch account location

tip

Nextflow uses the AZURE_MANAGED_IDENTITY_USER environment variable if the managed identity client ID is not provided in the Nextflow configuration file.

Service principals

Service principal credentials provide access to Azure Batch and Storage accounts.

Similar to a managed identity, a service principal is an application identity that can have specific permissions and role-based access. Unlike managed identities, service principals require a secret key for authentication. The secret key is less secure, but you can use it to authenticate with Azure resources from outside the Azure environment, such as from your local machine or a non-Azure cloud.

To create a service principal, follow the steps in the Register a Microsoft Entra app and create a service principal guide and add the credentials to your Nextflow configuration file. For example:

azure {
    activeDirectory {
        servicePrincipalId = '<service-principal-client-id>'
        servicePrincipalSecret = '<service-principal-client-secret>'
        tenantId = '<tenant-id>'
    }
    storage {
        accountName = '<storage-account-name>'
    }
    batch {
        accountName = '<batch-account-name>'
        location = '<batch-account-location>'
    }
}

tip

Nextflow uses the following environment variables if service principal settings are not set in the Nextflow configuration file:

• AZURE_CLIENT_ID: your Azure service principal application ID
• AZURE_CLIENT_SECRET: your Azure service principal secret
• AZURE_TENANT_ID: your Microsoft Entra tenant ID

Access keys

The basic authentication method uses direct access keys. While easy to set up, it is less secure than managed identities or service principals because it provides full access to your resources. This approach requires you to obtain and manage the access keys for your Azure Batch and Storage accounts.

You can find your access keys in the Azure portal:

• For Storage accounts, go to your Storage account and select Access keys in the left-hand navigation menu.
• For Batch accounts, go to your Batch account and select Keys in the left-hand navigation menu.

Keep access keys secure and rotate them regularly. Use environment variables instead of hardcoding keys in configuration files, especially in shared or version-controlled environments. Nextflow uses the following environment variables if the corresponding settings are not provided in the Nextflow configuration:

• AZURE_STORAGE_ACCOUNT_NAME: your Azure Storage account name
• AZURE_STORAGE_ACCOUNT_KEY: your Azure Storage account key
• AZURE_STORAGE_SAS_TOKEN: your Azure Storage account SAS token
• AZURE_BATCH_ACCOUNT_NAME: your Azure Batch account name
• AZURE_BATCH_ACCOUNT_KEY: your Azure Batch account key

Alternatively, specify the access keys in the Nextflow configuration:

azure {
    storage {
        accountName = '<storage-account-name>'
        accountKey = '<storage-account-key>'
    }
    batch {
        accountName = '<batch-account-name>'
        accountKey = '<batch-account-key>'
        location = '<batch-account-location>'
    }
}

You can also use a shared access signature (SAS) instead of an account key to provide time-limited access to your storage account:

azure.storage.sasToken = '<sas-token>'

tip

When you create a SAS token, enable Read, Write, Delete, List, Add, and Create permissions for the Container and Object resource types. Remove the leading ? character from the sasToken value.

Required permissions

The following role assignments are required to use Azure services:

For Azure Storage:

• Storage Blob Data Reader
• Storage Blob Data Contributor

For Azure Batch:

• Batch Data Contributor

To assign roles to a managed identity or service principal, see the Azure documentation.

Advanced configuration

See the azure config scope for all available options.