Today, we will look at using Azure OpenAI and 'Bring Your Data' to allow recent documentation to be referenced and bring life (and relevance) to your data.

The example we are going to use today is the Microsoft Learn documentation for Microsoft Azure.

Our scenario is this:

• We would like to use ChatGPT functionality to query up-to-date information on Microsoft Azure; in this example, we will look for information on Azure Elastic SAN (which was added in late 2022).

When querying ChatGPT for Azure Elastic SAN, we get the following:

Just like the prompt states, ChatGPT only has data up to September 2021 and isn't aware of Elastic SAN (or any other changes/updates or new (or retired) services after this date).

So let us use the Azure OpenAI and bring in outside data (ground data), in this case, the Azure document library, to overlay on top of the GPT models, giving the illusion the model is aware of the data.

To do this, we will leverage native 'Bring Your Own Data' functionality, now in Azure OpenAI - this is in Public Preview as of 04/07/2023.

To be clear, I don't expect you to start downloading from GitHub; this is just an example I have used to add your data. The ability to bring in updated data on Azure, specifically, will be solved by Plugins, such as Bing Search.

To do this, we will need to provision a few Microsoft Azure services, such as:

1. Azure Storage Account (this will hold the Azure document library (markdown files) in a blob container)
2. Cognitive Search (this search functionality, is the glue that will hold this solution together, by breaking down and indexing the documents in the Azure blob store)
3. Azure OpenAI (to do this, we will need GPT3.5 turbo or GPT4 models deployed)
4. Optional - Azure Web App (this can be created by the Azure OpenAI service, to give users access to your custom data)

Make sure you have Azure OpenAI approved for your subscription

We will use the following tools to provision and configure our services:

1. Azure Portal
2. PowerShell (Az Modules)
3. AzCopy

Download Azure Documents​

First, we will need the Azure documents to add to our blob storage.

The Microsoft Learn documentation is open-sourced and constantly updated using a git repository hosted on GitHub. We will download and extract the document repository locally (roughly 6 GB). To do this, we will use a PowerShell script:

     $gitRepoUrl = "https://github.com/MicrosoftDocs/azure-docs"
     $localPath = "C:\temp\azuredocs"
     $zipPath = "C:\temp\azurdocs.zip"
     #Download the Git repository and extract
     Invoke-WebRequest -Uri "$gitRepoUrl/archive/master.zip" -OutFile $zipPath
     Expand-Archive -Path $zipPath -DestinationPath $localPath

Create Azure Storage Account​

Now that we have a copy of the Azure document repository, it's time to create an Azure Storage account to copy the data into. To create this storage account, we will use PowerShell.

     # Login to Azure
     Connect-AzAccount
     # Set variables
     $resourceGroupName = "azuredocs-ai-rg"
     $location = "eastus"
     $uniqueId = [guid]::NewGuid().ToString().Substring(0,4)
     $storageAccountName = "myaistgacc$uniqueId"
     $containerName = "azuredocs"
     # Create a new resource group
     New-AzResourceGroup -Name $resourceGroupName -Location $location
     # Create a new storage account
     New-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName -Location $location -SkuName Standard_LRS -AllowBlobPublicAccess $false
     # Create a new blob container
     New-AzStorageContainer -Name $containerName -Context (New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName).Value[0])

We have created our Resource Group and Storage account to hold our Azure documentation.

Upload Microsoft Learn documentation to an Azure blob container​

Now that we have the Azure docs repo downloaded and extracted and an Azure Storage account to hold the documents, it's time to use AzCopy to copy the documentation to the Azure storage account. We will use PowerShell to create a SAS token (open for a day) and use that with AzCopy to copy the Azure repo into our newly created container.

     # Set variables
     $resourceGroupName = "azuredocs-ai-rg"
     $storageAccountName = "myaistgacc958b"
     $containerName = "azuredocs"
     $storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName).Value[0]
     $localPath = "C:\Temp\azuredocs\azure-docs-main"
     $azCopyPath = "C:\tools\azcopy_windows_amd64_10.19.0\AzCopy.exe"
     # Construct SAS URL for destination container
     $sasToken = (New-AzStorageContainerSASToken -Name $containerName -Context (New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey) -Permission rwdl -ExpiryTime (Get-Date).AddDays(1)).TrimStart("?")
     $destinationUrl = "https://$storageAccountName.blob.core.windows.net/$containerName/?$sasToken"
     # Run AzCopy command as command line
     $command = "& `"$azCopyPath`" copy `"$localPath`" `"$destinationUrl`" --recursive=true"
     Invoke-Expression $command

Note: I took roughly 6 minutes to copy the Azure docs repo from my local computer (in New Zealand) into a blob storage account in East US, so roughly a gigabyte a minute.

Create Cognitive Search​

Now that we have our Azure Blob storage accounts, it's time to create our Cognitive Search. We will need to create a Cognitive Search, with an SKU of Standard, to support the 6GBs of Azure documents that must be indexed. Please check your costs; this is roughly NZ$377.96 a month to run; you can reduce this cost by limiting the amount of data you need to index (i.e., only certain documents, not an entire large repository of markdown files). Make sure you refer to the Pricing Calculator.

     # Set variables
     $resourceGroupName = "azuredocs-ai-rg"
     $searchServiceName = "azuredocssearchservice" #Cognitive Service name needs to be lowercase.
     # Create a search service
     Install-Module Az.Search
     $searchService = New-AzSearchService -ResourceGroupName $resourceGroupName -Name $searchServiceName -Location "eastus" -Sku Standard

Now that the cognitive search has been created, we need to create the index, and indexers, which will index our Azure documents to be used by Azure OpenAI by creating the index and linking it to the Azuredocs blob container, we created earlier.

Note: There is no PowerShell cmdlet support for Azure Cognitive Search indexes; you can create using the RestAPI - but we will do this in the Azure Portal as part of the next step.

Create Azure Cognitive Search Index​

It's time to time to create the Cognitive Search Index, an indexer that will index the content.

We will move away from PowerShell and into the Microsoft Azure Portal to do this.

1. Navigate to the Microsoft Azure Portal
2. In the top center search bar, type in Cognitive Search
3. Click on Cognitive Search
4. Click on your newly created Cognitive Search
5. Select Import Data
6. Select Azure Blob Storage
7. Type in your data source name (i.e., azuredocs)
8. For the Connection string, select Choose an existing connection
9. Select your Azure Storage account and a container containing the Azure document repository uploaded earlier.
10. Click Select
11. Click Next: Add cognitive skills (Optional)
12. Here, you can Enrich your data, such as enabling OCR (extracting text from images automatically) or extracting people's names, and translating text from one language to another; these enrichments are billed separately, and we won't be using any enrichments so we will select Skip to: Customize target index.
13. Here is the index mapping that was done by Cognitive Search automatically by scanning the schema of the documents. You can bring in additional data about your documents if you want, but I am happy with the defaults, so I click: Next: Create an indexer
14. The indexer is what is going to create your index, which will be referenced by Azure OpenAI later; you can schedule an indexer to run hourly, if new data is being added to the Azure blob container where your source files are sitting, for my purposes I am going leave the Schedule as: Once
15. Uncollapse Advanced Options and scroll down a bit
16. Here, we can select to only index certain files; for our purposes, we are going to exclude png files, the Azure document repository contains png images files that aren't able to be indexed (we aren't using OCR), so I am going to optimize the indexing time slightly by excluding them. You can also exclude gif/jpg image files.
17. Finally, hit Submit to start the indexing process. This could take a while, depending on the amount of data
18. Leave this running in the background and navigate to the Cognitive Search resource, Overview pane to see the status.

Note: You can also run the Import Data in Azure OpenAI Studio, which will trigger an index - but you need to keep your browser open and responsive. Depending on how much data you are indexing, doing it manually through this process could be preferred to avoid browser timeout. You also get more options around the index.

Create Azure OpenAI​

Now that we have our base Azure resources, it's time to create Azure OpenAI. Make sure your region and subscription have been approved for Azure OpenAI.

Run the following PowerShell cmdlets to create the Azure OpenAI service:

To create the Azure OpenAI service, we will be using the Azure Portal.

1. Navigate to the Microsoft Azure Portal
2. In the top center search bar, type in Azure OpenAI
3. In the Cognitive Services, Azure OpenAI section, click + Create
4. Select your subscription, region, name, and pricing tier of your Azure OpenAI service (remember certain models are only available in specific regions - we need GPT 3.5+), then select Next
5. Update your Network Configuration; for this demo, we will select 'All Networks' - but the best practice is to restrict it. Click Next
6. If you have any Tags, enter them, then click Next
7. The Azure platform will now validate your deployment (an example is ensuring that the Azure OpenAI has a unique name). Review the configuration, then click Create to create your resource.

Now that the Azure OpenAI service has been created, you should now have the following:

• An Azure OpenAI service
• A Storage account
• A Cognitive Search service

Deploy Model​

Now that we have our Azure OpenAI instance, it's time to deploy our Chat model.

1. Navigate to the Microsoft Azure Portal
2. In the top center search bar, type in Azure OpenAI
3. Open your Azure OpenAI instance to the Overview page, and click: Go to Azure OpenAI Studio
4. Click on Models, and verify that you have gpt models (ie, gpt-36-turbo, or gpt-4). If you don't, then make sure you have been onboarded.
5. Once verified, click on Deployments
6. Click on + Create new deployment
7. Select your model (I am going to go with gpt-35-turbo), type in a deployment name, and then click Create
8. Once deployment has been completed, you may have to wait up to 5 minutes for the Chat API to be aware of your new deployment.

Run Chat against your data​

Finally, it's time to query and work with our Azure documents.

We can do this using the Chat Playground, a feature of Azure OpenAI that allows us to work with our Chat models and adjust prompts.

We will not change the System Prompt (although to get the most out of your own data, I recommend giving it ago); the System Prompt will remain as: You are an AI assistant that helps people find information.

1. Navigate to the Microsoft Azure Portal
2. In the top center search bar, type in Azure OpenAI
3. Open your Azure OpenAI instance to the Overview page, and click: Go to Azure OpenAI Studio
4. Click Chat
5. Click Add your data (preview) - if this doesn't show, ensure you have deployed a GPT model as part of the previous steps.
6. Click on + Add a data source
7. Select the dropdown list in the Select data source pane and select Azure Cognitive Search
8. Select your Cognitive Search service, created earlier
9. Select your Index
10. Check, I acknowledge that connecting to an Azure Cognitive Search account will incur usage to my account. View Pricing
11. Click Next
12. It should bring in the index metadata; for example - our content data is mapped to content - so I will leave this as is; click Next
13. I am not utilizing Semantic search, so I click Next
14. Finally, review my settings and click Save and Close
15. Now we can verify our own data is getting checked by leaving the: Limit responses to your own data content checked
16. Then, in the Chat session, in the User message, I type: Tell me about Azure Elastic SAN?
17. It will now reference the Cognitive Search and bring in the data from the index, including references to the location where it found the data!

Optional - Deploy to an Azure WebApp​

Interacting with our data in the Chat playground can be an enlightening experience, but we can go a step further and leverage the native tools to a chat interface - straight to an Azure web app.

To do this, we need to navigate back to the Chat playground, ensure you have added your own cognitive search, and can successfully retrieve data from your index.

1. Click on Deploy to
2. Select A new web app...
3. If you have an existing WebApp, you can update it with an updated System Message or Search Index from the Chat playground settings, but we will: Create a new web app
4. Enter a suitable name (i.e., AI-WebApp-Tst - this needs to be unique)
5. Select the Subscription and Resource Group and location to deploy to. I had issues accessing my custom data, when deployed to Australia East (as AI services are in East US), so I will deploy in the same region as my OpenAI and Cognitive Search service - i.e., East US.
6. Specify a plan (i.e., Standard (S1))
7. Click Deploy

Note: Deployment may take up to 10 minutes to deploy; you can navigate to the Resource Group you are deploying to, select Deployments, and monitor the deployment. Once deployed, it can take another 10 minutes for authentication to be configured.

Note: By default, the Azure WebApp is restricted to only be accessible by yourself; you can expand this out by adjusting the authentication.

Once it is deployed, you can access the Chat interface, directly from an Azure WebApp!

If you navigate to the WebApp resource in the Azure Portal, and look at the Configuration and Application Settings of your WebApp, you can see variables used as part of the deployment. You can adjust these, but be wary as it could break the WebApp, I would recommend redeploying/updating the WebApp for major changes, from Azure OpenAI studio.

I recently, ran into an error when attempting to use Azure OpenAI and custom data, in the Chat Playground. The error I was getting was:

Error Cannot connect to host aidocsopenaiaccount.openai.azure.com:443 ssl:default [Name of Service not known] Call to LLM failed.

As I was calling data, that was indexed by Azure Cognitive Search, I thought the index was corrupted or invalid, but after mulitple index attempts, continuted to have the same error, all other prompts (not using my custom data) succeeded.

I recreated the Azure OpenAI service, and was able to succesffuly call the custom data, so started to look at the comparisons between the Azure OpenAI instance, that was working and the Azure OpenAI instance that wasn't.

I did this by taking a look at the JSON of each Azure OpenAI instance, then did a comparison, and discovered a major difference.

Note: The one on the left is the working one, the one of the right is the failed one.

The version that was working, had an endpoint of https://*.openai.azure.com , and the version that didn't work, had a different endpoint: eastus.api.cognitive.microsoft.com.

You can check the JSON output of a resource, by navigating to it on the Azure Portal, in the Overview Pane, clicking JSON view.

So why would this have been the case?

When I originally created the Azure OpenAI instance that had this issue, I used PowerShell to create the instance:

$skuName = "S0"
$kind = "OpenAI"
# Create Cognitive Services resource
New-AzCognitiveServicesAccount -ResourceGroupName $resourceGroupName -Name $accountName -Location $location -SkuName $skuName -Kind $kind

This seemed to call an older API or different schema.

The version that worked, I created using the Azure Portal - which was correct.

If you run into the same problem, then recreate your Azure OpenAI instance using the Azure Portal, or one of the currently supported methods: Create a resource and deploy a model using Azure OpenAI, currently (as of the 4th of July 2023) PowerShell is not a supported method of creating an Azure OpenAI instance, until the cmdlets have been updated.

When logging into the Microsoft Azure Portal, to view your Management Group You might have found it blank or constantly attempting to load.

It looks like a potential bug in the Portal interface, especially if you have the correct permissions to see those Management Groups. Here is a few things to look for:

Elevated rights - Global Administrator​

By default, if you are a Global Administrator, it does not allow you to see Azure Resources and manage Azure management groups.

If you have a Global Administrator role, and this is the first time you are setting up Management Groups, then you can elevate your rights, to manage the Azure Management Groups.

Note: Because of the elevated nature of this role, it is recommended to enable it only for the period you need to do your work, and make sure you have specific roles assigned to manage your Azure infrastructure, as necessary.

If this is not the first time you set up Management Groups, ensure you have the right to see the Management Groups.

Create an Azure Management Group using PowerShell​

One of the fixes (workaround) discovered is that creating a Management Group triggers the Azure Portal to update, allowing management.

You can do this using the Azure PowerShell cmdlets, by running:

     New-AzManagementGroup -GroupName 'Contoso'

Once the Management Group is created, you should be able to refresh the Azure Management Group page in the Portal and view your Management Groups, if that doesn't work then log out and back into the Portal.

You can then use the Remove cmdlet to delete the new Management Group you created.

     Remove-AzManagementGroup -GroupName 'Contoso'

Note: This article was based on findings from the Microsoft Q&A article: Management Groups Unavailable in Tenant: Limited Account Control and Organization

Cleanup your unwanted Azure resources on a schedule

Every few months, I get that dreaded email "Your Microsoft Azure subscription has been suspended" - this is due to creating resources, and leaving them provisioned, so I needed a method of deleting the resources I didn't need, or wanted to spin up for a few days. I also needed away to creating resources that can stay, either for learning or a demo, independent of how the resources were deployed into the environment (via the Azure Portal, Terraform, Bicep).

Naturally I went straight to Azure Automation and using PowerShell.

What I ended up with was a Runbook capable of EXTREME AZURE DESTRUCTION which was exactly what I wanted.

This script is provided as-is with no warranties or guarantees. Use at your own risk. This is not intended to be a script to use in Production, mainly test environments, as this WILL CAUSE massive destruction and irretrievable data loss... You have been warned.

I am not going to go into setting up Azure Automation, if interested you can refer to a few of my blog posts I have done previously that goes through the process:

• Deallocate ‘Stopped’ Virtual Machines using Azure Automation
• Turn on a Azure Virtual Machine using Azure Automation
• Disable SFTP support on an Azure Storage account on a Schedule

The script named: Invoke-DakaraSuperWeapon, aptly named as a reference to the Dakara weapon from the TV series Stargate SG1 - a weapon if great power.

The Dakara superweapon was a Ancient device capable of reducing all matter to its basic elemental components, and/or restructuring it. Possessing the ability to pass through the shields of known ships it also functions (and has been used) as a devastating weapon to kill the entire crew of orbiting ships or wipe out all life on the surface of hundreds of planets at a time. "It is not only capable of destroying the Replicators but all life in the galaxy."

Using the latest Windows PowerShell release - 7.2 (Preview), this script is built around the following capabilities:

• Delete ALL resource groups (without a specific Tag) under all subscriptions, under a specific Management Group
• Delete all resources within those resource groups
• Delete Azure Recovery Vaults and their backed up items
• Delete any Azure policy assignments, assigned directly to any subscription under the Management Group
• Delete any Azure RBAC role assignments, assigned directly to any subscription under the Management Group.

In my demo environment, I have a range of Management Groups, and 2 Azure subscriptions.

`

For my purposes, I created a System Managed Identity from the Azure Automation account, and applied it to the: 'mg' Management Group as 'Owner' (Contributor will work, as long as you don't plan on removing the rights from the Azure subscriptions - theoretically, so could Contributor + User Access Administrator roles).

Again - this was created for my own environment - if you decide to run this, TEST IT! And Make sure it has as limited permissions as possible, potentially the Managed Identity will only have access to a specific test Subscription that you may not care about. I take no responsibily.

The System Identity will be used to execute the runbook.

I also needed a Tag (ie a Safe word) to save the Resource Groups that I need to remain, an example is a project I am working on, demo etc. This Tag is in name only - as Tags are Key/Value pairs in Azure - in this case I only cared about the Key (ie NotDelete) - what was in the value, didn't matter.

Important: When importing the Runbook it is imperative that you Tag the Resource Group it is in, with your safe word! Or else could will be deleted!

The script has a couple of parameters:

As you can tell, you can enable or disable specific parts of the script, for example - if you just want to use it to clean up direct role assignments on your subscriptions, while not deleting Azure resources you can by entering True or False.

When ran it will stream the Logs to the Azure Automation Log Stream, there is no waiting time or approval - it will just run.

As below, you can see the Resource Groups get removed (at the time of this recording, I had a limit on the amount of parallel delete tasks:

# This runbook deletes all resource groups under a management group except for the ones with a specific tag.
<#
.SYNOPSIS
Deletes all resource groups under a management group except for the ones with a specific tag.

.DESCRIPTION
This script deletes all resource groups under a specified management group except for the ones with a specific tag. It can also delete policy assignments and subscription role assignments if specified.

.PARAMETER ManagementGroupId
The ID of the management group to delete resource groups under. WARNING: This script will delete all resource groups under the specified management group except for the ones with the specified tag. Make sure you have specified the correct management group ID, or you may accidentally delete resources that you did not intend to delete.

.PARAMETER TagName
The name of the tag to check for. WARNING: This script will delete all resource groups that do not have this tag. Make sure you have specified the correct tag name, or you may accidentally delete resources that you did not intend to delete.

.PARAMETER RemoveResourceGroups
If specified, deletes the resource groups that do not have the specified tag.

.PARAMETER DeletePolicyAssignments
If specified, deletes the policy assignments for the management group and all child subscriptions.

.PARAMETER DeleteSubRoleAssignments
If specified, deletes the subscription role assignments for all child subscriptions.

.EXAMPLE
.\Initiate-DakaraSuperWeapon.ps1 -ManagementGroupId "my-management-group" -TagName "my-tag" -RemoveResourceGroups -DeletePolicyAssignments -DeleteSubRoleAssignments
Deletes all resource groups under the "my-management-group" management group that do not have the "my-tag" tag, and deletes the policy assignments and subscription role assignments for all child subscriptions.

.NOTES
This script requires the Azure PowerShell module to be installed. It also requires Owner rights (or User Administrator role) in order to remove roles from a subscription. Make sure your rights are set to be inherited from a management group before running this script.
#>

param (
    [Parameter(Mandatory = $true, HelpMessage = "The ID of the management group to delete resource groups under. WARNING: This script will delete all resource groups under the specified management group except for the ones with the specified tag. Make sure you have specified the correct management group ID, or you may accidentally delete resources that you did not intend to delete.")]
    [string]$ManagementGroupId,
    
    [Parameter(Mandatory = $true, HelpMessage = "The name of the tag to check for. WARNING: This script will delete all resource groups that do not have this tag. Make sure you have specified the correct tag name, or you may accidentally delete resources that you did not intend to delete.")]
    [string]$TagName,
       
    [Parameter(Mandatory = $false)]
    [switch][bool]$RemoveResourceGroups = $false,
    
    [Parameter(Mandatory = $false)]
    [switch][bool]$DeletePolicyAssignments = $false,

    [Parameter(Mandatory = $false, HelpMessage = "This will need Owner rights (or User Administrator role) in order to remove roles from a Subscription. Make sure your rights are set to be inherited from an Management Group, before running this.")]
    [switch][bool]$DeleteSubRoleAssignments = $false
)

# Convert string values to boolean values
$RemoveResourceGroups = [System.Boolean]::Parse($RemoveResourceGroups)
$DeletePolicyAssignments = [System.Boolean]::Parse($DeletePolicyAssignments)
$DeleteSubRoleAssignments = [System.Boolean]::Parse($DeleteSubRoleAssignments)

# Ensures you do not inherit an AzContext in your runbook
Disable-AzContextAutosave -Scope Process

#Toggle to stop warnings with regards to Breaking Changes in Azure PowerShell
Set-Item -Path Env:\SuppressAzurePowerShellBreakingChangeWarnings -Value $true

# Connect to Azure with system-assigned managed identity
(Connect-AzAccount -Identity).context

# Write an initial log message
Write-Output "Initilizing superweapon...."

# Get the subscription IDs under the specified management group AND child management groups
function Get-AzSubscriptionsFromManagementGroup {
    param($ManagementGroupName)
    $mg = Get-AzManagementGroup -GroupId $ManagementGroupName -Expand
    foreach ($child in $mg.Children) {
        if ($child.Type -match '/managementGroups$') {
            Get-AzSubscriptionsFromManagementGroup -ManagementGroupName $child.Name
        }
        else {
            $child | Select-Object @{N = 'Name'; E = { $_.DisplayName } }, @{N = 'Id'; E = { $_.Name } }
        }
    }
}
$mgid = Get-AzManagementGroup -GroupId $ManagementGroupID -Expand

$subIds = (Get-AzSubscriptionsFromManagementGroup -ManagementGroupName $mgid.DisplayName).id


# Delete the policy assignments

if ($DeletePolicyAssignments -eq $true) {
    Write-Output "Deleting management group policy assignments..."
    Get-AzPolicyAssignment -Scope $mgid.Id | Remove-AzPolicyAssignment -Verbose
    Write-Output "Deleting subscription group policy assignments..."

    foreach ($subId in $subIds) {
        Write-Output "Setting subscription context..."
        Set-AzContext -Subscription $subId
        Write-Output "Deleting subscription group policy assignments..."
        Get-AzPolicyAssignment -Scope "/subscriptions/$($subId)" | Remove-AzPolicyAssignment -Verbose

    }
}
else {
    Write-Output "Skipping policy assignment deletion..."
}

# Delete the resource groups
if ($RemoveResourceGroups -eq $true) {
    Write-Output "Deleting resource groups..."

    if ($null -ne $subIds -and $subIds.Count -gt 0) {

        foreach ($subId in $subIds) {
            Write-Output "Setting subscription context..."
            Set-AzContext -Subscription $subId

            $ResourceGroupsfordeletion = Get-AzResourceGroup | Where-Object { $_.Tags -eq $null -or $_.Tags.ContainsKey($tagName) -eq $false }
            Write-Output "The following Resource Groups will be deleted..."
            Write-Output -InputObject $ResourceGroupsfordeletion

            ## Checks to see if a Recovery Services Vaults exists, the Recovery Services Vault and backups need to be deleted first.
            $RSV = Get-AzRecoveryServicesVault | Where-Object { $_.ResourceGroupName -in $ResourceGroupsfordeletion.ResourceGroupName }
            if ($null -ne $RSV) {

                ForEach ($RV in $RSV) {
                    Write-Output  "Backup Vault deletion supports deletion of Azure VM backup vaults ONLY currently."
                    #Credit to Wim Matthyssen for reference in the backup section of the script - https://wmatthyssen.com/2020/11/17/azure-backup-remove-a-recovery-services-vault-and-all-cloud-backup-items-with-azure-powershell/
                    Set-AzRecoveryServicesVaultProperty -Vault $RV.ID -SoftDeleteFeatureState Disable
                    Set-AzRecoveryServicesVaultContext -Vault $RV
                    $containerSoftDelete = Get-AzRecoveryServicesBackupItem -BackupManagementType AzureVM -WorkloadType AzureVM | Where-Object { $_.DeleteState -eq "ToBeDeleted" }
 
                    foreach ($item in $containerSoftDelete) {
                        Undo-AzRecoveryServicesBackupItemDeletion -Item $item  -Force -Verbose
                    }

                    $containerBackup = Get-AzRecoveryServicesBackupItem -BackupManagementType AzureVM -WorkloadType AzureVM  | Where-Object { $_.DeleteState -eq "NotDeleted" }
                    foreach ($item in $containerBackup) {
                        Disable-AzRecoveryServicesBackupProtection -Item $item -RemoveRecoveryPoints -Force -Verbose
                    }
                    Remove-AzRecoveryServicesVault -Vault $RV -Verbose

                }

            }
            ## Checks to see if a Azure Resource Mover resource exists, as this need to be deleted first.

            $ARM = Get-AzResource | Where-Object { $_.ResourceGroupName -in $ResourceGroupsfordeletion.ResourceGroupName -and $_.ResourceType -eq 'Microsoft.Migrate/moveCollections' }

            Write-Output -InputObject $ARM

            if ($null -ne $ARM) {

                ForEach ($RM in $ARM) {
                    Write-Output  "Azure Resource Mover collections exists."
                    Write-Output -InputObject $RM
                    $a = Get-AzResourceMoverMoveResource -ResourceGroupName $RM.ResourceGroupName -MoveCollectionName $RM.Name
                    Foreach ($b in $a) {
                        Write-Output -InputObject $b
                        # Remove a resource using the resource ID
                        Invoke-AzResourceMoverDiscard -ResourceGroupName $RM.ResourceGroupName -MoveResourceInputType $b.Id -MoveResource $b.Name
                        Remove-AzResourceMoverMoveResource -ResourceGroupName $RM.ResourceGroupName -MoveCollectionName $RM.Name -Name $b.Name -Verbose
                    }
                
                    Remove-AzResourceMoverMoveCollection -ResourceGroupName $RM.ResourceGroupName -MoveCollectionName $RM.Name
                }

            }

            Write-Output "Deleting resource groups..."
            $ResourceGroupsfordeletion | ForEach-Object -Parallel {
                Remove-AzResourceGroup -Name $_.ResourceGroupName -Force
            } -ThrottleLimit 20 -Verbose

    
            # Remove the Network Watcher resource group - if remaining - in some scenarios the script left this RG behind.
            # Get the resource group with the specified tag
            $networkWatcherRG = Get-AzResourceGroup | Where-Object { $_.ResourceGroupName -eq 'NetworkWatcherRG' }
            if ($null -ne $networkWatcherRG -and $null -ne $networkWatcherRG.Tags -and $networkWatcherRG.Tags.ContainsKey($tagName) -eq $false) {
                Remove-AzResourceGroup -Name $networkWatcherRG.ResourceGroupName -Force -ErrorAction Continue -Verbose
            }     
        }

        # Write a final log message
        Write-Output "Resource group deletion process completed."
    }
    else {
        Write-Output "No child subscriptions found under the specified management group."
    }

}
else {
    Write-Output "Skipping resource group deletion..."
}

if ($DeleteSubRoleAssignments -eq $true) {
    if ($null -ne $subIds -and $subIds.Count -gt 0) {

        foreach ($subId in $subIds) {
            Write-Output "Setting subscription context..."
            Set-AzContext -Subscription $subId
            $roleAssignments = Get-AzRoleAssignment -Scope "/subscriptions/$($subId)" -IncludeClassicAdministrators
            Write-Output -InputObject $roleAssignments
            # Loop through each role assignment and delete it if it is not inherited a management group
            foreach ($roleAssignment in $roleAssignments) {
                if ($roleAssignment.Scope -like "/subscriptions/*" -and $null -ne $roleAssignment.ObjectId -and $roleAssignment.ObjectId -ne "") {
                    Write-Output "Deleting role assignment..."
                    Remove-AzRoleAssignment -Scope $roleAssignment.Scope -ObjectId $roleAssignment.ObjectId -RoleDefinitionName $roleAssignment.RoleDefinitionName -Verbose -ErrorAction Continue 
                }
            }
            Write-Output "Deleting subscription role assignments..."
        }

    }

}
else {
    Write-Output "Skipping policy subscription role assignments deletion..."
}

Using the Azure Automation schedule, I can then set this Runbook to run every Day, Week etc - knowing my environment will be fresh for my next project, learning exercise.

Today we are going to take a look at the Microsoft Azure Elastic SAN.

Introduction​

“Azure Elastic SAN is a unique cloud-native and fully managed storage area network (SAN) service. Combining SAN-like capabilities with the benefits of being a cloud-native service, Azure Elastic SAN offers a massively scalable, cost-effective, high-performance, and resilient storage solution. It can connect to a variety of Azure compute services, enabling you to seamlessly transition your SAN data estate to the cloud without having to refactor your application architectures.”

Azure Elastic SAN provides a storage solution that is highly scalable, cost-effective, high-performing, and resilient. It caters to various storage needs, whether you're migrating your on-premises SAN to the cloud or creating your application directly in the cloud.

As Azure Elastic SAN is still in the preview stage, as of 28/05/2023, it is important to note that its features and functionality may change before it reaches production. Microsoft continues to actively gather feedback from users and refine the offering to ensure a seamless experience when it finally becomes generally available. Request access to the Preview by filling out this form. This feature should not be used for production workloads until General Availability (GA).

A Storage Area Network (SAN) typically comprises one or more physical appliances equipped with multiple drive bays, which are used to create volumes – it is considered a high-performance and low-latency connectivity storage solution.

The benefits of a SAN are:

• Grow storage footprint independent of Compute
• Low latency and high storage throughput
• Cost efficient with massive scale.
• Built for databases and IOPS-intensive applications.
• Supports large virtualization deployments.

Introducing Azure Elastic SAN.

With the Azure Elastic SAN, we can the elasticity of the Microsoft Azure block storage systems, to supply expandable block storage capabilities to workloads via iSCSI (Internet Small Computer Systems Interface), or services such as Azure Kubernetes Services through Azure Container Storage.

When looking at some of the benefits of an Azure Elastic SAN, over a traditional SAN, we will delve into several common user stories around SAN provisioning and capacity management, with key differences around time to deployment and skills required.

Architecture and Components​

The Azure Elastic SAN consists of 3 layers:

The Elastic SAN​

The Elastic SAN itself, the Elastic SAN consists of the control plane, where you create and manage your Volume Groups from. The Elastic SAN is where the resources are provisioned, and the Cost Management takes place (i.e., Tags on the Elastic SAN resource).

Volume Group​

An Azure Elastic SAN can have up to 20 volume groups, the volume group is where your security, encryption, and data protection configurations get applied.

The volume group is where your Network Security rule and service endpoints are applied. Any settings or configurations applied to a volume group, such as virtual network rules, are inherited by any volumes associated with that volume group.

Volume​

The volume in an Azure Elastic SAN is the actual storage, that gets delivered and mapped to your workload or service.

Multiple volumes can be a part of a single-volume group, or separate groups – depending on requirements, such as accessibility across different virtual networks.

You partition the SAN's storage capacity into individual volumes. These individual volumes can be mounted to your clients with iSCSI. The name of your volume is part of their iSCSI IQD

Deployment and Configuration​

Now that we know what Azure Elastic SAN is, let's deploy it using the Azure Portal.

At the time of this article, the Azure Elastic SAN is only available in specific regions – and whether the SAN is capable of ZRS or LRS storage redundancy. As I am based in New Zealand, the closest region at this time to me is Australia East, this region only supports LRS so this is what I will be configuring.

If you haven’t already – as part of Public Preview, your Azure subscription needs to be enabled to provision Azure Elastic SAN.

Deployment​

1. Login to the Microsoft Azure portal
2. In the search box at the top of the portal, type in Elastic SAN, and navigate to the Elastic SAN resource page.
3. Click Create Elastic SAN
4. 
5. I will create a new Resource Group named: AzureElasticSAN-dev-rg
6. I will name my Azure Elastic SAN: azelasticsan_aue (Name has to be between 3 to 24 characters in length, and may only contain lowercase letters, numbers, hyphens, and underscores (hyphens and underscores must be surrounded by letters or numbers).
7. 
8. Now we need to specify the base and capacity size, the base size will determine what your iOPS and throughput your SAN will support. It’s cheaper to go with, a lower Base size, and higher additional storage – but it will affect your IOPS and bandwidth. These values can be changed later (start with a minimum and increase as needed, as you can’t downsize) – I will set my Base as the minimum of 1 TB and add size of 1TB.
9. 
10. Click Next
11. This is where we can create a volume group, click + Create volume group
12. The volume group will be used to contain our volumes, I will name a volume group as demo
13. I will then allow the volume group, to connect to my DevBox virtual network and set up a service endpoint, on my devbox subnet
14.  
15. Click Create, and finally Review + Create to create your Azure Elastic SAN.
16. Configuration

Now that we have an Azure Elastic SAN, it’s now time to add some volumes. We can partition the SAN's total storage into individual volumes, used for block storage.

A volume can only be part of one volume group, but you can have multiple volumes, across multiple volume groups – that equals the total size of the SAN (in my example 2 TB, the 1 TB Base size, and 1 TB additional capacity), however unlike the SAN – a volume can be Gigabytes in size.

A volume cannot be higher than the total allocated capacity assigned to the SAN.

Remember when you create a new Volume, you can increase the size later (but you can’t downsize the volume).

The volume name is part of your volume's iSCSI Qualified Name and can't be changed once deployed.

1. To create a new Volume, let's navigate to our Azure Elastic SAN.
2. Click on Volumes (under SAN Management)
3. Click Create a volume.
4. We are going to give the volume a name, in this example I will go with: vol1
5. For the size, I will select 500GB.
6. Click Save
7. Once the volume has been created, we can see the volume and the assigned volume group, including the size of the volume and the remaining capacity of the SAN.

Connect​

Now that the SAN, Volume Group, and Volume have been setup – it’s time to connect to the storage.

In my demo environment, I have a Volume Group, assigned to a storage service endpoint on my devbox virtual network and subnet, so I will connect to the SAN through a Windows Server 2022 server, that is attached to the same virtual network.

Following this guide will set up a single path, for more production scenarios to achieve higher IOPS and throughput, and configure mulipathing using the Microsoft document ‘here’.

1. To connect, we need to navigate to our volume, lets navigate to our Azure Elastic SAN.
2. Click Volumes
3. Click your volume, you want to connect to and select Connect.
4. Microsoft Azure, gives us the PowerShell/Bash cmdlets to run on Windows and Linux. Copy the scripts, we will need to run both.
5. Login to your Windows Server that you want to connect to the volume. If this is a new Windows Server, that hasn’t been configured to iSCSI – you will need to start the iSCSI Initiator service and set it to Automatic start.
6. Open a PowerShell prompt as Administrator and run the Add Target command, you should have ‘Operation completed successfully’.
7. Then run the login to target script.
8. 
9. Once completed, you can confirm your connectivity, by opening up Server Manager and launching iSCSI Initiator, you should see 'Connected'.
10. 
11. Now that your iSCSI target has been mapped – lets create a volume we can actually use!
12. Right Click your Start Menu and click Disk Management
13. You should see an Unallocated Disk
14. 
15. The unallocated disk here, is our 500GB Azure Elastic SAN volume, it’s time to initialize it, right click and select Create New Simple Volume, I will go through the defaults and now I have a new drive I can use within Windows
16. 

Scalability and Performance​

Now that we have provisioned the Azure Elastic SAN, and connected to it, let’s take a high-level look at the scalability and performance.

For performance, I ran a single threaded test on my 500GB iSCSI target using Diskspd, using the following parameters:

.\diskspd.exe - d60 - W15 - C15 - c128M - t4 - o4 - b8k - L - r - Sh - w50 f: \disk - speed - test.dat

The output was:

Total IO:

• I/Os: 109,541
• I/O per second: 1,825.30

Read IO:

• I/Os: 54,547
• I/O per second: 908.93

Write IO:

• I/Os: 54,994
• I/O per second: 916.38

The output does not directly provide the exact IOPS (Input/Output Operations Per Second) value, but we can calculate it based on the total number of I/Os and the test duration.

To calculate the overall IOPS, divide the total number of I/Os by the test duration:

• Total IOPS: 109,541 I/Os / 60 seconds = 1,825.68 IOPS

Similarly, you can calculate the read and write IOPS:

• Read IOPS: 54,547 I/Os / 60 seconds = 909.12 IOPS
• Write IOPS: 54,994 I/Os / 60 seconds = 916.57 IOPS

Therefore, based on the given Diskspd output, the SAN is performing approximately 1,825 IOPS in total, with 909 IOPS for reads and 917 IOPS for writes.

Note: This was a very basic test, running on a Windows Server 2022 Azure Virtual Machine (Standard D4s v3 (4 vcpus, 16 GiB memory)), single threaded.

At the time I ran this test – the IOPS limit of the SAN was 5000, and bandwidth was 80. I highly encourage you to run your own tests. This was more intended as a baseline for reference.

Pathping, reported 0 hops for DNS resolution, this Virtual Machine is in another Availability Zone from the SAN.

In terms of Scalability, a few things to note:

• You cannot increase the volume size, while an active session (i.e., in use by a workload) is in use.
• You can increase the Base and Additional size of the SAN, while sessions are in use.
• The Additional size of the Azure Elastic SAN has to be less than the Base size (for example, you cannot have an Azure Elastic SAN with a Base size of 3, and an Additional size of 4).

Data Protection and Redundancy​

At the moment, Azure Elastic SAN, only supports specific regions and configurations, for example in Australia East, the Azure Elastic SAN only supports LRS, so the volumes sit within a single datacenter, replicated across 3 storage stacks.

LRS is still decent in terms of resiliency but for those who are use to cross SAN replication - I expect, we will see ZRS start to become common place, as this service is rolled out.

At the time of this writing, there is no Azure Backup Support, I expect this capacity to be released closer to GA (Generally Available). For day to day operations, I was able to enable the Volume Shadow Copies, which allowed me to go back to previous versions of files and folders.

Security and Compliance​

Other than the usual filesystem permissions, you can use with your filesystems across Windows and Linux on the Azure Elastic SAN, there are built-in roles that can be leveraged for Azure Elastic SAN management.

Assigning least privilege operations or creating your own custom role is possible with Azure Elastic SAN, and roles such as Volume Group Owner will be key to how the SAN is architectured for an organisation.

All data stored in an Elastic SAN is encrypted at rest using Azure storage service encryption (SSE). Storage service encryption works similarly to BitLocker on Windows: data is encrypted beneath the file system level. SSE protects your data and to helps you meet your organizational security and compliance commitments. Data stored in Elastic SAN is encrypted with Microsoft-managed keys, that are rotated frequently.

Integration with Azure Services​

At the moment, Azure Elastic SAN supports, Service Endpoints only, Private endpoints are not yet available, but enabling service endpoints on subnet/s was easily done.

Connectivity to Windows and Linux machines, will be done through native iSCSI protocols, as the Azure Portal displayed when attempting to connect.

At the time of this writing, Azure Container Storage – connecting to Azure Kubernetes Service, can be leveraged by Azure Elastic SAN (Azure Container Storage is also in Public Preview).

Cost and Pricing​

The Azure Elastic SAN is charged by the amount you provision based on units:

• Base unit
• Capacity-only unit

They are designed to offer a simple bulk provisioning experience while also providing flexibility to expand your data footprint.

The Base unit has a capacity of 5000 IOPs and a throughput of 80 MBps per TiB.

The Capacity only unit, on the other hand, allows you to provision capacity only without provisioning performance at a lower cost.

As a result, you can cost-effectively scale performance while migrating workloads from on-premises to Azure.

The total price of Azure Elastic SAN depends on the base and capacity scale unit (LRS and ZRS dependent).

Elastic SAN will need to be provisioned with at least one base unit of 1TiB. It is also important to note that the SAN-provisioned resources are shared by all volume groups and volumes.

Tips​

I will cover a few titbits, I found in my discovery.

• Authentication Failure – if you get ‘Authentication Failure’, when attempting to connect to your iSCSI target (Azure Elastic SAN Volume), from your Windows or Linux workload, make sure that the Volume Group has had the Service Endpoint enabled for the subnet and virtual network you are connecting from, by default all traffic is denied to the Azure Elastic SAN – this includes traffic from other subnets, the storage endpoint needs to be enabled on the subnet that hosts your virtual machines.
• Persistent Login - When connecting to the iSCSI target (the Volume) this is a once off session, if your system is rebooted, the volume won’t automatically remap. You can set a Persistent Login, by replacing ‘LoginTarget’ in your iscsicli command to: PersistentLoginTarget (for example: iscsicli PersistentLoginTarget iqn.2023-05.net.windows.core.blob.ElasticSan.es-rsip05eo4sx0:vol1 t es-rsip05eo4sx0.z40.blob.storage.azure.net 3260 Root\ISCSIPRT\0000_0 -1 * * * * * * * * * * * 0), then the next time you reboot your Windows server, the volume will be automatically mounted.

Updates and New Features​

As this is a Private Preview service, there will be updates on functionality and features that I have gone through today and may have missed.

• aka.ms/ElasticSANOverview
• aka.ms/ElasticSANPlanning
• aka.ms/ElasticSANPricing
• Keep an eye out on the Azure Updates page for updates.
• Or if you are Linkedin member, follow Azure Feeds on Linkedin.