Skip to main content

188 posts tagged with "Azure"

View All Tags

Azure WebApp 500 Errors reporting from AspNetCoreModule

· 2 min read

Issue Description

Intermittent issues with Azure WebApp constantly stop functioning, a Stop/Start operation brings it back online.

Root Cause

Further investigation using Azure Application Insights, reveals the Azure WebApp was experiencing a few FailedRequestCount, with HTTP 500 Errors. An exception was thrown by a TaskScheduler. Exception of type 'System.OutOfMemoryException' was thrown.

Resolution

In my case, the service that was running on the Azure WebApp was using .NET Core 2.0, the fix was to upgrade to the latest version.

.NET Core 2.0 is an unsupported version and we highly recommend upgrading to the latest version (3.1). Please take a look at this information of the .NET Core official support policy: https://dotnet.microsoft.com/platform/support/policy/dotnet-core

For .NET Core applications I suggest enabling the stdout logs, as those will capture some important errors: https://learn.microsoft.com/en-us/aspnet/core/test/troubleshoot-azure-iis?view=aspnetcore-2.2#aspnet-core-module-stdout-log-azure-app-service-1

If those OutOfMemory exceptions come with a 5xx status code, I would suggest as well using the AutoHeal feature as it will allow setting rules based on that status code to capture a Memory Dump, you can check more information here: https://azure.github.io/AppService/2018/09/10/Announcing-the-New-Auto-Healing-Experience-in-App-Service-Diagnostics.html

Allow Azure DevOps Microsoft Hosted Agent to communicate with Azure KeyVault

· 3 min read

It is best practice to lock down Azure resources to be accessible by location and services that is only to what's required and, the Azure Key vault is no exception.

When using Microsoft Hosted Agents in Azure DevOps, you need to make sure that the AzureCloud IPs for the Azure DevOps regions are opened on the Firewall.

In my case, I was in the: AustraliaEast region and needed to identify and add the following 'AzureCloud' Address Ranges to the KeyVault firewall:

  • "name": "AzureCloud.australiaeast",
  • "id": "AzureCloud.australiaeast",
  • "properties": {
  • "changeNumber": 13,
  • "region": "australiaeast",
  • "regionId": 3,
  • "platform": "Azure",
  • "systemService": "",
  • "addressPrefixes": [
  • "13.70.64.0/18",
  • "13.72.224.0/19",
  • "13.73.192.0/20",
  • "13.75.128.0/17",
  • "13.104.211.128/26",
  • "13.105.16.192/26",
  • "13.105.20.128/26",
  • "13.105.52.192/26",
  • "13.105.53.128/26",
  • "20.37.192.0/19",
  • "20.38.112.0/23",
  • "20.40.64.0/20",
  • "20.40.80.0/21",
  • "20.40.120.0/21",
  • "20.40.176.0/20",
  • "20.42.192.0/19",
  • "20.43.96.0/20",
  • "20.47.37.0/24",
  • "20.47.122.0/23",
  • "20.53.32.0/28",
  • "20.53.40.0/21",
  • "20.53.64.0/18",
  • "20.53.128.0/17",
  • "20.58.128.0/18",
  • "20.60.72.0/22",
  • "20.60.182.0/23",
  • "20.70.128.0/17",
  • "20.135.120.0/21",
  • "20.150.66.0/24",
  • "20.150.92.0/24",
  • "20.150.117.0/24",
  • "20.157.44.0/24",
  • "20.188.128.0/17",
  • "20.190.142.0/25",
  • "20.190.167.0/24",
  • "20.191.192.0/18",
  • "20.193.0.0/18",
  • "20.193.64.0/19",
  • "23.101.208.0/20",
  • "40.79.160.0/20",
  • "40.79.211.0/24",
  • "40.82.32.0/22",
  • "40.82.192.0/19",
  • "40.87.208.0/22",
  • "40.90.18.0/28",
  • "40.90.30.0/25",
  • "40.90.130.80/28",
  • "40.90.130.208/28",
  • "40.90.140.32/27",
  • "40.90.142.160/27",
  • "40.90.147.64/27",
  • "40.90.150.0/27",
  • "40.112.37.128/26",
  • "40.126.14.0/25",
  • "40.126.39.0/24",
  • "40.126.224.0/19",
  • "52.108.40.0/23",
  • "52.108.83.0/24",
  • "52.109.112.0/22",
  • "52.111.224.0/24",
  • "52.113.88.0/22",
  • "52.113.103.0/24",
  • "52.114.16.0/22",
  • "52.114.58.0/23",
  • "52.114.192.0/23",
  • "52.115.98.0/24",
  • "52.120.158.0/23",
  • "52.121.108.0/22",
  • "52.143.199.0/24",
  • "52.143.200.0/23",
  • "52.147.0.0/19",
  • "52.156.160.0/19",
  • "52.187.192.0/18",
  • "52.232.136.0/21",
  • "52.232.154.0/24",
  • "52.237.192.0/18",
  • "52.239.130.0/23",
  • "52.239.226.0/24",
  • "52.245.16.0/22",
  • "104.44.90.64/26",
  • "104.44.93.96/27",
  • "104.44.95.48/28",
  • "104.46.29.0/24",
  • "104.46.30.0/23",
  • "104.209.80.0/20",
  • "104.210.64.0/18",
  • "191.238.66.0/23",
  • "191.239.64.0/19",
  • "2603:1010::/46",
  • "2603:1010:5::/48",
  • "2603:1010:6::/48",
  • "2603:1016:1400:60::/59",
  • "2603:1016:2402::/48",
  • "2603:1016:2500:c::/64",
  • "2603:1017:0:60::/59"

You only need to add the IP ranges of the Region that your Azure DevOps instance sits in.

You can find the region that your Azure DevOps instance sits in by following the article below:

You can retrieve the list of Azure IP Ranges and Service Tags from the following Microsoft JSON file:

Note: These IP ranges can change and update, depending on Microsoft removing and adding new datacenter capability, it is always worth rechecking the list if you find you start having problems with intermittent connectivity to check whether new ranges have been added that haven't been whitelisted.

Failed to delete the private endpoint. Error: Call to Microsoft.Storage/storageAccounts description: "Failed to delete the private endpoint. Error: Call to Microsoft.Storage/storageAccounts failed" failed

· 2 min read

Issue Description

Failed to delete the private endpoint. Error: Call to Microsoft.Storage/storageAccounts failed

Root Cause

Azure Backup locks the storage account when you configure protection for any file share in the corresponding account. This provides protection against accidental deletion of a storage account with backed-up file shares.

Resolution

In my case, the Storage account I was attempting to remove the Private Endpoint from was an Azure File Sync storage account, that had Azure File Shares that were getting Backuped Up.

  • Found and removed the lock on the storage account
  • Then successfully delete the private endpoint

More info

Generally, it is recommended that keep the lock taken on the storage account by Azure Backup. If you delete the lock, your storage account will be prone to accidental deletion and if it's deleted, you'll lose your snapshots or backups.

https://learn.microsoft.com/en-us/azure/backup/backup-afs#best-practices

https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources

Unable to start Windows Azure Guest Agent (it's in a disabled state)

· 3 min read

Azure Backup Overview

Issue Description

Unable to start Windows Azure Guest Agent (it's in a disabled state). When trying and set the service to auto the following error occurs 'The specified service has been marked for deletion.'

VM Agent is unable to communicate with the Azure Backup service.

Root Cause

This may occur if Windows Communication Framework (WCF) profiling is enabled. WCF profiling should only be enabled while debugging a WCF issue. It should not be left enabled while running a production workload.

Resolution #1

1. Restart your workload, I would recommend to Stop (deallocate first) to make sure that the workload starts correctly on a new hypervisor, the Azure Backup agent starts and checks for agent updates during the boot process.

Resolution #2

Disable WCF profiling:

1. Launch an elevated CMD prompt. 2. Run the following commands to back up the existing: C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config file:

   cd C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config

copy machine.config machine.config.bak

3. Run notepad machine.config to edit the file in Notepad.

Remove this text, being careful not to also remove any additional text that may be on the same line:

<add name="Microsoft.VisualStudio.Diagnostics.ServiceModelSink.Behavior" type="Microsoft.VisualStudio.Diagnostics.ServiceModelSink.Behavior, Microsoft.VisualStudio.Diagnostics.ServiceModelSink, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"/>

Also remove this text, being careful not to also remove any additional text that may be on the same line:

<commonBehaviors><endpointBehaviors><Microsoft.VisualStudio.Diagnostics.ServiceModelSink.Behavior/></endpointBehaviors><serviceBehaviors><Microsoft.VisualStudio.Diagnostics.ServiceModelSink.Behavior/></serviceBehaviors></commonBehaviors>

4. Save and close the file. 5. Restart the guest agent services:

net stop Rdagent

net stop WindowsAzureGuestAgent

net stop WindowsAzureTelemetryService

net start Rdagent

6. In some cases the VM may need to be restarted for the WCF disablement to take effect.

Resolution #3

From time to time the Azure backup agent may fail. Sometimes this will self-resolve but on the odd occasion, additional steps may be needed.

1. Uninstall the agent via the Control Panel. 2. Open CMD as Admin. 3. Stop the following services:

net stop rdagent

net stop WindowsAzureGuestAgent

net stop WindowsAzureTelemetryService

4. Delete all the services of the agent:

sc delete rdagent

sc delete WindowsAzureGuestAgent

sc delete WindowsAzureTelemetryService

5. Create a folder called OLD in "C:\ WindowsAzure" and move the old version of the agent to it and the folders that say Packages. 6. Install the service again using the link: https://go.microsoft.com/fwlink/?LinkID=394789&clcid=0x409 or the latest agent available.
7. Restart the server.

Resolution #4

1. Migrate the Pagefile to a new disk 2. Set a limit on the pagefile

First look at Universal Automation Desktop

· 8 min read

There are many ways to do automation scheduling – whether its Jenkins or even Windows Task Scheduler. Each toolset has its place or specialization today we are looking at the son of Ironman Software’s PowerShell Universal Automation platform – the Desktop Edition!

The Desktop edition replicates some of the same functionality as the Universal Automation platform, however, aimed more at being able to drive automation scheduling from your Desktop! Entirely for someone like me who likes to do a certain amount of automation from the Desktop but has a particular distaste for scheduled tasks – like the Universal Automation platform this is entirely driven for PowerShell!

“Desktop edition comes packaged as an Electron app that provides all the great automation features of UA without role-based access, remote access or authentication.”

You can use Universal Automation Desktop for free as a trial:

  • 25 Jobs per day

  • Up to 2 concurrent jobs

We are going to be using the Trial here – however, per-user pricing can be found at the following link: Universal Automation

The toolset is quite intuitive a lot of below isn’t worth going into how do use it - as it seems to be easy to pick up but its always pleasant to have it documented and referable! In my example below, we are going to create a Resource Group in Azure.

  • TOC {:toc}

Install Universal Automation Desktop:

Unlike Universal Automation & Universal Dashboard, installation of the Universal Automation Desktop is packaged into an executable.

  1. Download the latest Universal Automation Desktop installer (bottom of the download pages – at time of writing the installer is 120MB and version 1.0.0)
  2. Installation of Universal Automation Desktop is pretty straight forward, just run the downloaded installer: UniversalAutomationInstall
  3. Once complete, Universal Automation Desktop will load.

Use & Configure Universal Automation Desktop:

Add Scripts

Universal Automation supports git, so a Repository folder is created automatically – any scripts that you add will automatically be added to it:

%LOCALAPPDATA%\UniversalAutomation\Repository

  1. On the Scripts pane select Add Scripts UniversalAutomationScriptsPane
  2. Select the script you want to upload – in my example; I am using ‘New-AzureResourceGroup.ps1’ the script I created for quickly testing some of the functionality.

Gist of script found below, but its also in my GitHub Repository under Azure (GitHub link on the site menu).

  1. Once added you should see the script appear and you should be able to see it in the Repository folder now: UniversalAutomationScriptsPanePopulated

Add variables

Universal Automation Desktop supports variables.

  1. Click on the Variables menu item
  2. Select Add Variable
  3. In my example, I am adding the location that the Resource Group will be created, so it is going to be the following Key = Value pair: Location = Australia East
  4. Click Ok to save

Note: Location is a variable in my script, I also tested manually setting the name of the Resource Group as well with the Name value as well and worked well.

UniversalAutomationVariablesPopulated

Note: The Variables are not encrypted! They are in plain text under:

Repository\.ua\variables.ps1

I did have a few issues with the UI freezing on me, so also discovered that I can manually add variables to this file and after a restart, it seemed to be picked up by Universal Automation as well.

UniversalAutomationVariablesVSCode

Change PowerShell version

This is an interesting feature, that allows you to specify what Version of PowerShell you can have the scripts run under (in this example I will be adding PowerShell 7 preview). We do not need this for my example.

Note: If you do not see the below, you may need to update – Automation Desktop will update automatically and should notify you – close and restart Automation Desktop to continue (if you get an error message – navigate to your notification tray by the time and Quit any open Universal Automation Desktop applications you have open and then relaunch).

  1. Click Settings
  2. Navigate down to PowerShell versions
  3. Click Add New Version
  4. A new Table row will appear (Version\Path)
  5. In Version we are going to type in: PowerShell 7-preview (x64)
  6. In path type in: C:\Program Files\PowerShell\7-preview\pwsh.exe
  7. Press Enter UniversalAutomationPowerShellVersions

Now when you run your scripts, you can now specify what Version of PowerShell to use!

Run the script

Now that the variables have been set up and the script has been added, we can then Run it.

  1. On the scripts pane select ‘New-AzureResourceGroup.ps1’ and select Run Universal_Automation_ScriptsRun
  2. Specify the PowerShell version – I believe automating this selection is currently in the backlog: - and click Run Universal_Automation_ScriptsRunVersion
  3. The script will now go to the Jobs screen: Universal_Automation_ScriptsRunJob
  4. Usually, the script would just run – but in my case, I have a parameter in my PowerShell script to request the name of the Resource Group we are going to create, click on Response to Feedback icon
  5. Type in the name of the Resource Group we are going to create – in my example I am going with: UAutomationRGTest and click Ok Universal_Automation_VariablesFeedback
  6. It will now run the script: Universal_Automation_Script Runs
  7. My new Resource Group has been created in Azure, using the name specified in the Parameter (UAutomationRGTest) and the Location (Australia East) that was set in the Variables!

Universal_Automation_Azure Resource Group created

Scheduling scripts

Although at this stage, I am not scheduling any of my scripts to run – it is a core function of the toolset.

  1. Click on Scripts
  2. Select the script you want to schedule and select View
  3. On the right-hand side blade next to Edit, click on the ellipsis (i.e.…) Universal_Automation_Schedule
  4. Select Schedule Universal_Automation_Schedule
  5. Specify the schedule you want and click Ok

You should now see the Schedule under Schedules and view the Job history under Jobs.

Overall opinion

Long story short - Universal Automation has a place and is a toolset I will be looking at more closely and using!

I see myself using it to utilize PowerShell and automation a bit more in completion of general day to day activities (both personal and professional) and service requests - without having to worry about moving to the next step with a bigger toolset.

If I use a script often enough – then there will be a definite need to move to another team based toolset with RBAC tools such as the Universal Automation offering by Adam Driscoll of Ironman Software.

Word of warning – and it should go without saying :

DO NOT RUN UNIVERSAL AUTOMATION DESKTOP ON YOUR DOMAIN COMPUTER FOR PRODUCTION OR SHARED SCRIPTS! PLEASE LOOK AT UNIVERSAL AUTOMATION FOR THAT! YOU DON’T WANT TO GO HOME OR SHUTOFF YOUR PC OR LEAVE FOR BETTER AND BRIGHTER THINGS AND GET CALLED UP BECAUSE SOME VERY IMPORTANT PROCESS DIDN’T RUN!

My Test Script - New-AzureResourceGroup

I created this function to quickly test 2 things:

  • How does Universal Automation work with 3rd party modules?

  • How does Universal Automation work with parameters and variables?

Universal Automation Desktop does not touch your scripts, in fact depending on what your use case is your git repository should be inline with Automation Desktop and you can sync the Variables across multiple installs.

My script is using 2 modules:

  • CredentialManager

  • Azure (AZ)

I thought CredentialManager would be a good test here as Universal Automation is intended to be run from your Desktop (in my case Windows 10) and using Credential Manager to store my Azure SPN details – without revealing it in plan text was a good test. More information can be found below:

ToastIT - Safe Credentials

New-AzureResourceGroup.ps1

#requires -Version 2.0 -Modules Az.Accounts, Az.Resources, CredentialManager


function New-AzureResourceGroup
{
<#
.SYNOPSIS
Creates Azure Resource Group
.DESCRIPTION
Creates Azure Resource Group function, created as a test function for Universal Automation Desktop
.EXAMPLE
New-AzureResourceGroup
#>
param
([Parameter(Mandatory = $true, HelpMessage = 'Enter the name of the Resource Group you want to create', Position = 0)]
[ValidateNotNullorEmpty()]
[string] $Name,
[Parameter(Position = 1)]
[string]
$Location = 'Australia East'

)

$tenantId = (Get-StoredCredential -Target 'MSDN SPN Demo').GetNetworkCredential().UserName
$pscredential = (Get-StoredCredential -Target 'MSDN SPN Demo Key')

Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant $tenantId

New-AzResourceGroup -Name $Name -Location $Location -Force
}

New-AzureResourceGroup

Luke - GitHub