How to resolve “An exception was forcibly closed by the remote host” in a .NET app running on an azure VM behind a load balancer.

I get the error “An exception was forcibly closed by the remote host” occasionally at work. This is an I.O exception and happens when a HTTP request fails to reach the destination host. To troubleshoot, I often ping or tracert the destination IP/URL, and the error usually comes down to the firewall restricting the connection, and goes away once we have updated the firewall to handle the connection. However, another instance when I get this error is when my app that runs on an azure VM of a load balancer tries to send a request to another app which has a DNS that points to the same load balancer.

We host both of the web apps on the nodes of the load balancer, and one of them calls the other. For each app, we use a DNS entry that points to the load balancer. It had taken me some good hours of debugging why the apps could not talk to each other before I stumbled across Microsoft’s document. Per the document,

If your application hosted in the backend VM of a Load Balancer is trying to access another application hosted in the same backend VM over the same Network Interface, it is an unsupported scenario and will fail.

Troubleshoot Azure Load Balancer

Another cause of failure is when one of the nodes of the load balancer accesses the front end of the load balancer, and the load balancer routes the request back to the originating node instead of another nodes in the same pool.

If an internal Load Balancer is configured inside a VNet, and one of the participant backend VMs is trying to access the internal Load Balancer frontend, failures can occur when the flow is mapped to the originating VM. This scenario is not supported.

Troubleshoot Azure Load Balancer

The document has suggestions for fixing the issue. Besides those suggestions, you can also try one of the following simple workarounds.

• On each of the VMs, add entry in the host file to bypass the load balancer.

The hosts file translates hostnames to IP addresses. The OS checks this file to see if it can get the destination IP address based on the DNS. On Windows, you can find the hosts file at: C:\Windows\System32\drivers\etc. As an example, if your receiving app is reachable by the DNS name “app.company.com”, you can add an entry similar to below to map the DNS to the same VM:

127.0.0.1 app.company.com

• If the app is a .NET application and runs on IIS, you can add a binding for the receiving app and use that when making requests from the sending app.

For example, you can add a binding to bind to localhost on a custom port and use that in the receiving app.

That’s it.

References

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-troubleshoot