Deploying a load-balanced, high-available FTP Server with Azure Files

FTP has been around for a very long time and it looks like it won’t go away soon, especially in “the enterprise” with products like BizTalk, BizTalk Services, … which still use FTP to transfer large chunks of data between different systems. And then we didn’t even mention all those legacy applications using FTP for their core processes. Now depending on your scenario FTP could be a critical component, so in that case you’ll want it to be high-available to avoid any downtime in your business processes.

A typical high-available setup will look like this:

Image may be NSFW.
Clik here to view.

Two or more Virtual Machines will host the FTP Server behind a load balanced endpoint. Now in order to be high-available you have to make sure the data is available from multiple servers (in case a server goes offline), so a possible solution is to use a DFS. DFS replication is a feature which is available in Windows Server, but the only downside to it is that it requires Active Directory Domain Services:

Can I use DFS Replication in a workgroup?
No. DFS Replication relies on Active Directory® Domain Services for configuration. It will only work in a domain.

This means you need to configure and maintain Active Directory just for the sake of running a DFS. In this post we’ll see how we can replace the DFS and Active Directory with Azure Files and make it easier to host a high-available FTP Server.

Adding Azure Files to the mix

In the next few steps we’ll use an Azure File Share as the central storage for our FTP Servers. The advantage here is that this is a managed service so we don’t need to deploy Active Directory or configure a DFS. The only thing we’ll need to do is configure IIS and link it to an Azure File Share.

Image may be NSFW.
Clik here to view.

So the first thing we’ll do is create a storage account (assuming we don’t have one already). One of the endpoints that will be available in our Storage Account is the endpoint for the File Service (since this is a preview feature you’ll need to activate it first).

Image may be NSFW.
Clik here to view.

After the storage account is created we’ll use PowerShell to create a new File Share:

Image may be NSFW.
Clik here to view.

And that’s how easy it is. We now have a File Share with a capacity of 5TB which we can mount from multiple Virtual Machines.

Setting up the Virtual Machines

The first thing we need to do is deploy 2 Virtual Machines that will host the FTP Servers. I created 1 Cloud Service and 1 Virtual Network in which I deployed both Virtual Machines (in the same Availability Set):

Image may be NSFW.
Clik here to view.

Now that our machines are running we’ll configure the endpoints. Each Virtual Machine will have its own set of data ports and port 21 will be load-balanced over both Virtual Machines:

• lbftp1: port 21 and port range 10000-10050
• lbftp2: port 21 and port range 20000-20050

The following script will help you to automate the creation of the load-balanced endpoint and the data endpoints:

# Arguments.
param
(
	[Microsoft.WindowsAzure.Commands.ServiceManagement.Model.PersistentVMRoleContext]$vm = $(throw "'vm' is required."),
	[int]$publicPort = $(throw "'publicPort' is required."),
	[int]$dynamicPortFirst = $(throw "'dynamicPortFirst' is required."),
	[int]$dynamicPortLast = $(throw "'dynamicPortLast' is required.")
)

$totalPorts = $dynamicPortLast - $dynamicPortFirst + 1
if ($totalPorts -gt 150)
{
	$(throw "You cannot add more than 150 endpoints (this includes the Public FTP Port)")
}

# Add endpoints.
Write-Host -Fore Green "Adding: FTP-Public-$publicPort"
Add-AzureEndpoint -VM $vm -Name "FTP-Public-$publicPort" -Protocol "tcp" -PublicPort $publicPort -LocalPort $publicPort -LBSetName "FTP-LB" -ProbePort $publicPort -ProbeProtocol "tcp"
for ($i = $dynamicPortFirst; $i -le $dynamicPortLast; $i++)
{
	$name = "FTP-Dynamic-" + $i
	Write-Host -Fore Green "Adding: $name"
	Add-AzureEndpoint -VM $vm -Name $name -Protocol "tcp" -PublicPort $i -LocalPort $i
}

# Update VM.
Write-Host -Fore Green "Updating VM..."
$vm | Update-AzureVM
Write-Host -Fore Green "Done."

And here’s how you can call the script (assuming you saved it as Add-AzureFtpEndpoints.ps1):

$vm = Get-AzureVM -Name lbftp1 -ServiceName lbftpservers
.\Add-AzureFtpEndpoints.ps1 $vm 21 10000 10050

$vm = Get-AzureVM -Name lbftp2 -ServiceName lbftpservers
.\Add-AzureFtpEndpoints.ps1 $vm 21 20000 20050

After executing these scripts port 21 will be open on our Cloud Service, load-balanced over our 2 FTP Servers and each server will have 50 open ports used for the data channel.

Configuring the FTP Server

The next thing we’ll do is install IIS on each server and create an FTP Server which points to Azure Files. We need to do a few things in orde to make this work:

1. Create a Windows User where the username matches the storage account name and the password matches the storage account key
2. Point the FTP Server to the Azure File Share
3. Set the new Windows User which matches the storage account on the FTP Server (disable pass-through authentication)

The following script installs IIS, creates an FTP Server and takes care of the required configuration for setting up the FTP with Azure Files. The script is based on my post from a few years ago:
Passive FTP and dynamic ports in IIS8 and Windows Azure Virtual Machines.

REM Variables.
SETLOCAL EnableDelayedExpansion
SET FtpSiteName=%1%
SET StorageName=%2%
SET StorageKey=%3%
SET ShareName=%4%
SET PublicPort=%5%
SET DynamicPortFirst=%6%
SET DynamicPortLast=%7%
SET DynamicPortRange=%DynamicPortFirst%-%DynamicPortLast%
SET PublicIP=%8%

REM Install FTP.
START /w pkgmgr /iu:IIS-WebServerRole;IIS-FTPSvc;IIS-FTPServer;IIS-ManagementConsole

REM Add user.
net user %StorageName% /delete
net user %StorageName% %StorageKey%== /add

REM Configuring FTP site.
pushd %windir%\system32\inetsrv
appcmd add site /name:%FtpSiteName% /bindings:ftp://*:%PublicPort% /physicalpath:"\\%StorageName%.file.core.windows.net\%ShareName%"
appcmd set vdir /vdir.name:"%FtpSiteName%/" /userName:%StorageName% /password:%StorageKey%==
appcmd set config -section:system.applicationHost/sites /[name='%FtpSiteName%'].ftpServer.security.ssl.controlChannelPolicy:"SslAllow"
appcmd set config -section:system.applicationHost/sites /[name='%FtpSiteName%'].ftpServer.security.ssl.dataChannelPolicy:"SslAllow"
appcmd set config -section:system.applicationHost/sites /[name='%FtpSiteName%'].ftpServer.security.authentication.basicAuthentication.enabled:true
appcmd set config %FtpSiteName% /section:system.ftpserver/security/authorization /-[users='*'] /commit:apphost
appcmd set config %FtpSiteName% /section:system.ftpserver/security/authorization /+[accessType='Allow',permissions='Read,Write',roles='',users='*'] /commit:apphost
appcmd set config /section:system.ftpServer/firewallSupport /lowDataChannelPort:%DynamicPortFirst% /highDataChannelPort:%DynamicPortLast%
appcmd set config -section:system.applicationHost/sites /siteDefaults.ftpServer.firewallSupport.externalIp4Address:"%PublicIP%" /commit:apphost

REM Configure firewall.
netsh advfirewall firewall add rule name="FTP Public Port" dir=in action=allow protocol=TCP localport=%PublicPort%
netsh advfirewall firewall add rule name="FTP Passive Dynamic Ports" dir=in action=allow protocol=TCP localport=%DynamicPortRange%

REM Restart the FTP service.
net stop ftpsvc
net start ftpsvc

The final step is to execute the script on both servers:

On lbftp1:
Install-FTP.bat nameOfMyFtpServer myaccount mykey== nameOfTheShare 21 10000 10050 publicIp

On lbftp2:
Install-FTP.bat nameOfMyFtpServer myaccount mykey== nameOfTheShare 21 20000 20050 publicIp

If you don’t have IIS installed already it can take a few minutes for the script to execute. Once you’ve executed on all of your FTP Servers you’re all done.

Image may be NSFW.
Clik here to view.

Connecting to the FTP Server

You can now use any FTP client to connect to your Cloud Service and the load balancer will route you to one of the FTP Servers. Go ahead and upload a few files:

Image may be NSFW.
Clik here to view.

Now go back to your Virtual Machine and create a mapped drive which points to the File Share. Here you should see all the files you just uploaded:

Image may be NSFW.
Clik here to view.

And that’s it. You now have a high-available FTP Server without having to configure or manage a DFS with Active Directory.

Enjoy!