Scanning and analyzing invoices using Xamarin.Forms and Azure

I stumbled upon this neat service from Azure, namely the Azure Form Recognizer. This allows you to analyze documents (like pictures or PDF’s) of invoices and extract the structure and text fields of the invoice, f.ex. vendor name, total amount and due date. I figured it would be neat if you could combine this with a mobile app to easily scan invoices, like some of the newer mobile bank applications allow you to do these days. So, here is how you can do it using a Xamarin.Forms app and a combination of Azure services.

Create the app

Start off by creating a new blank Xamarin.Forms app. Make sure it uses the newest version of Xamarin.Essentials. As of writing this would be 1.7.0.

Take the picture

Use Xamarin.Essentials to take or pick a photo. I modified the MainPage.xaml with a button where the Clicked-event does something like this:

var photoResult = await MediaPicker.PickPhotoAsync();

Upload the picture

A small disadvantage with the Form Recognizer API is that the document to be analyzed has to be in the form of a URI, which means that the image has to be hosted somewhere. There are multiple ways of doing this, but since we already are going to use Azure, we can use Azure Blob Storage for this purpose.
The idea here is that when we’ve taken the picture, we upload it to the blob storage, retrieve the public URL for it and use that for the analysis. When we’re done analyzing, we clean up our resources by deleting the blob and its container.

We’ll start off by retrieving the file name and the full path of the image we took, which we’ll use further on:

var fullPath = photoResult.FullPath;
var fileName = photoResult.FileName;

Next we’ll upload the image to the blob storage. Follow this Microsoft quickstart for how to set up a storage account, how to retrieve the connection string for it and which NuGet package to add to your project. We’ll use some of the example code from the quickstart with some slight modifications:

var blobServiceClient = new BlobServiceClient("yourconnectionstring");

var containerName = "formrecognizerblobs" + Guid.NewGuid().ToString();

BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName, PublicAccessType.Blob);

var blobClient = containerClient.GetBlobClient(fileName);

await blobClient.UploadAsync(filePath, true);

var absoluteUri = blobClient.Uri.AbsoluteUri;

Notice that when creating the blob container we set its access type to PublicAccessType.Blob. This is so that we can use the public URI anonymously. We’ll store and use the absoluteUri-variable for this later.

Analyze the picture

Now we can finally analyze the picture and retrieve the structured data. The following quickstart for how to use the Form Recognizer C# client library shows us what we need to set up in Azure and how we should use the Prebuilt model with the invoice model. We’ll use the same code example, only we’ll swap out the invoiceUri with the link to our uploaded blob:

Uri invoiceUri = new Uri(absoluteUri);
...

By using the example you can see how we can extract the desired values. In our MainPage.xaml, let’s add a Label with an x:Name set to InvoiceTotalLabel. In the example code where the invoice total is being extracted, let’s add an extra line at the end:

if (document.Fields.TryGetValue("InvoiceTotal", out DocumentField invoiceTotalField))
{
    if (invoiceTotalField.ValueType == DocumentFieldType.Double)
    {
        double invoiceTotal = invoiceTotalField.AsDouble();
        Console.WriteLine($"Invoice Total: '{invoiceTotal}', with confidence {invoiceTotalField.Confidence}");
        InvoiceTotalLabel.Text = $"Invoice Total: {invoiceTotal}";
    }
}

Finally, after we’re done analyzing and saving all the data that we need further on, we should delete the blob and its surrounding container as we no longer need it.

await containerClient.DeleteAsync();

And here you can see it all in action! I added some extra labels to show some more of the extracted data.

Summary

This guide showed you how to create a Xamarin.Forms app, take or select a picture of an invoice and how to extract the structured data from it using the Azure Form Recognizer. I’m sure this can be very helpful for business applications, f.ex. those that needs to expense invoices. This Azure service is super powerful and, by the looks of the documentation, can be used for so much more than just invoices. You can also play around with the Form Recognizer Studio to see its capabilities without having to set up everything locally. Give it a spin!

I’ve provided a sample code on GitHub if you want to clone it and check it out.