{"id":7275,"date":"2022-01-31T07:00:00","date_gmt":"2022-01-31T15:00:00","guid":{"rendered":"https:\/\/newed.any0.dpdns.org\/en-us\/power-platform\/blog\/power-automate\/using-the-power-platform-to-automate-the-classification-and-validation-of-us-vaccination-record-cards\/"},"modified":"2025-06-11T07:51:03","modified_gmt":"2025-06-11T14:51:03","slug":"using-the-power-platform-to-automate-the-classification-and-validation-of-us-vaccination-record-cards","status":"publish","type":"post","link":"https:\/\/newed.any0.dpdns.org\/en-us\/power-platform\/blog\/power-automate\/using-the-power-platform-to-automate-the-classification-and-validation-of-us-vaccination-record-cards\/","title":{"rendered":"Using the Power Platform to Automate the Classification and Validation of US Vaccination Record Cards"},"content":{"rendered":"

Hello Automation Enthusiasts! My name is John Straumann and I am an Automation Specialist with the Mighty South region here at Microsoft.<\/p>\n

In this post we will learn how to easily and quickly build a solution to automate the validation and data extraction of COVID-19 vaccination cards. We will use vaccination cards from the US, but the same steps can be used for vaccination cards or documents from other countries. We are going to go step-by-step to build an image classification model, an information extraction model, and then use those models in a Power Automate automation.<\/p>\n

We will seek to demonstrate the tremendous value the Power Platform can provide for Microsoft Customers, in that every step we will see is part of the platform, and can be performed by a Citizen Developer, i.e., professional coding skills NOT required!<\/p>\n

The solution showcased in this article uses several components from the Microsoft Power Platform:<\/p>\n

Power Automate:<\/b> lets users create automated workflows between applications and services.<\/p>\n

AI Builder<\/b>: a Microsoft Power Platform capability that allows you to easily add intelligence to your automations and apps and predict outcomes to help improve business performance without writing code.<\/p>\n

LOBE:<\/b> Lobe helps you train machine learning image classification models with a free, easy to use tool. Just show it examples of what you want it to learn, and it automatically trains a custom machine learning image classification model that can be shipped in your app.<\/p>\n

In addition to common scenarios addressed by AI Builder such as Invoice and PDF processing, lately we are seeing requests from customers to help with the automated processing of COVID-19 vaccination cards. One such scenario we saw recently was a customer who had employees emailing COVID-19 vaccination cards, and the customer wanted the cards analyzed for completeness of information, and then have the card document stored to a central location on their intranet. Please note that such a scenario could have implications regarding the storage of Personal Health Information, so please be sure your organization understands and complies with all local, state, and federal laws regarding such information. In this example all of the personal information on the cards is simulated.<\/p>\n

The solution follows these process steps:<\/p>\n

    \n
  1. A Power Automate automation flow scans Outlook for incoming emails with COVID cards attached. The automated flow retrieves the attachment from the email, and if the subject is \u201cCOVID Vaccination Card\u201d then uses an AI Builder model to determine if the attachment is a COVID-19 Vaccination card<\/li>\n
  2. If yes, the information from the card is extracted using the AI Builder model. If no, the automation can either stop or report an error, in this example we will simply stop processing.<\/li>\n
  3. The automation then uses an Image Classification model to detect whether the COVID-19 vaccination card has a QR code or not.<\/li>\n
  4. If the card has a QR code, an AI Builder model scans the card and extracts the information of the card, validates it, and saves the card to OneDrive for Business. At this point the automated flow could connect to a service and validate the QR card, depending on what services are available in the area.<\/li>\n
  5. The final step is an AI Builder model scans the card and extracts the information entered on the card, validates the saves the card to OneDrive for Business.<\/li>\n<\/ol>\n

    So, let\u2019s look at how this example was built.<\/p>\n

    \u00a0<\/p>\n

    Building an image classification model to classify vaccination cards with or without QR codes<\/h2>\n

    Step 1:<\/b> Obtain some sample COVID Vaccination Card images.\u00a0You can download sample cards\u00a0here<\/a> . Be sure to get at least 5 cards with QR codes, and 5 cards without QR codes, and fill in the relevant information with demo data as shown:<\/p>\n

    \"\"
    \n\"\"
    \nStep 2:<\/b> Build your LOBE prediction model. Go to     https:\/\/www.lobe.ai\/<\/a> and download and install the Lobe desktop app. Once installation is done, run the Lobe application and select New Project.<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 3:<\/b> Click Import and browse to the directory where the training images are stored on your local hard drive.<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 4<\/b>: Click \u201cImport Images from your Computer\u201d<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 5:<\/b> Lobe classifies images according to Labels, in this example we are using 2 simple labels: QR and NOQR. Select the Images for the QR label and click Open.<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 6:<\/b> Label each Card with a QR Code image with the QR label<\/p>\n

    \"\"
    \n\"\"<\/p>\n

    \u00a0<\/p>\n

    Step 7:<\/b> Follow the same steps and import and label the NOQR images.<\/p>\n

    \"\" \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 8:<\/b> This screen shows what the LOBE app should look like when you\u2019re done. Next, we will export the model for use in Power Automate, click Use.<\/p>\n

    \"\"
    \nStep 9:<\/b> Click Export<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 10:<\/b> Click Export your model to AI Builder<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 11:<\/b> You\u2019ll be asked to sign in<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 12:<\/b> Pick your account<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 13:<\/b> Click Continue at this prompt<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    \u00a0<\/p>\n

    Step 14:<\/b> At this prompt, select the Check Box and Click Open<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 15:<\/b> Name your Model, pick your environment, and click Export.<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 16:<\/b> Click \u201cOptimize and Export\u201d or just click \u201cExport\u201d, in production you should always optimize the model, however since this is a demo\/lab it’s not necessary.<\/p>\n

    \"\"
    \n\"\"<\/p>\n

    \u00a0<\/p>\n

    Step 17:<\/b> Click Close<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Step 18:<\/b> Go to Power Automate on the Web: Microsoft Power Automate | Microsoft Power Platform<\/a>, expand AI Builder (1.), and your Model should show up (2.). If you do not see your model, ensure you are in the same environment as the Model was published to (3.)<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    Now the LOBE image classification model is ready to be used in an Automation. The next step is to build and train our COVID-19 vaccination card form processing model in AI Builder.<\/p>\n

    \u00a0<\/p>\n

    Building a form processing model to extract the information from the vaccination cards<\/h2>\n

    The first step in defining an AI Builder form processing model is to determine the fields you need. Looking at our image of a COVID-19 Vaccination card:<\/p>\n

    \"\"<\/p>\n

    \u00a0<\/p>\n

    We can define our needed fields list as:<\/p>\n

      \n
    1. Header<\/li>\n
    2. LastName<\/li>\n
    3. FirstName<\/li>\n
    4. DOB<\/li>\n
    5. FirstDoseProduct<\/li>\n
    6. FirstDoseLot<\/li>\n
    7. FirstDoseDate<\/li>\n
    8. SecondDoseProduct<\/li>\n
    9. SecondDoseLot<\/li>\n
    10. SecondDoseDate<\/li>\n
    11. FirstDoseSite<\/li>\n
    12. SecondDoseSite<\/li>\n<\/ol>\n

      With our fields list defined, we can build our model in AI Builder. The steps are:<\/p>\n

        \n
      1. Login to Power Automate<\/a>, expand AI Builder in the left nav, select “Build”, and then select\u00a0 “Form Processing”
        \n\"\"\"\"<\/li>\n
      2. Type a descriptive name (1), then select Create (2)
        \n\"\"<\/li>\n
      3. We need to set up an element for each of the fields we defined. So, for each field, select the Add drop-down, and select Field.
        \n\"\"<\/li>\n
      4. Enter each name (1), then select Done (2) and repeat the steps until all the fields are defined
        \n\"\"<\/li>\n
      5. When you are done defining fields, the screen should look like this screen, select Next
        \n\"\"<\/span><\/span><\/span><\/li>\n
      6. The next steps are where we see some of the magic of AI Builder in mapping the fields we defined to the documents. Because we have 2 distinct vaccination card layouts, we are going to create 2 Collections. Click \u201cNew Collection\u201d
        \n\"\"<\/li>\n
      7. Select the \u201c+\u201d button to add documents to the collection
        \n\"\"<\/li>\n
      8. Select \u201cAdd Documents\u201d
        \n\"\"<\/li>\n
      9. Select \u201cUpload from Local Storage\u201d
        \n\"\"<\/li>\n
      10. Browse to the folder where your documents are, select each document to upload (1), then select Open (2)
        \n\"\"<\/li>\n
      11. Select \u201cUpload X Documents\u201d, where “X” will equal the number of documents you selected for upload, in this case, 6
        \n\"\"<\/li>\n
      12. When you see this dialog showing the upload is finished,\u00a0 select \u201cClose\u201d
        \n\"\"<\/li>\n
      13. Select the name field (1) and rename the first collection. Then select New Collection (2).
        \n\"\"<\/li>\n
      14. Name the 2nd collection, and then follow the same steps to upload the next set of documents.
        \n\"\"<\/li>\n
      15. When you\u2019re done, you\u2019ll see this screen. select Next.
        \n\"\"<\/li>\n
      16. These next steps are where the real magic of AI Builder occurs! For each field we defined, we need to highlight that field on the documents and then map the field. This will teach the AI model how to extract the data we want. Select the first Collection.
        \n\"\"<\/li>\n
      17. Each document that needs to be mapped is marked with a red circle. Select the first document, and then we need to select each data field and map it to its corresponding element. To do this, simply click on the data field on the document, and then select the field you need from the list that appears. For a field that is multiple words, such as the Header field which is something like \u201cCOVID-19 Vaccination Record Card\u201d, use the mouse to draw a box around the entire line and then the correct field from the list that appears. Sor for example, we want the header field as shown:
        \n\"\"<\/li>\n
      18. So, we use the mouse to draw a box around all the words in the header, then select \u201cHeader\u201d from the list that appears:
        \n\"\"<\/li>\n
      19. When done, you will see the selection box around the data turns green, and a green check mark appears next to the selected field.
        \n\"\"<\/li>\n
      20. Next, we follow these steps for each field and for every sample document. Note that if a data element is not on the document, you select \u201cNot Available in Document\u201d from the menu to the right of the field. When you are finished with a document, the screen will look like the screen below, and then you select the next document in line to continue the process. When you select the next document, AI Builder will save your progress. Repeat the steps for all the documents you uploaded as part of the collection.
        \n\"\"
        \n\"\"<\/li>\n
      21. When you\u2019re done, the screen will look like this, select the back arrow next to the name of the first collection.
        \n\"\"<\/li>\n
      22. And repeat the steps for the 2nd collection of documents
        \n\"\"<\/li>\n
      23. When you\u2019re done with all the documents in both sets, select Next
        \n\"\"<\/span><\/span><\/span><\/li>\n
      24. On this screen select \u201cTrain\u201d
        \n\"\"<\/span><\/span><\/span><\/li>\n
      25. You will see this dialog while AI Builder trains your new model
        \n\"\"<\/li>\n
      26. OK, our model is done training so now we can see it in action! Before publishing your model for use, it\u2019s a best practice to run several tests on sample documents. Depending on the success rate of the tests, you can then decide to go back and continue to refine and improve your model by adding more documents and re-training, or you can publish the Model. Select \u201cQuick Test\u201d
        \n\"\"<\/span><\/span><\/span><\/li>\n
      27. Drag-and-drop an image
        \n\"\"<\/li>\n
      28. Success! Our model successfully recognized all the fields on our sample. Note if you mouse over each field AI Builder will tell you the confidence in the prediction of that field. If you\u2019re not happy with the results, you can continue to refine your model. For production, more tests would be a best practice, and it’s a good idea to test images of varying quality, picture angles, etc. For this example let’s test our other document type. Select \u201cStart Over\u201d\"\"<\/li>\n
      29. Drag and drop a sample of the 2nd document type and see the results, then select Close.
        \n\"\"<\/li>\n
      30. Select \u201cPublish\u201d
        \n\"\"<\/span><\/span><\/span><\/li>\n
      31. You will see this screen:
        \n\"\"<\/span><\/span><\/span><\/span><\/p>\n

        Putting everything together in an automation flow<\/h2>\n<\/li>\n
      32. When publishing is finished, select \u201cUse Model\u201d
        \n\"\"<\/li>\n
      33. Select \u201cNew Flow in Power Automate\u201d
        \n\"\"<\/li>\n
      34. And for some more Power Platform magic, we are presented with several template options to kick start our automation configuration! Select \u201cDocuments are received attached to emails sent to a Microsoft 365 Outlook Work account\u201d. So in this case the “trigger”, or the event that runs a Power Automate flow, is an email arriving.\u00a0 There are many kinds of triggers, such as a schedule, online documents being created or updated, etc.\"\"<\/li>\n
      35. One of the most powerful advantages of the One-Microsoft Power Platform solution is the integration between the various components of the platform such as Office, Power Automate, etc. This platform provides tremendous potential for solutions and value for our Customers! On this screen we see that Power Automate has detected it needs 3 connectors for this template, and has confirmed the connections. If a needed connection did not exist, the platform would step you through setting up the required connection. In addition to out-of-the box connections on the Microsoft Power Platform, we have over 500 connectors to products available here: https:\/\/powerautomate.microsoft.com\/en-us\/connectors\/<\/a> For each connector select Sign In and thanks to the integration of the One Microsoft platform, the authorizations will occur automatically!\"\"<\/li>\n
      36. After authorizing the connections, select Continue<\/span><\/span>\"\"<\/li>\n
      37. Ready for some more magic that shows the power of the platform? Power Automate actually builds almost all of the required steps for us to have a functioning automation, and we can choose to either use the configuration as is, or drag-and-drop more steps as needed to automate our process. Important note, be sure to save your Automation using the \u201cSave\u201d button before leaving the page in your browser.<\/span><\/span>\"\"
        \n\"\"<\/li>\n<\/ol>\n

        The step \u201cExtract information from forms\u201d has been preconfigured to us the AI Builder model we created, \u201cCOVIDVaxCards\u201d and process the email attachment:<\/p>\n

        \"\"<\/p>\n

        Now while this template process is actually quite useful and already runs, it does not meet our business requirements of determining if the attachment is actually a COVID card, and if so, determining if the card has a QR code using our Lobe prediction model, and then of storing the COVID card image on our secure Microsoft OneDrive for Business (https:\/\/newed.any0.dpdns.org\/en-us\/microsoft-365\/onedrive\/onedrive-for-business<\/a>). To achieve our business outcome, we need to modify the automation flow, so let\u2019s do that.<\/p>\n

        First let\u2019s change the Subject Filter on the email trigger so this automation only runs when an email with subject \u2018COVID Vaccination Card\u2019 is received.<\/p>\n

        \"Update<\/p>\n

        Next, let\u2019s define 3 Variables, all of type String, to hold some values in this automation:<\/p>\n

        a<\/strong> A variable to set if a QR code is detected
        \nb<\/strong> A variable for the filename to store the card
        \nc<\/strong> A variable to create a message to the user<\/p>\n

        \"\"<\/p>\n

        \u00a0<\/p>\n

        Now, let\u2019s setup Condition after the \u2018Extract information from forms actions\u2019 that proceeds only if the attachment contains the COVID Vaccination Card header. \u00a0Just click \u201cAdd an Action\u201d, and select \u201cControl\u201d, then select \u201cCondition\u201d
        \nTo set the condition, click in the input labeled \u201cChoose a Value\u201d, and Power Automate will give you a list of choices. We want to check if the header value extracted by AI Builder contains COVID, so select \u201cHeader\u201d.<\/p>\n

        Change the condition in the middle box to \u201ccontains\u201d, and in the last \u201cChoose a value\u201d box, type COVID. So now the Flow will analyze the value of the Header output set by AI Builder, and if it contains the word \u201cCOVID\u201d, the flow will continue.<\/p>\n

        \"Check<\/p>\n

        Now we are going to use the image classification model we built with LOBE and AI Builder to detect if the card has a QR Code or not. Under the \u201cIf Yes\u201d path, click \u201cAdd an action\u201d.<\/p>\n

        \"Add<\/p>\n

        Follow the same steps as before to select the AI Builder connector, but this time select \u201cPredict\u201d
        \n\"\" \"\"<\/p>\n

        Select the “Model” drop down, and select the model we built earlier, “COVIDQR Model”. Note this time we are using the Image Classification Model.
        \n\"\"<\/p>\n

        Click in the “Image” input, and select “Attachments Content”
        \n\"\"<\/p>\n

        For this example we are not going to take any further action if a QR code is found because there is no online service to connect with to perform validation, except to set a message. So, we need to set the value of our QR Variable based on the result from our LOBE prediction model. Click the \u201cAdd an Action\u201d button, search Variable, and select \u201cSet Variable\u201d<\/p>\n

        \"\"<\/p>\n

        We saw earlier that inputs in Power Automate can be set to dynamic values and outputs from earlier steps. So, in this case we want to set our QR variable to the output from our LOBE model, so under the name drop-down, select the QR Variable we created earlier. Then click in the \u201cValue\u201d input, and from the Dynamics Content list, under \u201cPredict\u201d, select \u201cPrediction\u201d. We will use this value later to create a message
        \n\"\"<\/p>\n

        The next step we will take is to save the file to our secure internal OneDrive for Business site. I created a folder on my OneDrive called \u201cCOVID Vax Cards\u201d. We must build a path to the file and save it in the Filename variable we created before. Click the \u201cAdd an Action\u201d button, search Variable, and select \u201cSet Variable\u201d
        \n\"\"<\/p>\n

        For <\/span>our save filename we need to build a path consisting of the URL to our <\/span>OneDrive<\/span>, and a filename. To accomplish that, we now <\/span>introduce Power Automate expressions.\u00a0<\/span> In Power Automate, expressions are a set of simple functions that enable you to return data. Put simpler, you use expressions to write a formula to get your data the way you want it. Expressions allow you to do things like convert a date to UTC, divide two numbers, create that perfect string by combining fields, and much more all by using various functions.<\/span> Click the drop-down next <\/span>to the \u201cName\u201d <\/span>label, and<\/span> select the \u201cFilename\u201d variable we created earlier. <\/span>Then, click in the <\/span>\u201cValue\u201d <\/span>input for the variable, and instead of <\/span>choosing <\/span>\u201dDynamic<\/span> Content\u201d as we did before, select \u201cExpression\u201d.\u00a0<\/span><\/span>\"\"<\/p>\n

        You will see headers for each expression type, and luckily the first one in the list is the one we need, the String \u201cconcat\u201d expression, which combines a number of strings into 1 string.
        \n\"\"<\/p>\n

        Click it, and you will see the expression pasted into the input box. Now we can use dynamic content just like we did before.\u00a0 We need to build a filename of <firstname><lastname>.png. Click the mouse inside the brackets of the expression as shown, then click the \u201cDynamic Content\u201d tab.
        \n\"\"<\/p>\n

        Scroll down to find the AI Builder \u201cExtract information from forms\u201d value for Last name and click it. NOTE be sure to select \u201cLastName value\u201d, not just \u201cLastName\u201d.\"\"<\/p>\n

        Then, in the Fx input box, scroll all the way to the right, and put a comma after the first input, then find and click the \u201cFirstname value\u201d data item.\"\"<\/p>\n

        \u00a0<\/p>\n

        The expression should look like this:<\/p>\n

        concat(outputs(‘Extract_information_from_forms’)?[‘body\/responsev2\/predictionOutput\/labels\/LastName\/value’],outputs(‘Extract_information_from_forms’)?[‘body\/responsev2\/predictionOutput\/labels\/FirstName\/value’])<\/em><\/p>\n

        Put another comma after the last ] bracket, and type \u2018.png\u2019, so now the expression is:<\/p>\n

        concat(outputs(‘Extract_information_from_forms’)?[‘body\/responsev2\/predictionOutput\/labels\/LastName\/value’],outputs(‘Extract_information_from_forms’)?[‘body\/responsev2\/predictionOutput\/labels\/FirstName\/value’], \u2018.png\u2019)<\/em><\/p>\n

        So, for example, if the name on the COVID-19 Card is John Doe, the file name will be JohnDoe.png.<\/p>\n

        Click OK and the expression will be saved and the component will look like this:<\/p>\n

        \"\"<\/p>\n

        Next, click the \u201cAdd an action\u201d button, search OneDrive, and select \u201cOneDrive for Business\u201d, then click \u201cCreate File\u201d\"\"<\/p>\n

        If this is your first time connecting to OneDrive you\u2019ll be asked to sign in, and we see the power of the One Microsoft Platform and the integrated security. We can sign in with our account and we are connected!
        \n\"\"<\/p>\n

        For the Create file inputs, set it up as shown. By now you should be familiar enough with this that you can do these inputs on your own! Note the Folder Path is the name of the OneDrive for Business folder I created, so use whatever folder name you created on your OneDrive.
        \n\"\"<\/p>\n

        Ok we are almost done! The last few steps are:<\/p>\n