We covered how to train and publish the object detection AI model in another blog. This blog will go through how to use the AI model in a real-world situation.

• We’ll submit the input from SharePoint in our demonstration.
• A user must first build a list item and then upload the image to SharePoint.
• The Power automate flow will be activated next.
• The next step is to deliver the data or file content to the AI Model via Power automation.
• The result/output is returned by the Next AI model.
• Finally, Power automates the modification of the result in the SharePoint List, allowing us to double-check the data

How to use Power Automate to consume AI models step by step

1. Generate a SharePoint list with the fields listed below. These are the picture and AI objects/data elements that will be collected.

a. Soccer Ball Count – Integer

b. Confidence – Decimal

c. Analyze Complete – Boolean

 

2. Develop automated cloud flow

a. Go to https://make.powerapps.com to get started.

b. Choose a suitable setting.

c. Make a new solution (If not exist already)

Note: There are some limitations in the default setting.

d. Click New to create Cloud Flow

e. Source: SharePoint

f. It Triggers when an item is created or modified

g. Type in the name flow.

h. To get SharePoint Attachments, add a new action.

• Choose the same Site and list
• The id from the Trigger step should be used.

I. Add an action to access the content of a SharePoint file.

• Choose the same site.
• The ‘Id’ from the “Get attachments” step should be used as the File Identifier.

j. Add action to AI Builder

• Action: Predict

k. Choose your AI model for object detection that we trained in our previous article.

• Set the image from Step #9 as the file content.

l. Add a SharePoint action to update an item.

• Source: SharePoint
• Action: Update Item
• From the trigger, set the Id and Title.
• Update other fields
• Soccer Ball Count
• Formula: length(outputs(‘Predict’)?[‘body/responsev2/predictionOutput/results’])
• Confidence %: mul(items(‘Apply_to_each_2’)?[‘confidence’],100)
• Analyze Complete – This is to avoid recursive (infinite loop)
• Because we may gather or anticipate several objects from the same image, another “Apply to each 2” has been included.
  m. Put the AI Model to the Test
• To create a list in SharePoint,
• Go to the Add List item
• Click save

3. Test Image

4. We should be able to observe the outcome in a matter of minutes at most.

5. The end result is as follows:

6. The degree of confidence is 78 percent. To boost confidence, we need to practice with more photos.

Detect, find, and identify specific objects in any image using the object detection AI model. This model may be used with Power Automation to gather data from images from any data source, including SharePoint, Data Verse, and others. Alternatively, we can create this model with the Canvas app from camera images.

We propose at least 15 photographs per object to train the model, but more than 50 images are preferable. Two objects in the same photograph can be tagged. We can, for instance, use the same image to tag a basketball and a soccer ball.

The image should be in a.jpg,.png, or.bmp format, with a maximum file size of 6 MB. In this case, AI detects the soccer ball that we trained it to detect.

Step-by-Step Instructions for Training the Model

1. Visit https://make.powerapps.com to get started.
2. Choose a setting

3. Expand AI Builder from the left – hand menu and pick Build. Choose Object Detection from the drop-down menu.

4. Give your model a name and then press the Create button. Each object should have at least 15 images.

5. Choose the domain that defines the object categories. This option does not have to be exact. Selecting common things in our scenario.

6. Make a list of object names for our model to recognise. In our situation, there is only one option, which is “Soccer Ball.” We can undoubtedly locate or notice a wide range of objects. To continue, click next.

Instead of inputting each object name individually, we can add them from the database.

7. Insert images to train. A minimum of 15 pictures are required for each object. However, I suggest getting between 30 and 50 photographs. More examples in your training data could help us enhance our model.

8. Press “Next” once your uploads are complete.
9. Hover your mouse over the highlighted selection or select a specific region to double-check the objects.

10. Select “Done Tagging” on the top right hand corner after you ‘ve completed the selection of all objects on all pictures.

11. After that, click “Next” and then “Train.” The model will begin training right away.

12. Hold until the model has been trained. Find and open the models you’ve saved.

13. On the prototype, choose “Quick test.”

15. Close the window and submit the model for publication.

We explained how to train and publish the Form processing AI Model in another blog. This blog will go through how to use the AI model in a real-world situation.

We’ll submit the input from SharePoint in our demonstration.

• The user must first upload the content to SharePoint as the first step.
• The Power automate flow will be activated next.
• The next step is to deliver the data or file content to the AI Model via Power automation.
• The result/output is returned by the Next AI model.
• Finally, Power automates the modification of the result in the SharePoint List, allowing us to double-check the data.

How to use Power Automate to consume AI models step by step

1. Make a SharePoint list with the fields listed below. These are the objects/data elements that the form will gather.

• Passport Number – String
• Date of Birth – Date and Time (Date Only)
• Place of Birth – String
• Issue Date – Date and Time (Date Only)
• Expiry Date – Date and Time (Date Only)

2. Create a cloud flow that is automated.

• Go to https://make.powerapps.com to get started.
• Choose a suitable setting.
• Make a new solution (If not exist already)

 Note: There are several limitations in the default setting.

3. To build a Cloud Flow, go to File > New > Cloud Flow.

4. Source: SharePoint
5. Trigger: When an item is created or modified

6. Fill in the flow’s name.
7. Choose your SharePoint site and list.

8. To get SharePoint Attachments, add a new action.

• Select the same Site and list.
• The id from the Trigger step should be used.

9. Add a SharePoint File Content action

• Choose the same location.
• The ‘Id’ from the “Get attachments” step should be used as the File Identifier.

10. Send the contents of the file to AI Model.

• Choose the AI model that we designed.
• Choose a form type (this may be done dynamically to determine the file type/extension)

11. Add the results to a SharePoint list item.

• From the Model findings, we gathered the following 5 fields.

12. Full view of the cloud flow

13. In SharePoint, create a new list item.

• Include a file attachment (Test Passport)
• Save the list to your computer.

Flow should be activated at any time.

14. Check the results in the SharePoint list.

AI Form processing retrieves text from any matching form by identifying the structure of your documents based on examples you supply. We’ll develop and train a model to recognise passport documents in this demo.

This model can be used to process invoices, passports, driver’s license, and tax forms. There are six steps to creating an AI model.

For today’s demo, we’ll use the business requirement of processing or collecting information from passports, such as the passport number, issue date, and expiration date.

Step-by-Step Instructions to Train the Model

1. Go to https://make.powerapps.com to get started.
2. Choose an environment
3. Expand AI Builder and choose Build from the left menu. choose form processing
4. Create your model by giving it a name.
5. To collect data from the document, add tags (data elements).
6. To continue, click Next. Select “New collection” and give it a name.
7. To add documents, click “0 documents.” To train the model, you’ll need at least five documents. Confirm the selection and click Upload.
8. Select “Analyze” after your uploads are complete.

9. Hover over the highlighted fields to validate the fields that our trained model should return while processing the form.
10. After you’ve double-checked all of the fields on all of the documents, click Next.
11. Click “Train
12. Model is training
13. Wait until the model has been trained. Locate and open the models you’ve saved.
14. On the model, click “Quick test.”
15. Quick Test

• Perform a test using the test passport copy, which can be in either pdf or jpg format. To test, drag and drop a picture.

16. Examine the trained model results.
17. Publish the model

Below are five different types of AI models.

1. Form Processing
2. Object Detection
3. Entity Extraction
4. Category Classification
5. Prediction

1. Form Processing – This model extracts text from any matching form by identifying the structure of our documents based on examples we provide.

We have five data attributes to obtain from the passport in this example: Passport Number, Birth date, Date Of issue, Date Of expiry, and Birthplace. The image above depicts how we prepare the model to process forms.

Use case: Models can be trained to process invoices, passports, driver’s licence, and tax forms.

recommendation: Microsoft advises training with a minimum of five papers. However, it is preferable to train the model with more documents. I can tell that effectiveness is determined by the number of documents and the kind of documents we trained on, such as jpeg, pdf, and various formats.

2. Object Detection — This model aids in the detection and identification of certain items within any image. This model may be used in PowerApps to extract data from photos taken with the camera.

recommendation: Microsoft suggests 15 or more pictures per object, while 50 or more is preferable. In the same image, we can tag two items. Each image should be up to 6 MB in size and be in one of the following formats:.jpg,.png, or.bmp.

3. Entity Extraction — Depending on our business requirements, this model extracts relevant data from text. This model aids in the transformation of unstructured information into structured information.

Use case: The model can extract data from emails and save it as distinct metadata in Sharepoint, DataVerse, or any other application.

The maximum length of a document is 5,000 characters. English, Chinese-Simplified, French, German, Portuguese, Italian, and Spanish are all supported languages. We have a couple listings of entity kinds that are supported.

4. Classification by Category — This methodology divides the text into categories. It can be used with the canvas app or with Power Automate.

 

Use case: Sentiment analysis, spam detection, and customer request routing are examples of use cases.

Recommendations: Each tag must have at least 10 text entries.

5. Prediction – Based on the past data we provided, it makes predictions. This model forecasts future events.

Types:

1. There are two alternatives available. (binary)
2. From a variety of possible outcomes
3. When the response is a number.

What exactly is an AI model?

Models of artificial intelligence (AI) that are aimed to improve or automate our business processes. AI builder is a software that allows you to create AI models.

What makes Artificial Intelligence, Machine Learning, and Deep Learning different?

Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL) are three methods for analysing data.

Logic, if-then rules, and decision trees are all examples of AI techniques.

Machine Learning – This method allows machines to get better at tasks as they gain experience.

Follow the instructions or be supervised – task driven

Learn from Data Reinforcement – Learn from Errors or Learn from Experience Unsupervised – Data-Driven or Learn from Experience

Deep learning

This method allows the computer to train itself to accomplish certain tasks. Example Analysis of images, videos, and voices.

Flow/Process: The process entails sending data to the model and receiving output in some form as a result of the model.

To construct an AI model, we have to follow the methods outlined below.

There are five different types of AI models accessible. This is something we’ll talk about in another blog.

1. Form Processing – Extract text from any matching form
2. Object Detection – Count, Locate, and Identify objects on any Image
3. Entity Extraction – Specific data from the text
4. Category Classification – Classify the text into categories
5. Prediction – Predict future outcomes