cancel
Showing results for 
Search instead for 
Did you mean: 

Sentiment Analysis in Power BI

You may think that Sentiment Analysis is the domain of data scientists and machine learning experts, and that its incorporation to your reporting solutions involves extensive IT projects done by advanced developers. Well, today this is going to change.

 

Today I will show you how to gain Sentiment Analysis insights without the help of machine learning gurus or software ninjas. All you need to do is open your Power Bi Desktop and follow the steps below.

 

A short intro...

For the last six months I have been sharing Power BI scenarios, tips & tricks on my blog DataChant.com blog. One of the most popular scenarios described how to gain insights on your brands and your competitors through their presence on Facebook (Read more here). But I always had an itch to add Sentiment Analysis as new dimension for the insights. For example, with Sentiment Analysis we can better understand competitors' posts, or fans' comments and measure their emotional engagement or track down outliers of negativity in our campaigns.

 

So let's pretend we are the social media analysts of the US Presidential Candidates, and imagine we have a Power BI report like this one to analyze the reactions to Clinton's and Trump's Facebook posts.

 

 

Wouldn't it be cool, if we could also apply Sentiment Analysis on Clinton's and Trump's posts, and correlate the sentiment that arise from their messaging with the reactions they get?  Wouldn't it be awesome if we could drill down the posts by sentiment and high rate of specific Facebook reactions?

 

While you can already apply Sentiment Analysis in Excel using Azure Machine Learning Add-in (read more here), and there are already great posts on doing it the "Power Query" way (for example here), I think that there are many advantages to the technique I will share with you today (For example, you gain better integration in Power BI than with the Add-in; you can post more messages in a single API call; You are using the newest service rather than the soon-to-be deprecated Azure Marketplace),

 

Are you ready to begin?

To perform Sentiment Analysis on Clinton's and Trump's Facebook posts, we first need to pull down their posts. On my blog you can find several techniques to do so. Since our main focus today will be on the Sentiment Analysis part, let's start with an Excel workbook that I prepared in advance: Facebook Reactions Data Sample.xlsx (You can download it from the blog's attachments section).

The Facebook Reactions Data Sample.xlsx workbook contains 500 recent Facebook posts from each of the US Presidential candidates. We will import it to a Power BI Report, and apply Sentiment Analysis on it. Please download this workbook and save it to your computer. We will import it in a minute.

 

Are we going to see here some complex sentiment analysis algorithms?

Of course not. We don't need a mad scientist to help us this time. We will use out-of-the-box Sentiment Analysis API that is already offered for free by Microsoft Cognitive Services.

According to Microsoft, the Sentiment Analysis API "returns a numeric score between 0 and 1. Scores close to 1 indicate positive sentiment and scores close to 0 indicate negative sentiment. Sentiment score is generated using classification techniques. The input features of the classifier include n-grams, features generated from part-of-speech tags, and word embeddings. English, French, Spanish and Portuguese text are supported." (Read more here).

 

So before we start the tutorial itself, let's sign up for Microsoft Cognitive Services here.

 

Screenshot_4.png

After you click the Get started for free button above and follow the easy signup process for Text Analytics Preview, you will reach your account page (here), where you can obtain your Text Analytics API key. Click on the Copy link (as highlighted below) to obtain the key. You will need it soon.

 

 Screenshot_5.png

 

Open Power BI Desktop, click on Get Data drop down menu and select Excel.

 

Screenshot_12.png

 

After you downloaded the attached Facebook Reactions Data Sample.xlsx, select that workbook in the Open dialog and click Open.

 

Screenshot_13.png

 

In the Navigator dialog, select the tables: Candidates, Posts, Reactions and ReactionTypes and click Edit.

 

Screenshot_14.png

 

Now in the Query Editor you can see a preview of the four tables and get a glimpse of the data we have at hand.

 

Our main focus today will be in the technique that allows us to upload the text data to the Sentiment Analysis API and load it to our report. We start by selecting the Posts query which contains the text for analysis. 

Right click on Posts in the left Queries pane, and click Reference.

 

Screenshot_15.png

 

Rename the query Posts (2) to Sentiment Results. To rename the query, right click on it and click Rename.

 

Screenshot_16.png

 

Note: In the next step we will select the columns in our data that contain the actual text for analysis and a unique ID for each text. The unique ID is required by Microsoft Cognitive Services API. The unique ID will help us to map the Sentiment scores in the response to the relevant text. In our data, we use the Facebook Post IDs as unique IDs. If you don't have a unique ID for the text in your own dataset, you can always use Add Column --> Add Index Column to obtain unique ID. 

 

In Home tab of Query Editor, click Choose Columns, unselect all columns, then select Post ID and Message and click OK.

 

.Screenshot_37.png

 

Now, let's rename the columns. Post ID should be renamed to id. Message should be renamed to text. This step is critical. Don't miss it. The Sentiment Analysis API requires these names.

 

Screenshot_17.png

 

Note: If you need to perform analysis on text in French, Spanish or Portuguese, you can create another step here and define a custom column whose name is language and is value is "fr", "es" or "pt" (For French, Spanish or Portuguese). Since English is the default language in the API, we skip this step.

 

In the next step, we will remove rows with empty text messages. There is no point in sending such rows for Sentiment Analysis, and the service will return errors if we try to.

 

Click in the filter icon of the column text, and then click Remove Empty.

 

Screenshot_22.png

 

In the next step we will keep the top 1000 rows. We don't need to do it in our specific dataset that contains 1000 messages. We do it to prevent errors when you try these steps on your own data, since Microsoft Cognitive Services API allows only 1000 text messages in a single API request call.

 

But don't worry, if you have more than 1000 rows stay tuned to my next blog post here with a detailed walkthrough that will also show you how to perform multiple API calls. It was just too long for a single blog post.

 

In Home tab, click Keep Rows, and then click Keep Top Rows.

 

Screenshot_23.png

 

In the Keep Top Rows dialog, set 1000 as Number of rows and click OK.

 

Screenshot_24.png

 

Let's take a short stop, and let the query Sentiment Results have a break. We will return to it soon.

We will now create a new query that sends our data to the Sentiment Analysis service. 

In Home tab, click New Source drop down menu, and select Blank Query.

 

Screenshot_19.png

 

Rename the new query to GetSentimentResults, and In Home tab click Advanced Editor. Copy & paste the code below to the Advanced Editor main box, and then replace the part which is highlighted below in red with the API key that you obtained from Microsoft Cognitive Services (as mentioned above). 

 

(Source as table) as any =>
let
    JsonRecords = Text.FromBinary(Json.FromValue(Source)),
    JsonRequest = "{""documents"": " & JsonRecords & "}",

    JsonContent = Text.ToBinary(JsonRequest, TextEncoding.Ascii),
    Response =
        Web.Contents("https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment?",
            [
                Headers = [#"Ocp-Apim-Subscription-Key"= "[Paste your API key here]",
                           #"Content-Type"="application/json", Accept="application/json"],
                Content=JsonContent 
            ]),
    JsonResponse = Json.Document(Response,1252)
in
    JsonResponse

 

After you place the API key, click Done in the Advanced Editor.

 

Note: Since you have inserted your API key in the code above, make sure with whom you share the Power BI Desktop file (.pbix). The API key will remain visible. If you don't wish to share the API key, you can use  Power BI template with the API Key as a parameter.

 

Screenshot_38.png

 

Going back to the query Sentiment Results, click on the little fx icon on the formula bar. If you don't see the formula bar you can enable it from the View tab.

 

Screenshot_25.png

 

 Change the formula from:

 

= #"Kept First Rows"

 

To:

 

 

= GetSentimentResults(#"Kept First Rows")

Press Enter, and click Edit Credentials in the yellow business bar. In the Access Web Content dialog, keep the default Anonymous option, and click Connect.

 

 

Screenshot_40.png

 

You will now see a record of documents and errors, Click on the List object of documents.

 

Note: I will provide more details about error handling on a followup blog post here. For simplicity reasons, let;s ignore the few errors that the API returns (If you cannot wait for the followup blog, you can truncate the column text to 10240 characters to avoid those errors).

 

Screenshot_27.png

 

Click To Table in List Tools / Transform tab.

 

Screenshot_28.png

 

Click OK in the To Table dialog.

 

Screenshot_29.png

 

Expand the column Column1 by clicking on the little icon which is highlighted below.

 

Screenshot_30.png

 

Again, expand the column Column1. In the expand column pane, select score and id. Then uncheck Use original column name as prefix and click OK.

 

Screenshot_31.png

 

Right click on the header of column score and select Change Type, then select Decimal Number.

 

Screenshot_32.png

 

In Home tab, click Close & Apply.

 

Screenshot_33.png

 

That's it. With a single API call to Microsoft Cognitive Services, you got 1000 scores between 0 (Negative) to 1 (Positive) for the attached Facebook posts that were made by Clinton and Trump.

 

The next steps are the modeling and visualizations. There is nothing new there, so I will let you try it yourself before I publish a detailed walktrhough. Till I do, you can learn here how to create most of the visualizations below, including a cool Pulse Chart that triggers different events depending on your slicer manipulations.

 

 

 

Would you like to learn more? Follow my blog here where I will soon share with you how I created this dashboard.

You will also learn:

 

  • How to handle multiple API calls with 1000 messages on each call.
  • How to handle errors.
  • How to split the messages to sentences for better accuracy of the Sentiment Analysis alogirthm.
  • How to use DAX and a slicer to dynamically define the Negative, Neutral and Positive Sentiment thresholds.
  • How to corrolate Facebook reactions with Sentiment Analysis scores.

 

Thank you for reading,

Gil

Comments

Hi,

Thanks for the great post, I am getting following error when I try to follow the steps provided in above. 

 

Formula.Firewall: Query 'Sentiment Results' (step 'Custom1') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination.

 

Is there any reason to get this error?

Issue.png

Dear mohsenvafa,

Please set Fast Combine to ignore privacy levels. It should resolve this issue.

Here is how you can do it: https://powerbi.microsoft.com/en-us/documentation/powerbi-desktop-privacy-levels/

Thanks, it resolved the issue!

Check out this Power BI Data Gallery Story: Sentiment Analysis on your comments to Power BI Facebook Page

 

Screenshot_25.png.

Good post.  I did a demo of this last week for a webinar using Amazon Product reviews.  I loaded my JSON files to Azure Data Lake Store and parsed them with the JSON serde.  I also parsed a static JSON file with Power BI.  https://www.blue-granite.com/webinar-overview-of-text-analytics-microsoft-azure

Thank you , furlong46. I am looking forward to watching your webcast.

The fact that we can now perform Sentiment Analysis without external Hadoop and R, and use Power BI Desktop for the entire workflow, makes the solution much more accessible for any Excel / BI end-users. For many, the mention of R or Hadoop is enough to keep a distance, and be dependent on their rare big data specialists to proceed. Self-service BI is really "self" in my case.

Hi,

 

many thanks for the great tutorial.

 

Im ama almost through it, but I get the following error I do not understand...

 

 

Captura.PNG

 

 

Any way to solve this? 

 

Many thanks in advance!

BR

Hi DrBiDuro,

 

You have a network issue. Could you copy the URL to your browser and see if you can access the domain?

Please paste the URL here in your reply. The screenshot you shared is blurry.

 

Thanks for the quick reply DataChant!!!!

 

I finally resolved this issue it was indeed a type mistake in the URL and now I get the data perfectly.

 

Now I wanted to apply this method to an internal response database I have. I followed the exact same steps but I get Error 400 Bad request...  Any thoughts?

 

"An error occurred in the ‘’ query. DataSource.Error: Web.Contents failed to get contents from 'https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment' (400): Bad Request
Details:
DataSourceKind=Web
DataSourcePath=https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment
Url=https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment"

 

Again, thanks for your amazing work!

Thank you DrBiDuro.

 

Is there any chance you have more than 1000 messages? Try to keep only the top 1000 rows and try again. I am publishing soon here how to handle messages beyond the first 1000. 

Update:

Follow DataChant next blog post for Part 2 of this article, and learn how to handle how to refine the solution beyond 1000 messages.

DrBiDuro,

 

How did you resolve your URL type mistake?  I am getting the same error and cannot see where I am going wrong?

 

Many thanks

Hi JPC3,

 

I was missing the question mark at the end of the URL. Nevertheless, I think that I actually solved it doing the following:

 

When editing credentials (where you select anonymous autentication), y chose to the more detailed URL, which was "https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment" instead of ""https://westus.api.cognitive.microsoft.com". You can select the URL in this step.

Hope I was helpfull.

 

Hi DataChant,

 

yes, I actually filtered top 900 rows to make sure this was not the mistake but i get the same error...

 

I willl review your next post and see if there is a solution.

 

Thanks!

The issue that DrBiDuro and JPC3 report can be fixed. If you applied the #"Kept First Rows" step, please change the line:

= GetSentimentResults(#"Filtered Rows")

To:

= GetSentimentResults(#"Kept First Rows")

 

The error only happens when you feed the function with a table which is longer than 1000 rows.