Step-by-step instructions to configure Incremental refresh

– Use filters in your visualizations to reduce the amount of data that display: The more data a visual must display, the slower it will load, so apply filters in the Filters pane, or use Top N, to reduce the number of items displayed.

So, if your report is connected to a large data model and you do not apply any filtering, all the data will be loaded into memory and decompressed on each update, creating a huge memory demand.

– Only include the necessary number of visuals in your report: This will decrease the number of calculations that Power BI is performing when rendering your report. For example, you should use a multi-row card instead of using multiple single cards.

– Limit the use of slicers: Because they are visualizations, the underlying data will be updated every time the dashboard is refreshed, which could affect the performance.

– Use Microsoft-certified custom visuals: These have the Microsoft’s guarantee that these visual elements perform at their best, in addition have more options than non-certified ones. 

– Avoid unnecessary interactions between visuals: Power BI automatically creates interactions between all the visuals on a page, which require high loading capacity that could slow down the loading of reports.

Therefore, make sure to eliminate the unnecessary interactions that Power BI creates by default.

– Use Performance Analyzer to Analyze the performance of your report elements: such as visuals and DAX formulas when users interact with them, and which consume the most resources.

✅ Excel Checklist

Download the Excel checklist at the following link: https://github.com/NuricBI/Checklist

Conclusions

In this article we reviewed how to improve and analyze the performance of your Power BI solutions in the different development layers:

✔ Data source.

✔ Data model.

✔ DAX.

✔ Data visualization.

You must be careful in each of them, because in whatever could be the cause that affects the final performance of the report.

Keep handy the excel file with the summarized list of best practices and recommended tools to analyze the performance of each phase of your solution.

In this checklist you will find the tools you can use at every step of the way:

1. DATA SOURCE

Query Diagnostic

 This Power BI Desktop feature will help you understand what happens inside the Power Query when a query is executed, the parts of the query that are being performed locally or remotely, and the slowdowns you may encounter during:

– Loading and transforming your data.

– Refreshing your data in Power Query.

Start a Query Diagnostic:

• Go to the ‘Tools’ tab in the Power Query Editor ribbon. You will have the following options Diagnose Step and Start Diagnostics / Stop Diagnostics:

Start Diagnostics / Stop Diagnostics is more used because it provides complete information about each query while Diagnose Step analyzes the performance of a specific step without showing you the diagnostic of the other steps of the query.

• In Power Query Editor, select the query you want to analyze.
• Go to Tools in the Home ribbon, click “Start Diagnostics”.
• Perform the evaluations you want analyze, as: authoring, refreshing a preview, full refresh, etc.
• When you are finished, make sure that you select “Stop Diagnostics”.

This generates two new queries with the diagnostics results:

In my opinion the most important column is the Exclusive Duration, you should convert it to seconds in the Power Query Editor and then summarize it to get an idea of where time is spent and identify how you can optimize the queries in your power bi models:

2. DATA MODEL

DAX Studio

– Launch DAX Studio. Select the model that you want to analyze:

– Go to the advanced tab and click on “View Metrics”:

This will show metrics that help you quickly detect the performance issues.

Cardinality: This is the number of unique values in a column. High cardinality columns can be particularly expensive, especially when have a string or date/time format.

If possible, remove columns such as GUID and timestamp from a SQL Server table from your model. In addition, I recommend splitting Datetime columns into two columns, one for date and one for time.

Size information: The other important thing you can see is the space that is taking up each table and column of the model, for example:

In this case, the Work Order table is taking up almost 20% of the entire model, which helps you analyze where your model consumes the most memory and choose which columns to remove to optimize it.

Summary information: In the Summary tab you can see the total size of this model, in this example the analyzed PBIX is consuming 198.27 MB. You can also see the number of tables and columns:

Bravo for Power BI

This is a free open-source tool for users who do not need a high level of complexity. In case you need more advanced functionalities you should use other more advanced tools such as DAX Studio and Tabular Editor.

After downloading Bravo, go to the “Analyze Model” and connect to the data model:

You will get information about the tables and columns in your model that are consuming the most memory:

In addition, you can drill down to see more detail for any table:

To analyze the unreferenced columns, I recommend my tool Tabular Editor, which can help you identify the reports connected to the dataset where these columns are used.

3. DAX FORMULAS / VISUALIZATION

Performance Analyzer: To Analyze the performance of your report elements, such as visuals and DAX formulas when users interact with them, and which consume the most resources.

In the View ribbon you can find the Performance Analyzer. By opening this pane and clicking Record, this will show you how fast your report renders when performing different functions in your report.

You can review the results of your performance test in the Performance analyzer pane. To review the tasks in order of duration, longest to shortest, right-click the Sort icon next to the Duration (ms) column header:

The log information for each visual shows the time it has taken (duration) to complete the following categories of tasks, which will help you detect what is causing performance delays so you can work on improving those sections.

• DAX query– Time taken by the to send the query and the time it took Analysis Services to return the results.
• Visual display– Rendering time on the screen, including the time required to retrieve web images or geocoding.
• Other– The time it took the visual to prepare queries, wait for other visuals to complete, or perform other background processing tasks.

The only real way to reduce the long duration is to optimize DAX queries for other visuals or reduce the number of visuals in the report.

4. DATA MODEL / DAX FORMULAS / VISUALIZATION

Best Practice Analyzer within Tabular Editor.

Best Practice Analyzer is an external tool that scans your data model and easily validates if your data model complies with all best practices at formatting level, performance, naming conventions and DAX expressions.

I analyzed a test data model getting the following information which shows the potential issues and improvements that can be made in the data model in each category:

This can help you save a lot of time by avoiding having to validate all these recommendations manually.

In this article 👉 Best practice rules to improve your model’s performance you can read the rules that analyze which are segmented into categories. Some rules are more aesthetic-oriented while others are more focused on performance optimization.

Benefits.

• Reduce space and processing time.
• Faster refresh times.
• Lower latency reports.
• Reduce the time development.
• More user-friendly data models because the user will not navigate through a bunch of useless objects that are not needed to build the reports.

– Choose the appropriate dataset mode for your solution: You can use Import, DirectQuery, and Composite model.

There are a variety of factors to consider when deciding which method to use, so I recommend that you review the advantages, disadvantages, and limitations of each in the Microsoft documentation so that you can select the correct one.

– Build your data model using a star schema architecture avoiding snowflake schema:

Benefits

• More user friendly.
• The DAX formulas are simpler.
• The time refreshes are faster.
• Reduces redundant data

Star schema are specialized data models used when designing data warehouses, the tables are classified as either dimension or fact. In this schema the fact sits in the middle and the dimensions are all around related directly to the fact which gives it the shape of a star.

– Only include the essential columns to create the reports: You do not want a report larger than necessary. However, removing the unnecessary columns could be too complicated when you have a lot of reports connected to the dataset because they can break.

I created a tool to help you to clean up your data models easily and quickly, you can download it for free, visit the link: Tabular model cleaner.

– Load in your data model the necessary tables: Probably you need to import a table in Power Query to make intermediate calculations but if you won’t use this table in your reports remember unchecking the “enable load” option in Power Query:

– Apply an Incremental Refresh policy to reduce the amount of data: Power BI desktop is limited for your computer resources, if you are working with very large models will be necessary configure the Incremental refresh.

Benefits

• Optimize your time development because you work over a smaller data model.
• Shorter data refresh time.
• Lower resource consumption in the Power BI Service.

Check our article: Configure an Incremental Refresh Policy in seven steps

– Group by and summarize combined with composite models: Depending on the level of granularity that you need to handle in your reports you can group your data by month, division, city for example.

In case you have to visualize a detail of the data, you can create a composite model and adding a drill through page that retrieves the detail data from a DirectQuery table.

For more detail check: Composite Models in Power BI Desktop

– Avoid many-to-many relationships: Most of the time, this can be avoided by creating a bridge table with unique values and one-to-many relationships from the bridge table to the fact tables, thus adopting star schema.

– Avoid both-directional relationships in your data model: You need to have a solid data modeling knowledge to measure the impact of this type of relationship in your data model, otherwise this could cause:

• Unexpected results
• Performance problems

Instead of creating a bi-directional relationship, try using a measure like the one below to get the value you need:

– Add a date table to your model and mark it as a date table.

– Disable auto date/time option: This avoids Power BI automatically create a lot of date dimension tables in the background.

– Disable the “Autodetect new relationships after data is loaded” option: Because you could mess up your data model if you do not have solid experience in data modeling.

– Remove columns with high cardinality: High cardinality columns can be particularly expensive. If possible:

• Remove columns such as GUID and timestamp from a SQL Server table from your model.
• Splitting Datetime columns in two columns, one for date and one for time.

– Select the correct data type: Don’t use decimal type if you only need the whole number or if your data granularity is at day level don’t use Date/Time data.

– Create measures instead of Calculated Columns: Using measures where applicable will help reduce data model size and increase the efficiency of the calculations.

 – Create custom columns in Power query instead of DAX calculated columns: The custom columns created in Power Query achieve a better and more efficient compression.

– Use tools as DAX Studio and Bravo you to analyze where your model consumes the most memory: and choose which columns and tables you should remove to optimize it.

– Use the tool Tabular model cleaner to analyze the impact in your reports of make changes or removing objects from your data models.

Click here for part 3 of this series.

– Convert Multiple Measures to Variables: This will minimize the number of measures on your data model and improve their performance.

– Use the DIVIDE function for division instead divide operator (/): This function is optimized to check division by zero which generates a significant performance gain.

– Avoid using the IFERROR function: It’s better to avoid using the ISERROR and IFERROR functions. Instead, apply defensive strategies when developing the model as:

• • Apply Power Query transformations to remove or substitute invalid or missing values, and to set correct data types.
  • Use the IF function to determine whether an error result would occur or error-tolerant functions.

– Reduce usage of calculated columns that use the RELATED function.

– Use SELECTEDVALUE() instead of HASONEVALUE().

– Use COUNTROWS instead of COUNT because it’s more efficient and performs better.

– Write Explicit measures: You will have more flexibility and will be able to reuse them within other DAX formulas.

Implicit measure: This is when you use a column with numeric values directly in a visualization, you will identify it because each numeric column has the Sigma (or sum) icon next to it.

Explicit measure: These are the ones created using a DAX formula, they show a calculator icon beside them.

I recommend that you use explicit measures even for simple measures such as sum, count, average, etc. and then hide the corresponding column to avoid confusing users by having both the column and the measure in the data model.

– Use DAX Formatter to format your code: The code will be easier to read and maintain.

– Use performance analyzer to identify the DAX formulas with longer rendering time.

– Use DAX Studio to troubleshoot the slowest formulas: This tool will help you to speed up the troubleshoot process as you can modify the DAX code there directly and when running it know in detail the performance impact. I recommend you check out this article that goes deeper in this topic.

Click here to read the last part of this series.

According to official documentation the following data sources support query folding:

– Relational databases (SQL Server, Oracle, etc.).

-OData sources (including SharePoint lists).

– Active Directory.

Data sources like flat files (Excel, CSV, txt, etc.), blobs, and web usually do not support it. Check here the list of transformations that prevent query folding.

You can check if the query folding occurs by right-clicking on the last applied steps and validating if View Native Query option is enabled, when the View Native Query option is grayed out, it means that this specific step or transformation is executed with local resources.

Query Folding importance:

– Tables in import mode will have a lower resource utilization and faster refresh time.

– Direct Query and Dual storage mode table only admit Power Query queries that can be folded.

– To be able to configure the incremental refresh in a table, it is essential that the query folding can be achieved.

Query folding good practices:

– Fully Folding: The preferred scenario is to achieve a query folding fully, that is the source executes all required transformations and returns to Power BI the desired result set.

– Partial Folding: Place steps that support query folding at the beginning and move the steps that prevent query folding later because these transformations will be loaded into the local cache and processed locally by Power BI engine.

The idea is to maximize the performance getting as much query folding as possible. However, the recommendation is to avoid this option whenever you can.

• Native Query: If you use a native query include all the logic of the required transformations there, because any other transformations applied to the results of this function will be processed with local resources.
• Apply the transformations in the data source: When you identify that your query requires a large number of steps that can’t be folded, apply the transformations in the data source using a database view or by physically preparing and materializing data.

I recommend you create a view because it can provide you with multiple advantages over regular tables. Check this interesting article about that.

Conclusion

When you are working with a relational data source (such as SQL Server) always check the query folding. If most of the steps do not support it, use another approach for the transformations.

Not applying these recommendations in your data model could exponentially increase resource usage and refresh time and will prevent you from using Direct query or Dual storage mode and apply incremental refresh policies in your data model.

At the end of this series of articles, you will be able to download the summary checklist in Excel format, I recommend you keep it handy to review it every time you build a Power BI solution from scratch or need to improve the performance of your existing developments.

Part I. Optimizing the Data source

This includes the source, connections, and power query transformations performed prior to loading a table to the data model.

– Ensuring query folding occurs whenever possible: Applying query folding in your power query tables could exponentially reduce resource utilization and refresh time.

When you are working with a relational data source (such as SQL Server) always check the query folding. If most of the steps do not support it, consider applying transformations on the data source using a database view or by physically preparing and materializing the data.

You must have the query folding enabled, to check it right-click on the last applied step in Power Query:

If the “View Native Query” option is grayed out, it means that this specific step or transformation is executed with local resources.

– Minimize the Power Query steps: Consolidate repeated steps, put all “remove columns” together, all “rename columns” together and so on. This change will improve the execution performance of the query.

– If your source is an Excel File, you could convert it to CSV before importing it into Power BI: The performance of reading data from a CSV is better.

To convert the Excel files, you could use the CloudConvert API to CSV files, and then save them to Azure Blob storage.

However, the recommendation for big projects is always to move the data to a database.

– Disable Power Query background refresh: By default, when Power Query imports data it retrieves and stores 1000 rows of preview data for each query.

If your Power BI Desktop file contains many queries, this option could extend the time it takes to complete a refresh, in these cases it is convenient to disable it.

– If safe, Ignore the Privacy levels or set both sources to Organizational: This setting can potentially improve performance and functionality, however, Power BI Desktop cannot ensure the privacy of data merged into the Power BI Desktop file.

Don´t deactivate this option in the following scenarios:

• If the data source contains sensitive data.
• Queries that accept query folding like SQL.
• If the dataset will be published.

– Use Query Diagnostics to analyze the query performance and identify the bottlenecks exist while loading and transforming your data and refreshing your data in Power Query.

Click here for part 2 of this series.

Or after cleaning up your data model, do you find that connected reports break so you manually go through each one to find the cause?

In this post I will introduce you to the first version of a tool I created to help you in the process of clean your data models, which can become a nightmare if you don’t have an effective and fast way to do it.

Challenge

When developers remove unnecessary objects from large data models, connected reports break a and the validation process is manual, tedious and time-consuming.

Solution:

When I tried to clean my data models I couldn’t find any tool with all the functionalities I needed, so decided to create my own using Power BI, which allows:

– Obtain all the dependencies of the data model in an efficient and automated way.

– Identify the list of objects (measures, columns, calculated columns, tables) in use and those that are not, in the reports connected to the Power BI or SSAS Data model.

I hope it can help you in the process of keeping your data models optimized 😊.

How It Works

This tool contains two parts:

1. Get the dependencies between all objects in the data model: This tool identifies how all the objects in your data model are connected to each other. For example, if you want to remove a measure, you will be able to detect the impact on your reports.

To get the dependencies I used Dynamic Management Views (DMV) which are queries that return information about model objects, server operations and server status. If you want to know more that DMVs, check our article What are Dynamic Management Views (DMVs) and what are they for?.

The main DMV that I used is the DISCOVER_CALC_DEPENDENCY, which Returns information about the calculation dependency for an object that is specified in a Tabular database or in a DAX query that is executed against a Tabular database.

2. Identify the objects used in the data model: For that I used the tool “Report Analyzer” explained in detail in this article: Report Analyzer

Finally, I related the information provided for the DMVs and I merged it with the information generated for the Report Analyzer in a Power BI report.

How to Use the Tool

You can use this tool in any Power BI data model with the following simple steps:

1. Get the connection details of a Power BI Dataset

Local Power BI Dataset:

Open a Power BI Dataset that you want to document. Open DAX Studio and choose the “PBI / SSDT Model” data source option, then click on Connect:

In the lower right-hand corner of the screen is the name of the server, and the port number you have connected to. In this case it is localhost:52223

Run the following DMV query in a DAX query window. This will give you the GUID, the name of the only database in the Power BI data model:

SELECT [CATALOG_NAME] FROM $SYSTEM.DBSCHEMA_CATALOGS

Power BI Premium

– Server: Go to “Workspace settings” and copy the Workspace Connection.

– Database: Is the dataset name.

2. Update the Dataset parameters in the Data model cleaner tool:

3. Get all the objects used in the Power BI reports connected to the data model:

– Save all the PBI report files connected to the golden dataset in a folder.

– Open External tool: Report Analyzer:

– Click on “Select folder”:

– Select the respective folder, and select “Export Report Metadata”:

– You will see the following message:

The files in .txt format will be saved in the same folder where the reports are.

4. Update the “LocalSoucePBITemplates” parameter in Power BI:

Type the location of the folder with the .txt files generated by the Report Analyzer:

Information provided by the Data Model Cleaner tool

– Object dependencies: On this tab you can consult the list of calculated objects affected by the selected object:

– Calculated Objects Breakdown: This tab shows the breakdown of the queried calculated object.

– Objects Used / Unused List: On this tab you can easily consult the Used / Unused list of the objects that will help you make decisions about hiding or removing them from the model.

If you consult by table additionally you will see the “M” or DAX expression of the table.

To determine the used list of objects this tool consider:

• Visuals and custom visuals
• Report-Level measures
• Filters added to the visuals, pages, and reports
• Bookmarks

You can also drill through on any object used and consult the reports, pages and in which of the visualizations this object is being used:

– Used Objects in the Reports: List of objects used by report.

Download Tabular Model Cleaner tool here 👇:

Tabular Model Cleaner tool.zip