In Power BI development in Microsoft Fabric, understanding and utilising source control mechanisms is crucial for efficient collaboration and version management. This blog post delves into the essential aspects of source control for Power BI. This blog also includes the recording of my session at Saudi Arabia’s Excel User Group on the 26th of August 2023. The event was organised by Microsoft MVP, Faraz Sheik, where we walked through all the topics discussed in this blog.
Understanding Source Control
At its core, source control is a system that records changes to a file or set of files over time. This lets developers recall specific versions later, ensuring efficient collaboration and error management. It’s particularly vital for development teams, allowing multiple contributors to work on the same codebase without overwriting each other’s work.
For Power BI developers, this means tracking changes made to reports, and data models that are the most crucial components of every Power BI project.
This is the second part of the series of blog posts showing how to integrate Power BI with Azure DevOps, a cloud platform for software development. The previous post gave a brief history of source control systems, which help developers manage code changes. It also explained what Git is, a fast and flexible distributed source control system, and why it is useful. It introduced the initial configurations required in Azure DevOps and explained how to integrate Power BI (Fabric) Service with Azure DevOps.
This blog post explains how to synchronise an Azure DevOps repository with your local machine to integrate your Power BI Projects with Azure DevOps. Before we start, we need to know what a Power BI Project is and how we can create it.
What is Power BI Project (Developer Mode)
Power BI Project (*.PBIP) is a new file format for Power BI Desktop that was announced in May 2023 and made available for public preview in June 2023. It allows us to save our work as a project, which consists of a folder structure containing individual text files that define the report and dataset artefacts. This enables us to use source control systems, such as Git, to track changes, compare revisions, resolve conflicts, and review changes. It also enables us to use text editors, such as Visual Studio Code, to edit the artefact definitions more productively and programmatically. Additionally, it supports CI/CD (continuous integration and continuous delivery), where we submit changes to a series of quality gates before applying them to the production system.
PBIP files differ from the regular Power BI Desktop files (PBIX), which store the report and dataset artefacts as a single binary file. This made integrating with source control systems, text editors, and CI/CD systems difficult. PBIP aims to overcome these limitations and provide a more developer-friendly experience for Power BI Desktop users.
Since this feature is still in public preview when writing this blog post, we have to enable it from the Power BI Desktop Options and Settings.
Enable Power BI Project (Developer Mode) (Currently in Preview)
As mentioned, we first need to enable the Power BI Project (Developer Mode) feature, introduced for public preview in the June 2023 release of Power BI Desktop. Power BI Project files allow us to save our Power BI files as *.PBIP files deconstruct the legacy Power BI report files (*.PBIX) into well-organised folders and files. With this feature, we can:
Edit individual components of our Power BI file, such as data sources, queries, data model, visuals, etc.
Use any text editor or IDE to edit our Power BI file
Compare and merge changes
Collaborate with other developers on the same Power BI file
To enable Power BI Project (Developer Mode), follow these steps in Power BI Desktop:
Power BI is a powerful tool for creating and sharing interactive data visualizations. But how can you collaborate with other developers on your Power BI projects and ensure quality and consistency across your reports? In this series of blog posts, I will show you how to integrate Power BI with Azure DevOps, a cloud-based software development and delivery platform. We can integrate Azure DevOps with Power BI Service (Fabric) as well as Power BI Desktop. The current post explains how to set up Azure DevOps and connect a Power BI Workspace. The next blog post will explain how to use it on your local machine to integrate your Power BI Desktop projects with Azure DevOps.
A brief history of source control systems
Before we dive into the details of Power BI and Azure DevOps integration, let’s take a moment to understand what source control systems are and why they are essential for any software project.
Source control systems, also known as version control systems or revision control systems, are tools that help developers manage the changes made to their code over time. They allow developers to track, compare, and roll back changes when necessary and collaborate with other developers on the same project.
There are two main types of source control systems: centralised and distributed. Centralised source control systems use Client-server approach to store all the code and its history in a single server, and developers need to connect to that server to access or modify the code. Examples of centralised source control systems are Microsoft’s Team Foundation Server (TFS) which rebranded to Azure DevOps Server in 2018, IBM’s ClearCase, and Apache’s Subversion.
On the other hand, distributed source control systems use a peer-to-peer approach, allowing each developer to have a local copy of the entire code repository, including its history. Developers can work offline and sync their changes with other developers through a remote server. Examples of distributed source control systems are Git Software and Mercurial, which takes us to the next section. Let’s see what Git is.
What is Git, and why use it?
Git is one of the world’s most popular and widely used distributed source control systems. It was created by Linus Torvalds, the creator of Linux, in 2005. Git has many advantages over centralised source control systems, such as:
Speed: Git is fast and efficient, performing most operations locally without network access.
Scalability: Git can easily handle large and complex projects, as it does not depend on a single server.
Flexibility: Git supports various workflows and branching strategies, allowing developers to choose how they want to organise their code and collaborate with others.
Security: Git uses cryptographic hashes to ensure the integrity and authenticity of the code.
Open-source: Git is free and open-source, meaning anyone can use it, modify it, or contribute to it.
While Git is pretty good, it has some disadvantages compared with a centralised source control system. Here are some:
Complexity: Git has a steep learning curve, especially for users who are new to distributed version control systems. Understanding concepts such as branching, merging, rebasing, and resolving conflicts can be challenging for beginners and sometimes even seasoned Git users.
Collaboration challenges: While distributed version control systems like Git enable easy collaboration, they can also lead to collaboration issues. Multiple developers working on the same branch simultaneously may encounter conflicts that need to be resolved, which can introduce complexities and require extra effort.
Performance with large repositories: While Git performs pretty well on most operations, it can get abortive when working with large repositories containing many files or a long history of commits. Operations such as cloning or checking out large repositories can be time-consuming.
What is Azure DevOps, and what does it relate to Git?
Azure Boards: A tool for planning, tracking, and managing work items, such as user stories, tasks, bugs, etc.
Azure Repos: A tool for hosting Git repositories online, which is the main focus of this blog post.
Azure Pipelines: A tool for automating builds, tests, and deployments.
Azure Test Plans: A tool for creating and running manual and automated tests.
Azure Artifacts: A tool for managing packages and dependencies.
Azure DevOps also integrates with other tools and platforms, such as GitHub, Visual Studio Code, and now, Power BI. This takes us to the next section of this blog post, Integrating Power BI with Azure DevOps.
How to integrate Power BI with Azure DevOps
Now that we understand what Git and Azure DevOps are let’s see how we can integrate Power BI with Azure DevOps.
Integrating Power BI with Azure DevOps has two different integrations. Cloud integration and local machine integration have the following requirements.
Prerequisites
To follow along with this tutorial, you will need:
In the cloud:
An Azure DevOps Service
A Power BI account with one of the following licenses to enable Power BI Workspace integration with Azure DevOps.:
Power BI PPU (Premium Per User)
Premium Capacity
Embedded Capacity (EM/A)
Fabric Capacity
On your local machine:
The latest version of Power BI Desktop (June 2023 or later)
Either Visual Studio or VS Code
As stated earlier, this post explains the Cloud integration partTherefore, we require to have an Azure DevOps Service and a Power BI account with a Premium licencing plan in order to integrate Power BI with Azure DevOps.
In the following few sections, we look into more details and go through them together step-by-step.
One of the most important aspects of the software development life cycle is to have control over different versions of a solution, especially in a project where there is more than one developer involved in the implementation. Just like when you normally create a project in visual studio and you commit the changes back to a source control system like GitHub or Azure DevOps, it’s advised to keep the history of different versions of your Power BI reports. What we expect from a source control solution is to keep tracking of all changes happening in the source code while developing a project. So you can easily roll back to a previous state if you like to.
The other benefit of having a source control process in place is when multiple developers are working on a single project. Every single one of them makes changes in the source code then they commit all the changes into the source control server without overwriting each others’ work.
With Power BI things are a bit different though. Power BI report files are PBIX files which are stored in binary format (well, PBIX is basically a zip file isn’t it?) which at the time of writing this post, there is no official way to enforce Power BI source control in any source control solutions like GitHub or Azure DevOps (YET).
Microsoft announced a fantastic feature last week (6/05/2020) named “Deployment Pipelines” which does exactly what we’re after, but it is currently a preview feature which is only available only to organisations with Power BI Premium. So it is out of the game for the majority of us.
Having said that, there is still a way to keep history of changes in the shape of different versions of PBIX files. This is called Version Control.
There are several ways you can enable version control over your PBIX files while developing the report. Regardless of the version control platform you need to think about having multiple environments and who can access them for doing what.
Environment
Accessible to
Description
Development
Developers
Data modellers and report writers access this environment for development purposes.
User Acceptance Test (UAT)
Developers, SMEs, Technical Leads, Power BI Admins
After the development is finished the developers deploy the solution to the UAT environment. The solution will then be tested by SMEs (Subject Matter Experts) to make sure the business requirements are met.
Pre-prod (Optional but recommended)
Technical Leads, Power BI Admins
After the solution passed all UAT testing scenarios Technical Leads or Power BI Admins will deploy it to Pre-prod for final checks to make sure all data sources are correctly pointing to production data sources and all reports and dashboards are working as expected.
Production
Technical Leads, Power BI Admins, End Users
After pre-prod checks completed Technical Leads or Power BI Admins deploy the solution to the Production environment which is then available to the end users.
Version Control Options
If your organisation does not have a Premium capacity then “Deployment Pipelines” feature is not available to you. So you need to come up with a solution though. In this section I name some Version Control options available to you
