Discovering diagram of dependencies in Synapse Analytics and ADF pipelines

Discovering diagram of dependencies in Synapse Analytics and ADF pipelines

Documenting objects dependencies of ETL processes is a tough task. Regardless it is SSIS, ADF, pipelines in Azure Synapse or other systems. The reasons for understanding the current solution can vary either: handover to other team/member of the team, troubleshooting, refactoring, debugging, investigating dependencies due to error, performance issue or others, as well as keen to remove selected/duplicated pipelines or logic.

But there is never a good time to make documentation, and even if that has been done – no one knows how much it’s up to date. The situation is not improved by the fact that quite often there is a lack of (free or built-in) tools for generating such documentation. Sounds familiar? I bet it does.

Wiki Markdown and Mermaid diagrams

If you ever work with Markdown, you know that it is more and more popular format to develop and share documentation, often as a part of the code repository. Most popular tools support the format, including Azure DevOps and GitHub. Markdown supports diagrams in a few different forms, hence one can enhance documentation with a handful sequence diagram, Gantt charts or flowcharts, simply by writing a few lines of plain text.

Generate diagram with PowerShell

It has been some time now since I released the first version of #adftools. The main reason behind the tool is to deploy Azure Data Factory directly from code. But as the name suggests – it is a toolkit. With the tool, you can very easily generate mermaid code and copy/paste it to your markdown file. What do we need to do?

  1. Install/import PowerShell module (if you haven’t done that yet)
    1. azure.datafactory.tools – for Azure Data Factory
    2. azure.synapse.tools – for Azure Synapse pipelines
  2. Download or clone the code onto a local machine
  3. Run one line of code in PowerShell
  4. Copy the result and paste it into the target .md (markdown) file

Let’s do all these steps together.

Install/import PowerShell module

Depending on which code of which service you want to discover – install and import one of the following PS modules:

Download or clone the code

If you don’t have Git Integration set up for the pipelines – do it. I was explaining how to do that in this post, and once you achieve that – you can clone the code into your local machine. The folder should contain the structure of subfolders similar to this one:

Run one line of code in PowerShell

It’s time to run “the magic”. Execute the following code (either in Visual Studio Code or PowerShell ISE / session):


The first line is to set RootFolder variable with location of code. The second line (import) is going to read all files in subfolders. You’ll see similar result in output terminal:

The last line is to generate mermaid code for you:

Text representation of diagram

Copy the result and paste it into the target .md (markdown) file

Now, simply copy & paste the result into the existing or newly created MarkDown file (.md extension) and you should be able to see the diagram, whenever in Visual Studio Code or using Azure DevOps:

Preview ‘diagram.md’ file in Visual Studio Code

Alternatively, you can pass the output of the Get-SynapseDocDiagram command directly to a new file:

Summary

I hope the command will be useful for you. As always, should you have any comments or suggestions for improvements, let me know: Raise an issue.

Thanks for reading.

Previous Database projects with SQL Server Data Tools (SSDT)
Next Cloud Formations - A New MVP Led Training Initiative

About author

Kamil Nowinski
Kamil Nowinski 189 posts

Blogger, speaker. Data Platform MVP, MCSE. Senior Data Engineer & data geek. Member of Data Community Poland, co-organizer of SQLDay, Happy husband & father.

View all posts by this author →

You might also like

New blog, first post, big challenge!

Yeeaaahh! First post. First blog. I was waiting for it for at least 2 years. Yup, undoubtedly a “waiting” is the best definition for that. Waiting, because I was doing

General 1Comments

Skirmishes with automated deployment of Azure Data Factory

When it comes to Azure Data Factory deployment, I always have a few words to say. You might be aware that there are two approaches in this terms. I was

General 0 Comments

Cloud Formations – A New MVP Led Training Initiative

Hello, data friends! Let me spread a few words about the new MVP led training initiative started by my friend and fellow MVP – Paul Andrew. It’s called Cloud Formations

2 Comments

  1. AlbertoZGZ
    January 25, 16:04 Reply

    Hello,

    Im not sure if is working on Azure DevOps

    The code generated is:
    ::: mermaid
    graph LR
    pipeline.pipeline1 –> dataset.DelimitedText1
    dataset.DelimitedText1 –> linkedService.AzureBlobStorage1
    IntegrationRuntime.AutoResolveIntegrationRuntime –> managedVirtualNetwork.default
    :::

    The error generated is
    Parse error on line 4:
    …nagedVirtualNetwork.default
    ———————–^
    Expecting ‘SEMI’, ‘NEWLINE’, ‘SPACE’, ‘EOF’, ‘SQS’, ‘AMP’, ‘STYLE_SEPARATOR’, ‘PS’, ‘(-‘, ‘STADIUMSTART’, ‘SUBROUTINESTART’, ‘CYLINDERSTART’, ‘DIAMOND_START’, ‘TAGEND’, ‘TRAPSTART’, ‘INVTRAPSTART’, ‘START_LINK’, ‘LINK’, ‘DOWN’, ‘NUM’, ‘COMMA’, ‘ALPHA’, ‘COLON’, ‘MINUS’, ‘BRKT’, ‘DOT’, ‘PUNCTUATION’, ‘UNICODE_TEXT’, ‘PLUS’, ‘EQUALS’, ‘MULT’, ‘UNDERSCORE’, got ‘DEFAULT’

Leave a Reply