NotFound Status Code when working with ADFv2 via AzureRM.DataFactoryV2 PowerShell module

NotFound Status Code when working with ADFv2 via AzureRM.DataFactoryV2 PowerShell module

Azure Data Factory V2 is a great good tool for processing big data. It’s very scalable, flexible and modern cloud data integration service.
You can compose and manage reliable and secure data integration workflows at scale. Use native ADF data connectors to move and transform cloud and on-premises data that can be unstructured, semi-structured, and structured with Hadoop, Azure Data Lake, Spark, SQL Server, Cosmos DB and many other data platforms.
Version 2 of Azure Data Factory was released to general availability (GA) about three months ago. Comparing to previous ADF version – it’s completely new product, contains own UI improving developer’s experience, has many various connectors (including non-Microsoft systems) and capabilities to run Azure Databricks as one of the steps.
No wonder that completely refurbished product has a new module in PowerShell: AzureRM.DataFactoryV2.

Today, I would like to show you how to use one cmdlet to deploy DataFactory v2 using PowerShell script which can be used in order to automate the deployment process afterwards (next post will be about it).

Quite recently, I wanted to automate the deployment process using files being kept in a code repository.
This is important as the files are divided by type of object – as not as the same like ARM template files exported from Azure portal.
As I’m a big advocate of PowerShell for some time – it was obvious to use PowerShell in this case (not mentioning that it should be natural choice).
There are two separate modules: one for V1 and another one for V2.
Once checking documentation – I found that we must keep a specific order of deploying the pieces:

  • Create a data factory (that was obvious)
  • Create linked services
  • Create datasets
  • Create a pipeline

So, I have started preparing script to deploy ADFv2:

Then I have got the following error:

Increasingly the description of errors says you… nothing. I got used to.
After an hour of investigation – I have found the issue. One fault was me (the original one), but the second one was caused by module (lack of information).

Hint 1

Check the version of your installed module.

So, firstly I have checked the version of the module I had locally. The name of the module is AzureRm.DataFactoryV2, so run:

Then, let’s check the latest version fron the Internet:

Seems like need to be updated (you must run a PowerShell session in administrator mode):

OK, let’s try run the creation of ADF again:

I’ve received the error again, BUT with different, more valuable cause:

As you can see – the problem was trivial – I wanted to use non-existing Resource Group.
This can happen when you make just a typo or switiching between many subscriptions and used resource group that belongs to other subscription.

Hint 2

Check your contexts and what RG are available:

Switch to appropriate subscription or change the [ResourceGroupName] parameter and run it again:

If that’s doesn’t work for you – restart your PowerShell session. It turning out that even though your module seems updated – still might not work correctly. For example, reloading the module to previous version I had (0.5.3) is not possible:

Simply, restart the PowerShell session and the latest version of the module should works correctly.

Conclusion

Once you notice that error:

Do:
1) update AzureRm.DataFactoryV2 module first.
2) restart PowerShell session (if needed)
3) check the correctness of passing Resource Group Name

Please bear in mind that the module is still developing (0.something version) and thus might contain some errors.
I hope that helps and save you certain minutes/hours.

Cheers,
Kamil

Previous Last week reading (2018-09-16)
Next ASF 014 video invitation

About author

Kamil Nowinski
Kamil Nowinski 84 posts

Blogger, speaker. Data Platform MVP, MCSE. Senior Data Engineer & data geek. Member of Data Community Poland, co-organizer of SQLDay, Happy husband & father.

View all posts by this author →

You might also like

General 0 Comments

Last week reading (2018-02-25)

It’s was very busy and pleasant week for me. Tenth SQLBits which has taken in London this year finished yesterday. I have been working as a volunteer (one of the

General 0 Comments

BI Conference in Wroclaw

I would like to spread the word about a new conference which appears in my favourite city in Poland – Wroclaw. I’m talking to Jacek Biały, Business Intelligence Competency Center Manager

Community 0 Comments

BI Connected by SII – conference in Wrocław

Good day SQL folks! The conference season has begun. You can find very interesting conferences every week in almost every place in the globe. Of course you do not need

0 Comments

No Comments Yet!

You can be first to comment this post!

Leave a Reply

32 − = 29

Protected with IP Blacklist CloudIP Blacklist Cloud