NotFound Status Code when working with ADFv2 via AzureRM.DataFactoryV2 PowerShell module

NotFound Status Code when working with ADFv2 via AzureRM.DataFactoryV2 PowerShell module

Azure Data Factory V2 is a great good tool for processing big data. It’s very scalable, flexible and modern cloud data integration service.
You can compose and manage reliable and secure data integration workflows at scale. Use native ADF data connectors to move and transform cloud and on-premises data that can be unstructured, semi-structured, and structured with Hadoop, Azure Data Lake, Spark, SQL Server, Cosmos DB and many other data platforms.
Version 2 of Azure Data Factory was released to general availability (GA) about three months ago. Comparing to previous ADF version – it’s completely new product, contains own UI improving developer’s experience, has many various connectors (including non-Microsoft systems) and capabilities to run Azure Databricks as one of the steps.
No wonder that completely refurbished product has a new module in PowerShell: AzureRM.DataFactoryV2.

Today, I would like to show you how to use one cmdlet to deploy DataFactory v2 using PowerShell script which can be used in order to automate the deployment process afterwards (next post will be about it).

Quite recently, I wanted to automate the deployment process using files being kept in a code repository.
This is important as the files are divided by type of object – as not as the same like ARM template files exported from Azure portal.
As I’m a big advocate of PowerShell for some time – it was obvious to use PowerShell in this case (not mentioning that it should be natural choice).
There are two separate modules: one for V1 and another one for V2.
Once checking documentation – I found that we must keep a specific order of deploying the pieces:

  • Create a data factory (that was obvious)
  • Create linked services
  • Create datasets
  • Create a pipeline

So, I have started preparing script to deploy ADFv2:

Then I have got the following error:

Increasingly the description of errors says you… nothing. I got used to.
After an hour of investigation – I have found the issue. One fault was me (the original one), but the second one was caused by module (lack of information).

Hint 1

Check the version of your installed module.

So, firstly I have checked the version of the module I had locally. The name of the module is AzureRm.DataFactoryV2, so run:

Then, let’s check the latest version fron the Internet:

Seems like need to be updated (you must run a PowerShell session in administrator mode):

OK, let’s try run the creation of ADF again:

I’ve received the error again, BUT with different, more valuable cause:

As you can see – the problem was trivial – I wanted to use non-existing Resource Group.
This can happen when you make just a typo or switiching between many subscriptions and used resource group that belongs to other subscription.

Hint 2

Check your contexts and what RG are available:

Switch to appropriate subscription or change the [ResourceGroupName] parameter and run it again:

If that’s doesn’t work for you – restart your PowerShell session. It turning out that even though your module seems updated – still might not work correctly. For example, reloading the module to previous version I had (0.5.3) is not possible:

Simply, restart the PowerShell session and the latest version of the module should works correctly.

Conclusion

Once you notice that error:

Do:
1) update AzureRm.DataFactoryV2 module first.
2) restart PowerShell session (if needed)
3) check the correctness of passing Resource Group Name

Please bear in mind that the module is still developing (0.something version) and thus might contain some errors.
I hope that helps and save you certain minutes/hours.

Cheers,
Kamil

Previous Last week reading (2018-09-16)
Next ASF 014 video invitation

About author

Kamil Nowinski
Kamil Nowinski 69 posts

Data Architect, MCSE Data Platform, MS BI Developer, Member of PLSSUG, co-organizer of SQLDay, speaker. Happy husband & father. #SQLFamily member.

View all posts by this author →

You might also like

General 1Comments

SQLSaturday #656 Denmark – we were there!

It was a great weekend in Copenhagen. Kamil and I had the opportunity to give lectures during #656 SQL Saturday. I started in the morning talking about the U-SQL and

Events 0 Comments

My recap of PASS Summit 2017

When you doing something first time, predominantly jitters happens, right? Fine, it was not my first conference, I wasn’t in a speaker role this time, etc. However, certain voice in

General 0 Comments

Speaking at SQLSaturday #656 Denmark 2017

On Saturday, October 7th, I’m speaking at SQLSaturday #656 Denmark 2017! I’m presenting Maintenance of a DB project and Continuous Delivery using SSDT. I will unveil disadvantages of SSDT and

0 Comments

No Comments Yet!

You can be first to comment this post!

Leave a Reply

1 + 5 =

Protected with IP Blacklist CloudIP Blacklist Cloud