2

I am setting up an ADF pipeline to copy blob into an Azure SQL DB. I have a Iteration activity in my pipeline, where I have set up a counter to loop and copy only if the blob exists.

This works great except for some random PK violations, which I will have to check manually. So I edited my pipeline to log the error, and continue. So I set up the pipeline as such. re If the copy activity fails due to Primary Key Violation, (for now) ignore, but log the details using a stored procedure and continue as usual i.e. update the loop counter to get the next folder.

Unfortunately, the success of Log Failure does not execute the "Set Variable" activity. So it goes back in an infinite loop keep coming back with the same exception, but the Stored Procedure activity itself is logging the error message correctly.

If I create a new "Set Variable" and do exactly what the SetLoopVariable does, it seems to be okay. but that means I have to copy every activity after that to have two separate paths. Which I feel is redundant.

BACKGROUND: My file structure is container/YYYY/MM/dd/HH/mm, there will be at least one file per hour, but not for every minute of the day so I to do a check if the folder exists before attempting to copy.

PEF
  • 125
  • 2
  • 11
  • I issue the same scenario where I needed to add a separate activity. So, add one set variable for copy data on success and one for log failure on success. Hope someone from ADF team will explain this behaviour. – DraganB Jun 11 '19 at 06:29
  • 1
    Shouldn't arrow coming from Copy Data to Set Variable be 'On Completion' and not 'On Success'? – Alexey A. Sep 17 '19 at 13:58

2 Answers2

1

This is by design. SetVariable will only be called if Copy Data succeeds and fails, since Data Factory V2 Activity Dependencies are a Logical AND.

Thomas
  • 31
  • 5
1

Thomas answer is correct. I had this exact issue recently. In case it helps someone else, I realised it means the arrows don’t represent a flow but a dependency. The box only runs if all the preceding tasks are done, which is impossible in your case because it depends on the copy both succeeding and failing.

To solve your case just duplicate the ‘set loop variable’ in your error handling path.

However you might then have the same problem that I now have here Azure data factory: Handling inner failure in until/for activity

Jason Welch
  • 896
  • 5
  • 9