1

When changing the pipelines for my company I often see the pipeline breaking under some specific condition that we did not anticipate. We use yaml files to describe the pipelines (Azure Devops)

We have multiple scenarios, such as:

  • Pipelines are run by automatic triggers, by other pipelines and manually

  • Pipelines share the same templates

  • There are IF conditions for some jobs/steps based on parameters (user input)

In the end, I keep thinking of testing all scenarios before merging changes, we could create scripts to do that. But it's unfeasible to actually RUN all scenarios because it would take forever, so I wonder how to test it without running it. Is it possible? Do you have any ideas?

Thanks!

I already tried the Preview endpoints from Azure REST api, which is good, but it only validates the input, such as variables and parameters. We also needed to make sure which steps are running and the variables being set in those

videap
  • 66
  • 5
  • Please provide enough code so others can better understand or reproduce the problem. – Community Jan 06 '23 at 03:43
  • I'm afraid that you have to run the pipeline to check all the actions defined in the yaml files. See [Is there a tool to validate an Azure DevOps Pipeline locally?](https://stackoverflow.com/questions/53041678/is-there-a-tool-to-validate-an-azure-devops-pipeline-locally) and [How do you debug Azure DevOps YAML file?](https://stackoverflow.com/questions/58550914/how-do-you-debug-azure-devops-yaml-file/58577592#58577592) for details. – Andy Li-MSFT Jan 06 '23 at 09:38
  • Thanks guys.. Well, I actually got the alerts, but as the pipeline builds and tests the code, it already fail a lot and it's expected behavior. The idea of running it locally is great! I'll give it a try – videap Jan 09 '23 at 20:32

1 Answers1

0

As far as I know (I am still new to our ADO solutions), we have to fully run/schedule the pipeline to see that it runs and then wait a day or so for the scheduler to complete the auto executions. At this point I do have some failing pipelines for a couple of days that I need o fix.

I do get emails when a pipeline fails like this in the json that holds the metadata to create a job: "settings": { "name": "pipelineName", "email_notifications": { "on_failure": [ "myEmail@email.com" ], "no_alert_for_skipped_runs": true },

Theres an equivalent extension that can be added in this link, but I have not done it this way and cannot verify if it works.

Azure Pipelines: Notification on Job failure

https://marketplace.visualstudio.com/items?itemName=rvo.SendEmailTask

I am not sure what actions your pipeline does but if there are jobs being scheduled there on external computes like Databricks, there should be a email alert system you can use to detect failures.

Other than that if you had multiple environments (dev, qa, prod) you could test in non production environment. Or if you have a dedicated storage location that is only for testing a pipeline, use that for the first few days and then reschedule the pipeline in the real location after testing it completes a few runs.

Raie
  • 61
  • 6