We know that the test execution and update of test results in the test management tool are important activities in the test execution life cycle. Updates to the test results of the regression suite will take more time if we are following the manual test case execution strategy. Better yet, we should have a stable automation setup for the regression suite execution, which will help reduce the manual test case execution effort. The stable automation setup is not enough; the automation setup has the capability to update the test results in test management tools dynamically during the runtime. In this article, I would like to share the capability of the test result update in Azure DevOps for each test case execution based on the configuration, and also how it will track the bug in Azure DevOps during runtime when a test case fails.
Pre-requisites:
- Must have an Azure DevOps account with Test Plan feature access.
- Test Plan section should contain the test suites and test cases.
- Create a Maven project.
- Add the following dependencies in the pom.xml file:
- testng
- rest-assured
- Write a sample test case in a test class. The test case should be mapped with the test case ID and test suite name.
- Create an AzureDevOpsActions class to keep all the methods and logic needed to update the test results in Azure DevOps.
- Create AutomationReport class that implements ITestListener interface. This will helps to get the test case ID, call the update test results in DevOps based on the test case status (Pass/Fail/Skip).
To achieve the test result update in DevOps, we can use the Azure DevOps Services REST APIs. We need the following API services and the APIs are executing in sequential order to update test results and create the bug in DevOps.
- Get Test Plan ID
- Get Test Suite ID
- Get Test Point ID
- Create Test Run and Get Run ID
- Get Test Results ID
- Create Bug and Get Bug ID
- Update Test Results
Before hitting the APIs, we have to set preconditions. The PAT (Personal Access Token) is essential for the authorization. The PAT will get from the User Setting in Azure DevOps. We can encode the PAT at the constructor of the AzureDevOpsActions class to generate the authorization token and it required while hitting each APIs. Following is the code to encode the PAT:
public AzureDevOpsActions() {
auth = Base64.getEncoder().encodeToString(("" + ":" + personalAccessToken).getBytes());
}
In addition, we have to set the value for the following variables:
- apiVersion as 5.0
- baseURI as https://dev.azure.com/
- organizationName is the name of the organization that signed up with Azure DevOps.
- projectName is the name of your project.
- planName is the name of your test plan.
- suiteName is the name of your test suite.
- testRunName is the name of test run.
Once the variable’s values are ready, we can start with each APIs logic.
Get Test Plan ID
The Test Plan ID is the root and it is essential for subsequent API calls. To get the Test Plan ID, we can use the variable the planName. Following is the private method and logic to get the Test Plan ID:
private String getTestPlanID() {
String testPlanID = null;
try {
String endPointPath = organizationName + "/" + projectName + "/_apis/test/plans?api-version=" + apiVersion
+ "";
Response response = getRequestWithAuthToken(baseURI, endPointPath, "Basic", auth);
System.out.println(response.getStatusCode());
System.out.println(response.asPrettyString());
String count = getValueFromResponse(response, "count");
for (int i = 0; i < Integer.parseInt(count); i++) {
String id = getValueFromResponse(response, "value[" + i + "].id");
String name = getValueFromResponse(response, "value[" + i + "].name");
if (planName.equals(name)) {
testPlanID = id;
}
}
} catch (Exception e) {
e.printStackTrace();
}
return testPlanID;
}
Get Test Suite ID
The Test Suite ID is required to get the Test Point ID. We will get the Test Suite ID with help of Test Plan ID and the test suite name. Following is the private method and logic to get the Test Suite ID:
private String getTestSuiteID(String testPlanID, String testSuiteName) {
String testSuiteID = null;
try {
String endPointPath = organizationName + "/" + projectName + "/_apis/test/plans/" + testPlanID
+ "/suites?api-version=" + apiVersion + "";
Response response = _apiActions.getRequestWithAuthToken(baseURI, endPointPath, "Basic", auth);
System.out.println(response.getStatusCode());
System.out.println(response.asPrettyString());
String count = _apiUtilities.getValueFromResponse(response, "count");
for (int i = 0; i < Integer.parseInt(count); i++) {
String id = _apiUtilities.getValueFromResponse(response, "value[" + i + "].id");
String name = _apiUtilities.getValueFromResponse(response, "value[" + i + "].name");
if (testSuiteName.equals(name)) {
testSuiteID = id;
}
}
System.out.println(testSuiteID);
} catch (Exception e) {
e.printStackTrace();
}
return testSuiteID;
}
Get Test Point ID
The Test Point ID is required to create payload data for test run creation and get run ID. We will get the Test Point ID with help of test case ID, Test Plan ID, and the test suite name. Following is the private method and logic to get the Test Point ID:
private String getTestPointID(String testCaseID, String testPlanID, String testSuiteName) {
String testPointID = null;
try {
String endPointPath = organizationName + "/" + projectName + "/_apis/test/plans/" + testPlanID + "/suites/"
+ getTestSuiteID(testPlanID, testSuiteName) + "/points?testCaseId=" + testCaseID + "&api-version="
+ apiVersion + "";
Response response = _apiActions.getRequestWithAuthToken(baseURI, endPointPath, "Basic", auth);
System.out.println(response.getStatusCode());
System.out.println(response.asPrettyString());
testPointID = _apiUtilities.getValueFromResponse(response, "value[0].id");
} catch (Exception e) {
e.printStackTrace();
}
return testPointID;
}
Create Test Run and Get Run ID
Create test run and get run ID is important step in the test result update. The Test Run ID is required to get the Test Results ID and update test results in Azure DevOps. We will get the Test Run ID with help of test suite name and test case ID. Following is the private method and logic to create test run with given test run name and get the Test Run ID:
private String createTestRunAndGetRunID(String testSuiteName, String testCaseID) {
String testRunID = null;
Response response;
try {
String testPlanID = getTestPlanID();
String endPointPath = organizationName + "/" + projectName + "/" + "_apis/test/runs?api-version="
+ apiVersion + "";
testRunName = getAzureConfigurationData(AutomationConstants.AZURE_TEST_RUN_NAME);
DateFormat dateFormatt = new SimpleDateFormat("dd_MM_yyyy");
DateFormat timeFormat = new SimpleDateFormat("HH-mm-ss");
Date date = new Date();
String currdate = dateFormatt.format(date);
String currtime = timeFormat.format(date);
String finaldatetime = currdate + "_" + currtime;
String updatedTestRunName = testRunName + "_" + finaldatetime;
String payloadData = "{\"name\":\"" + updatedTestRunName + "\",\"plan\":{\"id\":" + testPlanID
+ "},\"pointIds\":[" + getTestPointID(testCaseID, testPlanID, testSuiteName) + "]}";
response = _apiActions.postRequestWithBody(baseURI, endPointPath, payloadData, "Basic", auth,
"application/json");
System.out.println(response.getStatusCode());
System.out.println(response.asPrettyString());
testRunID = _apiUtilities.getValueFromResponse(response, "id");
} catch (Exception e) {
e.printStackTrace();
}
return testRunID;
}
Get Test Results ID
The Test Results ID is required to create payload data for test result update in Azure DevOps. We will get the Test Results ID with help of the endpoint that having the Test Run ID. Following is the private method and logic to get the Test Results ID:
private String getTestResultsID(String endPointPath) {
String testResultID = null;
try {
Response response = getRequestWithAuthToken(baseURI, endPointPath, "Basic", auth);
System.out.println(response.getStatusCode());
System.out.println(response.asPrettyString());
testResultID = getValueFromResponse(response, "value[0].id");
} catch (Exception e) {
e.printStackTrace();
}
return testResultID;
}
Create Bug and Get Bug ID
Create bug and get bug ID is an important step in the test result update. Whenever a test case failed, the ITestListener interface will catch the failure details in onTestFailure method and the failure details will send to AzureDevOpsActions class via updateTestResultsInTestRun method. In the fail condition, the bug details are pass to createBugAndGetBugID private method to create bug and get bug ID. Following is the private method and logic to create bug in Azure DevOps and get Bug ID:
private String createBugAndGetBugID(String bugDetails) {
String bugID = null;
try {
String endPointPath = organizationName + "/" + projectName + "/" + "_apis/wit/workitems/$bug?api-version="
+ apiVersion + "";
String payloadData = "[{\"op\": \"add\",\"path\": \"/fields/System.Title\",\"from\":null, \"value\":\""
+ bugDetails + "\"}]";
Response response = postRequestWithBody(baseURI, endPointPath, payloadData, "Basic", auth,
"application/json-patch+json");
System.out.println(response.getStatusCode());
System.out.println(response.asPrettyString());
bugID = getValueFromResponse(response, "id");
} catch (Exception e) {
e.printStackTrace();
}
return bugID;
}
Update Test Results
Update Test Results is an important step that connect the AutomationReport and AzureDevOpsActions classes. updateTestResultsInTestRun is a public method and it is called from onTestSuccess and onTestFailure overriding methods of ITestListener interface. The arguments are test suite name, test case ID, test case status and bug details. The test suite name and test case ID will be getting from onTestStart overriding method of ITestListener. The test case status will be getting from onTestSuccess and onTestFailure overriding methods of ITestListener. The bug details will be getting from onTestFailure overriding method of ITestListener. The endpoint created using the Test Run ID. The payload created using Test Results ID and Bug ID. Following is the public method and logic to update test results in Azure DevOps:
public String updateTestResultsInTestRun(String testSuiteName, String testCaseID, String testCaseStatus, String bugDetails) {
String testPointID = null;
String payloadData = null;
String bugID = null;
Response response;
String endPointPath;
try {
String testRunID = createTestRunAndGetRunID(testSuiteName, testCaseID);
endPointPath = organizationName + "/" + projectName + "/" + "_apis/test/runs/" + testRunID
+ "/results?api-version=6.0-preview.6";
if (testCaseStatus.toLowerCase().equals("pass")) {
payloadData = "[{ \"id\": " + getTestResultsID(endPointPath)
+ " , \"outcome\": \"PASSED\" ,\"state\": \"Completed\",\"comment\": \"Execution Successful\" }]";
}
if (testCaseStatus.toLowerCase().equals("fail")) {
// Create bug in DevOps
String needToTrackBugInAzureDevOps = getAzureConfigurationData(
AutomationConstants.AZURE_NEED_BUG_TRACK);
if (needToTrackBugInAzureDevOps.equalsIgnoreCase("yes")) {
bugID = createBugAndGetBugID(bugDetails);
}
payloadData = "[{ \"id\": " + getTestResultsID(endPointPath)
+ " , \"outcome\": \"FAILED\" ,\"state\": \"Completed\",\"comment\": \"Execution Failed\", \"associatedBugs\": [{\"id\":"
+ bugID + "}]}]";
}
response = _apiActions.patchRequestWithBody(baseURI, endPointPath, payloadData, "Basic", auth,
"application/json");
System.out.println(response.getStatusCode());
System.out.println(response.asPrettyString());
testPointID = _apiUtilities.getValueFromResponse(response, "value[0].id");
} catch (Exception e) {
e.printStackTrace();
}
return testPointID;
}
Additional Steps
Following are the additional steps:
- Mapping of test case ID and test suite name in Test Suite:
In the test suite class we can map the test case ID and test suite name using @TestCaseConfig annotation with the attribute testCaseID and testSuiteName as below:
@TestCaseConfig(testCaseID = 12345, testSuiteName = "LoginTest")
@Test()
public void TC12345_testUpdate() {
Assert.assertTrue(true);
}
Below is the implementation logic for TestCaseConfig interface:
@Retention(RetentionPolicy.RUNTIME)
public @interface TestCaseConfig {
int testCaseID();
String testSuiteName() default "";
}
- Get the test case ID and test suite name in onTestStart:
Once we map the test case ID and test suite name, next we need to get the test case ID and test suite name on test start the execution. Below is the logic to get the test case ID and test suite name that needs to be written in onTestStart overriding method:
TestCaseConfig testCasePolicy = result.getMethod().getConstructorOrMethod().getMethod()
.getAnnotation(TestCaseConfig.class);
String testCaseIDValue = testCasePolicy.testCaseID();
String testSuiteName = testCasePolicy.testSuiteName();
- Calling updateTestResultsInTestRun from onTestSuccess and onTestFailure.
updateTestResultsInTestRun method helps to connect the AutomationReport and AzureDevOpsActions classes. Following is the logic inside onTestSuccess:
new AzureDevOpsActions().updateTestResultsInTestRun(testSuiteName, Integer.toString(testCaseIDValue), "pass", null);
Following is the logic inside onTestFailure:
new AzureDevOpsActions().updateTestResultsInTestRun(testSuiteName, Integer.toString(testCaseIDValue), "fail", result.getThrowable().toString());
Supporting Methods
Following are the supporting methods:
Following are the supporting methods:
- getRequestWithAuthToken. Below is the logic to authenticate and execute GET APIs:
private Response getRequestWithAuthToken(String baseURI, String endPointPath, String authenticationScheme,
String token) {
Response response = null;
try {
response = RestAssured.given().baseUri(baseURI)
.header(HttpHeaders.AUTHORIZATION, authenticationScheme + " " + token).when().get(endPointPath);
System.out.println("==========================================================");
System.out.println("Execution completed successfully");
} catch (Exception e) {
e.printStackTrace();
}
return response;
}
- postRequestWithBody. Below is the logic to authenticate and execute POST APIs:
private Response postRequestWithBody(String baseURI, String endPointPath, String requestBody,
String authenticationScheme, String token, String contentType) {
Response response = null;
try {
response = RestAssured.given().header(HttpHeaders.AUTHORIZATION, authenticationScheme + " " + token)
.contentType(contentType).body(requestBody).when().post(baseURI + endPointPath).then().log().all()
.extract().response();
System.out.println("==========================================================");
System.out.println("Execution completed successfully");
} catch (Exception e) {
e.printStackTrace();
}
return response;
}
- patchRequestWithBody. Below is the logic to authenticate and execute PATCH APIs:
private Response patchRequestWithBody(String baseURI, String endPointPath, String requestBody,
String authenticationScheme, String token, String contentType) {
Response response = null;
try {
response = RestAssured.given().header(HttpHeaders.AUTHORIZATION, authenticationScheme + " " + token)
.contentType(contentType).body(requestBody).when().patch(baseURI + endPointPath).then().log().all()
.extract().response();
System.out.println("==========================================================");
System.out.println("Execution completed successfully");
} catch (Exception e) {
e.printStackTrace();
}
return response;
}
- getValueFromResponse. Below is the logic to get the value from JSON responses
private String getValueFromResponse(Response responseData, String parameterName) {
String jsonValue = " ";
try {
jsonValue = responseData.then().extract().path(parameterName);
System.out.println("Value of " + parameterName + " is " + jsonValue);
} catch (Exception e) {
try {
JsonPath jsonPath = new JsonPath(responseData.asPrettyString());
jsonValue = Integer.toString(jsonPath.getInt(parameterName));
} catch (Exception ex) {
try {
JsonPath jsonPath = new JsonPath(responseData.asPrettyString());
jsonValue = jsonPath.getString(parameterName);
} catch (Exception exception) {
System.out.println("Failed to get the value from the response");
}
}
}
return jsonValue;
}
Test Result Update is one of the important activity in the test case execution life cycle. Most of the test management tools supports the integration to update the test results once the execution completed. In this article we discussed about implementation of test result update in Azure DevOps using REST APIs and Java programming language. I hope everyone enjoyed with the implementations. Please try to utilize this opportunity in your testing world.
make it perfect!
journey of quality 
Leave a comment