Getting Started

This is a good presentation to introduce t-Triage to teams: http://tiny.cc/sjnw9y

What is t-Triage?

t-Triage is a tool that adds Intelligence in your Automation Testing process providing visibility, setting new workflows and saving time.

Generally speaking, Automation Testing has 3 main phases:

1. Test creation
2. Test execution
3. Test Supervision / Validation

t-Triage focuses mainly on the third area, and overviews the first two. It validates how tests are being executed and it sets a workflow for failed tests.

Executed tests are not only pass and fail, there are deadlines, there are priorities, there are relations… Using Big Data, t-Triage adds Intelligence in your Automation by getting sure everything is executed and validated on time and classifying how the automation behaves along time. By using Machine Learning and Artificial Intelligence, t-Triage saves meaningful time on your failures, false positives and lets you focus on the most important things first.

What does Triage mean? The term Triage is related to the process of classification to determine, in this case, how important is each automated test revision, given that tests may be passing, failing over a stable area, on a Smoke test, facing a flaky test and so on, analyzing different kind of automation scenarios.

After t-Triage processes your automation suites, it will generate a report where automators will work on a prioritized and organized set of tests to triage.

Let’s show you in detail how it works!

How t-Triage works?

In general terms, by automation testing, we manage groups of tests, running them in suites, and that suites grouped by any condition in containers or Views (for example, smoke tests or regression tests).

The whole set of tests for an application is what we call Product (mainly, a group of Containers or Views). A company can manage more than one Product; for example: two different applications, or the cloud version and the on-premise version of an application.

t-Triage is grouping the automation results per Views, that is, for example, equivalent to a View in Jenkins or a Project in Circle.

When you open the t-Triage report, your workspace will show your “My Today’s List”, that contains the list of suites assigned to you, that should be triaged today. Below it, you’ll find a “Backlog” list, where you can choose a View and list its suites to be triaged in the next days.

Triage

My Today’s List

This list contains the tests you have assigned to triage:

• Suite name: suite name and view/container where it is
• To Triage|Status: number of tests to triage per status:
  • New fails: tests that failed for the first time after passing runs
  • Fails: tests that failed in this and previous runs
  • New pass: tests that passed after previous failures
  • Auto triaged: tests automatically triaged based in previous runs triage information
• Triage Deadline: when is expected to have the suite fully triaged: today, this week, etc. Hover overing the deadline you’ll see the due date and time.
• Priority: priority assigned to this suite triage

The list shows ordered by triage priority, so you can focus in the most critical first. And at the same priority for several suites, you should triage first those with a red clock in Triage Deadline.

The clock colour indicates also a triage priority; it is determined by an algorithm considering several variables, including the time remaining until the deadline and the number of failures to triage in the suite. I.e., less time and a big percentage of tests to analyze will have a higher priority.

Assignee, Triage Deadline and Priority are determined based on the parameters entered in the Container definitions, in the Admin Panel. Priority can be changed from the workspace, and will also apply for the next runs.

Clicking on the suite name, you'll go to the suite details view, in a Kanban format.

Backlog

In the “Backlog“ list, you’ll see a drop-down menu with the containers having suites to be triaged in the following days, also ordered by Triage Deadline and Priority. In this case, you’ll see all the triage pending suites, not only the assigned to you.

By default, it is selected the first container in the list, but you can select to view another from the list or all of them.

For the selected container/s, the backlog list shows this info:

• Suite name: suite name
• To Triage|Status: number of tests to triage per status: new fails, fails, new pass, auto triaged
• Triage Deadline: when is expected to have the suite fully triaged
• Priority: priority assigned to this suite triage
• Assignee: who is the responsible of this suite triage

Priority and Assignee can be changed from the workspace, and will also apply for the next runs.

You have a Filter and Sort menu, where you can:

• Filter by:
  • Failures → to only view suites with failed tests
  • My Assigned Jobs → to only view the suites assigned to the logged user
• Sort by:
  • Priority → jobs will show ordered by triage priority, highest priority first
  • Suite name → jobs will show in alphabetical order

Search: you can use the Search box where you need to find:

• an specific suite to work on
• an specific error

Clicking on the suite name, you'll go to the suite details view, in a Kanban format.

Kanban view of a suite

For the selected suite, in this view this is what you’ll find:

• Suite Name

• Summary: you’ll find general information about the suite and actions buttons to execute over the suite
  • Information:
    • Assignee: automator assigned to triage this suite
    • Total test: how many tests are running in this suite
    • Tests pass: number of tests passing
    • Tests skip: number of tests skipped
    • Triage Deadline: when is expected to have the suite fully triaged (today, this week, etc.). Hover overing the deadline you’ll see the due date and time
    • Stability Index:  this number indicates the health of the suite, considering also historical runs; higher number, the better performance
  • Action buttons:
    • Triage Suite: make all the tests as triaged
    • Pull: update the suite information based in a newest run
    • Skip Run: by this action the run is considered invalid and it will be discarded by t-Triage; this is useful, for example, for cases where the suite execution failed due to an invalid setup, wrong suite configuration, etc.
    • Disable Suite: whit this action, the suite will not longer be considered in the triage process

• Filter and Sort menu: here you can apply some selection and / or order criteria to facilitate your task:

• Filter by:
  • Hide Automated Triages → by turning it on, tests automatically triaged will not shown, so you only will view tests that requires manual triage
  • Show by bugs only → by turning it on, only tests with a product bug or a test issue will show
• Sort by:
  • Test Name → tests will show in alphabetical order
  • Test Priority → tests will show ordered by the triage priority, higher first

• Search panel: you can enter an specific test here, to easily find it to work on

• Failed tests: in this section are listed the failed tests, displaying a card per each test, and grouping them in columns depending on the test status. Also the total number of tests per status is shown. The possible test status are:
  • New Fail: tests that are failing the first time after successful runs
  • Fail: tests that have previous run also failing, or tests running for the first time failing
  • New Pass: tests that passed in this run but failed in the previous one/s
  • Triage Done: tests already reviewed

• Test cards: they are showing:
  • Test group
  • Test name
  • Pin icon: this icon will be shown only if the test is pinned
  • Information of triage icons: hoverovering each of them, will show the information entered in the “Previous Triage” or in the “In Progress Triage” (you have saved some triage information, but you have not set the test as triaged yet), related to:
    • Product: shows analysis about the test Functionality
    • Automated Test: shows analysis on the Automated Test
    • Comment: shows the comment added
  • Tick icon: this icon will show only if the test is being triaged. Its format is dependant on the triage progress:
    • a single tick if you have triaged Product or Automated Test behavior
    • a double tick if you already triaged both of them
    • “A” and a tick for Auto Triaged tests (only in Triage Done column)

                    Hoverovering the icon, a tooltip will show you the status (for example: “In progress: Test under analysis” or “Done: all options set”)

Clicking on a test card will open a detailed Test information page.

Test details

Clicking on a test card in the Kanban view, will open a new page with the Test details. It contains basically a failure information section and a triage section

Test information

• Summary:
  • Status: of the test (for example: fail, new pass) and a description of the test situation result of the t-Triage process
  • Assignee
  • Execution information: Suite where the test run, Date (week day, date and time), and Duration in milliseconds
• Error Details: error causing the failure, with the option to Copy to Clipboard
• Stack Trace: logged stack trace, also with the option to Copy to Clipboard
• Stats:
  • Consecutive Fails
  • Historic Passes
• Déjà vu Error, same error found in other tests: if exist, list of other tests failing by the same error, with a link to view details of those tests, and their status
• Déjà vu Test, same test found in other suites: if exist, list of other suites running the same test, test status there and a link to view that test details
• Previous Runs: information related to the test under analysis in the previous executions, helpful in the triage process; by default the latest previous run is shown, but you can navigate here between all of the available executions. This is the information shown:

• Previous run number / Total number of previous runs (for example: 10/10, 8/10, etc.)
• Date of execution of the previous run you are viewing (by default, the latest), and Previous and Next buttons, to navigate between executions
• Summary: status, assignee, execution date, triage information related to the Product and the Automation Test
• Errors, with the option to Copy to Clipboard
• StackTrace, with the option to Copy to Clipboard

Triage Action

In this section, there are several check boxes where you can set your diagnostics about the failure and actions took / to do, related to the Product, the Automated Test, and also the option to add your observations in a Comment:

Product analysis:

• For failed tests:
  • Product Works Fine
    • Manual Test Succeed
    • Test Environment Issue
  • Not working
    • Product Bug URL: link to the ticket filed in the tool you use for reporting application bugs (e.g Jira)
    • Skip
• For passing tests:
  • Product Works Fine
    • App is working 
  • Not Working
    • App Unstable, still In Dev

          The link to the previously filed application bug ticket is shown

Automated test analysis:

• For failed tests:
  • Automated Test Works Fine
    • Re run and worked
    • External Cause
  • Not working
    • File Issue: if you choose this option, you need to select also that issue Priority
    • Won’t Fix
    • Skip
• For passing tests:
  • Automated Test Works Fine
    • Test is working
  • Not working
    • Test still needs improvement

Comment:

• Here you can enter your notes/observations based in the analysis done, that will help you and other automators in next executions or similar failures

When triaging, you are evaluating the Product or application and the automated test. So you need to enter one option in Product Works Fine or in Not working, and one option in Automated Test Works Fine or in Not working.

Related to the “Skip” options, they should be selected in exceptional situations, where for any reason you need to set the test as triaged and you cannot research to determine the failure cause. In this way you are delaying the triage to the next execution.

Save and Triage Done buttons

Once you have reviewed the failure and triaged it, you need to save your analysis.At the bottom of the Triage Action section, you will find these buttons:

• Save: you can change what you’ve entered as part of your analysis, but the test is not marked as Triaged, and by that reason it will not be moved to the Triaged Done column in the Kanbas view of the suite

• Triage Done: your changes are saved and the test is set as Triaged, so it will shown under Triaged Done column in the Kanbas view of the suite

Automation Issues

In this section all the automated tests triaged as failed are listed, so you can get in a single view all the automation tests that require some work.
The tests are listed ordered by the priority assigned when triaged and for each test this information is shown:

• Test name, that is also a link to the test details
• Involved suite name
• Failed since: number of run and date of the first failure, and also how many times the test has failed
• Fix priority: the priority assigned at triage stage
• Issue type:
  • Open: the test needs work
  • Reopen: the test was fixed in the past, but it failed again
  • Passing: the test was fixed and it is passing now
  • Fixed: the test passed more than four consecutive times after its fix
• Error details: a portion of the error is shown, and clicking on “Error Details” a popup will be opened with the error of the test, where the “Copy to clipboard” option is available
• Stack trace: clicking here a popup will be opened with the stack trace info, where the “Copy to clipboard” option is available
• Assignee: automator assigned to the fix the test. Clicking on its name, a combo showing also other automators is shown, and the assignee can be changed here. The automators suggestion is initially done based on the people working on the suite, so that list could be empty. Changing assignment here doesn’t change assignee for triage tasks.
• User Comment: you can enter comments based in the work/analysis you have done
• Pin icon: it is shown if the test is pinned or it will be grayed out if the test is not pinned; you can also pin or unpin the test from here

t-Triage will show the fixed tests that are passing now, for four consecutive runs; afterwards the test will be considered stable, “Fixed”, and it will not longer be shown here.
At the top of this page, you’ll find a Filter and Sort icon and a search box.
By clicking the “Filter and Sort” icon, you can choose and apply your preferences:

• Filter by:
  • Pinned
  • Assigned to me
  • Hide passing: this is selected by default
• Sort by:
  • Priority: this is selected by default
  • Older first
  • Newer first
  • Suite name

In the search box you can look for an specific suite to work on. Clicking here will open a popup, with the list of the suites; you can select an specific one or “All suites”.

Insights

This page shows some statistics to have the general automation status in a quick view.

Totals

• Unique Tests: unique test cases in the system; uniqueness is identified by test name and suite name

• Triaged Fails: triage fails is the historic value of all the fails that have been manually or automatically triaged

• Saved Time: the hours saved to the automators by avoiding unnecessary triages, i.e. calculated upon the number of performed automatic triages

• Achieved Deadlines: the amount of test suites analyzed before the deadline

Ongoing Tests

The total number of test ran and a pie graph grouping the tests by their status: Fails, Triaged Fails, Passed. Hoverovering each status in the pie, the number of tests and the percentage of the total that represent are shown.

Fail Exceptions

For the top exceptions, a pie graph showing the total number of tests failing by exception. Hoverovering each of them in the pie, the number of tests and the percentage of the total that represent are shown.

How many exceptions to consider in this statistic is configurable, by default the top 7 exceptions are included.

Burndown: Fails vs. new tests and fixes

A line graph showing totals of fixes, fails and new tests per month. Hoverovering the graph, the total number of test is shown for each value.

Bugs filed

A line graph indicating the number of automation issues filed per month. Hoverovering any point in the graph, a popup showing the month and the number of bugs is opened.

Pending to triage

The total number of tests with triage pending is shown in a bar graph, showing New Fails, Fails and Now Passing not triaged tests by test suite. A different colour per each value is shown in the bar, and hoverovering it a popup showing the totals is opened.

Missing deadlines

A bar graph that shows the number of missing deadlines per month is shown. Hoverovering each bar, the month and the number of deadlines not reached are shown.