What is App Experience?

Getting Started

This user manual shows you on how you can use App Experience to test a native Android or iOS app. You can use the App Experience operations console to create KPIs, upload scripts, create orders, select devices, run or schedule a suite of standard tests, and then view the results on App Experience dashboard.

After successfully signing up, sign into Operations Console and App Experience Dashboard using your credentials.

Next Section: Working with Operations Console

Working with Operations Console

This section explains on how to use operations console. Operations console helps a user to manage KPIs, scripts, orders, tests, status for the automated tests used to measure app experience.

Here is a flow for a new user to create an order using operations console:

Flow 1: Create an Order

Here is a flow for a user to modify an existing order using operations console:

Flow 2: Modify an Order

Manage KPIs

This section explains on how to add, delete, and edit a KPI through operations console as given in Figure 01.

Figure 1: Manage KPIs

Add a new KPI

• To add a new KPI, click on “Add New KPI” as given in Figure 02.
• Populate all the mandatory fields:
  • KPI Category
  • KPI Name
  • KPI Unit
  • KPI Description
  • Default Thresholds
  • Outlier
• Filling ‘App Category Relevance’ is optional.

After clicking Add, the new KPI gets added and can be seen in the KPIs table.

Figure 2: Manage KPIs – Add KPI

Delete a KPI

To delete a KPI, click on X symbol as given in Figure 03. After clicking Yes, the KPI will be deleted.

Figure 3: Manage KPIs – Delete a KPI

Edit a KPI

To edit a KPI, click on the pen symbol as given in Figure 04. After editing the desired fields, the KPI will be edited.

Figure 4: Manage KPIs – Edit a KPI

Filter By

Filter by helps in finding the KPIs by giving KPI Name as an input. The corresponding KPIs will get reflected in the tale.

Manage Scripts

This section explains on how to add, delete, and edit a script through operations console as given in Figure 05.

Figure 5: Manage Scripts

Add a Script

• Click on “Add New Script” to add a script as given in Figure 06.
• Populate all the mandatory fields:
  • App Name
  • App Category
  • Script Name
  • OS
  • Platform
  • Framework
  • KPIs(after clicking next)
• Filling ‘Script Description’ is optional. After clicking Add, the new script gets added and can be seen in the scripts table.

Figure 6: Manage Scripts – Add a Script

Delete a Script

To delete a script, click on X symbol as given in Figure 07. After clicking Yes, the script will be deleted.

Figure 7: Manage Scripts – Delete a Script

Edit a Script

To edit a script, click on the pen symbol as given in Figure 08. After editing the desired fields, the script will be edited.

Figure 8: Manage Scripts – Edit a Script

Filter By

Filter by helps in finding the scripts by giving Script Name as an input. The corresponding scripts will get reflected in the table as given in Figure 09.

Figure 9: Manage Scripts – Filter By

Manage Orders

This section explains on how to add, delete, and edit an order(app or telco) through operations console as given in Figure 10.

Figure 10: Manage Orders

Add a new Order

There are two kinds of orders. One is App order and another one is a telco order.
App orders take into account a specific industry segment and telco orders take into account multiple industry segments.

Populate the below fields in order to create a new order.

• App Order
  • Select Script (multiple selection)
  • Specify if PCAP and Video needs to be processed
    PCAP: {default ON}
    Video: {default OFF}
  • App Category (auto-select)
  • Main App (select from a dropdown)
  • Benchmark app (provides apps (scripts) based on app category except for the main app)
  • Weights
• Telco Order
  • Select Script (multiple selection)
  • Specify if PCAP and Video needs to be processed
    PCAP: {default ON}
    Video: {default OFF}
  • Main Telco (free text)
  • Benchmark Telco (free text, multiple inputs)
  • Apps categories (auto-select, inputs)
  • Apps inside each category(auto-select, multiple inputs)
  • Weights for every category

Assign Weights to KPIs

While adding weights, select the KPI category and assign the appropriate weights in % to it. Submit only when the summation of all the weights are equal to 100%.

After clicking Submit, provide weights for KPIs. KPIs will get selected from the dropdown as given in Figure 11.

For Telco order, weights for different apps will be added in the same way the weights for different apps are added as given in Figure 12 and 13.

Figure 11: Assign Weights to KPIs for an App Order

Figure 12: Assign Weights to KPIs for an App Order

Figure 13: Manage Orders – Add Weights to a Telco Order

Delete an Order

To delete an order, click on X symbol as given in Figure 14. After clicking Yes, the order will be deleted.

Figure 14: Manage Orders – Delete an Order

Edit an Order

To edit an order, click on the pen symbol as given in figure 15. After editing the desired fields, the order will be edited.

Figure 15: Manage Orders – Edit an Order

Filter By

Filter by helps in finding the orders by giving Customer(Order) Name as an input. The corresponding orders will get reflected in the table as given in Figure 16.

Figure 16: Manage Orders – Filter by

Start Tests

This section explains on how to schedule both continuous and sporadic tests using operations console as shown in Figure 17.

Figure 17: Start Tests

To start a test, populate and click the below fields:

• Select Order
• Select OS
• Select Scripts
• Capture
• Time between start of two tests
• Recurrence
  • One Time
  • Schedule
• Get Devices
• Filter By
• Tray Description

Now click ‘Start Tests’. You will get a success message that the tests are scheduled.

Check Status

This section explains on how to check, delete, or edit the tests that are scheduled through operations console.

To view devices and test results, click as given in Figure 18.

Figure 18: Check Status – View Devices, Test Results

You can delete trays(a group of devices) and a device. But you cannot delete test results.

To delete a tray and to delete a device in a tray, click on X symbol as given in Figure 19. After clicking ‘Yes’ as given in Figure 20, the tray or a device in a tray will be deleted.

Figure 19: Check Status – Delete Devices, Delete a Device

Figure 20: Check Status – Click ‘Yes’ to delete a device or a tray(a group of devices)

To edit a tray, click on the pen symbol as given in Figure 21. After editing the desired fields, the tray will be edited.

Figure 21: Check Status – Edit a Tray

Filter By

‘Filter by’ helps in finding the trays by giving tray name as an input. The corresponding trays will get reflected in the table as shown in Figure 22.

Figure 22: Check Status – Filter By

User Management

This section explains the Operations Console features accessible by two different users – Admin and Normal.

Next Section: Working with Dashboard

Working with Dashboard

This section explains on how to use App Experience dashboard. App Experience Dashboard helps the users to view the results from the automated tests and analyse the test results to assess the performance of the apps taken into consideration.

Monitoring

This section explains on how to use ‘Monitoring’ section of App Experience dashboard. You can add, delete, edit different kinds of cards aggregating indices and metrics and charts such as comparison charts, distribution charts, bar graphs to aid in your analysis across different networks conditions, geographies, network operators, etc.

Add a Card

• To add a new card, click on ‘Add Card’ as given in Figure 23
• Select KPI type to choose from metric (raw value) and index (aggregated score)
• Select the KPI to display
• Choose relevant conditions for which you want to create the card
• Click on ‘Add Score’ to show the card on the dashboard as given in Figure 24

Figure 23: Monitoring – Add a Card

Figure 24: Monitoring – Add KPI Score

View More Cards

• To view all cards together, click on ‘View More’ as given in Figure 25.

Figure 25: Monitoring – View More Cards

Rearrange Cards

• To rearrange the cards after all cards are made, click on the green drag-and-drop button present over one of the cards and drag and drop the cards to their new position as given in Figure 26. Newly created cards are added at the end of the rows. First five cards from the View more section are displayed on the dashboard.

Figure 26: Monitoring – Rearrange Cards

Edit a Card

• To edit a card click on encircled i in the top right hand side of the card as shown in Figure 27.

Figure 27: Monitoring – Edit a Card

• Edit the details of the card as you require and click on ‘Add Score’ as given in Figure 28.

Figure 28: Monitoring – Add Score

Delete a Card

• To delete the card, click ‘Delete’ and your card will be deleted as given in Figure 29.

Figure 29: Monitoring – Delete a Card

Add a Graph

• To add a new Graph page, click on the + icon as shown in Figure 30.

Figure 30: Monitoring – Add a Graph page

• Double tap to rename the page tab and click outside the field to save it
• Click on ‘Add Graph’ to add a new graph as given in Figure 31.

Figure 31: Monitoring – Add Graph

• Choose from the 3 different graph types – Timeseries (trend analysis), Compare (condition wise analysis) and Distribution (histogram) as given in Figure 32.
• Select KPI type to choose between metric (raw value) and index (aggregated score)
• Select the KPI to display
• Choose relevant conditions for which you want to create the graph
• Click on ‘Add Score’ to show the graph on the dashboard

Figure 32: Monitoring – Choose different graph types

Edit a Graph

• To edit a graph, click on the pen located on top right hand side of the selected graphs in Figure 33.

Figure 33: Monitoring – Edit a Graph

• Edit your KPIs and conditions as required and click on ‘Update’ to publish the updated graph as shown in Figure 34.

Figure 34: Monitoring – Edit KPIs

Delete a Graph

• Click on the ‘Delete’ to delete a graph as shown in Figure 35.

Figure 35: Monitoring – Delete a Graph

Delete the Graphs Tab

• Click on the X on the tab to delete the complete Graph Tab as given in Figure 36.

Figure 36: Monitoring – Delete the Graphs tab

Benchmarking

This section explains on how to use ‘Benchmarking’ section of App Experience dashboard. You can add, edit, delete different kinds of cards aggregating indices and metrics and charts such as comparison charts, distribution charts, bar graphs to aid in your analysis across different peers.

Add a Card

• To add a new card, click on ‘Add Card’ as given in Figure 37.

Figure 37: Benchmarking – ‘Add a Card’

Add More Cards

• Select KPI type to choose from metric (raw value) and index (aggregated score)
• Select the KPI to display
• Choose relevant conditions for which you want to create the card
• Click on ‘Add Score’ to show the card on the dashboard as given in Figure 38.

Figure 38: Benchmarking – Add more cards

View More Cards

• To view additional cards click on ‘View More’ as given in Figure 39.

Figure 39: Benchmarking – View More Cards

Rearrange Cards

• To rearrange the cards after all cards are made, click on the green drag-and-drop button present over one of the cards and drag and drop the cards to their new position as shown in Figure 40.
• Newly created cards are added at the end of the rows. First five cards from the View more section are displayed on the dashboard

Figure 40: Benchmarking – Rearrange Cards

Edit a Card

• To edit a card click on pen at top right hand side of the card as given in Figure 41.

Figure 41: Benchmarking – Edit a Card

• Edit the details of the card as you require and click on ‘Add Score’ as given in Figure 42.

Figure 42: Benchmarking – Add Score

Delete a Card

• To delete the card, click ‘Delete’ as shown in Figure 43 and your card will be deleted.

Figure 43: Benchmarking – Delete a Card

Add a Graph

• To add a Graph, click on the + icon as shown in Figure 44.

Figure 44: Benchmarking – Add a Graph

• Double tap to rename the page tab and click outside the field to save it.
• Click on ‘Add Graph’ to add a new graph as given in Figure 45.

Figure 45: Benchmarking- Add Graph

• Choose from the 3 different graph types – Timeseries (trend analysis), Compare (condition wise analysis) and Distribution (histogram)
• Select KPI type to choose between metric (raw value) and index (aggregated score)
• Select the KPI to display
• Choose relevant conditions for which you want to create the graph
• Click on ‘Add Score’ to show the graph on the dashboard as given in Figure 46.

Figure 46: Benchmarking – ‘Add Score’

Edit a Graph

• To edit a graph, click on the pen located on top right-hand side of the selected graphs as given in Figure 47.

Figure 47: Benchmarking – Edit Graph

• Edit your KPIs and conditions as required and click on ‘Update’ to publish the updated graph as given in Figure 48.

Figure 48: Benchmarking – Edit KPIs

Delete a Graph

Click on ‘Delete’ to delete a graph as given in Figure 49.

Figure 49: Benchmarking – Delete a Graph

Delete the Graphs Tab

Click on the X on the tab to delete the complete Graph Tab as given in Figure 50.

Figure 50: Delete the Graphs Tab

Performance

This section explains on how to use ‘Performance’ section of App Experience dashboard. You can analyse the test results from different PCAPs processed and captured when testing was done through server wise analysis, capturing videos, waterfall analysis, burst analysis, timeseries analysis.

Pre-requisite

• To check performance of a particular Test ID, click on ‘Change’ as shown in Figure 51.

Figure 51: Performance – Change

• Select your KPIs to find a custom test you are looking for or Select a test from Results Part with us without ‘Video Capture’ available as shown in Figure 52.

Figure 52: Performance – Select your KPIs

• View KPI Values – Click on the + icon on left hand side of KPI to view its respective values as given in Figure 53.

Figure 53: Performance – View KPI Values

Server Name Analysis

This part shows you all types of server names being called by the app. (E.g.: YouTube). Either they are by CDN, 1st Party or 3rd Party. The count of each type of server name called is shown within the pie chart as given in Figure 54. In the table, you can see the details such as ‘IP’, ‘Bytes Exchanged’, ‘Packets Exchanged’. You can further view server name specific KPIs for Protocols, Latency, Errors by clicking on the check boxes for these server names.

Figure 54: Performance – Server Name Analysis

View Video Capture (VCAP)

In this section of the Performance dashboard, you can view the Video Capture of the whole test being conducted as marked in Figure 55. If the VCAP is not on, it will not show any video.

Figure 55: Performance – View Video Capture

Waterfall Analysis

In the waterfall analysis, you can identify issues with APIs or the endpoints. Here, you can see data test run wise as given in Figure 56. You can read the waterfall graph by different streams such as DNS, TCP, TLS, etc. You can find out the number of first party APIs called, third party APIs called and how many were sequential or parallel. You can also find out the important APIs that are called. You can spot if there is any time lag between DNS packets and TCP packets or TLS packets, if there is a long wait time at the server to respond to a request packet which typically happens in case the end point server is busy. You have the option to select the time frame depending on how detailed analysis you wish to do. You can check different protocols such as TCP packets, DNS, UDP and find out the sequence of packets exchanged in the waterfall analysis.

Figure 56: Performance – Waterfall Analysis

Timeseries Analysis

Select the Protocol from the drop down to view its Timeseries Analysis Score as given in Figure 57.

Figure 57: Performance – Timeseries Analysis

Diagnostics

This section explains on how to use ‘Diagnostics’ section of App Experience dashboard. You can analyse the test results from different PCAPs processed and captured when testing was done through comparison analysis, trend analysis across different peers, network conditions, network layers.

Select KPI

Click on ‘Change’ as shown in Figure 58 and select the KPIs you wish to diagnose.

Figure 58: Diagnostics – Select KPIs

Figure 59: Diagnostics – Update

Comparison Analysis

Select the Latency, Protocol and Error KPIs from individual dropdowns to compare with each other as given in Figure 60.

Figure 60: Diagnostics – Comparison Analysis

Drill Down by Server

Refer to ‘Waterfall Analysis’ where we identified important endpoints. To inspect more about these endpoints for performing metrics averaged across all tests we can use the ‘Drill down by Server’ section. Select an ‘App Name’, apply Filters – ‘column names’, ‘greater/lesser than’, ‘sort value’, search for a particular server, select a server and click on ‘Inspect’ given in Figure 61 and inspect server based on Protocol, Latency, Error KPIs as shown in Figure 62.

Figure 61: Drill Down by Server

Figure 62: Inspect Server

Trend Analysis

Select how do you wish to see the trends – Weekly, Daily, or Hourly as given in Figure 63.

Figure 63: Diagnostics – Trend Analysis

App KPI based – Select the Network, App and App KPI whose trend you want to analyse as shown in Figure 64.

Figure 64: Diagnostics – App KPI based

Network KPI based – Select the Network, App and Network KPI whose trend you want to analyse as given in Figure 65.

Figure 65: Diagnostics – Network KPI based

Wiki

In this section, you can view UX KPIs, Diagnostic KPIs and details regarding data collection.

UX KPIs

Scroll down to see the UX KPIs and their respective weightage scores as shown in Figure 66.

Figure 66: Wiki – UX KPIs

Diagnostic KPIs

When you scroll down to Diagnostic KPIs section, you will view the apps KPIs in their respective categories and units mentioned as shown in Figure 67.

Figure 67: Wiki – Diagnostic KPIs

Data Collection

In the date section, you can set the dates to view the data/results for a particular data collection period as shown in Figure 68.

Figure 68: Wiki – Data Collection

Settings

This section is only available to admin users. Click on ‘Continue’ under any card to open that section as given in Figure 69.

Figure 69: Settings – Continue

Define Threshold

• Select Index:
  In this section, you can change cut-off values in raw KPIs that determine index scores from a range of 1 to 4. Select the index whose thresholds you want to edit by clicking on the name as given in Figure 70.

Figure 70: Settings – Define Threshold

• Configure Index:
  Edit value by assigning values to each score. Keep note of not missing out on score range, i.e. the value in “greater than” for a score should be equal to the value in “less than or equal to” in the previous row. Proceed ahead by pressing on Continue as given in Figure 71.

Figure 71: Settings – Configure Index

Manage Users

In this section, you can Add (by pressing the create user button), edit and enable/disable users as given in Figure 72.

Figure 72: Settings – Manage Users

Define Weights

• Select Weight

Figure 73: Settings – Select Weight

• Configure Weight
  Configure weights assigned to base index to configure calculation methodology of aggregate index in this section. Click on the Weight name to proceed ahead. Assign weight out of 100 to the scores as per desired configuration. Please ensure that all weights add up to exactly 100%. Press Save to save configuration and continue as given in Figure 74.

Figure 74: Configure Weights

Classify ESNI

Configure ESNIs into 1st party, 3rd party, and CDN categories to view network performance across different ESNI on a categorical level. Select the App, whose ESNIs you want to edit. Next, select the ESNI class. Finish by assigning keywords from the ESNI that you want to assign under App-ESNI class combo. Previous settings can be edited/deleted as given in Figure 75.

Figure 75: Settings – Classify ESNIs

User Management

This section explains the Dashboard features accessible by two different users – Admin and Normal.

Next Section: Document History

Document History

S.No.	Date	Version No.
1	Dec 4, 2020	1.0

Next Section: Resources

Resources

This section contains links to product video tutorials that can help you in getting started with App Experience.

Product Videos [YouTube Channel]

1. How to use operations console?
   View on Microsoft SharePoint
2. How to perform experience testing on MOZARK?
   View on Microsoft SharePoint
3. How to measure app experience?

4. How to diagnose app experience?
View on Loom

Next Section: Support

Support

For further assistance, you can reach out to enquiry@mozark.ai.