Test automation execution

The JD says: execute automated tests with customer-provided frameworks, analyse the results, and report failures. You don't need to be a programmer. This chapter teaches you exactly enough Robot Framework and Python to do this job and answer the questions.

What test automation is

Words you'll hear:

• Framework: the ready-made structure that runs tests and produces reports (like Robot Framework or pytest).
• Test script / test case: one automated test.
• Test suite: a group of automated tests (often a folder or file).
• Library: a toolbox of reusable functions, e.g. "send CAN signal", "take screenshot".
• Keyword: one step in Robot Framework, like "Set Signal".
• Test run / execution: one run of a suite, producing results.
• CI (Continuous Integration): a server (e.g. Jenkins) that automatically runs the suites, e.g. every night or for each new build.

Frameworks you might meet in automotive HMI

Your role: execute, watch, analyse, report

1. PrepareCorrect build flashed, bench healthy, correct configuration/variables (bench ID, variant), phones/USB connected.
2. Start the runFrom the command line, the test tool or Jenkins. Choose the right suite or tags.
3. Watch the first minutesMake sure it really runs (bench responding, no immediate mass failures).
4. Collect resultsReports, logs, traces, screenshots, crash dumps.
5. Analyse every failureClassify: product bug, script problem, environment problem, flaky, known issue.
6. ReportTickets for product bugs; messages to the automation team for script problems; an execution report for the proxy team.

Robot Framework from zero

What a test file looks like

Robot tests live in .robot files. The file has sections that start with ***. Columns are separated by two or more spaces.

*** Settings ***
Documentation     Cluster temperature tests (REQ-HMI-TEMP-012)
Library           CanSimLibrary          # our keywords to send CAN signals
Library           ClusterScreenLibrary   # our keywords to check the screen
Suite Setup       Power On Bench         # runs once before all tests
Test Teardown     Collect Logs On Failure    # runs after each test
Test Tags         cluster    temperature

*** Variables ***
${BENCH}          B-03
${WAIT}           2s

*** Test Cases ***
TC-02 Snowflake appears below 5 degrees
    [Documentation]    3-value BVA, just below the boundary
    [Tags]    smoke    REQ-HMI-TEMP-012
    Set Signal    Outside_Temp    6.0
    Sleep    ${WAIT}
    Set Signal    Outside_Temp    4.9
    Sleep    ${WAIT}
    Icon Should Be Visible    snowflake

TC-03 Snowflake not shown at exactly 5 degrees
    [Tags]    boundary
    Set Signal    Outside_Temp    5.0
    Sleep    ${WAIT}
    Icon Should Not Be Visible    snowflake

*** Keywords ***
Power On Bench
    Set Power Supply Voltage    13.5
    Switch Ignition    ON
    Wait Until Keyword Succeeds    60s    2s    Screen Should Show    home

Where keywords come from

Keywords come from libraries. BuiltIn is always available (Log, Sleep, Should Be Equal, Should Contain…). Others: OperatingSystem, Collections, String, Process, SSHLibrary, AppiumLibrary. The customer usually has their own libraries written in Python, e.g. a method def set_signal(self, name, value): in Python becomes the keyword Set Signal.

Running tests: the commands

The exit code of robot is the number of failed tests (0 = all passed). Jenkins uses this to mark a job red or green.

The three result files

How to read log.html

1. Open the failed test (red).
2. Expand keywords until you find the red keyword: the step that failed.
3. Read the failure message, e.g. Icon 'snowflake' is visible but should not be.
4. Look at the screenshot and the timestamp (e.g. 20260930 10:15:32.124).
5. Check the steps before it: were the preconditions right (e.g. did "Set Signal" succeed)?
6. Use the timestamp to open the CAN trace and the ECU log at the same moment.

Statuses

PASS, FAIL, SKIP (test skipped on purpose, e.g. feature not in this variant). A failure in setup fails all tests in it — often a sign of an environment problem, not many product bugs.

Analysing failures (triage)

False positives and false negatives

• False positive (false fail): the test says FAIL, but the product is fine. Example: the screenshot comparison fails because a clock in the corner changed.
• False negative (false pass): the test says PASS, but the product is broken. More dangerous! Example: the script checks only that "an icon" is visible, not which icon.

Reporting automation results

A ticket for an automation-found bug includes

• Test name and ID, suite, run/job link, build, bench.
• The failing step and message from log.html; screenshot.
• Your manual reproduction result ("Reproduced manually 3/3") and manual steps.
• Trace and log excerpts around the timestamp.
• Since which build it fails (compare with previous runs).

Execution report example

Nightly regression – build 24.3.1 – bench B-03 – 30.09.2026
Total 412 | Passed 389 | Failed 17 | Skipped 6 | Pass rate 94.4%
Failures:
  4 product bugs  -> new: IVI-1432, IVI-1433 | existing: IVI-1301 (x2)
  7 environment   -> USB hub power loss at 02:14; fixed; re-run: 7/7 pass
  4 script issues -> icon ID renamed in 24.3.1; reported to automation team (AUT-88)
  2 flaky         -> TC-MED-221 (3/10 fails), TC-BT-017 (1/10) under analysis
Not run: none | Blockers: none

Python basics for testers

The 10 basics you need to recognise

# 1. Variables
speed = 50                 # a number
unit = "km/h"              # text (a string)
# 2. Printing
print("Speed:", speed, unit)
# 3. Lists
temps = [3.9, 4.0, 4.1, 4.9, 5.0, 5.1]
# 4. Dictionaries (key -> value)
expected = {4.9: True, 5.0: False, 5.1: False}
# 5. if / else
if speed > 120:
    print("Overspeed chime!")
else:
    print("OK")
# 6. for loop
for t in temps:
    print(t)
# 7. Functions
def snowflake_on(temp):
    return temp < 5.0
# 8. Using the function
for t, exp in expected.items():
    result = snowflake_on(t)
    print(t, "PASS" if result == exp else "FAIL")
# 9. Reading a file
with open("cluster_log.txt", encoding="utf-8") as f:
    lines = f.readlines()
# 10. Handling errors
try:
    value = float("abc")
except ValueError:
    print("Not a number")

Useful script 1: find errors in a log

errors = 0
with open("cluster_log.txt", encoding="utf-8", errors="replace") as f:
    for number, line in enumerate(f, start=1):
        if "error" in line.lower() or "exception" in line.lower():
            errors += 1
            print(number, line.strip())
print("Total error lines:", errors)

Useful script 2: send and decode CAN signals (python-can + cantools)

import can, cantools

db = cantools.database.load_file("car.dbc")          # the DBC dictionary
msg = db.get_message_by_name("Ambient_Info")
data = msg.encode({"Outside_Temp": 4.9})             # real value -> raw bytes

bus = can.Bus(interface="vector", channel=0, bitrate=500000)
bus.send(can.Message(arbitration_id=msg.frame_id, data=data, is_extended_id=False))

reply = bus.recv(timeout=1.0)                        # read one message
if reply:
    print(db.decode_message(reply.arbitration_id, reply.data))

Libraries: python-can talks to the CAN interface; cantools reads DBC files and converts values. The details (interface name, channel) depend on your bench.

Useful script 3: a tiny pytest test

# file: test_snowflake.py   run with:  pytest -v
from cluster_logic import snowflake_on

def test_just_below_boundary():
    assert snowflake_on(4.9) is True

def test_on_boundary():
    assert snowflake_on(5.0) is False

CI and Jenkins

• Jenkins is a server that runs jobs: e.g. "every night at 01:00: flash the latest build on bench B-03, run smoke, then regression, then publish the results".
• A job has stages (flash → smoke → regression → report) and artifacts (report.html, log.html, traces).
• Red job = something failed; open the console output first to see whether the tests ran at all (e.g. flashing failed = environment).
• Your daily routine: check the nightly jobs each morning, download the logs, triage the failures, send the status.

Keeping automation trustworthy

• Track flaky tests and get them fixed; a suite nobody trusts is worthless.
• Keep benches identical and documented.
• Update scripts when the HMI changes on purpose (new text, new layout).
• Review false negatives: if a bug escaped the automated test, improve the test.

Quick answers for the interview