Information

Background

Previous session at UDS-O
Previous session at UDS-M
Previous session at UDS-N
Laptop Testing page

Summary

"Laptop Testing" is a community-driven project started with Lucid Lynx Alpha 3. Our goal is get Ubuntu working great on as many different makes and models of laptops as possible, mainly (but not only) testing point releases of the development cycle to help devs in triaging bugs. This could be done both in all the Community and LoCos, sharing hardware info about Laptops and trying to attract as many new contributors as possible.

Project has a Testing Team on launchpad. It uses the QA Mailing List and IRC channel #ubuntu-testing. In the Testing Team Wiki page this project is reported under Activities along with an English page.

In this session we would like to present goals reached and some suggestions to improve the project.

Activity

Natty Cycle Data

Tests here (Wiki)
Tests here (Laptop Tracker)

Milestone	People/Laptop	Test cases performed *
11.04 Beta 1	19	174
11.04 Beta 2	17	191
11.04 Final	11	236

Oneiric Cycle Data

29 laptop tested at least one milestone

Tests here (Laptop Tracker)

Milestone	People/Laptop	Test cases performed *
11.04 A1	6	54
11.04 A2	4	56
11.04 A3	2	28
11.04 Beta 1	15	174
11.04 Beta 2	11	102
11.04 Final	7	58

Failures summary

18 bugs have been reported with 9 test cases failed:

Testcase

Total/Failed

Bugs

System

Suspend/Resume	25/13	10
Memory Usage	27/1	1
Internal Data Card Reader	25/3	6
Builtin Webcam	23/1	4

Input/Output

Builtin Keyboard	30/2	3
Hotkeys	20/10	9
Touchpad	30/1	1
USB Mouse	21	1

Audio

Internal Speakers	36/1	2
Headphones	31/1	5
Internal Microphone	33/1	1
External Microphone	28/1	1

Video

Builtin Screen	32/4	4
External Screen	10/3	6

Networking

Cabled	25	1
Wireless	30/1	2

External Expansion Port

USB Ports	27/3	5
Firewire Ports

Bug Tasks Importance

* Critical :	2 (Fix Released)
* High :	1 (Fix Released)
* Medium :	1 (Incomplete)
* Low :	6 (1 Fix released, 2 in progress, 1 Triaged, 2 Confirmed)
* Undecided :	46 (5 Fix Released, 3 Triaged, 13 Confirmed, 5 Incomplete, 11 New, 5 Invalid, 3 Expired, 1 Won't Fix)

P.S.: About half of bugs have been reported against "linux" package

Graph (dev cycles)

Laptop Tracker

Blueprint on Launchpad
Website: laptop.qa.ubuntu.com

Thoughts

About the same number of tests performed in Beta 1, slight drop in Beta 2.
People need time to test their laptop (see the increment in Natty Final)
Critical areas for laptops (most bugs/test failures) are Suspend/Resume and Hotkeys.
Internal Data Card Reader, Video (internal and external) need attention too.

What's next

Try to automate the testing process as much as possible with checkbox. It can be patched to detect a dev release, so it can be used to test/report a bug about a laptop for the tracker without sending reports for the UF project (that it's used to validate hardware for stable releases)
Integrate in some way LTT/UF projects?

CategoryHomepage

-  ⇤ ← Versione 1 del 19/10/2011 10.01.07 → 
  Dimensione: 7031
  Autore: SergioZanchetta
  Commento:
+   ← Versione 25 del 24/10/2011 16.41.34 → ⇥
  Dimensione: 6332
  Autore: SergioZanchetta
  Commento:
-Le cancellazioni sono segnalate in questo modo.
+Le aggiunte sono segnalate in questo modo.
 Linea 9:
+ * Previous session at [[  |UDS-O]]
-Linea 22:
+Linea 23:
-== Lucid Cycle Data ==
 * Tests [[https://wiki.ubuntu.com/Testing/Laptop/Lucid/Reports|here]]
||<tablestyle="font-size: 0,9em; text-align: center; border: none; width: 50%; font-weight: bold; background-color: #f1f1ed; table-layout: fixed;"> Milestone ||	People ||	Test cases performed ||
||< : #FFFFFF> 10.04 A3 ||< : #FFFFFF> 15  ||< : #FFFFFF>  167   ||
||< : #FFFFFF> 10.04 Beta 1 ||< : #FFFFFF> 16 ||< : #FFFFFF> 211  ||
||< : #FFFFFF> 10.04 Beta 2 ||< : #FFFFFF> 12 ||< : #FFFFFF> 171  ||
||< : #FFFFFF> 10.04 RC ||< : #FFFFFF> 9 ||< : #FFFFFF>  145  ||
||< : #FFFFFF> 10.04 Final ||< : #FFFFFF> 7 ||< : #FFFFFF>  117  ||
-Linea 31:
+Linea 24:
-== Maverick Cycle Data ==
 * Tests [[https://wiki.ubuntu.com/Testing/Laptop/Maverick/Reports|here]]
||<tablestyle="font-size: 0,9em; text-align: center; border: none; width: 50%; font-weight: bold; background-color: #f1f1ed; table-layout: fixed;"> Milestone ||	People ||	Test cases performed ||
||< : #FFFFFF> 10.10 A1 ||< : #FFFFFF> 4 ||< : #FFFFFF>  49  ||
||< : #FFFFFF> 10.10 A2 ||< : #FFFFFF> 4 ||< : #FFFFFF> 88 ||
||< : #FFFFFF> 10.10 A3 ||< : #FFFFFF> 2  ||< : #FFFFFF>  48   (Holidays) ||
||< : #FFFFFF> 10.10 Beta ||< : #FFFFFF> 7 ||< : #FFFFFF> 167  ||
||< : #FFFFFF> 10.10 RC ||< : #FFFFFF> 5 ||< : #FFFFFF>  81  ||
||< : #FFFFFF> 10.10 Final ||< : #FFFFFF> 5 ||< : #FFFFFF>  54  ||
-Linea 47:
+Linea 30:
-||< : #FFFFFF> 11.04 A1 ||< : #FFFFFF> 4 ||< : #FFFFFF>  120  ||
||< : #FFFFFF> 11.04 A2 ||< : #FFFFFF> 7 ||< : #FFFFFF> 164 ||
||< : #FFFFFF> 11.04 A3 ||< : #FFFFFF> 5  ||< : #FFFFFF>  86   ||
||< : #FFFFFF> 11.04 Beta 1 ||< : #FFFFFF> 19 ||< : #FFFFFF> 338  ||
||< : #FFFFFF> 11.04 Beta 2 ||< : #FFFFFF> 17 ||< : #FFFFFF> 366  ||
||< : #FFFFFF> 11.04 Final ||< : #FFFFFF> 11 ||< : #FFFFFF>  201 ||
+||< : #FFFFFF> 11.04 Beta 1 ||< : #FFFFFF> 19 ||< : #FFFFFF> 174   ||
||< : #FFFFFF> 11.04 Beta 2 ||< : #FFFFFF> 17 ||< : #FFFFFF> 191   ||
||< : #FFFFFF> 11.04 Final ||< : #FFFFFF> 11 ||< : #FFFFFF>  236  ||
-Linea 54:
+Linea 34:
-* N.B. testcases here stand for "case-id" to compare them with wiki data  (e.g. tracker has 18 testcases but most of them contain more than one case-id)
-Linea 58:
+Linea 37:
+laptop tested at least one milestone
-Linea 60:
+Linea 40:
-||< : #FFFFFF> 11.04 A1 ||< : #FFFFFF> 6 ||< : #FFFFFF>  120  ||
||< : #FFFFFF> 11.04 A2 ||< : #FFFFFF> 3 ||< : #FFFFFF> 164 ||
||< : #FFFFFF> 11.04 A3 ||< : #FFFFFF> 2  ||< : #FFFFFF>  86   ||
||< : #FFFFFF> 11.04 Beta 1 ||< : #FFFFFF> 15 ||< : #FFFFFF> 338  ||
||< : #FFFFFF> 11.04 Beta 2 ||< : #FFFFFF> 11 ||< : #FFFFFF> 366  ||
||< : #FFFFFF> 11.04 Final ||< : #FFFFFF> 4 ||< : #FFFFFF>  201 ||
+||< : #FFFFFF> 11.04 A1 ||< : #FFFFFF> 6 ||< : #FFFFFF>  54  ||
||< : #FFFFFF> 11.04 A2 ||< : #FFFFFF> 4 ||< : #FFFFFF> 56 ||
||< : #FFFFFF> 11.04 A3 ||< : #FFFFFF> 2  ||< : #FFFFFF>  28   ||
||< : #FFFFFF> 11.04 Beta 1 ||< : #FFFFFF> 15 ||< : #FFFFFF> 174  ||
||< : #FFFFFF> 11.04 Beta 2 ||< : #FFFFFF> 11 ||< : #FFFFFF> 102  ||
||< : #FFFFFF> 11.04 Final ||< : #FFFFFF> 7 ||< : #FFFFFF>  58 ||
-Linea 68:
+Linea 48:
+=== Failures summary ===
-Linea 69:
+Linea 50:
-Mixed reports, filed using wiki until Alpha 3, with Laptop Tracker from Beta 1.
+bugs have been reported with 9 test cases failed:
-Linea 71:
+Linea 52:
- Laptop tested (Natty only, the tracker has also 7 for Maverick and 4 for Lucid).
  * 8+33 = 41   Wiki method + Laptop tracker
+||<tablestyle="font-size: 0,9em; text-align: center; border: none; width: 50%; font-weight: bold; background-color: #f1f1ed; table-layout: fixed;"> Testcase || Total/Failed || Bugs  ||
<<BR>>
'''System'''
||<tablestyle="font-size: 0,9em;  border: none; text-align: center; width: 50%;font-weight: bold; table-layout: fixed;"> Suspend/Resume||<style="color:red"> 25/13 ||<style="color:red"> 10  ||
||Memory Usage|| 27/1 || 1  ||
||Internal Data Card Reader ||<style="color:gold"> 25/3 ||<style="color:gold"> 6 ||
||Builtin Webcam || 23/1 || 4 ||
<<BR>>
'''Input/Output'''
||<tablestyle="font-size: 0,9em;  border: none; text-align: center; width: 50%;font-weight: bold; table-layout: fixed;"> Builtin Keyboard|| 30/2 || 3 ||
||Hotkeys||<style="color:red"> 20/10 ||<style="color:red"> 9 ||
||Touchpad|| 30/1 || 1 ||
||USB Mouse|| 21 || 1 ||
<<BR>>
'''Audio'''
||<tablestyle="font-size: 0,9em;  border: none; text-align: center; width: 50%; font-weight: bold;table-layout: fixed;"> Internal Speakers  || 36/1 || 2 ||
||Headphones   || 31/1 || 5 ||
||Internal Microphone   || 33/1 || 1 ||
||External Microphone|| 28/1 || 1 ||
<<BR>>
'''Video'''
||<tablestyle="font-size: 0,9em;  border: none; text-align: center; width: 50%; font-weight: bold;table-layout: fixed;"> Builtin Screen||<style="color:gold"> 32/4 ||<style="color:gold"> 4 ||
||External Screen||<style="color:gold"> 10/3 ||<style="color:gold"> 6 ||
<<BR>>
'''Networking'''
||<tablestyle="font-size: 0,9em;  border: none; text-align: center; width: 50%; font-weight: bold;table-layout: fixed;"> Cabled|| 25 || 1 ||
||Wireless|| 30/1 || 2 ||
<<BR>>
'''External Expansion Port'''
||<tablestyle="font-size: 0,9em;  border: none; text-align: center; width: 50%;font-weight: bold; table-layout: fixed;"> USB Ports             ||<style="color:gold"> 27/3 ||<style="color:gold"> 5 ||
||Firewire Ports|| || ||

<<BR>>
-Linea 75:
+Linea 87:
- How many people filed a report (dev and stable releases):
  * 49 people out of 73 members of ubuntu-laptop-testing team in Launchpad.
+=== Bug Tasks Importance ===
-Linea 78:
+Linea 89:
- How results have been reported (natty):
 
         * 4 reports with test results marked as failed but bugs not reported to Launchpad (all using wiki method)
  * 45 ok
+||<tablestyle="font-size: 0,9em;  border: none; text-align: left;font-weight: bold; table-layout: fixed;"> * Critical  	:|| 2  (Fix Released)||
|| * High      	:|| 1  (Fix Released)||
|| * Medium    	:|| 1  (Incomplete)||
|| * Low       	:|| 6  (1 Fix released, 2 in progress, 1 Triaged, 2 Confirmed)  ||
|| * Undecided 	:|| 46 (5 Fix Released, 3 Triaged, 13 Confirmed, 5 Incomplete, 11 New,  5 Invalid, 3 Expired, 1 Won't Fix)||

P.S.: About half of bugs have been reported against "linux" package
-Linea 86:
+Linea 100:
-{{attachment:laptop-tests-graph.png}}
+{{attachment:laptop-graph.png}}
-Linea 88:
+Linea 102:
-##<<Immagine(/PaoloSammicheli/LaptopTesting-O/tests-graph.png,850,center)>>
+##<<Immagine(/SergioZanchetta/LaptopTesting-O/laptop-tests-graph.png,850,center)>>
-Linea 92:
+Linea 106:
- * [[https://blueprints.launchpad.net/ubuntu/+spec/ubuntutheproject-qa-n-laptop-testing-tracker| Blueprint on Launchpad]]
+ * [[ | Blueprint on Launchpad]]
-Linea 95:
+Linea 109:
-== How it works ==

 * Profile setup

  {{attachment:laptop-info550.png}}

<<BR>>

 * Main page

  {{attachment:releases450.png}}

<<BR>>

 * [[http://laptop.qa.ubuntu.com/qatracker/test/5759|Testcases]]

  {{attachment:testcase450.png}}

<<BR>>

 * [[http://laptop.qa.ubuntu.com/qatracker/laptop/Natty/28712|Report example]]

  {{attachment:laptop-report.png}}

<<BR>>

 * [[http://laptop.qa.ubuntu.com/qatracker/laptop|Laptops summary]]

  {{attachment:laptop-summary550.png}}
-Linea 130:
+Linea 114:
- * Sensible increase in partecipants/tests performed.
 * A consistent drop of errors in filing a report, using wiki the most common were:
  1. Various kind of typos, in some cases lack of important info,in others lots of unnecessary/unstructured or not properly formatted one etc...
  1. Tests marked as "Failed" without opening a bug or linking the bug on the page.
  1. Laptop report done not added in the Reports summary page.

== Achievements ==
  1. Much more easy and quick for a user to report a results.
  1. Reports are structured and data validated and can be aggregated or filtered.
  1. Testcases are grouped by category (Audio, Video, I/O etc.)
  1. Users can be easily contacted via tracker/launchpad and milestone and can be notificated by email when a new milestone is available.
+ * About the same number of tests performed in Beta 1, slight drop in Beta 2.
 * People need time to test their laptop (see the increment in Natty Final)
 * Critical areas for laptops (most bugs/test failures) are Suspend/Resume and Hotkeys.
 * Internal Data Card Reader, Video (internal and external) need attention too.
-Linea 145:
+Linea 122:
-. Extend the actual iso-testing tag system implementing a tag scheme specific for laptop (Audio-Video-I/O etc)
  2. Add more testcases.
  3. Try to automate the testing process using tools like [[https://wiki.ubuntu.com/Kernel/kteam-tools |kernel-qa]] script (using USB live disks for example).
  4. Patches are welcome! :-)
+  * Try to automate the testing process as much as possible with checkbox. It can be patched to detect a dev release, so it can be used to test/report a bug about a laptop for the tracker without sending reports for the UF project (that it's used to validate hardware for stable releases)
  * Integrate in some way LTT/UF projects?