Wiki Ubuntu-it

Indice
Partecipa
FAQ
Wiki Blog
------------------
Ubuntu-it.org
Forum
Chiedi
Chat
Cerca
Planet
  • Pagina non alterabile
  • Informazioni
  • Allegati
  • Differenze per "SergioZanchetta/LaptopTesting-P"
Differenze tra le versioni 1 e 34 (in 33 versioni)
Versione 1 del 19/10/2011 10.01.07
Dimensione: 7031
Commento:
Versione 34 del 03/11/2011 17.22.20
Dimensione: 6404
Commento:
Le cancellazioni sono segnalate in questo modo. Le aggiunte sono segnalate in questo modo.
Linea 9: Linea 9:
 * Previous session at [[http://wiki.ubuntu-it.org/PaoloSammicheli/LaptopTesting-O|UDS-O]]
Linea 22: Linea 23:
== Lucid Cycle Data ==
 * Tests [[https://wiki.ubuntu.com/Testing/Laptop/Lucid/Reports|here]]
||<tablestyle="font-size: 0,9em; text-align: center; border: none; width: 50%; font-weight: bold; background-color: #f1f1ed; table-layout: fixed;"> Milestone || People || Test cases performed ||
||< : #FFFFFF> 10.04 A3 ||< : #FFFFFF> 15 ||< : #FFFFFF> 167 ||
||< : #FFFFFF> 10.04 Beta 1 ||< : #FFFFFF> 16 ||< : #FFFFFF> 211 ||
||< : #FFFFFF> 10.04 Beta 2 ||< : #FFFFFF> 12 ||< : #FFFFFF> 171 ||
||< : #FFFFFF> 10.04 RC ||< : #FFFFFF> 9 ||< : #FFFFFF> 145 ||
||< : #FFFFFF> 10.04 Final ||< : #FFFFFF> 7 ||< : #FFFFFF> 117 ||
Linea 31: Linea 24:

== Maverick Cycle Data ==
 * Tests [[https://wiki.ubuntu.com/Testing/Laptop/Maverick/Reports|here]]
||<tablestyle="font-size: 0,9em; text-align: center; border: none; width: 50%; font-weight: bold; background-color: #f1f1ed; table-layout: fixed;"> Milestone || People || Test cases performed ||
||< : #FFFFFF> 10.10 A1 ||< : #FFFFFF> 4 ||< : #FFFFFF> 49 ||
||< : #FFFFFF> 10.10 A2 ||< : #FFFFFF> 4 ||< : #FFFFFF> 88 ||
||< : #FFFFFF> 10.10 A3 ||< : #FFFFFF> 2 ||< : #FFFFFF> 48 (Holidays) ||
||< : #FFFFFF> 10.10 Beta ||< : #FFFFFF> 7 ||< : #FFFFFF> 167 ||
||< : #FFFFFF> 10.10 RC ||< : #FFFFFF> 5 ||< : #FFFFFF> 81 ||
||< : #FFFFFF> 10.10 Final ||< : #FFFFFF> 5 ||< : #FFFFFF> 54 ||
Linea 44: Linea 27:
 * Tests [[https://wiki.ubuntu.com/Testing/Laptop/OldReports|here]] (Wiki)
* Tests [[http://laptop.qa.ubuntu.com/qatracker/laptop|here]] (Laptop Tracker)
 * Tests [[http://laptop.qa.ubuntu.com/qatracker/laptop|here]]
Linea 47: Linea 29:
||< : #FFFFFF> 11.04 A1 ||< : #FFFFFF> 4 ||< : #FFFFFF> 120 ||
||< : #FFFFFF> 11.04 A2 ||< : #FFFFFF> 7 ||< : #FFFFFF> 164 ||
||< : #FFFFFF> 11.04 A3 ||< : #FFFFFF> 5 ||< : #FFFFFF> 86 ||
||< : #FFFFFF> 11.04 Beta 1 ||< : #FFFFFF> 19 ||< : #FFFFFF> 338 ||
||< : #FFFFFF> 11.04 Beta 2 ||< : #FFFFFF> 17 ||< : #FFFFFF> 366 ||
||< : #FFFFFF> 11.04 Final ||< : #FFFFFF> 11 ||< : #FFFFFF> 201 ||
||< : #FFFFFF> 11.04 Beta 1 ||< : #FFFFFF> 19 ||< : #FFFFFF> 174 ||
||< : #FFFFFF> 11.04 Beta 2 ||< : #FFFFFF> 17 ||< : #FFFFFF> 191 ||
||< : #FFFFFF> 11.04 Final ||< : #FFFFFF> 11 ||< : #FFFFFF> 236 ||
Linea 54: Linea 33:
* N.B. testcases here stand for "case-id" to compare them with wiki data (e.g. tracker has 18 testcases but most of them contain more than one case-id)
Linea 58: Linea 36:
 * Tests [[http://laptop.qa.ubuntu.com/qatracker/laptop|here]] (Laptop Tracker) 29 laptop have tested at least one milestone
* Tests [[http://laptop.qa.ubuntu.com/qatracker/laptop|here]]
Linea 60: Linea 39:
||< : #FFFFFF> 11.04 A1 ||< : #FFFFFF> 6 ||< : #FFFFFF> 120 ||
||< : #FFFFFF> 11.04 A2 ||< : #FFFFFF> 3 ||< : #FFFFFF> 164 ||
||< : #FFFFFF> 11.04 A3 ||< : #FFFFFF> 2 ||< : #FFFFFF> 86 ||
||< : #FFFFFF> 11.04 Beta 1 ||< : #FFFFFF> 15 ||< : #FFFFFF> 338 ||
||< : #FFFFFF> 11.04 Beta 2 ||< : #FFFFFF> 11 ||< : #FFFFFF> 366 ||
||< : #FFFFFF> 11.04 Final ||< : #FFFFFF> 4 ||< : #FFFFFF> 201 ||
||< : #FFFFFF> 11.10 A1 ||< : #FFFFFF> 6 ||< : #FFFFFF> 54 ||
||< : #FFFFFF> 11.10 A2 ||< : #FFFFFF> 4 ||< : #FFFFFF> 56 ||
||< : #FFFFFF> 11.10 A3 ||< : #FFFFFF> 2 ||< : #FFFFFF> 28 ||
||< : #FFFFFF> 11.10 Beta 1 ||< : #FFFFFF> 15 ||< : #FFFFFF> 174 ||
||< : #FFFFFF> 11.10 Beta 2 ||< : #FFFFFF> 11 ||< : #FFFFFF> 102 ||
||< : #FFFFFF> 11.10 Final ||< : #FFFFFF> 10 ||< : #FFFFFF> 97 ||
Linea 68: Linea 47:

Mixed reports, filed using wiki until Alpha 3, with Laptop Tracker from Beta 1.

 Laptop tested (Natty only, the tracker has also 7 for Maverick and 4 for Lucid).
  * 8+33 = 41 Wiki method + Laptop tracker
=== Failures summary ===
Linea 75: Linea 50:
 How many people filed a report (dev and stable releases):
  * 49 people out of 73 members of ubuntu-laptop-testing team in Launchpad.
||<tablestyle="font-size: 0,9em; text-align: center; border: none; width: 50%; font-weight: bold; background-color: #f1f1ed; table-layout: fixed;"> Testcase || Total/Failed || Bugs ||
<<BR>>
'''System'''
||<tablestyle="font-size: 0,9em; border: none; text-align: center; width: 50%;font-weight: bold; table-layout: fixed;"> Suspend/Resume||<style="color:red"> 28/15 ||<style="color:red"> 10 ||
||Memory Usage|| 30/1 || 1 ||
||Internal Data Card Reader ||<style="color:gold"> 28/4 ||<style="color:gold"> 6 ||
||Builtin Webcam || 25/1 || 4 ||
<<BR>>
'''Input/Output'''
||<tablestyle="font-size: 0,9em; border: none; text-align: center; width: 50%;font-weight: bold; table-layout: fixed;"> Builtin Keyboard|| 32/2 || 3 ||
||Hotkeys||<style="color:red"> 20/10 ||<style="color:red"> 9 ||
||Touchpad|| 32/1 || 1 ||
||USB Mouse|| 24 || 1 ||
<<BR>>
'''Audio'''
||<tablestyle="font-size: 0,9em; border: none; text-align: center; width: 50%; font-weight: bold;table-layout: fixed;"> Internal Speakers || 38/1 || 2 ||
||Headphones || 33/1 || 5 ||
||Internal Microphone || 35/1 || 1 ||
||External Microphone|| 30/1 || 1 ||
<<BR>>
'''Video'''
||<tablestyle="font-size: 0,9em; border: none; text-align: center; width: 50%; font-weight: bold;table-layout: fixed;"> Builtin Screen||<style="color:gold"> 35/4 ||<style="color:gold"> 4 ||
||External Screen||<style="color:gold"> 11/3 ||<style="color:gold"> 6 ||
<<BR>>
'''Networking'''
||<tablestyle="font-size: 0,9em; border: none; text-align: center; width: 50%; font-weight: bold;table-layout: fixed;"> Cabled|| 28 || 1 ||
||Wireless|| 34/1 || 2 ||
<<BR>>
'''External Expansion Port'''
||<tablestyle="font-size: 0,9em; border: none; text-align: center; width: 50%;font-weight: bold; table-layout: fixed;"> USB Ports ||<style="color:gold"> 29/3 ||<style="color:gold"> 5 ||
||Firewire Ports|| || ||
Linea 78: Linea 82:
 How results have been reported (natty):
 
         * 4 reports with test results marked as failed but bugs not reported to Launchpad (all using wiki method)
  * 45 ok
<<BR>>
Linea 84: Linea 85:
= Graph (dev cycles) = === Bug Tasks Importance ===
Linea 86: Linea 87:
{{attachment:laptop-tests-graph.png}} ||<tablestyle="font-size: 0,9em; border: none; text-align: left;font-weight: bold; table-layout: fixed;"> * Critical :|| 2 (Fix Released)||
|| * High :|| 1 (Fix Released)||
|| * Medium :|| 1 (Incomplete)||
|| * Low :|| 6 (1 Fix released, 2 in progress, 1 Triaged, 2 Confirmed) ||
|| * Undecided :|| 46 (5 Fix Released, 3 Triaged, 13 Confirmed, 5 Incomplete, 11 New, 5 Invalid, 3 Expired, 1 Won't Fix)||
Linea 88: Linea 93:
##<<Immagine(/PaoloSammicheli/LaptopTesting-O/tests-graph.png,850,center)>> P.S.: About half of bugs have been reported against "linux" package


= Graph (last cycles comparation) =

{{attachment:laptop-graph-lastcycle.png}}


{{attachment:laptop-graph.png}}

##<<Immagine(/SergioZanchetta/LaptopTesting-O/laptop-tests-graph.png,850,center)>>
Linea 92: Linea 107:
 * [[https://blueprints.launchpad.net/ubuntu/+spec/ubuntutheproject-qa-n-laptop-testing-tracker| Blueprint on Launchpad]]  * [[https://blueprints.launchpad.net/ubuntu/+spec/ubuntutheproject-qa-n-laptop-testing-tracker|Blueprint on Launchpad]]
Linea 95: Linea 110:

== How it works ==

 * Profile setup

  {{attachment:laptop-info550.png}}

<<BR>>

 * Main page

  {{attachment:releases450.png}}

<<BR>>

 * [[http://laptop.qa.ubuntu.com/qatracker/test/5759|Testcases]]

  {{attachment:testcase450.png}}

<<BR>>

 * [[http://laptop.qa.ubuntu.com/qatracker/laptop/Natty/28712|Report example]]

  {{attachment:laptop-report.png}}

<<BR>>

 * [[http://laptop.qa.ubuntu.com/qatracker/laptop|Laptops summary]]

  {{attachment:laptop-summary550.png}}
Linea 130: Linea 115:
 * Sensible increase in partecipants/tests performed.
 * A consistent drop of errors in filing a report, using wiki the most common were:
  1. Various kind of typos, in some cases lack of important info,in others lots of unnecessary/unstructured or not properly formatted one etc...
  1. Tests marked as "Failed" without opening a bug or linking the bug on the page.
  1. Laptop report done not added in the Reports summary page.

== Achievements ==
  1. Much more easy and quick for a user to report a results.
  1. Reports are structured and data validated and can be aggregated or filtered.
  1. Testcases are grouped by category (Audio, Video, I/O etc.)
  1. Users can be easily contacted via tracker/launchpad and milestone and can be notificated by email when a new milestone is available.
 * About the same number of tests performed in Beta 1, slight drop in Beta 2.
 * People need time to test their laptop (see the increment in Natty Final)
 * Critical areas for laptops (most bugs/test failures) are Suspend/Resume and Hotkeys.
 * Internal Data Card Reader, Video (internal and external) need attention too.
Linea 145: Linea 123:
  1. Extend the actual iso-testing tag system implementing a tag scheme specific for laptop (Audio-Video-I/O etc)
  2. Add more testcases.
  3. Try to automate the testing process using tools like [[https://wiki.ubuntu.com/Kernel/kteam-tools |kernel-qa]] script (using USB live disks for example).
  4. Patches are welcome! :-)
  * Try to automate the testing process as much as possible with checkbox. It can be patched to detect a dev release, so it can be used to test/report a bug about a laptop for the tracker without sending reports for the UF project (that it's used to validate hardware for stable releases)
  * Integrate in some way !LaptopTestingTeam/UbuntuFriendly projects?

Information

Background

  • Previous session at UDS-O

  • Previous session at UDS-M

  • Previous session at UDS-N

  • Laptop Testing page

Summary

"Laptop Testing" is a community-driven project started with Lucid Lynx Alpha 3. Our goal is get Ubuntu working great on as many different makes and models of laptops as possible, mainly (but not only) testing point releases of the development cycle to help devs in triaging bugs. This could be done both in all the Community and LoCos, sharing hardware info about Laptops and trying to attract as many new contributors as possible.

Project has a Testing Team on launchpad. It uses the QA Mailing List and IRC channel #ubuntu-testing. In the Testing Team Wiki page this project is reported under Activities along with an English page.

In this session we would like to present goals reached and some suggestions to improve the project.

Activity

Natty Cycle Data

Milestone

People/Laptop

Test cases performed *

11.04 Beta 1

19

174

11.04 Beta 2

17

191

11.04 Final

11

236

Oneiric Cycle Data

29 laptop have tested at least one milestone

Milestone

People/Laptop

Test cases performed *

11.10 A1

6

54

11.10 A2

4

56

11.10 A3

2

28

11.10 Beta 1

15

174

11.10 Beta 2

11

102

11.10 Final

10

97

Failures summary

Testcase

Total/Failed

Bugs


System

Suspend/Resume

28/15

10

Memory Usage

30/1

1

Internal Data Card Reader

28/4

6

Builtin Webcam

25/1

4


Input/Output

Builtin Keyboard

32/2

3

Hotkeys

20/10

9

Touchpad

32/1

1

USB Mouse

24

1


Audio

Internal Speakers

38/1

2

Headphones

33/1

5

Internal Microphone

35/1

1

External Microphone

30/1

1


Video

Builtin Screen

35/4

4

External Screen

11/3

6


Networking

Cabled

28

1

Wireless

34/1

2


External Expansion Port

USB Ports

29/3

5

Firewire Ports


Bug Tasks Importance

* Critical :

2 (Fix Released)

* High :

1 (Fix Released)

* Medium :

1 (Incomplete)

* Low :

6 (1 Fix released, 2 in progress, 1 Triaged, 2 Confirmed)

* Undecided :

46 (5 Fix Released, 3 Triaged, 13 Confirmed, 5 Incomplete, 11 New, 5 Invalid, 3 Expired, 1 Won't Fix)

P.S.: About half of bugs have been reported against "linux" package

Graph (last cycles comparation)

laptop-graph-lastcycle.png

laptop-graph.png

Laptop Tracker

Thoughts

  • About the same number of tests performed in Beta 1, slight drop in Beta 2.
  • People need time to test their laptop (see the increment in Natty Final)
  • Critical areas for laptops (most bugs/test failures) are Suspend/Resume and Hotkeys.
  • Internal Data Card Reader, Video (internal and external) need attention too.

What's next

  • Try to automate the testing process as much as possible with checkbox. It can be patched to detect a dev release, so it can be used to test/report a bug about a laptop for the tracker without sending reports for the UF project (that it's used to validate hardware for stable releases)
  • Integrate in some way LaptopTestingTeam/UbuntuFriendly projects?


CategoryHomepage