User testing results

LeMill was tested with 10 teachers from 5 countries. On the first day we had think aloud test with 2 teachers. Initially we planned to have 4 think aloud sessions (each 30 minutes), but the tasks took more time than we expected.

Think aloud test

Task 1

One of your colleagues told you about the LeMill website. You searched for it in Google
and reached the LeMill front page.

- Without leaving this page, think aloud about your thoughts on the front page.
- What do you think you will find behind each of these links?
  • Both teachers understood the content section and community section.
  • Both teachers did not have any idea, what could be in activities section.
  • Both teachers were thinking that tools section contains tools for creating learning materials. They expect that it has the same function as Tools menu in MS Word.
  • Both teachers understood, that it is possible to find content by browsing or by clicking on featured content.
  • It is good, that front page does not have much text. Four main sections can be easily remembered. Large titles and images are good.
  • One of the teachers probably did not notice links to join and log in. 
Task 2

You are preparing a learning material about France and need some photos for illustrations.

- Use LeMill system to find a photo from Paris.
  • Both teachers were able to find the photo by browsing the tags. It was not a very good task, because with one teacher the paris tag was already visible on the front page. 
Task 3

After finding a nice photo from Paris you would like to share some of your photos.
You were visiting Rome last year and took some photos that might be useful for
other teachers.

- Publish one photo from Rome in LeMill system. The photos are in your desktop in a folder named Rome.
  • One of the teachers completed the task without problems. She was able to choose correct language (language neutral). She said, that she has been working with Plone before and that knowledge helped here. For first time users Plone is complicated to understand.
  • The other teacher did not understand immidiately that she must join the system to add a photo. She was expecting that it is possible to copy a photo to LeMill.
  • After logging in both teachers noticed New Content and My Content portlets
  • One of the teachers was thinking that the language must be language of metadata description, not language of the content
  • One of the teachers was a little bit confused by Creative Commons license information
  • Both teachers were able to add tags
  • Cover image section in Edit Piece Properties form is confusing for the teachers 
Task 4

Besides the photos you also found a text about Paris. You know that there is a
comprehensive article about Paris in Wikipedia, but the text in LeMill does not
have any links to other sources of information.

- Edit the text and add a link that points to Wikipedia article about Paris (http://en.wikipedia.org/wiki/Paris)
  • Both teachers noticed the formatting guidelines
  • They suggested that teachers with basic computer skills are not going to use any formatting syntax. For these teachers we should have buttons [ B ] [ I ] [ U ] etc. We do not have to have WYSIWYM editor - current syntax is short enough, teachers with basic skills probably would not even notice it.
  • Both teachers had some trouble with creating the hyperlink
  • Metadata page is too long - screen forms should not be longer than 1,5 screens
  • The educational metadata was quite confusing for one of the teachers. 
Task 5.

You have found several good photos and a text about Paris. You are not going to use all that
content for your learning material now, but you are will need them in a few weeks.

- Try to mark one of the photos so that you can find it easily next time.
  • One of the teachers was able to find collections quite quickly
  • The other teacher was a little bit confused, it took time for her to find 'add to collection'

Overall conclusions

Currently the system is intuitive only for advanced users. One of the test users had difficulties because the user interface was in English.

Group walkthrough

On the second day we had group walkthrough with 10 teachers from 5 countries. Some people from CALIBRATE project also participated the discussion, but we were focusing on the opinion of teachers.

  • Is it only pictures that we can find from LeMill?
  • Activities and Tools section are not understandable without visiting the front pages. Then the teachers will notice similarity with Activities and Tools section.
  • It should be possible to browse resources by two categories, for example by subject and age group.
  • On the front page there must be link to short introduction/tutorial/walkthrough
  • Just jigsaw? Teachers need high quality activities.
  • After login teachers noticed New Content and My Content portlets
  • Some teachers suggested, that instead of "New Content" we should have "Add new content" or "Create new content"
  • Teachers do not read long texts.
  • Most of teachers do not know nothing about metadata. They agree, that metadata vocabulary should be the vocabulary that teachers use, not the vocabulary that educational technologists use.
  • Most of teachers do not know Creative Commons. After explaining the conditions they agreed that they would accept Creative Commons Attribution-ShareAlike 2.5 license.
  • Stories is not intuitive label. We should rename it to "Tips for use".
  • Most of the teachers have broadband connection at home, online editing is not a problem for them.
  • Fullscreen view is confusing, it is difficult to find way back.
  • If the material is not editable in LeMill, we should display (based on mime-type) suggestion like this "You need OpenOffice?.org to edit this material".