User Testing of an SFX Menu Redesign
Bennett Ponsford
Digital Services Librarian
Texas A&M University Libraries
The existing Services Menu design dated from 2005 - a very traditional out-of-the-box SFX menu. Were unhappy with design and wanted/needed/planned a change. Data showed that 87% of Services Menu use was confined to full text linking. When no full text available, only 30% tried something else - usually ILL.
Bennett Ponsford
Digital Services Librarian
Texas A&M University Libraries
The existing Services Menu design dated from 2005 - a very traditional out-of-the-box SFX menu. Were unhappy with design and wanted/needed/planned a change. Data showed that 87% of Services Menu use was confined to full text linking. When no full text available, only 30% tried something else - usually ILL.
The testing process:
21 volunteers - 2 faculty, 1 researcher, 13 graduate students, 5 professional students (Medicine, Law, etc). Volunteers were from all departments. Were asked to find 3 articles - 2 chosen for them and a citation provided and 1 citation they brought themselves. The provided citations were targeted to the volunteers' research areas. The library was also interested in the volunteers' evaluations of bX. The library would have liked to also sample undergraduate experiences but this wasn't possible for this round since the testing took place over the summer.
Volunteers were given a choice to search by journal title (A-Z list) or article citation (Citation Linker). 12 used journal title search; 8 used Citation Linker; 1 used Google Scholar. When asked, those not using Citation Linker said they were concerned about making typos during data entry and that completing so many fields takes too much time. Most searches of the journal title (89%) were successful, where only 20% of the Citation Linker searches were successful. Errors in the journal title search were caused by typos, by entering the article title instead of the journal title, or sometimes by the A-Z AJAX autocomplete journal titles. For the last cause, the response time of the AJAX service was too slow to keep up with volunteers quickly typing the journal titles they know really well. Most of the unsuccessful Citation Linker searches were caused by entering the article author and article title rather than the journal title. Volunteers reported that they almost never had ISSNs to enter into the form, although many of them were either familiar with or heavy users of DOIs and/or PMIDs. They reported that they wanted to enter as little information as possible in order to get to the article.
Bennett reported that the results confirmed something she has suspected for a while - to today's students, the concept of a "journal" as a container for articles is foreign. Most have been searching electronic databases all their educational lives so they don't understand using a journal title to locate something. Their favorite tools, including Google Scholar and PubMed, don't work this way - why should the library?
Results:
Their default search will remain the journal title (A-Z) search. They are keeping the AJAX autocomplete because it seemed to help the volunteers who knew to wait for it. They are now downplaying the Citation Linker although they are keeping it available and made a few changes to the search form:
--removed fields for article title and article author (less to complete)
--removed starts with/contains choices for journal title field, since having the drop-down between the field label and the field seemed to confuse people. And no one ever used it, anyway.
First draft of a redesigned services menu:
Added a "Journal at a glance" window that screenscrapes Ulrich's to indicate information like "peer reviewed," "journal," etc. directly to the services menu. In subsequent testing, they found that their volunteers didn't see it. The faculty and doctoral candidates already knew this information about the journals important to them, although the ones who taught did say that it could be useful for undergraduates. New graduate students used it.
Pared down link choices to full text, catalog search, ILL, bX, and Google Scholar. They loaded print journal ISSNs from their catalogs (3 of them!) into SFX as object portfolios so the catalog search link would not appear if the journal wasn't present in the catalog. Bennett indicated that this has been helpful for users but is a lot of work to maintain. Removed "report a problem" link because they found that when users had problems they thought it was their own fault, so no one used the link. Since they have document delivery of owned items, most users who encountered problems with full text linking just requested the item through ILL.
bX evaluation:
For the most part, volunteers understood how the links worked but didn't understand where the recommendations were coming from. Many assumed that the recommendations were caused by keyword matching or were citation based (especially those used to Web of Science). When it was explained that the recommendations were like Amazon (i.e.: usage-generated), the volunteers got it. Although the library is unhappy with the default bX header - "Users who were interested in this item also expressed interest in the following items" (or something similar) - they did not change it because they couldn't think of anything better to use.
In terms of evaluating the quality of the recommendations, most users were satisfied. Sometimes they recognized the title or author of a recommendation and approved. Some quibbled with the recommendations being based on human usage, believing that machine-generated recommendations would be more reliable. Some faculty said that students didn't need more recommendations - they needed help focusing on the resources they already had.
Direct Linking and Direct Link Banner evaluation:
Most volunteers had trouble seeing the banner on top of a results screen in a full text target. Bennett theorizes this is because we're so used to tuning out web banners. Although students liked it when it worked, it didn't work often enough to justify keeping it in service. They are exploring some possible improvements so someday may revisit the decision to remove direct linking.
21 volunteers - 2 faculty, 1 researcher, 13 graduate students, 5 professional students (Medicine, Law, etc). Volunteers were from all departments. Were asked to find 3 articles - 2 chosen for them and a citation provided and 1 citation they brought themselves. The provided citations were targeted to the volunteers' research areas. The library was also interested in the volunteers' evaluations of bX. The library would have liked to also sample undergraduate experiences but this wasn't possible for this round since the testing took place over the summer.
Volunteers were given a choice to search by journal title (A-Z list) or article citation (Citation Linker). 12 used journal title search; 8 used Citation Linker; 1 used Google Scholar. When asked, those not using Citation Linker said they were concerned about making typos during data entry and that completing so many fields takes too much time. Most searches of the journal title (89%) were successful, where only 20% of the Citation Linker searches were successful. Errors in the journal title search were caused by typos, by entering the article title instead of the journal title, or sometimes by the A-Z AJAX autocomplete journal titles. For the last cause, the response time of the AJAX service was too slow to keep up with volunteers quickly typing the journal titles they know really well. Most of the unsuccessful Citation Linker searches were caused by entering the article author and article title rather than the journal title. Volunteers reported that they almost never had ISSNs to enter into the form, although many of them were either familiar with or heavy users of DOIs and/or PMIDs. They reported that they wanted to enter as little information as possible in order to get to the article.
Bennett reported that the results confirmed something she has suspected for a while - to today's students, the concept of a "journal" as a container for articles is foreign. Most have been searching electronic databases all their educational lives so they don't understand using a journal title to locate something. Their favorite tools, including Google Scholar and PubMed, don't work this way - why should the library?
Results:
Their default search will remain the journal title (A-Z) search. They are keeping the AJAX autocomplete because it seemed to help the volunteers who knew to wait for it. They are now downplaying the Citation Linker although they are keeping it available and made a few changes to the search form:
--removed fields for article title and article author (less to complete)
--removed starts with/contains choices for journal title field, since having the drop-down between the field label and the field seemed to confuse people. And no one ever used it, anyway.
First draft of a redesigned services menu:
Added a "Journal at a glance" window that screenscrapes Ulrich's to indicate information like "peer reviewed," "journal," etc. directly to the services menu. In subsequent testing, they found that their volunteers didn't see it. The faculty and doctoral candidates already knew this information about the journals important to them, although the ones who taught did say that it could be useful for undergraduates. New graduate students used it.
Pared down link choices to full text, catalog search, ILL, bX, and Google Scholar. They loaded print journal ISSNs from their catalogs (3 of them!) into SFX as object portfolios so the catalog search link would not appear if the journal wasn't present in the catalog. Bennett indicated that this has been helpful for users but is a lot of work to maintain. Removed "report a problem" link because they found that when users had problems they thought it was their own fault, so no one used the link. Since they have document delivery of owned items, most users who encountered problems with full text linking just requested the item through ILL.
bX evaluation:
For the most part, volunteers understood how the links worked but didn't understand where the recommendations were coming from. Many assumed that the recommendations were caused by keyword matching or were citation based (especially those used to Web of Science). When it was explained that the recommendations were like Amazon (i.e.: usage-generated), the volunteers got it. Although the library is unhappy with the default bX header - "Users who were interested in this item also expressed interest in the following items" (or something similar) - they did not change it because they couldn't think of anything better to use.
In terms of evaluating the quality of the recommendations, most users were satisfied. Sometimes they recognized the title or author of a recommendation and approved. Some quibbled with the recommendations being based on human usage, believing that machine-generated recommendations would be more reliable. Some faculty said that students didn't need more recommendations - they needed help focusing on the resources they already had.
Direct Linking and Direct Link Banner evaluation:
Most volunteers had trouble seeing the banner on top of a results screen in a full text target. Bennett theorizes this is because we're so used to tuning out web banners. Although students liked it when it worked, it didn't work often enough to justify keeping it in service. They are exploring some possible improvements so someday may revisit the decision to remove direct linking.
Comments