Report on: "Managing Statistics for Electronic Resources" -Presenters- Brinley Franklin, Joe Zucca, & Jennifer Weintraub Homer Babbidge Library, Class of 1947 Conference Room, University of Connecticut, Storrs, CT by Julie Linden, Yale University View the program agenda and biographies. [PDF File]
May 17, 2002

The ARL E-Metrics Project: Measures for Electronic Resources
Presented by:  Brinley Franklin
Director, University of Connecticut Libraries

By 1999, ARL libraries had gone "over the cliff" in spending on networked resources -- taken together, the ARL libraries were spending more than $100 million annually. ARL launched the E-Metrics project in an effort to measure usage of electronic resources. The ultimate goal is to measure outcomes, but the participating libraries first needed to be able to satisfactorily measure inputs and outputs. Challenges in measuring electronic usage include the fact that vendors supply the usage data, different vendors count different data in different ways, and clear and consistent definitions of data elements don't exist.

The first phase of the project surveyed the 24 participating ARL libraries to determine the state of the art for measuring electronic resources and services. A Working Group on Database Vendor Statistics met with 11 major vendors to work on standardization of usage data. Vendors have a vested interest in working with the ARL group rather than trying to respond to disparate requests from individual libraries, so that data is standardized in a way that will satisfy a wide customer base. Phase Two defined and tested 18 data elements, including input measures (such as number of full-text electronic journal subscriptions, technical infrastructure costs, etc.) and output measures (such as usage of digital collections, percentage of digitally based reference transactions, etc.). The next phase will involve more data-gathering and a shift to focus on outcomes.

View Brinley Franklin's MS Powerpoint presentation: The ARL E-Metrics Project

Through the Bytes Darkly: Management Information and the Digital Library
Presented by:  Joe Zucca
Assessment, Planning and Publications Librarian, University of Pennsylvania

Joe described technology as "hostile to measurement" for much the same reasons Brinley Franklin described -- lack of measurement standards or common metrics. He described Penn's effort to move from low-resolution usage data (such as expenditures, number of electronic resource subscriptions, server loads) to high-resolution data. In order to increase data resolution and achieve common metrics for database usage, Penn has established a redirect from the database link on its Web site to capture the information on the library's Web server log. This enables standardized measurement of number of logins per database, time of access, place of access (libraries, dorms, off-campus, etc.) and "communities" of access (as defined by campus sub-domains). The data can be used to determine cost per login for databases and electronic journals and allows the library to demonstrate accountability in its expenditures. The redirector that captures login information was written in-house at Penn, and Joe indicated that Penn is willing to share it with other libraries.

Joe led a Web tour of the Penn Library's Data Farm (http://metrics.library.upenn.edu/prototype/datafarm/), which is a repository of library statistics on electronic and physical library use. It is a space to "assemble, process, integrate, analyze, and disseminate data" to support Penn's goal of strategic planning based on empirical evidence. Joe discussed why the data are important and summed it up with a quote from Casey Stengel: "If you don't know where you're going, you'll probably end up somewhere else."

View Joe Zucca's MS Powerpoint presentation: Through the Bytes Darkly: Management Information and the Digital Library

Usage Statistics for Yale's Digital Collections
Presented by:  Jennifer Weintraub
Digital Collections Specialist, Yale University Library

Jennifer focused on usage of Yale's digital collections, which include locally loaded full-text databases, the Curiae database of Supreme Court records and briefs, and image databases. Collection of usage statistics for these collections is neither centralized nor standardized. For example, the Curiae database is unique in using Nedstat Basic, a free Web utility, to track page views and access location; it does not provide information about how users are searching the database.

Usage statistics are not gathered at all for some digital collections. For instance, the Visual Resources Collection, a digital image library tailored for Art History and American Studies courses, is a faculty-driven project with a specifically defined user base (students enrolled in the courses using the VRC); thus far, the library has relied on faculty feedback to assess the project rather than usage statistics. Quantitative statistics are not vital to the assessment of every digital collection; rather, the goals of each project drive whether and what statistics are gathered. Jennifer is encouraging librarians creating new digital collections to consider what usage statistics might be important for assessment as well as other qualitative measures.

View Jennifer Weintraub's MS Powerpoint presentation: Usage Statistics for Yale's Digital Collections

Report by:
Julie Linden
Data and Electronic Services Librarian and Librarian for Sociology and Statistics,
Social Science Libraries and Information Services,
Yale University
New Haven, CT
julie.linden@yale.edu
Comments Welcome!