Description and Analysis of survey

To help us understand and project the needs of scholarly users, the WWP surveyed a group of scholars and librarians using a survey instrument designed by Daniel Odess, Karen Murphy, and Catharine Hall (all from Brown University), with consultation from Professor Carole Palmer of the University of Illinois. The survey asked 38 questions about the respondents' use of and attitudes towards electronic texts, the cost and extent of their research-related travel, and their use of primary sources materials in research and teaching. We subjected the results of this survey to both qualitative and quantitative analysis, described below.

Survey Design and Aims

The survey was sent to 330 people, all of whom were prior users of WWP text printouts. This group was chosen not at random, but with the aim of representing a range of geographic locations, degrees of access to online technology, and professional positions--the latter largely academic, but also including a sample of librarians and independent scholars. Our response rate was 21% (69 responses).

Our aims for the survey were two-fold. First, we wanted to judge the practical economic impact of electronic texts on academic research, and test hypotheses about the long-term costs of using rare texts in research and teaching, as compared to the use of primary source textbases such as the WWP's. To this end, we wanted to collect some concrete data on the kinds of costs researchers incur when using rare texts, and the sources of funding that typically support such research.

Our second, broader aim was to understand the range of attitudes and concerns that academics currently feel towards the use of electronic resources, as compared with other forms of textual material. Such knowledge, we felt, could provide important insight both into the way electronic resources are likely to be adopted in research and teaching, and into the needs that such resources must serve in order to become a useful and habitual part of scholarly life.

To elicit the more specific data on costs, we designed a set of detailed questions with multiple-choice answers to limit variation and ensure consistency among responses. For the broader questions about attitudes and text usage, however, our approach was necessarily more complex. We wanted to guide or limit the responses as little as possible, to capture the full range of respondents' opinions. For this reason, these sections of the survey asked for open-ended comments, inviting the respondents to speak as frankly and fully as possible.

As we discovered, this approach to survey design has advantages and disadvantages. The advantages, as stated above, are that one does not artificially limit what can be said in response to the survey by anticipating a certain range of responses and foreclosing others. However, open-ended questions make it more difficult to derive quantifiable conclusions from the data. To overcome this problem, we created a set of codes that represented the specific themes and issues that we wanted to track in the responses. Each response could then be coded to indicate which themes were present, thus providing a way of identifying patterns and expressing conclusions more precisely.

Qualitative Survey Results

The Women Writers Project has already reported on the qualitative analysis of the survey results in "Scholarly Habits and Digital Resources: Observations from a User Survey," presented at the Digital Resources in the Humanities conference in September 1998 and published online here. A copy was sent to the Mellon Foundation shortly thereafter, and an excerpt with our findings is also included for convenience as an appendix to this report (see Appendix B). To sum up here very briefly, our conclusions from this analysis were as follows:

1. Attitudes towards online resources were generally positive or at least curious; only a small minority of respondents expressed negative feelings about digital materials or indicated that they would probably not adopt them in their own work. Enthusiasm for digital materials in teaching was evident even for those who were hesitant about using them for personal research. In particular, respondents cited increased functionality (searching, textual analysis, ability to manipulate the text) and increased access as strong advantages of digital materials and research tools.

2. Respondents voiced a number of specific concerns about digital resources (even respondents who reported strong interest in using them) which clearly must be addressed by digital resource providers. These included:

• concerns about the accuracy and reliability of the text, and the integrity of the editorial treatment;
• concerns about the loss of the physical book;
• concerns about technical obstacles or inconveniences surrounding the use of digital resources (e.g. network failure, installation difficulties, ergonomic issues, software incompatibilities);
• concerns about the role of the scholar in producing electronic resources, and how that role is evolving.

3. The current cost of research using rare primary source texts, even when very roughly estimated, is substantial. Nearly half of the respondents reported annual travel costs of $1000 or more, and the average amount reported by those who responded to this question was nearly $1400. Moreover, many respondents indicated that the travel they were able to afford was insufficient for their real research needs, and that their research was hampered by lack of access to rare materials. Although access to digital resources does not altogether obviate the need (or the desire) to consult original materials, it can make this consultation more efficient and productive. It may also render some kinds of research travel unnecessary, allowing available resources to go farther. Many scholars now spend days simply transcribing or reading texts that they have traveled to see, painstaking tasks which limit drastically the number of texts they are able to consult. With prior access to an electronic source, the scholar can perform these preliminary tasks at home and then use precious research time with the original to investigate questions which only the original object can answer (such as watermarks, details of binding, etc.).

Quantitative analysis

The quantitative analysis of the survey results was performed by Professor Walter Freiberger of Brown University with assistance from Vanja Ducik (also at Brown). Our quantitative analysis of the survey results revealed several additional points of interest by allowing us to look at the association between the replies to various survey questions, and between these replies and relevant demographic data.

Method

The most important measure of association between sets of data is the correlation coefficient. If its value is zero, there is no association (or correlation); if it is +1, there is perfect positive, if -1 perfect negative correlation. For each pair of replies, we tested whether the correlation was significantly different from zero (using a p-value of .10 at the cut-off). The pairs for which there was significant correlation are listed and discussed below.

We used the two most commonly used correlation coefficients for our analysis: Pearson's product-moment correlation coefficient r and Kendall's rank correlation coefficient tau. The latter is less sensitive to outliers and to assumptions about the underlying distributions. The results of the two analyses were quite consistent (i.e. gave significant p-values for almost identical pairs of data). We also computed Cramér's coefficient of association (a normalized chi-square) from a cross tabulation of the replies, again with consistent results.

Results

Rank in profession correlates with positive feelings about increased access (p= .022, r=.28), and with negative feelings about technical obstacles (p= .022, r= .28). It correlates negatively with general negative feelings (p= .066, r= -.22), and with use in classrooms (p= .06, r= -.23). That is, more senior faculty are not as likely to use etexts in their teaching, but these faculty are more likely to have positive feelings in general about electronic texts and about increased access to primary source materials in particular. They are, however, more likely to be apprehensive about possible technological problems.

People currently using digital materials are of course more likely to plan to use them in the future, and this was confirmed in our results regardless of whether the current use was teaching or research. Current research use was also associated with increased negative feelings or concerns about ergonomic issues (p= .098, r= .2), which makes sense, given likely first-hand experience with these problems, and with positive feelings about increased access to source materials (p= .09, r= .205) and increased functionality (p= .0001, r=.46). It is significant for us that these are the two areas which seem to emerge most strongly for current users, since these are our two primary goals for the RWO/WWO resource.

Future plans to use digital resources in teaching were correlated with a number of cost concerns: with high travel costs (p= .098, r= .2) and with a sense that research had been hampered by lack of access to rare texts (p= .042, r= .26). These future plans were also correlated with positive feelings about the possibility that digital materials would reduce the cost of access to rare texts (p= .024, r= .247).

Future plans to use digital materials in research were correlated with both positive and negative feelings about these resources. Specific concerns which correlated concerned the lack of scholarly apparatus (p= .08, r= .2); while on the positive side these respondents (like the current users) looked forward to increased access (p= .005, r= .34) and functionality (p= .0, r= .47).

Although these results are locally suggestive, it is difficult to generalize from them to draw larger conclusions beyond what the qualitative analysis yielded. This could be partly the fault of the survey design, which focused more on eliciting nuanced statements about attitudes and preferences; however, it could also be that at this stage clear patterns have not yet emerged.

Next to Academic Research and Electronic Publication

Back to Early User Response