s519 logo S519 Evaluation of Information Systems

Assignment Description

Home | About | Course Readings | Syllabus | Handouts | Assignment

Note: I expect you to use the APA style for references.

1. Two Measurement Memos (15% each) - individual

We will be discussing, in depth, the six categories of IS success described in DeLone & McLean (1992; 2002) [system quality, information quality, system use, user satisfaction, individual impact, and organizational impact], plus usability. Choose two of the categories that represent appropriate criteria for evaluating the system you could be studying (Think ahead about the final project, for instance, or any information systems that you are familiar with).

Identify a method (e.g., survey instrument or observation) that could be used to evaluate an information system, for each category of IS success in which you're interested. Read the assigned articles for the class to understand the various categories of IS success.

For each category of IS success you selected, write a one-page single-spaced memo explaining why the proposed measure would be applicable. Consider the applicability and appropriateness of the measurement methodology which you are proposing. Provide the necessary details about how the measurement methodology will be applied. Also note any problems in applying the measurement methodology to the system in questions (i.e., cost or technological). Be prepared to present your theoretical assessment to the class at the appropriate sessions.

Each memo is due at the beginning of the class session day in which it is discussed:

The essential components of the memo include, but not limited to: Specifications:


Many resources are available for identifying measurement instruments. First, check the studies listed by DeLone and McLean (1992), in Tables 1-6. If these measures are not satisfactory, then check the ISWorld page on "Survey Instruments in IS" ( http://www.isworld.org/surveyinstruments/surveyinstruments.htm).

A traditional literature search may also yield a useful instrument. The IU Libraries provide Web access to Business Source Premier (covering the business literature), PsycInfo (covering the psychological literature), Library Literature and Information Science (covering library and information science), Inspec (Axiom) (covering the computer science and engineering literature), and Library & Information Science Abstracts (LISA).

2. Evaluation Design Proposal & Draft Measurement Instrument (15%) - group

This assignment consists of three components:

A. Evaluation design proposal memo: Write a brief memo (no more than 3 single-spaced pages) describing the manner in which your evaluation study will be conducted. First, you need to describe why you are proposing the evaluation (i.e., the purpose of the evaluation). Then, specify the participants, when any assessments will be conducted, what aspects will be measured, and how the assessments will be conducted. Justify your design with the readings in this course. Provide a brief budget based on the staff hours required to conduct the evaluation study (e.g., the number of hours for each IS and end-user staff member that will participate and rough estimate of financial resources if possible). [Note: this memo should be written for the full evaluation study, not for the pilot data collection.]
B. Instrument: Attach a measurement instrument you propose for the evaluation study. Consider modifying existing instruments as necessary to make them applicable to your case study. Prepare a copy of the instrument that could be administered in a usability test (version 1). Identify appropriate respondents for your case study (if applicable). In addition, attach a copy of the instrument that was improved based on the feedback you received from the usability test (version 2). The instrument should include a space for each participant to indicate a name (the purpose of this is to know who participate in the pilot data collection).
C. Instrument memo: In a brief memo (no more than 2 single-spaced pages) accompanying your modified instruments (versions 1 & 2), describe the modifications you made from version 1 to version 2, explain why you believe they were necessary, and describe the effects that you believe they will have on the reliability and validity of the instrument.

3. Participation in Data Collection Activities (required, but not graded)

Each class participant will be expected to assist other class members in generating realistic data for later analysis. The data will be associated with a proposed evaluation study based on one of the selected case studies. Participants may be asked to respond to questionnaires or interviews, to participate in thinking-aloud protocols while using a software package, or to participate in other data-gathering activities. Such participation will be distributed as evenly as possible between all class participants.

Some of the participation will take place in class; additional participation may be required outside of class time.

4. Data Analysis (15%) - group

Prepare and analyze the data associated with your evaluation study. The analysis may require the use of statistical analysis software or may involve qualitative methods, such as analytic induction or content analysis. Please consult with the instructor if you need further information. You should submit the raw data set and the results. The results of the analysis will vary, depending on the method selected. For example: FAQs

Q: What format should we use for raw data?
A: The format is not specified as long as the instructor can identify the data collected from all the classmates.

Q: Should we include all the analyses we ran?
A: No, please include the selected analyses which you think are important/useful.

Q: What should we include in the written text?
A: Which tests, if any, you ran and why. A summary of what you found from the data based on the chosen analyses.

Q: What's the page limit?
A: 2-3 single-spaced pages.

5. Final Report - group (Oral presentation—10%; Written document--15%)

Prepare and present orally your interpretation of the findings of your evaluation study in 15 minutes. Provide a brief overview of the rationale for the study and the design of the study. Present the results and the conclusions that you would draw from them. Address at least two questions: What worked and what didn't work when evaluating your chosen system? Describe how the evaluation might be improved.

Prepare and present a written summary of your evaluation study. The final report is similar to what you would submit to clients/supervisors/stakeholders except for the how to improve part at the end of an evaluation study. The report should not exceed 4 single-spaced pages. Here is a suggested outline:

a. Introduction/overview of the evaluation study
b. Discussion of Rationale for the evaluation including purposes
c. Discussion of Findings/Analysis
d. Discussion of Recommendations/Conclusions
e. Discussion of Improvement for the Future

It is a document that compiles what you have done for the group project, so some of them might be a bit redundant, but in order to be a complete report, you need them. Please feel free to use what you have previously submitted. No need to have an elaborate section of c. (above), but you need to mention basic information, so that people who are not familiar with what you have done for the evaluation study would understand what you did. For example, I suggest you do not include the explanation of how you came up with the instruments. The section e is about how to improve the evaluation study for the future.

For any projects, you need to follow the APA style.

7. Participation and Enthusiasm (15%)

I expect you to attend all class meetings. If you cannot attend class, you must notify the instructor in advance (preferably more than 24 hours prior to the class). Attendance will factor into your final grade. Unexcused absences will not be tolerated; numerous absences are frowned upon, and if you foresee yourself missing multiple classes, be sure to see the instructor the first day after class. Make-up work may be negotiated only in cases of documented, excused absences. If you do not come to the class on time, it will affect the participation grade.

I will also grade your participation based on your contribution to the course. Throughout the semester, you are encouraged and expected to participate fully in class discussions and activities. Readings should be completed prior to the class sessions so that you can ask questions and benefit from the class discussion.

Academic Dishonesty

There is extensive documentation and discussion of the issue of academic dishonesty here in the Indiana University "Code of Student Rights, Responsibilities and Conduct". Of particular relevance is the section on plagiarism:
3. Plagiarism A student must not adopt or reproduce ideas, words, or statements of another person without appropriate acknowledgment. A student must give credit to the originality of others and acknowledge indebtedness whenever:
1. Directly quoting another person’s actual words, whether oral or written;
2. Using another person’s ideas, opinions, or theories;
3. Paraphrasing the words, ideas, opinions, or theories of others, whether oral or written;
5. Offering materials assembled or collected by others in the form of projects or collections without acknowledgment

Indiana University and the School of Library and Information Science policies on academic dishonesty will be followed. Students found to be engaging in plagiarism, cheating, and other types of dishonesty could receive an F for the course. As a rule of thumb, when in doubt, cite the source!

Home | About | Course Readings | Syllabus | Handouts | Assignment