BACKGROUND INFORMATION:
Research Design and Research Logistics
Remember my approach to course design. Use the background material that you feel is appropriate and create a reference system so that you can recall the information as needed. And, as always, you are encouraged to search for, use, and reference whatever other sources you feel are appropriate to complete the assignments.
REQUIRED RESOURCES
Research design
Hartley, Robert F.; Prough, George E.; & Flaschner, Alan B (1983). Essentials of Marketing Research. PennWell Books, PennWell Publishing Co.
PART I: Overview of Marketing Research
Chapter 2: The Research Process
http://sites.google.com/site/zaydeealan0/HPF-Ch02-23-30.zip
http://sites.google.com/site/zaydeealan0/HPF-Ch02-31-38.zip
http://sites.google.com/site/zaydeealan0/HPF-Ch02-39-45.zip
PART III: Performing the Research
Chapter 11: Experimentation in Market Research
http://sites.google.com/site/zaydeealan4/HPF-Ch11-299-307.zip
http://sites.google.com/site/zaydeealan4/HPF-Ch11-308-316.zip
http://sites.google.com/site/zaydeealan6/HPF-Ch11-317-32.zip
http://sites.google.com/site/zaydeealan6/HPF-Ch11-326-335.zip
http://sites.google.com/site/zaydeealan0/HPF-Ch02-31-38.zip
McGrath, Joseph E. (1981, November/December) Dilemmatics: The Study of Research Choices and Dilemmas. American Behavioral Scientist. 25(2). 179-211.
Available as a PDF file in ProQuest on July 23, 2008.
If you read nothing else on the topic, read the McGrath article, which is an excellent summary of many types of research design and categorization of almost all of the trade offs in research design.
In a rather simple and elegant fashion, McGrath (1981) highlights the essential tradeoff between the three criteria of generalizability, context sensitivity, and measurement precision. That is, choosing a method that maximizes any one of these criteria reduces the possibility of achieving the others. The two axes of the circle should be fairly self-explanatory; the four methods on the right (“universal”) are aimed mainly at more general application of their findings, while those on the left (“particular”) are aimed at getting the most possible out of specific situations. The vertical axis splits the four methods at the top, which involve more participant awareness of the research, from the four at the bottom, which largely if not completely do not require much from the participants (the two most unobtrusive don’t even involve participants at as such at all).
One thing I like about this framework is that it doesn’t fall into what I see as the error of classifying research into “quantitative” and “qualitative” categories. These terms do not refer to kinds of research but to kinds of data, with the distinction being whether the data content can appropriately be represented in numeric form or not — almost all research does (or at least appropriately could) involve data of both types. Certainly many projects are predominantly quantitative or qualitative in orientation, but this is not a feature of the design itself but rather of the data and analysis choices made by the researchers with the structure of their design.
Research design issues.ppt with voice is a short PowerPoint@ presentation on designissues
Author Unknown (2008). Research Design. PDF of PowerPoint@ Presentation – Department of Psychology – University of Western Ontario.
Available on 16 November 2009 at
http://psychology.uwo.ca/undergraduate/psych266a/lectureslides/Psych%20266%20Research%20Methods%20×3-%20web%20version.pdf
Right click on link, download, then open PowerPoint@.
Trochim, William (2007) Introduction to Research Design. The Social Research Methods Knowledge Base. Cornell University.
Available on 2010 July 26 at
http://socialresearchmethods.org/kb/desintro.htm
Kontaxakis, Evangelos (2009, Spring) Flow chart describing the research process, Center for Instruction, Research, and Technology (CIRT), Indiana State University.
Note when the design of the research occurs and all of the research activities included in research design.
Available on 2010 July 26 at
http://www1.indstate.edu/mcnairsch/summer_docs/HowToDesignResearchProjectSpring2009-Evangelos.pptx
Right click and download the file, then open the downloaded file.
Deakin (ND) Deakin-ResearchDesign.pdf. Paper deals with marketing, but the major subsections hit the main research format issues discussed by Sekaran (1992). Sekaran, U. (1992). Research methods for business: A skill building approach. John Wiley, New York.
Also available on 2012 May 29 at
http://www.deakin.edu.au/buslaw/management-marketing/research/docs/research-design.pdf
Sharda, Nalin (ND). Project and Research Methodology: Topic 2- Research Methodologies. Sahada-Project-2-ResearchV1-10.pdf
Steps in Conducting Research
Meyer, John (ND). Psychology 266a – section 001. Introduction to Industrial and Organizational Psychology. Department of Psychology. The University of Western Ontario.
Available on 2010 November 05 at
http://psychology.uwo.ca/undergraduate/psych266a/
Click on .
http://psychology.uwo.ca/undergraduate/psych266a/lectureslides.html
Click on which will open a pdf file
http://psychology.uwo.ca/undergraduate/psych266a/Fall%202007%20overheads/Psych%20266%20Research%20Methods%20-%202007%20web%20version.pdf
The above sequence of steps has been provided because clicking on the direct link to the pdf file does not work all of the time.
Trade-offs between research design and implementation of projects
NSF (1997). User-Friendly Handbook for Mixed Method Evaluations. Directorate for Education and Human Resources. Division of Research, Evaluation and Communication. National Science Foundation.
Available on 2010 January 20 at
http://www.nsf.gov/pubs/1997/nsf97153/start.htm
NSF (2002). The 2002 User-Friendly Handbook for Project Evaluations. Directorate for Education and Human Resources. Division of Research, Evaluation and Communication. National Science Foundation.
Available on 2010 January 20 at
http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_1.pdf
These NSF publications discuss the trade-offs between research design and implementation in some detail.
Research Process or Research Logistics
Field Research.ppt is a brief guide to field research planning and logistics
Here’s a good set of Research Data Management Do’s & Don’ts – Guidelines.pdf
External Validity
McGrath, Joseph E & Brinberg, David (1983, June). External Validity and the Research Process: A Comment on the Calder/Lynch Dialogue. The Journal of Consumer Research. 10(1). 115-124.
Available on 2012 May 29 in JStor via the Trident U / Touro C eLibrary
Human Subjects Protection
A brief introductory
Human Subjects Protection and Research Ethics.ppt on using human subjects
Office of Human Research Protections (2007) The website for the Office of Human Research Protections, U.S. Department of Health & Human Services.
Available on 2010 July 26 at
http://www.hhs.gov/ohrp/
The group in charge of policing the Federal standards for research participation
If you want to examine why IRB’s exist, look at the following studies
Author Unknown (2010, October 7). Anonymous-Guatamala-IRB.pdf How they see us: Guatemalans as medical guinea pigs: An American doctor involved with the Tuskegee Institute study also experimented with Guatemalans, infecting them with syphilis and gonorrhea to test new ways of preventing sexually transmitted diseases. The Week
Author Unknown (ND) Research Ethics: The Tuskegee Syphilis Study. Tuskegee University, Tuskegee, AL 36088.
The Tuskegee Syphilis Study is one of the most horrendous examples of research carried out in disregard of basic ethical principles of conduct. The publicity surrounding the study was one of the major influences leading to the codification of protection for human subjects.
Available on 2010 November 15 at
http://www.tuskegee.edu/global/story.asp?s=1207598
Format Expected in SLP1
Ternes, Reuben (2007). Writing with Statistics. The OWL (Online Writing Laboratory) at Purdue. The Purdue University. Last edited by Driscoll, Dana Lynn.
Available on 2008 January 8 at
http://owl.english.purdue.edu/owl/resource/672/01/
Useful guides for writing your CASEs and SLPs when they contain statistics.
Readings on statistical techniques to be used in SLP1
Author Unknown (ND). StatSoft Electronic textbook StatSoft. Available on 2011 January 20 at
http://www.statsoft.com/textbook
on 9 April 2010.
Using the >buttons< on the right hand side of the page, click on
,
,
Also examine
(available at
http://www.statsoft.com/textbook/stgrm.html#stepwise
Assignment
Please prepare a short paper discussing the lessons you’ve learned from the Case about the issues of research design, research practice, research logistics and research ethics. In this analysis, using the questions as titles, you should answer the following 9 questions.
1. What is the research design used by Dr. Eveland? The expectation is a short sentence describing the design followed by your defense of your position on this issue.
Eveland, JD & Bikson, TK (1988) Eveland-p354.pdf. Work Group Structures and Computer Support: A Field Experiment. Transactions on Office Information Systems. ACM (Association for Computing Machinery). 6 (4). 354-379.
It is frequently suggested that work groups that have computer technology to support activities such as text editing, data manipulation, and communication develop systematically different structures and working processes from groups that rely on more conventional technologies such as memos, phone calls, and meetings. However, cross-sectional or retrospective research designs do not allow this hypothesis to be tested with much power. This field experiment created two task forces, each composed equally of recently retired employees and employees still at work but eligible to retire. They were given the identical tasks of preparing reports for their company on retirement planning issues, but they were randomly assigned to different technology conditions. One group had full conventional office support; the other had, in addition, networked microcomputers with electronic mail and routine office software. Structured interviews were conducted four times during the year-long project; in addition, electronic mail activity was logged in the on-line group. Although both groups produced effective reports, the two differed significantly in the kind of work they produced, the group structures that emerged, and evaluations of their own performance. Although the standard group was largely dominated by the employees through the extensive reliance on informal meetings, the electronic technology used by the other task force allowed the retirees to exercise primary leverage. We conclude that use of computer support for cooperative work results in both quantitative and qualitative changes but that effective participation in such electronically supported groups requires significant investments of time and energy on the part of its members to master the technology and a relatively high level of assistance during the learning process.
Permission given by Dr. Eveland to make a copy of the article available.
In answering this question about Dr. Eveland’s study, at a minimum, I expect you to use Trochim (2007)
Trochim, William (2007). Introduction to Research Design. The Social Research Methods Knowledge Base. Cornell University.
Available on 16 November 2009 at
http://socialresearchmethods.org/kb/desintro.htm
If you have the time, you might want to include statements of how the research designfalls into the categories developed by:
McGrath, Joseph E. (1981, November/December) McGrath.pdf taken from Dilemmatics: The Study of Research Choices and Dilemmas. American Behavioral Scientist. 25(2). 179-211.
Available on 2012 May 29 in ProQuest via the Touro College eLibrary.
If you have even more time, look at the following pdf of slides covering research designissues.
Author Unknown (2008). Research Design. PowerPoint@ Presentation – Department of Psychology – University of Western Ontario.
2. After listing and distinguishing a variety of research designs, explain the advantages and limitations of each. Why did Dr. Eveland choose the research design he chose? The expectation is a short sentence describing the design followed by your defense of your position on this issue.
Research Design Advantages Disadvantages
3. What are the main phases of a research project and their normal sequence? The expectation is a list of the stages or phases followed by your defense of your position on this issue.
At a minimum, to answer the above Q, look at Gabaney (2008)
Gabany, Steve (2008). Flow chart describing the research process. Indiana State University. Gabany.pdf
4. What is the relationship between the research question and the research design? The expectation is a short sentence describing the relationship followed by your defense of your position on this issue.
The following reference, in addition, to Conference1 should be helpful in addressing the issues in the following two questions (questions 5 and 6):
Draper, Janet (2004). The relationship between research question and research design. In: Crookes,
Patrick A. and Davies, Sue eds. Research into Practice: Essential Skills for Reading and Applying Reasearch
in Nursing and Health Care. Edinburgh: Bailliere Tindall, pp. 69??”84.
For guidance
Draper, Janet (2004). The relationship between research question and research design. In: Crookes,Patrick A. and Davies, Sue eds. Research into Practice: Essential Skills for Reading and Applying Reasearchin Nursing and Health Care. Edinburgh: Bailliere Tindall, pp. 69??”84.For guidance
Available on 2012 May 29 at
http://oro.open.ac.uk/12053/4/jan’s_chapter_final_draft.pdf
5. External validity: What are the trade-offs inherent in any design of a field experiment, particularly between site, sample, and methods of data collection and analysis (research design)? The expectation is three short paragraphs describing the
(1) tradeoff between site and sample followed by your defense of your position on this issue,
(2) tradeoff between site and methods followed by your defense of your position on this issue, and
(3) tradeoff between sample and methods followed by your defense of your position on this issue.
In explaining the tradeoffs that need to be made between research design, cost, project implementability, and expected results, with regard to Dr. Eveland’s study, at a minimum, I expect you to use,
McGrath, Joseph E & Brinberg, David (1983, June). External Validity and the Research Process: A Comment on the Calder/Lynch Dialogue. The Journal of Consumer Research. 10(1). 115-124.
Available on 2012 May 29 in Science Direct via the Trident U / Touro C eLibrary.
To assist in answering Q5, note that you need to address the following:
In Dr. Eveland’s study:
1. Were the participants randomly selected?
2. What trade-offs were made in selecting the sample of participants (people about to retire and people recently retired) that both allowed a measure of generalizability to the population (people about to retire and people recently retired), and at the same time made it difficult to generalize to the workforce (as a whole).
3. Which of the findings do you consider to be still applicable or applicable outside of the strict definition of the population studied? Defend your position
6. Internal validity: What is internal validty? How does method of assignment relate to internal validty? Defend your position.
In Dr. Eveland’s study, were the participants randomly assigned to the experimental or control condition? Defend your position.
If participants were randomly assigned to the experimental or control conditions, is internal validity high, medium, or low? Defend your position.
Or, if participants were NOT randomly assigned to the experimental or control conditions, is internal validity high, medium, or low? Defend your position.
The expectation is a short sentence describing the internal validity of Dr. Eveland’s followed by your defense of your position on this issue.
7. What were the major human subject protection issues arising in Dr. Eveland’s project, including how they might have been addressed by the researchers (to the degree that you can tell from the published article)? The expectation is a short sentence describing the possible human rights violation discussed in class followed by your defense of your position on this issue.
For example, did they need to inform the potential participants that information on individuals would not be published and/or the actual purpose of the study?
In the process of answering the question, (1) describe the basic ethical principles that govern behavioral research, and some of the ethical dilemmas with which researchers are often confronted, (2) explain the basic Federal human subjects protection rules and how they are implemented, (3) explain the role of the Institutional Review Board in approving and overseeing research, and (4) fold into your analysis any other issues that you find relevant to an assessment of research logistics or ethics in this case.
Please remember that Dr. Bikson is (and was at the time) the Chair of RAND’s IRB, so she should have been expected to be familiar with IRB rules and procedures. RAND is a research institute that responds to requests for proposals and, if awarded, conducts the research. In this instance the research was paid for by an agency that was interested in learning the issues that the nearly retired and recently retired face and another agency that was interested in the impact that computers were going to have on the ways that people interact. Although the article doesn’t mention this specifically, the project was subjected to a VERY thorough RAND IRB review (with Dr. Bikson recusing herself), and all the normal rules (including Informed Consent statements) were applied. Published articles often don’t mention things like this, because it’s taken for granted that at any major research university or institution these procedures are followed. Human subject issues are generally mentioned in published research, only if there are major problems or potential difficulties that warrant going outside of the normal procedures. So please don’t go off on a tear because the article doesn’t specifically mention the IC statement or list it in an appendix. Publishing space is too precious to waste on things that are essentially common to all studies. There are in fact some fairly serious human subjects issues that were not covered or discussed (or I wouldn’t have asked you to question compliance), but the basics of IRB review and IC statements were not among them.
8. What modifications could have been made to the logistics (i.e., the management and operations) of the project to improve its quality? In the process of answering this question, explain the idea of “research logistics” and explain why it matters. Then, defend your position relative to how the quality of the project could have been improved.
Note that project was field research. Over a half a dozen organizations were approached but none committed to the research. One of the reasons that it took over a year to find an organization that could be studied, was the need to randomly assign people (the cases) to the groups. Consider alternative designs such as a quasi-experimental design where you would pick people in one building as the test group and the people in another building and the control and explain why you would have either chosen or rejected those designs. In other words, what advantages or disadvantages would you see if the researchers had created a quasi-experimental design for the research? What are the costs and advantages of doing research on another’s property?
Remember, you’ve already discussed the sample and the basic design. For this issue, concentrate on how the project was structured and how day-to-day operations were carried out. Issues such as sampling and the basic division into four groups are designissues, outside the scope of this question. Management issues are things like training procedures, personnel assignments, communications protocols, record keeping arrangements, etc. For example, if diaries about their experiences were kept by the respondents imagine what collecting that data, coding that data, and analyzing that data would entail. And with random assignment, how could the researchers prevent a person in the treatment group sharing his or her diary with a person in the control group (and vice versa).
9. What would you like to know (that the article doesn’t discuss) in order to be able to understand and assess the logistics of the project?.
There are concerns, but, due to journal article page limitations, the authors didn’t mention anything about those concerns.
Expectations
The purpose of this assignment is for you to acquire experience in critically reading, examining, and analyzing research at a high level and evaluating it. In that regard, the content of your report is to demonstrate an understanding of the assigned reading and the research. You will be accomplishing this type of work throughout the course.
In preparing CASE1, to ensure that you demonstrate your ability to evaluate research, it is expected that you have used Background1 to learn how to:
Identify the main phases of a research project and their normal sequence
Distinguish a variety of research designs, and their advantages and limitations
Match particular designs with the research questions to which they are best adapted.
Explain the idea of “research logistics” and why it matters.
Describe the tradeoffs that need to be made between research design, cost, project implementability, and expected results
Describe the basic ethical principles that govern behavioral research, and some of the ethical dilemmas with which researchers are often confronted
Explain the basic Federal human subjects protection rules and how they are implemented
Explain the role of the Institutional Review Board in approving and overseeing research.
Dr. Eveland (currently a member of the TUI faculty) and Dr. Tora Bikson of the RAND Corp. collaborated on a study of the effects of computer use on work group performance in geographically dispersed groups. Dr. Eveland and Dr. Bikson have been involved in research on Computer Supported Cooperative Work, individually and jointly, off and on, for 35 years or so.
After you’ve had a chance to consider this study carefully, and have consulted such items from the Background Information as you find useful (start with the Required Sources then dig into the remaining resources if things remain unclear or you are curious about the issues involved).
PLEASE NOTE: You can’t hurt Dr. Eveland’s feelings by being critical of his study. It was a long time back, and he knows that it could have been done better.