Data Collection: Quantitative Measurement
T. Bevins
3/14/99

Main goals for quantitative data collection:

1. measurement - quantifying information.  Link the researcher's abstractions or theoretical concepts to concrete variables that can be empirically or objectively examined.
2. systematic collection - meaning that all data is collected in a uniform way
3. objectivity - meaning that the data is not biased.  Remove the influence of the researcher from data collection.

1. identifiable
2. repeatable, collected with consistency, and systematically recorded
3. standardized
4. checked and controlled
5. define the major variables
6. appropriate to the problem, the hypothesis, the setting, and the population
7. consistent with the study's objectives, concepts and theories

• Operationalization: translating concepts into observable and measurable phenomena

• An operational definition translates the conceptual definition into behaviors or verbalizations that can be measured.
• Validity - (a general definition) the strength of the relationship between an indicator and an underlying concept

• Reliability - (a general definition) consistency of a measure

(more about reliability and validity later)

Levels of measurement:

 First consider these terms:

discrete variable: a variable with a finite number of distinct values (e.g.. gender)
continuous variable: a variable with an infinite number of values (e.g.. height)

1. nominal - assign mutually exclusive values to levels of this variable.  Values are not ordered.  Values are discrete.  This is a way of assigning a name to each level of the variable but the level of the variable does not imply any order.  Gender is an example where you could assign the value of 1 to males and 2 to females, but there is no implication here that females are twice as good as males (and of course there is no researcher bias here).
2. ordinal - assign mutually exclusive values to levels of this variable, but at this level there is also an ordering or ranking of these values. These values are also discrete.  The spacing between each category is not numerically equivalent.  The judging of figure skaters is an example here where getting first place is clearly better than getting second place (order counts), but there is no implication here that the first place skater is better than the second place skater by the same amount that the second place skater is better than the third place skater (no implication of equal spacing).
3. interval - assign mutually exclusive values to each level of the variable with the levels of the variable being ordered and with equal spacing between each level of the variable.  These values become continuous.  The zero point is arbitrary, not absolute.  Temperature on the Fahrenheit scale is a good example.  Here a temperature of 10 degrees is warmer than a temperature of 5 degrees.  A rise in temperature from 5 degrees to 10 degrees is equivalent to the rise in temperature from 10 degrees to 15 degrees, or even equivalent to a rise in temperature from 80 to 85 degrees.  However, a temperature of 0 is not a complete lack of "temperature" (although this concept is easier for a Vermonter to understand than a Floridian). Also, since the zero point is not absolute, you can't say that a temperature of 10 is twice as warm as a temperature of 5.
4. ratio - assign mutually exclusive values to each level of the variable with the levels of the variable being ordered and with equal spacing between each level of the variable and with the zero point being absolute.  You can say that a subject with a $40,000 income has a higher income than the individual with a $20,000 income (order), and you can say that the rise in income from $20,00 to $40,000 is equivalent to the rise in income from $40,000 to $60,000 (equal spacing).  You can also say that an individual with an income of $0 has no income (zero point is absolute).  Having data at the ratio level allows comparisons to be made in a ratio form.  For example you can speak of the one subject's income ($40,000) being twice as high as the another subjects income ($20,000).

 

 

Most researchers seek to measure their variables at higher levels on the measurement scale if possible.  We will discuss this more when we talk about statistics.  For now let's just say that more sophisticated forms of statistical analyses can be completed when measurement is conducted at higher levels on the measurement scale. Measurement at the higher levels on the measurement scale also allow more discrete inferences to be made about the data.  For example, if you were measuring income at an ordinal level you could only infer that one subject's income was more or less than another subject's income.  However, if you measure income at a ratio level, you have a lot more information about your subjects' income since you have the qualities of equal spacing and an absolute zero point.  With data measured at the ratio level, you can say exactly how much more income one subject has as compared to another, and you can make mathematical computations to compare or your subjects' income.  That is not possible when you measure at the ordinal level.

It can also be said that even within the deductive/quantitative process, there is a place for inductive thinking.  Nunnally says: "Although the data of science must be objective, the scientist must rely on his intuition for research ideas."
 

Methods of data collection on objects:  (research is conducted on objects or physical qualities, not subjects)

• Measurement of objects - can be of physical qualities such as temperature, hardness...
• You can also measure qualities of the environment such as loudness, brightness...

Methods of data collection on subjects: (not as objective as much of the measurement of objects)

1. physiological
2. tests
3. observational
4. interview
5. questionnaire
6. available data (records)

 

advantages:
1. objectivity
2. precision
3. sensitivity
disadvantages
1. expense
2. training required
3. presence of some types of devices might change the measurement

some examples of these tests are:
1. psychological tests to determine the amount of an attribute present in a subject (for example depression).  There is not a clear distinction between tests and questionnaires.  "Tests" tend to be more standardized, but a distinction between these two terms is not absolute, and varies from text to text.
2. tests of achievement or mastery (for example a licensing exam)
3. tests of aptitude (for example an IQ test)
advantages:
1. objectivity
2. precision
3. sensitivity
4. standardization
disadvantages:
1. bias
2. questions of validity and reliability
3. often times a lot more work must be done to carefully construct the test (the pay back will usually be that once the test is constructed, gathering and analyzing the data is a lot easier than with the non-standardized forms of assessment)

a. consider if the observer utilizes:
1. concealment (whether the subjects knows they are being observed or not) (remember informed consent)
2. intervention (whether the observer provokes an action or not)
b. consider if the observation is:
1. structured (specifying in advance what behaviors or events will be observed and preparing forms for record keeping, such as categorization systems, checklists, and rating scales)
2. unstructured (use field notes to record observations)

advantages:
1. sometimes observing the subject may give a more accurate picture of the behavior in question than asking the subject, or may be the only way to to gather the data
2. suitable in complex situations that are best viewed as total entities and that are difficult to measure in parts
3. allows great depth and variety of information to be gathered
disadvantages:
1. interaction between the observer and subject can introduce bias or reactivity
2. there may be a lot of work summarizing and interpreting the data
3. the more the observer needs to make inferences and judgments about what is being observed, the more likely it is that distortions of the data will occur

4. & 5. Interviews and Questionnaires - ask subjects to report data for themselves.

A. Interviews - data collector questions the subject verbally (face-to-face or by phone) Using the phone can be a more efficient way to reach more respondents.
• Group or individual subject interviews: group interview may be appropriate for couples, families or work groups. Group interviews of 5 or more individuals involve a focus group methodology in which the interaction of the individuals is key to the information the researcher wishes to obtain.
• Structured interviews - questions determined before the interview. Researcher controls the content and sequencing of questions.
• Unstructured interviews - used primarily in qualitative studies and exploratory quantitative studies. The researcher initially presents the topic area of the interview to a subject and then uses probing questions to obtain information. The probing must be neutral to avoid biasing the subject's response
advantages:
1. best when the researcher may need to clarify the task for the respondent, or is interested in obtaining more personalized information from each respondent. Developing a rapport with the subject may facilitate the discussion of a sensitive issue
2. response rate tends to be higher for interviews than for questionnaires. This may eliminate some sampling bias (Would there be a difference between the type of subject that would answer the interview or questionnaire as compared to the type of subject who would not answer it?)
3. allow for richer, more complex data to be collected
disadvantages:
1. social desirability - subject seeks to respond to a question in a way to make a favorable impression
2. subject to interviewer bias - just your body language or tone of voice can influence how a respondent thinks they should answer a question!
3. more expensive and time consuming than questionnaires
B. Questionnaires - paper-and-pencil instruments designed to gather data
advantages:
1. most useful when there is a finite set of questions to be asked and the researcher can be assured of the clarity and specificity of the questions
2. less expensive than interviews
3. statistical analysis can be used. Groups can be compared.
4. allow anonymity. Subjects are more likely to respond honestly on paper knowing that their confidentiality will be upheld.
5. no interviewer bias
disadvantages:
1. social desirability - subject seeks to respond to a question in a way to make a favorable impression
2. lower response rate than for interviews
3. researcher may miss information that could be obtained with probing interview questions


Interviews and Questionnaires:

• Both ways to gather information such as attitudes and beliefs.
• Questions can be open-ended or close-ended.
open-ended questions:
1. used when the researcher wants the subjects to respond in their own words. May need to use a technique called content analysis to objectively, systematically and quantitatively translate subjects' responses into usable data.
2. used when the researcher may not know all of the possible alternate responses
close-ended questions:
1. fixed number of alternate responses, and the subjects pick their answer(s)
• Questions can be direct (as in asking the subject's age) or indirect, where the researcher uses a combination of items to estimate to what degree the respondent has some trait or characteristic (combining items to obtain an overall score).
• survey research - relies almost entirely on questioning subjects by either interviews or questionnaires


Scales for subject responses on Interviews and Questionnaires:

Scales are tools for the quantitative measurement of the degree to which individuals possess a specific attribute or trait.
1. Dichotomous: the answer must be one of two responses, for example, yes/no

2. Rank Order Questions: (Guttman scale) unidimensional (homogeneous), with graduated intensity.  Items hierarchically arranged so that endorsement of one item means an endorsement of those items below it.

 
3. Likert Scale: a scaled response, usually with five levels. Can be used to evaluate the amount of agreement, for example:
1. strongly agree
2. agree
3. neither agree or disagree
4. disagree
5. strongly disagree
The researcher may be interested in responses to individual questions, or possibly a total score for the whole questionnaire.


4. Semantic differential: a series of rating scales in which the respondent is asked to give a judgment about something along an ordered dimension, usually of 7 points.  Ratings are bipolar in that they specified two opposite ends of a continuum (good-bad). the researcher usually sums the points across all items for a total score for the questionnaire.

Not all studies require the researcher to acquire new information. Sometimes existing information can be examined to study a problem.

Can show trends over time (longitudinal) or comparisons of different groups at the same time (cross-sectional)

advantages:

1. may be a lot cheaper, faster and efficient way to gather data
2. with historical data, it may be the only way to "go back in time and get the data"; review the phenomenon in the past or over time
3. decreases the problems of reactivity (Hawthorne effect) and response set bias. This may be especially true with sensitive issues.
disadvantages:
you may not be assured that the data is representative and unbiased if you had nothing to do with how the data was collected

1. should be guided by the nature of the research question
2. should be guided by the literature review were you may gain information about how other researchers have measured your variables of interest
3. should be guided by the nature of the sample of subjects being measured

Reliability

Synonyms

• constancy
• reproducibility
• precision

 

 

Most measurements that we take have some error.  We represent this by conceiving of the observed score being made up of the true score plus some random error of measurement.

O = T + E
Building on this concept of measurement we can describe reliability as a ratio between the true score and an observed score.
r = T/O = T/(T + E)
If we examine this ratio we can see that if the error is very small, then the ratio approaches 1.  If the error is very large, than the ratio approaches zero.  Reliability coefficients range from 0 to 1, with a researcher obviously seeking reliability that approaches 1.

DePoy discusses three forms of reliability.

1. stability refers to the consistency of repeated measurements.  This test-retest format examines the reliability of an instrument (test) in a situation where the same test is given twice to the same subjects under the same circumstances.
2. tests of internal consistency refer to tests of the homogeneity of the items within a test.  The two major forms of internal consistency are the split half coefficient and the alpha coefficient.
3. equivalence refers to neither inter-rater reliability or reliability of alternate forms of the same test.  Inter-rater reliability refers to a comparison of two observers measuring the same event.  This can be measured as a correlation or as a percent of agreement between the observers.  Alternate or parallel forms involves the comparison of two versions of the same test.  With alternate forms, equivalent forms of one test are administered to the same subjects on the same testing occasion.

• Validity refers to how well and instruments measures some underlying concept.  The closer the instrument comes to representing the true definition of the concept the more valid the instrument.
• It is important to note that a scale that is valid is necessarily reliable, but the opposite may not be true.  Systematic error makes a measurement invalid

1. content validity addresses the degree to which measurement reflects the basic content of the phenomenon or domain of interest.  In order to establish content validity the researcher should:
1. specify the full domain through a literature search
2. ensure adequate representation of the domain in the items on the test
3. have a panel of experts review each item on the test
2. criterion validity involves demonstrating a correlation between the measurement of interest and another standard that has been shown to be an accurate representation of the underlying concept of interest.  Two types of criterion validity:
1. concurrent validity involves administering the new instrument along with the already accepted and validated instrument to measure this same concept on the same sample of individuals.  The correlation between the two measures of this concept represent the validity of the new instrument.
2. predictive validity is using the measurement on an instrument to predict or estimate the occurrence of a behavior or event.  The correlation between the score on the instrument and the subsequent criterion behavior or event is a measure of this validity.
3. construct validity is the most complex and comprehensive form of validation.  with construct validity the researcher collects supporting evidence of the relationship of the test instrument to related and distinct variables associated with the construct of interest.  The correlation or relationship of the test score to other variables or measures expected to be related to the construct represents the degree of construct validity.

DePoy lists 7 steps in instrument construction:

1. review the literature for relevant instrumentation
2. identify the theory from which to develop a new instrument
3. specify the concept or construct to be operationalized into an instrument
4. conceptually define the full range and content of the concept
5. select an instrument format, or potential type of instrument
6. translate the concept into specific items or indicators with appropriate response categories
7. test the reliability and validity of the instrument

1. format or design of the instrument
2. clarity of the instrument
3. social desirability of the questions
4. variation in administration of the test
5. situational contaminants
6. response set biases (all "yes" or all "no" responses)

A concept that contrasts with objectivity and systematic data collection (associated with quantitative research) is that of the richness of data found with open-ended, unstructured exploration (associated primarily with qualitative research).  The data collected in the latter form can be extremely valuable, but it is often more challenging to analyze.  The work is not done beforehand in determining exactly how to collect and analyze the data, but there is certainly a challenge in determining during, and after the data collection, how to collect and analyze the data.  To reiterate, a major distinction between qualitative and quantitative research is that in quantitative research decisions about theory and method are made before data collection starts.  In qualitative research the method for data collection may evolve during the study, and theory is not formulated until after data collection and analysis are at least underway, if not completed.

 copyright,  Thomas Bevins (3/14/99)

Sources:
LoBiondo-Wood, G., & Haber, J. (1998). Nursing research: Methods, critical appraisal, and utilization
(4th ed.).  St. Louis, MO: Mosby.
DePoy, E. & Gitlin, L. (1998).  Introduction to research:  Understanding and applying multiple strategies. St. Louis:  Mosby.