Methodology and Technical Notes

This section describes features of the Teaching and Learning International Survey (TALIS) 2018 methodology including sample design and survey design with a particular focus on the U.S. implementation. For further details about the assessment and any of the topics discussed here, see the Organization for Economic Cooperation and Development (OECD)’s TALIS 2018 Technical Report (OECD 2019).

• International Requirements for Sampling, Data Collection, and Response Rates
• Sampling in the United States
• Survey Development
• Translation and Adaptation
• Recruitment, Survey Administration, and Quality Assurance
• Weighting
• Data Limitations
• Confidentiality and Disclosure Limitations
• Statistical Procedures
• International and U.S. Response Rates
• U.S. Nonresponse Bias Analysis
• References

International Requirements for Sampling, Data Collection, and Response Rates

The OECD required all participating education systems (countries and subnational regions) to adhere to the TALIS 2018 technical standards, which provided detailed information about the target population, sampling, response rates, translation, survey administration, and data submission. According to the standards, the population covered in each education system should consist of as many as possible of the eligible International Standard Classification of Education (ISCED) Level 2 teachers and school administrators. Developed by the United Nations Educational, Scientific and Cultural Organization (UNESCO), ISCED is used by countries to map education levels across countries and education systems. In the United States, ISCED Level 2 teachers are those of students in grades 7, 8, and 9 (also labeled lower secondary education for convenience). To provide valid estimates of teacher and principal characteristics, the sample of TALIS teachers had to be selected in a way that represented the full population of ISCED 2 teachers in each education system. The sample design for TALIS 2018 was a stratified systematic sample, with the school sampling probability proportional to the estimated number of ISCED 2 teachers within each school. Samples were drawn using a two-stage sampling process. In the first stage a sample of schools was drawn, and in the second stage a sample of teachers was drawn within each selected school. Statistics Canada (StatCan), one of the members of the TALIS consortium responsible for the design and implementation of TALIS internationally, drew the sample of schools for each education system.

A minimum sample size of 4,000 teachers from a minimum of 200 schools was required in each participating education system. Following the TALIS consortium guidelines, replacement schools were identified at the same time the TALIS sample was selected by designating as replacement schools the two neighboring schools in the sampling frame. The international guidelines specified that within schools, a sample of 20 teachers was to be selected in an equal probability sample unless fewer than 20 teachers were available (in which case all teachers were selected).

Each education system collected its own data following international guidelines and specifications. The technical standards required that eligible teachers were those teaching at least one ISCED Level 2 class, regardless of subject matter. School principals or head administrators of each sampled school were also asked to participate. School principal and teacher data were collected independently, so teacher eligibility was not dependent on principal participation (or vice versa).

The response-rate target was at least 75 percent of schools and at least 75 percent of teachers across the participating schools in each education system. A minimum participation rate of 50 percent of schools from the original school sample and 75 percent of schools after replacement, was required for an education system’s data to be included in the main international comparisons. Education systems were allowed to use replacement schools (selected during the sampling process) to increase the response rate as long as the 50 percent benchmark before replacement had been reached.

The data collected by each participating education system was adjudicated by the TALIS international consortium to ensure that the data met the TALIS technical standards for data collection. The principal and teacher data were adjudicated separately. For school-level data adjudication depended on only school data (the principal participation) and for teacher-level data, adjudication depended on only teacher data (50 percent of teachers in the school had to participate). Thus, for TALIS 2018, there are two datasets—a school dataset with data from the principal survey, and a teacher dataset with data for teachers, each defined by their respective participation standards.

Internationally, TALIS 2018 was designed to be as inclusive as possible. In-scope teachers, those comprising the international target population, were all classroom teachers teaching at least one ISCED Level 2 class and their school principals, and all subject matters are included. Out-of-scope teachers included:

• Teachers in schools for special education needs students and their principals.
• Substitute, emergency, or occasional teachers who are defined as teachers filling in on a temporary basis (no longer than 6 consecutive weeks) for a teacher who is still employed as either a full-time or part-time teacher at the school.
• Teachers teaching exclusively to adults who are defined as teachers teaching only to adults, whether the adult students follow a standard or an adapted curriculum.
• Teachers on long-term leave who are defined as teachers “on long-term leave” who are absent and not expected to be back during the survey administration period.
• Teacher aides who are typically non-professional or paraprofessional staff supporting teachers in providing instruction to students.
• Pedagogical support staff who provide services to students to support the instructional program, such as guidance counselors or librarians.
• Health and social support staff who are health professionals such as doctors, nurses, psychiatrists, psychologists, occupational therapists, and social workers.

National target populations aim for maximum coverage of eligible teachers and schools. However, in some rare cases, for reasons of practicality, safety or economy (e.g., remote schools, unique demographic groups, types of schools, areas under civil unrest, natural catastrophe), the national survey population may be reduced. The U.S. population was not reduced in any way. (See Table A-3 for coverage rates for each country.)

Sampling in the United States

The TALIS 2018 school sample of 220 schools was drawn for the United States by the TALIS consortium. The U.S. sample was stratified into five explicit groups based on school control (public/private) and grade structure (with three groups—middle/junior high schools with grades 6–8 or 7–9, high schools with grades 9–12, or other schools with grade structures that include at least one ISCED 2 grade). Given the small number of private schools with a middle/junior high school grade structure, this stratum was collapsed with the high school grade structure in private schools. Within each explicit stratum, schools were sorted by census region (Northeast, Midwest, South, and West), locale (urban/suburban/town-rural),¹ percent minority students,² state, and number of ISCED 2 students (measure of size—see paragraph below).

The school sampling frame included any school containing at least one of grades 7 through 9 (defined as ISCED 2 in the United States). The data for public schools were from the 2015–16 Common Core of Data (CCD) and the data for private schools were from the 2015–16 Private School Universe Survey (PSS). The U.S. TALIS 2018 national school sample consisted of 220 schools. This number represents an increase from the international minimum requirement of 200 and was implemented to offset anticipated school nonresponse and reduce design effects.

The CCD and PSS databases include estimates of the number of teachers per school. While the number of full-time equivalent teachers (FTE) was available from the CCD, the number of teachers by ISCED level was not. Following the advice of the TALIS consortium, Westat, a research organization under contract to the National Center for Education Statistics (NCES) in the U.S. Department of Education, used the number of estimated ISCED 2 students from the CCD and PSS databases as the measure of size. Student estimates rather than teachers were used due to the uncertainty of the number of ISCED 2 teachers in certain types of schools (e.g., grade 9 teachers in grades 9–12 high schools).

Overlap with the 2018 National Teacher and Principal Survey (NTPS) school sample was minimized for the 2018 TALIS. The NTPS school frame was developed from the 2014–15 CCD. The school sample for the NTPS was selected by May 18, 2017, by the U.S. Census Bureau, and the Census Bureau provided Westat with the complete frame of schools (public schools only) with NTPS probabilities of selection for every school on their frame. Westat provided Statistics Canada the NTPS probabilities for the schools that matched the TALIS frame, and in drawing the U.S. TALIS sample Statistics Canada used this information of minimize overlap with NTPS.

In order to obtain a sample of teachers within schools, participating schools provided a list of TALIS eligible teachers (typically in January and February of 2018), and with the data from this list the sample was drawn using sampling software provided by the international contractor.

---

¹ Town and rural were combined to create three equal-sized groups because the category “town” was much smaller than the other three categories.
² The percent minority students is defined as “15 percent and above” or “below 15 percent” Black, Hispanic, Asian, Hawaiian/Pacific Islander, American Indian and Alaska Native, and l students of Two or more races.

Survey Development

The 2018 survey instruments were developed to cover a set of predefined themes defined by the TALIS framework (Ainley and Carstens 2018). These themes were identified by representatives of the participating education systems, the Questionnaire Expert Group (QEG), and the OECD. Items were reviewed by the TALIS QEG and by representatives of each country for their relevance to TALIS’s goals and for possible biases. All participating education systems field-tested the survey items in spring 2017. After the field trial, the descriptive statistics and psychometric properties of the items were reviewed, and items that did not meet the established measurement criteria were dropped for the main assessment.

The field trial also served as a means to test the field operations for the TALIS survey. The field trial evaluated the efficiency, accuracy, and effectiveness of the online questionnaire system in capturing information.

The final U.S. 2018 main study principal survey included 48 questions and the U.S. 2018 teacher survey included 55 questions. Countries were permitted to add “national only” questions/answers and answer categories. Also, each country adapted the international questionnaire to fit national terms, definitions, spelling, and punctuation, which is explained further in the following section. The resulting final set of items covered all 9 content themes of the framework. The United States added two questions to both the principal and teacher surveys asking about the principal’s/teacher’s race and ethnicity. The U.S. principal survey also added a question asking principals to rate a series of educational goals. The final national principal and teacher surveys are available online at: https://nces.ed.gov/surveys/talis/questionnaire.asp.

Translation and Adaptation

International source versions of the principal and teacher surveys and school-level manuals were prepared in English and French and translated into the primary language or languages of instruction in each education system. The TALIS consortium recommended that each education system follow a double translation design and provided precise translation guidelines that included a description of the features each item was measuring and for the main study statistical analysis from the field trial. These translation procedures entailed having two independent translations, one from each of the source languages (English and French), with differences reconciled by a third party. When double translation was not possible, single translation was accepted.

Instrument adaptation was necessary even in nations such as the United States that use English as the primary language of instruction. These adaptations were primarily for cultural purposes. For example, words such as “lift” might be adapted to “elevator” for the United States. The TALIS consortium verified and approved the national adaptation of all instruments, including those of the United States.

Recruitment, Survey Administration, and Quality Assurance

The TALIS 2018 school recruitment strategy included: (1) starting recruitment at the beginning of the school year in 2017; (2) incorporating assistance from National Assessment of Educational Progress (NAEP) State Coordinators within the states with sampled TALIS schools or approaching schools directly, as well as sending information to relevant school districts and states; and (3) providing cash incentives to both schools and teachers.

Westat staff initiated school recruitment activities in September 2017. Contact with states and districts began through a mailing in late September 2017, after an initial virtual meeting with NAEP State Coordinators. Schools were contacted through an initial mailing shortly after the district mailing. These initial mailings were sent from the NAEP State Coordinators and included a letter from NCES, or the state chief, addressed to the appropriate district or school official, and a folder of TALIS materials that contained a study brochure, timeline, and a list of frequently asked questions (FAQs). Twenty-eight school districts required the formal review and approval of a research proposal before schools could be contacted. Formal research requests were prepared and sent to these districts. In those states where NAEP State Coordinators were assisting with school recruitment, NAEP State Coordinators indicated when follow-up contacts could be made by Westat field staff. In states where the NAEP State Coordinators were not assisting with school recruitment, Westat field staff contacted districts and schools independently. Schools were asked to identify school coordinators to facilitate the survey within the schools and liaise with Westat during the TALIS data collection. School coordinators were asked to prepare a list of eligible teachers for the within-school teacher sample, distribute paper instruments, track teacher and principal participation, and distribute teacher incentives. The school coordinators of participating schools were offered $200, teachers were offered $20 to complete the teacher questionnaire, and the schools were offered $200 for participating. The data collection window for TALIS began March 1, 2018 and ended May 31, 2018.

The TALIS consortium emphasized the use of standardized procedures in all education systems. Each education system collected its own data, based on detailed manuals provided by the TALIS consortium that explained the survey’s implementation including precise instructions for the work of school coordinators. Survey administration in the United States was performed primarily using an online instrument with paper-based surveys also available. The paper instruments were provided upon request and also sent to nonresponding teachers and principals to prompt their response.

In each education system, a TALIS International Quality Observer (IQO) who was contracted independently by the TALIS consortium reviewed survey administrations in a subsample of participating schools. The schools in which the independent observations were conducted were selected jointly by the TALIS consortium and the IQO. In the United States, there was one IQO who observed 24 schools from the national sample. The IQO’s primary responsibility was to document the extent to which survey procedures in schools were implemented in accordance with administration procedures. The IQO’s observations in U.S. schools indicated that international procedures for data collection were applied consistently.

Weighting

The use of sampling weights is necessary for computing statistically sound nationally representative estimates. Survey weights were adjusted for the probabilities of selection for individual schools and teachers, for school or teacher nonresponse, and for errors at the time of sampling in estimating the size of the school or the number of eligible teachers in the school. Survey weighting for all education systems participating in TALIS 2018 was coordinated by the international TALIS consortium.

The school base weight was defined as the reciprocal of the school’s probability of selection. (For replacement schools, the school base weight was set equal to the original school it replaced.) Within each explicit stratum, the school base weights were then adjusted for nonresponding schools that were not replaced by replacement schools to produce the final school weights.

The teacher weighting was more complicated than the school weighting because in addition to the teacher base weight it included the school base weight as well as four additional adjustment factors: school nonresponse, teacher nonresponse, incidental inclusions, and multiplicity. The teacher base weight was defined as the reciprocal of the within school probability of selection for each selected teacher. The teacher nonresponse adjustment was an adjustment that allocates the weight of the nonresponding teachers to responding teachers within each school. The teacher nonresponse adjustment included adjustments that accounted for nonresponding teachers as well as teachers that left the school after the teachers were selected for the sample. The incidental inclusion adjustment adjusts the teacher weight for teachers who were counted originally but then excluded because they were also principals. The final weighting adjustment was the teacher multiplicity factor that adjusted the weights for teachers working in more than one school. For most teachers, the adjustment factor was 1; for the others, it was the reciprocal of the number of schools in which they taught. The final teacher weights were the product of the final school weight, the teacher base weight, and the three teacher adjustment factors.

All TALIS analyses were conducted using these adjusted sampling weights. For more information on the nonresponse adjustments, see OECD’s TALIS 2018 Technical Report (OECD 2019).

Data Limitations

As with any study, there are limitations to TALIS 2018 that should be taken into consideration. Estimates produced using data from TALIS 2018 are subject to two types of error: non-sampling errors and sampling errors.

Non-sampling error is a term used to describe variations in the estimates that may be caused by population coverage limitations, nonresponse bias, and measurement error, as well as data collection, processing, and reporting procedures. For example, suppose the study was unsuccessful in getting permission from many rural schools in a certain region of the country. In that case reports of means for rural schools for that region may be biased. The sources of non-sampling errors are typically problems such as item nonresponse, the differences in respondents’ interpretations of the meaning of survey questions, and mistakes in data preparation, such as miskeying paper survey responses or errors in data cleaning.

Sampling errors arise when a sample of the population, rather than the whole population, is used to estimate some statistic. Different samples from the same population would likely produce somewhat different estimates of the statistic in question. This means that there is a degree of uncertainty associated with statistics estimated from a sample. This uncertainty is referred to as sampling variance and is usually expressed as the standard error of a statistic estimated from sample data. The approach used for calculating standard errors in TALIS is the Fay method of balanced repeated replication (BRR) (Judkins 1990). This method of producing standard errors uses information about the sample design to produce more accurate standard errors than would be produced using methods that assume simple random sampling.

Standard errors can be used as a measure of the precision expected from a particular sample.

Confidence intervals provide a way to make inferences about population statistics in a manner that reflects the sampling error associated with the statistic. Assuming a normal distribution and a 95 percent confidence interval, the population value of this statistic can be inferred to lie within the confidence interval in 95 out of 100 replications of the measurement on different samples drawn from the same population.

Confidentiality and Disclosure Limitations

Confidentiality analyses for the United States were designed to provide reasonable assurance that public-use data files issued by the TALIS consortium would not allow identification of individual U.S. schools or teachers when compared against other public-use data databases. To limit disclosure risk for TALIS schools and teachers, analyses were conducted to identify potential risks, and based on this analysis school and teacher identification were masked by randomly swapping identifying data elements within the teacher and school files to add a measure of uncertainty. Swapping was designed and implemented in such a way that accurate estimates of means and variances for the whole population and reported subgroups could be maintained (Krenzke et al. 2006).

Statistical Procedures

Comparisons made in the text of the TALIS Highlights have been tested for statistical significance. For example, in the commonly made comparison of TALIS averages to U.S. averages, tests of statistical significance were used to establish whether or not the observed differences from the U.S. average were statistically significant.

In almost all instances, the tests for significance used were standard t tests. These fell into two categories according to the nature of the comparison being made: comparisons of independent samples and comparisons of non-independent samples. In TALIS, education system groups are independent. We judge that a difference is “significant” if the probability associated with the t test is less than .05. If a test is significant this implies that difference in the observed means in the sample represents a real difference in the population.³ No adjustments were made for multiple comparisons.

In simple comparisons of independent averages, such as the average percentages of education system 1 with that of education system 2, the following formula was used to compute the t statistic:

where est₁ and est₂ are the estimates being compared (e.g., averages of education system 1 and education system 2) and se₁² and se₂² are the corresponding squared standard errors of these averages. The TALIS 2018 data are hierarchical and include school and teacher data from the participating schools. The standard errors for each education system take into account the clustered nature of the sampled data. These standard errors are not adjusted for correlations between groups since groups are independent.

The second type of comparison occurs when evaluating differences between non-independent groups within the education system. Because of the sampling design in which schools and teachers within schools are randomly sampled, the data within the education system from mutually exclusive sets of teachers or principals (for example, males and females) are not independent. For example, to determine whether the performance of females differs from that of males would require estimating the correlation between females’ and males’ scores. The BRR procedure previously mentioned was used to estimate the standard errors of differences between non-independent samples within the United States. Use of the BRR procedure implicitly accounts for the correlation between groups when calculating the standard errors.

To test comparisons between non-independent groups the following t statistic formula was used:

where est_grp1 and est_grp2 are the non-independent group estimates being compared and se_(grp1-grp2) is the standard error of the difference calculated using BRR to account for the correlation between the estimates for the two non-independent groups.

---

³ A .05 probability implies that the t statistic is among the 5 percent most extreme values one would expect if there were no difference between the means. The decision rule is that when t statistics are this extreme, they are sampled from a population in which there is a difference between the means.

International and U.S. Response Rates

This section describes the success of participating education systems in meeting the international technical standards on data collection. Information is provided for all participating education systems on their coverage of the target population, exclusion rates, and response rates. For TALIS 2018, the principal and teacher data were adjudicated separately and are provided as separate, independent datasets with separate response rates.⁴

Table A-1 provides information on weighted school participation rates based on principals that responded to the principal questionnaire before and after school replacement and the number of participating schools after replacement for each participating education system. Table A-2 provides information on weighted school participation rates based on teachers that responded to the teacher questionnaire before and after school replacement and the number of participating schools and teachers after replacement for each participating education system.

Table A-3 provides information on the coverage of the target population and overall exclusion rates for each participating education system.

For the school data based on the participation of school principals, out of a sample of 220 schools, 214 were determined to be eligible and of those, 123 original schools participated for an initial unweighted response rate of 58 percent (63 percent weighted) before replacements were added. An additional 41 replacements schools participated for a total of 164 participating schools in the U.S. administration of TALIS. The unweighted response rate increased to 77 percent (also 78 percent weighted).

For school data based on the participation of teachers, 125 original schools participated for an initial unweighted response rate of 58 percent (60 percent weighted) before replacements were added. An additional 40 replacements schools participated for a total of 165 participating schools in the U.S. administration of TALIS. The unweighted response rate increased to 77 percent (77 percent weighted).

Within the schools that participated, up to 20 ISCED Level 2 teachers within each school were selected to participate in TALIS 2018. For the U.S., 90 percent (weighted) of sampled teachers participated in TALIS.

In addition to the international response rate standards described in the prior section, the U.S. sample had to meet the statistical standards of the National Center for Education Statistics (NCES) of the U.S. Department of Education. Since the U.S. TALIS weighted school response rate fell below 85 percent, NCES required that an investigation into the potential magnitude of nonresponse bias at the school level in the U.S. sample be conducted. Because the U.S. TALIS weighted teacher response rate was above 85 percent, a nonresponse bias analysis at the teacher level was not required.

Table A-1. Weighted participation rates and number of schools based on participating principals, by education system: 2018

Education system	Estimated size of school population	Weighted school participation before replacement	Weighted school participation after replacement	Number of participating schools after replacement
Alberta–Canada	1,038	46.0	57.8	129
Australia	2,680	47.1	75.1	230
Austria	1,483	93.1	100.0	277
Belgium	1,169	86.0	95.8	307
Belgium–Flemish	721	81.4	94.0	188
Brazil	52,187	88.0	95.4	184
Bulgaria	1,730	96.5	100.0	200
Chile	5,214	78.9	87.6	169
Chinese Taipei	935	100.0	100.0	202
Colombia	10,392	66.6	69.6	141
Croatia	896	95.0	95.6	188
Cyprus	99	88.9	88.9	88
Czech Republic	2,606	99.0	99.0	218
Denmark	1,457	51.5	71.4	140
England–United Kingdom	3,990	70.1	81.8	157
Estonia	389	88.3	100.0	195
Finland	706	100.0	100.0	148
France	6,770	97.6	98.0	195
Georgia	2,151	91.7	91.7	177
Hungary	2,640	90.9	93.3	182
Iceland	136	74.3	74.3	101
Israel	1,196	90.9	93.7	184
Italy	5,622	92.4	98.6	190
Japan	10,071	93.9	99.4	195
Kazakhstan	6,302	100.0	100.0	331
Korea, Republic of	3,134	68.1	77.8	150
Latvia	653	73.4	90.8	136
Lithuania	833	100.0	100.0	195
Malta	58	93.1	93.1	54
Mexico	16,327	90.6	97.0	193
Netherlands	524	56.2	85.6	125
New Zealand	1,732	71.7	92.0	189
Norway	1,091	64.9	80.6	162
Portugal	1,255	97.7	100.0	200
Romania	4,658	100.0	100.0	199
Russian Federation	31,948	99.1	100.0	230
Saudi Arabia	6,119	96.2	96.2	192
Shanghai–China	630	100.0	100.0	198
Singapore	193	93.3	97.9	167
Slovak Republic	1,593	83.6	90.4	180
Slovenia	448	74.8	79.3	119
South Africa	8,026	91.1	91.1	169
Spain	6,861	98.1	98.5	396
Sweden	1,739	83.5	88.6	171
Turkey	16,100	98.9	98.9	196
United Arab Emirates	521	91.4	91.4	476
United States	65,095	63.1	77.6	164
Vietnam	10,799	100.0	100.0	196

NOTE: Italics indicate non-OECD countries and education systems.
SOURCE: Organization for Economic Cooperation and Development (OECD), Teaching and Learning International Survey (TALIS), 2018.

Table A-2. Weighted participation rates and number of schools and teachers based on participating teachers, by education system: 2018

Education system	Estimated size of teacher population	Weighted school participation before replacement	Weighted school participation after replacement	Number of participating schools after replacement	Number of participating teachers	Teachers’ participation in participating schools (%)
Alberta–Canada	9,991	48.9	60.3	122	1,077	83.0
Australia	116,679	48.5	75.1	233	3,573	77.7
Austria	45,869	85.7	88.6	246	4,255	84.4
Belgium	34,442	86.0	95.1	302	5,257	86.9
Belgium–Flemish	18,615	80.1	91.0	182	3,122	84.4
Brazil	568,510	89.9	96.6	185	2,447	94.9
Buenos Aries–Argentina	10,218	80.7	85.2	130	2,099	88.6
Bulgaria	21,208	97.1	100.0	200	2,862	98.3
Chile	55,969	82.6	91.5	179	1,963	94.3
Chinese Taipei	53,208	98.9	98.9	200	3,835	97.2
Colombia	164,225	73.1	77.1	154	2,398	93.4
Croatia	15,762	95.4	96.2	188	3,358	87.0
Cyprus	3,860	88.9	88.9	88	1,611	90.3
Czech Republic	42,348	100.0	100.0	219	3,447	93.8
Denmark	22,475	51.1	72.0	141	2,001	86.8
England–United Kingdom	193,134	72.7	81.5	149	2,376	83.6
Estonia	7,354	86.6	100.0	195	3,004	95.2
Finland	18,938	100.0	100.0	148	2,851	96.2
France	197,013	87.3	87.8	176	3,006	88.1
Georgia	38,195	99.5	99.5	192	3,101	95.8
Hungary	44,018	94.9	97.7	189	3,245	95.0
Iceland	1,883	89.7	89.7	123	1,292	75.8
Israel	32,603	84.9	86.4	172	2,627	84.9
Italy	190,447	92.8	99.1	191	3,612	93.8
Japan	230,558	92.5	99.5	196	3,555	99.0
Kazakhstan	195,383	100.0	100.0	331	6,566	99.8
Korea, Republic of	75,654	69.9	82.5	163	2,931	92.2
Latvia	12,003	77.1	91.2	135	2,315	87.9
Lithuania	19,848	100.0	100.0	195	3,759	97.4
Malta	1,941	92.5	92.5	55	1,656	86.5
Mexico	254,794	90.4	96.3	193	2,926	94.3
Netherlands	66,672	56.7	79.5	116	1,884	80.9
New Zealand	23,227	62.6	79.3	185	2,257	79.6
Norway	21,828	77.4	92.6	185	4,154	83.2
Portugal	39,703	97.9	100.0	200	3,676	92.7
Romania	66,039	100.0	100.0	199	3,658	98.3
Russian Federation	646,405	98.7	100.0	230	4,011	99.9
Saudi Arabia	99,661	89.7	89.7	179	2,744	86.0
Shanghai–China	38,902	100.0	100.0	198	3,976	99.5
Singapore	11,544	96.9	100.0	169	3,280	99.2
Slovak Republic	24,746	82.4	88.9	176	3,015	95.4
Slovenia	7,422	82.2	88.0	132	2,094	91.5
South Africa	92,127	92.1	92.4	170	2,046	89.7
Spain	186,171	99.7	100.0	399	7,407	94.6
Sweden	31,421	89.1	93.9	180	2,782	81.3
Turkey	277,187	99.0	99.0	196	3,952	98.5
United Arab Emirates	14,489	100.0	100.0	521	8,648	96.0
United States	1,144,751	60.1	76.8	165	2,560	89.6
Vietnam	295,033	100.0	100.0	196	3,825	96.3

NOTE: Italics indicate non-OECD countries and education systems.
SOURCE: Organization for Economic Cooperation and Development (OECD), Teaching and Learning International Survey (TALIS), 2018.

Table A-3. Coverage of ISCED Level 2 target population and survey population, by education system: 2018

Education system	Population and coverage:	Reasons for exclusions	Number of schools	Percentage of schools	Number of teachers	Percentage of teachers
Alberta—Canada	Target Population		1,158	100	—	—
	Exclusive:	Band-operated schools. These schools operate on First Nations’ reserves and are the responsibility of the federal government rather than the responsibility of Alberta Education	39	3.4	—	—
		Very small schools (fewer than three students in each of Grades 7 to 9)	100	8.6	—	—
	Survey Population:		1,019	88	—	—
Australia	Target Population		2,862	100	—	—
	Exclusive:	Small schools (fewer than four students)	263	9.2	—	—
		Schools that are geographically remote	32	1.1	—	—
		Small schools that are also geographically remote	31	1.1	—	—
	Survey Population:		2,536	88.6	—	—
Austria	Target Population		1,496	100	—	—
	Exclusive:	Slovene school where the language of instruction is not German	1	0.1	—	—
	Survey Population:		1,495	99.9	—	—
Belgium	Target Population		1,245	100	—	—
	Exclusive:	Very small schools (fewer than 20 students at ISCED level 2)	10	0.8	—	—
		Special needs schools	79	6.3	—	—
	Survey Population:		1,156	92.9	—	—
Belgium—Flemish	Target Population		718	100	—	—
	Exclusive:	Very small schools (fewer than 20 students at ISCED level 2)	2	0.3	—	—
	Survey Population:		716	99.7	—	—
Brazil	Target Population		58,303	100	870,737	100
	Exclusive:	Schools with fewer than six teachers. Because ISCED level 2 requires at least one teacher for each subject, most of these schools have only one class. The schools that fit this criterion are located in geographically remote areas	4,957	8.5	18,896	2.2
		Public-federal schools	38	0.1	1,724	0.2
	Survey Population:		53,308	91.4	850,117	97.6
Buenos Aires—Argentina	Target Population		488	100	—	—
	Survey Population:		488	100	—	—
Bulgaria	Target Population		1,834	100	23,168	100
	Exclusive:	Schools for students with special education needs	67	3.7	454	2.0
		Very small schools	45	2.5	126	0.5
	Survey Population:		1,722	93.9	22,588	97.5
Chinese Taipei	Target Population		939	100	—	—
		Schools with fewer than three teachers	7	0.7	—	—
	Survey Population:		932	99.3	—	—
Chile	Target Population		6,008	100	51,626	100
	Exclusions:	Schools with fewer than three teachers	681	11.3	1433	2.8
		Schools that are geographically remote	4	0.1	27	0.1
	Survey Population:		5,323	88.6	50,166	97.2
Colombia	Target Population		13,009	100	—	—
	Exclusions:	Small schools (fewer than three teachers at ISCED level 2)	322	2.5	—	—
	Survey Population:		12,672	97.4	—	—
Croatia	Target Population		860	100	—	—
	Exclusions:	National minority schools (Italian, Serbian schools)	11	1.3	—	—
	Survey Population:		849	98.7	—	—
Cyprus	Target Population		102	100	4,426	100
	Exclusions:	School that is geographically remote	1	1.0	13	0.3
		School where language of instruction is one other than Greek or English	1	1.0	8	0.2
		Very small school that has part-time teachers and no head teacher or assistant head teacher	1	1.0	5	0.1
	Survey Population:		99	97.1	4,400	99.4
Czech Republic	Target Population		2,645	100	39,690	100
	Exclusions:	Schools with a different language of instruction (Polish)	10	0.4	116	0.3
		Dancing conservatoire – specific education programs	5	0.2	122	0.3
		Very small schools (fewer than three teachers at ISCED level 2)	14	0.5	11	0.0
	Survey Population:		2,616	98.9	39,441	99.4
Denmark	Target Population		1,721	100.0	—	—
	Exclusions:	Small schools (fewer than 40 students and generally fewer than 5 teachers)	251	14.6	—	—
	Survey Population:		1,470	85.4	—	—
England—United Kingdom	Target Population		4,345	100	—	—
	Exclusions:	International schools	13	0.3	—	—
		Very small schools	57	1.3	—	—
		Schools proposed for closure	17	0.4	—	—
	Survey Population:		4,258	98.0	—	—
Estonia	Target Population		404	100	8,622	100
	Exclusions:	International schools	4	1.0	34	0.4
		Ballet school	1	0.2	21	0.2
	Survey Population:		399	98.8	8,567	99.4
Finland	Target Population		722	100	—	—
	Exclusions:	International//foreign/immersion schools, where all students are taught in languages other than Finnish or Swedish	8	1.1	—	—
	Survey Population:		714	98.9	—	—
France	Target Population		7,203	100	215,940	100
	Exclusions:	Schools located in overseas French territories (TOM)	79	1.1	1,342	0.6
		Schools located in La Réunion and Mayotte (Southern Hemisphere calendar)	103	1.4	5,529	2.6
		Private schools under different administration	193	2.7	—	—
	Survey Population:		6,828	94.8	209,069	96.8
Georgia	Target Population		2,265	100	42,757	100
	Exclusions:	Schools where the language of instruction is an excluded minority language	13	0.6	249	0.6
		Very small schools	2	0.1	6	0.0
	Survey Population:		2,250	99.3	42,502	99.4
Hungary	Target Population		2,844	100	37,938	100
	Exclusions:	Very small schools (fewer than three teachers at ISCED level 2)	85	3.0	126	0.3
	Survey Population:		2,759	97.0	37,812	99.7
Iceland	Target Population		142	100	—	—
	Survey Population:		142	100	—	—
Israel	Target Population		2,475	100	—	—
	Exclusions:	International/foreign schools where the language of instruction is a single minority language	5	0.2	—	—
		Ultra-Orthodox Jewish schools where the language of instruction is Hebrew	1,246	50.3	—	—
	Survey Population:		1,224	49.5	—	—
Italy	Target Population		5,783	100	154,071	100
	Exclusions:	Schools with fewer than three teachers at ISCED level 2	63	1.1	90	0.1
	Survey Population:		5,720	98.9	153,981	99.9
Japan	Target Population		10,426	100	264,356	100
	Survey Population:		10,426	100	264,356	100
Kazakhstan	Target Population		6,424	100	208,254	100
	Exclusions:	Uzbek, Uighur and Tadjik schools. These regions together account for less than 5% of Kazakhstan’s population. The TALIS questionnaire will be administered in the Kazakh and Russian languages in these schools	30	0.5	1,215	0.6
		Schools located in the Baikonur region (special permission is required to enter this territory	5	0.1	306	0.1
		Schools with fewer than three teachers at ISCED level 2	2	0.0	5	0.0
		School where the language of instruction is English	1	0.0	60	0.0
	Survey Population:		6,386	99.4	206,668	99.2
Korea, Republic of	Target Population		3,252	100	69,820	100
	Exclusions:	Schools with fewer than three teachers at ISCED level 2 Schools that are geographically remote	151	4.6	1,073	1.5
		Schools with no more than three teachers at ISCED level 2 and schools that are geographically remote	4	0.1	7	0.0
		Schools with alternative curricula, teacher licensure and school structure	31	1.0	382	0.5
	Survey Population:		3,059	94.1	68,341	97.9
Latvia	Target Population		695	100	—	—
	Exclusions:	International schools (e.g. for diplomats’ children)	2	0.3	—	—
		Special regime (criminal) school	1	0.1	—	—
	Survey Population:		692	99.6	—	—
Lithuania	Target Population		960	100	—	—
	Exclusions:	International schools where English and French are the languages of instruction and the education system	3	0.3	—	—
		Youth schools. These implement basic education programs but also have a particular focus on special education (emotionally or socially disadvantaged students)	12	1.3	—	—
		Small schools (fewer than ten students at ISCED level 2)	19	2.0	—	—
	Survey Population:		926	96.5	—	—
Malta	Target Population		63	100	3,271	100
	Exclusions:	Language schools	2	3.2	16	0.5
	Survey Population:		61	96.8	3,255	99.5
Mexico	Target Population		16,763	100	328,649	100
	Exclusions:	Schools with fewer than three teachers	41	0.2	95	0.0
	Survey Population:		16,722	99.8	328,554	100.0
Netherlands	Target Population		538	100	—	—
	Survey Population:		538	100	—	—
New Zealand	Target Population		1,696	100	—	—
	Exclusions:	National correspondence school	1	0.1	—	—
	Survey Population:		1,695	99.9	—	—
Norway	Target Population		1,154	100	23,542	100
	Exclusions:	Very small schools (fewer than three teachers)	68	5.9	168	0.7
		Sami schools	5	0.4	48	0.2
		International schools	15	1.3	216	0.9
		International schools where the language of instruction is French	2	0.2	31	0.1
	Survey Population:		1,064	92.2	23,079	98.0
Portugal	Target Population		1,273	100	36,912	100
	Exclusions:	Schools with non-Portuguese curricula	16	1.3	299	0.8
	Survey Population:		1,257	98.7	36,613	99.2
Romania	Target Population		4,776	100	—	—
	Exclusions:	Very small schools (fewer than 25 students at ISCED level 2)	88	1.8	—	—
	Survey Population:		4,688	98.2	—	—
Russian Federation	Target Population		41,893	100	760,196	100
	Exclusions:	Crimean schools	621	1.5	12,462	1.6
		Moscow schools	733	1.7	41,541	5.5
	Survey Population:		40,539	96.8	706,193	92.9
Saudi Arabia	Target Population		8,105	100	120,109	100
	Exclusions:	Schools in the Najran and Jazan regions (near the borders with Yemen where there is war)	767	9.5	9,028	7.5
		Private schools. These schools are not being included in TALIS at this stage because their system of education differs from that of the public system	951	11.7	11,075	9.2
		Schools with fewer than four teachers	121	1.5	240	0.2
	Survey Population:		6,266	77.3	99,766	83.1
Shanghai—China	Target Population		662	100	41,705	100
	Exclusions:	Special schools for students with behavioral problems, including delinquency	12	1.8	340	0.8
	Survey Population:		650	98.2	41,365	99.2
Singapore	Target Population		197	100	12,285	100
	Exclusions:	Schools where the language of instruction is not English and where teachers would have difficulty responding to the TALIS survey, which is in English	4	2.0	200	1.6
	Survey Population:		193	98.0	12,085	98.4
Slovak Republic	Target Population		1,591	100	24,841	100
	Exclusions:	Schools with fewer than three teachers at ISCED level 2	10	0.6	20	0.1
	Survey Population:		1,581	99.4	24,821	99.9
Slovenia	Target Population		451	100	9,090	100
	Exclusions:	Italian basic schools in an ethnically mixed area where the language of instruction is Italian and where the teachers are therefore a separate group of Slovenian teachers	3	0.7	42	0.5
	Survey Population:		448	99.3	9,048	99.5
Spain	Target Population		6,954	100	200,193	100
	Exclusions:	Small schools (fewer than three teachers)	43	0.6	59	0.0
		Schools that are geographically remote	2	0.0	42	0.0
	Survey Population:		6,909	99.4	200,092	100.0
Sweden	Target Population		1,768	100	—	—
	Exclusions:	Small schools (fewer than six students at ISCED level 2)	49	2.8	—	—
		International schools not following the Swedish curriculum	11	0.6	—	—
	Survey Population:		1,708	96.6	—	—
Turkey	Target Population		16,228	100	310,932	100
	Survey Population:		16,228	100	310,932	100
United Arab Emirates	Target Population		577	100	17,541	100
	Exclusions:	Private schools	14	2.4	350	2.0
	Survey Population:		563	97.6	17,191	98.0
United States	Target Population		63,795	100	—	—
	Exclusions:	Schools in detention, hospital and treatment centers	569	0.9	—	—
	Survey Population:		63,226	99.1	—	—
Vietnam	Target Population		10,843	100	303,171	100
	Exclusions:	Non-Vietnamese international schools	22	0.2	153	0.1
	Survey Population:		10,821	99.8	303,018	99.9

— Not available
NOTE: Detail may not sum to totals because of rounding. Italics indicate non-OECD countries and education systems.
SOURCE: Organization for Economic Cooperation and Development (OECD), Teaching and Learning International Survey (TALIS), 2018

---

⁴ Principal/school data were adjudicated on their own in the 2018 survey, an occurrence that resulted in the notion of a “participating school for principal/school data” being introduced. A school was considered “participating” if its principal returned his or her questionnaire with at least one valid response. For the teacher data, the minimum of 50% teacher participation remained the criterion for determining whether a school was “participating” or not. Consequently, and in contrast to TALIS 2008 and 2013, a school record remained on the school file if the principal responded to the questionnaire, even if fewer than 50% of the teachers in the school participated in the survey.

U.S. Nonresponse Bias Analysis

In addition to the international response rate standards described in the prior section, the U.S. sample had to meet the statistical standards of the National Center for Education Statistics (NCES) of the U.S. Department of Education. Since the U.S. TALIS weighted school response rate fell below 85 percent, NCES required that an investigation into the potential magnitude of nonresponse bias at the school level in the U.S. sample be conducted. Because the U.S. TALIS weighted teacher response rate was above 85 percent, a nonresponse bias analysis at the teacher level was not required.

Of the 220 original sampled schools in the national sample, 214 were eligible (six schools did not have any grade-eligible students or had closed), and 125 agreed to participate. The weighted school response rate before replacement was 60 percent requiring the United States to conduct a nonresponse bias analysis to evaluate the quality of the final United States sample.

A bias analysis was conducted to address potential problems in the data owing to school nonresponse. The general approach taken involves an analysis in three parts as described below.

• 1. Analysis of the participating original sample: The distribution of the participating original school sample was compared with that of the total eligible original school sample. The participating original sample is the sample before substitution. In each sample, schools were weighted by their school base weights and estimate of eligible teachers, referred to as a size-adjusted weight¹ excluding any nonresponse adjustment factor.
• 2. Analysis of the participating final school sample with substitutes: The distribution of the participating final school sample, which includes participating substitutes that were used as replacements for nonresponding schools from the eligible original sample, was compared to the total eligible final school sample. The total eligible final sample includes the participating final sample plus those original nonrespondents that were not replaced by substitutes. Again, the size-adjusted school base weights were used for both the eligible sample and the participating schools.
• 3. Analysis of the nonresponse adjusted final sample with substitutes: The same sets of schools were compared as in the second analysis, but this time, when analyzing the participating final schools alone, school nonresponse adjustments were applied to the size-adjusted school base weights of the participating schools. The size-adjusted school base weights were again used the eligible sample.

To compare TALIS participating schools to the total eligible sample of schools, it was necessary to match the sample of schools to the sample frame to identify as many characteristics as possible that might provide information about the presence of nonresponse bias. Frame characteristics were taken from the 2015–16 Common Core of Data for public schools and from the 2015–16 Private School Universe Survey for private schools. The available school characteristics included school control (public or private), locale (central city, suburb, town, rural), census region, poverty (high or low poverty for public schools based on whether 50 percent or more of the students are eligible for participation in the national free and reduced-price lunch (FRPL) program), grade structure (schools with grades 6–8, grades 9–12, other), School size (small, medium, and large), number of grade-eligible students, total number of students, and percentage of students in various racial/ethnic groups (White, non-Hispanic; Black, non-Hispanic; Hispanic; Asian; American Indian or Alaska Native; Hawaiian/Pacific Islander; and two or more races). The percentage of students eligible for free or reduced-price lunch was available for public schools only. The full text of the nonresponse bias analysis conducted for TALIS 2018 will be included in the technical report released with the U.S. national dataset (forthcoming).

For original sample schools, participating schools had a lower mean total school enrollment than the eligible sample (812.6 versus 908.6, respectively; and a lower mean grade-eligible enrollment than the eligible sample (374.5 versus 424.8, respectively). Additionally, the absolute value of the relative bias for private schools, schools in rural areas, schools in each of the four census regions, small sized and large sized schools, Asian, American Indian or Alaska Native, and Hawaiian/Pacific Islander is greater than 10 percent, which indicates potential bias even though no statistically significant relationship was detected. Although each of these findings indicates some potential for nonresponse bias, when all of these factors were considered simultaneously in a regression analysis, only town was a significant predictor of school participation. The percentage of students eligible for free or reduced-price lunch was not included in the logistic regression analysis as public and private schools were modeled together using only the variables available for all schools.

For the United States final sample schools (with substitutes), schools in the Northeast were underrepresented among participating schools relative to eligible schools (15.8 percent versus 19.6 percent, respectively), while schools in the South were overrepresented among participating schools (47.6 percent versus 43.1 percent, respectively). Participating schools had a lower mean total school enrollment than the eligible sample (767.3 versus 857.6, respectively) and a lower mean grade-eligible enrollment than the eligible sample (373.8 versus 405.1, respectively). Participating schools had a lower mean percentage of Asian students than the eligible sample (3.7 percent versus 4.3 percent, respectively). Additionally, the absolute value of the relative bias for schools in towns and rural areas and large sized schools are greater than 10 percent, which indicates potential bias even though no statistically significant relationships were detected. When all factors were considered simultaneously in a logistic regression analysis (again with free or reduced-price lunch eligibility omitted), no variables were statistically significant predictors of participation. The model among public schools only showed that South region, high poverty, interaction between the high-poverty indicator and free or reduced-price lunch eligibility, and summed race/ethnicity were significant predictors of school participation.

In the final sample schools with substitutes when school nonresponse adjusted weights were used for the participating schools, schools in the Northeast were underrepresented among participating schools relative to eligible schools (15.6 percent versus 19.6 percent, respectively), while schools in the South were overrepresented among participating schools (47.7 percent versus 43.1 percent, respectively). Participating schools had a lower mean total school enrollment than the eligible sample (785.9 versus 857.6, respectively) and a lower mean grade-eligible enrollment than the eligible sample (375.6 versus 405.1, respectively). Participating schools had a lower mean percentage of Asian students than the eligible sample (3.7 percent versus 4.3 percent, respectively). Additionally, the absolute value of the relative bias for schools in towns and rural areas and large sized schools is again greater than 10 percent, which indicates potential bias even though no statistically significant relationship was detected. The multivariate regression analysis cannot be conducted after the school nonresponse adjustments are applied to the weights. The concept of nonresponse adjusted weights does not apply to the nonresponding units, and, thus, we cannot conduct an analysis that compares respondents with nonrespondents using nonresponse adjusted weights.

In summary, the investigation into nonresponse bias at the school level in the U.S. TALIS 2018 provides evidence that there is some potential for nonresponse bias in the TALIS participating original sample based on the characteristics studied. It also suggests that there is no evidence that the use of substitute schools reduced the potential for bias. Moreover, after the application of school nonresponse adjustments, there is some evidence of resulting potential bias in the available frame variables in the final sample with the largest relative bias in census region.

---

¹ The size-adjusted weight modifies the school base weight so that each sampled school is weighted in proportion to the number of teachers that it represents in the population. For considering potential nonresponse bias in TALIS, the estimated number of teachers of a school is a better measure of the relative size of a school than the estimated grade-eligible enrollment of a school, described in an earlier section, as TALIS is a survey of the population of teachers.

References

Ainley, J. and R. Carstens. (2018). Teaching and Learning International Survey (TALIS) 2018 Conceptual Framework, OECD Education Working Papers, No. 187, OECD Publishing, Paris. https://doi.org/10.1787/799337c2-en .

Judkins, D. R. (1990). Fay’s Method for Variance Estimation. Journal of Official Statistics 6(3): 223-239.

Kastberg, D., Cummings, L., Lemanski, N., Ferraro, D., Perkins, R.C., and Erberber, E. (forthcoming). U.S. Technical Report and User Guide for the 2018 Teaching and Learning International Survey (TALIS). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

Krenzke, T., Roey, S., Dohrmann, S., Mohadjer, L., Haung, W., Kaufman, S., and Seastrom, M. (2006). Tactics for Reducing the Risk of Disclosure Using the NCES DataSwap Software. Proceedings of the Survey Research Methods Section 2006. Alexandria, VA: American Statistical Association.

Organization for Economic Cooperation and Development (OECD). (2019). TALIS 2018 Technical Report. Paris: Author.