ESS Impact

Bibliographic Monitoring

Each year, an evaluation of the academic impact of the ESS is produced by Core Scientific Team (CST) member, Brina Malnar (University of Ljubljana). In-depth analysis of academic publications listed on Google Scholar and the ESS Bibliography informs this annual report. In order to evaluate the academic impact of our data, this monitoring provides:

• Longitudinal empirical evidence on the scope, geographical and disciplinary patterns
  of ESS academic usage, theory development and policy references;
• Empirical support for informed decision-making by ESS bodies (CST, SAB, and QDTs)
  regarding questionnaire content (item and module selection and revision), targeting
  training, communication strategies and similar;
• Summary bibliographic reports, a comprehensive list of citations with the option to
  create customized sub-lists based on various criteria (see Appendix 1) and item usage
  statistics (see Appendix 2). These documents help demonstrate ESS’s academic
  relevance to European and national funders and users (NCs, GA);
• Bibliographic repository for other work packages, utilized for methodological testing,
  updating of ESS online bibliography, supporting ESS communication efforts, and
  creating relevant outreach materials.
• The source of annual updates for the new ESS online bibliography, providing
  bibliographic variables accessible to the general audience and enabling users to
  browse ESS publications according to various criteria.

The latest research - published in January 2024 - found that 6,585 English-language academic publications include substantial primary analysis of our data (2003-22). This includes 3,719 articles published in peer-reviewed journals and 892 books/chapters.

The report also includes analysis of the journal disciplinary fields; research topics; country affiliation of authors; which countries are included in the analysis; the use of our questionnaire items; methodological awareness; and policy relevance of articles.

A brief summary of this latest report is available in our News section.

The final report is supplemented with two appendices. The bibliography includes reference information for all 6,585 publications. The item usage report highlights the number of times data from each ESS question has been analysed.

Lessons in impact

As part of SUSTAIN-2 (Grant Agreement 871063) Work Package 3 - Promoting ESS data among key European policy makers - three Impact Showcase Workshops were held. These events were delivered to General Assembly representatives on research (April 2021), teaching (October 2021) and policy (March 2023).

Presentations were delivered by Mirza Balaj (Centre for Global Health Inequalities Research (CHAIN), Dr. Theoni Stathopoulou (National Centre for Social Research, EKKE), Aurelija Stelmokienė (Vytautas Magnus University), Vera Messing and Bence Ságvári (Hungarian Academy of Sciences) and Jelena Angelis (EFIS Centre).

The materials from the these meetings were collated into a ‘Lessons in Impact' document. This report summarises and reflects upon these activities, and includes presentation slides in the appendix.

Policy Consultant

Funded by the ESS-SUSTAIN-2 project (Horizon 2020, grant agreement number 871063), a policy consultant was appointed in November 2022.

An open call was published for a consultant to investigate the use of ESS data amongst policy making audiences. Following the call, EFIS Centre were appointed to conduct the study and produce a report of their findings.

Finalised in November 2023, the report summarised ESS data use in policy making. This included: an overview of existing policy use, analysis of policy documents listed in the Overton.io database and interviews with key stakeholders.

The main section focused on the pathways to impact: how exactly our data and survey methods directly and indirectly contribute to the policy process in the European context. A final section offers some conclusions and recommendations for the ESS to increase its visibility within the policy arena. 

Impact Study: 2022

In February 2022, ESS ERIC published a new report: SUSTAIN-2: Impact study of the European Social Survey. The report was authored by Technopolis Group (UK).

Funded by the ESS-SUSTAIN-2 project (Horizon 2020, grant agreement number 871063), the study identifies academic, non-academic and teaching impact of ESS data.

This study was commissioned by the ESS ERIC and conducted by Technopolis, with bibliometric analysis from the Centre for Science and Technology Studies (CWTS) at the University of Leiden.

The report is a follow-up to the original impact study of the ESS, which was published in 2017.

Registered users

To access our data, users have to complete a short registration form. Analysis of data from this registration form by Technopolis Group (UK) found that, as of June 2021, there had been 182,778 registrations- almost double the figure at the start of the original Impact Study (June 2016).

The number of registered users has grown consistently by 14-15% in each of the last five years and, in the year up to June 2021, 74% registered users had downloaded our data.

Of the 50,000 non-student registered users who had accessed the data up until 2020, 7,712 had logged in and downloaded data within the calendar year.

Teaching impact

The report found that our data is increasingly used as a teaching tool, across participating countries and beyond and in an increasing number of institutions for both methodological and thematic courses.

Technopolis Group (UK) report that the actual number of students using our data is likely higher than our data suggests, as many lecturers download data for their students.

In 2016, there were 18 institutions with more than 500 confirmed users (including four with over 1,000). By the start of 2021, this increased to 42 (including 15 institutions with over 1,000 confirmed users).

Academic impact

Technopolis Group (UK) report that the overall number of academic publications including significant analysis of our data has increased by at least 150% since the first Impact Study.

Including various different publication types and non-English language publications, University of Ljubljana data suggests that there are over 7,500 publications in existence (the first Impact Study reported 2,704).

Centre for Science and Technology Studies (CWTS) was able to identify 2,448 academic publications listed on Web of Science, which compares to around 1,000 in the previous study.

The citation impact of these publications is well above average, being about 70% more highly cited than average, with 21% of all ESS publications belonging to the top 10%. The journals in which work is being published have a citation impact of 40% above the world average.

Our data is mostly used by ESS member countries in terms of publication, producing almost three quarters these.

Non-academic impact

Technopolis Group (UK) found many examples of non-academic impact, of many different types and across different domains, including:

• General intelligence and insight for NGOs or government ministries, agencies or advisory bodies
• Agenda setting by using our data to highlight a particular problem or challenge, triggering various types of policy action
• Influence on public debate or highlighting certain issues to the general public through presentation of our data or findings in the news media
• Monitoring: using our data as indicators to track certain aspects of societal progress, to help assess whether certain policies are achieving the desired outcomes

Impact Study: 2017

Funded by the ESS-SUSTAIN project (Horizon 2020, Grant Agreement 676166), Technopolis Group (UK) were tasked with undertaking an ESS ERIC Impact Study in June 2016.

Published in September 2017, the report assessed the academic, non-academic and teaching impact that has been achieved through the ESS. It also considered how these were achieved, what constitutes best practice and made recommendations to ensure the project’s long-term future.

In undertaking the study, Technopolis examined academic resources, sought out examples of non-academic use of our data, interviewed stakeholders and carried out an online survey of data users.

The report found that the ESS continues to be highly regarded amongst international values and attitudes surveys, offering extremely high quality data that is a critical academic resource for researchers across the social sciences.

It also found that ESS data is notable as a teaching resource, particularly within materials developed for use by students who are learning about survey methodology and quantitative data analysis.

The final report included bibliometric analysis undertaken by the Centre for Science and Technology Studies (CWTS), which found that the ESS online Bibliography underestimates the actual number of journal articles using our data by around 20%.

CWTS also discovered that most academic output using ESS data tends to perform highly on bibliometric indicators across a broad range of topics.

The impact study also found evidence of non-academic impact. Across member countries, many cases where ESS data was used in many different policy and practice contexts was found. This has helped to influence policy decisions and inform public and political debates.

The Technopolis report found that in some current member/observer countries, there is widespread use of the ESS as a teaching resource and in others the data is widely used in parliaments, ministries or government agencies.

Examples of where ESS data has been used by Governments or their agencies include the Austrian Ministry for Social Affairs, Estonia’s Ministry of Social Affairs, the French governmental think tank, France Stratégie, the German Federal Government, the Irish police force, Poland’s Central Statistical Office, the Portuguese Centre for Judicial Studies and the Institute of Macroeconomic Analysis and Development of the Republic of Slovenia.

ESS data has also helped inform international organisations such as Eurofound, United Nations Economic Commission for Europe (UNECE), European Commission’s DG EMPL (Employment, Social Affairs and Inclusion), OECD as well as think tanks including the Centre for European Policy Studies and European Policy Centre.

In making recommendations, the Technopolis report stressed the importance of continued funding and consistent involvement of countries over time. The addition of other European countries not already involved in the ESS would further help to ensure the long-term sustainability of the project.

A supplementary report was published in December 2018.