Director of Research, School of Humanities, University of Iceland<\/strong>
Eir\u00edkur Sm\u00e1ri Sigur\u00f0arson<\/strong><\/h2>\n\n\n\n\n\n![\"\"](\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\")
<\/figure>\n<\/div>\n\n\n\n\nSigur\u00f0arson is also active in ENRESSH (European Network for Research Evaluation in the Social Sciences and the Humanities), a COST Action supported by the European Union. As a member of one of its work groups that focuses on the social impact and relevance of social science and humanities research, he writes about the impact of humanities research, the societal meaning of which is not always immediately apparent.<\/p>\n<\/div>\n<\/div>\n\n\n\n
You work on the social impact of research. I\u2019d like to hear how you think about this issue. Iceland\u2019s population is about a twentieth of that of Japan. When people talk about the impact of research on society, how broad is this \u201csociety\u201d? Is it domestic society, European society, or the world community? In Japan, if we talk about the social impact of research, most researchers imagine Japanese society.<\/h4>\n\n\n\n
Eir\u00edkur Sigur\u00f0arson: First, let\u2019s distinguish between scientific impact and social impact. Scientific impact is not necessarily only local. For example, medieval Icelandic studies and Icelandic literature research are actually global research fields. Even research on more local historical phenomena or periods can have a scholarly impact that is global.<\/p>\n\n\n\n
In Iceland, it\u2019s usually the local population that is the focus of discussions when it comes to social impact. Perhaps in some research on specific local populations, it would be the impact on smaller communities. This issue\u2014the kind of society people have in mind when they say \u201csocial impact\u201d\u2014relates to the language of publications. In Iceland, as in many other places, there is a pull towards publishing in English. Usually, you get more research points at the University of Iceland if you publish in English than in Icelandic. Still, people publish a lot in Icelandic. Also, some, but not many, even write Ph.D. theses in Icelandic.<\/p>\n\n\n\n
Do students aiming to be researchers tend to go overseas?<\/h4>\n\n\n\n
Sigur\u00f0arson: Because we have such a small population, only 850 thousand people, and a fairly small number of higher education institutes, almost every lecturer, scholar, and professor at our universities has a degree from abroad. You actually spend time there as a Ph.D. student, and even as a postdoc. So in that sense, we are very international, and we have contacts all around. It\u2019s only recently that we\u2019ve started hiring academics with Ph.D.s from Iceland because our Ph.D. programs are only about fifteen or twenty years old.<\/p>\n\n\n\n
Research and Faculty Evaluations in Iceland: Present Situation and the Future<\/strong><\/h3>\n\n\n\nWhat is the research and faculty evaluation system like in Iceland? Such systems are a major topic of interest and discussion in Japan.<\/h4>\n\n\n\n
Sigur\u00f0arson: There are two systems. One is a national system that evaluates the quality of higher education in Iceland. It operates on a five-year cycle and is organized by the Icelandic Research Center, the country\u2019s central research council. The actual quality evaluations of universities are done by an entirely international board appointed by the Ministry of Education.<\/p>\n\n\n\n
Does it evaluate individual faculty members or institutions?<\/h4>\n\n\n\n
Sigur\u00f0arson: In the end, it\u2019s an institutional evaluation in the sense that the quality board issues a report for the University of Iceland as a whole. But for this evaluation, faculty members have to submit a self-evaluation report. So it\u2019s also an evaluation of individual faculty members.<\/p>\n\n\n\n
In addition to the national system, there\u2019s the evaluation of research within the university. Every scholar has to submit a report every year, detailing all publications, presentations, public engagements, and so on in the year before. Based on this, they are rated. For example, if you publish in a top-class national journal, you get five to fifteen points. If you publish in a high-impact international journal, you get twenty research points. You can get up to one hundred points for a monograph, if it\u2019s one of the leading publishers, like Cambridge University Press or Oxford University Press. And for a presentation, like the one I gave yesterday at my workshop, I guess I get three points. (Laughs<\/em>).<\/p>\n\n\n\nHappy that it was useful!<\/h4>\n\n\n\n
Sigur\u00f0arson: Then you get a certain number, I think the average within universities is about 35 points per year and in the humanities, we\u2019re slightly above average.<\/p>\n\n\n\n
And then you actually get a bonus payment in September for the research points you earn. So it\u2019s a very direct incentive and strongly influences how you behave. I don\u2019t think it\u2019s an entirely bad system. It\u2019s very transparent, nothing hidden there\u2014just a list of what you can get points for. And it\u2019s rather open, so we get a lot of points in the humanities. However, some find it difficult to actually thrive within the system, such as researchers or teachers in the languages. This is because they are basically teaching faculty. The University of Iceland was basically a teaching university, but a few decades ago, the decision was made to change it into a research university. Part of this was adopting this publishing incentive system. And it has worked. Publications have been skyrocketing, as have citations.<\/p>\n\n\n\n
Of course, this system\u2019s problems have also been discussed a lot. The system pushes researchers to do something, possibly leading them away from other activities that would be more useful to society. A problem that has been particularly discussed lately is that this system is used for decision-making in different contexts. For example, when Ph.D. students apply for funding within the university, the research points of the supervisor count. This is also used when the university is distributing funding to its faculties. A considerable amount of the funding is divided based on research activities, affecting how research is done. So now the universities are trying to scale this point system back.<\/p>\n\n\n\n
With that said, in Iceland, the Association of Professors and the Association of University Lecturers sign employment contracts with the Treasury. And this evaluation system is a part of these contracts. So it is not something the university can change on its own. Scholars are very much for this system\u2014maybe also for changing it, perhaps for scaling back how it\u2019s used, but not for abolishing it. So I think we\u2019re going to keep it.<\/p>\n\n\n\n
The Research Network ENRESSH: Organization and Activities<\/h3>\n\n\n\nYou are participating in the European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH). Could you talk a bit about this network?<\/h4>\n\n\n\n
Sigur\u00f0arson: ENRESSH<\/a> is a European network of people doing research on research evaluations, on the social impact of research, and so on. The people who lead and started the network are actually scholars in the humanities. A part of the group is practitioners like us, from research administration.<\/p>\n\n\n\nThe network is divided into different work groups. So, for example, there is one work group heavily focused on bibliometrics. They are working on, for example, how to incorporate books into research evaluations. There is also a group working on the language of publications. They produce something called the Helsinki Initiative<\/a>, which is about respecting domestic languages, not just English, when it comes to research publications in social science and humanities. This is all really valuable for humanities and social sciences research.<\/p>\n\n\n\nAnd then there is a special interest group working on young scholars. It does research from the perspective of the metrics debate and social impact debate, asking how they can get recognition to have some kind of career in the social sciences and humanities. It also asks about the career obstacles they face.<\/p>\n\n\n\n
So the problems of metrics and career are closely linked. You\u2019re part of the group researching social impact. What are its accomplishments?<\/h4>\n\n\n\n
Sigur\u00f0arson: We\u2019ve collected social impact case studies from the social sciences and humanities. Most of them were from the social sciences, but a considerable bit from the humanities as well. The work has concentrated a lot on these cases. A number of papers have already been published based on them. There\u2019s one on social impact pathways. Another compares what they call impact capacity between Eastern Europe and Western Europe. In other words, between countries where the impact debate hasn\u2019t really taken off and countries where it has and people are used to it. Right now, the group\u2019s doing social impact research on Nordic countries, and there are also many other papers being produced.<\/p>\n\n\n\n
Is ENRESSH involved in policymaking?<\/h4>\n\n\n\n
Sigur\u00f0arson: The network is funded by COST, a funding scheme now within the European Framework program. So, you could say that it is a part of the European Union. And one of the objectives of this network is to produce policy papers. It networks scholars within Europe working on some particular field, collects the research they are doing, and then often compares countries in this regard. Among its output is supposed to be something that is policy relevant.<\/p>\n\n\n\n
So is there some kind of coordinator that translates the research into policy?<\/h4>\n\n\n\n
Sigur\u00f0arson: The coordinating and mediating work is actually part of our project. The network will produce some recommendations for how to include humanities and social sciences in framework program activities and European Research funding activities, as well as suggestions on how to evaluate the contribution of humanities and social sciences. So these should be concrete and applicable recommendations, more targeted, I think, at a bit lower policymaking level. ENRESSH\u2019s target is the people who are actually figuring out how to run the programs and evaluate research.<\/p>\n\n\n\n
Monographs are a case in point. It is difficult to have them properly recognized and evaluated in the current systems. So the idea is for us to produce concrete recommendations on how to include monograph publications in both pre-funding and post-funding evaluations. As for social impact, we wrote articles based on our papers from the 2017 RESSH conference in Antwerp. In 2019, they will be published in a special section of the journal Research Evaluation<\/a>. The ENRESSH policy group will submit recommendations that include their content to the commission. It will also lobby people who grant research funding and innovation funding in Europe, as well as other actors who are dealing with impact evaluations at the European level.<\/p>\n\n\n\nIs there a particular instance that comes to mind where this lobbying yielded concrete policies?<\/h4>\n\n\n\n
Sigur\u00f0arson: Not yet. I don\u2019t think I can point to anything that\u2019s already happened. But I know the ENRESSH policy group has been active in Brussels.<\/p>\n\n\n\n
There\u2019s a tendency for a group like this to focus very much on the social sciences. This is probably because most of the scholars involved in the field of research evaluation are social scientists. As a humanities scholar, you can feel slightly marginalized just by the number of things going on. Of course, though, overall it\u2019s a very comfortable environment. (Laughs).<\/p>\n\n\n\n
From Indicators and Metrics to \u201cWhat is the Good We Are Doing?\u201d<\/h3>\n\n\n\nToday\u2019s research evaluation problems appear to have come from research evaluation\u2019s means becoming the objective. In other words, meeting research metrics has become the goal of research, and, accordingly, research inspiration is tending to atrophy. People say there\u2019s a gap between evaluation for resource allocation, on the one hand, and evaluation for promoting quality research, on the other. What do you think should be done so that research evaluation is not an obstacle to research but instead advances it?<\/h4>\n\n\n\n
Sigur\u00f0arson: A major thing I\u2019m trying to promote is properly understanding the priority order of the questions we ask about research evaluations. This is explained and analyzed very well by Jerry Z. Muller in his book The Tyranny of Metrics<\/a>. So, the question goes from indicators and metrics to the question of, \u201cWhat is the good we are doing?\u201d<\/p>\n\n\n\nThis is because metrics and indicators are only applicable in some cases, not all. If the debate\u2019s priority is metrics and indicators, you risk losing out, or ignoring, many valuable aspects of research that do not fit the metrics debate. Therefore, first one should ask, \u201cWhat is the good that this researcher is doing?\u201d Then you discuss how we will evaluate the methods and quality of their work and whether we can use metrics and indicators. It would be just a waste of time to apply indicators to all cases. So our objective is to reverse the priorities.<\/p>\n\n\n\n
Then, we should divide research into (1) fields where we can use some kind of metrics or tools, (2) fields where we need to use peer review, and (3) fields where we simply cannot use any formal methods. In (3), we are left with an informal policy debate on what we actually want for society. In such cases, one applies the likes of Martha Nussbaum\u2019s capabilities approach to discuss the social impact of the research.(1)<\/sup><\/p>\n\n\n\nThis is the most effective method that we should use first. This debate can\u2019t be avoided. When it comes to the value of the humanities for society, humanities scholars need to have their own voice, they need to be able to speak on their own terms.<\/p>\n\n\n\n
Shifting the focus away from metrics and indicators sounds very useful for research evaluation that promotes research. Going back to the basics.<\/p>\n\n\n\n
The Role of URAs in Research Evaluation<\/strong><\/h3>\n\n\n\nWhat do you think is the role of research administrators and research managers in research evaluation?<\/h4>\n\n\n\n
Sigur\u00f0arson: It depends a lot on people\u2019s position and the context. Say you are a research administrator assisting people writing funding applications, and the fund is asking for cogent argument about the research\u2019s use to society. In that case, your role might be precisely assisting them in writing or describing\u2014and indeed just understanding\u2014the social impact of their research. If one follows the debate regarding social impact even a little bit, one can express it using more appropriate terms.<\/p>\n\n\n\n
In my experience discussing this with scholars, there is a tendency to understand the question very narrowly. When asked to describe their research\u2019s social impact, some think, \u201cOkay, they are asking me how I am going to benefit the economy,\u201d or that the question is, \u201cHow is this research going to have a direct influence on policy?\u201d Even some people say, \u201cNo, I\u2019m researching medieval Icelandic literature, and there\u2019s nothing in what I\u2019m doing that\u2019s going to change anything in the economy.\u201d So one role of URAs would be to assist people in describing their social impact more broadly, to understand in a wider context what the social impacts might be.<\/p>\n\n\n\n
If you\u2019re working on the humanities and social sciences in an institute that is more focused on the hard sciences in general, then a role would be to keep arguing, at the institute\u2019s policy level, for the social good produced by the social sciences and humanities. Talking about what would be lost by your institute if we stopped or reduced funding or supporting the humanities.<\/p>\n\n\n\n
I have been working a lot with Ph.D. students in the humanities in Iceland. If working with young researchers, that is, early career researchers, is part of your remit, your role would also be helping them develop ideas and formulate arguments about what they\u2019re doing. Maybe pointing to some questions that they might ask themselves about their research, that they may not have thought of themselves, as well.<\/p>\n\n\n\n
Do you think that using the social impact approach is effective even for young researchers moving on to something other than research? In other words, when they have to articulate what they have been doing up through the present.<\/h4>\n\n\n\n
Sigur\u00f0arson: Yes. One can ask, \u201cWhat is the social benefit of you as an individual getting a research education, writing a thesis, and developing yourself as a researcher?\u201d If they gain skills that they can use in other contexts, it would be a loss to society if these weren\u2019t developed. That personal development, that kind of capacity building of yourself as an individual and the people around you, is beneficial to society.<\/p>\n\n\n\n
\n\n\n\n(1)<\/sup> This interview was carried out on 21 May 2019. Sigur\u00f0arson was at Kyoto University to give a KURA workshop<\/a>. For information on concrete methods for advancing discussions, refer to presentation slides<\/a> from Sigur\u00f0arson\u2019s KURA workshop.<\/p>\n\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
*This article is based on content previously published on \u201cLife as a scholar: Kyoto University and Beyond,\u201d which has since been discontinued. Today, scholars are expected to return the fruits of their research to society and address social problems. Instead of fitting into society\u2019s demands and expectations, how can those working on humanities, social science, or basic science research talk about social impact? We discussed such issues with the University of Iceland\u2019s Eir\u00edkur Sigur\u00f0arson. Focusing on social impact, he is working to create new research evaluation methods.(1) Director of Research, School of Humanities, University of IcelandEir\u00edkur Sm\u00e1ri Sigur\u00f0arson Sigur\u00f0arson is also active in ENRESSH (European Network for Research Evaluation in\u2025<\/p>\n","protected":false},"author":2,"featured_media":0,"template":"","meta":{"_acf_changed":false,"_crdt_document":"","exclusive":false,"relatedArticles":[],"footnotes":""},"tags":[75],"class_list":["post-2256","blog","type-blog","status-publish","hentry","tag-next-generation-forum"],"acf":[],"yoast_head":"
Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration<\/title>\n<meta name=\"description\" content=\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration\" \/>\n<meta property=\"og:description\" content=\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\" \/>\n<meta property=\"og:site_name\" content=\"Kyoto University Office of Research Acceleration\" \/>\n<meta property=\"article:modified_time\" content=\"2025-07-15T01:26:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\",\"url\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\",\"name\":\"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration\",\"isPartOf\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\",\"datePublished\":\"2020-03-26T07:13:00+00:00\",\"dateModified\":\"2025-07-15T01:26:39+00:00\",\"description\":\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage\",\"url\":\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\",\"contentUrl\":\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Topics\",\"item\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Research Evaluation that Promotes Actual Research: Focusing on Social Impact\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website\",\"url\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/\",\"name\":\"Kyoto University Office of Research Acceleration\",\"description\":\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>","yoast_head_json":{"title":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration","description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","og_locale":"en_US","og_type":"article","og_title":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration","og_description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","og_url":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","og_site_name":"Kyoto University Office of Research Acceleration","article_modified_time":"2025-07-15T01:26:39+00:00","og_image":[{"url":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg","width":1024,"height":768,"type":"image\/jpeg"}],"twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","url":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","name":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration","isPartOf":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage"},"image":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage"},"thumbnailUrl":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg","datePublished":"2020-03-26T07:13:00+00:00","dateModified":"2025-07-15T01:26:39+00:00","description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","breadcrumb":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage","url":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg","contentUrl":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.research.kyoto-u.ac.jp\/en\/"},{"@type":"ListItem","position":2,"name":"Topics","item":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/"},{"@type":"ListItem","position":3,"name":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact"}]},{"@type":"WebSite","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website","url":"https:\/\/www.research.kyoto-u.ac.jp\/en\/","name":"Kyoto University Office of Research Acceleration","description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.research.kyoto-u.ac.jp\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"featured_media_url":false,"_links":{"self":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog\/2256","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/users\/2"}],"version-history":[{"count":3,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog\/2256\/revisions"}],"predecessor-version":[{"id":2621,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog\/2256\/revisions\/2621"}],"wp:attachment":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/media?parent=2256"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/tags?post=2256"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
<\/figure>\n<\/div>\n\n\n\nSigur\u00f0arson is also active in ENRESSH (European Network for Research Evaluation in the Social Sciences and the Humanities), a COST Action supported by the European Union. As a member of one of its work groups that focuses on the social impact and relevance of social science and humanities research, he writes about the impact of humanities research, the societal meaning of which is not always immediately apparent.<\/p>\n<\/div>\n<\/div>\n\n\n\n
You work on the social impact of research. I\u2019d like to hear how you think about this issue. Iceland\u2019s population is about a twentieth of that of Japan. When people talk about the impact of research on society, how broad is this \u201csociety\u201d? Is it domestic society, European society, or the world community? In Japan, if we talk about the social impact of research, most researchers imagine Japanese society.<\/h4>\n\n\n\n
Eir\u00edkur Sigur\u00f0arson: First, let\u2019s distinguish between scientific impact and social impact. Scientific impact is not necessarily only local. For example, medieval Icelandic studies and Icelandic literature research are actually global research fields. Even research on more local historical phenomena or periods can have a scholarly impact that is global.<\/p>\n\n\n\n
In Iceland, it\u2019s usually the local population that is the focus of discussions when it comes to social impact. Perhaps in some research on specific local populations, it would be the impact on smaller communities. This issue\u2014the kind of society people have in mind when they say \u201csocial impact\u201d\u2014relates to the language of publications. In Iceland, as in many other places, there is a pull towards publishing in English. Usually, you get more research points at the University of Iceland if you publish in English than in Icelandic. Still, people publish a lot in Icelandic. Also, some, but not many, even write Ph.D. theses in Icelandic.<\/p>\n\n\n\n
Do students aiming to be researchers tend to go overseas?<\/h4>\n\n\n\n
Sigur\u00f0arson: Because we have such a small population, only 850 thousand people, and a fairly small number of higher education institutes, almost every lecturer, scholar, and professor at our universities has a degree from abroad. You actually spend time there as a Ph.D. student, and even as a postdoc. So in that sense, we are very international, and we have contacts all around. It\u2019s only recently that we\u2019ve started hiring academics with Ph.D.s from Iceland because our Ph.D. programs are only about fifteen or twenty years old.<\/p>\n\n\n\n
Research and Faculty Evaluations in Iceland: Present Situation and the Future<\/strong><\/h3>\n\n\n\nWhat is the research and faculty evaluation system like in Iceland? Such systems are a major topic of interest and discussion in Japan.<\/h4>\n\n\n\n
Sigur\u00f0arson: There are two systems. One is a national system that evaluates the quality of higher education in Iceland. It operates on a five-year cycle and is organized by the Icelandic Research Center, the country\u2019s central research council. The actual quality evaluations of universities are done by an entirely international board appointed by the Ministry of Education.<\/p>\n\n\n\n
Does it evaluate individual faculty members or institutions?<\/h4>\n\n\n\n
Sigur\u00f0arson: In the end, it\u2019s an institutional evaluation in the sense that the quality board issues a report for the University of Iceland as a whole. But for this evaluation, faculty members have to submit a self-evaluation report. So it\u2019s also an evaluation of individual faculty members.<\/p>\n\n\n\n
In addition to the national system, there\u2019s the evaluation of research within the university. Every scholar has to submit a report every year, detailing all publications, presentations, public engagements, and so on in the year before. Based on this, they are rated. For example, if you publish in a top-class national journal, you get five to fifteen points. If you publish in a high-impact international journal, you get twenty research points. You can get up to one hundred points for a monograph, if it\u2019s one of the leading publishers, like Cambridge University Press or Oxford University Press. And for a presentation, like the one I gave yesterday at my workshop, I guess I get three points. (Laughs<\/em>).<\/p>\n\n\n\nHappy that it was useful!<\/h4>\n\n\n\n
Sigur\u00f0arson: Then you get a certain number, I think the average within universities is about 35 points per year and in the humanities, we\u2019re slightly above average.<\/p>\n\n\n\n
And then you actually get a bonus payment in September for the research points you earn. So it\u2019s a very direct incentive and strongly influences how you behave. I don\u2019t think it\u2019s an entirely bad system. It\u2019s very transparent, nothing hidden there\u2014just a list of what you can get points for. And it\u2019s rather open, so we get a lot of points in the humanities. However, some find it difficult to actually thrive within the system, such as researchers or teachers in the languages. This is because they are basically teaching faculty. The University of Iceland was basically a teaching university, but a few decades ago, the decision was made to change it into a research university. Part of this was adopting this publishing incentive system. And it has worked. Publications have been skyrocketing, as have citations.<\/p>\n\n\n\n
Of course, this system\u2019s problems have also been discussed a lot. The system pushes researchers to do something, possibly leading them away from other activities that would be more useful to society. A problem that has been particularly discussed lately is that this system is used for decision-making in different contexts. For example, when Ph.D. students apply for funding within the university, the research points of the supervisor count. This is also used when the university is distributing funding to its faculties. A considerable amount of the funding is divided based on research activities, affecting how research is done. So now the universities are trying to scale this point system back.<\/p>\n\n\n\n
With that said, in Iceland, the Association of Professors and the Association of University Lecturers sign employment contracts with the Treasury. And this evaluation system is a part of these contracts. So it is not something the university can change on its own. Scholars are very much for this system\u2014maybe also for changing it, perhaps for scaling back how it\u2019s used, but not for abolishing it. So I think we\u2019re going to keep it.<\/p>\n\n\n\n
The Research Network ENRESSH: Organization and Activities<\/h3>\n\n\n\nYou are participating in the European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH). Could you talk a bit about this network?<\/h4>\n\n\n\n
Sigur\u00f0arson: ENRESSH<\/a> is a European network of people doing research on research evaluations, on the social impact of research, and so on. The people who lead and started the network are actually scholars in the humanities. A part of the group is practitioners like us, from research administration.<\/p>\n\n\n\nThe network is divided into different work groups. So, for example, there is one work group heavily focused on bibliometrics. They are working on, for example, how to incorporate books into research evaluations. There is also a group working on the language of publications. They produce something called the Helsinki Initiative<\/a>, which is about respecting domestic languages, not just English, when it comes to research publications in social science and humanities. This is all really valuable for humanities and social sciences research.<\/p>\n\n\n\nAnd then there is a special interest group working on young scholars. It does research from the perspective of the metrics debate and social impact debate, asking how they can get recognition to have some kind of career in the social sciences and humanities. It also asks about the career obstacles they face.<\/p>\n\n\n\n
So the problems of metrics and career are closely linked. You\u2019re part of the group researching social impact. What are its accomplishments?<\/h4>\n\n\n\n
Sigur\u00f0arson: We\u2019ve collected social impact case studies from the social sciences and humanities. Most of them were from the social sciences, but a considerable bit from the humanities as well. The work has concentrated a lot on these cases. A number of papers have already been published based on them. There\u2019s one on social impact pathways. Another compares what they call impact capacity between Eastern Europe and Western Europe. In other words, between countries where the impact debate hasn\u2019t really taken off and countries where it has and people are used to it. Right now, the group\u2019s doing social impact research on Nordic countries, and there are also many other papers being produced.<\/p>\n\n\n\n
Is ENRESSH involved in policymaking?<\/h4>\n\n\n\n
Sigur\u00f0arson: The network is funded by COST, a funding scheme now within the European Framework program. So, you could say that it is a part of the European Union. And one of the objectives of this network is to produce policy papers. It networks scholars within Europe working on some particular field, collects the research they are doing, and then often compares countries in this regard. Among its output is supposed to be something that is policy relevant.<\/p>\n\n\n\n
So is there some kind of coordinator that translates the research into policy?<\/h4>\n\n\n\n
Sigur\u00f0arson: The coordinating and mediating work is actually part of our project. The network will produce some recommendations for how to include humanities and social sciences in framework program activities and European Research funding activities, as well as suggestions on how to evaluate the contribution of humanities and social sciences. So these should be concrete and applicable recommendations, more targeted, I think, at a bit lower policymaking level. ENRESSH\u2019s target is the people who are actually figuring out how to run the programs and evaluate research.<\/p>\n\n\n\n
Monographs are a case in point. It is difficult to have them properly recognized and evaluated in the current systems. So the idea is for us to produce concrete recommendations on how to include monograph publications in both pre-funding and post-funding evaluations. As for social impact, we wrote articles based on our papers from the 2017 RESSH conference in Antwerp. In 2019, they will be published in a special section of the journal Research Evaluation<\/a>. The ENRESSH policy group will submit recommendations that include their content to the commission. It will also lobby people who grant research funding and innovation funding in Europe, as well as other actors who are dealing with impact evaluations at the European level.<\/p>\n\n\n\nIs there a particular instance that comes to mind where this lobbying yielded concrete policies?<\/h4>\n\n\n\n
Sigur\u00f0arson: Not yet. I don\u2019t think I can point to anything that\u2019s already happened. But I know the ENRESSH policy group has been active in Brussels.<\/p>\n\n\n\n
There\u2019s a tendency for a group like this to focus very much on the social sciences. This is probably because most of the scholars involved in the field of research evaluation are social scientists. As a humanities scholar, you can feel slightly marginalized just by the number of things going on. Of course, though, overall it\u2019s a very comfortable environment. (Laughs).<\/p>\n\n\n\n
From Indicators and Metrics to \u201cWhat is the Good We Are Doing?\u201d<\/h3>\n\n\n\nToday\u2019s research evaluation problems appear to have come from research evaluation\u2019s means becoming the objective. In other words, meeting research metrics has become the goal of research, and, accordingly, research inspiration is tending to atrophy. People say there\u2019s a gap between evaluation for resource allocation, on the one hand, and evaluation for promoting quality research, on the other. What do you think should be done so that research evaluation is not an obstacle to research but instead advances it?<\/h4>\n\n\n\n
Sigur\u00f0arson: A major thing I\u2019m trying to promote is properly understanding the priority order of the questions we ask about research evaluations. This is explained and analyzed very well by Jerry Z. Muller in his book The Tyranny of Metrics<\/a>. So, the question goes from indicators and metrics to the question of, \u201cWhat is the good we are doing?\u201d<\/p>\n\n\n\nThis is because metrics and indicators are only applicable in some cases, not all. If the debate\u2019s priority is metrics and indicators, you risk losing out, or ignoring, many valuable aspects of research that do not fit the metrics debate. Therefore, first one should ask, \u201cWhat is the good that this researcher is doing?\u201d Then you discuss how we will evaluate the methods and quality of their work and whether we can use metrics and indicators. It would be just a waste of time to apply indicators to all cases. So our objective is to reverse the priorities.<\/p>\n\n\n\n
Then, we should divide research into (1) fields where we can use some kind of metrics or tools, (2) fields where we need to use peer review, and (3) fields where we simply cannot use any formal methods. In (3), we are left with an informal policy debate on what we actually want for society. In such cases, one applies the likes of Martha Nussbaum\u2019s capabilities approach to discuss the social impact of the research.(1)<\/sup><\/p>\n\n\n\nThis is the most effective method that we should use first. This debate can\u2019t be avoided. When it comes to the value of the humanities for society, humanities scholars need to have their own voice, they need to be able to speak on their own terms.<\/p>\n\n\n\n
Shifting the focus away from metrics and indicators sounds very useful for research evaluation that promotes research. Going back to the basics.<\/p>\n\n\n\n
The Role of URAs in Research Evaluation<\/strong><\/h3>\n\n\n\nWhat do you think is the role of research administrators and research managers in research evaluation?<\/h4>\n\n\n\n
Sigur\u00f0arson: It depends a lot on people\u2019s position and the context. Say you are a research administrator assisting people writing funding applications, and the fund is asking for cogent argument about the research\u2019s use to society. In that case, your role might be precisely assisting them in writing or describing\u2014and indeed just understanding\u2014the social impact of their research. If one follows the debate regarding social impact even a little bit, one can express it using more appropriate terms.<\/p>\n\n\n\n
In my experience discussing this with scholars, there is a tendency to understand the question very narrowly. When asked to describe their research\u2019s social impact, some think, \u201cOkay, they are asking me how I am going to benefit the economy,\u201d or that the question is, \u201cHow is this research going to have a direct influence on policy?\u201d Even some people say, \u201cNo, I\u2019m researching medieval Icelandic literature, and there\u2019s nothing in what I\u2019m doing that\u2019s going to change anything in the economy.\u201d So one role of URAs would be to assist people in describing their social impact more broadly, to understand in a wider context what the social impacts might be.<\/p>\n\n\n\n
If you\u2019re working on the humanities and social sciences in an institute that is more focused on the hard sciences in general, then a role would be to keep arguing, at the institute\u2019s policy level, for the social good produced by the social sciences and humanities. Talking about what would be lost by your institute if we stopped or reduced funding or supporting the humanities.<\/p>\n\n\n\n
I have been working a lot with Ph.D. students in the humanities in Iceland. If working with young researchers, that is, early career researchers, is part of your remit, your role would also be helping them develop ideas and formulate arguments about what they\u2019re doing. Maybe pointing to some questions that they might ask themselves about their research, that they may not have thought of themselves, as well.<\/p>\n\n\n\n
Do you think that using the social impact approach is effective even for young researchers moving on to something other than research? In other words, when they have to articulate what they have been doing up through the present.<\/h4>\n\n\n\n
Sigur\u00f0arson: Yes. One can ask, \u201cWhat is the social benefit of you as an individual getting a research education, writing a thesis, and developing yourself as a researcher?\u201d If they gain skills that they can use in other contexts, it would be a loss to society if these weren\u2019t developed. That personal development, that kind of capacity building of yourself as an individual and the people around you, is beneficial to society.<\/p>\n\n\n\n
\n\n\n\n(1)<\/sup> This interview was carried out on 21 May 2019. Sigur\u00f0arson was at Kyoto University to give a KURA workshop<\/a>. For information on concrete methods for advancing discussions, refer to presentation slides<\/a> from Sigur\u00f0arson\u2019s KURA workshop.<\/p>\n\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
*This article is based on content previously published on \u201cLife as a scholar: Kyoto University and Beyond,\u201d which has since been discontinued. Today, scholars are expected to return the fruits of their research to society and address social problems. Instead of fitting into society\u2019s demands and expectations, how can those working on humanities, social science, or basic science research talk about social impact? We discussed such issues with the University of Iceland\u2019s Eir\u00edkur Sigur\u00f0arson. Focusing on social impact, he is working to create new research evaluation methods.(1) Director of Research, School of Humanities, University of IcelandEir\u00edkur Sm\u00e1ri Sigur\u00f0arson Sigur\u00f0arson is also active in ENRESSH (European Network for Research Evaluation in\u2025<\/p>\n","protected":false},"author":2,"featured_media":0,"template":"","meta":{"_acf_changed":false,"_crdt_document":"","exclusive":false,"relatedArticles":[],"footnotes":""},"tags":[75],"class_list":["post-2256","blog","type-blog","status-publish","hentry","tag-next-generation-forum"],"acf":[],"yoast_head":"
Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration<\/title>\n<meta name=\"description\" content=\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration\" \/>\n<meta property=\"og:description\" content=\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\" \/>\n<meta property=\"og:site_name\" content=\"Kyoto University Office of Research Acceleration\" \/>\n<meta property=\"article:modified_time\" content=\"2025-07-15T01:26:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\",\"url\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\",\"name\":\"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration\",\"isPartOf\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\",\"datePublished\":\"2020-03-26T07:13:00+00:00\",\"dateModified\":\"2025-07-15T01:26:39+00:00\",\"description\":\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage\",\"url\":\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\",\"contentUrl\":\"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Topics\",\"item\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Research Evaluation that Promotes Actual Research: Focusing on Social Impact\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website\",\"url\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/\",\"name\":\"Kyoto University Office of Research Acceleration\",\"description\":\"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.research.kyoto-u.ac.jp\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>","yoast_head_json":{"title":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration","description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","og_locale":"en_US","og_type":"article","og_title":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration","og_description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","og_url":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","og_site_name":"Kyoto University Office of Research Acceleration","article_modified_time":"2025-07-15T01:26:39+00:00","og_image":[{"url":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg","width":1024,"height":768,"type":"image\/jpeg"}],"twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","url":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/","name":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact | Kyoto University Office of Research Acceleration","isPartOf":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage"},"image":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage"},"thumbnailUrl":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg","datePublished":"2020-03-26T07:13:00+00:00","dateModified":"2025-07-15T01:26:39+00:00","description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","breadcrumb":{"@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#primaryimage","url":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg","contentUrl":"https:\/\/www.research.kyoto-u.ac.jp\/app\/uploads\/sites\/2\/2020\/03\/societal-impact.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/889\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.research.kyoto-u.ac.jp\/en\/"},{"@type":"ListItem","position":2,"name":"Topics","item":"https:\/\/www.research.kyoto-u.ac.jp\/en\/blog\/"},{"@type":"ListItem","position":3,"name":"Research Evaluation that Promotes Actual Research: Focusing on Social Impact"}]},{"@type":"WebSite","@id":"https:\/\/www.research.kyoto-u.ac.jp\/en\/#website","url":"https:\/\/www.research.kyoto-u.ac.jp\/en\/","name":"Kyoto University Office of Research Acceleration","description":"Kyoto University Office of Research Acceleration, KURA, was established as a new support organization to advance strategic research promotion and strengthen organizational frameworks through collaboration between faculty, administrative staff, and specialized personnel, based on appropriate analysis of research activities. It supports the free and outstanding \u201ccreation of knowledge\u201d across all fields at Kyoto University.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.research.kyoto-u.ac.jp\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"featured_media_url":false,"_links":{"self":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog\/2256","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/users\/2"}],"version-history":[{"count":3,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog\/2256\/revisions"}],"predecessor-version":[{"id":2621,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/blog\/2256\/revisions\/2621"}],"wp:attachment":[{"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/media?parent=2256"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.research.kyoto-u.ac.jp\/en\/wp-json\/wp\/v2\/tags?post=2256"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Happy that it was useful!<\/h4>\n\n\n\n
Sigur\u00f0arson: Then you get a certain number, I think the average within universities is about 35 points per year and in the humanities, we\u2019re slightly above average.<\/p>\n\n\n\n
And then you actually get a bonus payment in September for the research points you earn. So it\u2019s a very direct incentive and strongly influences how you behave. I don\u2019t think it\u2019s an entirely bad system. It\u2019s very transparent, nothing hidden there\u2014just a list of what you can get points for. And it\u2019s rather open, so we get a lot of points in the humanities. However, some find it difficult to actually thrive within the system, such as researchers or teachers in the languages. This is because they are basically teaching faculty. The University of Iceland was basically a teaching university, but a few decades ago, the decision was made to change it into a research university. Part of this was adopting this publishing incentive system. And it has worked. Publications have been skyrocketing, as have citations.<\/p>\n\n\n\n
Of course, this system\u2019s problems have also been discussed a lot. The system pushes researchers to do something, possibly leading them away from other activities that would be more useful to society. A problem that has been particularly discussed lately is that this system is used for decision-making in different contexts. For example, when Ph.D. students apply for funding within the university, the research points of the supervisor count. This is also used when the university is distributing funding to its faculties. A considerable amount of the funding is divided based on research activities, affecting how research is done. So now the universities are trying to scale this point system back.<\/p>\n\n\n\n
With that said, in Iceland, the Association of Professors and the Association of University Lecturers sign employment contracts with the Treasury. And this evaluation system is a part of these contracts. So it is not something the university can change on its own. Scholars are very much for this system\u2014maybe also for changing it, perhaps for scaling back how it\u2019s used, but not for abolishing it. So I think we\u2019re going to keep it.<\/p>\n\n\n\n
The Research Network ENRESSH: Organization and Activities<\/h3>\n\n\n\nYou are participating in the European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH). Could you talk a bit about this network?<\/h4>\n\n\n\n
Sigur\u00f0arson: ENRESSH<\/a> is a European network of people doing research on research evaluations, on the social impact of research, and so on. The people who lead and started the network are actually scholars in the humanities. A part of the group is practitioners like us, from research administration.<\/p>\n\n\n\n The network is divided into different work groups. So, for example, there is one work group heavily focused on bibliometrics. They are working on, for example, how to incorporate books into research evaluations. There is also a group working on the language of publications. They produce something called the Helsinki Initiative<\/a>, which is about respecting domestic languages, not just English, when it comes to research publications in social science and humanities. This is all really valuable for humanities and social sciences research.<\/p>\n\n\n\n And then there is a special interest group working on young scholars. It does research from the perspective of the metrics debate and social impact debate, asking how they can get recognition to have some kind of career in the social sciences and humanities. It also asks about the career obstacles they face.<\/p>\n\n\n\n Sigur\u00f0arson: We\u2019ve collected social impact case studies from the social sciences and humanities. Most of them were from the social sciences, but a considerable bit from the humanities as well. The work has concentrated a lot on these cases. A number of papers have already been published based on them. There\u2019s one on social impact pathways. Another compares what they call impact capacity between Eastern Europe and Western Europe. In other words, between countries where the impact debate hasn\u2019t really taken off and countries where it has and people are used to it. Right now, the group\u2019s doing social impact research on Nordic countries, and there are also many other papers being produced.<\/p>\n\n\n\n Sigur\u00f0arson: The network is funded by COST, a funding scheme now within the European Framework program. So, you could say that it is a part of the European Union. And one of the objectives of this network is to produce policy papers. It networks scholars within Europe working on some particular field, collects the research they are doing, and then often compares countries in this regard. Among its output is supposed to be something that is policy relevant.<\/p>\n\n\n\n Sigur\u00f0arson: The coordinating and mediating work is actually part of our project. The network will produce some recommendations for how to include humanities and social sciences in framework program activities and European Research funding activities, as well as suggestions on how to evaluate the contribution of humanities and social sciences. So these should be concrete and applicable recommendations, more targeted, I think, at a bit lower policymaking level. ENRESSH\u2019s target is the people who are actually figuring out how to run the programs and evaluate research.<\/p>\n\n\n\n Monographs are a case in point. It is difficult to have them properly recognized and evaluated in the current systems. So the idea is for us to produce concrete recommendations on how to include monograph publications in both pre-funding and post-funding evaluations. As for social impact, we wrote articles based on our papers from the 2017 RESSH conference in Antwerp. In 2019, they will be published in a special section of the journal Research Evaluation<\/a>. The ENRESSH policy group will submit recommendations that include their content to the commission. It will also lobby people who grant research funding and innovation funding in Europe, as well as other actors who are dealing with impact evaluations at the European level.<\/p>\n\n\n\n Sigur\u00f0arson: Not yet. I don\u2019t think I can point to anything that\u2019s already happened. But I know the ENRESSH policy group has been active in Brussels.<\/p>\n\n\n\n There\u2019s a tendency for a group like this to focus very much on the social sciences. This is probably because most of the scholars involved in the field of research evaluation are social scientists. As a humanities scholar, you can feel slightly marginalized just by the number of things going on. Of course, though, overall it\u2019s a very comfortable environment. (Laughs).<\/p>\n\n\n\n Sigur\u00f0arson: A major thing I\u2019m trying to promote is properly understanding the priority order of the questions we ask about research evaluations. This is explained and analyzed very well by Jerry Z. Muller in his book The Tyranny of Metrics<\/a>. So, the question goes from indicators and metrics to the question of, \u201cWhat is the good we are doing?\u201d<\/p>\n\n\n\n This is because metrics and indicators are only applicable in some cases, not all. If the debate\u2019s priority is metrics and indicators, you risk losing out, or ignoring, many valuable aspects of research that do not fit the metrics debate. Therefore, first one should ask, \u201cWhat is the good that this researcher is doing?\u201d Then you discuss how we will evaluate the methods and quality of their work and whether we can use metrics and indicators. It would be just a waste of time to apply indicators to all cases. So our objective is to reverse the priorities.<\/p>\n\n\n\n Then, we should divide research into (1) fields where we can use some kind of metrics or tools, (2) fields where we need to use peer review, and (3) fields where we simply cannot use any formal methods. In (3), we are left with an informal policy debate on what we actually want for society. In such cases, one applies the likes of Martha Nussbaum\u2019s capabilities approach to discuss the social impact of the research.(1)<\/sup><\/p>\n\n\n\n This is the most effective method that we should use first. This debate can\u2019t be avoided. When it comes to the value of the humanities for society, humanities scholars need to have their own voice, they need to be able to speak on their own terms.<\/p>\n\n\n\n Shifting the focus away from metrics and indicators sounds very useful for research evaluation that promotes research. Going back to the basics.<\/p>\n\n\n\n Sigur\u00f0arson: It depends a lot on people\u2019s position and the context. Say you are a research administrator assisting people writing funding applications, and the fund is asking for cogent argument about the research\u2019s use to society. In that case, your role might be precisely assisting them in writing or describing\u2014and indeed just understanding\u2014the social impact of their research. If one follows the debate regarding social impact even a little bit, one can express it using more appropriate terms.<\/p>\n\n\n\n In my experience discussing this with scholars, there is a tendency to understand the question very narrowly. When asked to describe their research\u2019s social impact, some think, \u201cOkay, they are asking me how I am going to benefit the economy,\u201d or that the question is, \u201cHow is this research going to have a direct influence on policy?\u201d Even some people say, \u201cNo, I\u2019m researching medieval Icelandic literature, and there\u2019s nothing in what I\u2019m doing that\u2019s going to change anything in the economy.\u201d So one role of URAs would be to assist people in describing their social impact more broadly, to understand in a wider context what the social impacts might be.<\/p>\n\n\n\n If you\u2019re working on the humanities and social sciences in an institute that is more focused on the hard sciences in general, then a role would be to keep arguing, at the institute\u2019s policy level, for the social good produced by the social sciences and humanities. Talking about what would be lost by your institute if we stopped or reduced funding or supporting the humanities.<\/p>\n\n\n\n I have been working a lot with Ph.D. students in the humanities in Iceland. If working with young researchers, that is, early career researchers, is part of your remit, your role would also be helping them develop ideas and formulate arguments about what they\u2019re doing. Maybe pointing to some questions that they might ask themselves about their research, that they may not have thought of themselves, as well.<\/p>\n\n\n\n Sigur\u00f0arson: Yes. One can ask, \u201cWhat is the social benefit of you as an individual getting a research education, writing a thesis, and developing yourself as a researcher?\u201d If they gain skills that they can use in other contexts, it would be a loss to society if these weren\u2019t developed. That personal development, that kind of capacity building of yourself as an individual and the people around you, is beneficial to society.<\/p>\n\n\n\n (1)<\/sup> This interview was carried out on 21 May 2019. Sigur\u00f0arson was at Kyoto University to give a KURA workshop<\/a>. For information on concrete methods for advancing discussions, refer to presentation slides<\/a> from Sigur\u00f0arson\u2019s KURA workshop.<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" *This article is based on content previously published on \u201cLife as a scholar: Kyoto University and Beyond,\u201d which has since been discontinued. Today, scholars are expected to return the fruits of their research to society and address social problems. Instead of fitting into society\u2019s demands and expectations, how can those working on humanities, social science, or basic science research talk about social impact? We discussed such issues with the University of Iceland\u2019s Eir\u00edkur Sigur\u00f0arson. Focusing on social impact, he is working to create new research evaluation methods.(1) Director of Research, School of Humanities, University of IcelandEir\u00edkur Sm\u00e1ri Sigur\u00f0arson Sigur\u00f0arson is also active in ENRESSH (European Network for Research Evaluation in\u2025<\/p>\n","protected":false},"author":2,"featured_media":0,"template":"","meta":{"_acf_changed":false,"_crdt_document":"","exclusive":false,"relatedArticles":[],"footnotes":""},"tags":[75],"class_list":["post-2256","blog","type-blog","status-publish","hentry","tag-next-generation-forum"],"acf":[],"yoast_head":"So the problems of metrics and career are closely linked. You\u2019re part of the group researching social impact. What are its accomplishments?<\/h4>\n\n\n\n
Is ENRESSH involved in policymaking?<\/h4>\n\n\n\n
So is there some kind of coordinator that translates the research into policy?<\/h4>\n\n\n\n
Is there a particular instance that comes to mind where this lobbying yielded concrete policies?<\/h4>\n\n\n\n
From Indicators and Metrics to \u201cWhat is the Good We Are Doing?\u201d<\/h3>\n\n\n\n
Today\u2019s research evaluation problems appear to have come from research evaluation\u2019s means becoming the objective. In other words, meeting research metrics has become the goal of research, and, accordingly, research inspiration is tending to atrophy. People say there\u2019s a gap between evaluation for resource allocation, on the one hand, and evaluation for promoting quality research, on the other. What do you think should be done so that research evaluation is not an obstacle to research but instead advances it?<\/h4>\n\n\n\n
The Role of URAs in Research Evaluation<\/strong><\/h3>\n\n\n\n
What do you think is the role of research administrators and research managers in research evaluation?<\/h4>\n\n\n\n
Do you think that using the social impact approach is effective even for young researchers moving on to something other than research? In other words, when they have to articulate what they have been doing up through the present.<\/h4>\n\n\n\n
\n\n\n\n