Monitoring Citizen Science Performance: Methodological Guidelines

• Autorzy: Aelita Skaržauskienė , Margarita Angelidou , Christos Politis , Elli Roma-Athanasiadou , Monika Maciuliene
• Języki publikacji: EN

Abstrakty

EN

This research paper aims to co-design a theoretical framework and methodological guidelines to monitor the performance, outcomes, and impact of Citizen Science projects. Hence, the proposed methodology combines the previous research efforts on the engagement of external stakeholders into a composite monitoring tool, which allows measuring and comparing CS project progress towards defined objectives. The development of methodology adopted a pragmatic mixed-method research design. Firstly, analysis of the theoretical sources and available monitoring tools provided the basis for a conceptual framework and identification of measurement indicators. The rationale of the conceptual framework was based on the co-creation workshop and stakeholders` interviews. In addition, the aggregation of indicators during the expert Focus group research resulted in a meaningful design of the composite CS Performance Index. The designed methodology is a part of the three-year Horizon 2020 "Science with and for Society" project INCENTIVE (Grant Agreement No. 101005330) that brings on board four Research Performing and Funding Organizations: the University of Twente, the Autonomous University of Barcelona, the Aristotle University of Thessaloniki and the Vilnius Gediminas Technical University. Methodological guidelines set the structure for applying the monitoring in four national CS Hubs and support their coordination. In particular, the values of the indicators could be compared with the targets set, with the respective values of other Citizen Science Hubs, or to worldwide trends. (original abstract)

Słowa kluczowe

EN

Citizen involvement; Citizen; Academic teachers; Industry; Scientific research; Research methodology

PL

Zaangażowanie obywateli; Obywatel; Nauczyciele akademiccy; Przemysł; Badania naukowe; Metodologia badań

• Czasopismo: Contemporary Economics
• Rocznik: 2024
• Tom: 18
• Numer: nr 1
• Strony: 67--86

Twórcy

autor

Aelita Skaržauskienė

• Vilnius Gediminas Technical University, Lithuania

autor

Margarita Angelidou

• Aristotle University of Thessaloniki, Thessalonike, Kentrike Makedonía, Greece

autor

Christos Politis

• Aristotle University of Thessaloniki, Thessalonike, Kentrike Makedonía, Greece

autor

Elli Roma-Athanasiadou

• Aristotle University of Thessaloniki, Thessalonike, Kentrike Makedonía, Greece

autor

Monika Maciuliene

• Vilnius Gediminas Technical University, Lithuania

Bibliografia

• Ahmad, M. S., & Abu Talib, N. (2013). Local government systems and decentralization: evidence from Pakistan's devolution plan. Contemporary Economics, 7(1), 33-44. https://doi.org/10.5709/ce.1897-9254.72
• Akman, H., Plewa, C., Conduit, J. (2019). Co-creating value in online innovation communities, European Journal of Marketing, 53 (6), 1205-1233. https://doi.org/10.1108/EJM-12-2016-0780
• Baek, S.I., Kim, Y. M. (2015). Longitudinal analysis of online community dynamics. Industrial Management & Data Systems, 115 (4), 661-677. https://doi.org/10.1108/IMDS-09-2014-0266
• Bauer, A., Bogner, A., & Fuchs, D. (2021). Rethinking societal engagement under the heading of Responsible Research and Innovation:(novel) requirements and challenges. Journal of Responsible Innovation, 8(3), 342-363. https://doi.org/10.1080/23299460.2021.1909812
• Stilgoe, J., Peter, V., & Jäger, A. (2018). Monitoring the evolution and benefits of responsible research and innovation in Europe: Final Report. European Commission. https://doi.org/doi/10.2777/207020
• Chakravarty, S. R. A. (2003). Generalized human development index. Review of Development Economics, 7(1), 99-114. https://doi.org/10.1111/1467-9361.00178
• Coccia, M. (2018). An introduction to the theories of institutional change. Journal of Economics Library, 5(4), 337-344. https://doi.org/10.1453/jel.v5i4.1788
• Gorbaniuk, O., & Zalinski, A. (2019). Attribution asymmetry in perception of companies' successes and failures. Contemporary Economics, 13(1), 5-19. https://doi.org/10.5709/ce.1897-9254.295
• Hajighasemi, A., Oghazi, P., Aliyari, S., & Pashkevich, N. (2022). The impact of welfare state systems on innovation performance and competitiveness: European country clusters. Journal of Innovation & Knowledge, 7(4). https://doi.org/10.1016/j.jik.2022.100236
• Dart, J., & Davies, R. (2003). A dialogical, story-based evaluation tool: The most significant change technique. The American Journal of Evaluation, 24(2), 137-155. https://doi.org/10.1177/109821400302400202
• Drebee, H. A., Abdul-Razak, N. A., & Shaybth, R. T. (2022). The Impact of Governance Indicators on Corruption in Arab Countries. Contemporary Economics, 14(3), 354. https://doi.org/10.5709/ce.1897-9254.409
• Directorate General for Regional Policy (2018). Guidance document on monitoring and evaluation. https://guidance_monitoring_evaluation_en.pdf
• DITOs Consortium (2016). Doing It Together science: Terms of reference and evaluation templates. UCL. https://discovery.ucl.ac.uk/id/eprint/1560387/
• European Citizen Science Association (2020). The ten principles of citizen science. European Citizen Science Association. https://doi.org/10.17605/OSF.IO/XPR2N
• European Commission (2020). Citizen science policy. Publications Office of the European Union https://ec.europa.eu/digital-single-market/en/citizen-science
• Delaney, N., & Iagher, R. (2020). Institutional changes towards responsible research and innovation: achievements in Horizon 2020 and recommendations on the way forward. Publications Office of the European Union. https://data.europa.eu/doi/10.2777/682661
• European Commission. (2019). Responsible for research & innovation. Publications Office of the European Union. https://data.europa.eu/doi/10.2777/682661
• Foa, R., & Tanner, J. C. (2014). Methodology of the indices of social development. International Institute of Social Studies of Erasmus University Rotterdam.
• Fochler, M., & De Rijcke, S. (2017). Implicated in the indicator game? An experimental debate. Engaging Science, Technology, and Society, 3, 21-40. https://doi.org/10.17351/ests2017.108
• Freudenberg, M. (2003). Composite indicators of country performance: A critical assessment. (STI Working Paper No. 16), 2-34. https://doi.org/10.1787/405566708255
• Göbel, C., Cappadonna, J. L., Newman, G. J., Zhang, J., & Vohland, K. (2017). More than just networking for citizen science: Examining core roles of practitioner organizations. In Analysing the role of citizen science in modern research (pp. 24-48). IGI Global. https://doi.org/10.4018/978-1-5225-0962-2.ch002
• Grey, F., Wyler, D., Fröhlich, J., & Maes, K. (2016). Citizen science at universities: Trends, guidelines, and recommendations. LERU, 20, 1- 24. https://www.leru.org/files/Citizen-Science-at-Universities-Trends-Guidelines-and-Recommendations-Full-paper.pdf
• Haklay, M. M., Dörler, D., Heigl, F., Manzoni, M., Hecker, S., & Vohland, K. (2021). What is CSCS? The challenges of definition. In K. Vohland, A. Land-Zandstra, L. Ceccaroni, R. Lemmens, J. Perelló, M. Ponti, R. Samson, & K. Wagenknecht (Eds.), The Science of CSCS (pp. 13-33). Springer. https://doi.org/10.1007/978-3-030-58278-4_2
• Jakobsen, S. E., Fløysand, A., & Overton, J. (2019). Expanding the field of Responsible Research and Innovation (RRI)-from responsible research to responsible innovation. European Planning Studies, 27(12), 2329-2343. https://doi.org/10.1080/09654313.2019.1667617
• Jordan R.C., Ballard H.L., & Phillips T.B. (2012). Key issues and new approaches for evaluating citizen-science learning outcomes. Front Ecological Environment, 10(6), 307-309 https://doi.org/10.1890/110280
• Irwin, A. (1995). Citizen science: A study of people, expertise and sustainable development (1st ed.). Routledge.
• Kaiser, F. G. (2020). GEB-50. General ecological behavior scale. ZPID (Leibniz Institute for Psychology) - Open Test Archive. https://doi.org/10.23668/psycharchives.4489
• Kieslinger, B., Schäfer, T., Heigl, F., Dörler, D., Richter, A., & Bonn, A. (2018). Evaluating citizen science: Towards an open framework. In S. Hecker, M. Haklay, A. Bowser, Z. Makuch, J. Vogel, & A. Bonn (Eds.), CS: Innovation in Open Science, Society, and Policy (pp. 81-98). UCL Press. https://doi.org/10.14324/111.9781787352339
• Kieslinger, B., Schäfer, T., Heigl, F., Dörler, D., Richter, A., & Bonn, A. (2017). The challenge of evaluation: An open framework for evaluating citizen science activities. Open Science Framework. https://doi.org/10.17605/OSF.IO/ENZC9
• Liu, H.-Y., Dörler, D., Heigl, F., & Grossberndt, S. (2021). Citizen Science Platforms. The Science of Citizen Science, 22, 439. https://doi.org/10.1007/978-3-030-58278-4_22
• Lu, J. W., & Beamish, P. W. (2001). The internationalization and performance of SMEs. Strategic Management Journal, 22(6-7), 565-586. https://doi.org/10.1002/smj.184
• Lusch, R.F., Vargo, S.L. & O'Brien, M. (2007). Competing through service: Insights from service-dominant logic. Journal of Retailing, 83(1), 5-18. https://doi.org/10.1016/j.jretai.2006.10.002
• Mačiulienė, M.; Skaržauskienė, A.; Botteldooren, D. (2018). Developing a digital co-creation assessment methodology. Contemporary Economics, 12(4), 399-408. https://doi.org/10.5709/ce.1897-9254.285
• Manzoni, M., Vohland, K., Gobel, C., Pruse, B., & Schade, S. (2019). Citizen science strategies in Europe: Preliminary findings from the pan-European Survey of citizen science strategies and initiatives in Europe as part of a joint initiative, COST ACTION 15212 and the JRC. European Commission. https://d-nb.info/125076274X/34
• Multiple Indicator Cluster Surveys (2022). Developing metrics and instruments to evaluate citizen science impacts on the environment and society. Cordis EU Research Results. https://cordis.europa.eu/project/id/824711
• Mirowski, P. (2018). The future(s) of open science. Social Studies of Science, 48(2), 171-203. https://doi.org/10.1177/0306312718772086.
• Mittal, B., & Woodside, A. G. (2022). Nurturing urban innovation and knowledge in the ongoing COVID-19 world. Journal of Innovation & Knowledge, 7(3). https://doi.org/10.1016/j.jik.2022.100217
• MoRRI. (2022). Benefits of Responsible Research and Innovation. https://op.europa.eu/en/publication-detail/-/publication/1f32df40-4479-11e9-a8ed-01aa75ed71a1
• Mugdal, S., Turbe, A., Sanz, F., Barba, J., Pelacho, M., Serrano-Sanz, F., Robinson, L., & Gold, M. (2018). Citizen science for environmental policy: development of an EU-wide inventory and analysis of selected practices. Publications Office of the European Union. https://op.europa.eu/en/publication-detail/-/publication/842b73e3-fc30-11e8-a96d-01aa75e-d71a1/language-en
• O'Grady, M., & Mangina, E. (2022). Adoption of responsible research and innovation in citizen observatories. Sustainability, 14(12), 7379. https://doi.org/10.3390/su14127379
• Owen, R., Macnaghten, P., & Stilgoe, J. (2012). Responsible research and innovation: From science in society to science for society, with society. Science and Public Policy, 39(6), 751-760. https://doi.org/10.1093/scipol/scs093
• Passani, A., Janssen A. L., & Hoelscher, K. (2021). Impact assessment framework. Zenodo. https://doi.org/10.5281/zenodo.4432132
• Pérez, A., & Costa, N. (2018). Deliverable D3.1: Initial report on market analysis and market uptake. Ground Truth 2.0. https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e5b8e06f3b&appId=PPGMS
• Phillips, T., Ferguson, M., Minarchek, M., Porticella, N., & Bonney, R. (2014). Evaluating learning outcomes from citizen science. Cornell Lab of Ornithology. https://www.birds.cornell.edu/citizenscience/wp-content/uploads/2018/10/USERS-GUIDE_linked.pdf
• Ray, A. K. (2008). Measurement of social development: An international comparison. Social Indicators Research, 86(1), 1-46. https://doi.org/10.1007/s11205-007-9097-3
• Roman, D., Reeves, N., Gonzalez, E., Celino, I., El Kader, S. A., Turk, P., Soylu, A., Corcho, O., Cedazo, R., Calegari, G. R., Scandolari, D., Simperl, E., Kieslinger, B., Schäfer, T., Heigl, F., Dörler, D., Richter, A., & Bonn, A. (2021) An analysis of pollution Citizen Science projects from the perspective of Data Science and Open Science, Data Technologies and Applications, 55(5), 622-642. https://doi.org/10.1108/DTA-10-2020-0253
• Rosenbusch, N., Brinckmann, J., & Bausch, A. (2011). Is innovation always beneficial? A meta-analysis of the relationship between innovation and performance in SMEs. Journal of business Venturing, 26(4), 441-457. https://doi.org/10.1016/j.jbusvent.2009.12.002
• Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences, 1(1), 83-98. https://doi.org/10.1504/IJSSCI.2008.017590
• Saisana, M., & Tarantola, S. (2002). State-of-the-art report on current methodologies and practices for composite indicator development. Institute for the Protection and Security of the Citizen Technological and Economic Risk Management. https://doi.org/10.13140/RG.2.1.1505.1762
• Sauermann, H., Vohland, K., Antoniou, V., Balázs, B., Göbel, C., Karatzas, K., Mooney, P., Perelló, J., Ponti, M., Samson, R., Winter, S. (2020). Citizen science and sustainability transitions. Research Policy, 49(5), 103978. https://doi.org/10.1016/j.respol.2020.103978
• Schaefer, T., Kieslinger, B., Brandt, M., & van den Bogaert, V. (2021). evaluation in citizen science: the art of tracing a moving target. In K. Vohland, A. Land-Zandstra, L. Ceccaroni, R. Lemmens, J. Perelló, M. Ponti, R. Samson, & K. Wagenknecht (Eds.), The Science of CSCS (pp. 495-514). Springer. https://doi.org/10.1007/978-3-030-58278-4_25
• SciShops. (2022). Enhancing the responsible and sustainable expansion of the science shops ecosystem in Europe. Cordis EU Research Results. https://doi.org/10.3030/741657
• Skaržauskienė, A., & Mačiulienė, M. (2021). Citizen science addressing challenges of sustainability. Sustainability, 13(24), 1-11. https://doi.org/10.3390/su132413980
• Somerwill,L., & When, U. (2022). How to measure the impact of citizen science on environmental attitudes, behaviour and knowledge? A review of state-of-the-art approaches. Environmental Sciences Europe, 34(18). https://doi.org/10.1186/s12302-022-00596-1
• Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568-1580. https://doi.org/10.1016/j.respol.2013.05.008
• European Commission. (2015). Indicators for promoting and monitoring responsible research and innovation. Publications Office of the European Union. https://data.europa.eu/doi/10.2777/9742
• SUPER_MoRRI (2022). Scientific understanding and provision of an enhanced and robust monitoring system for RRI. Cordis EU Research Results https://cordis.europa.eu/project/id/824671
• Tagscherer, F., & Carbon, C. C. (2023). Leadership for successful digitalization: A literature review on companies' internal and external aspects of digitalization. Sustainable Technology and Entrepreneurship, 2(10). https://doi.org/10.1016/j.stae.2023.100039
• Topal H.F., Hunt D.V., & Rogers C.D. (2020), Urban sustainability and smartness understanding (USSU)-identifying influencing factors: A systematic review. Sustainability, 12(11), 4682. https://doi.org/10.3390/su12114682
• Udall, A. M., de Groot, J. I., de Jong, S. B., & Shankar, A. (2020). How do I see myself? A systematic review of identities in pro-environmental behavior research. Journal of Consumer Behaviour, 19(2), 108-141. https://doi.org/10.1002/cb.1798
• Urban Innovative Actions. (2020, March). Evaluation approaches. Urban Innovative Actions. https://www.uia-initiative.eu/en/evaluation-approaches
• United Nations. (2020). The sustainability development goals report 2020. Statistics Division, United Nations. https://unstats.un.org/sdgs/report/2020/
• Van De Klippe. (2019, March 20). From MoRRI to SUPER_MoRRI: Monitoring as reflection and learning, not representation and control. CWTS. https://www.cwts.nl/blog?article=n-r2w264&title=from-morri-to-supermorri-monitoring-as-reflection-and-learning-not-representation-and-control
• Vohland, K., Göbel, C., Balázs, B., Butkevičienė, E., Daskolia, M., Dužı, B., Hecker, S., Manzoni, M., & Schade, S. (2021). Citizen Science ́ in Europe. In K. Vohland, A. Land-Zandstra, L. Ceccaroni, R. Lemmens, J. Perelló, M. Ponti, R. Samson, & K. Wagenknecht (Eds.), The Science of Citizen Science (35-53). Springer. https://doi.org/10.1007/978-3-030-58278-4_3
• Von Schomberg, R. (2013). A vision of responsible research and innovation. In R. Owen, J. Bessant, & M. Heintz (Eds.), Responsible innovation: Managing the responsible emergence of science and innovation in society (pp. 51-74). Wiley. https://doi.org/10.1002/9781118551424.ch3
• Wehn, U., Gharesifard, M. & Ceccaroni, L. (2020). D2.3: Impact-assessment methods adapted to CS. Deliverable report of project H2020 MICS. https://zenodo.org/record/6542325/files/MICS_Deliver-ableD2.3%202020_06_29.pdf?download=1
• Wehn, U., Gharesifard, M., Ceccaroni, L., Joyce, H., Ajates, R., Woods, S., Bilbao, A., Parkinson, S., Gold, M., & Wheatland, J. (2021). Impact assessment of citizen science: State of the art and guiding principles for a consolidated approach. Sustainability Science, 16(5), 1683-1699. https://doi.org/10.1007/s11625-021-00959-2
• Wildevuur, S., Skaržauskienė, A., Mačiulienė, M., Van den Berg, M., Bakratsas, Th., Psaltoglou, A., Stylianidis, E., Tavantzis, I., & Karatzas, K. (2023). Developing citizen science ecosystem: Critical factors for quadruple helix stakeholders` engagement. Journal of enterprising communities: People and places in the global economy. Advance online publication. https://doi.org/10.1108/JEC-08-2022-0116
• Willetts, J., & Paul Crawford. (2007). The most significant lessons about the most significant change technique. Development in Practice, 17(3), 367-379. https://doi.org/10.1080/09614520701336907
• Wyler, D., & Grey, F. (2016). Citizen science at universities: Trends, guidelines, and recommendations. LERU. https://www.leru.org/publications/citizen-science-at-universities-trends-guidelines-and-recommendations
• Wyler, D., & Haklay, M. (2018). Integrating citizen science into university. Citizen Science: Innovation in Open Science, Society and Policy, 168-181. https://doi.org/10.2307/j.ctv550cf2.18
• Yadav, A., Rajesh, K. S., Ruchi, M., & Surajit, B. (2023). Perceived challenges affecting user engagement in the online community: An analysis of inter-relationships and interaction. An International Journal, 1463-5771. Advance online publication https://doi.org/10.1108/BIJ-11-2022-0710

Identyfikatory

DOI

10.5709/ce.1897-9254.526