11/18/2023 0 Comments Benchmark research studiesThis issue should be considered by computational researchers when designing and analysing their benchmark studies and by the scientific community in general in an effort towards more reliable benchmark results. In conclusion, based on previous literature and on our illustrative example, we claim that the multiplicity of design and analysis options combined with questionable research practices lead to biased interpretations of benchmark results and to over-optimistic conclusions. Strategic research include longitudinal studies. We then demonstrate how the impact of each choice on the results can be assessed using multidimensional unfolding. The study is structured around the 6 key benchmark parameters used to. To raise awareness for this issue, we use an example benchmark study to illustrate how variable benchmark results can be when all possible combinations of a range of design and analysis options are considered. the selective reporting of results or the post-hoc modification of design or analysis components) to fit their expectations or hopes. However, confidence in the results of such observational research is typically low, for example, because different studies on the same question often. As a consequence of this flexibility, researchers may be concerned about how their choices affect the results or, in the worst case, may be tempted to engage in questionable research practices (e.g. This includes the choice of data sets and performance measures, the handling of missing performance values and the way the performance values are aggregated over the data sets. While general advice on the design and analysis of neutral benchmark studies can be found in recent literature, certain amounts of flexibility always exist. Thus, all cited works should be duly referenced, all procedures performed in studies involving human participants should be in accordance with the ethical. Information Studies Identifiers URN: urn:nbn:se:kth:diva-304781 ISI: 000709638700135 Scopus ID: 2-s2.0-85112622329 OAI: oai:DiVA.Download a PDF of the paper titled Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results, by Christina Niel (1) and 10 other authors Download PDF Abstract:In recent years, the need for neutral benchmark studies that focus on the comparison of methods from computational sciences has been increasingly recognised by the scientific community. Proceedings of the International Conference on Scientometrics and Informetrics, ISSN 2175-1935 National Category Place, publisher, year, edition, pagesINT SOC SCIENTOMETRICS & INFORMETRICS-ISSI, 2021. Participate in current benchmark study by Best Practices, LLc and receive a free copy of benchmarking studies research survey results in presentation. We apply this strategy to a Swedish research center, and examine the effectiveness of the method. Three essential attributes for the evaluation of benchmarks are research topics, output, and coherence. We define a benchmark as a well-connected research environment, in which researchers work on similar topics and publish a similar number of publications compared to a given research center during the same period. This study aims to propose a bibliometric method to identify benchmarks. However, few studies have investigated this problem. Therefore, methods to identify benchmark research units are of practical significance. In addition to monitoring and evaluations, the identification of comparable benchmark organizations can also be used to pinpoint potential collaboration partners or competitors. Furthermore, research organizations, policymakers and research funding providers tend to use benchmark units as points of comparison for a given research center in order to understand and monitor its development and performance. While normalized bibliometric indicators are expected to resolve the subject-field differences between organizations in research evaluations, size still matters. 1229-1234 Conference paper, Published paper (Refereed) Abstract 2021 (English) In: 18 th International Conference on Scientometrics and Informetrics” (ISSI2021) / Glanzel, W Heeffer, S Chi, PS Rousseau, R, INT SOC SCIENTOMETRICS & INFORMETRICS-ISSI, 2021, p.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |