## Introduction These are the rubrics used to assess _Research Project 1_. There are two things worth emphasizing about these rubrics: 1. The weights of the aspects within each individual rubric are not predefined. 2. The normative descriptions are abstracted and might require reinterpration for any given project. Both these points can be explained by the diverse range of projects that are performed by students from the research master *Brain and Cognitive Sciences*. Different disciplines will place different requirements on student performance. Still, we believe that the use of rubrics contributes to comparable assessment of all students within the programme, regardless of their field or hosting lab. ## Rubrics There are three rubrics, corresponding to the three graded components: process, written report and oral presention. ### Process Process counts towards 40% of the final mark and can only be assessed by the internal assessor of the project. | Aspect | Poor (1.0 - 4.0) | Not sufficient (4.5 - 5) | Sufficient (6.0 - 7.0) | Good (7.5 - 8.5) | Excellent (9-10) | | | :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | | Collecting literature | The student did not engage with academic literature or demonstrated negligible understanding of the research project's context. This severely hindered their ability to comprehend the project's purpose or contribute meaningfully. | The student struggled to independently collect and/or critically evaluate appropriate literature in a timely manner. This resulted in an incomplete or superficial understanding of the project's context, as evidenced by interactions with the supervisor and negative impact on project performance. They lacked self-reliance in identifying key academic resources. | The student managed to collect relevant literature, demonstrating a basic understanding of the research context. However, their theoretical understanding remained somewhat superficial, limiting their ability to critically evaluate important research tasks, design choices, or the broader implications of their work. Independent synthesis of the literature was limited. | The student independently collected and evaluated relevant literature, leading to a comprehensive understanding of both the theoretical framework and the empirical landscape surrounding the research project. This robust understanding allowed them to grasp the relevance of project tasks, and justify their research questions and significant design choices based on existing knowledge. | The student demonstrated self-reliance and critical acumen in collecting and evaluating academic literature. They developed a nuanced overview of both the theoretical framework and the empirical state-of-the-art relevant to the research project. This understanding enabled them to justify all research project steps with clear academic grounding, and compare their project's approach with current methodologies found in peer-reviewed literature. | | | Experimental skills | The student has failed to consistently show proficiency in the relevant experimental skills, even under supervision. | Experimental work done by the student is not efficient, either because of low accuracy, low speed or the requirement of intensive supervision. | The student learned to employ technique(s) relevant to this project, but important elements of proficiency are still missing. Example elements are timeliness, accuracy, or demonstrated understanding of the principles behind the technique(s). | The student worked in a timely and accurate fashion when applying technique(s) relevant to this project. The student has a good idea of how the techniques work and what their principles are, but is not yet able to apply them independently or in a novel setting. | The student mastered the technique(s) relevant to this research project and understands their principles and their scope of applicability. The student works timely and accurately when doing experimental work. | | | Data Analysis | The student did not demonstrate the ability to perform data analysis steps relevant to the project. | The student struggles significantly with data analysis. They may not be able to select an appropriate method or, if a method is provided, cannot execute the analysis or interpret results without constant, step-by-step supervision. There's limited or no understanding of the method's principles or the meaning of the outputs. | The student is capable of selecting appropriate data analysis methods for the acquired data within the research project's domain. They can perform the analysis and derive interpretations, but this process requires considerable supervision. They frequently struggle to justify key analytical choices (e.g., choice of statistical test, handling of outliers, model selection) and may present interpretations without sufficient critical reflection on their validity. | The student demonstrates a solid ability to select appropriate data analysis methods for the acquired data within the research project's domain, providing sound reasoning for their choices. They have a good conceptual and practical understanding of the chosen analysis methods. They can independently perform the analysis, accurately interpret results, and articulate the implications of their findings. The student generally demonstrates awareness of potential limitations of the analysis. | The student exhibits exceptional proficiency in data analysis. They not only select the most appropriate data analysis methods for the acquired data, providing clear, well-argued justifications for all critical choices, but also demonstrate mastery of these methods. They can independently execute complex analyses, critically evaluate and interpret results within the broader context of the research, and proactively identify and address potential biases, assumptions, or limitations of their approach. Their analysis supports robust, evidence-based conclusions. | | | Learning skills | The student demonstrates significant difficulty in acquiring new knowledge or skills independently. They frequently rely on constant instruction and struggle to grasp new concepts or techniques relevant to the project, showing resistance to feedback or new approaches. Progress is severely hampered by an inability to overcome learning obstacles, indicating low ownership of the project. | The student shows limited initiative in acquiring new knowledge or skills. They may attempt to learn but often require significant guidance and explicit step-by-step instructions. They struggle to effectively identify necessary resources or apply new concepts without direct prompting, and may only partially integrate feedback. They are slow to adapt to new information or challenges. | The student demonstrates a willingness to learn new skills and acquire necessary knowledge, often seeking guidance when encountering difficulties. They can identify some relevant resources and apply newly learned concepts with moderate supervision. They generally integrate feedback, and can adapt to minor changes or new information, but may struggle with more complex or unexpected learning challenges independently. | The student actively seeks out and effectively acquires new knowledge and skills required for the project. They demonstrate resourcefulness in finding and utilizing appropriate learning materials and tools (e.g., tutorials, documentation, expert advice). They are generally quick to integrate feedback, adapt to unforeseen challenges or changes in the project direction, and independently apply new concepts to solve emerging problems. | The student demonstrates exceptional proactive learning and adaptability throughout the project. They independently identify knowledge gaps or skill requirements, and master new complex concepts or techniques with the right balance between independence and asking questions. They are resourceful, leveraging self-directed thinking to overcome significant challenges. They consistently and thoughtfully integrate feedback, rapidly adapt, and work independently, taking full ownership of the project. | | | Original contribution | The student demonstrated minimal or no initiative to contribute original ideas or improve the work, even when prompted or instructed. The project primarily reflects the supervisor's direction, with the student acting as a passive implementer. | Throughout the project, the student primarily followed instructions regarding experimental design, data collection, analysis, and interpretation. While tasks were completed, the student rarely contributed substantial independent ideas, insights, or modifications to the project's direction or execution. | The student demonstrated the ability to make informed choices regarding experimental design, data analysis, and interpretation of results, but typically only when directly prompted or guided by the supervisor. They could articulate reasoning for these choices once a direction was suggested, but lacked consistent proactive ideation. | The student consistently demonstrated the ability to make decisions about the projects independently, often initiating discussions with the supervisor to refine these ideas. They contributed valuable, independent thoughts for experimental setup, analytical approaches, or interpretation, actively shaping significant aspects of the project. | The student displayed exceptional initiative and made significant original contributions to the project. E.g. they formulated insightful hypotheses, conceived and articulated well-reasoned design choices or consistently brought forward novel ideas that demonstrably enhanced data collection, analysis, or interpretation. | | ### Technical report The [[Technical report]] counts towards 50% of the final mark and is assessed by both the internal and the external assessor of the project. | Aspect | Poor (1.0 - 4.0) | Not sufficient (4.5 - 5.0) | Sufficient (6.0 - 7.0) | Good (7.5 - 8.5) | Excellent (9-10) | | :--------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Structure of the report | Sections are poorly defined, illogical or entirely absent, leading to a fragmented report. The connections between ideas are unclear or non-existent and the reader struggles to follow the overall argument or purpose. | While somewhat coherent, the report deviates significantly from the technical report template or an alternative logical flow. Sections may be present, but are not always coherent or internal organisation is confusing. Understanding the connections between elements in the text requires too much effort from the reader. | The report adheres to a recognizable technical report template or employs a logical, consistent structure that generally serves the report's purpose. Main sections are present and ordered acceptably. While generally understandable, transitions between sections or ideas may sometimes be abrupt or require minor effort from the reader. | The structure of the report actively supports comprehension and guides the reader through the content. It adheres to technical report conventions (or a well-justified alternative) with clear, logical connections between major sections. Paragraphs are well-organized and contribute effectively to the larger narrative. The reader can follow the main arguments and findings without significant effort. | The structure of the text is clear and logical at all levels. Organisation of the text helps to highlight key findings, enhance readability and support the conclusion and recommendations in the report. Effective use of headers and logical sequencing create a compelling narrative flow, rendering the report a pleasure to read and helping with retention of the content. | | Introduction | It is insufficiently clear why the research project is relevant and/or the context is not explained clearly enough to warrant the research question and hypotheses (if any). | Some relevance of the research project is addressed, but it is unclear how the project fits into the ongoing work of the host environment. | The research question and hypothesis (if any) are shown to tie into the work of the host environment, but important context (e.g. theoretical framework) is missing. | The relevance of the research project is shown, both with regards to the hosting environment and the broader academic literature. | The relevance of the project is explicitly and clearly discussed, helping the reader to position the project in its larger picture. | | Methods | Operationalization is lacking: it is not clear how data was gathered and/or what the overall design of the project was. Replication would be impossible on the basis of this report. | It is clear what the experimental procedures were, how samples were collected (e.g. how subjects were included, cells were obtained, etc.), and what the overall design of the project was, but vital information that would be necessary for replication is missing. | The description of data acquisition is complete enough to allow other scientists to replicate the study. Operationalisation of key concepts is mentioned explicitly. The choice of methods is not fully justified through logical argumentation or citations. | The description of data acquisition is complete enough to allow other scientists to replicate the study. Operationalisation choices are clearly described and justified. Experimental design is justified through logical argumentation or citations. However, some errors or omissions remain in the methods section. | The description of data acquisition is complete enough to allow other scientists to replicate the study. The relation between the concepts under investigation and their operationalization is clear and the choice of methods is justified through logical argumentation or citations. There are no errors or omissions in the methods section. | | Results | The results section shows incomplete, inaccurate or irrelevant information, thus preventing the reader from evaluating the subsequent conclusion and recommendations. | There is information about the data that was collected, the data analysis that took place and the results, allowing the reader to evaluate the subsequent conclusion and recommendations, but the presentation of the results leaves much to be desired. | The information about the data that was collected, the data analysis that took place and the results are all clearly and efficiently communicated, preparing the reader for the subsequent conclusion and recommendations. However, because of incomplete statistics, data tabulation, use of graphs or other obstacles, it is difficult for the reader to critically examine the results independently. | The description of the data and subsequent analysis is accurate, complete and parsimonious, allowing to reader to critically evaluate the findings and prepare for the subsequent conclusion and recommendations. Still, some relevant statistics are missing or there are flaws with regards to graphs and figures. | The description of the data and subsequent analysis is accurate, complete and parsimonious, allowing to reader to critically evaluate the findings and subsequent conclusion and recommendations. All relevant statistics are given. Graphs are clear and informative and chosen for their relevance to answer the research question under consideration. | | Conclusion and recommendations | The results are not interpreted (or to a very limited degree), so that it is not clear what the project has added to the existing body of knowledge. Alternatively, the recommendation section offers no actionable directions. | The conclusion provides a clear interpretation of the results. This interpretation is linked to the aims of the study and some recommendations are made, but this is done in a too superficial manner, leaving important implications unmentioned. | The conclusion provides a clear interpretation of the results, and links this to the aims of the study effectively. Recommendations address the most important concerns, but the suggestions are either not concrete or are not actionable for other reasons. | The conclusion provides a clear interpretation of the results, and links this to the aims of the study effectively. Recommendations are concrete and actionable and address the most important concerns. | The conclusion provides a clear interpretation of the results, and links this to the aims of the study effectively. Recommendations are concrete and actionable and address a wide range of concerns. Alternatively, the recommendations are especially creative and demonstrate a forward-thinking mindset. | | Writing | The technical report is not readable at a glance. Language use is unclear or the structure of the text does not aid the reader to quickly parse the information. | While mostly readable at a glance, there are sections in which language use or structure negatively affect the flow of the text. As a consequence, the report as a whole suffers. | The report is clearly readable and anyone searching for specific information can quickly find it. Language use is in order, but there are multiple examples of sentences or paragraphs that should have been improved. | The report uses clear language and provides a structure that makes it easy to interpret the report at a glance and find the relevant information. There are hardly any comments to make about the writing of the text. | The writing is parsimonious and clear and the structure of the report is logical. It is easy to find specific information and the text is written in a way that engages the reader. | | Disclosure and transparency of GenAI use | No statement on GenAI is present as a distinct section in the technical report. | A statement on GenAI use is present, but it lacks specificity. It may mention the use of GenAI generally without identifying specific tools, or it provides a list of tools without clear indication of _how_ or _for what purpose_ they were used in the research or report writing process. Does not address potential limitations or verification. | A clear statement on GenAI use is present, accurately listing the specific GenAI tools (e.g., ChatGPT-4, Midjourney v5.2, UvA AI Chat) and their version numbers and models. It details the _specific purposes_ for which each tool was employed (e.g., "for brainstorming initial ideas for the literature review," "to assist with grammatical editing and phrasing of the introduction," "for generating sample data for testing a specific algorithm," "to create initial drafts of figures based on provided data"). The statement may briefly acknowledge the need for human oversight but does not elaborate on verification. | A comprehensive statement on GenAI use is present, providing precise details on: <br>- **Tools and Versions:** Specific GenAI tools (including version numbers and models). <br>- **Purpose and Scope:** Detailed explanation of _how_ and _for what specific tasks_ each tool was utilized. <br>- **Human Oversight & Verification:** Explicitly describes the process of critical evaluation, but may lack depth in explaining the _methods_ of verification. It does not consistently address limitations or provide prompts. | A comprehensive and transparent statement on GenAI use is present, providing precise details on: <br>- **Tools and Versions:** Specific GenAI tools (including version numbers and models) <br>- **Purpose and Scope:** Detailed explanation of _how_ and _for what specific tasks_ each tool was used.<br>- **Human Oversight & Verification:** Explicitly describes the process of critical evaluation and the limitations of specific GenAI use.<br>- **Prompts (optional but encouraged for key uses):** For significant contributions by GenAI, provides examples of prompts used to generate outputs in an appendix. | | Ethical considerations | Fails to identify significant ethical considerations relevant to the research project. | Presents a very limited and superficial understanding of ethical issues. May simply list general ethical principles without connecting them to the specific research context. | Identifies some general ethical considerations but may overlook specific or significant ethical challenges directly related to the research project. Discussion may be superficial, lacking depth in analyzing potential impacts or proposing adequate mitigation strategies. | Identifies most of the key ethical considerations relevant to the research project. Discusses potential impacts on stakeholders, data privacy, and informed consent. May miss some more subtle ethical nuances or broader societal implications. | Demonstrates a comprehensive and nuanced understanding of all relevant ethical considerations pertaining to the research project. Identifies both obvious and subtle ethical dilemmas, proactively addressing potential impacts on stakeholders, data privacy, bias, informed consent, and societal implications. | ### Oral presentation The presentation counts towards 10% of the final mark and is assessed by the internal assessor. | Aspect | Poor (1 - 4) | Not sufficient (4.5 - 5.0) | Sufficient (6.0 - 7.0) | Good (7.5 - 8.5) | Excellent (9 - 10) | | :---------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Structure of the presentation | The presentation lacks a discernible or logical structure, with essential elements (e.g., introduction, question, methods, results, conclusion) either missing, severely disorganized, or disproportionately allocated time. This makes it too difficult for the audience to follow the research narrative or understand the core message. | The presentation covers the necessary elements (e.g., research question, methods, results, discussion/conclusion), but their order or emphasis is illogical or poorly balanced. Key components may be rushed or excessively detailed, leading to confusion, disengagement, or an inability for the audience to grasp the main points effectively. | The presentation includes all necessary elements and generally follows a logical order (e.g., standard scientific presentation flow). The audience can generally follow the project description, but the flow between sections might be somewhat abrupt, or the differentiation between primary and secondary information is not consistently clear, requiring some effort from the audience to integrate ideas. | The presentation features a clear, logical, and engaging structure that effectively guides the audience through the research. It begins with a compelling opening, clearly articulates the research question, design, and findings, and leads to a coherent discussion and conclusion with recommendations. Transitions between sections are mostly smooth, and key information is appropriately highlighted, enabling the audience to follow the story and main arguments with ease. | The presentation offers a purposeful structure designed for maximum clarity and impact. It features a captivating opening, a logically unfolding narrative that connects the research question, detailed methodology, result, and discussion. Transitions are fluid and enhance comprehension, while crucial information is expertly emphasized. The structure ensures crystal-clear understanding of findings and arguments. | | Demonstration of relevance | The relevance of the research project is not made clear through the presentation. | The presentation shows how the question of the research project relates to existing knowledge and applications, but not (very) persuasively or comprehensively. | The presentation persuades the audience that the research question is (somewhat) important, but there are clear missed opportunities to do so. | The presentation demonstrates the relevance of the research question in a persuasive, comprehensive manner, but should have better situated the research project in its theoretical framework. | The presentation clearly shows why the research question is worth investigating. It comprehensively shows the place of the research project in the existing body of knowledge and clearly delineates what the research project adds to this body and/or its applications. | | Clarity of research design | The research design is not apparent from the presentation. There is information missing, too much unnecessary information or there are other reasons why overall design is unclear. | The presentation contains all necessary information about the design of the experiment, but important choices were left unexplained or were not communicated clearly. | The presentation offers a complete and motivated view of the research design, but it is insufficiently clear why this design was chosen over other options. | The presentation offers a complete, motivated and clear view of the research design. | The presentation clearly conveys the design that was used to answer the research question(s), including the underlying motivations. Design is clarified through graphs and illustrations, contains no unnecessary information and is easily understood. | | Results | The results are not clear from the presentation. | The presentation lists the results from the project, but the steps that were made to arrive at the results are (partially) omitted or not clearly explained. | The presentation lists the results from the project and it's clear how these results were obtained. However, the presentation of the results (graphs, illustrations) should have been better. | The presentation clearly conveys the important results from the project and how they were obtained. Whenever relevant or necessary, the presentation addresses the steps of analyses and this explanation is supported through good use of graphs and figures. | The presentation clearly conveys the results from the project and how they were obtained. The presentation clearly and succinctly addresses the steps of analyses whenever needed, and this explanation is supported through good use of graphs and figures. There is a clear focus on the most important results, showing the student could differentiate between major or minor importance. | | Q&A | The student is not able to provide satisfactory answers to questions about the research project. | The student is able to answer basic questions about aims and design choices, but is not able to critically reflect on the research, place it in its research context or elaborate on recommendations for the future. | The student is able to answer basic questions about aims and design choices and can place the research project in a broader context. The student can offer some recommendations, but is not able to critically reflect on the strengths and weaknesses of these suggestions or the research project itself. | The student is able to answer basic and advanced questions about aims and design choices and can place the research project in a broader context. The student can also offer recommendations, and is able to critically reflect on the strengths and weaknesses of the research project. | The student is able to engage in a productive discussion about all aspects of the research project, can place it in a broader context, can critically reflect on its strengths and weaknesses, and can aptly respond to questions and suggest new research directions that could further elucidate the research question. | | Form and style | The student mumbles or is otherwise difficult to follow, and makes too much use of notes or other supportive material to stay on track. | Student is mostly clear, but makes (too much) use of notes or other supportive material, or at times has trouble expressing ideas. Engagement with the audience is limited. | Student engages with the audience and is clear, but could improve elocution. | Student uses a clear voice and correct English, so that everyone can easily follow the presentation. Student makes little use of notes or presentation slides to stay on track. | Student uses a clear voice and correct English, so that everyone can easily follow the presentation. The student makes little use of notes or presentation slides to stay on track. Through engaging examples and good elocution, the audience is captivated during the presentation. | # Using the rubrics To assess work using the rubrics, first determine how much weight you'd give each aspect in the respective rubric. Once you have determined this, pick the normative descriptions that best describe the level of the work for each aspect. When doing so, you will end up in a specific column for each aspect, which corresponds to a bandwidth of grades. You have discretion to choose the lower bound, the upper bound or somewhere else within this bandwidth. Once you have determined all the grades, you can use the weights of the aspects to arrive at a partial marks per rubric. These partial marks per rubric are then averaged according to preset weights, to arrive at a final mark: 50% written work, 40% process and 10% presentation. You can fill in the rubrics on the [[Datanose project page]] of the Research Project, once the time to assess has come (i.e. once the student has submitted the report). # Dutch grading system The Dutch grading system scores on a 1-10 scale, with 6 being the minimum pass mark. See the following website for more information: https://www.nuffic.nl/en/education-systems/netherlands/grading-systems. The University of Amsterdam has listed how the different grades in the Dutch system compare to international standards. This list can be found here: https://student.uva.nl/en/topics/the-marking-assessment-and-registration-of-your-academic-results Note that works that score higher than 8.0 are remarkable and that the grade 9 is reserved for the top 3% of students, while only the top 0.5% scores 10. # Disclosure of GenAI use The above rubrics were created with the help of GenAI (UvA AI Chat, using ``gpt-3.5-turbo``). The purpose of this use was to rephrase descriptions and to ensure different levels of performance were sufficiently differentiated. The overall process of generation consisted of: 1. Manual writing of the rubrics 2. Row-by-row rephrasing using UvA AI Chat 3. Manual review and revision of the Gen-AI assisted formulations The prompting used for rephrasing was as follows. >I am creating grading rubrics at the level of a master's programme in brain and cognitive sciences. The following rubric row is meant to measure (insert ILO). Can you improve it?