eFiL Final Report
e–Feedback for interactive Lecture
LP3C- MSHB – Université Rennes 2 (Eric Jamet, Nicolas Michinov, Estelle Michinov)
INTUIDOC – IRISA – INSA Rennes (Eric Anquetil, Nathalie Girard)
DUKE – LS2N – Université de Nantes (Yannick Prié, Julien Blanchard, Antoine Pigeau, Olivier Aubert)
2-year CominLabs education Project (2017-2019)
This website presents the results of the eFiL project. Its main points are summarized in an Executive Summary (4 page PDF)
Note: the content of this website is also available as a 16 page PDF
1. Introduction
The eFiL project is a 2 year project consisting of 80% engineering and 20% research. In line with the CominLabs call, we chose to recruit 3 engineers and concentrate on the development of artifacts and preliminary studies. This approach gave us a better understanding of the issues and leads to meaningful research questions.
The eFIL project focused on the design, development, and assessment of KASSIS, a new interactive digital notebook for active learning in higher education. KASSIS had been designed and developed over the previous 2 years at INSA Rennes, in collaboration with lecturers and IRISA laboratory’s IntuiDoc team.
The main objectives of the eFiL project were to:
1) consolidate and improve the usability of the KASSIS solution (LP3C, IRISA-IntuiDoc) by applying a user-centered design (UCD) approach;
2) examine the uses of the KASSIS solution and its impact on active learning in higher education using, for example, innovative collaborative whiteboards, new types of graphic quizzes, and new visualization tools to deliver collective feedback such as heatmaps and clusters of thumbnails (LP3C);
3) enrich the solution by using student and teacher activity tracking (log) to design dashboards for the teacher and study appropriation and usage strategies for the students (LS2N + LP3C).
Summary of main achievements and research findings
Objective 1: Consolidate and improve the KASSIS solution
Main achievements
- Regular usability tests and surveys were conducted to consolidate and extend the KASSIS environment, applying a multidisciplinary UCD approach.
- KASSIS was extended by designing two innovative functionalities for the automatic graphic summary of collective feedback: an interactive saliency map (heatmap) and interactive graphic clustering.
- We conceived an automated generic clustering of handwritten graphics (Symbols, Mathematical equations, Drawings…) based on fuzzy C-Means approach with a dedicated feature selection using HBF49 features and t-SNE algorithm. The approach was validated using the “QuickDraw” dataset from Google.
Main achievements
- Many experimental studies were carried out to test KASSIS in various teaching situations (eg. peer instruction, collaborative drawing) and disciplines (e.g. computer science, ux design, psychology, physiotherapy).
- Together, these studies and tests involved more than 1,158 students and 14 different teachers (see Table 1).
Main research findings
- Note taking was faster with a keyboard, followed by paper and pen-based tablet; using a tablet with a pen may be particularly relevant for disciplines in which sketches or formulae are frequently used (studies 1 and 2).
-
A very high level of interest for KASSIS collaborative drawing activity was demonstrated among both teachers and learners in medical education but this activity needs to be guided to be efficient (studies 3 and 4).
- Regular quizzes administered with KASSIS during the lecture improved learning outcomes more than quizzes administered at the end of the lecture (studies 5a and 5b).
- Peer Instruction may be extended to graphic quizzes (instead of multiple-choice questions), and a collective feedback to the whole class can be given in a heatmap format (instead of bar charts) with positive effects on learning outcomes when the teacher guided students about how to use the heatmap (study 6).
- Students using the KASSIS solution (graphic quizzes) were more satisfied, interested in the lesson, and understood better the concepts than those in traditional teaching (graphic quizzes on slides), but no difference was observed on academic performance (study 8).
Objective 3: Enrich Kassis with Learning Analytics dashboards
Main achievements
- Two types of dashboard (realtime for in-class monitoring, and post-class for more in-depth analysis of how the session unfolded) were iteratively designed and implemented in a UCD multidisciplinary approach, using prototyping and user studies to validate the dashboard proposals.
- The KASSIS environment was instrumented to capture interaction traces, thereby providing material for research activities and the design of teacher dashboards.
- Preliminary data analyses were carried out, yielding a number of ideas that could not, however, be properly validated, owing to a lack of data.
Main research findings
- The iterative methodology we used for trace modelling and dashboard design seems sound, and we want to use it again in other contexts to assess its relevance.
- There clearly is value in fine-grained analysis of course unfolding with new types of visualizations, for teachers and for researchers.
- Dashboards were recognized as helpful by teachers.
2. Results
Results are described in dedicated sections of this website :
Results for Objective 1: Consolidate and improve the KASSIS solution
Results for Objective 3: Enrich Kassis with Learning Analytics dashboards
3. Conclusions and perspectives
We achieved our goals, which were to
1: Consolidate and improve the KASSIS solution;
2: Examine the uses and impact of the KASSIS solution on active learning in higher education;
3: Enrich Kassis with learning analytics dashboards.
The project’s main outputs are an enhanced and more robust version of KASSIS, along with multiple studies (some of them still pending analysis) evaluating the uses of KASSIS and its impact on active learning, and providing guidelines for future application deployments. The KASSIS application is now instrumented to capture interaction traces, which can then be provided to teachers (and analysts) through dashboards, thereby opening up enhanced opportunities for reflective teaching. These assets were obtained through the work of a genuinely interdisciplinary consortium, with good relations and high levels of mutual understanding (interdisciplinarity is always risky).
We could not test the dashboards as thoroughly as we would have liked, owing to the unavoidable amount of time needed for modelling/designing/implementation, as well as the need to have real traces for prototyping. Time is needed for bootstrapping before the anticipated indicators can be contrasted with the reality of actual traces. On the bright side, we now have an instrumented version of the application, that will shorten the amount of time required for further developments and experiments.
Some results are still pending. For instance, we are working on a cross-team article on the creation and usage of dashboards, and investigating the automation of some forms of learning analytics through trace mining, in order to gain a deeper (self)understanding of how the sessions unfold.
Regarding future prospects, the KASSIS application could be more widely deployed in higher education, to build on its existing features and allow larger-scale collection of learning analytics. It could also be deployed in other settings (high school, college, etc.) and/or in other disciplines.
The same interdisciplinary consortium has submitted a proposal for the new Cominlabs call, entitled GUIDE, which is aimed at providing tools and assistance for the drawing of figures in the context of medical studies-a field that we identified during the eFiL project. This would give teachers more sophisticated tools, and also allow for the continued capture of activity traces, in order to capitalize on this asset.
