Objectives

Objective 1 concentrates on TRUMAN’s AI and ML techniques and is the starting point as it defines the characteristics and specifications of these techniques. Accordingly, the three keywords that the project works around are:

(i) “human- in-the-loop (HITL)”, because the human plays an important role in the improvement of AI systems:

(ii) distributed because data are usually generated in a distributed manner as in the case of federated learning scenarios, as a function of where the service provider and consumer are located;

(iii) “dynamic” because data continuously evolve in time.

It is crucial to monitor the process by which previous data are managed and exploited for analysing the data generated during the current time slot via methods like time-aware learning. By HITL, we mean techniques where there is a continual bidirectional human-machine interaction and where the human can spontaneously provide information. These would include different scenarios such as tagging a photo, reporting information about the human’s current activities [6, 7], or the human and the machine engaging in a discussion because the machine thinks that the human has provided wrong information, e.g. mislabeling [8, 9]. Contrarily, such an exchange could also occur when the human needs to fix the machine’s reasoning that leads to a wrong conclusion [10]. The HITL approach can be adopted in all four main ML phases (i.e., data collection, data pre-processing, model training, and model execution) [11]. The enforcement of this HITL approach motivates our choice of using Knowledge Graphs (KGs) as a more abstract and knowledge-level representation of tabular or Internet of Things (IoT) data. Indeed, KGs enforce the move from variables (or at most tabular data) to entities, from features to data properties describing entities and to object properties describing relations and actions among entities [12]. In turn, LLMs, optionally fed by KGs, provide an even further level of abstraction. This becomes key where the end-user is not an expert such that entities, their properties and relations are explained to the user in natural language, also making explicit a huge amount of common- sense knowledge that they embed [13, 14].

We will study how to adopt the HITL approach at different levels of abstraction, from IoT sensors and features to data and object properties as embedded in KGs, where KGs will facilitate the user in the reuse of past data as well as data generated by third parties up to the interaction. The proper level of data abstraction will be tuned to the specific task and application scenario with the possibility of using two representation models (e.g., features and KGs), mutually reinforcing themselves in a meaningful interaction with the user. We will develop Personal KGs (PKGs) encoding the user’s subjective perspective on the world, and we will integrate this into KGs to encompass a multi-user perspective [15]. This will require the development of Temporal KGs (TKGs) and Spatio-Temporal KGs whenever there is a need to integrate IoT data [16, 17]. It is worth noting that Knowledge Graphs (KG) operate on two levels: the schema KG (SKG) and the data KG (DKG). The reliability of KG-based reasoning heavily depends on the quality of the SKG. If the SKG is flawed, any reasoning or explanations derived from it—whether concerning entities, time, or relationships—will be inaccurate, regardless of the data’s correctness. Current literature reveals a lack of theoretical and principled approaches in this area. To address this, our project will leverage techniques from information science. This approach will not only support a multi-dimensional representation of knowledge (known as facets) but also provide the flexibility needed to incorporate and update new facts and knowledge within the KG.

Ambition 1.1

We will analyse different steps, from local to global, by which data are first generated and then used to train and use ML models. The ultimate goal is to integrate robustness solutions and evaluate their impact on the performance and accuracy of the AI systems and hence the trade-off between their trustworthiness and accuracy. We will concentrate on two main techniques, that is, how to distribute in time and space the data generation and preparation, and federated learning for what concerns the model training and usage.

Ambition 1.2

We will develop a robust and adaptable federated continual learning framework capable of effectively addressing spatial-temporal catastrophic forgetting in resource-constrained environments. The framework will be designed to enhance communication efficiency, handle model heterogeneity, and explore alternative sharing modes. Through rigorous validation in diverse real-world scenarios, including healthcare, IoT, and finance, we will demonstrate the practical effectiveness of our approach. One fundamental focus will be on applying federated learning across individuals, for instance, with the goal of doing cross-individual activity recognition [29, 30]. A second fundamental focus will be on federated learning and temporal adaptation in the design of this framework.

Ambition 1.3

The performance of AI technologies relies on access to large datasets of good quality and on the training of an accurate model. Hence, AI systems’ dependence on large data makes them vulnerable to adversarial attacks that can manipulate inputs or model parameters in order to tamper with the training process [31]. Such attacks may be hard to detect because the manipulated model is designed to exhibit adversarial behaviour on inputs which usually are only known to the attacker. The literature features various proposals to mitigate poisoning attacks [32]. Nevertheless, these solutions usually work with neural networks and sometimes require considerable and computationally heavy modification to existing algorithms.

We will first investigate and design new adversarial attacks against the TRUMAN AI systems where the learning is distributed among multiple parties, is continual, and relies on the use of graph structures. We will further develop mitigation solutions to counter these newly proposed attacks and make sure that they protect against state-of-the-art attacks as well.

Ambition 2.1

The goal is to study and identify new privacy attacks or attack implementations and further design/develop customised defense strategies, both based on differential privacy mechanisms as well as other empirical defenses, while continuously addressing the trade-off between robustness-efficiency and model quality. Moreover, TRUMAN aims to clean the training data used for LLMs to ensure that personal and sensitive information is removed or replaced with synthetic data that maintains the same format. Where successfully applied this will become novel and highly relevant enabler for KI solutions by solving an essential data protection concern while retaining the data’s value for KI training.

Ambition 2.2

TRUMAN will first review both the attacks and the existing mitigation techniques and study their suitability and performance in the context of the project’s AI systems. We will also explore mitigation techniques based on differentially private mechanisms. While some studies have shown that the adoption of differentially private mechanisms results in a negative correlation between the privacy budget and fairness violation [40, 41], recent works [42, 43] show that when applied carefully, this trade-off can be addressed or at least bounded. We will take advantage of these studies to design and develop customised solutions for the newly developed AI models.

Ambition 2.3

While the system-focused approaches mentioned above are essential for mitigating bias and enhancing robustness, they are insufficient for human-centered AI systems. Rather, they must be accompanied by techniques that enhance the explainability (of system behaviours) from the user’s point of view. Explainability of system security, including privacy measures for achieving robustness, is essential for a multitude of reasons, from enabling the user to verify the system’s behaviour and the stated security measures to learning from the system and as a means for achieving compliance with the EU AI Act 3 regarding transparency and human oversight requirements for high-risk AI systems. Moreover, usable explanations of privacy and security protection goals and trade-offs between these goals that can be achieved by security and privacy measures are also important ingredients for developers and system administrators for designing and configuring systems for privacy and security (in line with GDPR Art. 25) and for conducting a data protection impact assessment (GDPR Art. 35). Transparency and explainability are also important for establishing reliable trust [44] and are key to user acceptance of secure and robust AI technology, as it helps align user expectations with the system’s capabilities [45]. A key challenge is that, increasingly, the end-users of a given AI system are diverse (e.g., in terms of their needs and intentions, but also, their abilities and backgrounds). Thus, to enhance usability and effectiveness, explanations must be tailored to the needs of individual users. Moreover, our previous research showed that functional explanations of security technologies (in the form of functional encryption) that explain the core functions of a technology are better understood than structural explanations by both lay and technical users, while structural explanations that explains how the technology works are more trusted by both lay and technical users [46].

We will conduct user studies for investigating suitable combinations for functional and structural explanations for the provided security/privacy/robustness protection that are trustworthy, understandable and meaningful and relevant for different types of users (e.g., end-users, system admins, decision makers). Furthermore, this task will develop user interface prototypes providing such usable explanations, which will be evaluated with user tests at KU and OUC (focus groups and user tests) in at least two iteration cycles involving test users with diverse backgrounds regarding their gender, age, education, and culture.

Ambition 3

The ultimate goal of TRUMAN is to develop trustworthy solutions for AI that are ready for real-world integration. The designed solutions will, therefore, be integrated and deployed into real-world use cases. In order to explore a large number of challenges raised by different AI methods involving different data structures, algorithms and actors, the project will consider four use cases, each of which will focus on a different AI method (continual learning, KGs, or LLMs) and a particular phase of the AI life cycle (data collection, data pre-processing, model training, model execution). Hence each use case will exhibit different challenges with respect to robustness and integrate dedicated robustness and human-centric solutions.

TRUMAN will integrate and demonstrate the developed solutions with four use cases: (i) the collection of data of various types for market and scientific research on household lifestyle; (ii) the development of phishing website detectors that will make use of deep neural networks that are collectively trained (meant to assemble ground truth), deployed over several machines collecting privacy-sensitive information, and executing the AI model locally; and validated by humans to improve both accuracy and explainability, (iii) the training of a model detecting fraudulent transaction carried out by bank employees where the technology will make use of graph-structured data; and, finally, (iv) an LLM-based chatbot that will engage interactions with patients suffering from Parkinson’s disease and based on these interactions, offer advanced functionalities disease management coaching.

Ambition 4

Within the existing literature, the initial steps towards trustworthy AI systems have been rather bottom-up, driven by the specific type of data and/or the specific AI/ML technique and/or the specific domain or application scenario. For example, various solutions already exist to address one difficulty, e.g., the lack of fairness or transparency, given a specific AI technique. In parallel to addressing the problems of security, privacy, and fairness of a given AI technology, individually, we will also follow a top-bottom and holistic approach and develop a generic methodology and a road map to build trustworthy AI technologies. Indeed, the ultimate goal is to take a more principled approach towards building trustworthy AI systems.

We will analyse the problem along three dimensions, namely with respect to: (i) the different AI phases (data collection, preparation, model training, and model execution), (ii) the actual AI techniques which can exhibit different robustness issues at different phases; and, (iii) the underlying data which can be used at different levels of abstraction, depending on the specific application (for example KGs are very useful in corporate applications while LLMs are successful for applications involving human-machine interactions). The approach will also be three-fold and spiral: once the individual solutions for Objective 1 and Objective 2 are developed, selected solutions will be integrated together and evaluated using appropriate metrics, and finally, the individual solutions will be revised accordingly. Hence, while the design will follow a bottom-up approach, starting from the specific ML techniques and types of data, the results, however, will be analysed and interpreted top-down, aimed at providing the general methodology mentioned above. The ultimate goal is to produce guidelines towards building robust and human-centric AI systems effectively.

Ambition 5

Throughout TRUMAN’s duration, its research topics and results addressing the innovation challenge of the call will continuously be publicised and promoted to raise awareness about the requirements of Trustworthy AI covering all the aspects of robustness (security, privacy, and fairness).

This final objective will accommodate directions for ethics and privacy through various channels of dissemination tools such as publications, workshops, and organisation of events that have the potential of bringing different communities together. Among these communities, TRUMAN targets: (i) researchers in the various disciplines TRUMAN covers, including AI, security, privacy, ethics & law; industrial stakeholders; and the European R&D community. In particular, TRUMAN will seek collaboration with European projects within the same call as well as the HORIZON-CL4-2023-HUMAN-01-04 and HORIZON-CL4-2023-HUMAN-01-03 ones. Communication and dissemination activities will be performed in parallel with the technical and theoretical work with project management to efficiently coordinate the various activities concerned with knowledge transfer.

Ambition 6