This is a very big and interesting idea to grapple with. In terms of **A**, I think building a robust audit process starts with building a not-so robust audit process. I would say that there's a practical answer to this question and also a more 'philosphical' one in terms of how you do the practical process (ways of working). So the first stage of the practical process is creating your initial, relatively naive ethical AI approach: A good jumping off point is to begin conducting some research and using this to build an *ethical* framework for your audit process. First port of call is to select an ethical/policy basis that you believe in or you know is going to legally apply to your use case(s), e.g. AI for Good initiative recommendations. You can use this policy basis to codify the standards, transparency processes, & alignment with regulation you need to meet, and which you will need to test for adherence to in your *technical* audit process. Then you can build a simple technical auditing framework by answering the following questions: what statistical methods will be used to test data/model fairness/quality, and how will this vary based on the type of data/model? How do these tests relate back to the standards outlined in your ethical framework, and what is your acceptance threshold/definition of done? Again, you will most likely need to perform some kind of literature review here to select tools & algorithms that are the most robust in criteria that make sense to your team. You should discuss with technical & non-technical stakeholders as you do this to make sure as much is captured as possible. Crucially, these frameworks should be living artefacts that are regularly reviewed and reflected on. This should enable them to grow alongside changing science, policy, & your learning from auditing mistakes (there will be many of these, especially early on). This is the very basic level you need to reach before you can start making things more robust. Once you've got a simple approach down you can start to test out more layers & techniques, & based on their efficacy introduce them permanently into your frameworks. The sky is the limit here, but we have used the following approaches at my company to great success: Red teaming is a concept in cybersecurity wherein ethical hackers simulate cyberattacks against your organisation's systems. You can do the same thing for ethical AI; for example attempt to build an adversarial model that can infer protected characteristics from your model. Introducing documentation requirements is also helpful, because A it encourages people to think about things and B it makes it easier to decide on testing methods from your framework to apply. You can use Model Cards for model reporting and Datasheets/Spec Sheets for datasets to capture details like intended use, limitations, & testing conditions. Requiring audit trails during your audit & development processes can also make them more robust. Log all design decisions, training data sources, & evaluation processes. Where possible, you should connect them to artefacts & sign-offs so they are verifiable & misuse is discouraged. So practically, this is how I think you can build a robust auditing process. On a more 'ways of working' level, I think a great process can only be built in an augmented captial-a Agile or SCRUM kind of way (i.e. cycling but retaining quite comprehensive documentation so learning gets embedded, maybe you could even knowledge graph on top of this?): I think there needs to be a repeating cycle of research, discussion, trying, learning, & documenting in order to develop anything close to a solid audit process. I also think that if organisations/teams do this, share their approach publically, or at least across the organisation in sensitive contexts, & then discuss/reject/adopt incoming critique this process can be supercharged. On top of this, I believe that adopting the Agile/SCRUM idea of 'living' the Agile values & processes is also necessary. Accepting failure, never assigning individual blame in your team, & accepting that you will have to grow your process & knowledge by working iteratively will ultimately lead to a better process & a happier AI team! In short, building an acceptable audit process is quite easy to do, building a robust audit process is really hard to do; & certainly cannot be done by siloed individuals in silence!