MIT 6.S191: Robust and Trustworthy Deep Learning

Transcription for the video titled "MIT 6.S191: Robust and Trustworthy Deep Learning".


Note: This transcription is split and grouped by topics and subtopics. You can navigate through the Table of Contents on the left. It's interactive. All paragraphs are timed to the original video. Click on the time (e.g., 01:53) to jump to the specific portion of the video.


Introduction and Themis AI (00:00)

I'm really excited especially for this lecture which is a very special lecture on robust and trustworthy deep learning by one of our sponsors of this amazing course Themis AI and as you'll see today Themis AI is a startup, actually locally based here in Cambridge. Our mission is to design, advance, and deploy the future of AI and trustworthy AI specifically. I'm especially excited about today's lecture because I co-founded Themis right here at MIT. Right here in this very building, in fact. This all stemmed from really the incredible scientific innovation and advances that we created right here, just a few floors higher than where you're sitting today. And because of our background in really cutting-edge scientific innovation, stemming from MIT, Themis is very rooted deeply in science and, like I said, innovation. We really aim to advance the future of deep learning and AI, and much of our technology has already grown from published research that we've published at top-tier peer review conferences in the AI venues around the world. And our work has been covered by high-profile international media outlets. This scientific innovation, with this scientific innovation, Themis, we are tackling some of the biggest challenges in safety-critical AI that exist today, and really that stems from the fact that we want to take all of these amazing advances that you're learning as part of this course and actually achieve them in reality as part of our daily lives. And we're working together with leading global industry partners across many different disciplines ranging from robotics, autonomy, healthcare and more to develop a line of products that will guarantee safe and trustworthy AI. And we drive this really deeply with our technical engineering and machine learning team. And our focus is very much on the engineering, very flexible and very modular platforms to scale algorithms towards robust and trustworthy AI. This really enables this deployment towards grand challenges that our society faces with AI today. Specifically, the ability for AI solutions today are not very trustworthy at all, even if they may be very high performance on some of the tasks that we study as part of this course. So it's an incredibly exciting time for Themis in specific right now. We're VC backed, we're located, our offices are right here in Cambridge, so we're local. And we have just closed a round of funding. So we're actively hiring the best and the brightest engineers, like all of you, to realize the future of safe and trustworthy AI. And we hope that really today's lecture inspires you to join us on this mission to build the future of AI. And with that, it's my great pleasure to introduce Sadhana. Sadhana is a machine learning scientist at Themis. She's also the lead TA of this course, Intro to Deep Learning, at MIT. Her research at Themis focuses specifically on how we can build very modular and flexible methods for AI and building what we call a safe and trustworthy AI. And today, she'll be teaching us more about specifically the bias and the uncertainty realms of AI algorithms, which are really two key or critical components towards achieving this mission or this vision of safe and trustworthy deployment of AI all around us. So thank you and please give a big warm round of applause for Sadhana.

Deep Dive Into Deep Learning Challenges And Solutions

Background (03:46)

Thank you and please give a big warm round of applause for Sadhana. Thank you so much, Alexander, for the introduction. Hi everyone, I'm Sadhana. I'm a machine learning scientist here at Themis AI and the lead TA of the course this year. And today, I'm super excited to talk to you all about robust and trustworthy deep learning on behalf of Themis. all about robust and trustworthy deep learning on behalf of Themis. So over the past decade we've seen some tremendous growth in artificial intelligence across safety critical domains. In the spheres of autonomy and robotics we now have models that can make critical decisions about things like self-driving at a second's notice and these are paving the way for fully autonomous vehicles and robots. And that's not where this way for fully autonomous vehicles and robots. And that's not where this stops. In the spheres of medicine and healthcare, robots are now equipped to conduct life saving surgery. We have algorithms that generate predictions for critical drugs that may cure diseases that we previously thought were incurable. And we have models that can automatically diagnose diseases without intervention from any healthcare professionals at all. These advances are revolutionary and they have the potential to change life as we know it today. But there's another question that we need to ask, which is where are these models in real life? A lot of these technologies were innovated five, 10 years ago, but you and I don't see them in our daily lives. So what's the gap here between innovation and deployment? The reason why you and I can't go buy self-driving cars or robots don't typically assist in operating rooms is this. These are some headlines about the failures of AI from the last few years alone. In addition to these incredible advances, we've also seen catastrophic failures in every single one of the safety-critical domains I just mentioned. These problems range from crashing autonomous vehicles to healthcare algorithms that don't actually work for everyone, even though they're deployed out in the real world so everyone can use them. Now, at a first glance, this seems really demoralizing. If these are all of the things wrong with artificial intelligence, how are we ever going to achieve that vision of having our AI integrated into the fabric of our daily lives in terms of safety critical deployment? But at Themis this is exactly the type of problem that we solve. We want to bring these advances to the real world and the way we do this is by innovating in the spheres of safe and trustworthy artificial intelligence in order to bring the things that were developed in research labs around the world to customers like you and me. And we do this by our core ideology is that we believe that all of the problems on this slide are underlaid by two key notions. The first is bias. Bias is what happens when machine learning models do better on some demographics than others. This results in things like facial detection systems not being able to detect certain faces with high accuracy, Siri not being able to recognize voices with accents, or algorithms that are trained on imbalanced data sets. So what the algorithm believes is a good solution doesn't actually work for everyone in the real world. And the second notion that underlies a lot of these problems today is unmitigated and uncommunicated uncertainty. This is when models don't know when they can or can't be trusted. And this results in scenarios such as self-driving cars continuing to operate in environments when they're not a hundred percent confident instead of giving control to users, or robots moving around in environments that they've never been in before and have high unfamiliarity with. And a lot of the problems in modern-day AI are the result of a lot of the problems in modern day AI are the result of a combination of unmitigated bias and uncertainty.

Challenges for Robust Deep Learning (07:29)

So today in this lecture, we're going to focus on investigating the root causes of all of these problems, these two big challenges to robust deep learning. We'll also talk about solutions for them that can improve the robustness and safety of all of these algorithms for everyone. And we'll start by talking about bias. Bias is a word that we've all heard outside the context of deep learning. But in the context of machine learning, it can be quantified and mathematically defined. Today, we'll talk about how to do this and methods for mitigation of this bias algorithmically and how Themis is innovating in these areas in order to bring new algorithms in this space to industries around the world. Afterwards we'll talk about uncertainty which is can we teach a model when it does or doesn't know the answer to its given task and we'll talk about the ramifications for this for real-world AI.

What is Algorithmic Bias? (08:24)

So what exactly does bias mean and where is it present in the artificial intelligence life cycle? The most intuitive form of bias comes from data. We have two different two main types of bias here. The first is sampling bias, which is when we oversample from some regions of our input data distribution and undersample from others. A good example of this is a lot of clinical data sets where they often contain fewer examples of diseased patients than healthy patients because it's much easier to acquire data for healthy patients than their disease counterparts. In addition, we also have selection bias at the data portion of the AI lifecycle. Think about Apple's series voice recognition algorithm. This model is trained largely on flawless American English, but it's deployed across the real world to be able to recognize voices with accents from all over the world. The distribution of the model's training data doesn't match the distribution of this type of language in the real world because American English is highly over-represented as opposed to other demographics. But that's not where bias in data stops. These biases can be propagated towards models training cycles themselves, which is what we'll focus on in the second half of this lecture. Then once the model is actually deployed, which means it's actually put out into the real world and customers or users can actually get the predictions from it, we may see further biases perpetuated that we haven't seen before. The first of these is distribution shifts. Let's say I have a model that I trained on the past 20 years of data, and then I deploy it into the real world. In 2023, this model will probably do fine because the data input distribution is quite similar to data in the training distribution. But what would happen to this model in 2033? It probably would not work as well because the distribution that the data is coming from would shift significantly across this decade. And if we don't continue to update our models with this input stream of data, we're going to have obsolete and incorrect predictions. And finally, after deployment, there is the evaluation aspect. So think back to the Apple Siri example that we've been talking about. If the evaluation metric or the evaluation data set that Siri was evaluated on was also mostly comprised of American English, then to anybody this model will look like it does extremely well, right? It can detect, it can recognize American English voices with extremely high accuracy and therefore is deployed into the real world. But what about its accuracy on subgroups, on accented voices, on people who, for whom English is not their first language? If we don't also test on subgroups in our evaluation metrics, we're going to face evaluation bias. So now let's talk about another example in the real world of how bias can perpetuate throughout the course of this artificial intelligence lifecycle. Commercial facial detection systems are everywhere. You actually played around with some of them in lab 2 when you trained your VAE on a facial detection data set. In addition to the lock screens on your cell phones, facial detection systems are also present in the lock screens on your cell phones, facial detection systems are also present in the automatic filters that your phone cameras apply whenever you try to take a picture, and they're also used in criminal investigations. These are three commercial facial detection systems that were deployed and we'll analyze the biases that might have been present in all of them for in the next few minutes. So the first thing you may notice is that there is a huge accuracy gap between two different demographics in this plot. This accuracy gap can get up to 34%. Keep in mind that this facial detection is a binary classification task. Everything is either a face or it's not a face. This means that a randomly initialized model would be expected to have an accuracy of 50% because it's going to randomly assign whether or not something is a face or not. Some of these facial detection classifiers do only barely better than random on these underrepresented data, on these underrepresented samples in this population. So how did this happen? Why is there such a blatant gap in accuracy between these different demographic groups? And how did these models ever get deployed in the first place? What types of biases were present in these models? So a lot of facial detection systems exhibit very clear selection bias. This model was likely trained mostly on lighter skin faces and therefore learned those much more effectively than it learned to classify darker skin faces. But that's not the only bias that was present. The second bias that's often very present in facial detection systems is evaluation bias because originally this data set that you see on the screen is not the data set that these models were evaluated on. They were evaluated on one big bulk data set without any classification into subgroups at all. And therefore, you can imagine if the data set was also comprised mostly of lighter skinned faces, these accuracy metrics would be incredibly inflated and therefore would cause incredibly inflated and therefore would cause unnecessary confidence and we could deploy them into the real world. In fact, the biases in these models were only uncovered once an independent study actually constructed a data set that is specifically designed to uncover these sorts of biases by balancing across race and gender. However, there are other ways that data sets can be biased that we haven't yet talked about.

Class imbalance (14:13)

So, so far we've assumed a pretty key assumption in our dataset, which is that the number of faces in our data is the exact same as the number of non-faces in our data. But you can imagine, especially if you're looking at things like security feeds, this might not always be the case. You might be faced with many more negative samples than positive samples in your data set. So what's the problem here? In the most extreme case, we may assign the label non-face to every item in the data set because the model sees items that are labeled as faces so infrequently that it isn't able to learn an accurate class boundary between the two classes. So how can we mitigate this? This is a really big problem and it's very common across a lot of different types of machine learning tasks and datasets. And the first way that we can try to mitigate class imbalance is using sample reweighting, which is when instead of uniformly sampling from our dataset at a rate, um, we instead sample at a rate that is inversely proportional to the incidence of a class in our dataset. So in the previous example, if, um, the likelihood, if faces were much- if the number of faces which was much lower than the number of non-faces in our dataset, we would sample the faces with a higher probability than the negatives so that the model sees both classes equally. The second example, the second way we can mitigate class imbalance is through loss reweighting, which is when instead of having every single mistake that the model makes contribute equally to our total loss function, we reweight the samples such that samples from underrepresented classes contribute more to the loss function. So instead of the model assigning every single input face to a as a negative, it'll be highly penalized if it does so because the loss of the faces would contribute more to the total loss function than the loss of the negatives. And the final way that we can mitigate class imbalance is through batch selection, which is when we choose randomly from classes so that every single batch has an equal number of data points per class.

Latent feature imbalance (16:25)

So is everything solved? Clearly, there are other forms of bias that exist even when the classes are completely balanced, because the thing that we haven't thought about yet is latent features. So, if you remember from Lab 2 and the last lecture, latent features are the actual representation of this image according to the model. And so far we've mitigated the problem of when we know that we have underrepresented classes, but we haven't mitigated the problem of when we have a lot of variability within the same class. Let's say we have an equal number of faces and negative examples in our data set. What happens if the majority of the faces are from a certain demographic or they have a certain set of features? Can we still apply the techniques that we just learned about? The answer is that we cannot do this and the problem is that the bias present right now is in our latent features. All of these images are labeled with the exact same label so according according to the model, all we know is that they're all faces. So we have no information about any of these features only from the label. Therefore, we can't apply any of the previous approaches that we used to mitigate class imbalance because our classes are balanced but we have feature imbalance now. However, we can adapt the previous methods to account for bias in latent features, which we'll do in just a few slides. So let's unpack this a little bit further. We have our potentially biased data set, and we're trying to build and deploy a model that classifies the faces in a traditional training pipeline. This is what that pipeline would look like. We would train our classifier and we would deploy it into the real world, but this training pipeline doesn't de-bias our inputs in any way. So one thing we could do is label our biased features and then apply resampling. So let's say in reality that this data set was biased on hair color. Most of the data set is made up of people with blonde hair, with faces with black hair and red hair underrepresented. If we knew this information, we could label the hair color of every single person in this dataset and we could apply either sample reweighting or loss reweighting just as we did previously. But does anyone wanna tell me what the problem is here? Yeah, if you go through each samples, it takes a lot of time. Yeah. So there are a couple of problems here, and that's definitely one of them. The first is, how do we know that hair color is a biased feature in this data set? Unless we visually inspect every single sample in this data set, we're not going to know what the biased features are. And the second thing is exactly what you said, which is once we have our biased features, going through and annotating every image with this feature is an extremely labor-intensive task that is infeasible in the real world. So now the question is, what if we had a way to automatically learn latent features and use this learned feature representation to de-bias a model. So what we want is a way to learn the features of this data set and then automatically determine the samples with the highest feature bias and the samples with the lowest feature bias. We've already learned a method of doing this. In the generative modeling lecture, you all learned about variational autoencoders, which are models that learn the latent features of a dataset. As a recap, variational autoencoders work by probabilistically sampling from a learn latent space, and then they decode this new latent vector into back into the original input space, measure the reconstruction loss between the inputs and the outputs and continue to update their representation of the latent space. And the reason why we care so much about this latent space is that we want samples that are similar to each other in the input to decode to latent vectors that are very close to each other in this latent space. And samples that are far from each other or samples that are dissimilar to each other in the input should decode to should encode to latent vectors that are far from each other in the input should encode to latent vectors that are far from each other in the latent space.

Debiasing variational autoencoder (DB-VAE) (20:30)

So now we'll walk through step-by-step a de-biasing algorithm that automatically uses the latent features learned by a variational autoencoder to undersample and oversample from regions in our data set. Before I start, I want to point out that this de-biasing model is actually the foundation of Themis' work. This work comes out of a paper that we published a few years ago that has been demonstrated to de-bias commercial facial detection algorithms, and it was so impactful that we decided to make it available and work with companies and industries, and that's how Themis was started. So let's first start by training a VAE on this data set. The Z shown here in this diagram ends up being our latent space and the latent space automatically captures features that were important for classification. So here's an example latent feature that this model captured. This is the facial position of an input face. And something that's really crucial here is that we never told the model to calculate the, uh, to encode the feature vector of, um, the facial position of a given face. It learned this automatically because this feature is important for the model to develop a good representation of what a face actually is. So now that we have our latent structure, we can use it to calculate a distribution of the inputs across every latent variable and we can estimate a probability distribution depending on, that's based on the features of every item in this data set. that's based on the features of every item in this dataset. Essentially, what this means is that we can calculate the probability that a certain combination of features appears in our dataset based on the latent space that we just learned. And then we can oversample denser or sparser areas of this dataset and undersample from denser areas of this dataset. So let's say our distribution looks something like this. This is an oversimplification, but for visualization purposes, and the denser portions of this dataset, we would expect to have, um, a homogenous skin color and pose and hair color and very good lighting. And then in the sparser portions of this dataset, we would expect to see diverse skin color, pose, and illumination. So now that we have this distribution and we know what areas of our distribution are dense and which areas are sparse, we want to undersample areas from the undersample samples that fall in the denser areas of this distribution and oversample data points that fall in the sparser areas of this distribution. So for example, we would probably undersample points with the very common skin colors, hair colors, and good lighting that is extremely present in this data set, and oversample the diverse images that we saw on the last slide. And this allows us to train in a fair and unbiased manner.

DB-VAE mathematics (23:24)

To dig in a little bit more into the math behind how this resampling works, this approach basically approximates the latent space via a joint histogram over the individual latent variables. So we have a histogram for every latent variable z sub i, and what the histogram essentially does is it discretizes the continuous distribution so that we can calculate probabilities more easily. Then we multiply the probabilities together across all of the latent distributions. And then after that, we can develop an understanding of the joint distribution of all of the samples in our latent space. Based on this, we can define the adjusted probability for sampling for a particular data point as follows. The probability of selecting a sample data point x will be based on the latent space of x, such that it is the inverse of the joint approximated distribution. We have a parameter alpha here, which is a debiasing parameter, and as alpha increases, this probability will tend to the uniform distribution, and as alpha decreases, which is a debiasing parameter. And as alpha increases, this probability will tend to the uniform distribution. And if alpha decreases, we tend to debiase more strongly. And this gives us the final weight of the sample in our dataset that we can calculate on the fly and use it to adaptively resample while training. And so once we apply this debiasing, we have pretty remarkable results. This is the original graph that shows the accuracy gap between the darker males and the lighter males in this dataset. Once we apply the debiasing algorithm, where as alpha gets smaller, we're debiasing more and more as we just talked about, this accuracy gap decreases significantly. And that's because we tend to oversample samples with darker skin color, and therefore the model learns them better and tends to do better on them. Keep this algorithm in mind because you're going to need it for the Lab 3 competition, which I'll talk more about towards the end of this lecture. So, so far, we've been focusing mainly on facial recognition systems and a couple of other systems as canonical examples of bias. However, bias is actually far more widespread in machine learning. Consider the example of autonomous driving. Many data sets are comprised mainly of cars driving down straight and sunny roads in really good weather conditions with very high visibility. And this is because the data for these cars, for these algorithms, is actually just collected by cars driving down roads. However, in some specific cases you're going to face adverse weather, bad visibility, near collision scenarios. And these are actually the samples that are the most important for the model to learn. Because they're the hardest samples and they're the samples where the model is most likely to fail. But in a traditional autonomous driving pipeline, these samples are often extremely low rep- have extremely low representation. So this is an example where using the unsupervised latent de-biasing that we just talked about, we would be able to up sample these important data points and under sample the data points of driving down straight and sunny roads. Similarly, consider the example of large language models. An extremely famous paper a couple years ago showed that if you put terms that imply female or women into a large language model powered job search engine, you're going to get roles such as artist or things in the humanities. But if you output similar things but of the male counterpart, you put things like male into the the search engine, you'll end up with roles for scientists and engineers. So this type of bias also occurs regardless of the task at hand for a specific model. And finally, let's talk about healthcare recommendation algorithms. These recommendation algorithms tend to amplify racial biases. A paper from a couple years ago showed that black patients need to be significantly sicker than their white counterparts to get the same level of care. And that's because of inherent bias in the data set of this model. And so in all of these examples, we can use the above algorithmic bias mitigation method to try and solve these problems and more.

Uncertainty in deep learning (27:40)

So we just went through how to mitigate some forms of bias in artificial intelligence and where these solutions may be applied. And we talked about a foundational algorithm that Themis uses that you all will also be developing today. And for the next part of the lecture, we'll focus on uncertainty, or when a model does not know the answer. We'll talk about why uncertainty is important and how we can estimate it, and also the applications of uncertainty estimation. So to start with, what is uncertainty and why is it necessary to compute? Let's look at the following example. This is a binary classifier that is trained on images of cats and dogs. For every single input, it will output a probability distribution over these two classes. Now, let's say I give this model an image of a horse. It's never seen a horse before. The horse is clearly neither a cat nor a dog. However, the model has no choice but to output a probability distribution because that's how this model is structured. However, what if in addition to this prediction, we also achieved a confidence estimate? In this case, the model should be able to say, I've never seen anything like this before and I have very low confidence in this prediction, so you as the user should not trust my prediction on this model. And that's the core idea behind uncertainty estimation. So in the real world, uncertainty estimation is So in the real world, uncertainty estimation is useful for scenarios like this. This is an example of a Tesla car driving behind a horse-drawn buggy, which are very common in some parts of the United States. It has no idea what this horse-drawn buggy is. It first thinks it's a truck and then a car and then a person. And it continues to output predictions, even though it is very clear that the model does not know what this image is. And now you might be asking, OK, so what's the big deal? It didn't recognize the horse-drawn buggy, but it seems to drive successfully anyway. However, the exact same problem that resulted in that video has also resulted in numerous autonomous car crashes.

Types of uncertainty in AI (29:50)

So let's go through why something like this might have happened. There are multiple different types of uncertainty in neural networks, which may cause incidents like the ones that we just saw. We'll go through a simple example that illustrates the two main types of uncertainty that we'll focus on in this lecture. So let's say I'm trying to estimate the curve y equals x cubed as part of a regression task. The input here, x, is some real number, and we want it to output f of x, which should be ideally x cubed. So right away you might notice that there are some issues in this data set. Assume the red points in this image are your training samples. So the boxed area of this image shows data points in our data set where we have really high noise. These points do not follow the curve y equals x cubed. In fact, they don't really seem to follow any distribution at all, and the model won't be able to compute outputs for the area points in this region accurately because very similar inputs have extremely different outputs, which is the definition of data uncertainty. We also have regions in this dataset where we have no data. So if we query the model for a prediction in this part of, in this region of the dataset, we should not really expect to see an accurate result because the model has never seen anything like this before. And this is what is called model uncertainty, when the model hasn't seen enough data points or cannot estimate that area of the input distribution accurately enough to output a correct prediction. So what would happen if I added the following blue training points to the areas of the dataset with high model uncertainty. Do you think the model uncertainty would decrease? Raise your hand. Okay. Does anyone think it would not change? Okay. So yeah, most of you were correct. Model uncertainty can typically be reduced by adding in data into any region, but specifically regions with high model uncertainty. by adding in data into any region, but specifically regions with high model uncertainty. And now what happens if we add these blue data points into this data set? Would anyone expect the data uncertainty to decrease? You can raise your hand. That's correct. So data uncertainty is irreducible. In the real world, the blue points and the noisy red points on this image correspond to things like robot sensors. Let's say I'm I have a robot that's trained to, that has a sensor that is making measurements of depth. If the sensor has noise in it, there's no way that I can add any more data into the system to reduce that noise unless I replaced my sensor entirely.

Aleatoric vs epistemic uncertainty (32:48)

So now let's assign some names to the types of uncertainty that we just talked about. The blue area or the area of high data uncertainty is known as aleatoric uncertainty. It is irreducible as just mentioned, and it can be directly learned from data, which we'll talk about in a little bit. The green areas of the green boxes that we talked about, which were model uncertainty, are known as epistemic uncertainty. And this cannot be learned directly from the data.

Estimating aleatoric uncertainty (33:29)

However, it can be reduced by adding more data into our systems into these regions. Okay, so first let's go through aleatoric uncertainty. So the goal of estimating aleatoric uncertainty is to learn a set of variances that correspond to the input. Keep in mind that we are not looking at a data distribution and we are, as humans, are not estimating the variance. We're training the model to do this task. And so what that means is typically when we train a model, we give it an input x and we expect an output y hat, which is the prediction of the model. Now, we also predict an additional sigma squared. So we add another layer to our model of the same output size that predicts a variance for every output. So the reason why we do this is that we expect that areas in our dataset with high data uncertainty are going to have higher variance. And the crucial thing to remember here is that this variance is not constant. It depends on the value of x. We typically tend to think of variance as a single number that parameterizes an entire distribution. However, in this case, we may have areas of our input distribution with really high variance and we may have areas with very low variance. So our variance cannot be independent of the input, and it depends on our input x. So now that we have this model, we have an extra layer attached to it. In addition to predicting y hat, we also predict a sigma squared. How do we train this model? Our current loss function does not take into account variance at any point. This is your typical mean squared error loss function that is used to train regression models. And there's no way, training from this loss function, that we can learn whether or not the variance that we're estimating is accurate. So in addition to adding another layer, to estimate aleatoric uncertainty correctly, we also have to change our loss function. So the mean squared error actually learns a multivariate Gaussian with a mean yi and constant variance. And we want to generalize this loss function to when we don't have constant variance. And the way we do this is by changing the loss function to the negative log likelihood. We can think about this for now as a generalization of the mean squared error loss to non-constant variances. So now that we have a sigma squared term in our loss function, we can determine how accurately the sigma and the y that we're predicting parameterize the distribution that is our input. So now that we know how to estimate aleatoric uncertainty, let's look at a real-world example. For this task, we'll focus on semantic segmentation, which is when we label every pixel of an image with its corresponding class. We do this for scene understanding and because it is more fine-grained than a typical object detection algorithm. So the inputs of this data to this data set are known as it's from a data set called cityscapes and the inputs are RGB images of scenes. The labels are pixel-wise annotations of this entire image of which label every pixel belongs to. And the outputs try to mimic the labels. They're also predicted pixel-wise masks. So why would we expect that this data set has high natural aleatoric uncertainty? And which parts of this data set do you think would have aleatoric uncertainty? Because labeling every single pixel of an image is such a labor-intensive task, and it's also very hard to do accurately, we would expect that the boundaries between objects in this image have high aleatoric uncertainty. And that's exactly what we see. If you train a model to predict aleatoric uncertainty on this dataset, corners and boundaries have the highest aleatoric uncertainty, because even if your pixels are like one row off or one column off, that introduces noise into the model. The model can still learn in the face of this noise, but it does exist and it can't be reduced. So now that we know about data uncertainty or aleatoric uncertainty, let's move on to learning about epistemic uncertainty.

Estimating epistemic uncertainty (37:42)

As a recap, epistemic uncertainty can best be described as uncertainty in the model itself, and it is reducible by adding data to the model. So with epistemic uncertainty, essentially what we're trying to ask is, is the model unconfident about a prediction? So a really simple and very smart way to do this is, let's say I train the same network multiple times with random initializations, and I ask it to predict the exact- I- I call it on the same input. So let's say I give model 1 the exact same input and the blue x is the output of this model. And then I do the same thing again with model 2, and then again with model 3, and again with model 4. These models all have the exact same hyperparameters, the exact same architecture, and they're trained in the same way. The only difference between them is that their weights are all randomly initialized, so where they start from is different. And the reason why we can use this to determine epistemic uncertainty is because we would expect that with familiar inputs in our network, our networks should all converge to around the same answer, and we should see very little variance in the logits or the outputs that we're predicting. However, if a model has never seen a specific input before or that input is very hard to learn, all of these models should predict slightly different answers, and the variance of them should be higher than if they were predicting a similar input. So creating an ensemble of networks is quite simple. You start out with defining the number of ensembles you want, you create them all the exact same way, and then you fit them all on the same training data and, uh, training data. And then afterwards, when at inference time, we call ev- all of the models, every model in the ensemble on our specific input. And then we can treat, um, our new prediction as the average of all of the ensembles. This results in a usually more robust and accurate prediction, and we can treat the uncertainty as the variance of all of these predictions. Again, remember that if we saw familiar inputs or inputs with low epistemic uncertainty, we should expect to have very little variance, and if we had a very unfamiliar input or something that was out of distribution or something the model hasn't seen before, we should have very high epistemic uncertainty or variance. So what's the problem with this? Can anyone raise their hand and tell me what a problem with training an ensemble of networks is? So training an ensemble of networks is really compute expensive. Even if your model is not very large, training five copies of it or 10 copies of it tends to, uh, it takes up compute and time, and that's just not really feasible when we're training, um, if- on specific tasks. However, the key insight for ensembles is that by introducing some method of randomness or stochasticity into our networks, we're able to estimate epistemic uncertainty. So another way that we've seen about introducing stochasticity into networks is by using dropout layers. We've seen dropout layers as a method of reducing overfitting because we randomly drop out different nodes in our layer. And then we continue to propagate information through them, and it prevents models from memorizing data. However, in the case of epistemic uncertainty, we can add dropout layers after every single layer in our model. And in addition, we can keep these dropout layers enabled at test time. Usually we don't keep dropout layers enabled at test time because we don't want to lose any information about the network's process or any weights when we're at inference time. However, when we're estimating epistemic uncertainty we do want to keep dropout enabled at test time because that's how we can introduce randomness at inference time as well. So what we do here is we have one model, it's the same model the entire way through, we add dropout layers with a specific probability, and then we run multiple forward passes. And at every forward pass, different layers get dropped, different nodes in a layer get dropped out. So we have that measure of randomness and stochasticity. So again, in order to implement this, what we have is a model with the exact one model. And then when we're running our forward passes, we can simply run t forward passes where t is usually a number like 20. Um, we keep dropout enabled at test time and then we use the mean of these, um, samples as the then we use the mean of these, um, samples as the new prediction and the variance of it, these samples as a measure of epistemic uncertainty. So both of the methods we talked about just now involve sampling. And sampling is expensive. Ensembling is very expensive. But even if you have a pretty large model, um, having or introducing dropout layers and calling 20 forward passes might also be something that's pretty infeasible. And at Themis, we're dedicated to developing innovative methods of estimating epistemic uncertainty that don't rely on things like sampling so that they're more generalizable and they're usable by more industries and people. So a method that we've developed to estimate or a method that we've studied to estimate epistemic uncertainty is by using generative modeling. So we've talked about VAEs a couple times now but let's say I trained a VAE on the exact same data set we were talking about earlier which is only dogs and cats. The latent space of this model would be comprised of features that relate to dogs and cats, and if I give it a prototypical dog, it should be able to generate a pretty good representation of this dog, and it should have pretty low reconstruction loss. Now if I gave the same example of the horse to this VAE, the latent vector that this horse would be decoded to, would be incomprehensible to the decoder of this network. The decoder wouldn't be able to know how to project the latent vector back into the original input space. Therefore, we should expect to see a much worse reconstruction here, and we should see that the reconstruction loss is much higher than if we gave the model a familiar input or something that it was used to seeing.

Current Innovations And Future Perspectives

Evidential deep learning (44:11)

So now let's move on to what I think is the most exciting method of estimating epistemic uncertainty that we'll talk about today. So in both of the examples before, sampling is compute intensive, but generative modeling can also be compute intensive. Let's say you don't actually need a variational autoencoder for your task, then you're training an entire decoder for no reason and other than to estimate the epistemic uncertainty. So what if we had a method that did not rely on generative modeling or sampling in order to estimate the epistemic uncertainty. That's exactly what a method that we've developed here at Themis does. So we view learning as an evidence-based process. So if you remember from earlier when we were training the ensemble and calling multiple ensembles on the same input, we received multiple predictions and we calculated that variance. Now the way we frame evidential learning is what if we assume that those data points, those predictions, were actually drawn from a distribution themselves? If we could estimate the parameters of this higher order evidential distribution, we would be able to learn this variance or this measure of epistemic uncertainty automatically without doing any sampling or generative modeling. And that's exactly what evidential uncertainty does. So now that we have many methods in our toolbox for estimating epistemic uncertainty, let's go back to our real world example. Let's say, again, the input is the same as before. It's a RGB image of some scene in a city. And the output again is a pixel level mask of what every pixel in this image belongs to, which class it belongs to. Which parts of the dataset would you expect to have high epistemic uncertainty? In this example, take a look at the output of the model itself. The model does mostly well on semantic segmentation. However, it gets the sidewalk wrong. It assigns some of the sidewalk to the road and other parts of the sidewalk are labeled incorrectly. And we can, using epistemic uncertainty, we can see why this is. The areas of the sidewalk that are discolored have high levels of epistemic uncertainty. Maybe this is because the model has never seen an example of the sidewalk that are discolored have high levels of epistemic uncertainty. Maybe this is because the model has never seen an example of a sidewalk with multiple different colors in it before, or maybe it hasn't been trained on examples with sidewalks generally. Either way, epistemic uncertainty has isolated this specific area of the image as an area of high uncertainty.

Recap of challenges (46:44)

So today we've gone through two major challenges for robust deep learning. We've talked about bias, which is what happens when models are skewed by sensitive feature inputs, and uncertainty, which is when we can measure a level of confidence of a certain model. Now we'll talk about how Themis uses these concepts to build products that transform models to make them more risk-aware, and how we're changing the AI landscape in terms of safe and trustworthy AI.

How Themis AI is transforming risk-awareness of AI (47:14)

So, at Themis, we believe that uncertainty and bias mitigation unlock a host of new solutions to solving these problems with safe and responsible AI. We can use bias and uncertainty to mitigate risk in every part of the AI life cycle. Let's start with labeling data. Today we talked about aleatoric uncertainty, which is a method to detect mislabeled samples, to highlight label noise, and to generally maybe tell labelers to re-label images or samples that they've gotten that may be wrong. In the second part of this cycle, we have analyzing the data. Before a model is even trained on any data, we can analyze the bias that is present in this dataset, and tell the creators whether or not they should add more samples, which demographics, which areas of the dataset are underrepresented in the current dataset before we even train a model on them. Then let's go to training the model. Once before we even train a model on them. And then let's go to training the model. Once we're actually training a model, if it's already been trained on a biased dataset, we can de-bias it adaptively during training using the methods that we talked about today. And afterwards, we can also verify or certify deployed machine learning models, making sure that models that are actually out there are as safe and unbiased as they claim they are. And the way we can do this is by leveraging epistemic uncertainty or bias in order to calculate the samples or data points that the model will do the worst on. The model has the most trouble learning or samples that are the most underrepresented in a model's data set. If we can test the model on these samples, specifically the hardest samples for the model, and it does well, then we know that the model has probably been trained in a fair and unbiased manner that mitigates uncertainty. And lastly, we can think about, we are developing a product at Themis called AI Guardian. And that's essentially a layer between the artificial intelligence algorithm and the user. And the way this works is this is the type of algorithm that, if you were driving an autonomous vehicle, would say, hey, the model doesn't actually know what is happening in the world around it right now. As the user, you should take control of this autonomous vehicle. And we can apply this to spheres outside autonomy as well. autonomous vehicle. And we can apply this to spheres outside autonomy as well. So you'll notice that I skipped one part of the cycle.

Capsa: Open-source risk-aware AI wrapper (49:30)

I skipped the part about building the model. And that's because today we're going to focus a little bit on Themis AI's product called CAPSA, which is a model agnostic framework for risk estimation. So CAPSA is an open source library. You all will actually use it in your lab today, that transforms models so that they are risk aware. So this is a typical training pipeline, you've seen this many times in the course by now. We have our data, we have the model, and it's fed into the training algorithm, and we get a trained model at the end that outputs a prediction for every input. But with CAPSA, what we can do is by adding a single line into any training workflow, we can turn this model into a risk-aware variant that essentially calculates biases, uncertainty, and label noise for you. Because today, as you've heard by now, there are so many methods of estimating uncertainty and bias, and sometimes certain methods are better than others. It's really hard to determine what kind of uncertainty you're trying to estimate and how to do so. So CAPSA takes care of this for you by inserting one line into your training workflow, you can achieve a risk-aware model that you can then further analyze. So this is the one line that I've been talking about. After you build your model, you can just create a wrapper or you can call a wrapper that CAPSA has an extensive library of, and then in addition to achieving prediction or receiving predictions from your model, you can also receive whatever bias or uncertainty metric that you're trying to estimate. And the way CAPSA works is it does this by wrapping models. For every uncertainty metric that we want to estimate, we can apply and create the minimal model modifications as necessary while preserving the initial architecture and predictive capabilities. In the case of aleatoric uncertainty, this could be adding a new layer. In the case of a variational autoencoder, this could be creating and training the decoder and calculating the reconstruction loss on the fly. And this is an example of CAPSA working on one of the datasets that we talked about today, which was the cubic dataset with added noise in it, and also another simple classification task. And the reason why I wanted to show this image is to show that using CAPSA, we can achieve all of these uncertainty estimates with very little additional added work.

Trustworthy Ai Implications

Unlocking the future of trustworthy AI (51:51)

So using all of the products that I just talked about today and using CAPSA, Themis is unlocking the key to deploy deep learning models safely across fields. We can now answer a lot of the questions that the headlines were raising earlier, which is, when should a human take control of an autonomous vehicle? What types of data are underrepresented in commercial autonomous driving pipelines? We now have educated answers to these questions due to products that Themis is developing. And in spheres such as medicine and healthcare, we can now answer questions such as, when is a model uncertain about a life-threatening diagnosis? When should this diagnosis be passed to a medical professional before this information is conveyed to a patient? Or what types of patients might drug discovery algorithms be biased against? And today, the application that you guys will focus on is on facial detection. You'll use CAPSA in today's lab to thoroughly analyze a common facial detection dataset that we've perturbed in some ways for you so that you can discover them on your own. And we highly encourage you to compete in the competition, which the details are described in the lab. But basically, it's about analyzing this dataset, creating risk-aware models that mitigate bias and uncertainty in the specific training pipeline. And so at Themis our goal is to design, advance, and deploy trustworthy AI across industries and around the world. We're passionate about scientific innovation, we release open-source tools like the ones you'll use today, and our products transform AI workflows and make artificial intelligence safer for everyone. We partner with industries around the globe, and we're hiring for the upcoming summer and for full-time roles. So if you're interested, please send an email to careers at or apply by submitting your resume to the Deep Learning resume drop, and we'll see those resumes and get back to you. Thank you.

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to Wisdom In a Nutshell.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.