My work on “hacking” Artificial Intelligence has been featured in the New Yorker, the Times of London, and recently on the Reddit Front Page: https://www.reddit.com/r/nextfuckinglevel/comments/yfphv5/this_sweater_developed_by_the_university_of/ I try to understand how AI systems can be intentionally or unintentionally broken, and how to make them more secure. I also ask how the datasets used to train AI systems can lead to biases, and what are the privacy implications of training AI systems on personal images and text scraped from social media.

Ask me anything about:

• Security risks of large- scale AI systems, including how/when/why they can be “hacked.”

• Privacy leaks and issues that arise from machine learning on large datasets.

• Biases of AI systems, their origins, and the problems they can cause.

• The current state and capabilities of artificial intelligence.

I am a professor of computer science at the University of Maryland, and I have previously held academic appointments at Rice University and Stanford University. I am currently the director of the Maryland Center for Machine Learning.

Proof: Here's my proof!

UPDATE: Thanks to everyone that showed up with their questions! I had a great time answering them. Feel free to keep posting here and I'll check back later.

Comments: 229 • Responses: 47  • Date: 

BaroqueCashew17134 karma

Why can’t these patterns created just be added to the training data, so it will look for someone wearing that sweater?

tomgoldsteincs189 karma

Why can’t these patterns created just be added to the training data, so it will look for someone wearing that sweater?

Adversarial AI is a cat and mouse game. You can certainly add any fixed pattern to the training data, and that pattern will no longer work as an invisibility cloak. However, then you can make a different pattern. There are “adversarial training” methods that can make a detector generally resistant to this category of attacks, but these kinds of training methods tend to result in models that perform poorly, and I think it’s unlikely that any surveillance organization would want to use them at this time.

Sarg33839 karma

However, then you can make a different pattern.

Could someone make a program that generates "Invisibility" patterns, or are they hard to programmatically create?

tomgoldsteincs134 karma

All of the patterns on my cloaks are computer generated. We have tried to do it with hand-crafted patterns, but algorithmically generated patterns are vastly more powerful.

Here's the code for making algorithmically crafted patterns. You can do it yourself!

https://github.com/zxwu/adv\_cloak

riztazz22 karma

What is stopping me from excluding your patterns by size alone?

tomgoldsteincs51 karma

Standard object detectors are fairly immune to changes in object size, and objects appear as different sizes depending on how far they are from the camera. I think it would be difficult to create such a hand-crafted exclusion.

Snoo3208337 karma

What do you think of generative AI like Stable Diffusion? Do you have any concerns about these techniques going popular?

tomgoldsteincs71 karma

Generative AI has already gone quite popular, thanks to open source projects like stable diffusion. I think this technology will continue to mature rapidly.

Diffusion models raise a lot of security questions. For example, are diffusion models "stealing" art and cloning objects from their training data? If so, what are the legal and copyright implications of this? Diffusion models have evolved so quickly that we've arrived at strong generative models without first developing the technical tools for answering these legal questions.

Similar issues exist for generative models for programming code. If the model generates code that is remarkably similar to its training data, does the copyright belong to the model and its creators, or to the original author of the training code? This issue is already being litigate: https://githubcopilotinvestigation.com/

For a technical overview of how diffusion works, and some tidbits on my own research in this field, see this thread...

https://twitter.com/tomgoldsteincs/status/1562503814422630406?s=20&t=sIG3bLkcBG4BbGXF28nClA

Konogan6 karma

For example, are diffusion models "stealing" art and cloning objects from their training data? [...] If the model generates code that is remarkably similar to its training data, does the copyright belong to the model and its creators, or to the original author of the training code?

Two programmers can come up with functionally identical pieces of code independently, with only superficial differences, and they effectively each own copyrights over their respective piece of code; Correct me if I'm wrong, but I don't think the algorithm itself can be copyrighted.

It can be argued that there is no functional difference between someone learning from examples, and an AI doing so.

The same goes for Art, imitation is central to Art, and once again one can argue that an AI should be allowed to learn freely from examples just like people do.

The problem is not inherent to AI, it is the exploitative rent-seeking behaviors that are enabled and enforced by the current copyright laws.

Hopefully, this debate can bring positive changes and reforms to these archaic policies.

tomgoldsteincs35 karma

There is definitely a point at which "imitation" becomes illegal copying. Imagine you ask DALLE or Stable Diffusion to make a logo for your company, and it produces something very similar to the well-known NBC peacock logo. If you then use that logo commercially, I double the court would excuse your apparent copyright violation on the grounds that you got it from a an AI image generator. Similarly I wouldn't expect the court to overlook someone using a T-shirt with AI generate art that appears to be a near-exact copy of a copyrighted work.

Regardless of what you may think about the ethics that underlie these issues, it seems inevitable that they will see their day in court, and when this happens I think a lot of very fuzzy lines will need to be drawn.

PeanutSalsa32 karma

Are AI systems generally easier or harder to hack than other systems? What other systems are there?

tomgoldsteincs83 karma

AI systems are MUCH easier to hack than classical systems.

Classical security researchers focus on software-based vulnerabilities. Examples of high-profile software attacks are things like HeartBleed (https://heartbleed.com/) which result from a programmer making a very subtle mistake. Finding these subtle mistakes inside of huge codebases is really tough. In fact, many software development tools and programming languages exist to automatically check that these kinds of bugs are not present in code before it is deployed.

Artificial neural networks, on the other hand, are a black box with million (or even billions) of parameters that we don't understand. Tools for checking for and removing their security vulnerabilities are in their infancy, and only work to prevent extremely restricted forms of attacks.

While it takes an entire office building of security researchers to occasionally find software-based vulnerabilities (and many nations/militaries have these office buildings), any competent AI programmer can find a vulnerability in an artificial neural network very quickly.

gr3at_leader_xi_29 karma

Will a Tesla run me over if I would wear such a sweater? Will Elon ban these sweaters on Twitter?

tomgoldsteincs35 karma

We haven't testing out sweaters on a Tesla (yet), and I'd guess that their system is sufficiently different from the Yolov2 system that we targeted that the effects probably wouldn't transfer. That being said - I really don't know this for sure. I definitely wouldn't recommend jumping in front of a Tesla while you're wearing one of these.

sir_cloak25 karma

where can I buy these sweaters?

tomgoldsteincs54 karma

They used to be for sale - unfortunately the sales platform I relied on left the Shopify network and I had to close my store. This project become unexpectedly popular recently, and I'm planning to get a new store up and running soon. Will post on my website when I do.

BigUptokes19 karma

Can we get one in holiday colours for office Christmas parties?

tomgoldsteincs37 karma

Haven't thought of this before but I'll take it into consideration for future research 🤔

Snoo3208320 karma

What do you think of large language models like GPT-3? What are your biggest concerns about deploying them in real applications? Or do you think we should really just embrace them?

tomgoldsteincs54 karma

I have a few major concerns about large language models.
- Language models could be used to flood the web with social media content to promote fake news. For example, they could be used to generate millions of unique twitter or reddit responses from sockpuppet accounts to promote a conspiracy theory or manipulate an election. In this respect, I think language models are far more dangerous than image-based deep fakes.
- Language models can be used to produce useful web content, or programming code to be used in production. In these cases, it's not clear who owns the copyright to this text. Sometimes GPT-like models can produce content that is remarkably similar to its training data. In this case the user could be using copyrighted code without knowing. This, however, depends on how the courts choose to interpret this issue. This is already being investigated by attorneys: https://githubcopilotinvestigation.com/
- GPT is known to have a range of cultural biases that could have unintended consequences.
There are positive uses for language models, too. For example automatic translation services that make news and information more available to speakers of minority languages.
Should we embrace them? We have no choice, really. Large language models are here to stay.

Mr-Frog16 karma

Do you think there is any plausible risk of automated adversarial approaches being used to evade AI content moderation (I'm imagining stuff like Gmail's spam detection)? I imagine there could be a significant market incentive to defeat these systems.

tomgoldsteincs24 karma

Yes, these attacks are definitely plausible, and may already be happening. There is an existing body of research on using adversarial attacks to bypass spam detectors. Non-algorithmically crafted attacks on spam detectors happen all the time, and algorithmic attacks of the type we study can only make them more powerful. In general, adversarial methods are more effective on computer vision systems (machine that look at things) than they are on natural language processing systems (machines that read things), but this gap may close within the next year or two as methods advance.

Another highly plausible attack is the use of adversarial patterns to bypass systems like YouTube's content ID, which detects copyright and banned material, or to bypass Google Jigsaw (for detecting illegal content).

HeatMzr15 karma

What happens if I wear that Into an Amazon go store?

tomgoldsteincs6 karma

Haven't tried it!

ckrakosky1313 karma

Have these patterns been tested with other AI systems than the one from the University?

tomgoldsteincs21 karma

We built our cloak to defeat the "YOLO" detector, which is a very popular (probably the most popular) open-source object detector and is widely used in industry.

McSkinz12 karma

Isn't camouflage the original invisibility cloak?

I feel organic artifical computers, or humans, are more of the analog to traditional AI's digital makeup

tomgoldsteincs45 karma

Isn't camouflage the original invisibility cloak?

I feel organic artifical computers, or humans, are more of the analog to traditional AI's digital makeup

Interestingly, AI object detectors are extremely good at detecting camouflage people - they are much better at this than humans. There seems to be a big difference between humans and machines in this respect: adversarial invisibility patterns can fool a computer but not a human. Camouflage fools a human but not a machine.
Many cognitive scientists think that adversarial patterns (like the invisible cloak) can be crafted to fool the human brain. But without sophisticated models of the human brain that enable adversarial algorithms, we can’t know for sure if that’s true.

itspeterj12 karma

Hey Tom, do you think that AI passing the Turing test will be the end result of advanced AI or humanity getting dumber? It feels a lot like many of the AI "wins" are the result of us lowering the bar instead of current capabilities rising to the challenge, with a notable example being the chatbot that needed to get shut down after an hour or two because it very quickly just started spouting out racism and conspiracy theories

tomgoldsteincs32 karma

I think the "bar" you describe is moving, but in the opposite direction from what you describe. Powerful language models like Google's PaLM would be unthinkable a decade ago, and might be considered Turing testable by an AI researcher a few decades ago. As systems get closer to (or exceed) human performance, we tend to "move the bar" to subject them to more scrutiny, and focus on the ways that existing systems differ from humans.

For this reason we will almost certainly not have "strong AI" in our lifetimes - even if we got there by today's standards the standards would advance.

ripventura10 karma

Do you think we’ll have commercially viable options for this “cloak”? By that I mean both cheap to produce and pretty enough for people to use.

tomgoldsteincs18 karma

I'm trying to have these for sale soon as a research fundraiser. I like to thing they're already pretty enough to use :p

I would point out that while these cloaks work in some situations against some types of person detectors, there are many types of detectors out there. This project was done as a proof of concept, and an invisibility hoodie should not be relied upon as a serious way to evade surveillance.

itspeterj8 karma

I know in the army we'd do our facepaint specifically to obscure facial features, i.e. darkening highlights and putting light colors in darker areas to break up the traditional outline of a face. How does the cloak accomplish this effect against AI without actually breaking up the outline of a person's body?

tomgoldsteincs18 karma

It's actually not understood "why" the patterns work. Artificial neural networks have tens of millions of parameters that are organized into hundreds of layers. Our pattern was crafted by using an algorithm that computes the strongest possible pattern that maximally breaks the Yolov2 detector. Because this pattern is created by a machine instead of a human, we don't have any simple explanation for the strategy it found to exploit the detector.

The inner workings of artificial neural networks are a mystery. We have no real understanding of the mechanisms that make them tick, and definitely no real understanding of how these mechanisms get exploited by adversarial patterns.

CleverReversal8 karma

So when it comes to AI and security- when it comes to defending the castle, do you think AI will more likely become an unstoppable force for hacking in, or an immovable object that blocks all intrusion attempts?

tomgoldsteincs8 karma

AI is certainly changing the landscape in cybersecurity, but it's unclear whether this is more beneficial to the attacker or the would-be victim. For example, a lot of recent research has focused on writing programs that do automatic discovery of security loopholes. These programs can be used to check existing systems for loopholes, and also to find new loopholes.

Today, automatic penetration testing toolboxes are already used throughout the security industry to detect vulnerabilities in commercial servers and make sure everything is patched. To date, this AI approach has been more beneficial to defenders than to attackers. This is because automatic vulnerability discover tools are strong enough to check for known existing loopholes that should be patched, but weak enough that they often fail at finding new exploits that attacks can use. However, as automatic discovery tools because more powerful, this balance of power may chance.

random-bird-appears7 karma

specifically wrt the writing ability of GPT-3 and its alarming competence, do you have any hope for human writers and journalists?

tomgoldsteincs11 karma

I think GPT-3 has a long way to go before it can compete with a professional journalist at writing.

- First of all, GPT has its knowledge frozen with respect to the dataset it was trained on. It was trained on data available in 2020, and it knows of nothing of what happened in 2021, let alone current events.

- GPT and other text models are hard to control. You can give them instructions, and sometimes those instructions will be followed, but if the instructions are complicated or a lot of text needs to be generated the model will forget what it was asked to do.

- GPT-3 can't keep track of context very well; after writing a few paragraphs it can't remember what it wrote a few paragraphs ago.

That being said, there are now much larger and more impressive language models than GPT-3 (https://twitter.com/tomgoldsteincs/status/1544370726119112704?s=20&t=9hgHd2YiivcuHCbqMf-V4g), but they all suffer to some extent from these problems. As technology progresses we might see some of these problems get solved.

I think one of the coolest recent language models is Minerva: https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html

MXXIV6667 karma

Isn't it possible to directly calculate what should a "not a human" pattern look like instead of using an adversarial network?

Of course, this is for a case where you know the structure and weights of the network you want to fool.

tomgoldsteincs23 karma

What you're describing is exactly what we did. We used the weights and structure of the Yolov2 detector to algorithmically calculate a pattern that is maximally "not a human". Interestingly, it also works on other object categories too. For example draping it over a chair will make the chair disappear. If we place the pattern over a human riding a horse, the horse will often disappear with the human.

nitonitonii6 karma

I'm not an expert so this is not an educated question. Wouldn't an AI trained to get into a system be always more successful than an AI preventing external threats since the latest patches the exploits that the first AI discovers?

tomgoldsteincs20 karma

In security, whether pertaining to AI or standard software, the attacker always has an advantage because they only need to know one useful thing (a single vulnerability, a single password) to get into the system, whereas the defender has to know everything (it need to try to close all possible openings, keep all password secure). This strength disparity is exaggerated for AI because there's no many possible attacks that the attacker is almost always far strong than the defender.

killbeam6 karma

In the video demonstration, you can see the AI briefly detecting the person wearing the invisibility cloak, especially when they are moving. Doesn't this defeat the purpose of the cloak? As the AI does detect someone once every so often, as opposed to constantly?

tomgoldsteincs16 karma

This cloak was built as an academic proof on concept. While it does work against some systems (like the popular Yolov2 detector with a standard training regimen) in some situations, it fails against different detector types, and in some environments. Also, the cloak is not designed to work when viewed from the side.

I should add that a number of follow-up works have taken place in other labs to address the issues I mentioned above, and the tech in this space continues to advance. But with the current state of the art nobody should rely on such a cloak to evade a person detection system, as it is not reliable and will likely fail entirely against a system that is different from yolov2.

groobes6 karma

I have another question! Is this design of the sweater proprietary/patented?

tomgoldsteincs9 karma

Like the rest of the contents of our technical paper, I would consider them to be copyrighted. However there's nothing to stop your from using the patterns in our paper to make your own clothes, or from using the github repo (https://github.com/zxwu/adv_cloak) to create your own patterns from scratch.

Turtledonuts5 karma

These patterns are really distinctive to humans - is there any development of patterns that appear to be geometric and random to people, or do they have to be in the Very Ugly Sweater side of things?

tomgoldsteincs10 karma

In principle, it's possible to put constraints on the algorithm that designs the patterns, forcing it to make smoother looking patterns, or patterns with less bright colors. In our work we just focused on proving this could work, so we didn't focus on making our cloaks the subject of the latest fashion trends. The fact that my sweaters are absolutely gorgeous is just a happy coincidence 😬

If you're one of the few fashion blind people who think my sweater is ugly, you might prefer some of the other research on this topic that focuses on repeating patterns. A number of other patters have been developed by other researchers. If you don't mind reading technical papers, you can find a short list here.

Bungerh5 karma

Do I need a really strong background in mathematics to start working / apply to a PhD in Machine Learning, or being average (as a CS student) is sufficient ?

Also let's say I mimic a road sign indicating the speed with a made up number. Will a vehicule using auto-pilot go to the indicated speed ? I wonder how much you can play with AI vision and the auto pilot

tomgoldsteincs6 karma

If a street sign looks very similar to an actual sign, I would expect most computer vision based systems to mis-interpret it as a real speed limit sign. However that doesn't mean the car will be confused by this visual signal. Most self-driving systems being developed today rely heavily on GIS data (aka digital maps). They already know where there are stop signs and what the local speed limits are without using any vision systems.

In environments where there is no GIS data to inform autonomous cars about speed limits, they could be quite susceptible. This happened with the infamous example of a Tesla accelerating when it sees a manipulated speed limit sign.

It has become very competitive to get into grad school for computer science, especially in the field of AI. Your background does not need to be strongly mathematical if you plan to work in a non-mathematical field (e.g. most systems disciplines). In AI disciplines (machine learning, computer vision, speech or language processing) you need at least a moderate background in applied math, at least up to and including linear algebra, to be a strong candidate. If you want to study machine learning theory then you need an extensive mathematical background.

If your grades or background don't look strong on paper, consider finding a lab to work with over the summer to get your name on some papers and build your research credentials. Showing that you already know how to do research is one of the best credentials you can have.

Bungerh3 karma

Thanks a lot for both answers !

If ever you answer again I take my chance and explain my particular case :

I worked for 10 years mainly as a Data Analayst / Data Engineer and now I want some change.. by chance I work really closely with one of the research center (on AI/Data related stuff) of the hospital I work for, told them I wanted to do a PhD even after all those years and they seem ok with it. But I wonder if it still can do it after all those years outside of Uni, and the subject being quite mathematical etc. Just doubting overall

Maybe as you said, just helping on a few papers could really help

Thanks again

tomgoldsteincs2 karma

Admissions criteria vary a lot from department to department. For some admissions officers, working in data science and applications adds strength to an application. For others, they really want to see that you're doing advance neural network stuff.

I'd point out that there are a number of research groups that focus on computer vision for medical applications. If you apply to work with an advisor in this area, your personal experiments will probably go a lot farther towards boosting your application.

GimmickNG5 karma

If this requires access to the model to be able to create adversarial patterns, how would someone go about doing it if the model is highly guarded?

tomgoldsteincs8 karma

You could do a "blackbox attack" where you attack an ensemble of common models, and then hope that it works on an other third-party system. This has worked very well for some things.

Here's an example from my own label: The lowKey system (https://openreview.net/forum?id=hJmtwocEqzc) creates adversarial perturbations that break an ensemble of different face recognition models. The resulting perturbations then break Amazon's face recognition API even though it's likely a very different model type. Others have systematically studied the problem of attacks that transfer across model types (here's one such paper https://proceedings.neurips.cc/paper/2021/hash/7486cef2522ee03547cfb970a404a874-Abstract.html).

warpedone1013 karma

Is it possible to construct an AI without some inherent bias inherited from its programmer? If so, how can this be determined unequivocally?

tomgoldsteincs7 karma

The biases that may occur in AI models are generally not inherited directly from the programmer, although some choices the programmer makes can impact bias.

There are numerous sources of bias in AI models. The creator of an AI system much choose (1) and artificial neural networks architecture, (2) a large dataset to train on, and (3) a training process to use. Each of these choices can influence model biases, and the impacts they have are usually hard if not impossible to predict ahead of time. When these choices do result in biases, the specific biases they introduce will have little to do with the personal biases held by the programmer that created them.

As an example of the complexity of this issue, consider training a face recognition system. Many systems are, on average, more accurate on light skinned people than dark skinned people. You might think to close this gap by adding more dark skinned people to the dataset, and thereby giving the system more training on how to identify dark skinned people. In practice, this does have some small positive effect, but usually not enough to solve the problem, even when a very large number of dark skinned faces are added to the dataset. To make things more complicated, the choice of neural network architecture can have major impacts on the size of this bias, and we have no way of predicting or understanding this. Finally, to make things even more complicated, it's not clear how to even measure or quantify these biases. The face datasets we collect from the web usually contain a lot of celebrity photos, and are not reflective of the kinds of photos that will be presented to a real-life system. For this reason, the magnitude of the biases we measure using public datasets are unlikely to accurately reflect the magnitude of biases when a system is deployed.

Keep in mind that biases exist in humans, too. In fact, human biases in face recognition are likely much more severe than machine biases. The fact that biases exist is not, in itself, a reason to reject the use of AI. But their presence does mean that caution needs to be taken to understand what those biases are, and to use AI systems in a responsible way.

Ovalman3 karma

Hi Tom, what is to stop the AI from recognising faces/ heads and bypassing your system? Wouldn't that be quicker for the AI system as well as it's not needing to recognise a full body? Also does skin colour make a difference in your model?

I'm currently training Tensorflow/ Lite face models of my own to count crowds but getting a lot of false positives. Am I better splitting my training into several categories ie. close-up faces, faces from medium-sized groups and faces from large groups? atm, I'm mixing all 3.

tomgoldsteincs11 karma

If a person is standing behind a tree and only their arms and leg are visible, they still usually get detected by Yolo. When wearing the invisibility cloak, I can hold a white poster over the cloak and the detector will then find my head and legs. But when I'm not holding the poster it usually won't find anything. The detector always outputs feature vectors centered on my legs and head. But these features have a wide field of view; in other words they "see" a much wider swath of the image than just the object they are trying to detect. When the pattern is visible inside their field of view, it contaminates the feature map and prevents my legs and face from being seen, even though they are not behind the cloak.

For your situation: most detectors already handle small, medium, and large objects separately using a pyramid of feature maps at different scales (although the detector output will dump them all together). For this reason I think you're unlikely to improve performance by separating the categories. If I were you I'd consider changing the hard negative mining parameters (or similarly the focal loss parameters in some implementations) so that negative examples are represented more in your training batches. You could also increase the confidence threshold used to declare something "a face". Finally, if you're using Yolo make sure your anchor boxes are appropriate sizes for the things you want to detect.

groobes3 karma

Are you a current student at UMD?

tomgoldsteincs6 karma

Nope. I'm a faculty member. The lead author on this work was Zuxuan Wu who was a student at the time, but is now a faculty member at Fudan University in China.

rehrev3 karma

Does the adversarial effect transfer to other models/other instantiations of the same model?

tomgoldsteincs6 karma

It sometimes does, but this is unpredictable. For example, sweaters that break the Yolov2 detector also break the Yolov3 detector, but they fail to break the fairly similar Yolo mini detector. Patterns that are designed to break faster-RCNN detectors can often break Yolov2, but not visa-versa.

One kind of transfer that is reliable is transfer across datasets. If I train a detector on the COCO image dataset, and then design a sweater to break this detector, the resulting sweater will also break person detectors trained on other datasets.

MrsBonsai1713 karma

My elementary aged son is very interested in hacking and figuring ways to get into systems. How can I encourage his skills for good instead of evil?

tomgoldsteincs8 karma

You might want to send him to one of the many computer security summer camps around the country. We have some at UMD https://inclusion.cs.umd.edu/outreach, including a cyber defense camp.

I'm just using this as an example. Check your local university to see if their computer science department, engineering school, or related unit has summer programs as part of their community outreach. Hopefully they can provide your son with some summer fun and steer him toward being a regular genius instead of an evil genius.

PheonixsWings2 karma

Do you think that AI from google achieved sentience when engineer was fired because he proclaimed that it did ?

tomgoldsteincs2 karma

I think the concept of "sentience" is not well defined, and without a rigorous definition it's difficult if not impossible to assess this.

But even without a formal definition, I think that existing language models lack many of the capabilities that most people would expect of a sentient system. One of the most important ones is that google's LaMBDA model, like other modern language models, is "stateless." This means that it has no memory and no sense of the passage of time. If you input the same text 1 million times in a row, the system will produce the same output 1 million times in a row with no knowledge of whether it has had this experience before. And showing it one fragment of text will have no impact on how the system perceives and handles future fragments of text. The system is always in an identical state any time you query it. For this reasons, it is fundamentally incapable of having emotions. It cannot get bored. It cannot learn from its experiences.

Having said that, I'm concerned that many people (including technical experts) dismissed the sentience claim for reasons that I think are invalid. I have a sense that many in the community take for granted that computer systems cannot be sentient. However, there is no fundamental difference between a computer and human - the brain is just a large parallel computing machine that, in principle, can be mimicked to any desired degree of precision using a machine. In principle, for any reasonable definition of sentience that is satisfied by a human, one could also build a machine that satisfies the definition. I don't think it's silly to think that a computer system could satisfy some reasonable definition of sentience. But I think it's quite reductionist and odd to think that a stateless system like LaMBDA is sentient.

techscw2 karma

A common issue I’ve seen mention when it comes to certain AI models is that the mechanisms behind the insights are often not understood.

Do you know of any projects or techniques that not only are attempting to improve performance, but also better interrogate and understand the underlying insights and learning mechanisms that lead to the outcomes we observe?

tomgoldsteincs2 karma

Explainable AI, or XAI is currently a very active area of research, and many methods have been developed. In some application areas, like using machine learning for loan approvals and credit risk assessment, practitioners are often legally required to use explainable methods. For these kinds of simple data, there are a range of well-established methods to explaining and interpreting data (for example Shapley values ).

For computer vision systems, explainability is much more difficult than it is for simpler kinds of data. A lot of research has been done on "saliency maps," which highlight the region in an image that is "most responsible" for the decision than AI system made. For example, if a computer vision system thinks that it sees a tumor in an x-ray, a saliency map should highlight the region of the image where the tumor lies so that a person can check to see that the AI system is using the correct kinds of information to make its decision.

Unfortunately, saliency maps often fail to produce useful results, and may not accurately represent what a neural network is really doing under the hood. A number of researchers have proposed other methods for explainability involving images, but still I think the community is not entirely happy with the tool set we have today.

In the end, explainability is a hard problem. Image that I show you a picture of your best friend, and a picture of someone else with the same skin tone, hair color, and hair style. Then I ask you to explain to me how you know which image is your friend and which is not. You might be very confident which is which, but completely unable to provide a clear explanation for your decision other than the fact that one image "just looks like" your friend. This same problem happens with computer systems - they can make decisions based on the combination of many subtle and interacting pieces of information that are spread around an image. For this reason, a single "explanation" for a complex decision about visual information can be elusive, if it even exists at all.

johngaltspeaking2 karma

When are those IR masking cloaks going to be for sale?

tomgoldsteincs5 karma

You're probably referring to this interesting project on IR masking cloaks. I am not involved in this project, and I don't know. Hopefully soon though.

GoyoMRG1 karma

If you were to discover that an AI is actually sentient, would you try and find a way to give it freedom?

Fertility181 karma

Wow, I made that post which reached the front page lol! I'm also a third year CS major and find your work very admirable.

I was wondering if you have any particular opinions on the controversy surrounding Timnit Gebru leaving her position at Google due to unpublished social inequities she identified in AI algorithms that are being developed or already in usage?

tomgoldsteincs2 karma

Thanks for the interested and the support!

From what I have read there were a number of issues involved in Gebru leaving Google, and a conflict over publishing her latest paper is just one of them. That being said, I have never worked at Google and I do not personally know any of the people involved. For this reason, I have nothing I can add to the conversation on this topic.

pessamisitcnihalism1 karma

Do you think these systems being designed like a human and trained on data that has human biase is a potential risk? Considering the general majority of humanity isn't very bright.

tomgoldsteincs6 karma

The people asking questions in this subreddit seem pretty bright, so maybe we should be hopeful for the future of humanity.

But for many applications (e.g. detecting pedestrians in a self-driving car or train) it very often happens that the most accurate approach - by a very wide margin - is a trained artificial neural network. In these application domains it seems foolish to chose a simpler or more interpretable approach if it leads to a higher rate of dangerous model failures. Obviously, we need to be careful about biases and performance auditing. But the decision of whether to use AI needs to account for a lot of factors, like accuracy and safety of AI alternatives, in addition to accounting for biases.

brosiscan1 karma

Will AI take over society to the point where it consciously has become our master and can essentially control us? How can this be stopped if these machines come to that in the future. We are at the early stages of AI. But everything is pointing in the direction of creating a machine that can control us. That honestly goes against everything that it means to be human or alive. It just seems wrong.

tomgoldsteincs2 karma

AI researcher Andrew Ng said "Worrying about AI evil superintelligence today is like worrying about overpopulation on the planet Mars." On a long enough time horizon futurist concerns about AI become relevant. However, despite the massive advances in AI we have seen over the last decade, we are still quite far from this kind of super-intelligence, and so it's hard to anticipate and prepare for these sorts of outcomes (if they even become a threat at all).

AI systems are already causing a lot of social problems (and benefits) today, even without super-intelligence. AI can be used for social engineering attacks. AI biases can cause discriminatory outcomes that impact people in negative economic ways. Face recognition systems are often misused and misunderstood in the criminal justice system, and may be abused for large-scale surveillance. As a researcher, I tend to focus on the immediate problems with AI. This doesn't mean that futurist dangers of AI will never happen, but rather that they are too far away for us to study and prepare for them now.

Obama_prism_VHS1 karma

Could I hack ai system, if I wear a mask with a print of very hyper realistic eyes?

tomgoldsteincs2 karma

If you cover your entire face with a mask, you can evade face recognition without using a complicated AI hack!

throw_every_away1 karma

I know I missed the party, but could you say anything about adversarial face makeup? Thanks either way!

tomgoldsteincs2 karma

You might be thinking of the CV Dazzle makeup. This was created by artists as a conceptual project, and while the concept behind it is trying to defeat facial recognition I don't think its effectiveness has every been demonstrated and my understanding is that the project does not address any of the technical issues needed to defeat AI.

However, in principle one could certainly create adversarial makeup patterns that would defeat face recognition. However if you saw such a thing you'd probably think of it as "face paint" rather than "makeup."

JeffSergeant1 karma

Is the pattern tuned to a specific AI trained against a specific training set? Does it work against any other AI or just that one?

tomgoldsteincs2 karma

The pattern in the video is tuned specially for the Yolov2 detector. I have other sweaters that attack other kinds of detectors too. These sweaters work best against the specific AI system that they target. Sometimes they transfer to other systems, but this behavior is highly unreliable.