duanesmallman

the natural end of unchecked mass surveillance technology is a police state that erodes the civil liberties of the public under the guise of safety. as a developer and steward of technology that works by actively identifying individuals as “threats”, what are you doing to ensure it can never be used in a manner that exploits the populace? under what conditions would you refuse to accommodate the requests of government and local law enforcement?

Does this answer your question fully?

No

We also don't provide this information directly to law enforcement - the information is provided to the institution's internal security teams, that acts as the "human in the loop" who analyzes the threat and makes a decision on further action

There's a couple things here:

• This sounds like you're saying that you don't pro-actively provide this information to law enforcement. What is your policy when they ask for it?
• By offloading stewardship of this information to "internal security teams", it sounds like you are washing your hands of responsibility with regard to what happens with the information afterwards. What are you doing to ensure that these teams don't exploit this information as well?

With regard to your responsibility to protect individuals and use this data in an ethical manner:

It's almost always the goal of tech startups to "scale", and long-term business decisions are often filtered through this lens, especially when investors are involved.

• When your investors ask you to pivot into areas that require sharing more of this data, training your AI on more than hands with weapons (say, on dark figures with large backpacks, or people wearing heavy coats), or going after lucrative government contracts that require watering down your privacy requirements, what will you say?
• If making this company profitable meant sharing more data with people in positions of power (bosses, law enforcement, the gov't), where do you draw the line at what you're personally willing to provide to them?

This whole thread should be at the top.

Because surveilling people is a way of wielding power over them.

Regardless of what a surveillance system tells you its purpose is, individuals and organizations that seek to hold power over people will always use it as a tool for achieving that goal.

The groups with the most power over us are the ones that govern us, and without any forces to check that power, they have shown over and over again that they are willing to trade our rights to privacy and dignity to further their agendas.

That's why anyone developing something that can be used to wield power over others, regardless of what their intentions are, must deeply consider how they will prevent it from becoming a tool of oppression.