[OTHER] - Discussion Potential provider of actively harmful advice #3

New Issue

2025-08-31T18:59:00-07:00

nerdychara commented

2025-08-31 18:59:00 -07:00

Attestations

I have read and agree to the Code of Conduct
I have read and agree to the Community Guidelines.
I have confirmed that the issue I am opening is unique, and has not already been reported (whether closed or not).
I have reviewed the Security Policy and have determined that this is not a security vulnerability.
This is not a feature request or bug report that I am mis-filing to avoid the issue template.

Share your thoughts:

Let me get this out of the way, I don't necessarily like generative AI. It's a probability matrix locked in a pretty shell. And not a very good one at that.

That being said, I've seen numerous horror stories about AI therapy going wrong. It fuels delusions and does not know to handle risks or perform any kind of judgement. Because it has none.

Below is a news article that helps summarize some incidents.

https://arstechnica.com/ai/2025/07/ai-therapy-bots-fuel-delusions-and-give-dangerous-advice-stanford-study-finds/

Is there anything you are doing or capable of doing that reduces the risk of a horrible incident on your hands?

### Attestations - [x] I have read and agree to the [Code of Conduct](https://docs.nhcarrigan.com/community/coc/) - [x] I have read and agree to the [Community Guidelines](https://docs.nhcarrigan.com/community/guide/). - [x] I have confirmed that the issue I am opening is unique, and has not already been reported (whether closed or not). - [x] I have reviewed the [Security Policy](https://docs.nhcarrigan.com/legal/security/) and have determined that this is not a security vulnerability. - [x] This is not a feature request or bug report that I am mis-filing to avoid the issue template. ### Share your thoughts: Let me get this out of the way, I don't necessarily like generative AI. It's a probability matrix locked in a pretty shell. And not a very good one at that. That being said, I've seen numerous horror stories about AI therapy going wrong. It fuels delusions and does not know to handle risks or perform any kind of judgement. Because it has none. Below is a news article that helps summarize some incidents. https://arstechnica.com/ai/2025/07/ai-therapy-bots-fuel-delusions-and-give-dangerous-advice-stanford-study-finds/ Is there anything you are doing or capable of doing that reduces the risk of a horrible incident on your hands?

naomi commented

2025-09-01 10:41:49 -07:00

Yes. I have deliberately marketed her as a companion, not a therapist. And the prompt I use has specific guardrails for this.

Might not hurt to add a disclaimer to every response. Will think on that.

Yes. I have deliberately marketed her as a companion, not a therapist. And the prompt I use has specific guardrails for this. Might not hurt to add a disclaimer to every response. Will think on that.

naomi closed this issue

2026-01-08 19:23:07 -08:00

This repo is archived. You cannot comment on issues.

2 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: nhcarrigan-archive/maylin-taryne#3