Thanks for this, Sam. I know you've written extensively about AI detection tools in the past, so it's good to get your measured take on this. It's also interesting to read the responses across the Substack community to these latest developments, especially since many people are ESL or neurodivergent.
Like you, I use AI for research, structure and editing, but the ideas and thinking behind my pieces are all mine. Surely we should rely on the reader to judge the quality of the work. It's like the library telling you which books to read!
I can't help thinking that Substack has shot itself in the foot here. I can understand their motivation, but I think they are missing the point somewhat. Because in my experience, Substack is about three things: content, connection and community and with this bungled attempt at protecting the quality of the content, they risk damaging the connections and alienating the communities the platform is built upon.
Absolutely. It's business 101: listen to the customers' voice. Instead, they plumped for adding a feature that instantly makes everyone on the platform distrust each other. Go figure.
That's not a translation problem, it's that Pangram has no way to see origin, only the finished text. It can't tell if a sentence came from you writing it straight, translating it, or a model generating it start to finish, because none of that history reaches it. All it sees is the shape of the words after the fact, and plenty of genuinely human writing reads the same way to it.
Look up things like rule of three, was x but was y, and metaphor stacking. Ai heavily use these formulas to write so when you ask for it to translate into natural English it could be applying these formulas. I’m noticing this when I ask it to rewrite my pieces to read more like an article. There are some drastic structural changes that I go back and fix.
The larger danger may be what this does to the reader. Instead of asking, “Did this help me think, feel, or see something differently?” we start inspecting every sentence for evidence of guilt.
A probability score cannot tell us who supplied the judgment, experience, or conviction behind a piece. Once suspicion replaces reading, strong human writers will get caught alongside the slop.
Well said. I support people explaining their process, and I agree that's a positive outcome. If they'd stopped with that, it would have been a great first step.
But what Substack actually did is exactly as you described: they brought witch-hunts to Substack.
Thank you for devoting so much space to us non-native writers! English is my third language, and now I have two options: be judged for the imperfections in my English, or be judged for using AI to improve it. Checkmate!
Another thought on discrimination: it seems obvious that groups already subject to more scrutiny and discrimination — women, people of color, and others — will also be judged more harshly for using AI. We can already see this pattern emerging in research. Sadly, that is how these things usually work: the weakest and most marginalized people end up paying the highest price.
I disagree with you slightly about the statement, though. On its own, a “How I make this” statement might not be a bad idea. But placed where it is, right next to the detector’s verdict, it starts to look like an explanatory note from a schoolkid who failed to do their homework. That is why mine is aggressively sarcastic. Sorry not sorry, but I refuse to put myself in the position of someone pleading their innocence.
And of course, none of this is really about protecting us from slop. It is about looking good in investor decks. Which is very very sad.
You are so welcome. And thank you for your disagreement and perspective! Where do you think we could place this statement instead? As I genuinely think it is a good idea, but I am also conscious that I always speak from the epicentre of White, male privilege...
I would place it somewhere that makes it part of the author’s normal context, e.g. on their profile, on the publication’s About page, or as an optional “How I made this” note at the end of the post. Somewhere the writer chooses to put it. Right now it feels more like a box for the accused to explain themselves.
What you have suggested, I have done with all of my Substack articles at the end of every post. Even before the introduction of this AI detection tool, I thought it was important to be transparent with my readers.
I’ve never met a techbro that used the line “if you have nothing to hide, you should not fear this product” that I liked. Mostly because they make obscene amounts of money at the expense of people’s privacy.
I'm repurposing an answer I shared with another author (mostly out of laziness):
I did a random “test” with some of my pieces. The AI scan said almost all my poems with AI written with 0% human input. 😂 Then I did the same thing with some of my essays and it said only 10% was human in most cases 😂😂
The false positives aren’t evidence of AI authorship; it's just evidence that I have high standards. AI detectors measure surface artifacts: grammatical symmetry, thematic continuity, and statistically probable word pairings.
Like you said, when an AI “matches” my work, it’s performing a probability trick, concluding that after a word like frequency, the mathematically sensible companions are loss or signal. Not every piece I write is a grand slam, but my baseline craft is tight, evocative, and conceptually disciplined enough that machines mistake rigor for automation. And NO. I’m not going to contort my future work to dodge these alarms. I’m not going to lace my sentences with artificial “human irregularities” or run pre‑publication diagnostics to appease a detector. That’s not my job. My job is to write with the methodologies that built my voice in the first place. This is an arms race I don't care to engage in.
I started two podcasts to show folks that the author of my pieces and the guy talking on video are one in the same. I also started sharing essays with accompanying drafts belonging to my poetry pieces. The essays trace the process from first impulse to final version, accounting for the logic of what stayed. I did think of being transparent; I just wanted to share some knowledge.
Here's the thing at the end of the day, If people think I’m using AI to write my pieces, they can listen to my podcast, hear how I speak and realize I write the same way I speak. It's really not that hard. And if they still want to troll me, I politely suggest they can blow me...( a kiss before they say goodbye).
Sam, you honestly make this place a better place mate. I really don't have the energy to give you a full on Lee Shand reply on this subject anymore, but please know this place is better for having you.
Sam… as you may have witnessed, I went on a 14-notes tirade about this integration. I was still in light testing mode, and figured many notes would be scanned, so I did scan my own notes. All 100% human.
Then… I found a better statistic. And I also decided to add sharper language on the founder (I added disclaimer-> exception, since he says it shouldn’t be the sole arbiter but has no qualms outing journalists and authors with the tool on Twitter).
I made two light edits. A reader later told me its scanning as 100% AI written. It flipped 100% in the other direction from 2 edits.
There is a big difference between the confidently wrong, uninterrogated, boldly published AI slop, being done by the laziest thinkers on this planet, and some of the best uses of AI that I have seen on Substack, people that edit and review and, most importantly, fact check that idiot AI. Because at the end of the day, this is a tool that is tuned for emotional resonance, not epistemic authority, every single output must be fully reviewed and questioned and researched.
But the problem with Pangram in particular is even when it’s confidence is LOW it flags as AI - so it is itself, boldly wrong some of the time. If they are going to hand out a % AI Written, % AI Assisted, and % Human Written, it should all be paired with % Confidence.
I have been working in commercial ML/AI implementations since 2013, and I have seen some really incredible use cases, if Substack wanted to do something really amazing for humanity, they would enable a Trust Score and a Fact Checker, THAT is the use case that humanity needs, because like every other platform that has ZERO responsibility for the quality of content published, this one is prone to absolute bullshit, even when it is 100% human written.
%100 guaranteed, all that spewed directly from my human brain 🫠
It would be ok if used for the right reasons and responsibly. It’s not perfect I mean like everyone states you are asking an ai tool to check for ai generated material lol I mean…
Thank you for writing this. I put a few of my articles through the tool and some came back human and others AI. I do use AI with editing and counter arguments (you helped me hone this skill) so I’m not surprised the flag of AI is raised. Like you, the arguments and ideas are mine, with some help of a tool that is available. I’m sure some of the Grammarly edits or even edits from any spellcheck will flag as AI.
2 be more hooman is two misspell and uz bad grammer
Sam, this gives sharper language to a concern I recently tried to name in my own essay. The deepest failure here is a category error. Pangram can identify linguistic patterns. It cannot measure authorship, integrity, lived experience, or whether a writer takes responsibility for what bears his name. Yet the interface invites readers to turn a probability into a moral verdict.
I use AI openly in my own process as a research aid, editor, and thinking partner. But my work begins in recovery, ministry, prayer, study, conviction, and questions I am actually living. I verify, revise, reject, and accept responsibility for every published word. No percentage can describe that process.
And the irony is nearly perfect: AI is now being used as a purity test for human expression, while human readers outsource their discernment to the machine. If the goal is trust, machine-generated suspicion cannot create it. Honest disclosure, accountable authorship, and careful reading might.
Detecting plagiarism is fair game: copying someone's argument whole cloth violates the rules and institutions that have been built, for good reason, around authorship.
Detecting whether writing "sounds AI" is a different act. There's no offense to verify, only a judgment about what a human is supposed to sound like, and that falls hardest on people whose natural voice reads as unusual: non-native speakers, the neurodivergent, anyone formally trained into a stiffer register. That's no longer plagiarism detection. That's punishing someone for how they choose to write.
Pangram: 0% accuracy, 100% confidence.
Just like all AI tools…
Pangram could run for office someday
It certainly has the colour scheme...
Thanks for this, Sam. I know you've written extensively about AI detection tools in the past, so it's good to get your measured take on this. It's also interesting to read the responses across the Substack community to these latest developments, especially since many people are ESL or neurodivergent.
Like you, I use AI for research, structure and editing, but the ideas and thinking behind my pieces are all mine. Surely we should rely on the reader to judge the quality of the work. It's like the library telling you which books to read!
I can't help thinking that Substack has shot itself in the foot here. I can understand their motivation, but I think they are missing the point somewhat. Because in my experience, Substack is about three things: content, connection and community and with this bungled attempt at protecting the quality of the content, they risk damaging the connections and alienating the communities the platform is built upon.
Thanks Des! And honestly all they needed to do was ask a few actual writers and readers on this platform for their input.
Absolutely. It's business 101: listen to the customers' voice. Instead, they plumped for adding a feature that instantly makes everyone on the platform distrust each other. Go figure.
💯💯💯
I had the same concern.
My first language is Japanese, but I write my newsletter in English for an international audience.
I don’t want to rely on automatic translation, so I write in Japanese first and then use ChatGPT and DeepL to help translate it into natural English.
As a result, AI detectors often classify my writing as AI-generated.
But the ideas, analysis, and judgment are still mine. AI helps me express them in another language—it doesn’t replace my thinking.
Exactly this Taka! I also wish I could use AI effectively to improve my Japanese!
That's not a translation problem, it's that Pangram has no way to see origin, only the finished text. It can't tell if a sentence came from you writing it straight, translating it, or a model generating it start to finish, because none of that history reaches it. All it sees is the shape of the words after the fact, and plenty of genuinely human writing reads the same way to it.
Look up things like rule of three, was x but was y, and metaphor stacking. Ai heavily use these formulas to write so when you ask for it to translate into natural English it could be applying these formulas. I’m noticing this when I ask it to rewrite my pieces to read more like an article. There are some drastic structural changes that I go back and fix.
Taka, that really is an excellent point. This too is part of an important conversation - about all ESL writers.
The larger danger may be what this does to the reader. Instead of asking, “Did this help me think, feel, or see something differently?” we start inspecting every sentence for evidence of guilt.
A probability score cannot tell us who supplied the judgment, experience, or conviction behind a piece. Once suspicion replaces reading, strong human writers will get caught alongside the slop.
Exactly my worries as well Todd. 😢
Well said. I support people explaining their process, and I agree that's a positive outcome. If they'd stopped with that, it would have been a great first step.
But what Substack actually did is exactly as you described: they brought witch-hunts to Substack.
And as you so brilliantly pointed out in your article I suspect it is the law which will cause them to back peddle…
Thank you for devoting so much space to us non-native writers! English is my third language, and now I have two options: be judged for the imperfections in my English, or be judged for using AI to improve it. Checkmate!
Another thought on discrimination: it seems obvious that groups already subject to more scrutiny and discrimination — women, people of color, and others — will also be judged more harshly for using AI. We can already see this pattern emerging in research. Sadly, that is how these things usually work: the weakest and most marginalized people end up paying the highest price.
I disagree with you slightly about the statement, though. On its own, a “How I make this” statement might not be a bad idea. But placed where it is, right next to the detector’s verdict, it starts to look like an explanatory note from a schoolkid who failed to do their homework. That is why mine is aggressively sarcastic. Sorry not sorry, but I refuse to put myself in the position of someone pleading their innocence.
And of course, none of this is really about protecting us from slop. It is about looking good in investor decks. Which is very very sad.
You are so welcome. And thank you for your disagreement and perspective! Where do you think we could place this statement instead? As I genuinely think it is a good idea, but I am also conscious that I always speak from the epicentre of White, male privilege...
I would place it somewhere that makes it part of the author’s normal context, e.g. on their profile, on the publication’s About page, or as an optional “How I made this” note at the end of the post. Somewhere the writer chooses to put it. Right now it feels more like a box for the accused to explain themselves.
This is a great suggestion. I would find this really valuable as a writer and a reader.
What you have suggested, I have done with all of my Substack articles at the end of every post. Even before the introduction of this AI detection tool, I thought it was important to be transparent with my readers.
This is great to hear. Well done! 💪💪💪
You have articulated something that I realise I was feeling uncomfortable about Valentina. Thank you.
We're not even tracking the right operator.
I read for content. I don't care how it's produced.
Bring me good logic and argumentation — teach me — and it could have been written by an elephant for all I care.
Exactly the same! 💪💪💪
Amen!
Exactly!
I’ve never met a techbro that used the line “if you have nothing to hide, you should not fear this product” that I liked. Mostly because they make obscene amounts of money at the expense of people’s privacy.
I'm repurposing an answer I shared with another author (mostly out of laziness):
I did a random “test” with some of my pieces. The AI scan said almost all my poems with AI written with 0% human input. 😂 Then I did the same thing with some of my essays and it said only 10% was human in most cases 😂😂
The false positives aren’t evidence of AI authorship; it's just evidence that I have high standards. AI detectors measure surface artifacts: grammatical symmetry, thematic continuity, and statistically probable word pairings.
Like you said, when an AI “matches” my work, it’s performing a probability trick, concluding that after a word like frequency, the mathematically sensible companions are loss or signal. Not every piece I write is a grand slam, but my baseline craft is tight, evocative, and conceptually disciplined enough that machines mistake rigor for automation. And NO. I’m not going to contort my future work to dodge these alarms. I’m not going to lace my sentences with artificial “human irregularities” or run pre‑publication diagnostics to appease a detector. That’s not my job. My job is to write with the methodologies that built my voice in the first place. This is an arms race I don't care to engage in.
I started two podcasts to show folks that the author of my pieces and the guy talking on video are one in the same. I also started sharing essays with accompanying drafts belonging to my poetry pieces. The essays trace the process from first impulse to final version, accounting for the logic of what stayed. I did think of being transparent; I just wanted to share some knowledge.
Here's the thing at the end of the day, If people think I’m using AI to write my pieces, they can listen to my podcast, hear how I speak and realize I write the same way I speak. It's really not that hard. And if they still want to troll me, I politely suggest they can blow me...( a kiss before they say goodbye).
I love that you are creating podcasts but hope that you (and others) weren't 'forced' into it to prove your humanity. 😢
It ended up serving a dual utility. It gave me a chance to share my thoughts and for people to see I'm real. But the main driver wasn't proving it :)
Sam, you honestly make this place a better place mate. I really don't have the energy to give you a full on Lee Shand reply on this subject anymore, but please know this place is better for having you.
Thanks so much Lee. I really appreciate that pal. 🙏
Sam… as you may have witnessed, I went on a 14-notes tirade about this integration. I was still in light testing mode, and figured many notes would be scanned, so I did scan my own notes. All 100% human.
Then… I found a better statistic. And I also decided to add sharper language on the founder (I added disclaimer-> exception, since he says it shouldn’t be the sole arbiter but has no qualms outing journalists and authors with the tool on Twitter).
I made two light edits. A reader later told me its scanning as 100% AI written. It flipped 100% in the other direction from 2 edits.
Make it make sense!
This is absolute insanity! And thank you for all the digging and work you have been doing here JHong!
There is a big difference between the confidently wrong, uninterrogated, boldly published AI slop, being done by the laziest thinkers on this planet, and some of the best uses of AI that I have seen on Substack, people that edit and review and, most importantly, fact check that idiot AI. Because at the end of the day, this is a tool that is tuned for emotional resonance, not epistemic authority, every single output must be fully reviewed and questioned and researched.
But the problem with Pangram in particular is even when it’s confidence is LOW it flags as AI - so it is itself, boldly wrong some of the time. If they are going to hand out a % AI Written, % AI Assisted, and % Human Written, it should all be paired with % Confidence.
I have been working in commercial ML/AI implementations since 2013, and I have seen some really incredible use cases, if Substack wanted to do something really amazing for humanity, they would enable a Trust Score and a Fact Checker, THAT is the use case that humanity needs, because like every other platform that has ZERO responsibility for the quality of content published, this one is prone to absolute bullshit, even when it is 100% human written.
%100 guaranteed, all that spewed directly from my human brain 🫠
Love this! And also you absolutely highlight how they seem to have picked one of the worst AI tools in terms of false confidence and fabrications. 🫠🫠🫠
Just another tool to bully or falsely accuse I can so this happening lord lol
It's not good is it Diane. 😢
It would be ok if used for the right reasons and responsibly. It’s not perfect I mean like everyone states you are asking an ai tool to check for ai generated material lol I mean…
Thank you for writing this. I put a few of my articles through the tool and some came back human and others AI. I do use AI with editing and counter arguments (you helped me hone this skill) so I’m not surprised the flag of AI is raised. Like you, the arguments and ideas are mine, with some help of a tool that is available. I’m sure some of the Grammarly edits or even edits from any spellcheck will flag as AI.
2 be more hooman is two misspell and uz bad grammer
Thank you Billy! And exactly. Also if we all wanted 💯'human scores' we could just used a humaniser without any engagement or thought. Go figure. 😬
Right. So use an AI tool to trick the other AI tool that I am a human. Not what I’m imagining Substack had in mind here! 😂
Exactly! And yet… here we are. 🫠🫠🫠
Very funny!
Sam, this gives sharper language to a concern I recently tried to name in my own essay. The deepest failure here is a category error. Pangram can identify linguistic patterns. It cannot measure authorship, integrity, lived experience, or whether a writer takes responsibility for what bears his name. Yet the interface invites readers to turn a probability into a moral verdict.
I use AI openly in my own process as a research aid, editor, and thinking partner. But my work begins in recovery, ministry, prayer, study, conviction, and questions I am actually living. I verify, revise, reject, and accept responsibility for every published word. No percentage can describe that process.
And the irony is nearly perfect: AI is now being used as a purity test for human expression, while human readers outsource their discernment to the machine. If the goal is trust, machine-generated suspicion cannot create it. Honest disclosure, accountable authorship, and careful reading might.
Thank you for your perspective and insight.
Thank you Kyle. It is great to hear about your own process as well. Ans it sounds completely driven by human judgement. 💪
The first post I wrote and put through their tool scored 70% human. I rewrote it substantially. The score dropped to 62%.
My human editing scores lower than the Claude project I use for editing (and then re-edit).
Thanks, Brian. This says it all, really, doesn't it...
Detecting plagiarism is fair game: copying someone's argument whole cloth violates the rules and institutions that have been built, for good reason, around authorship.
Detecting whether writing "sounds AI" is a different act. There's no offense to verify, only a judgment about what a human is supposed to sound like, and that falls hardest on people whose natural voice reads as unusual: non-native speakers, the neurodivergent, anyone formally trained into a stiffer register. That's no longer plagiarism detection. That's punishing someone for how they choose to write.
Great distinction Leif. And even the plagiarism tools we have had for decades now is not really fully fit for purpose.