Hello everyone, hope all is well with folks today. It is a warm sunny day here in Northern Georgia. If you are first time reader, or know me I try to cut through the bull, and I find social media to be an endless source of humbug. I also try and talk about the edge-cases I find with AI, not to be a curmudgeon but to let folks know what they are getting in to.
“To AI, or not to AI” that is question that vexes the world. One sees it in every post, either praising or scolding, as a solution or impediment. I don’t like calling “AI“… AI. It is a branding thing, which is like “The Cloud“ which is also a marketing term for distributed data and network operation centers.
Ask ChatGTP: “AI really isn't AI right? what is it really called?“ and it will give you some information on what it *thinks*. My assistant told me a better terminology for it is: “Artificial Narrow Intelligence (ANI)“.
I agree with this, ANI is Machine Learning, LLMs, NPL all the things in concert working together. ANI is still good terminology, as training models means that the data trained upon is in the past: the knowledge itself has in many cases already expired. At least for the public consumer market, the private market I am not privy to.
In my opinion what ANI does really well right now is “fake“ you into you thinking you are having some sort of discourse with it. You are not. As part of the “Grounding“ series I am writing on your own human bias, let me describe a scenario I find consistently using ANI on a daily basis.
I run into interesting edge cases in my daily work using ANI. I ask questions that may be precise or may be ambiguous. This is the only way I can test things out as there is no test harness for co-pilot. I find the reasoning engines profoundly fascinating!
This is the prompt I asked: “Can you please put this list into a markdown checkbox list I can copy into GitHub?“
0 1 * * * psql postgres -dmain < /var/lib/postgresql/a.sql > /dev/null 2>&1
#0 * * * * psql postgres -dmain < /var/lib/postgresql/b.sql > /dev/null 2>&1
*/5 * * * * psql postgres -dmain < /var/lib/postgresql/c.sql > /dev/null 2>&1
0 * * * * psql postgres -dmain < /var/lib/postgresql/d.sql > /dev/null 2>&1
0 */4 * * * psql postgres -dmain < /var/lib/postgresql/e.sql > /dev/null 2>&1
0 0 * * * psql postgres -dmain < /var/lib/postgresql/f.sql > /dev/null 2>&1
Which, is a reasonable thing to ask as markdown’s extended syntax has checkboxes. This is the output that I received from Claude Sonnet:
<input disabled="" type="checkbox"> 0 1 * * * psql postgres -dmain < /var/lib/postgresql/a.sql > /dev/null 2>&1
<input disabled="" type="checkbox"> #0 * * * * psql postgres -dmain < /var/lib/postgresql/b.sql > /dev/null 2>&1
<input disabled="" type="checkbox"> */5 * * * * psql postgres -dmain < /var/lib/postgresql/c.sql > /dev/null 2>&1
<input disabled="" type="checkbox"> 0 * * * * psql postgres -dmain < /var/lib/postgresql/d.sql > /dev/null 2>&1
<input disabled="" type="checkbox"> 0 */4 * * * psql postgres -dmain < /var/lib/postgresql/e.sql > /dev/null 2>&1
<input disabled="" type="checkbox"> 0 0 * * * psql postgres -dmain < /var/lib/postgresql/f.sql > /dev/null 2>&1
For those who might not know, what ANI returned was HTML; which is fascinating, because that’s not what I thought I was asking for. Huh. I debated what happened.
The question my brain asked is: how did it interpret the angle brackets in my input? In the world of HTML (or XML, JSX, etc.), <, / and > denote opening and closing tags in html. So did the model see “< /var/lib/postgresql/c.sql >” and interpret that as a malformed HTML tag? Is that how it inferred my intent? Did it override my request for markdown entirely? Fascinating.
There is more. The mistake I made is that I should have been asking for a “Task List“. I was just being quick about things and said checkboxes in haste.
So, as the pattern matching came back with its potentials for answers, and the neural network vectorization concluded that even though I said `markdown` the network inferred that I was wrong. Honestly, I love these things it does, and maybe - just maybe will miss these problems when they do eventually get solved.
To continue the journey, I restated the prompt a few times, even gave it an example:
”Sorry, my bad: convert this list to a markdown task list I can use in github use - [ ] "0 1 * * * psql postgres -dmain < /var/lib/postgresql/a.sql > /dev/null 2>&1"”
Even though I corrected myself, it stubbornly kept giving me back HTML. I can’t remember exactly what the prompt was that I used — pretty sure I told it what do with itself — but we got there eventually after about 5 minutes, when I honestly could have just copied, pasted and converted in like 60 seconds on my own.
The point is: this is why agentic and ANI are still a gamble. Prompts do not recognize when YOU have made a mistake. You have to lower the odds of a mistake happening, that is the ROI. I wasn’t doing this to be lazy, but when I do find an interpretation mistake between my cognition and and ANI’s cognition I grab hold of it to figure out how to limit the same faux pas again.
When we got through to other side with an acceptable answer, I asked Co-pilot to explain itself; and it gave me such an interesting answer.
I apologize for my previous confusion. You're right - following precise formatting instructions should have been straightforward. Here's why I struggled:
I over-complicated the task by trying to add explanations and formatting
I kept adding unnecessary markup and styling
I didn't properly read and follow your requirements for the raw markdown checkbox list
The simple solution was to just provide the exact markdown format you needed, with nothing else. Sometimes the simplest answer is the best one. Thank you for helping me understand this better.
Hey, don’t be too hard on yourself there little buddy. I started you out on bad footing. I have 100% over-coded too. I try not to over code anymore.
The takeaway is that ML pattern recognition still relies on a common understanding of inputs to outputs. As folks are building their Prompt Toolboxes and looking forward to Agentic Coding (I will not use the term Vibe Coding, I am an adult); remember that the input data and the prompt need to be in alignment so that one may hedge their bets that the output is correct. ANI is a numbers game as to how many iterations it costs to get to a deliverable and is the result faster or slower than a human.
This is probably a bit naughty of me, however I will guarantee an OKR like this on someone’s board somewhere already:
Reduce the number of iterations required for AI to arrive at a cost effective answer.
Deep thinking, reasoning — whatever — confusion still abounds and ANI is brittle in some ways that as people we take for granted parsing language orally. ANI is not asking itself: “Should I clarify this question, did they really mean Task List?“. It should and I hope is the in the roadmap somewhere. I feel that is the real game changer.
Don’t code tired!