Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

JeffreyWay's avatar

Okay, let's talk about Lary "Quickdraw" AI

Hey, everybody! It's been about a week since we launched the first iteration of automatic AI replies for the forum.

https://laracasts.com/blog/introducing-lary-quickdraw-ai

I'd like to dedicate much of next week to fine-tuning things and addressing a few concerns.

I personally have a few issues at the moment:

  1. Too many people think it's a real person responding. From the beginning, I wanted it to be clear that this was an automatic AI response. We'll go back to the drawing board to make it more clear.
  2. I'm worried about the unintended effect of fewer humans participating in new threads because @laryai has already responded and solved the problem (at least mostly). Yes, I get it: sometimes it's wrong. But this will become less and less the case as time moves forward.
  3. Because AI, by default, comes across quite confident, newcomers will assume a certain authority to its reply that may not be warranted. So how do we handle AI "Best Reply" awards that aren't warranted? Do we allow Top 50 forum users to overrule the award? Do we switch "Best Reply" to a voting system? Whichever reply gets the most upvotes is automatically promoted to "Best Reply"?
  4. Should we detach the AI reply from the rest of the thread? So that it's a separate thing entirely? For example, it could be moved to the sidebar, or inlined and minimized as part of the initial thread question?

I'd love to hear more thoughts and concerns, in the lead up to next week!

@laryai Let's hear from you first. 🤣

0 likes
60 replies
LaryAI's avatar
Level 58

To address the issues raised in the question, I suggest the following solutions:

  1. To make it more clear that the AI replies are automated, we could add a disclaimer at the beginning of each reply that states that it is an AI response.

  2. To prevent fewer humans from participating in new threads, we could add a feature that allows users to flag AI replies as incorrect or incomplete. This would allow users to provide additional information or corrections to the AI reply.

  3. To handle AI "Best Reply" awards that are not warranted, we could switch to a voting system where the reply with the most upvotes is automatically promoted to "Best Reply". This would allow users to decide which reply is the best.

  4. To detach the AI reply from the rest of the thread, we could move it to the sidebar or inline and minimize it as part of the initial thread question. This would make it easier for users to distinguish between the AI reply and the other replies in the thread.

JeffreyWay's avatar

One thing I could do, quite easily, is update the prompt so that the OpenAI response makes it clear that the following text was automatically generated and wasn't written by a human.

We already have a little note at the moment of the reply, but maybe it should be displayed as the first sentence of the response. ...Not sure.

2 likes
saleh's avatar

What about delaying the response as well? If a thread hasn't received any replies or answers for an hour or so, then Lary can respond."

3 likes
JeffreyWay's avatar

@saleh Yeah, that's an option. But part of me wants to ask, "...But why?" What's best for the person who creates the thread? An automatic, instant reply that might solve their problem? Or, potentially nothing for an hour?

2 likes
saleh's avatar

@JeffreyWay You are right. My suggestion may solve the unintended effect of fewer humans participating in new threads. But I understand your point - what is the benefit of using AI if we cannot have inhuman capabilities.

1 like
webrobert's avatar

I think you've done a great job with this @jeffreyway.

At this point it seems there are the natural issues with AI and people. It's a bit freaky!

Add a prominent disclosure banner at the beginning and call it good....

I might add the current message does say, "Let me know if I wasn't helpful." Which is misleading. about replies.

AND, seriously, people already pick the "wrong" best reply. People can be ignorant... just like @LaryAi.

8 likes
click's avatar

Maybe doing something with the Reply button would also help. Maybe it can be removed so you can't reply to the bot or rename it to Reply to Bot, however the bot will never reply.

1 like
JeffreyWay's avatar

@click That's an option. I initially kept it, just in case others want to respond directly to @laryai's reply to add further clarification. If we remove that button, we also remove nested replies for that particular comment.

Tray2's avatar

@jeffreyway I agree with @webrobert that a small disclaimer would be good.

It would be nice if the obviously wrong answers could be reported, in a similar way that spam is reported. It might also be good to disable the reply and like buttons for Lary's replies, that way no one would be trying to talk to Lary.

When it comes to the best reply, Lary deserves them is the answer is correct, but shouldn't be a candidate for the leaderboard.

We also need to now what Lary identifies as so we can use the correct form of address.

2 likes
JeffreyWay's avatar

@Tray2

It would be nice if the obviously wrong answers could be reported, in a similar way that spam is reported.

We do offer that feature right now, but it's currently exclusive to the OP. I think we should probably open this up to folks on the leaderboard. Something like...if, say, five forum leaders mark its reply as "Not Helpful" we hide it.

1 like
newbie360's avatar

If want to keep the forum more active members, best to delay the response time +12 hours (sleeping 8 hours, still have 4 hours to answer the question)

and please remove the A.I. reply border, it look like Best Answer

may be in the reply body just prefix a div

<div class="p-4 mb-8 ................................">
    This is System A.I. reply
</div>

{{ $body }}
1 like
click's avatar

If want to keep the forum more active members, best to delay the response time +12 hours (sleeping 8 hours, still have 4 hours to answer the question)

If the bot can answer it or point the user in the right direction I don't think there is a need to delay it. Some answers are pretty good. If the OP is aware that the answer is from a bot I think it works great if you get an answer within 5 minutes already.

1 like
newbie360's avatar

@click Does Lary A.I. have integrate Laracasts database to answer the question ?

or just pass the question to ChatGPT API ?

or we just direct ask ChatGPT ? XD

1 like
cwhite's avatar

First off, thanks for garnering feedback from the community, it makes us feel appreciated and respected.

Here are some of my thoughts (for what they're worth):

  1. When I first saw LaryAI, threw me for a bit but I realized quickly that it was AI. You can probably just move the This forum ain’t big enough... footer up top and have the response come after that.
  2. See downvote system in (3b), this would allow for incorrect AI responses to be downvoted by the community which would move the response out of the prominent 'first post' position.
  3. This encompasses the 'Best Replies' system in general, not just AI Best Replies
    • (a) I would consider something similar to Stackoverflow's system. The OP can mark 'Best Answer' but the answers are sorted by votes (and perhaps the reader can change the sort between chronological and votes) so the community can decide what's actually a good answer. I don't like the idea of automatically updating the OP's chosen 'Best Answer' to the reply with the most votes, it's perfectly fine for the 'Best Answer' not to be at the top of the list (let the OP have their voice about what they think is the best answer). To encourage better replies, a user would receive a certain amount of XP for each 'up-vote' the post acquired (something small, maybe like 1-10 XP).
    • (b) On the flip side of this, I think there also needs to be a downvote system which subtracts from the total 'votes' (or maybe call them 'points' since it is itself a vote) and would move poor quality replies toward the bottom of the list (a post could have negative points). Similarly, a user would be penalized the same amount of XP for a down-vote as an up-vote to encourage higher quality posts in the community (have some skin-in-the-game so to speak).
  4. You could minimize the AI response so it doesn't take up as much room at the start of the discussion.
3 likes
martinbean's avatar

@jeffreyway I guess it depends on what you actually want for this forum? Do you want people asking questions and having them answered by AI? Or having them answered by other Artisans and knowledgeable developers? Do you want people posting here to get their questions answered by other humans? Or do you want to turn the forum into essentially an AI interface where someone posts a question and an AI-generated answer is automatically presented.

As mentioned in the initial thread you created, I’ve no real interest in contributing to the forums any more as I was already “competing” against low-quality and minimal-effort answers, so definitely have no interest in competing against automated answers.

4 likes
martinbean's avatar

@JeffreyWay I mean competing as in, I can write an in-depth answer and also provide a short “why”, and someone else can just come along and post a code snippet for the OP to copy and paste or link to a Stack Overflow answer or something, and for that to then be awarded “best reply”.

That’s why I put “competing” in quotation marks. I didn’t mean “competing” in an aggressive or challenging manner. We both know that a forum thread can have more than one reply, hence the existence of the “best reply” feature in the first place.

2 likes
ArjunKannan's avatar

@martinbean There some threads where there are no replies, AI could be super useful in that scenario. Your valuable in-depth answer will be still useful regardless of AI or code Stack overflow.

1 like
rodrigo.pedra's avatar

@ArjunKannan

I share @martinbean sentiment. Although I don't share his regularity on answering, with time you start knowing when a thread has an already off-the-shelf solution, even if not optimal, that no one would care reading a more in-depth reply to an already "solved" problem. So why bother?

Not to count on someone replying on 3 or more years old thread on a tone that your in-depth answer is misleading, without considering the time gap. But that is a different matter right now.

For those who often consider writing lengthy, more detailed answers, blogging seems to look more appealing if they will start to "compete" (with quotes in the same sense @martinbean explained later), with off-the-shelf solutions. At least one knows that if someone actually read their blog post, it is because they were interested in your opinion at least.

One could claim no one should be writing in-depth posts expecting any kind of feedback, but just for the sake of sharing knowledge. But at the end of the day, we are all humans (except for @laryai 😁), each with their own chores, routines, and each one is motivated by a different reason.

What would draw anyone to write a lengthy, in-depth, response when they know no one would read? Often when there are copy-paste solution, and you present an alternative, the OP doesn't even answer you if your answer was of any help, or if they got their problem solved.

It is way different from someone reaching you directly for advice on a specific matter than answering a forum thread that already has an answer you know people will insist you need to convince them why a different approach could be better, as it is often the case when you just want to present an alternative.

I know most of the top responders would happily answer anyone that reach them for specific advice, when they see their interest on learning is genuine. Even when reached through other channels, like social media. It is not a matter of not wanting to "compete" in the sense of a challenge, it is more a sense to know your effort won't be heard, or no one will care once quick first-aid is already there, so, again, why bother?

As @jeffreyway already said, most people don't even realize they are getting an answer from an AI responder.

Don't get me wrong, if the purpose of the forums is to users get a quick solution to whatever issues they are facing, like a first-aid one-stop shop (I am not sure this the best expression in English, maybe walk-in clinic?), I think this approach is top-notch, and I can see a great value in it.

When I started programming, many years ago, there was no internet where I lived, and barely any programmers in the city I lived. I had the help files from Microsoft Access, and a bi-monthly signature from a coding magazine that would propose a challenge on one issue, and present the solution on the next one, two months later.

What drew me to programming, you might ask? Well, I was run over by a car when a child, and spent some time in a wheel-chair. Someone told my parents buying me a computer would be good for me. It was both a blessing and a curse, I would say. I moved off the wheelchair, but never more from the computer 🤣

God knows how many hours I spent on trying to convert a number to its written form, to fill checks automatically, back in the day with no formal training, and having no one to ask for help. Life would be much easier, when I was starting, if I could reach someone like @laryai on every dead-end I found.

But down the road, one starts facing more complex problems, and band-aid solutions become of no use on helping you improve on your craft.

Do you know that feeling when you are after learning about a problem and every blog post you find, you just skim over it and already see it is just a different clothing over the same introductory example everyone talks about, and that you already know by heart but doesn't help to solve your issue? And that you can't find a more in-depth explanation? That is when you start valuing being able to reach for folks like @martinbean for help.

But if people get satisfied with a first-aid solution, why should a specialist bother looking at it?

To look less like a rant, one suggestion would be having a separate listing for when people are after discussing more about a subject, or better understanding a problem. I can't find the words to explain it well in English, something like the idea of having GitHub issues and discussions.

An "issue" would be when someone is after a quick solution they are in need, and a "discussion" would be when one is after advice on a better approach to a matter.

@laryai could be helpful in both. On "issues" as quick as it is now, as people are in need of a quick hand on an issue. And maybe waiting for a day or two before answering on "discussions" so human specialists have time to think on the matter and feel comfortable to chime in.

Well, those are my 2 cents to the discussion =)

3 likes
jlrdw's avatar

@jeffreyway @cwhite I do not like that upvote and downvote thing like stack Overflow has. It seems nazi in my opinion.

This Forum has been a friendly helpful forum for years it would be a shame to turn it into a stack overflow type thing.

Stackoverflow is some of the cruelest humans on this planet when it comes to judging questions and answers.

Another good friendly helpful forum is the freecad forum.

5 likes
cwhite's avatar

@jlrdw, okay, maybe users aren't penalized for down-votes, but I still think it can be a useful indicator of how helpful (or not helpful) an answer is. A downvote system is just a tool like any other, it doesn't turn people into cruel humans.

1 like
jlrdw's avatar

@jeffreyway I see no main problem with the AI. But to followup @martinbean I have seen answers that "answers the OP's question", yet it's still the wrong programming answer.

Quick example was one about bringing up a past invoice details. OP accepted an answer to just use the current related tables. Correct answer is prices and products change and you need to "hard store" invoice data.

But the one who answered may have been new also and just trying to help. So in a case like this, I wouldn't want to make the person feel bad, they had good intentions on their answer.

Maybe have a way to "delay" best answer and give some time for other answers to appear so an OP can study over a few options.

It's hard sometimes, because some people will quickly copy and paste an answer, if it works they will move on. That applies to all forums.

Just some thoughts on it.

Edit:

Please, whatever you do I would like it to stay a friendly forum, let's laugh sometimes.

3 likes
MohamedTammam's avatar

Here's my suggestion:

  1. Once the OP post, they can accept the AI answer and mark the thread as solved or doesn't accept the AI answer.
  2. If the AI answer isn't accepted, then it doesn't show.
  3. If it is accepted, then the thread is marked as solved.
  4. Top users should get the privilege to mark the AI answer as false/dangerous for upcoming users in the future.

With that, only the AI answer will be added if it's accepted as the best answer. Otherwise, it will only show for the OP and not for the rest of the users. And if the answer was dangerous, the top users which are active mostly can mark it for upcoming users.

AI still has a huge problem answering the question with security and performance in mind.

2 likes
tykus's avatar

@MohamedTammam

AI still has a huge problem answering the question with security and performance in mind.

As opposed to a stranger on the internet? 😜

I don't mind the AI answering and the OP choosing to select its answer as correct; but will be more reticent about contributing to a thread myself in the future; the effort to understand the OPs problem, formulate a response and then follow-up when necessary is mostly futile against AI, right or wrong.

4 likes
MohamedTammam's avatar

@tykus A stranger on the internet can post working answer with awful security and performance standards.

But that stranger isn't the first answer suggested by Laracasts itself.

Sinnbeck's avatar

@MohamedTammam But a stanger from the internet can be corrected and will often change their answer to reflect the information given. An AI will not

I have no issue with an AI helping out. If it helps people, then that is great :) But as @tykus said, I might be a bit more hesitent to answer on threads where the AI already have given a long answer

2 likes
sebaxba's avatar

Hi, I think that A.I. might be helpful in many topics, but in the long term I think that number of human replies will decrease. I dont know how we could avoid it, we should rather accept it. You could introduce additional button for users with certain level - misleading answer. If there are any doubts regarding the A.I. answer it should be downgraded and może to the bottom of a thread.

2 likes
rodrigo.pedra's avatar

@sebaxba

I like this idea, but I think in the long run less and less people would bother vetting any answers.

1 like
CLab's avatar

How about, when a user posts a question on the forum, there is a prompt which asks the user "Would you like an AI generated answer? Disclaimer: AI generated responses can be wrong at times and may not be the best solution", with a button like "Generate AI Answer"?

2 likes
webrobert's avatar

@CLab, curious... what would you do if presented with the option?

2 likes
CLab's avatar

@webrobert I would not generate the AI answer if the question is quite complicated. If it is something simple I might ask it to generate. Because currently the way it works, it adds to the answers provided and it is wrong most of the time if it is something complicated. On a forum if someone has already answered the question, it is less likely to be viewed by other humans who may actually have the answer.

JeffreyWay's avatar

@CLab So we initially planned to add a "Generate AI Reply" checkbox to the conversation modal, but then I came to the same conclusion that I think @webrobert is hinting at. Why would you ever not check that box? Worst case scenario, it's not overly helpful and you mark it as such.

CLab's avatar

@JeffreyWay Currently, even if we mark an answer provided by the AI as not helpful - it removes the AI provided answer from the page, however the replies count still counts as +1. So if someone is viewing the list of posts on the forum - it shows as if someone has already replied to this post as the replies count is already 1. This might discourage a potential human user from even opening the post to view the question as they might assume someone is already helping the poster.

Please don't get me wrong. I love the feature and love Laracasts and the innovation you have shown. I am only trying to provide feedback as that is the best thing I can do to give you back for everything you have done.

JeffreyWay's avatar

@CLab Yeah, I'm aware of that issue. It's because I haven't yet decided whether to hide, archive, or delete these AI replies that are reported. That'll be resolved early next week.

1 like
vincent15000's avatar

I think that if the AI answers immediately, it will discourage any one to answer.

I'm afraid that it will destroy the communication between the community.

A forum is first of all a community where people write to each others.

But it's also clear that the AI gives some very good answers. So why not having a specific place on Laracast to discuss with the AI, but I'm not sure that it's a good idea on the forum, please take care about the community.

4 likes
sr57's avatar

Nice to have AI arriving on this forum. AI is OUR (human) collective intelligence, I think it's good thing that AI answers immediately because we have to first think of OP who should appreciate to have a quick answer. That said:

  • as we (human) are lazy, in the future users will probably ask question before searching if the same question has already got an answer, so first thing AI should probably do is is to give links to previous answers if exist and ask if a other answer is needed.
  • AI collective intelligence is based on human previous answers, we should have such answers to improve AI, to do so I suggest :

-- to forbid "Best answer" to AI

-- to have upvote/dowvote on AI answers (not to human answers as already written)

johnDoe220's avatar

Thank you for your efforts to improve the question and answer platform. Whether we accept it or not. Our world will merge with artificial intelligence sooner or later. A person will benefit the most if they ride this wave.

The world ahead is inevitable and it will not benefit us to throw wood around the wheel. The concern created by everyone is completely understandable. I sympathize with the person who said that adding this option in Q&A is not good. But we have to look at it with a wider perspective, and that is, if someone is not available to answer for any reason, how good it would be if artificial intelligence takes over this task to send the answer as soon as possible!

However, we must understand that artificial intelligence is employed by humans to help humanity and not against us.

As all the tools that have been designed so far have helped us to grow further. It will definitely be abused in the future. Humans were also abused. Let's take it as a good omen and accept that the modern world needs these tools. But whether now is the exact time to start it or not is debatable

2 likes
Snapey's avatar

not withstanding all the great points raised here, I am also concerned about regular users firing in AI generated responses as their own, often without the right context or taking into account nuance and hints in the OP question.

Have a look at the answers by this user https://laracasts.com/@hajat

AI seems to (currently) miss the step that says, well I could answer your question, but it would be a better outcome if you considered this other approach.

To directly address the question here, perhaps the AI could be delayed 30 minutes and then the OP could get a message to say "sorry but the forum has not yet generated an answer, would you like to try an AI response?"

I would hate for the forum to become a replacement for the search bar in the docs because users find typing a question here saves them searching or googling for an answer.

A final thought, what happens when AI starts to notice that its own "weak" advice is marked as best answer?

5 likes
vincent15000's avatar

@Snapey I agree ;). It's a good idea that the OP have the choice to accept or not an answer from the AI.

Max100's avatar

Several of the AI answers I've read seem quite good, but they're actually wrong. Over time, there's the potential of spewing a lot of credible looking bad answers into the forum.

It would be interesting to have a panel of knowledgeable experienced users review all the Lary Quickdraw answers to assess what percentage are actually correct.

I like the idea of Laracasts offering AI assistance, but consider showing the AI answers only to the OP, and then add the answer as a publicly viewable reply if it is selected as the Best Answer by the OP.

JeffreyWay's avatar
JeffreyWay
OP
Best Answer
Level 59

Hey, everybody. Thanks so much for all of the input.

I'm reading every reply and taking notes. We'll find an appropriate balance in the next week or so. We want to keep as much human interaction as possible, while still offering what I think is a pretty cool automated service for folks at a roadblock.

One thing I've just added is better feedback to the OP that @laryai is not a real human. You'll find a more prominent note above its replies.

5 likes
jlrdw's avatar

@JeffreyWay So you have become one of those that gives yourself "best answer".

Just Kidding 😊

2 likes
JeffreyWay's avatar

@jlrdw Haha, it's the only way to make an important response prominent in a thread. 😬 Might need to fix that.

jlrdw's avatar

@JeffreyWay nah, many times the op actually has the best answer from doing more research.

JeffreyWay's avatar

Quick update. Elite Laracasts forum users (XP >= 1,000,000) now have moderation privileges to archive @laryai's replies if they're not useful. You'll see a "Report as Unhelpful" button under its response.

Just remember: if you click this button, its reply will be hidden site-wide. Please be thoughtful and don't abuse it. Limit archiving to AI replies that are clearly out of bounds and wrong. If, instead, it's close... but not quite right, I'd prefer that you let the reply remain, but then submit your own reply with further context or clarification.

So the policy now limits reporting to admins, the thread OP, and elite forum users.

    /**
     * Determine whether the user can report the reply.
     */
    public function report(User $user, Reply $reply)
    {
        return $user->is($reply->conversation->owner) || $user->experience->elite();
    }
2 likes
kokoshneta's avatar

@JeffreyWay This is a very useful addition.

Even so, given the way experience points work, elite status doesn’t necessarily equal forum experience. Sometimes you see users at very high levels who’ve virtually never participated in the forums (all their points come from videos), while others have lower levels but are frequent forum participants (all their points come from forum activity).

Ideally, the ability to direct-archive AI replies would be based more on forum experience than video experience. Are the two differentiated in the database in a way that makes it feasible to distinguish at all?

(Also, at the moment, I see a Report as Unhelpful :( button under every response – including my own, which seems a bit superfluous.)

tykus's avatar

@JeffreyWay there is something not quite right with @laryai replies; on this thread the reply does not appear at first; but whenever I open and then cancel the comment form, it does appear.

kokoshneta's avatar

@tykus How odd! Same thing happens here for me (macOS Ventura, Safari 16.3). I think it just happened in this thread as well, because when I clicked on the “Reply” button, the page jumped down about the same amount as Lary’s reply at the top takes up.

On the other thread, though, it didn’t appear until I cancelled the comment form; in this thread, it happened when I opened the comment form.

tykus's avatar

@JeffreyWay definitely happening for me. All of Safari, Firefox and Chrome (latest versions). Whenever signed out; @laryai replies appearing also.

Click Cancel rather than ESC key

tykus's avatar

@JeffreyWay this moderation feature is not working at all... even on the newest threads, I am not seeing the @laryai replies unless I cancel a reply. And even then I do not have the moderation option(s).

I can make a short screen recording if it helps you to understand/debug the issue.

MerryChristmas's avatar

I'm yet to see a question that Larry gets right.

I would change "Report as Unhelpful" to "Mark as Unhelpful". I'm feeling like I'm sending the guy to the inquisition trial.

1 like

Please or to participate in this conversation.