OverflowAI Search is now available for alpha testing (September 13, 2023)

You folks seem to be searching desperately for a way to use a chatbot. What users need is a search that actually finds what they are looking for.

The first step of your chatbot enhanced search is to make the search function return relevant results.

If the search would actually return relevant results, then the chatbot "enhancement" isn't needed. Questions and answers with relevant information are more useful than a chatbot generated summary of the same questions and answers.

Fix the real problem (poor search) instead of tossing some useless chatbot "enhancement" into the mix.

All we need (and want) is for the search to actually work.

Joke:

Q: How do you find information on Stackoverflow using the built-in Stack site search?

A: Ignore it and use google with "site:https://stackoverflow.com".

Seriously:

Fix the lousy site search feature. That's it. That's all.

This feature is in violation of the share by license. It is easily correctable.

There is no attribution to code that is provided. It is broadly "sourced" in a hidden section for sources. There needs to be citation for use.

These are verbatim copies of the sourced answers.

Hidden and overly broad sources do not count for actual attribution when it is verbatim like this.

Each one of the "copy" sections needs to have a citation source in there. It can be small, out of the way, or even just a [1] which references the answer it came from.

Solution: AI Summarization for Quick Insights

What's the license of the machine-generated summaries?

• I don't think "Number of searches performed" is a particularly great way to measure success. What if I try to do more searches because the search results are bad?

• First, we want to provide more transparency within the product development process

  Great!

• The second knob is the Question selection strategy. Once we have a number of questions that fit your query, we need to decide on the parameters we use to select amongst those questions. For example, we might prioritize questions that are non-deleted, non-closed, and have an accepted answer.

  Hold on a sec. Why would you ever show content from a deleted question? Why would you even consider such content as a candidate for a search result?

• The third knob is the Answer selection strategy. This involves deciding how we should select answers from the set of questions. Should we consider them per-question or as a whole?

  What do "per-question" and "as a whole" mean?

• For instance, we might prioritize accepted answers first, then by highest score, etc.

  Going just by score is probably better on average than prioritizing the accepted answer first. Prioritizing the accepted answer is letting a single person determine what is considered best rather than letting the masses speak with their votes. See also the earlier feedback you got (including the comments there)

• The fifth knob is Prompt engineering. Once we have the answers, we need to decide how we communicate to the LLM how it should summarize those answers into one concise response, possibly with examples.

  I thought the whole point of this was "trust"? Prashanth keeps talking about trust in his blogposts and in the WAD fireside video and OverflowAI announcement video, and linking that trust to being able to cite to answers. And I thought that the generative part of AI was what undermined trust?

  • Why risk generating nonsense examples that mislead readers and erode that trust?
  • How will you know whether it even makes sense to give examples for a question? Not every Q&A is the type that needs examples.
  • Who will the examples be attributed to, and where will they be shown in the UI? I hope you don't mislead readers into thinking that generated content was written by the people who wrote the posts you feed to the LLM.

• The final knob is Temperature. This refers to how random the LLM results should be. We generally operate on a very low temperature, but it could make sense to leverage the LLM more here.

  How is increasing the temperature "leveraging the LLM more"? I know people say that higher temperature can make ChatGPT more "creative", but what's to say that's a good thing here? You literally cite that a major pain point is having to search through multiple answers, but won't you be making that worse if you put readers in a situation where the random dice rolls of higher temperature make it so that you can get a potentially infinite number of answers by regenerating responses?

• Are you going to follow OpenAI's sharing and publication policy this time?

I'd hate to see randomness introduced into the search. A core feature of the public web is that content and pages can be linked; that includes search results. Sure, the results change over time as the underlying data changes (slowly), but when I link a search I would expect that multiple users visiting that link around the same time will see the same results. Similarly, making search conversational might break this, as the URL needs to carry the context with it.

Please add a permalink or "share" feature to this, which allows others to reproduce the search. This would also benefit discussions about how to improve the search.

As a user with a gold tag badge I am often using search to identify duplicates. I see questions I know have been answered numerous times before, but when looking through search results, I find answers that are all slightly off. They tackle the same basic problem, but zoom in on a different detail, are outdated, or explain the same principle as an answer to a different premise. I, as an expert in my field, can see the similarity, but the asker I am trying to point to a duplicate would surely be confused.

Currently, I often find I am faster just writing the same answer for what feels to be the tenth time, than digging through search results that all are just not good enough.

And that is what I will be looking for when taking part in this testing phase: do I mark questions as duplicates more often?

These points may have been already said in previous answers but I want to emphasize them:

1. I see great value in using semantic search and think it'll greatly enhance and potentially fix the historically broken SO search so thanks for this!

2. I'm somewhat skeptical of the AI helping in refining the search query but I think it's worth a try.

3. I'm extremely skeptical of the AI summarization part and still think you haven't addressed the elephant in the room regarding its validity, intrinsic LLM hallucinations.

   I also learned a lot and gained a lot of value as a beginner developer a few years back (and still definitely do) from reading the answers in their entirety myself and even the discussions in the comments of answers and questions. Consequently, I really think abstracting this away is a very bad idea.

4. Since you are going to do the summarization anyway, please don't prioritize accepted answers. They are the opinion of a single user at a single point of time and in many cases an inexperienced user. Instead prioritize highly upvoted and trending (frequently upvoted recently) answers.

There's plenty of info here (and everywhere this is talked about) how it will supposedly help people looking for an answer, but how will it help people looking for questions? Will it get out of my way when I'm looking for a specific well known question rather than trying to spoon-feed me a generated answer I'm not looking for?

I don't typically use SO's search when I'm looking for an answer because SO isn't the only source of answers. There's an entire internet out there with other places answers may exist that this tool can't access. Git repositories with issues to search through. Official docs with documented methods/params/returns and examples. I'd be quite frustrated if new devs only used SO's search to find solutions given how much existing information it can't access.

I recently saw the alert inviting me to try the new search experience, and immediately opted out of doing so. For what it's worth, my reasoning for doing so was this:

As a user, I don't care about the mechanism underlying the search functionality I'm using. No other site I use makes user-facing boasts about their search tech, I don't really want to have to know or think about it – I just want a search functionality that works in the conventional, as-expected manner: searching content and presenting it exactly as it exists, surfacing me the content most relevant to my query without editorialization.

The idea of a search function that sounds to me like may not return the same results every time I use it, or might synthesize content in a way that changes the author's intent? That's deeply off-putting, and frankly just not something I want to be bothered spending even one second learning or adjusting to. The popup for this "new experience" feels like a big neon sign hung up by the marketing department that profoundly misses the point of what users expect from a well-established UI norm like a search box, which is quiet, familiar simplicity and predictability – and for me, it's one more reason to just keep using Google as the de facto Stack Overflow search functionality.

I have to say that the language you're using in the popup, "New search experience powered by AI" and "leveraging AI to summarize", is offputting and undercuts confidence that stackoverflow knows what it's doing. (Independently of whether using LLM chatbots or what have you will improve search experience, or whether bot summaries cite code properly, etc.) This comes from being keyed in to the current LLM hype cycle, how it's being promoted as "AI", and what that implies vs. what the actual capabilities are.

You're a technical knowledge site whose answers are written by your users, many of whom are sophisticated enough to know that "powered by AI" can mean "we've integrated a LLM somewhere in our site", or "we've integrated what superficially seems like machine intelligence somewhere in our site, but isn't quite there yet so uses human assistance behind the scenes", or even "we want our new feature to catch the eye of gullible investors, so we'll call it ‘AI’". If your users think less of you for the way you're promoting this new search feature, it's going to undercut what gets your users to answer questions here or trust answers as authoritative.

You should be very careful & specific with how you describe these features if you want to maintain trust. As of right now, with the "OverflowAI" messaging, you are not. This comes off as trying to sell snake oil to investors by jamming LLMs into your product. Like other users have mentioned, SO’s search function is not great right now; this messaging seems like waving AI at it in the hopes that'll fix it. It's not an effective way to approach the problem. At this point, I don't trust the company's judgement at all, and I doubt I'm the only one.

There seems to be a way to opt in to seeing the alpha, but as the feature goes wider, an option for users to opt out of both:

• ever being shown potentially incorrect summaries powered by guessing engines of any kind (LLMs or otherwise)
• their content being used by summarization features, to avoid potentially incorrect summaries of their content being served to novice users, with the original poster's name on it (as per 'CC BY-SA')

would be ideal.

An example that springs to mind of the danger here is an answer like this:

Here is how to do X; {SOME COMMENTED CODE THAT CORRECTLY HANDLES BAD INPUTS}.

But doing X is bad practice because security/performance/etc, ideally you would do Y instead. For more on Y, see [this other question].

-UserName

being summarised to:

to do X; {SHORTENED UNSAFE CODE} -UserName

The summarized version is shorter and direct so it could improve the number-of-searches and copy-button-clicks metrics, but perpetuates bad practice, hides useful information, and makes UserName appear reckless.

At present the alternative available to users who strongly object to guessed summarization is to delete content, and given according to the quote included above "we might prioritize questions that are non-deleted", deleted content is only lower ranked and not excluded, some users might be motivated to edit their content to be nonsensical before leaving the site. This could add moderation tasks, cause confusion for novice users, and so damage the StackExchange brand.

Is there a way to permanently dismiss the banner asking me to test that feature? It reappears every time I log in (i.e., every day), even though I closed it several times via the "X" button.

I gave it a couple of tries so far.

Can we get a free-response box for feedback even if I say "yes" it was helpful? In one response there were three short sections, the last of which was garbage but the first two appeared reasonable.

The Featured on Meta panel is currently showing me two entries with the term "AI":

1. This one which explains an alpha release of "AI" powered search.
2. Temporary policy: Generative AI (e.g., ChatGPT) is banned which contains:

   All use of generative AI (e.g., ChatGPT1 and other LLMs) is banned when posting content on Stack Overflow.

If "AI" is considered an unreliable source of posted content, why is considered useful to provide summarised search results?

The total number of upvotes is sometimes incorrect, e.g.:

while the search on SO is keyword based IMO and returns exact matches in most cases (which can be different from what user is actually trying to search),

can you guys consider the search feature of popular software /docs website where they use https://www.algolia.com/ and shows as (powered by algolia). it is very effective and finds alternate similar words too, and accounts for typos. in some cases it perform better than Google or bing as the content is very site specific.

let me know what you guys think.