Introduction
GPT-4 has been proven to be useful to answer questions. For domain specific questions, we need to incorporate domain knowledge to GPT-4 to make it smarter to answer domain specific questions. Today we launched Website GPT at GPTs Store, which provides an effective solution to provide domain knowledge to GPT-4. Imagine that you are a website owner and you plan to promote your website or serve your customers at GPTs Store. The uniqueness of your custom GPT is that it can leverage the data from your own website. Website GPT is designed to achieve this goal by providing the following features.
- You can build a knowledge retriever (flow 1 in the diagram) at denser.ai by specifying a website URL. All web pages under this website are used to the build the knowledge retriever.
- Once you have a website knowledge retriever (whether you build by yourself or by others), you can use this website retriever to ask GPT-4 any questions (flows 2 or 3). While flow 2 is to use your own retriever to answer questions, flow 3 is to use pre-built retrievers (maybe by others) to answer questions. GPT-4 answers these questions not only based on its global knowledge, but also on the the website knowledge obtained through a retriever. In doing so, GPT-4 can provide more accurate answers and also reduce the risk of hallucinations.
- You can launch a GPT (flow 4) at GPT Store and configure it to use the website retriever you built. In doing so, you can release a GPT which is customized on your website knowledge.
How to test Website GPT
Note that you need a ChatGPT plus account to conduct the tests outlined in the page, as GPT-4 model access requires a ChatGPT plus account. The four workflows as illustrated in the above diagram are described as follows.
1. Build Retriever
Go to here (you need to sign up for a free account if you do not already have one) and type in the website you would like to build a retriever on (for example, www.example.com) to the Website Url box and click Build Now.
A free tier user can build an index using up to 100 web pages under your specified website. The retriever build process takes a few minutes to complete. Once it id done, you will be directed to a chatbot window. Take a note of the chatbot ID (highlighted in blue box) which will be used to query this retriever at GPT Store. Visit Web Search and Chat Guide for additional details on how to build a chatbot.
2. Test Website GPT with the Retriever you built
You first need to build a retriever at Denser as described in workflow 1 Build Retriever. Once you build the chatbot, take a note of the Retriever_ID (that is the Chatbot ID at denser.ai). Go to Website GPT at GPT Store.
You can input a query by following the format of
Retriever_ID: Your_questionsThat is, you separate the Retriever_ID and your query with a colon. It does not matter if you add spaces before or after colon. As an example, you can try the following query to query Website GPT.
46711c45-8726-4e1e-8c33-c63de5b9b66d: what applications does denser ai support?The retriever 46711c45-8726-4e1e-8c33-c63de5b9b66d was built using denser.ai, and thus understands the content from denser.ai. With the relevant information, GPT-4 can generate accurate results for this question (see the following figure).
On the other hand, if you do not include a retriever, the GPT-4 cannot answer this question correctly due to the lack of knowledge from denser.ai website.
3. Test Website GPT with Pre-built Retrievers
Similar to test your built retriever, you can test any retrievers built by others. Here we list a few pre-built websites for test.
- https://en.wikipedia.org/: 6f5831a0-b949-4562-8ca6-159fe4d7c20e
- https://openai.com/: 5a3e28e4-3d75-475f-8adb-f9fd0badb211
- https://denser.ai/: 46711c45-8726-4e1e-8c33-c63de5b9b66d
For example, you can ask what is the sora using OpenAI website (retriever 5a3e28e4-3d75-475f-8adb-f9fd0badb211) with the following query:
5a3e28e4-3d75-475f-8adb-f9fd0badb211: what is the sora?GPT-4 with a retriever returns results on the recently released Sora model as shown below.
This is more useful than the GPT-4 model without a retriever, which is struggling understand the user intent.
The reason is explained as following. GPT-4 and all other LLMs have a cut-off date. Any news or events (for example, the release of Sora model) after the cut-off date was not included in the LLM knowledge. As a result, GPT-4 is struggling understand Sora is a LLM capable of generating videos. The LLM normally takes a long period of time to re-train, it is thus difficult to keep its knowledge up to date. That is where a retriever shines: It can be quickly refreshed to provide latest event information to help LLM derive up-to-date answers.
Here we list queries which you can try on those pre-built retrievers:
- https://en.wikipedia.org/: 6f5831a0-b949-4562-8ca6-159fe4d7c20e
- who sings does he love me with reba?
- where do the great lakes meet the ocean?
- when does the new my hero academia movie come out?
- https://openai.com/: 5a3e28e4-3d75-475f-8adb-f9fd0badb211
- what is sora?
- who are the founders of openai?
- what is the pricing for gpt-4?
- https://denser.ai/: 46711c45-8726-4e1e-8c33-c63de5b9b66d
- What use cases does Denser support?
- Is denserbot free?
4. How to launch your GPT with a retriever
Once you build a a retriever as described in workflow 1 is complete, take a note of the Retriever_ID (that is the Chatbot ID at denser.ai). You then follow the OpenAI Creating a GPT instructions to build your custom GPT at GPT Store. To set up the retriever at Configure Tab, you click create new action button (highlighted in blue color) as follows.
It would direct you to the action configuration page below.
You can leave None in Authentication field, add a postRequest with method of POST and Path of /engine/, and fill in https://denser.ai/privacy.html to the Privacy policy filed. An important step is to copy the following JSON to Schema filed. Note that the action defined by this JASON requires two inputs: input and chatbot_id. The input refers to the query input and the chatbot_id is the retriever id you build at denser.ai. Replace YOUR_RETRIEVER_ID with your real retriever ID, all queries issued to your custom GPT would invoke the an call to your custom retriever, and the retrieved results are used to help GPT-4 better understand and answer your questions :)
{
"openapi": "3.0.0",
"info": {
"title": "Denser Retriever",
"description": "Retrieve Website knowledge to help GPT-4 answer questions",
"version": "1.0.0"
},
"servers": [
{
"url": "https://denser.ai/",
"description": "Main API server"
}
],
"paths": {
"/engine/": {
"post": {
"operationId": "postRequest",
"summary": "Submit a query request to retrieve relevant information from your website",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"input": {
"type": "string"
}
"chatbot_id": {
"type": "string",
"default": YOUR_RETRIEVER_ID
}
},
"required": [
"input",
"chatbot_id"
]
}
}
}
},
"responses": {
"200": {
"description": "Successful response"
},
"default": {
"description": "Unexpected error"
}
}
}
}
}
}FAQs
Do we need ChatGPT plus account to do the tests?
- Yes. ChatGPT plus account is required as you need it to access GPT-4, according to ChatGPT Plus subscription.
Why do I see "[denser_ai__jit_plugin.postRequest*source)" at the end of the response from Website GPT?
- This is a known bug. This is generated by GPT-4 and we do not an effective solution to remove it. The hypothesis is that GPT-4 generates that due to the invocation of Denser Retriever API, although it does not provide much useful information.
For more specific inquiries, reach out to support@denser.ai.