Custom App not working as expected.

[Arshil-Akkala]

I was trying to create a custom app, the app is getting created, but after giving & structuring all the input placeholders, I am not getting any result. Adding screenshot and code for reference.

default_prompt = """System: <|begin_of_text|><|start_header_id|>user<|end_header_id|>
Generate a SQL query to answer this question:
{input_question}

{instructions}

DDL statements:

<|eot_id|><|start_header_id|>assistant<|end_header_id|>

===Additional Context

{sample_rows}

Human: ### SQL
Given the database schema, here is the SQL query that answers {input_question}:

CUSTOM_LLMS = [
    LLM_MODEL_NAME
]

ag.init()
ag.config.default(
    prompt_template=ag.TextParam(default_prompt),
    temperature=ag.FloatParam(LLM_TEMPERATURE),
    model_name=ag.MultipleChoiceParam(LLM_MODEL_NAME, CUSTOM_LLMS),
    top_p=ag.FloatParam(LLM_TOP_P),
    top_k=ag.IntParam(0, 0, 10),
    repetition_penalty=ag.FloatParam(LLM_REPEATITION_PENALTY),
    max_tokens=ag.IntParam(LLM_MAX_TOKENS, -1, LLM_MAX_TOKENS),
    # task=ag.TextParam(LLM_TASK),
)


@ag.entrypoint
def generate(inputs: ag.DictInput = ag.DictInput(default_keys=["input_question", "instructions", "sample_rows"])) -> str:
    llm = SeldonCore(repo_id= ag.config.model_name, endpoint_url=MODEL_HOST,
            task=LLM_TASK,
            model_kwargs={
                    "temperature": ag.config.temperature,
                    "top_p": ag.config.top_p,
                    "top_k": ag.config.top_k,
                    "repetition_penalty": ag.config.repetition_penalty,
                    "max_new_tokens": ag.config.max_tokens
                }, verbose=False)
    
    final_prompt =ag.config.prompt_template.format(**inputs)
    prompt = PromptTemplate(
        input_variables=[], template=final_prompt
    )
    chain = LLMChain(llm=llm, prompt=prompt)
    # output = chain.run(inputs)
    output = chain.invoke(inputs)

    return output['text']
    =======================================
    
    Need to know where I am going wrong, if I am.

[Arshil-Akkala]

this is the response

[dosubot[bot]]

Hey @Arshil-Akkala, I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's tackle this together!

Your custom app might not be producing any results due to a few potential issues. Here are some steps to troubleshoot and resolve the problem:

1. Ensure .env File is Set Up:
   Make sure you have a .env file with your OpenAI API key:

   OPENAI_API_KEY=sk-xxxxxxx
   

   You can find your API keys at: https://platform.openai.com/account/api-keys

2. Initialize Agenta SDK:
   Ensure you have initialized the Agenta SDK in your application code:

   import agenta as ag
   ag.init()

3. Register Default Configuration:
   Verify that your default configuration is correctly registered:

   ag.config.default(
       prompt_template=ag.TextParam(default_prompt),
       temperature=ag.FloatParam(LLM_TEMPERATURE),
       model_name=ag.MultipleChoiceParam(LLM_MODEL_NAME, CUSTOM_LLMS),
       top_p=ag.FloatParam(LLM_TOP_P),
       top_k=ag.IntParam(0, 0, 10),
       repetition_penalty=ag.FloatParam(LLM_REPEATITION_PENALTY),
       max_tokens=ag.IntParam(LLM_MAX_TOKENS, -1, LLM_MAX_TOKENS),
   )

4. Entrypoint Decorator:
   Ensure the @ag.entrypoint decorator is used correctly:

   @ag.entrypoint
   def generate(inputs: ag.DictInput = ag.DictInput(default_keys=["input_question", "instructions", "sample_rows"])) -> str:
       llm = SeldonCore(repo_id= ag.config.model_name, endpoint_url=MODEL_HOST,
               task=LLM_TASK,
               model_kwargs={
                       "temperature": ag.config.temperature,
                       "top_p": ag.config.top_p,
                       "top_k": ag.config.top_k,
                       "repetition_penalty": ag.config.repetition_penalty,
                       "max_new_tokens": ag.config.max_tokens
                   }, verbose=False)
       
       final_prompt = ag.config.prompt_template.format(**inputs)
       prompt = PromptTemplate(
           input_variables=[], template=final_prompt
       )
       chain = LLMChain(llm=llm, prompt=prompt)
       output = chain.invoke(inputs)
   
       return output['text']

5. Serve the Application:
   Use the Agenta CLI to serve your application:

   agenta init
   agenta variant serve myapp.py

6. Check Environment Variables:
   Ensure your .env file is in the same folder as your application code and includes the necessary environment variables.

By following these steps, you should be able to resolve any issues with producing results ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[Arshil-Akkala]

I am already doing all these, still not able to get a response. Cannot determine any issue in the container logs as well.

[Arshil-Akkala]

2024-09-20 15:20:03 Using Agenta Python SDK version 0.24.4 2024-09-20 15:20:12 Closing trace None 2024-09-20 15:20:12 Cannot close trace, no trace to close 2024-09-20 15:20:12 INFO: 172.18.0.4:41654 - "POST /generate HTTP/1.1" 200 OK

These are the logs I see in the container.

[mmabrouk]

Hey @Arshil-Akkala

I have created a simplified version of your example. I basically removed all the LLM functionality and focused on the agenta part:

import agenta as ag


default_prompt = """xxx
"""



CUSTOM_LLMS = [
    "xxx"
]

ag.init()
ag.config.default(
    prompt_template=ag.TextParam(default_prompt),
    temperature=ag.FloatParam(1.),
    model_name=ag.MultipleChoiceParam("x", CUSTOM_LLMS),
    top_p=ag.FloatParam(1.),
    top_k=ag.IntParam(0, 0, 10),
    repetition_penalty=ag.FloatParam(1.),
    max_tokens=ag.IntParam(1000, -1, 1000),
    # task=ag.TextParam(LLM_TASK),
)


@ag.entrypoint
def generate(inputs: ag.DictInput = ag.DictInput(default_keys=["input_question", "instructions", "sample_rows"])) -> str:


    return "OUTPUT"

This worked fine in my setup (agenta --version->Agenta CLI version: 0.24.4) used with cloud.

Can you please try first doing the same. Returning a constant string instead of the LLM output to determine whether the interface created by agenta works correctly. If it does not, please let me know your setup (version of CLI + OSS or CLOUD / version). Also, please try updating any if out of date.

In case the simple version works fine. The issue is in the llm call. What I would do is removing the function argument and using instead constants then using the application within the terminal to debug it ( running python app.py simply) and making sure the LLM code works fine.

Let me know if this helps and if you find a solution.

[Arshil-Akkala]

Hey @mmabrouk , thanks for reaching out. I did try it out with a simple setup. So, my agenta --version --> Agenta CLI version: 0.24.4 running on local machine. What I am getting in the container logs is,

2024-09-20 19:47:34 Using Agenta Python SDK version 0.24.4 2024-09-20 19:47:34 Closing trace None 2024-09-20 19:47:34 Cannot close trace, no trace to close 2024-09-20 19:47:34 INFO: 172.18.0.4:48576 - "POST /generate HTTP/1.1" 200 OK

[Arshil-Akkala]

No, I am not.

[mmabrouk]

Are you are using the latest version for the self-hosted agenta?

[Arshil-Akkala]

I am actually using it via docker images.

[mmabrouk]

I think you are using a deprecated version with a new version from the CLI. The two versions are incompatible. You need to update the self-hosted version. You can do this the following way:

To update to the last version of agenta. Please run docker compose -f docker-compose.gh.yml up -d --pull always Which forces docker to pull the last version of the image

This should resolve the problem. However note, that you might need to run migrations depending on the versions you were using. (see migrations to postgres and migrations).

[mmabrouk]

@Arshil-Akkala just checking, where you able to resolve the problem?

[Arshil-Akkala]

Hey @mmabrouk , thanks for the heads-up. That resolved my issue, I was running it on older images.