Model Garden Gemma Deployment on Vertex - incomplete documentation about prediction response format

[afirstenberg]

Environment

• Deployed a gemma-7b-it model on Vertex AI Model Garden using the "Deploy" button from the Gemma card. No additional tuning was done.
• I have an instance running on a g2-standard-12 machine with I4 GPU. It is visible in the Online Prediction section of my Cloud Console.
• I am able to reach the endpoint without any issues.

Unable to find any good documentation on what needs to be sent to the model and what to get back, I used the "Model Garden Gemma Deployment on Vertex" notebook to try and get an idea. It did provide an example for what to provide to the prompt:

vertex-ai-samples/notebooks/community/model_garden/model_garden_gemma_deployment_on_vertex.ipynb

Line 621 in b37ed6e

"instances = [\n",

However, it does not indicate what to expect for the reply. So it wasn't clear that the reply includes the original prompt as well as part of the reply string and this would need to be parsed out:

{
  "predictions": [
    "Prompt:\nWhat is a car?\nOutput:\nA car is a motor vehicle that is propelled by gasoline. It has four wheels, a steering wheel, and a seat."
  ],
  "deployedModelId": "xxx",
  "model": "projects/111/locations/us-central1/models/gemma-7b-it-google",
  "modelDisplayName": "gemma-7b-it-google",
  "modelVersionId": "1"
}

The documentation should make clear what the output will be.

[gericdong]

@kathyyu-google: could you please assist? Thanks.