Populating custom fields when indexing documents in an Azure AI Search index

[AsmaaMHadir]

I have an Azure AI index defined with the following schema:

  "name": "newindex5",
  "fields": [
    {
      "name": "id",
      "type": "Edm.String",
      "key": true,
      "retrievable": true,
      "stored": true,
      "searchable": false,
      "filterable": true,
      "sortable": false,
      "facetable": false,
      "synonymMaps": []
    },
    {
      "name": "content",
      "type": "Edm.String",
      "key": false,
      "retrievable": true,
      "stored": true,
      "searchable": true,
      "filterable": false,
      "sortable": false,
      "facetable": false,
      "synonymMaps": []
    },
    {
      "name": "content_vector",
      "type": "Collection(Edm.Single)",
      "key": false,
      "retrievable": false,
      "stored": true,
      "searchable": true,
      "filterable": false,
      "sortable": false,
      "facetable": false,
      "synonymMaps": [],
      "dimensions": 1536,
      "vectorSearchProfile": "myHnswProfile"
    },
    {
      "name": "metadata",
      "type": "Edm.String",
      "key": false,
      "retrievable": true,
      "stored": true,
      "searchable": true,
      "filterable": true,
      "sortable": false,
      "facetable": false,
      "synonymMaps": []
    },
    {
      "name": "title",
      "type": "Edm.String",
      "key": false,
      "retrievable": true,
      "stored": true,
      "searchable": true,
      "filterable": true,
      "sortable": false,
      "facetable": false,
      "synonymMaps": []
    },
    {
      "name": "source",
      "type": "Edm.String",
      "key": false,
      "retrievable": true,
      "stored": true,
      "searchable": true,
      "filterable": true,
      "sortable": false,
      "facetable": false,
      "synonymMaps": []
    },
    {
      "name": "page_number",
      "type": "Edm.String",
      "key": false,
      "retrievable": true,
      "stored": true,
      "searchable": true,
      "filterable": true,
      "sortable": false,
      "facetable": false,
      "synonymMaps": []
    }
  ],
  "scoringProfiles": [],
  "suggesters": [],
  "analyzers": [],
  "tokenizers": [],
  "tokenFilters": [],
  "charFilters": [],
  "normalizers": [],
  "similarity": {
    "@odata.type": "#Microsoft.Azure.Search.BM25Similarity"
  },
  "vectorSearch": {
    "algorithms": [
      {
        "name": "default",
        "kind": "hnsw",
        "hnswParameters": {
          "m": 4,
          "efConstruction": 400,
          "efSearch": 500,
          "metric": "cosine"
        }
      },
      {
        "name": "default_exhaustive_knn",
        "kind": "exhaustiveKnn",
        "exhaustiveKnnParameters": {
          "metric": "cosine"
        }
      }
    ],
    "profiles": [
      {
        "name": "myHnswProfile",
        "algorithm": "default"
      },
      {
        "name": "myExhaustiveKnnProfile",
        "algorithm": "default_exhaustive_knn"
      }
    ],
    "vectorizers": [],
    "compressions": []
  },
  "@odata.etag": "\"0x8DD1EAD2AA16D56\""
}

I run the following indexing script to add sample docs to my index:

texts = ["Document content 3", "Document content 4"]
metadatas = [
    { "title": "Title 1", "source": "Source A", "page_number": "1"},
    { "title": "Title 2", "source": "Source B", "page_number": "2"},
]


vector_store.add_texts(texts, metadatas=metadatas)

However, when I check the documents in my Azure index I see that the field title,``source,and page_number` are all null:

{
  "@odata.context": "https://phoenix-hudhud.search.windows.net/indexes('newindex5')/$metadata#docs(*)",
  "@odata.count": 2,
  "value": [
    {
      "@search.score": 1,
      "id": "ZGJlZTBhMDYtMmIzZC00YmYwLWEzMzItMDIzYjk4YTRlNmQw",
      "content": "Document content 3",
      "metadata": "{\"title\": \"Title 1\", \"source\": \"Source A\", \"page_number\": \"1\"}",
      "title": null,
      "source": null,
      "page_number": null
    },
    {
      "@search.score": 1,
      "id": "ZmQ3YzZjMzktOTU3OC00YmJiLWE3OTYtNGVlMzAzMjVlYzE2",
      "content": "Document content 4",
      "metadata": "{\"title\": \"Title 2\", \"source\": \"Source B\", \"page_number\": \"2\"}",
      "title": null,
      "source": null,
      "page_number": null
    }
  ]
}

How do I index my documents such as the values in the metadata dictionary I provide to the method are passed to my "title", "source", and "page_number" fields defined in my index' schema?

[dosubot[bot]]

To ensure that the values in your metadata dictionary are correctly populated in the "title", "source", and "page_number" fields in your Azure AI Search index, you need to make sure that the metadata fields are mapped correctly when adding documents to the index. Here’s how you can do it:

1. Ensure Correct Field Mapping: When you add documents to the index, make sure that each field in the metadata dictionary is explicitly mapped to the corresponding field in the index schema. This means that the metadata keys should match the field names in the index schema.

2. Modify the Indexing Script: Instead of passing the metadata as a single JSON string, you should ensure that each field is directly mapped. Here’s an example of how you might modify your script:

texts = ["Document content 3", "Document content 4"]
metadatas = [
    { "title": "Title 1", "source": "Source A", "page_number": "1"},
    { "title": "Title 2", "source": "Source B", "page_number": "2"},
]

# Add texts with metadata to the index
for text, metadata in zip(texts, metadatas):
    document = {
        "content": text,
        "title": metadata["title"],
        "source": metadata["source"],
        "page_number": metadata["page_number"]
    }
    vector_store.add_document(document)

3. Check the Data Types: Ensure that the data types in your metadata match those defined in your index schema. For example, if "page_number" is defined as a string in the schema, make sure it is passed as a string in the metadata.

By ensuring that each field in the metadata is directly mapped to the corresponding field in the index schema, you should be able to populate the "title", "source", and "page_number" fields correctly.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[AsmaaMHadir]

To ensure that the values in your metadata dictionary are correctly populated in the "title", "source", and "page_number" fields in your Azure AI Search index, you need to make sure that the metadata fields are mapped correctly when adding documents to the index. Here’s how you can do it:

1. Ensure Correct Field Mapping: When you add documents to the index, make sure that each field in the metadata dictionary is explicitly mapped to the corresponding field in the index schema. This means that the metadata keys should match the field names in the index schema.
2. Modify the Indexing Script: Instead of passing the metadata as a single JSON string, you should ensure that each field is directly mapped. Here’s an example of how you might modify your script:

texts = ["Document content 3", "Document content 4"]
metadatas = [
    { "title": "Title 1", "source": "Source A", "page_number": "1"},
    { "title": "Title 2", "source": "Source B", "page_number": "2"},
]

# Add texts with metadata to the index
for text, metadata in zip(texts, metadatas):
    document = {
        "content": text,
        "title": metadata["title"],
        "source": metadata["source"],
        "page_number": metadata["page_number"]
    }
    vector_store.add_document(document)

3. Check the Data Types: Ensure that the data types in your metadata match those defined in your index schema. For example, if "page_number" is defined as a string in the schema, make sure it is passed as a string in the metadata.

By ensuring that each field in the metadata is directly mapped to the corresponding field in the index schema, you should be able to populate the "title", "source", and "page_number" fields correctly.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosu

This is not the correct syntax.

[dosubot[bot]]

To correctly populate the "title", "source", and "page_number" fields in your Azure AI Search index, you should ensure that the metadata fields are directly mapped to the corresponding fields in the index schema. Here's how you can modify your script:

texts = ["Document content 3", "Document content 4"]
metadatas = [
    { "title": "Title 1", "source": "Source A", "page_number": "1"},
    { "title": "Title 2", "source": "Source B", "page_number": "2"},
]

# Add texts with metadata to the index
documents = [
    {
        "content": text,
        "title": metadata["title"],
        "source": metadata["source"],
        "page_number": metadata["page_number"]
    }
    for text, metadata in zip(texts, metadatas)
]

vector_store.add_texts(documents)

Make sure that the add_texts function in your vector store is designed to handle documents with these fields. If the function expects a different structure, you may need to adjust the way documents are passed to it.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}