Trying to Serve ModelA and ModelB at the same Bento Service Endpoints

[hailindu]

I have been checking for similar discussion posts and my question is still not being answered, so have to open this post.

I have two models, ModelA and ModelB for XgboostEven though discussion 1260 said parallel inference is not yet supported by BentoML, some discussion does provide a similar coding structure as I expected so I wrote the following code.

%%writefile Predictions.py
@env(pip_packages=['xgboost']

@artifacts([XgboostModelArtifact('model1'),
XgboostModelArtifact('model2)
])

class ModelPredcition(BentoService):
        @api(input=DataframeINput(), mb_max_latency=50000, mb_max_batch_size=5000, batch=True)
        def predict_model1(self, df):
                dmatrix = xgb.Dmatrix(df)
                return self.artifacts.model1.predict(dmatrix)

        @api(input=DataframeINput(), mb_max_latency=50000, mb_max_batch_size=5000, batch=True)
         def predict_model2(self, df):
                dmatrix = xgb.Dmatrix(df)
                return self.artifacts.model2.predict(dmatrix)

from Predictions import ModelPredcition
bento_svc = ModelPredcition()
bento_svc.pack('model1', model1)
bento_svc.pack('model2', model2)

saved_path = bento_svc.save()

\anaconda3\lib\site-packages\bentoml\service\__init__.py in pack(self, name, *args, **kwargs)
772 - Lreturn this BentoService instance
773 - """
774 - self.artifacts.get(name).pack(*args, **kwargs)
775 - return self

\anaconda3\lib\site-packages\bentoml\service\__init__.py in get(self, item)
243 - """Access the BentoServiceArtifact instance by artifact name
244 - """
245 - return self[item]
247 - def get_artifact_list(self) -> List[BentoServiceArtifact]:
KeyError: 'model1'

I assume the Artifact does not recognize the model name I have defined but according to this post it should work

I want to access two models at the same time.

localhost:5000/modelA 
localhost:5000/modelB

Not

localhost:5000/modelA 
localhost:5001/modelB

Many thanks!

[aarnphm]

``@artifacts([XgboostModelArtifact('model1'),
XgboostModelArtifact('model)
])```
The second artifact should be model2 instead of `model`

You haven't defined the model1 and model2 for packing process, thus It returns KeyError.

However, the team is working hard on bentoml 1.0, and the behaviour is expected to change, so stay tuned for that 😄

[hailindu]

Hi Aaron, thank you so much for your reply.

Does that mean I can go into the packing process.py and adding a new function for model2? So does that help to serve ModelA and ModelB at the same time using one BentoService?

I would definitely wait, just wondering if I can explore that myself. The team is fantastic!