Import error for pymongo

[aclasky]

I worked through the tutorial to set up AWS Lambda with an API gateway and associated MongoDB database. I ran the updated main.py code after adding the associated credentials to my_secrets.py (and environment variable for the Lambda function).

I received the following error:

Sending document to AWS Lambda Response: (200), msg = {"errorMessage": "Unable to import module 'lambda_function': No module named 'pymongo'", "errorType": "Runtime.ImportModuleError", "requestId": "", "stackTrace": []} Added Successfully

The tutorial notes to add the pymongo layer to the lambda function, but I don't see an option for it:

Is this a custom layer that should be added? Or is it included as part of one of the powertools options? Thanks!

[sgbaird]

We are thinking that we might remove the AWS part from the hello world course. What this would mean is the orchestrator would handle uploading and downloading of data, and there would be no direct upload of data via the microcontroller. Instead, that concept, which is still important for some applications, would be covered in a later course. Thoughts?

For the specific error mentioned, @SissiFeng could you advise?

[SissiFeng]

They actually thinking that we might remove the AWS part from the hello world course. What this would mean is the orchestrator would handle uploading and downloading of data, and there would be no direct upload of data via the microcontroller. Instead, that concept, which is still important for some applications, would be covered in a later course. Thoughts?

For the specific error mentioned, @SissiFeng could you advise?

If students find aws operations too complex, we can consider using the built-in functionality provided by MongoDB Atlas to perform CRUD operations directly in the database without relying on an external back-end server. I found that it should be possible to automate database operations and event responses using Triggers and Functions.

[sgbaird]

Interesting! I would have expected this to have come up in one of the suggested alternatives when they announced the deprecation.

[SissiFeng]

I worked through the tutorial to set up AWS Lambda with an API gateway and associated MongoDB database. I ran the updated main.py code after adding the associated credentials to my_secrets.py (and environment variable for the Lambda function).

I received the following error:

Sending document to AWS Lambda Response: (200), msg = {"errorMessage": "Unable to import module 'lambda_function': No module named 'pymongo'", "errorType": "Runtime.ImportModuleError", "requestId": "", "stackTrace": []} Added Successfully

The tutorial notes to add the pymongo layer to the lambda function, but I don't see an option for it:

Is this a custom layer that should be added? Or is it included as part of one of the powertools options? Thanks!

Select Custom Layer.
Upload pymongo_layer.zip file, and ensure the correct Python version is selected.

[aclasky]

@SissiFeng thanks for your reply. I had checked the custom layers before but there are none compatible for my runtime (Python 3.12). Do you know for which runtime the custom pymongo layer is compatible? I'm wondering if a different version of Python might be required.

Additionally, I tried to create a layer and upload the tar.gz file from the pymongo website (link) but it was flagged as incompatible. Is there a separate pymongo_layer.zip file that I'm not finding on my end?

[SissiFeng]

The current AWS Lambda stable supports Python versions 3.7-3.10. If aws suggests an incompatible error, you can also install pymongo in local environment and package it as a Lambda layer.

[sgbaird]

Related issue: #108.

Hopefully will be better resolved in #132.

[aclasky]

Hi all, thank you to both @sgbaird and @SissiFeng for their help. Also, thanks to the poster from #108 who provided insight as well. Glad to hear #132 is being put together. I do think learning the Lambda/API Gateway/MongoDB pipeline was worthwhile and, for what it's worth, I do support keeping this as part of the course. I just wanted to update here for completion's sake since the solution was more involved than expected.

Firstly, since pymongo was added manually, the .zip file had to be reorganized to reflect the proper path in order for the python runtime to access it:

This technically solved the import error I had originally flagged when I opened this discussion.

There was also a mismatch between the main.py file and lambda function, but this was also flagged by #108 so I won't elaborate on it.

Finally, connecting with MongoDB using the connection string generated from MongoDB directly raised timeout errors which were difficult to debug (MongoDB doesn't raise exceptions, there is just no connection, so the timeout error is from the Lambda function). Some things I noted:

1. mongodb+srv requires dnspython, which is not included in the pymongo package
2. The password needs to be for the database user and not the actual account (needed to set this up separately on MongoDB).
3. The password must be URL encoded when used in the connection string (in the case that special characters like #$%^ are used)
4. Converting the URL to mongodb:// requires specification of the hosting IP address, which I was not able to figure out.

Ultimately, I was not able to work with the mongodb:// URL directly and instead downloaded dnspython as a separate layer (with the same path correction I mentioned above) that I load before pymongo in order to process the mongodb+srv URL. With the updated database user password credentials, I was able to connect and write to the database successfully.

I'm sure there is a more elegant way to do it but this workaround did solve the issue so I thought it would be worth documenting here.

[sgbaird]

Thank you for the thorough description! I'll let @SissiFeng handle follow up on the technical bits.

I do think learning the Lambda/API Gateway/MongoDB pipeline was worthwhile and, for what it's worth, I do support keeping this as part of the course.

If this were to be moved to course 4 (focused on software development), with a note or link to that portion included in 1.5, would that be preferred over keeping it in 1.5 directly?