Uploading JSON files to DynamoDB from Python

Posting JSON to DynamoDB through the AWS CLI can fail due to Unicode errors, so it may be worth importing your data manually through Python.

Fortunately this is relatively simple – you need to do this first:

pip install boto3

Then you can run this code to loop over all files in a directory and upload them:

from __future__ import print_function
import boto3
import json
import decimal
import os 

dynamodb = boto3.resource('dynamodb', region_name='us-east-1')

table = dynamodb.Table('talks')

rootdir = "d:\\projects\\image-annotation\\data\\talks\\json\\1"
for subdir, dirs, files in os.walk(rootdir):
  for file in files:
    selected = subdir + "\\" + file
    try:
      with open(selected, encoding="UTF-8") as json_file:
        row = json.load(json_file, parse_float = decimal.Decimal)
        row['id_i'] = str(row['id_i'])

        #print(row)

        table.put_item(
          Item=row
        )
    except:
      print("error: " + selected)

Note here that I’ve chosen “id_i” as my primary key and set it up as a string in AWS, which is why it needs to be monkey-patched here.

One Reply to “Uploading JSON files to DynamoDB from Python”

Hello Gary,

Hope you ding good. I m stuck in one thing and I believe I will get help from you.
I ma loading the Jason file data from s3 to dynamo DB using the lambda function which automatically loads the file data to the DB.

But my piece of logic is working out if the Jason file is having only one object. If a file is having more than one object/record it is failing due to the below issue.

Parameter validation failed:
Invalid type for parameter Item, value: [{‘id’: ‘9a’, ‘Name’: ‘Ram’, ‘Age’: 25, ‘Location’: ‘Bangalore’}, {‘id’: ‘7a’, ‘Name’: ‘Raju’, ‘Age’: 29, ‘Location’: ‘Bangalore’}], type: , valid types: : ParamValidationError
Traceback (most recent call last):
File “/var/task/lambda_function.py”, line 12, in lambda_handler
table.put_item(Item=jsonDictionary)
File “/var/runtime/boto3/resources/factory.py”, line 520, in do_action
response = action(self, *args, **kwargs)
File “/var/runtime/boto3/resources/action.py”, line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File “/var/runtime/botocore/client.py”, line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File “/var/runtime/botocore/client.py”, line 586, in _make_api_call
api_params, operation_model, context=request_context)
File “/var/runtime/botocore/client.py”, line 621, in _convert_to_request_dict
api_params, operation_model)
File “/var/runtime/botocore/validate.py”, line 291, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Item, value: [{‘id’: ‘9a’, ‘Name’: ‘Ram’, ‘Age’: 25, ‘Location’: ‘Bangalore’}, {‘id’: ‘7a’, ‘Name’: ‘Raju’, ‘Age’: 29, ‘Location’: ‘Bangalore’}], type: , valid types:

Could you please suggest me ? Hoping for the positive reply. Many thanks.

Regards,
Harish BS

One Reply to “Uploading JSON files to DynamoDB from Python”

Leave a Reply Cancel reply