How to get the Bson size of records in MongoDb?
Using Mongo Queries:
Querying via the DB :
- This will give the ApplicationID:
db.getCollection("Applications").find({'Name':Application Name"},{"_id":1})
- To get the RecordId:
db.Records.find({'Name':"TIIA-211993"},{"_id":1})
- Then you can run the following:
- Object.bsonsize(db.getCollection("Records").findOne({"_id": "{{RecordId }}"},{'{{Values}}':1})
Make sure to use 'findOne' instead of 'find' to get the records for fetching the record size as the 'find' statement gives a cursor as output.
'Values' to retrieves only the memory occupied by fields 'ValuesDocument' to retrieve the memory occupied by documents
Object.bsonsize(db.getCollection("Records").findOne({"_id": "{{RecordId }}"},{'{{ValuesDocument}}':1})
Using Python Script:
import bson
from pymongo import MongoClient
import datetime
import ssl
from swimlane import Swimlane
import pprint
from bson.raw_bson import RawBSONDocument
"""
Connecting to Swimlane
"""
sw = Swimlane(host="{host url}", access_token=sw_context.inputs['swimlane_pat'], verify_ssl=False)
"""
Connecting to mongo
document_class=RawBSONDocument is added to fetch the record as bson
"""
client = MongoClient(host=sw_context.inputs['mongo_hostname'],#
port=sw_context.inputs['mongo_port'],
username=sw_context.inputs['mongo_username'],
password=sw_context.inputs['mongo_password'],
authSource=sw_context.inputs['mongo_auth_source'],
authMechanism='SCRAM-SHA-1',
ssl=True,
ssl_cert_reqs=ssl.CERT_NONE,
document_class=RawBSONDocument)
"""
Assigning the Database
"""
db = client.Swimlane
list_database_names = client.list_database_names()
"""
All the records are in collection called Records
"""
collection = db['Records']
"""
Fetch the record using _id
the condition can be altered w.r.t the available params
Please note that the whole record size include some infrastructural data,
To get the data occupied by the field values and field documets specifically
Up the value of fields dict below to 1
You can also include any more data points required like ApplicationId,ReferenseRecords etc which vary according to the application built
Do a print(collection.find({'_id':'aKzuC1WBtsVcK1Wb4'}) to get the whole record structure
PS: if you have set RawBSONDocument as document_class the output has to be decoded eg: bson.decode(doc.raw)} { # doc----> for doc in document: print(len(doc.raw)) }
"""
record_id = "xxxxxxxxxxx" #Eg:aKzuC1WBtsVcK1Wb4
fields = {'Values':0,'ValuesDocument':0}
document = collection.find({'_id':record_id},fields)
"""
Document will be a cursor with the list of results
"""
for doc in document:
print(len(doc.raw))
Use Cases:
- To find if the record is exceeding the size limit of 16mb:
""" Document fields are excluded here as the limit is confined to 16mb excluding the attachments"""
fields = {'Values':1,'ValuesDocument':0}
document = collection.find({'_id':'{recordid}'},fields)
for doc in document:
doc_len = len(doc.raw)
#16777216 = 16mb
memory_used = (100-((16777216 -doc_len)/16777216)*100)
print(f"The record occupies {memory_used}% of the available limit")