Understanding BSON in MongoDB and How It Actually Works

Introduction and Fundamental Terms

As for the formation of modern data management systems, the move from static tables to dynamic documents has greatly influenced the way applications handle data. One of the fundamental features that enable all of this is the flexible data model. An excellent example of NoSQL document-oriented database is MongoDB. MongoDB uses binary data exchange and storage format called BSON. It stands for Binary JSON and it is the primary data language used by MongoDB. In order to know the effectiveness and flexibility of MongoDB, it is essential to understand the operation of BSON. If you are a professional and wish to gain a complete understanding of such fundamental architectural concepts so as to make a career out of them, then mongodb online classes will be highly helpful for you.

Only then will one be able to comprehend the relationship between JSON and BSON, thus becoming able to understand BSON. JSON stands for JavaScript Object Notation and is a lightweight and readable format used for the purpose of exchanging data on networks. JSON came into popularity because of its ease in terms of parsing and reading while at the same time being a format understood by all the languages. A JSON format document consists of an unsorted list of name value pairs and sorted list of values in the form of objects and arrays. Although JSON format is easily readable, the disadvantage of this format is that it is a text-based format and thus machine readable.

The design of BSON has been made by the engineers working at MongoDB for addressing the problem of matching the flexibility of JSON documents with the needs of database engine to be efficient. Unlike the parsing of plain-text strings, BSON turns JSON-like documents into binary encoded serialization format. The format in question is designed specifically to be fast to traverse, compact and includes types that do not exist in plain JSON. Therefore, MongoDB is transforming human-readable structures into binary structures in order to process complex documents using minimum amount of CPU processing.

The Rationale behind the Architecture of BSON

In the course of designing database management system, the architect faces the necessity to make compromises regarding the storage format, memory consumption and traversal speed. The classical database management systems use tightly packed binary records that are either fixed in size or may vary in size depending on the pre-defined schema. It ensures quick positioning and indexing of information. In case of the document-oriented database that should handle semistructured or polymorphic data where each document has unique fields definitions, JSON text-based was the right choice.

Nevertheless, storing and manipulating the database information as the text JSON format poses quite a number of performance problems. For example, consider the case when we have to get some field which is located deep inside the text JSON object. As strings may have different lengths, and we have curly brackets, colon and comma as field delimiters within the JSON object, the database engine will have to go through all characters from the beginning of the JSON object. In order to access the tenth field of the JSON object, the engine will have to parse all preceding fields and determine their length.

The above architectural problem is addressed by the inclusion of metadata information in the binary format itself. The length information and data type information are put at the beginning of each BSON object. Thus a solution is created in which the problem of sequential scanning is solved by making it a direct access problem. In other words when MongoDB query engine wants to see a nested field then it simply uses the length prefix from the parent document/sub-document and jumps over all those bytes without parsing them and reaches to the required field. This particular tradeoff enables us to save a lot in terms of traversal at the cost of some extra space.

Key Features and Design Philosophies of BSON

In order to understand how BSON works, one has to understand the key design philosophies of BSON that include lightweight construction, efficient traversal, and rich data typing.

The simplicity of creating BSON is manifested by the fact that it is lightweight in spite of the binary overhead known as metadata in this case. Unlike XML, whose verbosity is attributed to the closing tags, the overhead in BSON is limited to very specific bytes only. The field name and the value itself are found in the document, thus, making it a redundant structure compared to the relational database whose columns exist only once in the database dictionary. This is the reason why MongoDB is considered to have a flexible schema.

Efficient traversal is achieved by employing the length-prefix for each field. All documents, strings, arrays, and binary subtypes in BSON documents begin with an integer of four bytes that contains the exact number of bytes in this field. While loading the BSON document to the memory from disk, there is no need for MongoDB to eagerly deserialize string values into objects. It is possible to simply map the underlying binary buffer to memory and traverse the document using pointer arithmetic. This ability, referred to sometimes as "zero-copy" or "near zero-copy" deserialization, is what greatly increases the efficiency of the read operation and helps avoid garbage collection problems.

Richness of data types is another major difference between JSON and BSON. The type system of vanilla JSON is extremely restricted. It allows only strings, numbers (with no distinction made between integers and floats), booleans, arrays, objects, and null. Such design makes it hard to write enterprise applications. For instance, one needs special string encoding in order to work with such constructs as timestamps, 64-bit integers, binary data, or regular expressions. BSON, unlike vanilla JSON, adds rich data types to the JSON type system.

Complete Analysis of BSON Data Types

The most important aspect of the data types offered by BSON is flexibility. Since BSON provides particular data types, programmers can work with complex objects within the domain without any problem regarding the type system and avoid the tedious process of serialization.

There are different primitive data types in BSON, such as floating point numbers, 32-bit and 64-bit integers. However, all data types in JSON are numeric; that is why, precision is lost while working with 64-bit IDs or calculations with integer math.

In addition to the other data types, there is another important data type of BSON named ObjectId. If you wish to store documents in the MongoDB, then you need a field that must be unique, and that field is called id, which acts as the primary key for the documents in the database. There are no restrictions on the developers on what value should be put in id field (email, UUID, or an increasing integer), but by default, the MongoDB creates a 12-byte BSON object Id. The object Id is constructed in such a manner that it is globally unique and has some metadata. The object Id has a 4-byte value of the timestamp when it was created and a 5-byte unique value of the machine process.

Handling dates and times is another scenario where BSON proves itself to be far better than normal JSON. JSON lacks any date format inherently, so there are two ways to serialize the date – either by using the ISO-8601 format or the Unix timestamp. Although ISO-8601 is human-readable, finding a specific range of dates becomes tough and tricky since either string comparison or string parsing is needed for that purpose.

Moreover, BinData is also a part of BSON which supports different types of binary data subtypes. There comes the need to store binary data as payloads in the documents of the database in different situations, such as cryptographic keys, serialized objects, compressed data streams, media file formats, and so on. The concept of BSON binary subtypes plays an important role in identifying the type of binary data payload stored in the database, thus helping in efficient processing, indexing, and retrieving the same. Some other specific types of BSON include regex for pattern matching, JavaScript fragments to run on the server-side, timestamps created specifically for the purpose of replica sets, and Decimal128 for performing mathematical operations on money-related data without having problems related to binary floating point operation.

The Internal Mechanisms: The Process of Serialization and Deserialization of BSON

In order to understand BSON completely, it becomes necessary to understand the lifecycle of a document right from sending the document by the application driver to the database storage engine through the network wire.

Serialization entails the process of converting the data structure stored in memory to the binary BSON format. In a situation where there is a call to execute a write operation in MongoDB, then the driver that corresponds to the programming language is invoked. The driver analyzes each key-value pair and identifies the correct BSON type tag and calculates the size of bytes required for strings, arrays, and sub-documents.

When proceeding with the serialization process, the driver follows the strict formatting as defined by the BSON document. First, the BSON document starts with the 4-byte integer describing the size of the whole document in bytes. Then follow the elements, which are made up of the one-byte identifier representing the type of the element and specifying whether it contains string data, integer data, object data, array data, etc. Then follows the name of the field, which is presented in the form of the null-terminated string. Following the name of the field comes the value of the element depending on its data type. In case the element is a sub-document or an array, the same process repeats itself with the length of the element and its elements.

Next, the byte stream goes through the network socket from the driver to the process of the MongoDB server. As the data is in the BSON format, it implies that there is no necessity for the parsing and translation of the byte stream into the intermediate form for the interpretation of the data by the database. It is enough to read the headers from the wire protocol, the size of the BSON document and process the command and write the bytes in the memory cache of the database engine's storage system.

The inverse operation is deserialization and occurs in case of extraction of data from the database via the queries. The raw BSON bytes go from the server through the network socket to the driver and with the help of the byte buffer and length prefixes, type tags, and field names in the BSON structure, it creates the native objects from the BSON structure. This operation may be performed lazily when accessing the sub-documents in some modern drivers.

BSON and Disk Storage: Data persistence in MongoDB

While BSON is applied for network and database queries, it is important to understand how BSON is connected with disk storage in order to get information about the working principles. There is a pluggable storage engine in MongoDB. As a default option, it is WiredTiger and it is the most popular engine in the latest releases of MongoDB.

When WiredTiger writes BSON documents into the disk, it saves them as BSON-serialized records in data files. Memory and disk storage management by WiredTiger is carried out through cache management, compression and logging (write-ahead logging). Since the binary format of BSON is predictable and easily compressible, compression of data blocks using algorithms like Snappy and zlib is quite easy. Binary format compresses extremely well, optimizing the requirements for the disk storage and reducing I/O bandwidth requirements.

Also, MongoDB creates indexes on the fields of documents according to the BSON values. When creating the indexes on certain fields, the storage engine extracts the BSON values corresponding to the fields from the documents, sorts them and creates indexes using the B-tree or WiredTiger indexes. As BSON values always have the same order of sorting regardless of the data type, the database engine is capable of performing efficient range searches, equalities and sorting of BSON values of the indexes without any costly run-time conversions of data types.

Performance Implications and Trade-offs of BSON

In computer science, every design decision has its price. Analysis of the performance of BSON should involve understanding the implications of its use in terms of the following characteristics: CPU utilization, memory bandwidth, network throughput, and disk space usage.

On the other hand, the BSON format provides unrivaled levels of traversal and lookup performance as far as document databases are concerned. The presence of explicit typing and lengths for fields allows query operations on deeply nested hierarchies to bypass unnecessary portions of documents without the need to perform any additional processing. Compared to systems that rely on encoding of data in rigid binary forms which must be compiled into corresponding schemas, BSON strikes the best compromise between flexibility and speed of access. Schema migration and restructuring of database tables by programmers in order to introduce additional fields is not required.

Regarding the drawbacks of the BSON data format, there is one that should be noted, and that is the increased size of storage needs. Due to the fact that field names are included multiple times within each document in which they appear, the collections that use field names of large sizes can take more storage space than those in relational databases where the names of fields are used once in the metadata table. While compression algorithms at the database level could somewhat help to solve this issue, programming standards for MongoDB applications encourage using short and descriptive field names.

Another aspect to consider is how much CPU time does serialization/deserialization take. Although the speed of BSON parsing is significantly greater than the speed of parsing of JSON as text, it still takes some CPU time to convert binary representation of data into language-specific objects. When a microservices architecture deals with millions of documents per second, the issue can be related to serialization cost since in this case the application will have to deserialize whole documents to get several required fields. There are different ways to solve this problem offered by MongoDB drivers and query projections which allow selecting only necessary fields on the server side.

Comparison of BSON with Other Serialization Formats

To understand better the technology of BSON, it is needed to compare this format with other binary serialization formats like Protocol Buffers, Apache Avro, and MessagePack.

Avro by Apache is another framework that is widely used in big data for serialization purposes within frameworks such as Apache Hadoop and Apache Kafka. The schema must be included along with the data that is being serialized with Avro. Similarly to Protocol Buffers, Avro works within a statically typed system, where the producer and consumer of the data agree on the type contract of the data. Even though Avro has great performance and compressions capabilities, it does not offer the flexibility provided by BSON in MongoDB.

It can be considered that BSON is related to MessagePack as well because of the similar ideas. MessagePack is binary serialization format which is similar to JSON format but very lightweight and efficient. MessagePack enables to encode data according to common JSON types and it is binary serialization format with minimal overhead without schema requirements. Nevertheless, MessagePack was created as a general purpose binary wire protocol for RPC and messaging and it was not designed as a storage format for data. Contrary to MessagePack, BSON has been specially created to function within database environment. Hence, there are special types of data such as ObjectIds, Date and precise decimal type and length prefixes which can be used for indexing, execution planning and fast in-memory traversal.

Advanced Patterns in BSON and Engineering Practices

To understand BSON and to work with MongoDB it is necessary to know how to use format properly. There are some practices which are used by database specialists while working with BSON.

First, efficient field naming is an important factor in the process of optimizing the storage of data. As far as our case is concerned, storage of the field names in BSON adds additional space requirements. If we use long field names for a collection having several million records, we will occupy additional space both in hard drive and memory because of the large number of field names that we have. The use of short field names instead of the explanations of the meaning of the fields makes our database compact.

Second, correct use of BSON data types helps to optimize the storage and prevent bugs in the code. For example, in case of dealing with numbers, it is necessary to make sure that numbers will be stored in numerical form and not in string form. If we store IDs and money values in string form, we compare strings, and not make calculations and, thus, slow down index and range searches.

Thirdly, it is important to understand the projection concept and the level of nesting of sub-documents in order to ensure the efficiency of the memory and bandwidth usage. Although the use of BSON ensures deep nesting of sub-documents and arrays, which in turn leads to hierarchy modeling of complex relationships in the domain, but too much nesting can make querying hard and can increase the size of documents. The engineers need to find the balance between advantages of data nesting in a single document by means of BSON and limitations on the document size in MongoDB.

Conclusion

BSON format is not merely the binary form of JSON format, but the format which allows MongoDB to combine its schema flexibility, fast queries and data types. With the help of BSON, the documents are encoded into the binary stream with length prefixes, type identifiers and unique data types, thus omitting CPU-intensive character scans of text formats.

From the way it manages ObjectIds, which are unique around the world and its precision in decimal values, to its integration with the current storage and networking systems, BSON is a perfect illustration of systems design which perfectly marries two seemingly contradicting features of usability and performance. Knowledge in BSON becomes very vital for the developers, DBAs, as well as system infrastructure architects who have to design infrastructure for applications in use currently, since this will help them make maximum use of MongoDB. To know everything about databases starting from scratch, you need to check out some unique mongodb dba online training.