Master MongoDB Architecture, Data Modeling, Scaling, and Modern Engineering

An Overview of the Modern Data Systems

For many years, RDBMS was the standard when it came to enterprise application development. These systems, which are largely based on Edgar F. Codd’s relational model with their strict schemas, normalization, and SQL, provided trustworthy transactional functionality and data consistency. Yet, Due to the growth of web applications, the way the industry operated transformed fundamentally.

Applications started generating huge amounts of semi-structured and polymorphic data kinds: social media posts, clickstream events, states of mobile apps and IoT telemetry. Within this framework, the traditional systems with their rigid tables, foreign keys and expensive operations with multiple tables had their share of engineering hurdles. Developers’ teams would have to face the same difficulties over and over again when migrating their relational models in the process of fast agile release cycles or horizontally scaling their RDBMS in their distributed storage systems.

Late in the 2000s, a movement named NoSQL (Not Only SQL) came out to deal with these problems. Among the many types of DBMSs that evolved—such as key-value stores, wide-column stores, graph databases, and document databases—MongoDB appeared to be the front runner among document-oriented databases. Founded by three computer whizzes—Dwight Merriman, Eliot Horowitz, and Kevin Ryan—MongoDB was designed around the idea that data should be stored in a database the same way that application developers use the objects in their program, without limiting the system’s querying ability and the possibilities for indexing and horizontal scaling.

MongoDB has become a completely functional general-purpose data platform that is capable of being widely used in various applications that are hosted in the cloud, as well as in any data center. For developers seeking to master these distributed patterns of data management, structured mongodb training online can help in obtaining practical knowledge of navigating legacy migration processes and contemporary architectures.

Basic Philosophy and Document Model

The document model is at the center of MongoDB. Rather than storing data in rows and columns across tables, MongoDB organizes it into flexible, JSON-like units called documents. This means documents grouped together into collections are included in the database.

From JSON to BSON

Although developers handle MongoDB using normal JSON, the engine runs through the process of storing, indexing, and sending messages using BSON.

Extended Data Types: Standard JSON only handles strings, numbers, booleans, arrays, objects, and null values. Furthermore, BSON expands this range of types to include for instance 64-bit integers, 128-bit decimal floating points for high-precision financial calculations, UTC timestamps, regular expressions, binary data payloads (like raw bytes or UUIDs), and 12-byte ObjectIds.
Rapid Traversal and Parsing: BSON documents are equipped with explicit prefixed lengths and index headers. Because of this, storage engines easily traverse documents and skip unrequested nested elements as well as build queries without the need to parse the entire stream of characters.
Space-efficient Encoding: Since the data is represented as binary, it does not take much space during transfer through network sockets and while being stored on persistent storage devices.

The Object-Relational Impedance Mismatch

While developing classical relational applications, engineers create significant mapping layers – usually Object-Relational Mappers (ORMs) – that allow them to convert in-memory data representations into tabular rows linked through foreign keys. The architectural tension that arises from this process is termed the object-relational impedance mismatch, resulting in additional costs for maintenance, serialization time, and query optimization issues.

This problem is addressed by MongoDB through the alignment of its database model with the way applications store data in memory. A customer’s profile containing several delivery addresses, various payment information and his preferences can be treated as one document. This makes it easier for the application to retrieve the customer without multiple joins in the process of reading the data.

Dynamic Schema and Flexible Validation

MongoDB has moved to the concept of fixed schema used in SQL databases where every record in the table must follow all the rules. Although both documents are in the same collection and created on the basis of the same document type, their fields, or types of information described in them may be different.

This approach gives developers an ability to deliver new features without any downtime due to required database changes. When a solution requires strict control and laws are to apply in some specific conditions MongoDB enables introducing validation based on declarative JSON schema.

Core Architecture and the Storage Engine

To comprehend how MongoDB addresses data persistence, concurrent access, and memory management issues, it is necessary to evaluate the storage engine it is based on.

WiredTiger Storage Engine

The WiredTiger storage engine became available in MongoDB version 3.2. It has been created to provide high-level performance, productivity, and fast response times. As a result, WiredTiger replaced outdated memory-mapped engines and adopted a new architecture based on lock-free data structures, multi-version concurrency control, and cache use optimization.

Concurrency and Document Locking

Older types of document databases would either lock the entire database or restrict access to all documents in the database (or collection). This means that during a writing operation, all reading operations conducted simultaneously would be blocked. WiredTiger provides more granular levels of access.

WiredTiger applies optimistic concurrency control mechanisms along with hazard pointers, which allows multiple users to read and modify different documents in the same collection at the same moment without any conflicts. However, if conflicting users try performing conflicting actions on the same document at the same time, WiredTiger will recognize the incident and automatically stop any operation that would conflict with other ongoing actions.

Memory Management, Caching, and Checkpointing

WiredTiger uses a special in-memory cache to manage memory:

Allocation of Cache: WiredTiger, by default, uses approximately 50 percent of the total available system memory for its internal data cache. Remaining memory is left for page cache in the file system, currently active connections, and buffers used for processing queries.
Life Cycle of Page: Read operation reads compressed B-Tree pages from the disk and loads them into the memory where they are uncompressed and put into cache memory.
Checkpoints: A background thread makes a checkpoint every certain period of time (every 60 seconds or after 2 gigabytes of erased data).

The Journaling Process

Since checkpoints occur at intervals, power failures or hardware errors may lead to data loss if the unrecorded data existed only in volatile memory. WiredTiger uses a mechanism called Journaling to ensure that the write transactions are durable.

Every write operation (insert, update, delete) is recorded onto the journal file that resides on the disk either before or during the moment when data is changed in the RAM. Once a server crashes, all the activities performed after the last successful checkpoint will be repeated during recovery using the journal. A deep knowledge of low-level technologies such as wiring tiger cache tuning and recovery issues belongs to the most basic points of an advanced mongodb online course.

Searching, Indexing and Aggregation

Data retrieval and modification are important functions of a database system, so MongoDB offers the features of querying and indexing, as well as an engine for data aggregation.

Filtering and Querying Expressions

MongoDB supports rich query operators that allow:

Exact Match and Comparison: Equality, inequality, and comparison of ranges (more than, less than), and membership in sets.
Logical Operators: Logical constructs using expressions AND, OR, NOR, and NOT.
Navigation of Arrays: Querying for specific elements by index, matching elements that meet a certain criterion, and match on more than one conditions at the same time along with querying the sizes of arrays.
Querying of Nested Documents: Access to nested attributes by using standard dot-notation.

Indexing Techniques

In the event that the database lacks the indexes, it will then have to perform a full collection scan, looking through every document available in the disk or cache to make a query. MongoDB utilizes several sources of indexes which use B-Tree structures:

Single-field indexes: These indexes make use of a singular field which could be top level or embedded and have the possibilities of ascending and descending traversals.
Compound indexes: These are multi-field indexes that can be used during the filtering of queries using several filters or ranges and sorts. These compound indexes follow the Equality, Sort, Range principle of operations whereby the fields going through equality testing are placed close to each other followed by fields which have gone through sorting and lastly have fields which have undergone filtration.
Multi-key indexes: These indexes come through automatically during indexing.
Text indexes: These indexes are specifically used to perform a series of keyword searches.
Spatial indexes: The indexes include the two-dimensional indexes and two-dimensional indexes.
Indexes based on Time-To-Live: It is a simple index which is unique on date fields in documents that have an expiry date after a certain number of seconds and is used mainly for session expiration and caching purposes.
Partial and Sparse Indexes: These are indexes that store the reference information of the documents that satisfy the criteria mentioned.
Wildcard Indexes: These dynamic indexes index all fields that match an arbitrary subdocument path pattern, including data governed by user-defined attribute schemas.

The Aggregation Framework

MongoDB offers the Aggregation Framework for the purpose of complex data transformation, multi-step data processing, and analytical computations. Based on the logic of the data pipeline, documents will get through an aggregation chain and pass one stage after another.

The important stages are:

Filtering and Projection: The match stage enables the limiting of the document set back at the very beginning in order to diminish the size of data while the project or addFields stages provide modifications of the structure of documents and creation of new fields.
Unwinding and Grouping: The unwind stage is a process of getting flat arrays into the streams of documents while in the group stage documents will become grouped around the same keys in order to obtain the sum, average, maximum, standard deviation, etc.
Joining and Merging: The lookup stage makes left-outer-joins to other collections while with the help of merge or out the data will be inserted into the destination collections and analytical views.
Window Functions and Faceting: Some particular functions will be incorporated at this stage to enable the use of moving averages, growing totals, rank orders, etc. all in one processing.
The question optimizer analyzes the whole process, moving filter steps in front of alteration ones and applying indexes wherever possible to lessen utilization of CPU and memory.

Architecture for Distributed Systems: Replication and Availability

In the cases of project-critical deployments, architectures based on a single node are not acceptable, because of their potential failure within the project. Using Replica Sets, MongoDB achieves high availability and disaster recovery and ensures data redundancy.

Replica Set Topologies and Roles of Members

A replica set represents a cluster of mongod instances that have identical copies of database data. A normal production replica set comprises at least three nodes that store data:

Primary node: one and only one primary node operates, and it is the one on which write documentation occurs; after that, all the changes are applied to the data and archived.
Secondary nodes: secondary nodes are the ones that do the replication of the primary’s operations.

Specialized instances:

Hidden nodes: Secondary instances with complete replicas of data but unfathomable to client apps. Such nodes usually supply functions for business intelligence and extraction of backups in real-time.
Delayed nodes: Secondary instances set to function a specific fixed delay (for example, 6 hours) behind the main node that provides protection against human errors.
Arbiters: Instances that neither hold copies of data nor process any information but which fulfill the role of the odd vote in elections. Nowadays, the use of hundreds of data-bearing nodes is encouraged over using arbiters to avoid split brain cases.

Consensus in elections and failover

The nodes in a replica set are constantly tracking the health of the cluster using bidirectional heartbeats issued every two seconds. If the primary node becomes unreachable due to hardware failure, a network partition, or a software crash, the secondary nodes become aware of the problem within a configurable election timeout (by default it is set to 10 seconds). The elections among secondary nodes are held according to a Raft-based algorithm, namely:

The secondary nodes vote for a candidate with the latest record in its oplog.
As soon as the candidate gets more than half of the votes (a strict majority), it becomes a primary node.
The application driver automatically identifies the topology changes and starts sending writes to the newly elected primary node without needing any manual actions from the administrators.

The Operational Log (Oplog)

Oplog is a capped collection that exists in the local database of all replica set members. Any operation changing the primary is converted into an idempotent operation and added to the oplog.

Idempotent means that the oplog entry can be applied several times but entails single application results. Secondary nodes keep on reading the oplog of their primary node and applying the oplog entries on their local threads.

Horizontal Scaling: Sharding and Distributed Data Partitioning

When the volume of the data goes beyond the capacity limit of a single device, or when the transaction speed surpasses the capability threshold of a single computer, it would then result in the inconvenience both economically and in terms of physical space if vertical scaling is used. MongoDB tries to counter this issue through the implementation of horizontal scaling with the use of sharding. Therefore, mastering methods such as partition key and routing strategy of MongodB through mongodb training is extremely important for backend engineers trying to design distributed systems.

Elements of a Sharded Cluster

A completely decentralized MongoDB sharded cluster comprises of three distinct architectural layers:

Shards: Every shard refers to a separate replica set in charge of preserving a unique part of the overall data from the cluster. It is through the use of replica sets for individual shards that the system has high availability and durability in each of the partitions.
Config Servers: This term refers to a dedicated three-member replica set which is charged with maintaining cluster metadata, routing tables, chunk ranges, and configuration states. It is the config servers that keep an authoritative mapping of data chunks on various physical shards.
Query Routers (mongos): This definition stands for stateless routing processes that work as a connecting link between client and sharded cluster. The client drivers connect to mongos instead of the shards directly. The mongos caches metadata from config servers, analyzes incoming queries, identifies the shards that keep the required files, applies operations accordingly, and combines the results received.

Sharding Techniques

Partitioning of data across the shards is done using a Shard key defined by one or multiple immutable fields that are present in each and every document in the entire collection that is shared:

Range based Sharding: Data is partitioned into blocks using the raw value of the shard key. This technique is useful for range queries like getting records between two dates. However, if the shard key as in the case of self-incrementing IDs or current timestamps is ever-increasing, the different writes are done to one shard only where the highest piece of the range is located.
Hashed Shard: MD5-based hashing is performed on the shard key and it is used to collect data in chunks. This technique allows distributing the writes all over the cluster without hot-spotting although range queries might need to be run on all shards in parallel.
Zone-based sharding: With this technique, administrators can link particular shards with regions or infrastructure levels, enabling organisations to comply with the mandates of data sovereignty (e.g. storing the data of European users on physical servers located in the territory of the EU) as well as implement data tiering (for example, storing the frequently accessed hot data on fast NVMe drives and pushing low-demand archived data onto cheaper drives).

Dividing, Balancing, and Splitting

A group of data in a collection is divided into a continuous block of information called Chunks (by default, chunks are 64 megabytes in size). When the data in the chunk starts to exceed the specified limit, the chunk is split into smaller chunks.

If the chunks are unevenly distributed across shards, the Balancer, an automated function, disperses the chunks among shards, maintaining a uniform distribution across the network without any disruption in the application.

7. Consistency, Durability, and Distribution Transactions

When building the distributed application, there is a need for a precise understanding of the tradeoff in consistency and durability, based on CAP theorem and PACELC model.

Tunable Consistency: Reading Concerns and Writing Concerns

MongoDB offers flexible means for ensuring data safety and consistency during operations:

Write Concern Parameters

Write concern indicates the acknowledgement needed from the cluster before an operation can be deemed successful.

w: 1: Acknowledgement takes place when the write is committed to the memory cache of the primary.
w: majority: Acknowledgement is given after the writer has been committed to a strict majority of the members of the replica set.
j: true: The writer must be also committed to the journal for being successful, so that it can survive the crash of the host.

Read Concern Levels

The way a read operation receives its data depends heavily on its read concerns:

local read concern: It allows for reading data from the primary without confirming whether the obtained data has been acknowledged by the majority of servers. It is fast but can be reverted in case of a failover event.
majority read concern: It allows for obtaining data that is confirmed by more than half of the servers, which means that the obtained data can’t be reverted.
linearizable read concern: It means that the read operation requires the primary node to get responses from more than half of the nodes during the read operation to avoid stale data being read.
snapshot read concern: This read concern allows holding multiple documents consistent at the same time.

ACID Transactions in Multi-documents

One of the major features of MongoDB is its native single-document atomicity: every write operation made to a single document is atomic.

Starting in version 4.0 for replica sets and 4.2 for sharded clusters, MongoDB introduced Multi-document distributed ACID Transactions, enabling developers to perform distributed updates of multiple documents across different collections and shards of a database.

Atomicity: The entire set of operations performed is rolled back if any operation fails or succeeds for the given transaction.
Consistency: Data obtained within the transaction is read in a snapshot mode without the risk of dirty, non-repeatable, and phantom reads.
Isolation: Various transactions that are not confirmed are not known to outside transactions until they are finalised.
Durability – Once a transaction has been committed the data is saved in the database even if a failure occurs at the nodes.

Next Steps: Introduction to Current Database Technologies

MongoDB can support transactions from a single document to distributed transactions with ACID properties across shards. If you are getting ready for production or want to improve your engineering skills, taking a top-notch best mongodb online course is the perfect way to learn how to design great database solutions.

WEEKEND SPECIAL - UPTO 60% OFF