The 100GB Illusion :
A 100 gb file on a machine with just 16 gb ram sounds impossible to work with. You can still read a byte from the middle though using a simple c++ thing and it only takes one instruction really. The file is way bigger than memory so how does that pointer even work without running out of space. It has to do with mmap which a lot of programs use like elasticsearch and kafka and some others.
Virtual Addresses and Empty Promises:
Pointers are not pointing to real physical memory directly. They use virtual addresses that the kernel manages with page tables. When you map a big file the kernel just sets up a small record for it and gives you an address that is basically empty at first. I think this is the part that feels weird at first because nothing is actually loaded yet.
Page Faults:
Then when you try to read from it a page fault happens and the kernel loads only the needed 4kb page from disk if its not already in the cache. That way you only use ram for the parts you actually touch which makes sense for large files. Some pages might stay unloaded for a while if you never reach them.
Shared Memory and Forking:
mmap also lets processes share memory easily with the same physical pages. Or with copy-on-write it can make forking cheap because changes only copy the page being written. It seems like this is used in more places than people realize.
The Database Problem:
But there are issues if you use it for databases. You lose control over when things get written to disk and errors can crash your program instead of giving nice messages. That is probably why some databases moved away from it. The OS decides the timing and that can mess with strict ordering needs.
inspired by Youtube video : mmap Shouldn't Be Possible