Storage overhead of GridFS

Let’s have a look how MongoDB stores files with GridFS.

There are two collections for each GridFS bucket, one is bucket.fs.files and the other is bucket.fs.chunks.

While fs.files collection stores metadata about the file, fs.chunks collection stores the actual file data.

The file is divided into chunks and stored in the chunks collection. The default chunk size is 256KB. That means if a file size is more than 256KB than more than one chunk will be needed. The last chunk won’t be filled up to 256KB, that means if a file size is 10KB than only one chunk will be enough and it will hold only 10KB. MongoDB allocates files on the disk to persist the documents. The first file has 16 MB in size. Next file has double size of the previous. It gets bigger and bigger until it gets 2 GB in size. If the smallfiles parameter is used, the max file allocation size is 512 MB.

[root@gridfsserver db1]# ll
total 6321488
drwxr-xr-x. 2 root root      4096 Nov 18 02:54 journal
-rw-------. 1 root root  16777216 Nov 18 01:32 local.0
-rw-------. 1 root root  16777216 Nov 18 01:32 local.ns
-rwxr-xr-x. 1 root root         0 Nov 18 02:54 mongod.lock
-rw-------. 1 root root  16777216 Nov 18 02:52 mydb.0
-rw-------. 1 root root  33554432 Nov 18 02:52 mydb.1
-rw-------. 1 root root 536608768 Nov 18 02:25 mydb.10
-rw-------. 1 root root 536608768 Nov 18 02:30 mydb.11
-rw-------. 1 root root 536608768 Nov 18 02:40 mydb.12
-rw-------. 1 root root 536608768 Nov 18 02:47 mydb.13
-rw-------. 1 root root 536608768 Nov 18 02:52 mydb.14
-rw-------. 1 root root 536608768 Nov 18 02:47 mydb.15
-rw-------. 1 root root  67108864 Nov 18 02:52 mydb.2
-rw-------. 1 root root 134217728 Nov 18 02:51 mydb.3
-rw-------. 1 root root 268435456 Nov 18 02:52 mydb.4
-rw-------. 1 root root 536608768 Nov 18 01:59 mydb.5
-rw-------. 1 root root 536608768 Nov 18 02:52 mydb.6
-rw-------. 1 root root 536608768 Nov 18 02:52 mydb.7
-rw-------. 1 root root 536608768 Nov 18 02:52 mydb.8
-rw-------. 1 root root 536608768 Nov 18 02:01 mydb.9
-rw-------. 1 root root  16777216 Nov 18 02:52 mydb.ns
drwxr-xr-x. 2 root root      4096 Nov 18 02:47 _tmp
[root@gridfsserver db1]#

This is a typical fs.files document :

db.fs.files.findOne()
{
"_id" : ObjectId("528945ef4324fc1e4d863749"),
"chunkSize" : NumberLong(262144),
"length" : NumberLong(4096),
"md5" : "cce04522a1c82b93cf6e23b196754578",
"filename" : "tmp0000c73287606d8.rand",
"contentType" : null,
"uploadDate" : ISODate("2013-11-17T22:40:47.780Z"),
"aliases" : null
}

And this is a typical fs.chunks document for a file less than 256 KB, if it would have a size more than 256 KB than more than one chunk would be needed:

db.fs.chunks.findOne()
{
 "_id" : ObjectId("528945ef4324fc1e4d86374a"),
 "files_id" : ObjectId("528945ef4324fc1e4d863749"),
 "n" : 0,
 "data" : BinData(0,"kHIE2s7oot1C42...........shortened for brevity..............FlyQBcA==")
}

While the database is empty, it takes 32800 KB space on the disk. Note that mongod is started with the smallfiles option to limit the size of the allocated up to 512 MB :

[root@grifsserver db1 ]# /usr/local/mongodb/mongod --port 17017 --dbpath /data/db1/ --logpath /var/log/mongod.log --fork --smallfiles
about to fork child process, waiting until server is ready for connections.
forked process: 937
all output going to: /var/log/mongod.log
child process started successfully, parent exiting
[root@grifsserver db1 ]# du -xsk
32800    .
[root@grifsserver db1 ]#

I have generated 4 KB sized 10.000 files to insert into GridFS. Filenames and contents are randomly generated with a script.

This code is used to add files:

public void SaveFiles() throws Exception{
  MongoClient mongoClient = new MongoClient("gridfsserver:27017");
  DB db = mongoClient.getDB("mydb");
  GridFS g = new GridFS(db);

  File dir = new File("e:/generatedFiles");
  assert dir.isDirectory();
  File[] files = dir.listFiles();
  for (int i = 0; i < files.length; ++i) {
    GridFSInputFile gf = g.createFile(files[i]);
    gf.setFilename(files[i].getName());
    gf.save();
    db.getLastError().throwOnError();;
    if (i % 1000 == 0) {
      Long t1 = (new Date()).getTime();
      System.out.println((new Date())+". Saved " + i + " files in " + (t1 - t0) + "ms: " + (files.length * 1000 / (t1 - t0)) + " files/sec.");
    }
  }
  mongoClient.close();
}

After adding 10.000 files, checked the size again:

[root@grifsserver db1]# du -xsk
 339084 .
[root@grifsserver db1]#

So for the 10.000 files, sizes are:
before inserting files = 32800 KB,
inserted files = 40000 KB,
after inserting files = 339084.

To see the the difference more clear, I will try to add 512.000 files (2000 MB).

db.fs.files.count()
512000

Now the folder size is:

[root@grifsserver db1]# du -xsk
4460332 .
[root@grifsserver db1]#

So for the 512.000 files (2000 GB), sizes are:

  • before inserting files = 32800 KB,
  • inserted files = 2048000 KB,
  • after inserting files = 4460332.

Let’s calculate the storage overhead.
Before inserting 2048000 KB files, the storage size was 32800 KB. After inserting files, storage size is 4460332 KB.
There is a raise of 2379532 KB. So while we inserted 2048000 KB, there is additional 2379532 KB disk space is used. Wow! that’s more than 100% overhead! There is something wrong!

The problem here is that MongoDB pre allocates files to store JSON data in BSON. So you can’t really be sure how much each file is filled up.

In order to calculate the storage overhead correctly, we have to fill the last allocated file up to full size. So I come up with an idea to do it properly.

I have printed the file count and the time information in the java program and started to insert 512000 more files. In the same time, I checked the mongod log file to see when the new file is allocated.

So I would know when the new file is allocated and how many file would be put in the GridFS collection.

I came with these values:

Mon Nov 18 02:47:11.475 [FileAllocator] allocating new datafile /data/db1/mydb.15, filling with zeroes...
Mon Nov 18 02:47:11.485 [FileAllocator] done allocating datafile /data/db1/mydb.15, size: 511MB,  took 0.009 secs

Mon Nov 18 02:47:10 EET 2013. Saved 480000 files in 4774ms: 2094 files/sec.
Mon Nov 18 02:47:12 EET 2013. Saved 481000 files in 7014ms: 1425 files/sec.
Mon Nov 18 02:47:15 EET 2013. Saved 482000 files in 9624ms: 1039 files/sec.

So that means roughly 992000 files (512000+480000) saved before allocating the the last file.

Now as you can see the folder size is 6321492 KB:

[root@grifsserver db1]# du -xsk
6321492 .
[root@grifsserver db1]#

We should subtract the last file size from this result. The file size is 524032 KB.

In the end, we saved 992000 files and it took 5797460 KB (6321492-524032=5797460) size on the disk. Let’s don’t forget the initial size of the database which was 32800 KB. So we will en up 5764660 KB size change on the disk for the 992000 x 4 KB files.

The ultimate result:

  • We saved 3875 MB and spent 5629 MB disk size.
  • The actual file size saved in the GridFS is 68% of the spent disk size (3875/5629 = 68%).
  • Storage overhead for 3875 MB is 1754 MB (5629-3875).
  • Storage overhead in percentage is 45% (1754/3875).

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>