1

So this answer got me most of what I needed:

https://stackoverflow.com/a/49433633

Basically what I'm trying to do is pre-allocate disk space for a set of files in their given directories. I'm ensuring that each directory and sub directory is created recursively, and I do actually have files being saved to disk that appear to be of the right size. The problem I'm seeing is that while the file created is 100GB, it's only taking up 4KB on the actual disk. Is there something I can do so it actually occupies the full space? For reference, here's the method I've created to pre-allocate disk space for a file:

export const preallocateFile = (rootPath: string, file: File): Promise<boolean | Error> => {
  const { Filelocation: fileLocation, FileName: fileName, FileSize: size } = file
  return new Promise((resolve, reject) => {
    console.log('allocate!', rootPath, file)
    // process asynchronously
    setTimeout(() => {
      try {
        // Create the necessary directories ahead of writing the file
        fs.mkdirSync(`${rootPath}${fileLocation}`, { recursive: true })
        // Open the file for writing; 'w' creates the file
        // (if it doesn't exist) or truncates it (if it exists)
        const allocatedFile = fs.openSync(`${rootPath}${fileLocation}${fileName}`, 'w')
        if (size > 0) {
          // Write one byte (with code 0) at the desired offset
          // This forces the expanding of the file and fills the gap
          // with characters with code 0
          fs.writeSync(allocatedFile, Buffer.alloc(1), 0, 1, size - 1)
        }
        // Close the file to commit the changes to the file system
        fs.closeSync(allocatedFile)

        console.log('about to resolve!')
        resolve(true)
      } catch (error) {
        console.log('nope!', error)
        reject(error)
      }
      // Create the file after the processing of the current JavaScript event loop
    }, 0)
  })
}

While my code is working, if I'm not actually occupying disk space then the pre-allocation will not work properly.

tbourne89
  • 77
  • 5

1 Answers1

2

First off, the reason you are seeing only 4KB allocated is due to file system design, there is a lot to talk about, you can read more here.

In short, in newer file systems like ext4 and NTFS, as you directly write to the last block, the file system will only allocate 1 harddisk block (4KB) at some position X on disk, then record in file table "3.999GB offset from file begin, the block of data is at position X", and when reading the file, the file system will intelligently "expand" the first 3.999GB as 0 because there is no data record and is never written. This design allows the file system to save space in cases like you are doing.

So to circumvent this optimization, you should writeSync a bunch of 0 over the whole size, instead of just writing the last byte.

Sidenote: If you are on Linux only, you might want to checkout the dd command.

Eric Wong
  • 1,388
  • 7
  • 17
  • I ended up changing the function so that the full size is broken up into chunks, and each chunk is appended to the file with `Buffer.alloc(chunk)`. This gives me the full disk space allocation that I'm looking for. Thanks for the detailed explanation of what's happening on the file system, turns out that's really the information that I was looking for! – tbourne89 Oct 02 '20 at 17:28