Files, WaitGroup

In Go programs we often want to open multiple files at once, on separate threads (goroutines). This can be done safely, provided we use a WaitGroup.

In some operating systems like macOS, a common error is reached when too many files are open. We must limit the number of simultaneous threads.

Example

The design of this code involves calling a func ReadFileSafely on multiple threads with the "go" keyword. We use a global chan to limit the number of threads.

package main

import (
    "fmt"
    "os"
    "io/ioutil"
    "sync"
)

// For limiting threads.
var tokens = make(chan struct{}, 10)

func ReadFileSafely(fileNameHere string, folder string, wg *sync.WaitGroup) {

    // For wait group (wait until all threads done).
    defer wg.Done()

    // Acquire token.
    tokens <- struct{}{}

    // Get full path.
    fullPath := folder + fileNameHere

    // Copy file.
    inputFile, err := os.Open(fullPath)
    if err != nil {
        panic(err)
    }
    data, err := ioutil.ReadAll(inputFile)
    if err != nil {
        panic(err)
    }
    // Close file.
    inputFile.Close()

    // Debug info.
    fmt.Println("FILE:", fullPath, len(data))

    // Release token.
    <-tokens
}

func main() {

    // Read in all files in this folder.
    folder := "/Users/sam/test/"

    // For waiting on threads.
    var wg sync.WaitGroup

    // Get files in stage.
    dirRead, _ := os.Open(folder)
    dirFiles, _ := dirRead.Readdir(0)
    for dirIndex := range dirFiles {
        fileHere := dirFiles[dirIndex]
        fileNameHere := fileHere.Name()

        // Increment the WaitGroup counter.
        wg.Add(1)

        // Thread.
        go ReadFileSafely(fileNameHere, folder, &wg)
    }

    // Wait on threads to end.
    wg.Wait()
}
FILE: /Users/sam/test/top.svg 308
FILE: /Users/sam/test/search.svg 306

Wait

We call Wait() on the WaitGroup to ensure all threads have finished before the program exits. This ensures every file is read in correctly.

With the "go" keyword, each call to read in a file can be placed on a separate thread. This can vastly improve performance.