Home
Search
Golang Read All Files (WaitGroup, go Keyword)Read all files in a directory on threads with the go keyword and the WaitGroup type.
dot net perls
Files, WaitGroup. In Go programs we often want to open multiple files at once, on separate threads (goroutines). This can be done safely, provided we use a WaitGroup.
In some operating systems like macOS, a common error is reached when too many files are open. We must limit the number of simultaneous threads.
Example. The design of this code involves calling a func ReadFileSafely on multiple threads with the "go" keyword. We use a global chan to limit the number of threads.
Info To start we specify a directory containing the files we wish to open. Any number of files may be present in the folder.
ReadDir We read in all the files in the directory. We call Add() on the WaitGroup to ensure ReadFileSafely() is part of the group.
File
Defer In ReadFileSafely, we use defer and call Done() to indicate we are done with the func.
Tokens When we enter ReadFileSafely, we acquire a token, and when we leave, we release the token. This limits the number of open files.
Golang program that reads files in parallel
package main import ( "fmt" "os" "io/ioutil" "sync" ) // For limiting threads. var tokens = make(chan struct{}, 10) func ReadFileSafely(fileNameHere string, folder string, wg *sync.WaitGroup) { // For wait group (wait until all threads done). defer wg.Done() // Acquire token. tokens <- struct{}{} // Get full path. fullPath := folder + fileNameHere // Copy file. inputFile, err := os.Open(fullPath) if err != nil { panic(err) } data, err := ioutil.ReadAll(inputFile) if err != nil { panic(err) } // Close file. inputFile.Close() // Debug info. fmt.Println("FILE:", fullPath, len(data)) // Release token. <-tokens } func main() { // Read in all files in this folder. folder := "/Users/sam/test/" // For waiting on threads. var wg sync.WaitGroup // Get files in stage. dirRead, _ := os.Open(folder) dirFiles, _ := dirRead.Readdir(0) for dirIndex := range dirFiles { fileHere := dirFiles[dirIndex] fileNameHere := fileHere.Name() // Increment the WaitGroup counter. wg.Add(1) // Thread. go ReadFileSafely(fileNameHere, folder, &wg) } // Wait on threads to end. wg.Wait() }
FILE: /Users/sam/test/top.svg 308 FILE: /Users/sam/test/search.svg 306
Wait. We call Wait() on the WaitGroup to ensure all threads have finished before the program exits. This ensures every file is read in correctly.
Important We must be careful to call Close() on a file after we call Open() on it. This can reduce "too many files open" errors.
A summary. With the "go" keyword, each call to read in a file can be placed on a separate thread. This can vastly improve performance.
Home
© 2007-2021 sam allen. see site info on the changelog