Batch A Channel By Size Or Time in Go

Photo by Perry Grone on Unsplash

There are some cases where you need to read enough items from a channel to begin processing. However, you also want to a TTL to expire with a smaller batch if there are not more items for a while.

Here’s an example of streaming a channel of string into an output channel of []string that guarantees:

  1. Each batch contains at least one string.
  2. A batch will never contain more than maxItems.
  3. If there is at least one string to be read at any given time, it will always send a new batch within maxTimeout .
  4. A batch will contain less than maxItems if maxTimeout has occurred since the the last batch finished (not since the first item in the batch). This is important because it means that if the stream is scarce and infrequent — such as one item per maxTimeoutBatchStrings will prefer to send a consistently timed batch of one item rather than more volatile intervals that might contain two items.

Try it now:

The above example will halt when the channel is closed. It can also be written in a way that is context aware:

Now the stream can be stopped by either cancelling the context or closing the channel.

Try it now:

I’m a data nerd and TDD enthusiast originally from Sydney. Currently working for Uber in New York. My thoughts here are my own. 🤓

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store