Run Dotnet Tasks Concurrently

There are times when programming where you will need to do large batches of things very quickly. Dotnet Tasks using the WhenAll function are a great way of doing this. If you run Dotnet tasks concurrently, rather than sequentially, you can often reduce the time it takes to do a large number of tasks by a pretty large factor.

When to Use Concurrent Tasks

The best time to use concurrent tasks is when:

• The things you need to do are not dependent on each other.
• You have a lot of things to do and/or
• Each of the things could take a long time.

For example, I was recently tasked with improving the speed of an application at work that runs nightly and downloads large media files from an API. There would usually be less than 100 files, but they would range in size from a few MB to 10+GB. Due to how much database and API querying the application was doing, along with the downloads taking quite a while due to bandwidth issues, the application would sometimes run for several hours. This was a perfect candidate for running multiple downloads concurrently.

Another quick example is we had to run a quick check to verify files existed on a network folder. Checking the existence of the file was quick, but we were verifying a couple hundred thousand files. Concurrency to the rescue.

Concurrency Examples

This is a really quick example of how affective running tasks in parallel is. Let's say you have to do 10 tasks that each take 1 second. If you do them sequentially, it will take a little over 10 seconds. However if you do them concurrently it will only take a little over 1 second.

If you would like the quick and dirty (and ugly) code I used to test this, here is a gist.

This is an example of the sequential way in .Net:

for (int i = 0; i < 10; i++)
{
    await Task.Run(() => Thread.Sleep(1000));
}

Running this code with a Stopwatch timing it took just over 10 seconds. Compare that to running them in parallel using Task.WhenAll:

List<Task> tasks = new List<Task>();
for (int i = 0; i < 10; i++)
{
    tasks.Add(Task.Run(() => Thread.Sleep(1000)));
}
await Task.WhenAll(tasks);

This only took 1 second!

Side by side comparison of sequential vs concurrent tasks.

Task.WhenAll will let all tasks passed to it run. Once all of them have completed, it will return. This is great for heavily IO bound work, such as database queries, API calls, and file system reads/writes.

One thing to keep in mind is that all of the tasks can't depend on each other. An example of when running concurrent tasks would not work is a process where each step depends on information from the previous step. A made up (and probably bad) example: You query a database for a record. You then use the returned records ID to call some other API to get more data. Then you take the results and save them to a file. It's a bad example because it's not very many steps, but I think you get the idea.

Conclusion

As developers we often need to write apps that make a lot of queries or check for a lot of files. If time is of the essence, consider running the tasks concurrently using .Net Tasks and using the WhenAll function.