How to write an asynchronous recursive walkdir function with an asynchronous callback

Question

I'm trying to write an async function that will traverse the filesystem tree, recursively, and calls an asynchronous callback for each file found. This is for a learning effort, I have no real use case.

Here is what I have so far:

use async_std::{
    fs::{self, *},
    path::*,
    prelude::*,
}; // 1.5.0, features = ["unstable"]
use futures::{
    executor::block_on,
    future::{BoxFuture, FutureExt},
}; // 0.3.4
use std::{marker::Sync, pin::Pin};

fn main() {
    fn walkdir<F>(path: String, cb: &'static F) -> BoxFuture<'static, ()>
    where
        F: Fn(&DirEntry) -> BoxFuture<()> + Sync + Send,
    {
        async move {
            let mut entries = fs::read_dir(&path).await.unwrap();
            while let Some(path) = entries.next().await {
                let entry = path.unwrap();
                let path = entry.path().to_str().unwrap().to_string();
                if entry.path().is_file().await {
                    cb(&entry).await
                } else {
                    walkdir(path, cb).await
                }
            }
        }
        .boxed()
    }

    let foo = async {
        walkdir(".".to_string(), &|entry: &DirEntry| async {
            async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
        })
        .await
    };

    block_on(foo);
}

I get this far by some sort of trial and error, but now I'm stuck on async closure callback with this error

warning: unused import: `path::*`
 --> src/main.rs:3:5
  |
3 |     path::*,
  |     ^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused import: `pin::Pin`
  --> src/main.rs:10:25
   |
10 | use std::{marker::Sync, pin::Pin};
   |                         ^^^^^^^^

error[E0308]: mismatched types
  --> src/main.rs:33:54
   |
33 |           walkdir(".".to_string(), &|entry: &DirEntry| async {
   |  ______________________________________________________^
34 | |             async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
35 | |         })
   | |_________^ expected struct `std::pin::Pin`, found opaque type
   |
   = note:   expected struct `std::pin::Pin<std::boxed::Box<dyn core::future::future::Future<Output = ()> + std::marker::Send>>`
           found opaque type `impl core::future::future::Future`

Shepmaster · Accepted Answer · 2020-02-18T19:02:22.987

use async_std::{
    fs::{self, *},
    path::*,
    prelude::*,
}; // 1.5.0
use futures::{future::{Future, FutureExt, LocalBoxFuture}, executor}; // 0.3.4

fn main() {
    async fn walkdir<R>(path: impl AsRef<Path>, mut cb: impl FnMut(DirEntry) -> R)
    where
        R: Future<Output = ()>,
    {
        fn walkdir_inner<'a, R>(path: &'a Path, cb: &'a mut dyn FnMut(DirEntry) -> R) -> LocalBoxFuture<'a, ()>
        where
            R: Future<Output = ()>,
        {
            async move {
                let mut entries = fs::read_dir(path).await.unwrap();

                while let Some(path) = entries.next().await {
                    let entry = path.unwrap();
                    let path = entry.path();
                    if path.is_file().await {
                        cb(entry).await
                    } else {
                        walkdir_inner(&path, cb).await
                    }
                }
            }.boxed_local()
        }

        walkdir_inner(path.as_ref(), &mut cb).await
    }

    executor::block_on({
        walkdir(".", |entry| async move {
            async_std::println!(">> {}", entry.path().display()).await
        })
    });
}

Notable changes:

Take in AsRef<Path> instead of a String and a generic closure instead of a trait object reference
Change the closure type to be FnMut as it's more permissive
The closure returns any type that is a future.
There's an inner implementation function that hides the ugly API required for recursive async functions.
The callback takes the DirEntry by value instead of by reference.

How to write an asynchronous recursive walkdir function with an asynchronous callback

1 Answers1