Sharepoint Tree Iteration & Performance

[sukamat]

In AEM, the "Promote" option in Floodgate copies the content from FG tree to the main tree. Something similar needs to be implemented for FG in Milo.

In order to copy the files from one tree to the other, I need to iterate the FG tree to grab the files first and then copy/overwrite them in the main tree.

From my discussions with the Sharepoint/MS support, there is no API that will iterate a given root folder. There is however a driveitem-list API that can list the immediate children at a given path.
https://learn.microsoft.com/en-us/graph/api/driveitem-list-children?view=graph-rest-1.0&tabs=http

In order to achieve what we need, i.e. to iterate the tree until the leaf node, I am using the driveitem-list API, finding the immediate children and then detecting the item type. If the item is a folder, I iterate further in that folder and so on until all folders are iterated.

Below is a snippet of iteration code for reference. Note that the below code is grabbing the path to the word doc blob. This can be further used to perform the file copy which it is currently not doing. Also we might need to grab all files eventually for copy and not just word docs.

let folders = [''];
let docs = [];

async function findAllDocxV2(sharePointBaseURI, options) {
  while (folders.length != 0) {
    const uri = `${sharePointBaseURI}${folders.shift()}:/children`;
    const res = await fetch(uri, options);
    if (res.ok) {
      const json = await res.json();
      const files = json.value;
      if (files) {
        for (let fileObj of files) {
          if (fileObj.folder) {
            // it is a folder
            const folderPath = fileObj.parentReference.path.replace('/drive/root:/milo', '') + '/' + fileObj.name;
            folders.push(folderPath);
          } else if (fileObj.file?.mimeType === 'application/vnd.openxmlformats-officedocument.wordprocessingml.document') {
            const downloadUrl = fileObj['@microsoft.graph.downloadUrl'];
            const docPath = fileObj.parentReference.path + '/' + fileObj.name;
            docs.push({ docDownloadUrl: downloadUrl, docPath: docPath });
          }
        };
      }
    }
  }

  return docs;
}

Here are some test results:

• Iterating /milo-pink tree with 219 files took ~3.5 seconds.

• Iterating /milo tree with ~2800 files took ~6 minutes

• Iterating /bacom tree with ~28k files took ~30 minutes

As seen from the above, the Sharepoint tree iteration is very slow. And this is just for the READ operation. The copy action (which will be WRITE operation) is limited to just couple hundred files before the API gets throttled.

I am open to any ideas to speed this up, perhaps a different API that you are aware of or a different way of iteration that might give us better outcome in terms of performance.

Thanks!