Example to compress/decompress

[hitecSmartHome]

Hi!

Great library. I wonder if you could create examples for compressing and decompressing nested folders.
Currently I want to make a backup manager class which would use your lib to compress folders ( multilple ) into a tar file which a user can download and upload later and the esp could decompress it into the fs.

So far it halts with error -6 which is an fs issue. Still wondering how to do this. ( this is an esp32 with psram )

[tobozo]

hi,

thanks for your feedback 👍

Do I understand you want to be able to compress a list directory entitties (files+folders) as .tar.gz?

This feature depends on the implementation of TarPacker, which isn't published yet, but should be ready soon.

... or maybe not so soon 😅 it depends how long it'll take to spot where that #@?!§% memory leak comes from, it's currently losing ~10KB on every run so I can't push the changes yet.

[hitecSmartHome]

Wow! I will check your example ASAP. Here is my implementation:

compressTask

void BackupHandler::compressTask(void *params) {
    BackupHandler *handler = static_cast<BackupHandler *>(params);

    ESP_LOGI(BACKUP_HANDLER_DEBUG_TAG, "Starting compression...");

    File backupFile = fileSys.open(BACKUP_FILE_PATH, FILE_WRITE);
    if (!backupFile) {
        ESP_LOGE(BACKUP_HANDLER_DEBUG_TAG, "Failed to create backup file");
        handler->inProgress = false;
        vTaskDelete(nullptr);
        return;
    }

    addDirectoryToTar("/", backupFile);

    backupFile.close();

    ESP_LOGI(BACKUP_HANDLER_DEBUG_TAG, "Compression completed successfully");

    handler->inProgress = false;
    vTaskDelete(nullptr);
}

decompressTask

void BackupHandler::decompressTask(void *params) {
    BackupHandler *handler = static_cast<BackupHandler *>(params);

    ESP_LOGI(BACKUP_HANDLER_DEBUG_TAG, "Starting decompression...");

    File backupFile = fileSys.open(BACKUP_FILE_PATH, FILE_READ);
    if (!backupFile) {
        ESP_LOGE(BACKUP_HANDLER_DEBUG_TAG, "Failed to open backup file");
        handler->inProgress = false;
        vTaskDelete(nullptr);
        return;
    }

    TarGzUnpacker tarGzUnpacker;
    if (tarGzUnpacker.tarExpander(LittleFS, BACKUP_FILE_PATH, LittleFS, "/")) {
        ESP_LOGI(BACKUP_HANDLER_DEBUG_TAG, "Decompression completed successfully");
    } else {
        ESP_LOGE(BACKUP_HANDLER_DEBUG_TAG, "Decompression failed");
    }

    backupFile.close();
    handler->inProgress = false;
    vTaskDelete(nullptr);
}

utils

bool BackupHandler::shouldExclude(const char *path) {
    for (const char **excludedPath = excludedPaths; *excludedPath != nullptr; excludedPath++) {
        if (strcmp(path, *excludedPath) == 0) {
            return true;
        }
    }
    return false;
}

void BackupHandler::addFileToTar(File &file, Stream &tarStream) {
    if (!file || file.isDirectory()) {
        return;
    }

    ESP_LOGI(BACKUP_HANDLER_DEBUG_TAG, "Adding file: %s", file.path());

    writeTarHeader(tarStream, file.path(), file.size(), '0');

    uint8_t buffer[512] = {0};
    size_t bytesRead;
    while ((bytesRead = file.read(buffer, sizeof(buffer))) > 0) {
        tarStream.write(buffer, bytesRead);
    }

    // Pad to 512-byte boundary
    size_t padding = 512 - (file.size() % 512);
    if (padding < 512) {
        memset(buffer, 0, padding);
        tarStream.write(buffer, padding);
    }
}

void BackupHandler::addDirectoryToTar(const char *path, Stream &tarStream) {
    if (shouldExclude(path)) {
        return;
    }

    ESP_LOGI(BACKUP_HANDLER_DEBUG_TAG, "Adding directory: %s", path);

    // Add the directory entry to the TAR
    writeTarHeader(tarStream, path, 0, '5'); // Type '5' indicates a directory

    File dir = fileSys.open(path);
    if (!dir || !dir.isDirectory()) {
        return;
    }

    File entry;
    while ((entry = dir.openNextFile())) {
        if (entry.isDirectory()) {
            addDirectoryToTar(entry.path(), tarStream);
        } else {
            addFileToTar(entry, tarStream);
        }
        entry.close();
    }
    dir.close();
}

void BackupHandler::writeTarHeader(Stream &tarStream, const char *filename, size_t filesize, char type) {
    char header[512] = {0};

    // Ensure filename fits within 100 characters and ends with '/' for directories
    char formattedFilename[101] = {0};
    strncpy(formattedFilename, filename, 100);
    if (type == '5' && formattedFilename[strlen(formattedFilename) - 1] != '/') {
        strncat(formattedFilename, "/", 100 - strlen(formattedFilename));
    }

    // Write the filename
    strncpy(header, formattedFilename, 100);

    // File mode
    snprintf(header + 100, 8, "%07o", type == '5' ? 0755 : 0644);

    // Owner and group IDs
    snprintf(header + 108, 8, "%07o", 0);
    snprintf(header + 116, 8, "%07o", 0);

    // File size
    snprintf(header + 124, 12, "%011o", static_cast<unsigned int>(filesize));

    // Modification time will come from the actual time from the file later...
    snprintf(header + 136, 12, "%011o", static_cast<unsigned int>(time(nullptr)));

    // Set checksum field to spaces initially
    memset(header + 148, ' ', 8);

    // File type
    header[156] = type;

    // Compute checksum
    unsigned int checksum = 0;
    for (int i = 0; i < 512; ++i) {
        checksum += static_cast<unsigned char>(header[i]);
    }
    snprintf(header + 148, 8, "%06o", checksum);

    // Write the header to the stream
    tarStream.write((uint8_t *)header, 512);
}

I must say that it is incomplete and untested so far. I just jumped in to brainstorm

[hitecSmartHome]

If I understand it correctly, this whole thing which I have created can be replaced by this line TarGzPacker::compress(&tarGzFS, src_path, targz_path, dst_path );

[hitecSmartHome]

It would also be good to pass a filter callback or a list of paths which would be excluded from the compress. There can be sensitive files and folders in the file system. Also I excluded the already created tar if it has one ( so it does not compress itself ). I made a little mistake also where the "/" default path is also represented in the tar so it is also a bug in my implementation

[hitecSmartHome]

I'm thinking on something like this

bool filterCB(const char* path){
    return strcmp(path, "/logs") == 0);
}

size_t dstLen = TarGzPacker::compress(&tarGzFS, src_path, targz_path, dst_path, filterCB );

Or the user can keep a list of paths to exclude

const char *excludedPaths[] = {
    "/backup.tar.gz",
    "/logs",
    nullptr // Marks the end of the array
};

// Return true if the path should be excluded
bool filterCB(const char* path){
    for (const char **excludedPath = excludedPaths; *excludedPath != nullptr; excludedPath++) {
        if (strcmp(path, *excludedPath) == 0) {
            return true;
        }
    }
    return false;
}

size_t dstLen = TarGzPacker::compress(&tarGzFS, src_path, targz_path, dst_path, filterCB );

[tobozo]

the task-in/task-out is very elegant, it was my first choice but I lost myself into semaphores and queues, eventually gave up with a headache 🤣

I'm not sure this would work on all supported architectures though, RP2040 and ESP8266 have a slightly different flavour of FreeRTOS tasks, a bit like fs::FS with file.name(). Maintaining conditional macro blocks is a permanent tech debt I'd rather avoid whenever possible.

ESP32-targz/src/libpacker/common.hpp

Lines 130 to 138 in 5d0dd5a

	const char* file_path =
	#if defined ESP32
	file.path()
	#elif defined ESP8266 || defined ARDUINO_ARCH_RP2040
	file.fullName()
	#else
	#error
	#endif
	;

If I understand it correctly, this whole thing which I have created can be replaced by this line TarGzPacker::compress(&tarGzFS, src_path, targz_path, dst_path );

Only if you want to compress a whole folder.
That function performs no filtering except for the tar/gz output file itself.

It would also be good to pass a filter callback

This need is already satisfied with TarGzStream class taking a vector as second argument.

You can apply your filters on that vector before feeding it to the the class, or you can create your own vector and compose:

  struct dir_entity_t
  {
    String path{""};
    bool is_dir{false};
    size_t size{0};
  };

  std::vector<dir_entity_t> directoryEntities;

Adding your own directory entities:

   directoryEntities.push_back{"/", true, 0}; // first element is always root directory
   directoryEntities.push_back{"/my/folder1", true, 0}; 
   directoryEntities.push_back{"/my/folder1/blah.txt", false, 12345}; 
   // ....

Creating the TarGzStream object:

  TarGzStream myTarGzStream( &tarGzFS, &dirEntities, src_path, targz_path, dst_path);

Triggering compression:

  File tarGzOutput = tarGzFS.open(targz_path, "w"); 
  size_t dstLen = TarGzPacker::compress( &myTarGzStream, &tarGzOutput );
  tarGzOutput.close();

[hitecSmartHome]

I'm not sure this would work on all supported architectures though, RP2040 and ESP8266 have a slightly different flavour of FreeRTOS tasks, a bit like fs::FS with file.name(). Maintaining conditional macro blocks is a permanent tech debt I'd rather avoid whenever possible.

Yeah, I understand. The task thing should be left to the user of the library.

You can apply your filters on that vector before feeding it to the the class, or you can create your own vector and compose:

I will try this ASAP. Thank you very much. Will make a feedback about this!

[hitecSmartHome]

TARGZUnpacker->setPsram( true ); is not changing the game.
Also I have created the TARGZUnpacker on init, and calling it in the task but this has also no effect. ( so I have created it outside of the task ). Still crash

[tobozo]

I'm not sure what we're debugging here, is that the compression, the decompression, FreeRTOS tasks?

Do the examples at least work for you?

btw I can't decode your stack traces since I don't have the source code of the application, so maybe it's time to share some basic project so I can actually reproduce the issue.

It would also be good if we could pass an std::function to the callbacks so we could pass or bind class methods to it like this

I don't know well enough this capture thing to use it with confidence, but feel free to submit a PR 😉

[hitecSmartHome]

Its the decompression. Iam sure the problem is not the task, nor the ram or the fs space. You dont have to decode, I did it.

• I have 1.8mb free space on my fs.
• 108kb free ram
• 60kb free contiguous ram
• tried even with 32k stack for the task
• 3mb free psram

It is always crashing when I call decompression. I will create a sketch. I don't know where I screw it up. :/

[hitecSmartHome]

Sometimes I got an error from my system E (157604) System: heap_caps_malloc was called but failed to allocate 32000 bytes with 0x2052 capabilities. but the TARGZUnpacker is configured to use psram and it's just silently fails.

[tobozo]

well the ::noDict() function should take care of not allocating the 32KB for the dictionary, and use the checksum-based slower decompression instead