Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE] OV file utils file_size use std file path #28462

Open
wants to merge 44 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
887dd2b
move wide character string conversions to separate file
barnasm1 Jan 15, 2025
04671c7
refactor
barnasm1 Jan 15, 2025
b334528
refactor file size functions to use ov::util::Path and handle Android…
barnasm1 Jan 15, 2025
c0ec93e
add unit tests for cut_android_path function
barnasm1 Jan 15, 2025
dce5d59
add unit tests for file size functions with various file types and en…
barnasm1 Jan 15, 2025
701cd96
code format
barnasm1 Jan 15, 2025
97f2a96
prevent variable redefinition for android
barnasm1 Jan 15, 2025
c0345b1
refactor test to use ov::util::Path for Unicode file output
barnasm1 Jan 15, 2025
66615dc
Merge branch 'master' into ov_file_path_util_file_size
mlukasze Jan 16, 2025
a1e8053
add Windows-specific includes for wstring_cast_util
barnasm1 Jan 16, 2025
f34b2d2
fix comment typo
barnasm1 Jan 16, 2025
2d071bf
refactor cpp_version.hpp and file_path.hpp for compiler version checks
barnasm1 Jan 16, 2025
3b2fbae
refactor WPath function to use lambda syntax, to meet Naming style check
barnasm1 Jan 16, 2025
eed127e
update copyright year to 2025, fix typos,
barnasm1 Jan 16, 2025
067224b
rename wstring_cast_util to wstring_convert_util and update includes
barnasm1 Jan 16, 2025
22d6712
add Android-specific file size test
barnasm1 Jan 16, 2025
2e7cf52
fix typo in CMakeLists.txt for wstring_convert_util source file path
barnasm1 Jan 16, 2025
e9c2cca
deprecate old file_size function and provide alternative usage guidance
barnasm1 Jan 16, 2025
71e2206
update file size tests to use string literals
barnasm1 Jan 16, 2025
c8bdb3b
skip file size tests based on C++ version
barnasm1 Jan 16, 2025
dd13c04
revert: refactor WPath function to use lambda syntax, to meet Naming …
barnasm1 Jan 16, 2025
74640d8
Merge branch 'master' into ov_file_path_util_file_size
barnasm1 Jan 21, 2025
0f2aa4a
clang format
barnasm1 Jan 22, 2025
76d0a49
Merge branch 'master' into ov_file_path_util_file_size
barnasm1 Jan 23, 2025
f9cd47d
Merge branch 'master' into ov_file_path_util_file_size
barnasm1 Jan 27, 2025
0794bcd
Refactor namespace declaration
barnasm1 Jan 27, 2025
8d745c5
Replace file size calculation with std::filesystem
barnasm1 Jan 27, 2025
a82aef0
Refactor cut_android_path function for improved readability
barnasm1 Jan 27, 2025
77dd50f
Remove deprecated file_size function
barnasm1 Jan 27, 2025
d45867d
Update file_size function to use ov::util namespace for consistency
barnasm1 Jan 27, 2025
8837c2e
Refactor file path handling and update file size functions
barnasm1 Jan 28, 2025
d9a9487
Revert "Remove deprecated file_size function"
barnasm1 Jan 28, 2025
6c8b9a5
Fix preprocessor directive formatting for GCC and Clang version checks
barnasm1 Jan 28, 2025
8acae29
Fix file_size function to handle Android paths correctly
barnasm1 Jan 28, 2025
b45c26b
Update file size tests to use string literals to avoid deprecation wa…
barnasm1 Jan 28, 2025
65cc6d5
Add Unicode content tests for file size functionality
barnasm1 Jan 28, 2025
bab5a77
Add test for file size calculation of raw byte files
barnasm1 Jan 29, 2025
b8cf8dc
Remove Unicode content tests and related setup from FileUtilTest
barnasm1 Jan 29, 2025
f4cb551
cut_android_path function restrict
barnasm1 Jan 29, 2025
feab0bd
check if cut_android_path is required at new impl
barnasm1 Jan 29, 2025
07f5a1b
Refactor file size and clean up unused includes
barnasm1 Jan 29, 2025
3c90e81
Update file_size function to accept std::wstring and adjust related t…
barnasm1 Jan 29, 2025
664b578
Remove deprecated file_size function for std::wstring and update path…
barnasm1 Jan 29, 2025
3227347
Simplify file_size function implementation
barnasm1 Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/common/util/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ endif()

set_source_files_properties(
"${CMAKE_CURRENT_SOURCE_DIR}/src/file_util.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/src/wstring_convert_util.cpp"
PROPERTIES COMPILE_DEFINITIONS OpenVINO_VERSION="${OpenVINO_VERSION}")

source_group("src" FILES ${LIBRARY_SRC})
Expand Down
34 changes: 30 additions & 4 deletions src/common/util/include/openvino/util/file_path.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,14 @@
#include <cstdio>

#include "openvino/util/filesystem.hpp"
namespace ov {
namespace util {

namespace ov::util {

#if defined(OPENVINO_HAS_FILESYSTEM)
// There are known issues with usage of std::filesystem::path unicode represenataion:
// * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95048
// * https://stackoverflow.com/questions/58521857/cross-platform-way-to-handle-stdstring-stdwstring-with-stdfilesystempath
// Working compiler versions has been designated with godbolt.
using Path = std::filesystem::path;
#elif defined(OPENVINO_HAS_EXP_FILESYSTEM)
// Known issues:
Expand All @@ -30,5 +34,27 @@ using Path = std::filesystem::path;
using Path = std::experimental::filesystem::path;
#endif

} // namespace util
} // namespace ov
#if !defined(__GNUC__) || (__GNUC__ > 12 || __GNUC__ == 12 && __GNUC_MINOR__ >= 3)
# define GCC_NOT_USED_OR_VER_AT_LEAST_12_3
#endif

#if !defined(__clang__) || defined(__clang__) && __clang_major__ >= 17
# define CLANG_NOT_USED_OR_VER_AT_LEAST_17
#endif

#if defined(__GNUC__) && (__GNUC__ < 12 || __GNUC__ == 12 && __GNUC_MINOR__ < 3)
# define GCC_VER_LESS_THAN_12_3
#endif

#if defined(__clang__) && __clang_major__ < 17
# define CLANG_VER_LESS_THAN_17
#endif

} // namespace ov::util

#if defined(GCC_VER_LESS_THAN_12_3) || defined(CLANG_VER_LESS_THAN_17)

template <>
ov::util::Path::path(const std::wstring& source, ov::util::Path::format fmt);
barnasm1 marked this conversation as resolved.
Show resolved Hide resolved

#endif
61 changes: 29 additions & 32 deletions src/common/util/include/openvino/util/file_util.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
#include <string>
#include <vector>

#include "openvino/util/file_path.hpp"
#include "openvino/util/util.hpp"
#include "openvino/util/wstring_convert_util.hpp"

namespace ov {
namespace util {
Expand Down Expand Up @@ -89,19 +91,6 @@ const std::string& path_to_string(const Path& path) {
}

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
/**
* @brief Conversion from wide character string to a single-byte chain.
* @param wstr A wide-char string
* @return A multi-byte string
*/
std::string wstring_to_string(const std::wstring& wstr);
/**
* @brief Conversion from single-byte chain to wide character string.
* @param str A null-terminated string
* @return A wide-char string
*/
std::wstring string_to_wstring(const std::string& str);

/**
* @brief Convert path as wide character string to a single-byte chain.
* @param path Path as wide-char string.
Expand Down Expand Up @@ -172,12 +161,26 @@ bool directory_exists(const std::string& path);
bool directory_exists(const std::wstring& path);
#endif

#if defined(__ANDROID__) || defined(ANDROID)
inline ov::util::Path cut_android_path(const ov::util::Path& file_name) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is not required can be still internal part of file_size

const auto& file_name_native = file_name.native();

if (const auto pos = file_name_native.find('!'); pos != std::decay_t<decltype(file_name_native)>::npos) {
return ov::util::Path(file_name_native.substr(0, pos));
} else {
return file_name;
}
}
#endif

/**
* @brief Returns file size for file
* @param[in] path The file name
* @return file size
*/
inline int64_t file_size(const char* path) {
barnasm1 marked this conversation as resolved.
Show resolved Hide resolved
[[deprecated("This function is deprecated use file_size(const ov::util::Path& path) instead. Will be removed in "
"2026.0")]] inline int64_t
file_size(const char* path) {
#if defined(OPENVINO_ENABLE_UNICODE_PATH_SUPPORT) && defined(_WIN32)
std::wstring widefilename = ov::util::string_to_wstring(path);
const wchar_t* file_name = widefilename.c_str();
Expand All @@ -194,6 +197,12 @@ inline int64_t file_size(const char* path) {
return in.tellg();
}

inline int64_t file_size(const ov::util::Path& path) {
std::error_code ec;
const auto size = std::filesystem::file_size(path, ec);
return ec ? -1 : static_cast<int64_t>(size);
}

/**
* @brief Returns file size for file
* @param[in] path The file name
Expand All @@ -218,15 +227,6 @@ inline bool file_exists(const char* path) {

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT

/**
* @brief Returns file size for file
* @param[in] path The file name
* @return file size
*/
inline int64_t file_size(const std::wstring& path) {
return file_size(wstring_to_string(path).c_str());
}

/**
* @brief Returns true if file exists
* @param[in] path The file name
Expand All @@ -237,15 +237,6 @@ inline bool file_exists(const std::wstring& path) {
}
#endif // OPENVINO_ENABLE_UNICODE_PATH_SUPPORT

/**
* @brief Returns file size for file
* @param[in] path The file name
* @return file size
*/
inline int64_t file_size(const std::string& path) {
barnasm1 marked this conversation as resolved.
Show resolved Hide resolved
return file_size(path.c_str());
}

/**
* @brief Returns true if file exists
* @param[in] path The file name
Expand Down Expand Up @@ -397,5 +388,11 @@ inline std::basic_string<C> make_path(const std::basic_string<C>& folder, const
return folder + ov::util::FileTraits<C>::file_separator + file;
}

#if defined(_MSC_VER) || defined(GCC_VER_LESS_THAN_12_3) || defined(CLANG_VER_LESS_THAN_17)
inline ov::util::Path make_path(const wchar_t* file) {
return {std::wstring(file)};
}
#endif

} // namespace util
} // namespace ov
29 changes: 29 additions & 0 deletions src/common/util/include/openvino/util/wstring_convert_util.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
// Copyright (C) 2018-2025 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//

#pragma once

#include <string>

#include "openvino/util/util.hpp"

namespace ov::util {

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
/**
* @brief Conversion from wide character string to a single-byte chain.
* @param wstr A wide-char string
* @return A multi-byte string
*/
std::string wstring_to_string(const std::wstring& wstr);
/**
* @brief Conversion from single-byte chain to wide character string.
* @param str A null-terminated string
* @return A wide-char string
*/
std::wstring string_to_wstring(const std::string& str);

#endif

} // namespace ov::util
15 changes: 15 additions & 0 deletions src/common/util/src/file_path.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Copyright (C) 2018-2025 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//

#include "openvino/util/file_path.hpp"

#include "openvino/util/wstring_convert_util.hpp"

#if defined(GCC_VER_LESS_THAN_12_3) || defined(CLANG_VER_LESS_THAN_17)

template <>
ov::util::Path::path(const std::wstring& source, ov::util::Path::format fmt)
: path(ov::util::wstring_to_string(source), fmt) {}

#endif
40 changes: 0 additions & 40 deletions src/common/util/src/file_util.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -338,46 +338,6 @@ void ov::util::convert_path_win_style(std::string& path) {
std::replace(path.begin(), path.end(), '/', '\\');
}

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT

# ifdef __clang__
# pragma clang diagnostic push
# pragma clang diagnostic ignored "-Wdeprecated-declarations"
# endif

std::string ov::util::wstring_to_string(const std::wstring& wstr) {
# ifdef _WIN32
int size_needed = WideCharToMultiByte(CP_UTF8, 0, &wstr[0], (int)wstr.size(), NULL, 0, NULL, NULL);
std::string strTo(size_needed, 0);
WideCharToMultiByte(CP_UTF8, 0, &wstr[0], (int)wstr.size(), &strTo[0], size_needed, NULL, NULL);
return strTo;
# else
std::wstring_convert<std::codecvt_utf8<wchar_t>> wstring_decoder;
return wstring_decoder.to_bytes(wstr);
# endif
}

std::wstring ov::util::string_to_wstring(const std::string& string) {
const char* str = string.c_str();
# ifdef _WIN32
int strSize = static_cast<int>(std::strlen(str));
int size_needed = MultiByteToWideChar(CP_UTF8, 0, str, strSize, NULL, 0);
std::wstring wstrTo(size_needed, 0);
MultiByteToWideChar(CP_UTF8, 0, str, strSize, &wstrTo[0], size_needed);
return wstrTo;
# else
std::wstring_convert<std::codecvt_utf8<wchar_t>> wstring_encoder;
std::wstring result = wstring_encoder.from_bytes(str);
return result;
# endif
}

# ifdef __clang__
# pragma clang diagnostic pop
# endif

#endif // OPENVINO_ENABLE_UNICODE_PATH_SUPPORT

std::string ov::util::get_absolute_file_path(const std::string& path) {
std::string absolutePath;
absolutePath.resize(MAX_ABS_PATH);
Expand Down
52 changes: 52 additions & 0 deletions src/common/util/src/wstring_convert_util.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
// Copyright (C) 2018-2025 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//

#include "openvino/util/wstring_convert_util.hpp"

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT

# include <codecvt>
# include <locale>

# ifdef _WIN32
# include <windows.h>
# endif

# ifdef __clang__
# pragma clang diagnostic push
# pragma clang diagnostic ignored "-Wdeprecated-declarations"
# endif

std::string ov::util::wstring_to_string(const std::wstring& wstr) {
# ifdef _WIN32
int size_needed = WideCharToMultiByte(CP_UTF8, 0, &wstr[0], (int)wstr.size(), NULL, 0, NULL, NULL);
std::string strTo(size_needed, 0);
WideCharToMultiByte(CP_UTF8, 0, &wstr[0], (int)wstr.size(), &strTo[0], size_needed, NULL, NULL);
return strTo;
# else
std::wstring_convert<std::codecvt_utf8<wchar_t>> wstring_decoder;
return wstring_decoder.to_bytes(wstr);
# endif
}

std::wstring ov::util::string_to_wstring(const std::string& string) {
const char* str = string.c_str();
# ifdef _WIN32
int strSize = static_cast<int>(std::strlen(str));
int size_needed = MultiByteToWideChar(CP_UTF8, 0, str, strSize, NULL, 0);
std::wstring wstrTo(size_needed, 0);
MultiByteToWideChar(CP_UTF8, 0, str, strSize, &wstrTo[0], size_needed);
return wstrTo;
# else
std::wstring_convert<std::codecvt_utf8<wchar_t>> wstring_encoder;
std::wstring result = wstring_encoder.from_bytes(str);
return result;
# endif
}

# ifdef __clang__
# pragma clang diagnostic pop
# endif

#endif // OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
Loading
Loading