JuliaImages 1.0 and beyond — a two year roadmap

[johnnychen94]

For years I've been thinking what 1.0 means for JuliaImages and what I can do for it. I just started my third year in my Ph.D. career, and I expect to graduate in another two years. It's hard to know what will happen after graduation and whether I'll be able to further commit to JuliaImages, so I think, maybe, I should set a two-year target for JuliaImages and reach some healthy status and so that newcomers can become regular contributors, and regular contributors can become maintainers. This is why I drafted this roadmap.

The role of JuliaImages ecosystem

Sometimes the answer is simple, make JuliaImages another OpenCV. But it requires more than we can afford to achieve this goal. This is often less a priority here because we don't want to reinvent the wheel.

Eventually, I want to make JuliaImages a platform for research and education before I step away from being an active maintainer. JuliaImages will try to provide high-performance implementations, but won't commit to embedding systems because the Julia ecosystem isn't ready yet.

What problems can(should) JuliaImages solve

As a dynamic language, Julia makes prototyping even easier than Python/MATLAB, and as a compiled language, Julia runs as performant as C/C++. There are many cases in image processing where prototyping doesn't require much on performance, thus it doesn't make much difference whether the researchers doing the prototyping work in Julia, Python, or MATLAB. Thus if we want to distinguish JuliaImages from other image processing frameworks, we should target:

• solving tasks where even prototyping requires high performance. For instance, real-time video encoding/decoding, 3D image processing, and 3D modeling. In the domain of 3d image processing, very few frameworks are as well optimized as JuliaImages is currently.
• supporting new computational patterns where optimization is not yet done in C/C++. For instance, automatic differentiation through filters, warps, and other primitives.
• doing more math-related and not engineering-related tasks because Julia is math ready. For instance, if we want to solve some optimization and PDE problems when doing image processing, Julia should be the best choice.

The key properties of JuliaImages and how we achieve these

Usability, Generality and Composability

The most satisfying point I enjoy about JuliaImages is its usability and its integration with the entire Julia ecosystem; when you learn JuliaImages you are also learning the language Julia. This makes JuliaImages a perfect platform for education.

The only secret here is to not blocking ourselves as an isolated ecosystem; if we can make generic implementation in an independent upstream package, we make it there. For example, to deal with padding we provide OffsetArrays and PaddedViews, to deal with the iteration of N-dimensional arrays we introduce CartesianIndices to upstream Julia, and to deal with patches and blocks we introduce TiledIterations. This definitely requires more effort than "fixing things right here as patches", but the gain is that the ecosystem is so easy to intuit and use while still as generic as it can be.

Performance

Image processing itself is a computationally intensive field and also has a long history. Many classic algorithms have been highly optimized in low-level languages (such as C/C++) or DSL languages (such as Halide) very well. Thus a key question for JuliaImages to solve is: how do we achieve competitive or even better performance than existing frameworks?

There are already many cases where JuliaImages excels compared to the competition. Where we do not, there are several possible solutions:

• Take advantage of low-level codegen and compiler optimizations (e.g., LoopVectorization)
• Take advantage of hardware and concurrency (e.g., GPU, threads)
• Implement new efficient algorithms that require less computation

All of them are important, but I might want to emphasize more on the 3rd point here because this is where Julia can distinguish itself from other languages. And this requires JuliaImages to be a good platform for research and education.

JuliaImages target feature set:

This list is by no means complete and I can't make sure everything can be achieved, but I'd like to list it here so that people know what is under the scope of the JuliaImages ecosystem.

• Spatial transformations: warp, rotate, resize, zoom, projective geometry
• Registration: feature matching, error minimization, optical flow
• Image restoration and enhancement: denoise, deblur, super-resolution, rain removal, dehaze, contrast adjustment, smooth, binarization, inpainting
• Image reconstruction: CT, MRI, etc
• Image morphology
• Segmentation, edge detection
• Feature descriptors, object detection, template matching
• Filters: convolution, wavelets, etc
• Visualization tools
• IO utils: PNG, JPEG, TIFF

Image processing and computer vision is a complicated field with so many possible tools that no single person knows them all, e.g., filters, PDEs, variational methods, statistics, deep learning. And it is also an evolving field that new applications and methods are developed, e.g., image style transfer.

I can't foresee what JuliaImages will contain in the future, but I'd like to still give a simple criterion on whether an algorithm should live in JuliaImages:

• It should either be from known literature, or it solves one specific question for other parts of JuliaImages.
• It is reliably reproducible because we want to test it.
• If it has dependencies outside JuliaImages' current scope, that dependency should be relatively small and reliable.

For instance, bilateral filters, QR code support, MRI reconstruction are good candidates for JuliaImages.

We would also want to extend JuliaImages to support GPUs, AD, and deep learnings. But they might live in some bigger projects, e.g., the not-yet-existed CuImages.jl and/or DiffImages.jl.

I'm pretty sure that I still missed a lot of fields in the above list. If people have ideas and want to develop packages under JuliaImages, I'm glad to have them here so that maintainers here can help to watch them. A very nice recent example is DitherPunk.jl.

Code organization

Currently, there is one meta package Images.jl that provides an out-of-box experience for many users of JuliaImages. This is supported by Reexport.jl via e.g., @reexport using ImageCore. In the future, we might have CuImages.jl for CUDA.jl support, or DiffImages.jl for AD (e.g., Zygote.jl) support.

I'd like to also briefly introduce some of the packages in JuliaImages so that people know how we organize things:

• Low-level core types:
  • Pixel abstractions: every pixel is a struct Colors.jl for colorants (e.g., RGB), FixedPointNumbers.jl for N0f8
  • Image abstraction: every image is an abstract array with pixel elements. ImageCore can be seen as a blessed umbrella package of various arrays (e.g., MappedArrays, OffsetArrays, PaddedViews) and related traits that we use among JuliaImages.
• Intermediate-level utilities:ImageBase, ImageFiltering, ImageContrastAdjustment, ImageMorphology, HistogramThresholding provide some core utilities and functions that you might use to process images and build a higher level pipeline.
• High-level functions and solutions: ImageBinarization, ImageNoise, ImageSegmentation , ImageTransformations and others provides some out-of-box solution for specific problems and fields.

All these packages are composed together as our meta package Images.jl using Reexport.jl.

Where one specific function lives is not fixed. It is possible that we first develop a field-specific implementation of the function in high-level packages, and then we polish and generalize it, and finally move to lower and smaller packages. For instance, forwarddiff was developed as a specific utils for Images.imROF(deprecated) and then we generalized and moved to ImageBase as fdiff. It is also possible that we first develop one algorithm in some package, and then we add more implementations and algorithms and making it an independent package. For instance, otsu_threshold (deprecated) once lived in Images.jl and then @zygmuntszpak added a lot of similar threshold operations and created HistogramThresholding.jl.

Packages are(will be) hosted either in the Images.jl repo as sub-dir packages, or hosted as independent packages (e.g., OffsetArrays.jl, Colors.jl)

Tests in the Images.jl repository consist of two parts:

• Unit tests on each subpackage
• Integration tests on Images.jl. There will be some comprehensive applications that we can't build with only Images, thus we want to test if some predefined demos and examples are still working.

Documentation:

• Documentation for packages that live in Images.jl should be hosted in https://juliaimages.org/
• For independent packages, we should at least have some demos or tutorials to show how it's used in JuliaImages.

There are two key issues that drive us to this scattered code organization:

• Tradeoff between out-of-box solution and package loading time: users enjoy the simple using Images solution to get everything ready, while developers want to cut down as many dependencies as possible to get faster loading time. For example, using FileIO should be sufficient to load an image.
• Reliability: some dependencies can be out of the control or understanding of JuliaImages maintainers. If it is too opaque or unstable to coordinate with, we would refrain from adding them as direct dependencies to packages under JuliaImages. In some sense, we want to only depend on trusty packages so that we can build a stable ecosystem on top of them. If we want to implement a feature that depends on unstable packages, it would be better to do this first as an independent experimental package.

Hopefully, when Julia has first-class conditional package loading, we don't need to make these tradeoffs.

JuliaImages roadmap

When writing down the roadmap, I'm thinking about the question: why are people duplicating the functionalities and basic utils outside JuliaImages? I get two answers:

• The documentation is not comprehensive or clear enough; it's hard to understand the core principles of JuliaImages simply from the documentation. And because people don't understand it, they want something more transparent to them.
• JuliaImages doesn't provide fundamental support for some specific fields, e.g., image reconstruction.

I set the roadmap as it is here so that when people develop new functionalities whether inside or outside JuliaImages, they can reuse the infrastructure that JuliaImages provides without duplicating the codes.

1.0 roadmap

The main tasks of 1.0 roadmap is to revisit and clean our codebase, improve documentation, and eventually set a common "modern" ground for future improvement.

• Move almost everything from Images.jl to smaller packages (🧹 code organization and dead code elimination #971, Sort out our dependencies and moving to version 1.0 #825)
• Revisit packages from bottom to up: check the names, traits, data structures, and APIs. Some packages (e.g., ImageFiltering and ImageMorphology) can be separated to a more generic package and an image-specific package
• Reunite Images into one big repo (Fixing the workflow: reuniting JuliaImages, Colors development #898)
• Revisit and organize our exported symbols and entry points. (Symbol hierarchy and reorganization #906)
• Set up benchmarks: for both regression tests and comparison with other frameworks.
• Documentations for users: for every main function in JuliaImages, we should add docs and demos for them.
• Documentation for developers: performance guides, workflows, best practices.
• Revisit our "Comparison with other image processing frameworks" juliaimages.github.io#230
• improve/remove AxisArrays stack from JuliaImages

1.x features

Here I just list things that interest me most; things that I will eventually try to support given enough free time. As an open-source project without the full-time commitment, there are really no hard requirements or timelines here.

• Extensible GPU and multi-threading support
• LoopVectorization support
• AD supports for some primitives
• More IO supports: Libjpeg wrapper and etc
• Better visualization tools for video, 3d rendering, and even computational graphics (focus more on the computing and not plotting)
• New tools: wavelets, framelets (e.g., tightframes)
• Revisit ImageAxes, better support on unitful pixels, and see if we can provide better support for MIRT.jl Better support for spatial/temporal images: ImageAxes, Unitful or others? #978

[Tokazama]

I'm probably going to register SpatioTemporalTraits.jl soon. It's far from perfect but it's at least a start if there is interest in gradually integrating it.

EDIT: just triggered registrator on this