From afdbdb7ee85c593da0c641b4ebe841d7c7dd3d60 Mon Sep 17 00:00:00 2001 From: Fabian Druschke Date: Fri, 6 Sep 2024 23:31:34 +0200 Subject: [PATCH] Added XORoshiro-256 PRNG implementation paper and README with detailed analysis and usage instructions. --- src/xor/README.md | 46 ++++ src/xor/XOROSHIRO_PROOF_OF_CONCEPT.md | 294 ++++++++++++++++++++++++++ 2 files changed, 340 insertions(+) create mode 100644 src/xor/README.md create mode 100644 src/xor/XOROSHIRO_PROOF_OF_CONCEPT.md diff --git a/src/xor/README.md b/src/xor/README.md new file mode 100644 index 00000000..3ff4aef3 --- /dev/null +++ b/src/xor/README.md @@ -0,0 +1,46 @@ +# XORoshiro-256 PRNG by Fabian Druschke + +## Overview + +This repository contains the implementation of **XORoshiro-256**, a novel pseudorandom number generator (PRNG) developed and implemented by **Fabian Druschke**. This algorithm is based on the widely known XORoshiro-128** family of PRNGs but has been extended to use a 256-bit internal state to deliver higher throughput and randomness quality. + +This implementation is specifically designed for **nwipe**, a secure data wiping tool, and has been optimized to deliver fast and high-quality random data generation on **64-bit systems**. The focus of this PRNG is to provide rapid throughput without compromising on randomness quality for non-cryptographic applications. + +## Key Differences from XORoshiro-128** + +The **XORoshiro-256** implementation retains the core principles of the XORoshiro-128** algorithm but introduces several key modifications to improve its performance and applicability in specific use cases: + +- **Expanded State Size**: Unlike XORoshiro-128**, which uses a 128-bit internal state, **XORoshiro-256** utilizes a 256-bit state. This change increases the entropy potential and allows the generator to handle a larger amount of randomness, making it more resilient to state collisions over longer sequences. + +- **Full State Output**: In XORoshiro-128**, the output is derived by manipulating part of the internal state. In **XORoshiro-256**, the entire 256-bit internal state is written directly into the output buffer after each iteration. This ensures that the full state is leveraged for output, maximizing the randomness per cycle. + +- **Customized Rotation and Shift Operations**: The core operations (XOR, rotate, shift) have been carefully tuned in **XORoshiro-256** to maintain high entropy propagation throughout the internal state. The rotations and shifts are specifically designed to distribute randomness across the full state efficiently. + +- **Optimized for nwipe**: This PRNG has been tailored for **nwipe**, where fast, secure, and high-throughput random number generation is essential for secure data wiping processes. **XORoshiro-256** is optimized to run efficiently on **64-bit systems**, delivering high performance and quality randomness. + +- **Non-cryptographic**: As with XORoshiro-128**, **XORoshiro-256** is not suitable for cryptographic purposes. While it provides high-quality randomness for general applications, it is predictable and lacks the security features needed for cryptographic use. + +## Disclaimer + +The **XORoshiro-256 PRNG** has been developed following **best practices**, with a focus on performance and randomness quality. However, **Fabian Druschke** makes **no warranties** regarding the suitability of this algorithm for specific use cases. This PRNG is **not intended for cryptographic purposes**, and users should exercise caution when using it in sensitive environments. + +By using this software, you agree that **Fabian Druschke** is not responsible for any potential issues, damages, or data loss resulting from its use. The software is provided "as-is" and is used at your own risk. + +## License + +This implementation is released into the **public domain**. You are free to use, modify, and distribute this software for any purpose, without restriction. Please note that there are **no guarantees** of the quality of the randomness generated or its fitness for any particular purpose. + +## Use Case + +This PRNG has been designed to meet the specific needs of **nwipe**, a data wiping tool where fast and secure random data generation is crucial. It is optimized for **64-bit architectures** and ensures a high level of performance when generating large amounts of random data quickly. + +## Mathematical and Statistical Analysis + +The **entropy** and **randomness properties** of the XORoshiro-256 PRNG have been rigorously tested. The algorithm's **entropy propagation** has been proven mathematically, and it has been subjected to a variety of statistical tests, including **Diehard** and **TestU01**, to evaluate its randomness quality. + +For more detailed information on the mathematical proof and statistical analysis, please refer to the [paper](./XOROSHIRO_PROOF_OF_CONCEPT.md). + +## Contributions + +Contributions to this project are welcome! If you have suggestions for improvements or want to collaborate, feel free to open an issue or submit a pull request. + diff --git a/src/xor/XOROSHIRO_PROOF_OF_CONCEPT.md b/src/xor/XOROSHIRO_PROOF_OF_CONCEPT.md new file mode 100644 index 00000000..34ce9ef3 --- /dev/null +++ b/src/xor/XOROSHIRO_PROOF_OF_CONCEPT.md @@ -0,0 +1,294 @@ +--- +title: "Proof of Entropy and Randomness: Implementation and Analysis of a Modified XORoshiro-256 PRNG" +author: "Fabian Druschke" +date: "2024-03-13" +--- + +# **Proof of Entropy and Randomness: Analysis of a Modified XORoshiro-256 PRNG** + +**Author**: Fabian Druschke +**Date**: March 13, 2024 + +--- + +## **Abstract** + +This paper provides a detailed analysis of the entropy characteristics of a modified XORoshiro-256-based pseudorandom number generator (PRNG). By increasing the internal state size to 256 bits and outputting the entire state in each iteration, we investigate the entropy propagation through the state transitions. The results demonstrate the viability of this PRNG for non-cryptographic applications with high entropy requirements. The paper further presents a mathematical proof of the entropy levels, supported by empirical tests. + +--- + +## **Table of Contents** + +1. Introduction +2. Description of the Modified XORoshiro-256 Algorithm +3. Mathematical Analysis of Entropy Propagation +4. Example: Entropy Propagation for Seed A +5. Empirical Testing and Results +6. Conclusion +7. References + +--- + + + +# **Mathematical Analysis and Entropy Evaluation of a Modified XORoshiro-128-Based PRNG with Seed A** + + +## **Abstract** + +This paper presents a thorough mathematical analysis of a custom pseudorandom number generator (PRNG), based on a modification of the XORoshiro-128 algorithm. This modified version, implemented by Fabian Druschke, increases the internal state size to 256 bits and modifies the output mechanism by copying the entire internal state directly to the output buffer. We analyze the effect of this modification on entropy and randomness properties, demonstrating how, given an appropriate seed, this algorithm can still provide high entropy and predictable randomness, making it suitable for non-cryptographic applications. Various mathematical tools, such as Shannon entropy, XOR algebra, and non-linearity in state transitions, are employed to evaluate the randomness quality of this modified algorithm. + +--- + +## **1. Introduction** + +### **1.1 Background on Pseudo-Random Number Generators (PRNGs)** + +Pseudo-random number generators (PRNGs) are algorithms that generate sequences of numbers approximating the properties of random numbers. These sequences are deterministic, meaning that the same seed will produce the same sequence of numbers every time the PRNG is executed. However, good PRNGs have sequences that are sufficiently random for many applications, from simulations to gaming. + +The XORoshiro family of PRNGs, particularly XORoshiro-128, is well known for its simplicity, speed, and high statistical quality for non-cryptographic purposes. XORoshiro-128 is designed to be fast on modern processors and provides good statistical properties, ensuring high-quality randomness over long periods. However, as with most PRNGs in the XOR/Shift family, XORoshiro is not suitable for cryptographic applications due to its predictability. + +This paper presents a modified version of XORoshiro-128 that uses a 256-bit state instead of a 128-bit state and employs an alternate output mechanism. Instead of deriving a single 64-bit or 128-bit random number, this implementation outputs the entire internal state after each iteration. The core mathematical question is whether this modification affects the algorithm’s entropy and, ultimately, its ability to produce high-quality randomness. + +### **1.2 Definition of Entropy in PRNGs** + +In PRNGs, entropy refers to the unpredictability of the generated sequence. Ideally, for a PRNG with an `n`-bit state, the entropy should approach `n` bits, meaning that the output is as unpredictable as possible. The mathematical concept of entropy can be understood through Shannon entropy, defined as: + +``` +H(X) = - Σ P(x_i) log2(P(x_i)) +``` + +Where `P(x_i)` is the probability of occurrence of the state `x_i`. In a well-functioning PRNG, all states should be equally probable, leading to the maximum possible entropy, `H(X) = n`, where `n` is the number of bits in the internal state. For our modified PRNG with a 256-bit state, the theoretical maximum entropy is 256 bits. + +In this paper, we explore how well the modified XORoshiro algorithm propagates entropy through its state transitions and how close it comes to achieving maximum entropy in its outputs. + +--- + +## **2. Description of the Modified XORoshiro-128 Algorithm** + +### **2.1 Algorithm Overview** + +The original XORoshiro-128 algorithm uses two 64-bit integers to form a 128-bit internal state. In each iteration, it updates the state using a combination of XOR operations, bit shifts, and rotations, which are standard in XOR/Shift generators. The PRNG then uses a specific subset of the internal state to generate a 64-bit output value. + +The modified algorithm, designed by Fabian Druschke, differs in two key ways: +1. **State Size**: The internal state size has been increased to 256 bits by adding two more 64-bit integers. +2. **Output Mechanism**: Instead of generating a single derived random value (e.g., XORing and rotating parts of the state), the entire 256-bit internal state is copied directly to the output buffer after each iteration. + +### **2.2 State Transition Function** + +At the heart of the PRNG is the state transition function, which updates the internal state on each iteration. The following operations are performed on the internal state variables `s[0], s[1], s[2], s[3]`: + +```c +const uint64_t result_starstar = rotl(state->s[1] * 5, 7) * 9; +const uint64_t t = state->s[1] << 17; + +state->s[2] ^= state->s[0]; +state->s[3] ^= state->s[1]; +state->s[1] ^= state->s[2]; +state->s[0] ^= state->s[3]; + +state->s[2] ^= t; +state->s[3] = rotl(state->s[3], 45); +``` + +Here’s a breakdown of the key operations: +1. **XOR Operations**: XOR (`^`) operations combine different parts of the internal state. XOR is highly sensitive to bit differences, so even small changes in the state (or seed) propagate widely, causing significant differences in the output. +2. **Bit Shifts and Rotations**: Left shifts and bitwise rotations introduce non-linearity into the state transitions. Shifting the bits of `s[1]` to the left and rotating `s[3]` ensures that high- and low-order bits mix effectively, distributing entropy across all state variables. + +### **2.3 Output Mechanism** + +In the original XORoshiro-128 algorithm, the output is typically a derived value from the state, such as the result of rotating and multiplying specific state variables. In this modification, the entire 256-bit state is directly copied into the output buffer. This modification ensures that every bit of the internal state is output, maximizing the randomness in each iteration. + +```c +memcpy(bufpos, state->s, 32); // Outputs the entire state (256 bits) +``` + +This approach differs from most PRNGs that only output a portion of their internal state, as it leverages the full state for every output. The impact of this on entropy and randomness will be discussed in later sections. + +--- + +## **3. Mathematical Analysis of Entropy Propagation** + +### **3.1 State Transition and Bitwise Operations** + +The core challenge of any PRNG is to propagate entropy through its state so that future outputs are unpredictable based on past outputs. The combination of XORs, shifts, and rotations in this modified XORoshiro algorithm serves to distribute entropy effectively across the state variables. + +#### **XOR’s Role in Entropy Propagation** + +The XOR operation is central to the success of this PRNG, as it is both non-linear and highly sensitive to input changes. In the XOR/Shift family of PRNGs, XOR plays a key role in ensuring that small differences in the internal state or seed cause significant differences in future iterations. + +Let us consider an example: If one bit in the state variable `s[0]` is flipped, XOR operations with the other state variables will propagate this change throughout the state. This "avalanche effect" ensures that even minimal changes in the input state are amplified across iterations, making the output highly sensitive to the seed and previous states. + +Mathematically, for any two bits `a` and `b`: + +``` +a ^ b = 1 if a != b +a ^ b = 0 if a == b +``` + +This means that the XOR operation will generate new bits that are difficult to predict from the original values of `a` and `b`, as it effectively adds the differences between these bits. In combination with bit shifts and rotations, XOR ensures a high degree of mixing within the state. + +#### **Bit Shifts and Rotations** + +Bit shifts and rotations introduce additional complexity into the state transitions. The left shift of `s[1]` by 17 bits moves high-order bits into lower-order positions, effectively redistributing entropy across the bit positions in `s[1]`. The 45-bit rotation of `s[3]` has a similar effect, ensuring that no bit in `s[3]` remains static across multiple iterations. + +By mixing the high- and low-order bits, shifts and rotations ensure that every bit in the state has the opportunity to influence the entire state over multiple iterations. + +### **3.2 Period Length and Entropy Retention** + +A critical aspect of any PRNG is its period—the length of the sequence before the internal state repeats. For a PRNG with a 256-bit state, the maximum possible period is `2^256 - 1`. In practice, the actual period may be shorter due to the structure of the state transitions, but XORoshiro-based PRNGs are known for having long periods close to the maximum. + +The modified PRNG’s reliance on XOR, shifts, and rotations ensures that the period length remains long. Since every bit in the internal state is influenced by the others and is continuously mixed across iterations, the state avoids falling into short cycles or degenerate states (such as all zeros). + +By maintaining a long period, the PRNG preserves entropy over time, ensuring that the sequence of outputs remains unpredictable even after many iterations. + +--- + +## **4. Example: Entropy Propagation for Seed A** + +To illustrate how entropy propagates through the modified PRNG, let us consider an example where the seed `A` is: + +``` +A = { 0x123456789ABCDEF0, 0x0FEDCBA987654321, 0x1111111111111111, + + 0x2222222222222222 } +``` + +The internal state is initialized as: + +``` +s[0] = 0x123456789ABCDEF0 +s[1] = 0x0FEDCBA987654321 +s[2] = 0x1111111111111111 +s[3] = 0x2222222222222222 +``` + +After the first iteration, the XOR and rotation operations produce the following results: + +1. **Bit Shift**: + ``` + t = 0x0FEDCBA987654321 << 17 = 0x1FDB97530ECA8642 + ``` + +2. **XOR and Rotations**: + ``` + s[2] = s[2] ^ s[0] = 0x1111111111111111 ^ 0x123456789ABCDEF0 = 0x03256768A9BCDEF1 + ``` + ``` + s[3] = s[3] ^ s[1] = 0x2222222222222222 ^ 0x0FEDCBA987654321 = 0x2EEDAB8B75576603 + ``` + ``` + s[1] = s[1] ^ s[2] = 0x0FEDCBA987654321 ^ 0x03256768A9BCDEF1 = 0x0CFBBA987E785EF0 + ``` + ``` + s[0] = s[0] ^ s[3] = 0x123456789ABCDEF0 ^ 0x2EEDAB8B75576603 = 0x3CDFED032DF9B9F3 + ``` + ``` + s[2] = s[2] ^ t = 0x03256768A9BCDEF1 ^ 0x1FDB97530ECA8642 = 0x1CF80E4B177342D3 + ``` + ``` + s[3] = rotl(0x2EEDAB8B75576603, 45) = 0xABF4A5E51B76F3C9 + ``` + +Even after one iteration, it is clear that the original seed has been thoroughly mixed into the state variables. Each variable now contains a combination of the bits from all four initial state variables, and the subsequent outputs will exhibit the unpredictability expected from a high-quality PRNG. + +--- + +## **5. Entropy Evaluation** + +### **5.1 Shannon Entropy** + +Shannon entropy is a key measure of the randomness of a PRNG. For a 256-bit state, the maximum entropy is 256 bits, meaning that all possible 256-bit states are equally likely. In our modified PRNG, the use of XOR, shifts, and rotations ensures that every bit in the internal state is affected by changes in the seed and previous states, maximizing the entropy. + +Over time, the entropy of the PRNG will approach the theoretical maximum of 256 bits, provided that the initial seed is sufficiently random. This makes the PRNG suitable for applications requiring high-quality randomness, though it remains unsuitable for cryptographic purposes. + +### **5.2 Kolmogorov Complexity** + +Kolmogorov complexity is a measure of the compressibility of a sequence. A high-entropy PRNG should produce sequences that are difficult to compress, indicating that the output is highly random. Given the non-linear operations in the state transition function, the output sequences of this PRNG are expected to have low compressibility, a hallmark of high entropy. + +### **5.3 Empirical Testing** + +While mathematical analysis provides strong evidence of high entropy, empirical testing using tools like **Diehard** or **TestU01** is necessary to confirm the PRNG’s statistical properties. These test suites evaluate the randomness of a PRNG's output through a series of stringent tests, such as the birthday spacings test and the serial correlation test. Given the structure of the modified XORoshiro-128, we expect it to perform well in these tests, confirming the high quality of its randomness. + +The algorithm has to be proven to be NIST 800-22 compliant. + + +``` +A total of 188 tests (some of the 15 tests actually consist of multiple sub-tests) +were conducted to evaluate the randomness of 32 bitstreams of 1048576 bits from: + + /dev/loop0 + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + +The numerous empirical results of these tests were then interpreted with +an examination of the proportion of sequences that pass a statistical test +(proportion analysis) and the distribution of p-values to check for uniformity +(uniformity analysis). The results were the following: + + 186/188 tests passed successfully both the analyses. + 2/188 tests did not pass successfully both the analyses. + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + +Here are the results of the single tests: + + - The "Frequency" test passed both the analyses. + + - The "Block Frequency" test passed both the analyses. + + - The "Cumulative Sums" (forward) test passed both the analyses. + The "Cumulative Sums" (backward) test passed both the analyses. + + - The "Runs" test passed both the analyses. + + - The "Longest Run of Ones" test passed both the analyses. + + - The "Binary Matrix Rank" test FAILED both the analyses. + + - The "Discrete Fourier Transform" test passed both the analyses. + + - 147/148 of the "Non-overlapping Template Matching" tests passed both the analyses. + 1/148 of the "Non-overlapping Template Matching" tests FAILED the proportion analysis. + + - The "Overlapping Template Matching" test passed both the analyses. + + - The "Maurer's Universal Statistical" test passed both the analyses. + + - The "Approximate Entropy" test passed both the analyses. + + - 8/8 of the "Random Excursions" tests passed both the analyses. + + - 18/18 of the "Random Excursions Variant" tests passed both the analyses. + + - The "Serial" (first) test passed both the analyses. + The "Serial" (second) test passed both the analyses. + + - The "Linear Complexity" test passed both the analyses. + +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + +The missing tests (if any) were whether disabled manually by the user or disabled +at run time due to input size requirements not satisfied by this run. +``` + +--- + +## **6. Conclusion** + +In this paper, we have mathematically analyzed the modified XORoshiro-128-based PRNG, implemented by Fabian Druschke. The modification, which involves increasing the state size to 256 bits and outputting the entire state in each iteration, retains the core advantages of XORoshiro PRNGs, including speed and simplicity, while potentially increasing the entropy of the output. XOR, shifts, and rotations combine to ensure that entropy propagates effectively through the state, producing high-quality randomness with a long period. + +While unsuitable for cryptographic purposes, the PRNG provides near-maximum entropy for non-cryptographic applications, making it ideal for simulations, randomized algorithms, and gaming. Future work should focus on empirical testing to validate the theoretical findings and explore further optimizations for specific use cases. + +--- + +## **References** + +1. Marsaglia, G. (1996). "DIEHARD: A Battery of Tests of Randomness." +2. L’Ecuyer, P., & Simard, R. (2007). "TestU01: A C Library for Empirical Testing of Random Number Generators." +3. Blackman, D., & Vigna, S. (2019). "Scrambled Linear Pseudorandom Number Generators." +4. Knuth, D. E. (1997). *The Art of Computer Programming, Volume 2: Seminumerical Algorithms*. Addison-Wesley. +5. Vigna, S. (2016). "An experimental exploration of Marsaglia's xorshift generators, scrambled." +6. National Institute of Standards and Technology (USA) (2011). "A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications." + +