The name of the challenge hints what the program might be doing (operations on "squares"). Try using a decompiler (such as Ghidra) to get a rough idea of what the program is doing. Keeping in mind that the program is working with squares, try to look for relevant data structures or control flows (for example 2 dimensional arrays, square numbers, doubly nested loops, etc.).
The arithmetic operations performed on your input is eventually checked against some hardcoded values in the binary. These operations are simple and give constraints on the input values which can be solved automatically.