Prompt

Math is often tricky and hard, which is why we make computers do math for us. That being said, it's important to know how to solve the math when the bidding is essential.

Walk-Through

This is a reverse-engineering challenge designed to analyze what a C program does. In this challenge, you will reverse the encryption process that changes an input text into an output text.

Main Function

Start by looking at the C code given for the challenge. Notice that there are two functions. In the C programming language, main is the program’s entry-point function where execution starts.

int main() {
    char input[64]; 
    printf("Enter the secret sequence: ");
    fgets(input, sizeof(input), stdin);
    check_flag(input);
    return 0;
}

Open the drop-down arrows for more explanations on what each line does!

check_flag

Now, let’s look at the check_flag function. Every other step of the main function falls within the C standard library, and nothing has been modified to what we can see. The check_flag function is where a majority of the work is performed.

Briefly view the function as shown below:

Use the dropdown toggles to understand what a section of code does to a deeper degree:

Guide

The process of reverse-engineering involves taking the long, complicated, and extensive final form of an engineered piece of work (such as the code above), and breaking it apart in detail to understand how it was created. The next part is understanding how to perform the opposite of the processes used in the original code.

Questions 1-5 for this challenge can be solved by analyzing the code. Question 3 may require some outside research, but all of the answers are found in the code provided for this challenge.

Question 6 is essentially asking for the input (or the correct flag) that will return the ‘Correct…’ statement. This will take a bit more work because it will involve performing the opposite of the processes used in the code.

Solving for the Flag

How do we figure out the flag from this code? The input submitted to the program initially becomes transformed_text. At the end,transformed_text is compared to target_string (EOC-ZBMH-5761) to see that they match. Therefore, we need to use target_string and functions performed to transformed_text to get the input that will cause the function to return “Correct”

After analyzing the code, there are a few obvious places where these elements of the flag are modified.

Solving else if (i == 7)

At this portion, the code uses the 8th character in transformed_text and applies the letter_r_xor_key (which is 0x1A ). From the looking at target_string , it can be concluded that this transformation will result in the letter ‘H’.

What character will result in ‘H’ when XORed with 0x1A?

Cyberchef can show that H XOR 0x1A = R. Therefore, the input must be ‘R ’ to get ‘H’ in the final target_string.

Solving else if (i >= 9 && i <= 12)

From the last else if block, we can determine what characters 9-12 of the index should be. Recall that digit_xor_keys[] = {0x1, 0x2, 0x3, 0x4}; and the last four digits of the target_string are 5761.

Perform the functions as follows: 5 XOR 0x1, 7 XOR 0x2, 6 XOR 0x3, and 1 XOR 0x4 . You can use cyberchef to perform the XOR. The result will be 4555.

Solving for *

So far, we have ***-***R-4555 as our flag (where * corresponds to an unknown value).

Let’s go back to the ‘complicated’ portion of the code. This portion is specifically for transforming the first 3 characters of the flag and the next three characters after the first hyphen (basically, the characters marked with * that we have left to solve for).

This portion of the code is where the transform_matrix matrix is being used and block_size is utilized. After this point, all variables and arrays in the code are accounted for.

This portion of the code is performing a Hill Cipher transform on 2 separate blocks of 3 characters. Those 2 blocks are EOC (index 0, 1, and 2 of target_string) and ZBM (index 4, 5, and 6).

To decrypt a Hill Cipher, and basically ‘reverse’ the multiplication done that produced the ciphertext, the ciphertext (in this case, portions of target_string) will need to be multiplied by the inverse of the key matrix (or mod 26).

After completing this operation, we will have the entire flag string. Anything else in the code performs comparisons to make sure the transformation happened correctly, and adds a \0 null terminator character in order to end the string.

Shown in the dropdown menus below is the process that occurs on EOC. Solving for ZBM will be done is a similar process but will not be shown for brevity.

Hill Cipher Decode

Assuming you determine this function in the code is using a Hill Cipher, you would not need to complete the math by hand or using Python. You can use a hill cipher decoder and pass in the matrix. Functionally, a Hill Cipher is a normal cipher, except that the key for the encryption is represented as a matrix. This can be any matrix with the same numbers of rows and columns.

Tutorial Video

Watch our full Tutorial Video to learn more about reverse engineering and see a walkthrough of how to solve this challenge:

Cyber Skyline Live: Reverse-Engineer a C Program for Enumeration & Exploitation

In Cyber Skyline Live: Reverse-Engineer a C Program for Enumeration & Exploitation, you’ll learn from Franz Payer, CEO of Cyber Skyline, about how to analyze a C program that uses XOR and matrix transformations to reverse-engineer a flag. This video is for educational purposes only.

www.youtube.com

Questions

1. What are the dimensions of the transform_matrix variable?

2. What is the integer value stored at transform_matrix[0][1]?

3. What is the header file that provides functions like isdigit and isalpha?

4. What is the hex value assigned to the letter_r_xor_key variable?

5. We apply a matrix change to the letters at indices 0 to 2. What are these three letters in the target_string?

6. What is the input that correctly works for this code?

©️ 2026 Cyber Skyline. All Rights Reserved. Unauthorized reproduction or distribution of this copyrighted work is illegal.