SLAE Assignment #4 – Customer Encoder

The goals for Assignment #4 are:

  • Create a custom encoding scheme like the “InsertionEncoder” we showed you
  • PoC with using execve-stack as the shellcode to encode with your schema and execute

For my custom encoder, I decided to make what would be a “ROT13” or rotate 13 encoder. A python script would take the hex values of each byte of the shellcode a user provides, and add 13 to that hex value. A decoder stub would be created that looks at each byte of the encoded shellcode, subtracts 13, and then jumps to the decoded shellcode after all of the bytes had been decoded.


The encoder was created using python. The shellcode bytes would be hard coded into the python script in the format “\x00”. To make the encoder more configurable, the user would provide a number they wanted to “rotate” the original shellcode to encode it.

I used objdump to get the hex values of the bytes for my shellcode that used execve to launch /bin/bash. I then added the bytes to the python script, and ran the python script to “rotate” the shellcode by a value of 2.

The encoded bytes were written to the screen and would be used when creating the assembly code for the decoder.

The python encoder script can be found here:

Assembly code for the decoder stub

The encoded shellcode would be defined in memory, and the decoder stub will modify the bytes while iterating through each byte. To provide the decoder stub with the memory address of the encoded shellcode, a JMP-CALL-POP was used.

; jump to the segment of code that defines the encoded shellcode in memory to start the JMP-CALL-POP process to load the encoded shellcode bytes into memory
jmp short call_decoder

; “pop” the memory address to encoded shellcode off the stack into a registers
pop esi


; call the decoder section of the code. This puts memory location of the encoded shellcode on the stack
call decoder

shellcode: db ; bytes of the encoded shellcode

After this is performed, the memory address of “shellcode” would be popped into the ESI register.

To decode the bytes, the decoder stub will loop through the bytes at the memory address of “shellcode” and subtract 2. To do this, the number of bytes needed to be decoded will be required. My encoded shellcode was 30 bytes in length. The ECX register is used for the loop counter, so it was set to the value 30.

; set up the counter. The shellcode is 30 bytes long, so set the counter to 30
xor ecx,ecx
mov cl,30

The decoder would then loop through 30 bytes by increasing the value of ESI with each loop. Once the loop had been performed 30 times, it would jump to the decoded shellcode.

; this will subtract the encoded byte by the amount rotated by the encoder
sub byte [esi], 0x2 ; replace 0x2 with the number you used to encode with
inc esi
loop decode
jmp short shellcode

Testing the shellcode

The assembly was compiled and then executed. When I tried to run the decoder, it first gave a segmentation fault error. This is due to the decoder trying to modifying memory space that was read-only. To get around this, the decoder’s shellcode bytes were extracted with objdump, and a C shellcode runner was used to run the decoder. The shellcode runner placed the shellcode in memory space that could be written to, and the decoder ran successfully.

This seemed less than ideal, so the decoder was re-written in assembly so that instead of the encoded shellcode being defined in memory and using JMP-CALL-POP to get the memory address, the encoded shellcode bytes were pushed onto the stack. The stack should be writable by the decoder, so the segmentation fault shouldn’t occur. No-operation or “NOP” bytes were added to the shellcode to make it divisible by four when pushing to the stack.

; push the encoded shellcode onto the stack
push 0x909082cf
push 0x0db2e38b
push 0x55e48b52
push 0xe58b3131
push 0x31316a31
push 0x706b646a
push 0x6a756364
push 0x6a52c233

Then, the stack pointer (ESP) value was moved into ESI. The decoder then proceeded to decode the encoded shellcode in the same manner as before.

The decoder was compiled, and then executed successfully.

The code for the encoder that uses defined bytes in memory and JMP-CALL-POP can be found here:

The code for the encoder that uses the stack can be found here:

SLAE Student Disclaimer

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: SLAE- 1373