Why You Should Learn Reverse Engineering and How to Get Started
I am WittsEnd2 — Founder of Ragnar Security. My mission is to give everyone to make complex cybersecurity topics easier to learn. Particularly, I focus on teaching Binary Exploitation and Reverse Engineering.
Why did I start doing Reverse Engineering and Binary Exploitation?
I started to learn Reverse Engineering as a way to solve difficult challenges. Take for example CTF Challenges focused on Reverse Engineering. They are designed to find a flag (generally in a format like FLAG{}), and if you are successful, you get points for your team. Professionally, this is similar to any other software. Malware analysts figure out how what is the malware actually doing so that software security engineers can protect against that strain of malware.
Reverse Engineering also helped me with my software engineering as I became better at debugging programs. For example, I improved my use of debuggers such as GDB and WinDBG because they are useful in discovering what’s happening when running a program. Also, I learned how to use new tools to understand how a binary works. Ghidra, strace, and ltrace are perfect examples of this because each of these tools gathers different information about a binary.
Now that you know about my interests in Reverse Engineering, let's get started on your Reverse Engineering journey.
Why should you get started in Reverse Engineering?
It depends, there is no one true answer to this question, nor should I claim that I have the right answer. Assess your goals and see how Reverse Engineering fits. Most people, that I know, get into reverse engineering because for career purposes. Others want a challenge and solve puzzles.
Even if you are a developer, you can find reasons to learn reverse engineering. From my own experience, I have become a better debugger because I learned reverse engineering. GDB, Ghidra, and tools for determining code behavior. This allows me to figure out problems with my code quicker and
How to start Reverse Engineering!
This isn’t going a step by step process, but some ways that you can build skills. While these are separate strategies, these can be blended. Everyone has their own learning style and saying that the way I learned is the only/best way to learn reverse engineering isn’t correct. That being said, here are some resources and strategies that you should try:
Learn by doing it! — Get a decompiler/disassembler and just start!
This is the most adventurous way of doing it and might be the best way to integrate it into your memory, but it is also the most challenging way to do it. If you are not familiar with assembly (ARM, x86, etc), you need to constantly look up instructions. You will also be working towards getting used to the tools you are working with as well.
Some places where you can get some practice:
- ctftime.org — list of CTFs that has happened or about to happen
- picoctf
- overthewire.org
- Googling!
When doing it this way, you will need to supplement your learning experience with writeups and videos. Undoubtedly, there will be times when you struggle and unable to find the solution; thus, writeups and videos will help you learn the things that you missed, and how to take a different approach.
Watch YouTube Videos and Follow Along!
There is a lot of content on YouTube and there are so many things that you can learn from other people. This could range from CTF walkthrough videos, embedded systems reverse engineering, to even malware analysis. Whatever topic you are interested in, there is probably a video (or channel) for it. Here are a few that I recommend:
- LiveOverflow — https://www.youtube.com/channel/UClcE-kVhqyiHCcjYwcpfj9w
- StackSmashing — https://www.youtube.com/channel/UC3S8vxwRfqLBdIhgRlDRVzw
- Colin Hard (Malware Reverse Engineering) — https://www.youtube.com/channel/UCND1KVdVt8A580SjdaS4cZg
By taking the time to watch videos, you can explore topics more topics and broaden your skills as a reverse engineer! I challenge you to search around YouTube and see what you find!
Read Books and Papers About Different Topics!
I don’t learn very well from reading; however, there are times when books and papers teach about certain topics better than videos or hands-on practice. The most relatable example for me was when I wanted to add anti-reversing techniques to a CTF challenge I wrote for UMDCTF. Particularly, the book Programming Linux Anti-Reversing Techniques helped me with this. I learned how to apply anti-debugging techniques with ptrace, obfuscate the code, and ways to mess with the file format. This book also had examples for implementing it on an x86 binary, so when I tried to do it for x86_64, it was simple. Here are some good book(s) that I recommend:
- Programming Linux Anti-Reversing Techniques- https://leanpub.com/anti-reverse-engineering-linux
- No Starch Press Books — (their practical series has a ton of great reverse engineering tips) — https://nostarch.com/
- The “Ultimate” Anti-Debugging Reference (Windows) — https://anti-reversing.com/Downloads/Anti-Reversing/The_Ultimate_Anti-Reversing_Reference.pdf
Learn by taking a course! — Have someone teach you the basics before starting!
For those that are visual learners, taking a course might be a really good way to get started. These courses teach the foundations, such as learning how to use a tool or an assembly language, and they also teach tips and strategies. For example, the Reverse Engineering course I took taught me how to use IDA and how to break anti-reversing techniques. Anti-reversing techniques is particularly a niche topic and there isn’t nearly as much coverage on it as IDA. By having someone show what it could look like, you will be able to research the topic further and know what to look for.
Another reason to take a course is that it builds credibility if you are looking to do reverse engineering professionally. It will differentiate you from others who are applying for reverse engineering opportunities because you were professionally trained. This does not mean, however, that you don’t need to practice outside of class. You still need to do some external practice like CTF Challenges because the basics aren’t enough for a majority of available reverse engineering positions.
Reverse Engineering is something that I think everyone should learn! It is extremely adventurous, can lead to an awesome professional career, and it can help with other aspects of your job (if not doing reverse engineering professionally). You will not regret the decision!
