Introduction to ciphers


#1

During this ARG I noticed that being new to this, I often had a harder time on basic ciphers and recognizing them, making me fall behind. E.g.: I didn’t know that ‘SCHRODINGEABFJKLMPQTUVWXYZ’ is a keyword, neither did I know how to use it… Or if I look at the first pdf, I do not know instantly which cipher is which type.

With your help I hope to make a starting point for all the other people who are having the same trouble. I’ll edit this opening post with your answers and questions. It is not the goal to make an in depth analysis of all possibilities, but a basic guide. Please ask or add anything you think is important! Below some websites and questions to start off the topic. As I lack so much knowledge these have to be added to by you!

Q(&A) (will be updated with your Q’s and A’s)

Encrypting/Decrypting

Encryption: the process of encoding the original message. These encrypted messages are what we find.
Decryption: Transforming the encrypted message back to the original. Press this button on any website where you’ve inputted the encrypted text.

Recognizing basic ciphers

Thanks to @Shota

Binary:
A combination of 0 and 1. Most commonly each letter is represented by a 8 bit string.
E.g.: ‘Example’ = ‘01000101 01111000 01100001 01101101 01110000 01101100 01100101’

Octal:
As the name suggest only the number 0-7 are used. Each character is represented by a string consisting of 3 numbers. This type is not often used.
E.g.: ‘Example’ = ‘105 170 141 155 160 154 145’

Decimal(/ASCII numbers):
Uses numbers 0-9. The length of the encoded string depends on the character. Most often the length will be 2 or 3.
E.g.: ‘Example’ = ‘69 120 97 109 112 108 101’

Hexadecimal:
16 characters can be used: 0-9 and a-f. Each encoded letter is 2 of these characters long.
E.g.: ‘Example’ = ‘45 78 61 6d 70 6c 65’

Base64:
64 characters are possible: a-z, A-Z, 0-9 and +/. The encoded strings often, but not always, end in = or ==.
E.g: ‘Example’ = ‘RXhhbXBsZQ==’

Morse:
Known by most. Consisting of long and short signals, most often represented by - and . Case insensitive.
E.g.: ‘Example’= ‘. -…- .- – .–. .-… .’

Recognizing the cipher:
Recognizing them is not always easy and generally, in puzzles like this people make assumptions:
only 0 or 1, it’s binary,
only numbers, it’s decimal
letters in your numbers, but only in the a~f range, it’s hexadecimal.
This doesn’t need to be true, since 123901541 could be either decimal or hexadecimal, but in puzzles it’s most commonly as the above assumptions show.

Recapping the example:

ASCII :      EXAMPLE
BINARY:      01000101 01111000 01100001 01101101 01110000 01101100 01100101'
OCTAL:       105 170 141 155 160 154 145
DECIMAL:     69 120 97 109 112 108 101
HEXADECIMAL: 45 78 61 6d 70 6c 65
BASE64:      RXhhbXBsZQ==
MORSE:       . -..- .- -- .--. .-.. .
Caesar shift

A basic encryption method in which two alphabets are aligned. One alphabet is shifted X positions. Each letter of the original message is then replaced by the corresponding one from the cipher alphabet. This is a type of substitution cipher. A special caesar shift is called ROT13, which rotates the alphabet with 13 places.

Example. Shifted 3 places:
Plain : ABCDEFGHIJKLMNOPQRSTUVWXYZ
Cipher: DEFGHIJKLMNOPQRSTUVWXYZABC
This would transform ‘example’ into ‘hadpsoh’

Substitution with keyword

Basic:
Like the Caesar shift two alphabets are aligned. The cipher alphabet is however transformed by using a keyword. This keyword is added in front of the alphabet. All letters used in the keyword are then removed in the alphabet:

Example with keyword ‘testing’
Plain : ABCDEFGHIJKLMNOPQRSTUVWXYZ
Cipher:TESINGABCDFHJKLMOPQRUVWXYZ
This would transform ‘example’ into ‘nxtjmhn’

Note how the keyword ‘testing’ become ‘tesing’ as each letter can only be used once in the alphabet

More complex:
Thanks to @Shota
If you find a password (or phrase) in the form of an alphabet (such as “SCHRODINGEABFJKLMPQTUVWXYZ”), it is usually used to make a polybius square https://en.wikipedia.org/wiki/Polybius_square2 and usually this means it is one of the following: ADFGVX cipher, Nihilist cipher, bifid cipher or fairplay cipher. (Trial and error will determine which.)
Notice that the polybius square only has 25 places, usually either the X, the Q, the I or the J is dropped. The default for a lot of these (as it was in our schrodinger example) is to treat the I and the J as the same character, and thus drop 1 of them (usually the J), you can easily see which parts of your text were supposed to be J’s and which are properly I’s. Also notice that these ciphers ignore all interpunction (spaces, dots, commas, etc).

Book ciphers/Book code

Answer thanks to @DevilinPixy

With book ciphers, the ‘key’ used to encode/decode the message is an existing text, document, book or alike. Let’s use a book for example. When the sender encodes a message using a book, the receiver requires to have the exact same book (version) to be able to decode the message, since it is the key used to encrypt it.

Encoding can be done in several ways and results in a series of separated numbers that refer to either words in the book or the starting letter of that word. The encoder, can refer to these words or letters in different ways, thus ending up with a variety of possible number sequences.

Beale Cipher
This is an example of a book cipher, where you encode a text, by translating each letter to a number that matches the starting letter of a word in the book. The resulting number for each letter is the word count from the start of the text in the book. So if you encode the letter ‘T’, you could start counting words in the book until you find a word that starts with a ‘T’ and write down that number you counted to get to that word.

Other variants

  • Full Words:
    Instead of using only the starting letter, you could actually use complete words. So if your message to encode contains the word ‘THE’, you could count from the start of the book text, until you find this word and use this counted number for your encoded message. This may be faster and even easier to do, but it requires a book that has all the words contained in the message you wish to encode.

  • Different Count Method:
    When using a book to encode your message that contains a lot of pages and sections, it is possible to use a different counting mechanism. The receiver of the encoded message needs to know the counting mechanism used. You could for example not count words from the start of a book, but use a decided on set of numbers, referring to the Page, Line and Word for example. This would result in sets of 3 numbers instead. This could still decode into the first letter of that word, or possibly the full word.

How to recognise a Book Cipher
They can be recognised by the fact these only contain numbers, which are separated. These numbers are often higher than for example 26 (amount of letters in the alphabet) and can go even into the thousands. With truly high numbers being used, it is therefor likely a big ‘book’ and less likely to have a different count method where sets of numbers are being used.

Decoding a Book Cipher
Requires knowing the book, text, document or similar, being used. This is called the key. Counting is required, although tools can sometimes be used, but may often be limited.

Useful websites to decypher text

ARGonauts - For players (seriously: tools for everything)
http://rumkin.com/tools/cipher/
http://www.dcode.fr/tools-list
Easily check the basic ciphers
CyberChef: easy to use with a lot of functions
Test all possible caesar shifts at once

Please add your expertise here!

Mods: This topic is not completely in approval with the forum section’s guidelines, yet I believe it might be useful here. Feel free to delete or move it though.


#2

Related, how much have we explored the idea of a Hamlet book cypher? I remember seeing some earlier progress along those lines but it didn’t quite pan out. Maybe now that we have some more numbers from Multiverse-75? I’m going to take a crack at it myself, but my knowledge of book cyphers and the like is limited to what I’ve learned from spy movies. :grinning:


#3

Forgot abour bookciphers! Added a question in the main post about it. Thanks for the idea. The question still needs a answer though. Any things that typical guide us to recognizing an book cipher? How does a book cipher exactly work??

Maby it’ll draw some attention to the possible hamlet cipher. But i think the idea itself is best discussed in the main forums


#4

Thanks for posting this. This thread will be a useful tool for the ARG. I started looking around for more info about ciphers the other night and have found a couple links.

This is the “Black chamber.” It walks you through different ciphers and provides tools to create them.
http://www.simonsingh.net/The_Black_Chamber/chamberguide.html

The link you, @ohwell, posted seems to be the best tools I’ve found so far. It has tools for the most common types of cyphers.
http://rumkin.com/tools/cipher/

For some, using a tool might take the fun out of solving them. I only have solved simple ceasar shift ciphers on my own, where the alphabet is shifted a numbers of spaces, before. Knowing that “A” and “I” are the only single letter words, the most common letters of the alphabet are “E” and “T,” and figuring out the common two and three letter words are the basics.

It would be nice to find a simple troubleshooting tree for identifying ciphers. I’m still looking.


#6

I would like to add the following website, which can be viewed in English and has loads of tools and info:

Description: dCode is a toolkit website for decryption, ciphertexts, cheating at letter games, solve riddles, treasure hunts, etc.
http://www.dcode.fr/about

Book cipher aka Book Code

With book ciphers, the ‘key’ used to encode/decode the message is an existing text, document, book or alike. Let’s use a book for example. When the sender encodes a message using a book, the receiver requires to have the exact same book (version) to be able to decode the message, since it is the key used to encrypt it.

Encoding can be done in several ways and results in a series of separated numbers that refer to either words in the book or the starting letter of that word. The encoder, can refer to these words or letters in different ways, thus ending up with a variety of possible number sequences.

Beale Cipher
This is an example of a book cipher, where you encode a text, by translating each letter to a number that matches the starting letter of a word in the book. The resulting number for each letter is the word count from the start of the text in the book. So if you encode the letter ‘T’, you could start counting words in the book until you find a word that starts with a ‘T’ and write down that number you counted to get to that word.

Other variants

  • Full Words: Instead of using only the starting letter, you could actually use complete words. So if your message to encode contains the word ‘THE’, you could count from the start of the book text, until you find this word and use this counted number for your encoded message. This may be faster and even easier to do, but it requires a book that has all the words contained in the message you wish to encode.

  • Different Count Method:
    When using a book to encode your message that contains a lot of pages and sections, it is possible to use a different counting mechanism. The receiver of the encoded message needs to know the counting mechanism used. You could for example not count words from the start of a book, but use a decided on set of numbers, referring to the Page, Line and Word for example. This would result in sets of 3 numbers instead. This could still decode into the first letter of that word, or possibly the full word.

How to recognise a Book Cipher
They can be recognised by the fact these only contain numbers, which are separated. These numbers are often higher than for example 26 (amount of letters in the alphabet) and can go even into the thousands. With truly high numbers being used, it is therefor likely a big ‘book’ and less likely to have a different count method where sets of numbers are being used.

Decoding a Book Cipher
Requires knowing the book, text, document or similar, being used. This is called the key. Counting is required, although tools can sometimes be used, but may often be limited.

Note

I agree that discussion of specific cipher methods may need their separate topic within the #knowledge-center


#7

Well, decimal is kinda obvious, that’s just any number.
Binary is any combination of 0 and 1 (01101101).
Hex is 0~9 + a~f where each position represents a number between 0 and 15 (f). Notice that this also means that 10 in hex is actually 16 decimal and 20 hex is 32 dec.
octal is not usually used much, but only contains the numbers 0~7.
Notice that these systems are all used to represent numbers using a different base (or radix: https://en.wikipedia.org/wiki/Radix ). Recognizing them is not always easy and generally, in puzzles like this people make assumptions:
only 0 or 1, it’s binary,
only numbers, it’s decimal
letters in your numbers, but only in the a~f range, it’s hexadecimal.
This doesn’t need to be true, since 123901541 could be either decimal or hexadecimal, but in puzzles it’s most commonly as the above assumptions show.
I should also mention Base64, it is a special variety where each position in the number can represent 0~63. usually by using A-Z, a-z, 0-9 and 2 more characters. Base64 numbers sometimes end in either = or ==, in which case you can easily identify them, but might also look just like “aJ3PsQ” in which case you will just have to try and see if it decodes into anything.
A site like: http://www.asciitohex.com/ can help you when trying to see if a number has any meaning as either hex, base64, binary or just decimal.

Morse then is anything that can be used to represent “long” and “short” signals. Typically, —…-.- or any variation there of. _–__, etc

Finally, any encrypted texts one encounters are almost first tried as Rot13, and then as any of the other 24 shifts.

If you find a password (or phrase) in the form of an alphabet (such as “SCHRODINGEABFJKLMPQTUVWXYZ”), it is usually used to make a polybius square https://en.wikipedia.org/wiki/Polybius_square and usually this means it is one of the following: ADFGVX cipher, Nihilist cipher, bifid cipher or fairplay cipher. (Trial and error will determine which.)
Notice that the polybius square only has 25 places, usually either the X, the Q, the I or the J is dropped. The default for a lot of these (as it was in our schrodinger example) is to treat the I and the J as the same character, and thus drop 1 of them (usually the J), you can easily see which parts of your text were supposed to be J’s and which are properly I’s. Also notice that these ciphers ignore all interpunction (spaces, dots, commas, etc).

Now there are more forms of ciphers, but so far they’ve not been shown in this ARG yet. Typically the forums used in online puzzles are all encryption that was used in times before the computer existed. As you can imagine, a bifid cipher would be difficult to decode without using a computer. Even if you had the key, you’d still need some time to write it all out by hand.

A final note on ciphers or hex/binary/etc, it does come down a lot to just trying to decode it and see if you get any meaningful text. You can sometimes see that something looks like it could be written in X, but until you try to decode it, you won’t know for sure.

Finally, a note regarding brute-force attacks, since I’ve seen them mentioned once or twice now on these forums. A brute-force attack is simply trying all possible combinations instead of actually using (or even knowing) the key. An example of it can be easily seen when using the ceasar cipher, for example if you go to this page: http://www.dcode.fr/caesar-cipher and select the “Test all possible shifts (brute-force attack)” option, and then pick “decrypt”, you will see all possible solutions on the left of the screen. For a human it is usually easy to instantly pick the right answer from all the random letter mess.

Now this is also be tried with more complex ciphers such as bifid. However, instead of “just” 25 possible keys, a bifid cipher can have 262524…etc (or 26!) which is about 4.0329146e+26 possible outcomes. (e+26 is basically saying, move the . 26 places to the right) That’s a huge number. But a computer can do a lot of attempts quite fast (actually, it will still take forever, but just to illustrate the idea of brute-force). since you’re not going to check all those answers yourself, you need a computer to check the texts it gains from random attempts against a dictionary, and if more than half appears to be real words, it should show the texts as a candidate. Like this you can do a lot of attempts in a relatively small time. You can even speed it up more if you know (or suspect) the password is an actual word, in which case you can just let a computer try all English words (about 600k of them).

The actual time you spend on a brute-force attack really depends on your own system. As a technicality, manually trying a lot of random possible passwords is also considered brute-force by some even though it resembles “guessing” more in English. :slight_smile:

Finally, as you can see from the above example, some ciphers are more vulnerable to brute-force attacks than other. Rot13 will give in quickly, but bifid is going to be quite tough to crack. Generally speaking, any passwords that are more than 8 or 9 characters long will yield too many possibilities to easily break using a brute-force attack.

Edit: Added Base64


#8

If anyone has any questions on how Vigenere, Affine/Atbash and Bifid ciphers work, be sure to ask me since I’ve spent quite a while programming them over the past few months.


#9

Thanks for the very well formatted answer @DevilinPixy. I’ve added it to the OP!
I’ve added yours as well @Shota. Nice note regarding brute-forcing too!
@ScienceofArt I also added your site. I’ve found CyberChef to be very useful too.


#10

You want tools? I got tools for you.


#11

Oh boy, those are some serious tools you have there


#13

Although I do agree it is a nice list of tools, this topic is focused on ciphers specifically. Maybe we should have a separate topic for other ARG related tools, possibly split by type, similar to that list?

-Could not find a way to reply to the correct person instead, so deleted previous post. Now it is too similar to that post to be directed at the correct person … Hope this is enough of a change to get it posted-


#14

offtopic:
if you want to specifically mention someone, use @DevilinPixy (or the name of whichever person you wish to refer to.


#15

I know, but I already used the ‘reply’ functionality, but the wrong one. This resulted in no longer being able to change this.
See the topic I created in the #knowledge-center here:


#16

Aye, well, personally, I’d say just editing the reply to mention the intended person would be the closest thing one can do. Changing who you reply to is like changing which topic you reply to. (Though there might be a hidden way to do it, I am not aware of any.)