Why Certain Special Characters Reduce The SMS Character Limit To 70

I recently noticed that the character count of a text message I was drafting on my iPhone suddenly changed from “x/160” to “x/70” (here’s how to display a character count in Messages, if you didn’t already know).

Perplexed, I turned to the Internet for an answer, and found one quite quickly on this MacRumors thread.

It basically boils down to this: An SMS may contain up to 140 bytes (= 1120 bits) of data. UK mobile networks use the GSM standard. The basic GSM character set is encoded using 7 bits per character, which allows for a text message to consist of 1120/7 = 160 characters.

It is only possible to represent 128 different characters with 7 bits. This suffices to capture all common English characters. A few additional special characters (mostly punctuation) can be specified using the basic character set “extension”, which requires 14 bits for every character in the extended set.

However, support for the vast majority of foreign language characters comes in the form of 16-bit UTF-16 alphabet. If you have a mix of English and foreign language characters in your text message, the entire message must be sent in UTF-16, which reduces the number of available characters to 1120/16 = 70 characters. This explains the phenomenon I was experiencing.

I know what you’re thinking: this sucks for those who text in languages other than English. Thankfully, the GSM standard has a solution called “national language shift tables”. In this scheme, several 7-bit character sets are recognised, each corresponding to the most commonly used characters of a particular language. The first four bytes of the text message indicate the specific character set to use, and the rest of the message (136 bytes, or 1088 bits) can be used for the actual content of the message, allowing for a respectable compromise of 1088/7 ≅ 155 characters.

Using characters that belong to multiple shift tables in the same text triggers a fallback to UTF-16, but the idea is to capture the vast majority of text communication.

If this has piqued your interest, Wikipedia has a fairly comprehensive article about GSM 03.38.


Civil Partnerships are Discriminatory

Civil partnerships are often incorrectly viewed as the panacea for reconciling the views of those in favour of human rights with the views of champions of “traditional” marriage. It’s easy to see the appeal: the same-sex couples get to enjoy the same legal benefits of marriage (which, by the way, they often don’t), and the tragically misguided get to cling to the vacuous notion that somehow, the “real” meaning of marriage remains sacrosanct. Everyone wins! Unfortunately, it isn’t that convenient.

The fallacy is concisely stated: having the same legal rights is not equality. The ostensibly well-intentioned civil partnership is a step in the right direction, but ultimately fails to satisfy some very basic, primitive human needs, and is therefore not a solution to the problem of marriage inequality.

But they’re different!

A common argument is that different things call for different names. This is nothing but a rehash of the separate but equal argument, attempting to hide the proponent’s homophobia under a thin veneer of semantics. It reaches for ad absurdum: in the interest of political correctness, why bother with semantic distinctions at all? Instead of having different words for “man” and “woman”, why not call them all “persons”?

In short, the reason it makes no sense to have distinct words to express two different kinds of marriage is because there is no great utility to be found in making that distinction. This is why we don’t have separate words for interracial marriage, marriage between old people, marriage between wealthy people, or marriage between unpleasant people, even though these are all different things. Can we move on now?

The word can only be marriage.

Words do not have meaning. Rather, they convey meaning, and what a word conveys is a matter of soft social convention. No single party can claim global authority on the definition of a word.

The word “marriage” carries social weight which makes it absolutely essential for complete equality. A marriage is not simply a contract which entitles the underwritten to certain legal provisions. Marriage means confidence in a relationship and the ability to commit to one. Marriage means willingness and ability to reconcile differences. Marriage means the opportunity for mutual growth and support.

A marriage is a publicly visible and recognised milestone in a relationship. It is a deeply ingrained human aspiration in civilised society. Like anything else that has heavy aspirational value placed on it, we are taught the worth of marriage through our upbringing, through our friends and relatives, through experiences and conversation, and through media. Everyone is searching for meaning in life, and we have taught ourselves that marriage is a most important and meaningful human experience.

The value of marriage is a complex creature, evolved through centuries and across several cultures, and realised through arbitrary conventions and rituals. A contemporary Western marriage, for instance, becomes much less meaningful without arbitrary rituals such as stag nights and honeymoons.

I understand that no one is stopping civil partners executing identical rituals in an effort to make their experience of civil partnership a better approximation for the marriage experience to which they aspire. However that is exactly the problem: no matter how painstakingly planned and executed, the civil partnership remains an approximation. The value of the experience, the life-meaning that the individuals can glean, can never quite match the aspirational value they have imbibed from society.

Depriving an entire category of people of an opportunity to obtain meaning in life is discriminatory. It sends a message that certain people are unworthy of feeding such an aspiration because of something in which they had no choice.

Our soft social convention for what the word “marriage” conveys is easily lenient enough to accommodate same-sex couples, and the legal definition needs must catch up.

Data Science and Other Diversions