As I was getting ready to enjoy a couple relaxing hours this evening on the 8th of July, 2019, a notification popped up via KDE Connect from my phone. Ordinarily, if it’s an email (which this was), I’d ignore it and go about my business. But something caught my eye: It said “invoice” somewhere in the text and also mentioned “GoDaddy.”

Puzzling, I thought, because I have all GoDaddy-related emails go to a separate folder, and they typically say nothing about an “invoice” in the message text. I quickly clicked on my email and, there, at the bottom of the window, sat a new message from PayPal with the title “Invoice from GoDaddy” with one of my domains in parenthesis at the end.

Before I continue, I’ll confide a small secret: I panicked. Oh, yes, I panicked. I don’t know why, because I use a password manager for everything, and 2FA where possible, but there’s always a small seed of doubt lurking in the darkness, desperately trying to convince you of the worst.

Escaping from my brief delirium, shortly after rationality finally kicked back in, I thought to myself “Ah-hah! It’s most likely this is a phishing email! This is the first one I’ve received in quite a while!”

I won’t deny that I felt the pangs of confidence–and a healthy sprinkling of arrogance–as I clicked through to examine the email headers in their entirety. Of course it was going to show up as an email that neither originated from PayPal nor from any reputable email service except, perhaps, from a hacked account being exploited for spam.

As I scrolled through the DKIM signatures and the SPF validation, not to mention the SMTP exchanges that clearly identified this as a legitimate PayPal emailing, reality set in. This wasn’t going to be quite so simple as an email scam. This was, in fact, a legitimate mailing from PayPal themselves.

Now, I’d be lying to you if I said that I was completely free of my panicked state. Nay, it returned, with somewhat more strength, to concern me even more deeply that perhaps my PayPal account was victim of an as-yet unknown attack or exploit. Quickly, or as fast as fumbling and vaguely worried hands could manage, I logged in to my PayPal account. There, at the bottom of the activity list, was an invoice–for $56.00 USD.

First, I’ll point out that this is just an invoice. It doesn’t mean that any money has exchanged hands. Yet. But it was still cause for alarm, because someone had decided it might be cute to exploit trust and the general imposition people feel toward settling outstanding debts for services rendered. To say this is a scummy, disgusting practice would be something of an understatement.

However, here’s where the scammer made a couple of critical mistakes (ignoring the more obvious ones–more on that momentarily). Of these, the most obvious was their account name on the requested transaction: It was written in Russian. Second, the string they used for “GoDaddy” did not match what GoDaddy actually uses for their billing statements. I don’t expect most people would consider the latter until it was too late, but I think the Russian name might’ve been something of a flashing neon sign that really ought to give pause for thought.

There were a couple of other clues that immediately shouted “SCAM!” (in capital letters), but they might not be helpful toward potential victims that have dozens of domains or are otherwise pressed for time and simply cannot consider these alternatives. The first of these was the timing. The domain they were targeting was indeed up for renewal, but they missed the expiration date by one day. I had already received an email from GoDaddy about the pending (automatic) renewal several days before and had it floating around in the back of my head. This invoice was therefore something of a surprise. As such, considering this background provided an immediate indication that something wasn’t quite right. The second was that all of my domains automatically renew. I don’t receive invoices from GoDaddy–only receipts. Oh, and finally, I don’t use PayPal to pay for my domains.

Admittedly, that last one was something of a dead ringer for potential scam (or a cracked account) material.

Before doing anything, I immediately started scouring the Internet for clues. Surprisingly, I couldn’t find anything about fake invoices from GoDaddy. I found some from buyers looking for shoes (of all things), and dozens of examples of phishing emails. This wasn’t a phishing email–this was a legitimate notification from PayPal informing me of an invoice that had been fraudulently sent. So, I did what any self-respecting (lol) person would do in a time of abject puzzlement: Take to Twitter.

It didn’t take me long to find someone else complaining to both the GoDaddy and PayPal Twitter accounts about receiving an invoice for $47 on a renewal that wasn’t up yet. I replied, suggesting that it might’ve been a scam, and that I received something similar.

Of course, I don’t know that the Twitter user in question was complaining about a fraudulent invoice. They didn’t provide enough information to deduce whether or not there was anything off about the invoice they received. But hey, why not offer it up as a possibility?

About 5 minutes later, I had a notification waiting for me on Twitter. It was PayPal’s support account asking for details via DM. I’m still a bit shocked in retrospect, to be completely honest, because I didn’t expect to hear from anyone much less one of the companies in question. I certainly can’t complain, either.

As expected, they asked for account information, location, and the nature of the issue. However, they also asked for screenshots of the offending invoice (couldn’t they see it?). After a brief back-and-forth, they strongly recommended I report it to their abuse department. I was quite pleased with the immediacy of their interest, but it remains to be seen what happens with the abuse report. (I’ll have to wait until later in the week for a reply, if any; I’m not holding out much hope.)

But the saga doesn’t end there.

I’ve heard mixed things about GoDaddy’s customer support. I’ve had a wide array of experiences myself but limited mostly to their sales department (they’re rabid up until the moment you turn off the whole “I’d like to be contacted for sales purposes,” which was apparently re-enabled at some point in my account’s history). I mused for a while about whether GoDaddy should know their name was being exploited for the gains of less savory individuals. I strongly considered against it, I won’t lie, but my conscience got the better of me.

I loaded up their web chat and almost immediately got in touch with one of their support representatives. She (I’ll assume it was a she, based on the feminine accents on the name; if not, for privacy, we’ll just roll with it) asked for my name and a description of the problem.

Well, this was awkward. I hadn’t really thought that far ahead, because the problem wasn’t really a problem with GoDaddy. It wasn’t a problem with my account. It wasn’t a problem with my domains, customer service, or any particular product offering. I told her as much. The problem was weird, I can’t deny that, but I felt someone needed to know. Even if it didn’t matter, at least I could sleep better at night knowing that I tried to do something about it. After all, I can’t be the only one targeted in this scam. What if someone else were to fall for it?

I explained the issue, and she quickly escalated the ticket through the account verification process, and then asked for some additional information. I explained a couple of times that the problem wasn’t with an account or domain per se so much as it appeared to be a new-ish scam, and that I mostly wanted to report it for my own satisfaction.

We went back and forth with a couple of relevant questions, and then she asked for a copy of the scam email. I was somewhat surprised, because I hadn’t exactly received a scam email from anyone. I asked if she meant the PayPal message; she said yes. So, off went the PayPal message (as an attachment to preserve headers), and I asked if she would like screenshots of the PayPal account pages with the invoice. Much to my surprise, she also wanted copies of those.

At this point, I’ll be honest. I don’t know what good any of this is going to do. I do know that the GoDaddy support representative was incredibly helpful, and she seemed genuinely interested in my concern (even going so far as to say “You are such a responsible person” to sate my worries). I was a bit taken back by her kindness, to say the least.

What surprised me with this whole ordeal was GoDaddy’s interest in the problem. They weren’t the ones who were dealing with invoices to third parties masquerading as someone else. They were merely the third party whose name was being exploited to commit fraud. It remains to be seen if PayPal expresses concern outside social media. I hope they do, but for now, it’s been awfully surprising to me that I received far more customer care from a company who couldn’t do anything about the problem. (I say “couldn’t,” even though technically they could, as the scammer was using their name and logos–i.e. trademarks–without permission.) Nevertheless, PayPal’s social team responded to me very quickly, so they at least get a few points for expediency.

All things considered, I feel the night ended on mostly positive terms. The initial shock of receiving a fraudulent invoice that wasn’t via a phishing attempt certainly took me by surprise, but in the end, the positive experiences with a random customer service representative probably half-way across the world expressing concern and compassion for others who could become victims of this scam more than made up for it. It’s a reminder that no matter how big a company is or how variable its reputation is viewed by customers on the receiving end, there are still humans who work for them. Sure, there are humans who typically see it as just another job. That’s normal.

However, no matter how rare it may be, it’s worth noting that there are those who see it as their duty to help. It may be woefully uncommon in our society today, but there are genuinely people who want to do the Right Thing™.

I don’t expect I’ll ever know if the representative who helped me managed to escalate the ticket and share the information about this scam to others who might be able to do something. Even if they could, there isn’t anything GoDaddy could do about this scam in the first place. This is clearly in PayPal’s sphere of influence, but perhaps if they know about it they can inform their customers when they inevitably receive the calls asking “Why am I receiving this invoice?”

I sometimes encounter the question “should I use a VPN?” with the inevitable shower of comments along the lines of “yes, it’ll make you more secure!” or “it’ll protect your privacy!” Occasionally, I see VPNs recommended as a solution against doxxing, such as when someone comments about their profession or business, competitors, or potential employers. Perhaps someone in industry has quipped about sending “anonymous” emails criticizing a particular organization or offered unsavory political opinions that would otherwise get them fired.

First, I should state that I am no security expert. I just happen to write software, and I have a vague curiosity into the world of information security. I enjoy reading the opinions of individuals who are considered experts in the field, and they almost uniformly warn of the same folly: VPNs are no panacea!

I think this advice is offered as therapeutic more than curative. In particular, it seems plausible users attracted to VPNs may place unwarranted trust in the software and provider, engaging in activities that suggest a degree of carelessness. Caution is nevertheless a desirable trait even under the warm embrace of cryptography. I’ll explain why.

A VPN may be useful to disguise your activity if you’re posting on Twitter, and you wish to avoid the danger of clicking on links that may be able to collate information about your activities or track usage behavior. VPNs may also provide some limited protection if you’re prone to torrenting your entertainment (up to a certain dollar amount, after which legal recourse against you becomes economically viable–maybe even necessary). However, even in the latter case, use of VPNs is of dubious utility, and they may not always keep you anonymous. Just this month (April 2019), NordVPN has been the subject of increased scrutiny over sending information to a series of unusual domains (billed as an anti-censorship strategy). Three months ago, NordVPN was also accused of tracking its users. None of this is surprising. Usage of a VPN is surrendering your privacy to a single firm (in most cases) in the hopes they will protect you from others doing naughty things with your browsing habits while simultaneously doing nothing of the sort themselves.

Nota bene: This behavior isn’t limited to NordVPN. They’re simply one of the most popular providers and therefore examined by more people with an equivalent increase in negative press. Regardless of intent, I find I can’t fault them for running analytic tracking on their user base: There are cases where traffic analysis (latency, throughput, timeouts, etc) may be useful for providing better quality-of-service and improving customer experience. In the event of an endpoint failure, I’m sure such analysis can be incredibly helpful re-routing packets toward other endpoints within a margin of acceptable latency. If the unusual domains their applications have directed traffic to this month is an anti-censorship countermeasure, I have to commend them for a bold strategy, even if it makes a few users nervous. To be clear: I neither endorse NordVPN nor am I overly critical of their decisions. I don’t care either way. I don’t use VPNs.

Now we can get to the meat of this discussion.

I believe the most important consideration as a user of a VPN service is to quantify your threat model. To illustrate, let’s take an example from earlier: For most people, doxxing isn’t a significant threat. Those at greatest risk often draw attention to themselves, either deliberately through their actions (whistle-blowers), or through online interactions (gaming, comments, etc.) that turn sour. Some may be victims of cyberstalking. In these cases, a VPN may be useless, because the victims often post sufficient information online to identify who they are, where they live, and numerous other details about their lives that a determined third party can piece together. Simply put: VPNs aren’t magic and they cannot protect you from yourself.

For most of us, our opinions and online interactions aren’t important or interesting enough to attract attention. If you think your opinions are interesting enough to be the target of a harassment campaign, then perhaps a VPN may be useful, but it isn’t the only tool you should rely on. To put it another way, if you’re afraid you might be identified online, you must firewall everything about your life that may be exposed through writing, your interactions with other people, and the media you post.

Ask yourself this question: What’s your threat model?

Most of the people I’ve spoken with who espouse the use of a VPN do so because they’re concerned about their identity being leaked, they may worry about employers identifying them online, they don’t want to become targets of harassment, or they simply wish to share politically unpopular opinions online that might draw the ire of one group or another (this may cross over into any of the prior points). As a free speech advocate, I can sympathize with their desire for further anonymity; losing your job because you’re the subject of a targeted harassment campaign is the antithesis of a free society. Neither should people be subject to hecklers or harassment, especially of the sort that crosses over from the online world to their front doorstep. Unfortunately, this is the world we live in.

A VPN isn’t going to provide unlimited protection against adversaries, and neither will a VPN protect users from disseminating information about themselves to interested but malevolent third parties. They can provide an additional layer of security when using Internet connections in a public location (airports, hotels, coffee shops, etc.), and they may be able to circumvent regional restrictions on entertainment or information (Google, YouTube, Netflix) by the state or licensing institutions. You should not expect a VPN to keep you completely anonymous, but they may be useful as part of a defense-in-depth strategy.

However, cautious use of the Internet can bring you 80% of the way toward a safer online presence. In particular: Don’t click links you don’t trust; avoid sharing secure information with services that are not offered via TLS (HTTPS); if you reply to an unknown third party via email, be cautious of using SMTP with your provider (this may divulge your client IP) and stick with the web or mobile interfaces; and don’t post information about yourself you don’t want publicly accessible. You may not have a choice in some cases, depending on your line of work, so this advice may not be applicable. I do believe it is broadly useful for the majority of people. Take heed.

There are limitations to VPNs that less technologically-inclined consumers may not be aware of. Key to understanding this is to understand the technology behind VPNs (typically IPsec with some authentication layer) and their history as a tool to extend company or school network boundaries off-premise, providing employees and students a means of connecting to internal services. It was never designed as a mechanism preventing the controlling institution (in this case the VPN provider) from classifying or logging traffic. Partial anonymity is a useful side effect but it wasn’t the design goal. Neither was complete security.

VPNs can have surprising utility if your adversary is intermediate tracking, or you don’t trust your ISP. Providers like Comcast have demonstrated this by injecting advertisements into sites their users visit, and others have been accused of using traffic analysis to track user behavior, possibly for targeted advertising. VPNs can protect against this threat by acting as a secure tunnel between your computer and your VPN provider’s endpoint.

Before I conclude this post, I should leave my readers with some particularly interesting tidbits of research that may be helpful in deciding whether your use case justifies paying for a commercial VPN. There was a paper written in 2016 titled “Characterization of Encrypted and VPN Traffic using Time-related Features.” This paper discusses techniques in traffic analysis to determine the protocol and type of traffic transmitted over encrypted connections, including VPNs, and could differentiate between VoIP, browsing, or streaming behaviors. There are other related papers including “Realtime Classification for Encrypted Traffic” (2010) and “Analyzing HTTPS Encrypted Traffic to Identify User’s Operating System, Browser, and Application” (2017); the latter describes attacks capable of defeating countermeasures intended to obfuscate payloads in transit. Although I cannot find it at this time, I also recall reading a paper that presented deep packet analysis techniques to defeat random noise injected into streams, successfully categorizing the encrypted traffic despite efforts to thwart would-be adversaries. This is an area of active research, and I expect with advancements in deep learning and greater access to GPUs capable of training neural networks tuned toward traffic analysis, VPNs may not present significant defense against adversaries that couldn’t already be achieved via other forms of encryption, e.g. TLS. Yes, I am aware of SNI-related information leaks due to how TLS presently works.

To put it more succinctly: You have to decide on your threat model.

A spurt of curiosity this evening–more specifically, one of those circumstances we each have from time to time wherein a handful of unrelated thoughts flutter about the conscious mind like a pair of butterflies flitting from flower to flower–consumed me sufficiently that I decided to do a brief Google search on prepared statements. I’m unsure where such a motive originated, but I’m fairly convinced that it was at least tangentially related to some misinformation I’ve heard of late related to web programming advice and also possibly due to my surprise that few commercial PHP bulletin board packages actually use prepared statements.

Before I begin, let’s consider for a moment that last and most disconcerting statement: Few commercial PHP forums use prepared statements. To the uninitiated, this might seem to be a matter of nick-picking unimportant to the real world. To the rest of you, it may come as a sad commentary on the state of modern programming and commercial software (perhaps, fittingly, as a commentary on the average run-of-the-mill PHP programmer). Prepared statements certainly aren’t new, and while they’ve been a part of PHP for a number of years now, it’s infuriating that they hardly see common use.

PHP first shipped PDO with PHP 5.1 (available as a PECL extension for PHP 5.0, circa 2004-2005). Intriguingly, for systems that don’t provide PDO support (or the appropriate drivers for PDO), the MySQLi and PostgreSQL functions and classes have provided prepared statements for quite some time, and the SQLite 3 drivers have provided a prepare() method since PHP 5.3. Commercial bulletin boards, like vBulletin and IPB, have seen many revisions in the years since, and several free/open source packages including phpBB have been part of similarly major overhauls. Yet the overwhelming majority of them still make no use of prepared statements. Humorously, as of this writing, vBulletin does provide a misleadingly-named sql_prepare method in its database class, but it doesn’t emulate prepared statements–it simply provides an escape wrapper with data type introspection and casting.

PDO has been available for nearly 8 years and many RDBMS drivers for PHP have offered prepared statements for at least that long (longer in the case of PostgreSQL if memory serves correctly). Yet every year or two, new major versions of popular PHP message boards are released, and every major release sees the same legacy database code under the hood. Perhaps it’s intentional. Perhaps the developers still want to support PHP 4.x in spite of the fact that it went EOL in 2008. Perhaps they just don’t know any better. Who knows!

Why Prepared Statements?

A prepared statement or parameterized statement, as it is occasionally known by in DBA parlance, is a specially-formatted SQL string that utilizes placeholders, either question marks (?), special named parameters (such as “:name”), or other database-specific strings, to indicate to the database or the driver where data is to be inserted. This has the benefit that, in theory at least, any data managed by a prepared statement is unlikely to serve as a vector for SQL injection attacks. The reason this works is because most drivers dispatch the prepared statement and its data separately on the wire and process them independently providing a certain degree of isolation. But wait, there’s more! Because of the implementation nature of prepared statements on most platforms, the query planner can often optimize and partially compile the statement such that, if it runs again, much of the legwork has already been completed and the query can run faster. Software like forums or blogs often execute the same query multiple times–with different data–so one might think it would be a natural fit. If it’s such a good thing, why do so many popular packages forgo such a benefit?

While I can’t answer for many developers, I think I know what at least part of the answer might be. First, for enormous code bases like vBulletin (and phpBB to a lesser extent), virtually no effort is made to separate the application logic from the underlying model. I’ll be fair in my distinction: The presentation layer is thankfully separated from the mess in the form of templates, but the remaining code is a bowl of spaghetti not unlike that of many of the very first PHP applications (and Perl!) that first graced the Internet over a decade ago. Because the model (and, by extension, the SQL) is so deeply entrenched in the functional logic of the application, reworking it to use prepared statements–and consider, also, that many of these queries are generated programmatically–would be a tremendous undertaking of many man-hours. Cleaning up the code properly such that it is more of a structurally sound framework (think MVC) is most certainly out of the picture. It isn’t impossible, of course, but when you consider that some functions in many of these software packages have persisted since the dawn of time, such refactoring becomes the thing of fairy tales.

To illustrate some of my displeasure, vBulletin version 4.2 still provides an iif function which is little more than a wrapper for the ternary operator (?:) in PHP. The ternary operator has been around since at least PHP4, yet there it is, in all its glory, a legacy function still available from the early days of PHP3 when such a beast didn’t exist!

One might think that it would simply be a matter of adding some logging code to old function calls, tracing the source that called them, and then reworking the culprit code to use built in language features. It might even take less than an afternoon.

The Drawbacks Programmer Mistakes

While prepared statements (parameterized queries for those of you who are embarrassingly excited by more elaborate verbiage) aren’t a panacea (I did it again) for all things SQL injection-like, they’re a good mitigation strategy, but it’s important to use them with caution. As Jason Lam states on the ISC Diary, “I still remember 4-5 years ago when SQL injection just started to become popular, the common mitigation suggested [was] to use prepared statement [sic] as if [they were] a magic bullet. As we [now] understand the SQL injection problem better, we realize that even prepared statement can be vulnerable to SQL injection as well.”

Well, yeah. This is where I smack my forehead. Maybe I’m being overly critical as I re-read a post from 5 years ago, because I’ve had the tremendously good fortune of witnessing some magnificently terrible code in my time as a web application developer.

Mr. Lam goes on to explain the insertion of unchecked user input, but I can’t shake the feeling that there is an implicit overtone in the article that it is somehow the fault of prepared statements. Perhaps more accurately, the article is faulting most of us for having championed prepared statements as a welcome solution to a very common and widespread problem. Realistically, though, it’s not an issue with prepared statements–they work just fine. It’s an issue with developers inappropriately using the tools at their disposal and doing so in a manner that simply transfers the vulnerability from query() to prepare() by forgetting to properly manage incoming data. Though, I should say that I’m inclined to suggest that programmatically assembling a prepared statement is somewhat counter-productive. More on this later.

Ironically, while doing some research for this article, I ran across a couple of posts on Stack Overflow that presented this problem of unchecked user input as one of the primary drawbacks of prepared statements. Really? Drawbacks? If you’re not using named parameters or placeholders for your query data, you’re probably not using prepared statements correctly! But drawbacks? Gee, maybe we were a little too vigilant in telling people to use prepared statements–so much so that they did a find/replace for query and swapped it with prepare. (I’m being facetious; so, to head off any comments to the contrary, it’s not at all possible to simply swap some text, because prepared statements do require a little more work.)

The problem I have with labeling unchecked user input as a drawback of prepared statements is that it is no longer a “real” prepared statement whenever such data is concatenated with the resulting query. Yes, it is still a prepared statement, insofar as calling prepare() on the driver’s end, but it’s no longer being used like a prepared statement. Here’s a hint to new developers, particularly PHP developers since a huge percentage of them are guilty of doing stupid things like this: Never concatenate unchecked input in any query–prepared or otherwise. If you’re using a prepared statement, use it like a damned prepared statement. The moment you start piping data into the query string itself, it’s no longer going to have the benefits of a prepared statement. (I’ll give you a special exception if you’re using LIMIT and concatenating integers with your queries since not all of you may be running MySQL 5.0.7 or later.)

Will the Real Prepared Statement Please Step Forward?

In my mind, and trust me, it’s a very strange place in here, a prepared statement is one that may contain parameters and is “prepared” ahead of time for reuse (that is, compiled) by the driver or the RDBMS (usually the RDBMS). Nothing more, nothing less. The instant some unfiltered data is slapped on to the end of the query, it’s no longer a pure prepared statement; instead, it becomes a mistake. Again: Prepared statements are parameterized queries that are usually compiled by the backend for a little extra speed. A query can contain anything else that the programmer adds into it, but fundamentally, a prepared statement is something that dictates a very specific structure. It certainly cannot overcome the mistakes of a naive developer who, believing that a prepared statement will magically fix all of their (singular they, sorry linguistic purists) security-related woes, use such a tool in addition to dangerous techniques like concatenating unchecked input. Another way to look at it is thus: If prepared statements are prepared (that is, compiled) by the database for reuse, and the developer is concatenating a dynamic value to the statement, the entire benefit of preparing (compiling) that statement is immediately lost, because the RDBMS has to re-compile the statement every single time it’s sent along the wire. Please, don’t do this.

Of course, there may be reasons not to use prepared statements all the time. For one, prepared statements in MySQL versions prior to 5.1 can no longer be managed by the query cache which may impact performance (High Performance MySQL, 2nd ed., p229). DBMSs that don’t support prepared statements can, in PDO at least, can have them emulated by the PDO driver at the cost of some pre-processing performance, and using older PHP functions like the popular-but-now-deprecated mysql_* ones just outright don’t support anything but basic queries (they also don’t use the binary interface, making them somewhat slower). If you’re using only a single query with absolutely no intention of reusing it, prepared statements may incur some overhead since the query must be compiled. Furthermore, for MySQL at least, if you’re not using stored procedures, the database has no way to share compiled prepared statements among multiple connections. Yet, while a prepared statement is no substitute for caution–particularly with programmatically-generated queries–it is a useful tool in the developer’s arsenal to protect against attacks like SQL injection. Plus, if you make it a habit to use PDO (you should), not only do you get emulated prepared statements for databases that don’t support them, you also get to use the modern MySQL APIs under the hood and some consistency, which says a lot in the PHP world.

Tangentially, this is also why it boggles my mind that many sites (banks, news agencies, airlines, and even some game companies) limit what characters the user can enter for their password, and how so many companies with an online presence often have draconian limits of less than 16 characters, inclusive. Seriously: If you’re correctly storing a secure hash of the password (HMAC, bcrypt, scrypt, or at least SHA256 or similar), you don’t need to store the password directly, nor does it matter if the password is 5 or 500 characters. It’s going to be comprised of a fixed length of some limited set of ASCII characters representing hexadecimal numbers which can be stored without much fuss. The 1990s were over two decades away. I think it’s time we stopped writing code like Y2K is still a viable problem.

Also, let’s start using prepared statements a little more often.