60% of top threads are now spam.

[Omicron]

Here's a candid screenshot: https://dl.dropboxusercontent.com/u/44754370/spambots.png
Taken on Monday the 15th of September, 07:40 UTC.

Of the top 15 threads (below the stickies) in General FTB Discussion, 9 are advertising threads. That is no longer "sorry for the occasional spam post", this is "a solid 60% of our content is spam". This number is rising daily. Whatever you're doing against it, it appears to not be working. At all.

I wish I could offer suggestions, but I have no experience administrating forum software. =/

 

[ratchet freak]

Problem is that those spam bots are designed to circumvent the filters in this forum, and they frankly don't care about whether people click link, they just care about plugging it so web crawlers pick it up

One suggestion is getting more mods able to handle the downtime of the others

another is having the first few posts of each new user be audited to minimize exposure

 

[Omicron]

More moderators to squash the spam is treating the symptoms, not the illness.

The forum needs an adequate, effective measure to prevent automated registration. Most forums on the net succeed in doing so, including other large gaming communities. The problem here is that the spammers have sussed out and automated the login procedure of the FTB forums like Direwolf20 automates charcoal production. Unless the login procedure is changed and locked down, you will not be able to defeat an automated spamscript no matter the number of moderators you employ. The machine is always faster, and never gets tired nor bored.

 

[ljfa]

Aren't Captchas a thing?
Well, they can be circumvented as well

 

[HeilMewTwo]

I actually saw one last week that had an avatar. We may be getting spammed even more by smarter ones soon.

 

[bounding star]

my favourite captcha is the one that only spambots see, its hidden in the code so humans dont see it but the bots fall for it every time. also, a lot of the spam threads have the same url at the end, surely that can be picked up on

 

[ratchet freak]

my favourite captcha is the one that only spambots see, its hidden in the code so humans dont see it but the bots fall for it every time. also, a lot of the spam threads have the same url at the end, surely that can be picked up on

that url changes from day to day

 

[Vauthil]

I've spoken about this at length before, but I'll note some things again now.

CAPTCHAs are regularly cracked and trampled on. Once an appropriate "firing solution" is calculated, they're easily circumvented. Build a better mousetrap and better mice will show up. That's life.

The alternative, human verification questioning, tends to only function until somebody human can take 15 minutes out to quickly read the netted questions and provide functional answers. Any given set of questions tends to have a life of 3-6 months before the bots are back in.

Omicron is correct in noting that moderation is a last bastion of defense. This is why ratchet's suggestion of moderation queue'ing everybody's first few posts is actually even worse than the current situation. Once one of us has eyeballs on forum, we can eliminate a given spam wall within about 5 minutes. Make us have to review and approve every single new account's first few posts? That's not reducing the hands-on moderation requirement, it's increasing it in a manner that scales poorly and would fall apart pretty much as soon as the next registration surge happens (which coincides with the release of a new cluster of modpacks).

I'm still a strong fan of just banning URL/IMG links for the first 10 or so posts, but that has other complications due to other functions the site serves. I use that kind of filter on sites where I admin and it just means I do an account cleaning every year or so since accounts do stack up but do nothing since they just slam headfirst into a "nope" wall with every post.

XenForo 1.4 went public last week and, in addition to some updated CAPTCHA options, uses the hidden randomized honeypot methodology on registration, but the chances of seeing that update here are probably slim (that's a decision between people paying bills and people administrating the server itself, two groups I am not in).

 

[HeilMewTwo]

I'm still a strong fan of just banning URL/IMG links for the first 10 or so posts, but that has other complications due to other functions the site serves. I use that kind of filter on sites where I admin and it just means I do an account cleaning every year or so since accounts do stack up but do nothing since they just slam headfirst into a "nope" wall with every post.

Yeah but if they care enough to figure out the questions, couldn't they just program the bots to spam ten posts?

 

[Vauthil]

Yeah but if they care enough to figure out the questions, couldn't they just program the bots to spam ten posts?

You'd think so, but with the post delay function that's wasting a lot of bot time. Questions are easy because it's just one more routine to add in the sequence, making the bot have to idle for X minutes posting at intervals before attempting a URL post is problematic as bots for SEO tend to run as one-off processes and not as idle keep-alive sessions (we get the keep-alive "true spam" bots once in a blue moon, largely because they're ineffective at actually achieving their end goals). It also vastly increases the chance of interception prior to getting that URL link up in the first place, which is what they're aiming for: having the URL up at a moment where a search engine spider is crawling that post.

Once you increase the computational costs of running the bot effectively, they don't bother with getting more complex. There's easier prey out there and higher value targets to spend those kinds of resources on.

 

[HeilMewTwo]

You'd think so, but with the post delay function that's wasting a lot of bot time. Questions are easy because it's just one more routine to add in the sequence, making the bot have to idle for X minutes posting at intervals before attempting a URL post is problematic as bots for SEO tend to run as one-off processes and not as idle keep-alive sessions (we get the keep-alive "true spam" bots once in a blue moon, largely because they're ineffective at actually achieving their end goals). It also vastly increases the chance of interception prior to getting that URL link up in the first place, which is what they're aiming for: having the URL up at a moment where a search engine spider is crawling that post.

Once you increase the computational costs of running the bot effectively, they don't bother with getting more complex. There's easier prey out there and higher value targets to spend those kinds of resources on.

Hmmm, spam botting is more complicated than I thought... BTW I report every spam bot I see, does this get annoying or is it helpful?

 

[Vauthil]

Hmmm, spam botting is more complicated than I thought... BTW I report every spam bot I see, does this get annoying or is it helpful?

It's helpful. The first thing I see when I log in is the top left corner of my screen, where there are counters on things in the Moderation Queue (i.e. Server Promotion threads that need approval, usually) and Reports. If I log in and see a handful of Reports, I'll just go straight to handling them and then check the What's New list; if I see double-digits in Reports then I head straight to What's New and run a survey of threads from the last time I logged in, basically using Reports to "keep score" on making sure I'm being thorough enough with a clear-out. Other moderators may handle it differently in the specifics, but that's how I handle it.

What isn't helpful is when folks reply to the spambots. (1) The spambot doesn't care that you Reported it, once it has posted it is long gone and off somewhere else; (2) Everybody else doesn't need to know either as Reports are consolidated. If ten people Report the same post, it's a single Report with ten comments, not ten Reports; (3) Congratulations, replying to a spambot is one of the worst things someone can do because now you're making the post look legitimate to any crawlers that catch it before deletion, helping the spammer out.

I do keep meaning to screenshot the workflow on this some day just so folks can see how it all works. Problem is, we can't just bring anybody on board to do it because it is in fact granting the ability to delete whole threads and ban accounts.

 

[ratchet freak]

Hmmm, spam botting is more complicated than I thought... BTW I report every spam bot I see, does this get annoying or is it helpful?

from another post I can gather all reports on a post get collated into a single super report, and the mods typically do a sweep before tackling the queue anyway

 

[buggirlexpres]

\o/
Glad to see I'm legit.

 

[Omicron]

Gideon, you stopped being legit the moment you claimed you'd leave if Microsoft took over Mojang, while your title very specifically states "Never Leaves". Clearly you are an imposter who has replaced the true Gideonseymour with malicious intents!

@Vauthil - thanks for the explanations, I love hearing how things work under the hood.

 

[SatanicSanta]

I've spoken about this at length before, but I'll note some things again now.

CAPTCHAs are regularly cracked and trampled on. Once an appropriate "firing solution" is calculated, they're easily circumvented. Build a better mousetrap and better mice will show up. That's life.

The alternative, human verification questioning, tends to only function until somebody human can take 15 minutes out to quickly read the netted questions and provide functional answers. Any given set of questions tends to have a life of 3-6 months before the bots are back in.

Omicron is correct in noting that moderation is a last bastion of defense. This is why ratchet's suggestion of moderation queue'ing everybody's first few posts is actually even worse than the current situation. Once one of us has eyeballs on forum, we can eliminate a given spam wall within about 5 minutes. Make us have to review and approve every single new account's first few posts? That's not reducing the hands-on moderation requirement, it's increasing it in a manner that scales poorly and would fall apart pretty much as soon as the next registration surge happens (which coincides with the release of a new cluster of modpacks).

I'm still a strong fan of just banning URL/IMG links for the first 10 or so posts, but that has other complications due to other functions the site serves. I use that kind of filter on sites where I admin and it just means I do an account cleaning every year or so since accounts do stack up but do nothing since they just slam headfirst into a "nope" wall with every post.

XenForo 1.4 went public last week and, in addition to some updated CAPTCHA options, uses the hidden randomized honeypot methodology on registration, but the chances of seeing that update here are probably slim (that's a decision between people paying bills and people administrating the server itself, two groups I am not in).

We use QuestyCAPTCHA on the wiki, and have for at least 5 months, and have only had a single spam account created since then. Although, we also haven't had a lot of real users join, and have had to create accounts for people because the way the CAPTCHA software works is awful and broken, but hey, we don't have spam!

Basically, QuestyCAPTCHA asks you custom questions that the server admins have to set themselves. Unfortunately there are so many possible answers for each question that it's just unreasonable.

 

[Vauthil]

We use QuestyCAPTCHA on the wiki, and have for at least 5 months, and have only had a single spam account created since then. Although, we also haven't had a lot of real users join, and have had to create accounts for people because the way the CAPTCHA software works is awful and broken, but hey, we don't have spam!

Basically, QuestyCAPTCHA asks you custom questions that the server admins have to set themselves. Unfortunately there are so many possible answers for each question that it's just unreasonable.

That's part of what I refer to in speaking of "human verification questions". And yeah, having those barriers to entry are more permissible on a project site like a wiki.

Granted, some days I'd really like to institute some barriers to registering/posting here... but that's on the order of barring anybody from replying to an announcement/news post unless they have at least 20 posts, because entirely too many people register an account and immediately treat those posts like tech support/"talk about what I want to talk about even though it's irrelevant" threads.

 

[Celestialphoenix]

Would it be possible to automatically hide a post/thread if its reported [as spam?] X times?

Could a [new?] user's entire post count be hidden if Y number of their posts are hidden?

 

[Captainnana]

This will be mostly fixed when xenforo is updated but I've been told to update the main site first so this is going to have to wait