Update The Neural Network that Generated Thousands of Technical Questions

One developer recently gave the world another example of a neural network doing what humans do — or at least, simulating what humans do. It generates artificial questions about computing programming by mimicking posts on Stack Overflow, a popular community Q&A site run by Stack Exchange.

The results were laughable, eerie, and ultimately thought-provoking — raising some real questions about what AI will be doing in our future…

Learning to Learn

“Stack Roboflow” was the brainchild of Des Moines-based developer Brad Dwyer, who’d been taking fast.ai’s online course on practical deep learning for developers.

“I tend to learn best by working on my own projects,” he explains on his project’s site, “and creating Stack Roboflow was a way to apply my learning to a novel problem.”

Using fast.ai’s open source library for deep learning, along with the open-source machine learning library for Python (PyTorch), Dwyer began teaching the neural network how to talk like a programmer. He started with Salesforce’s AWD-LSTM language model, which determines the probability of words in an English sentence based on sentences from Wikipedia pages.

But then using millions of questions from Stack Overflow as input data, he trained his neural network to speak geek.

Along the way, Dwyer had to overcome all the technical challenges faced by a one-man project — sometimes, even considering the use of cloud computing. For example, it took about 30 hours of training using a 2080 Ti graphics card, Dwyer explained on Reddit, though “I was only able to use ~1/16 of the data I had available.” The project’s page explains that he only had 32GB of system memory, though he’d calculated that it would take 700GB to hold everything — and then several additional weeks to train everything with his graphics card. This “100% ‘serverless’ site” used AWS’ Simple Storage Service (S3), the AWS CloudFront content delivery network, and [email protected]”

“I ultimately abandoned the idea of cloud training due to cost considerations,” he explained on his site. Though on Reddit he added that he hasn’t given up on the idea — if he can make it work. “I’d like to train on a Google cloud instance with more memory at some point but my preemptible instance kept getting terminated and the non-preemptible instances are too expensive for me to justify using on a toy project like this.”

Virtual Programming Questions

He got the site up and running — the questions are generated in real time and uploaded to the site on-demand and within a few days it received over 40,000 unique visitors, Dwyer reported on Twitter — and generated over 100,000 questions.

It’s a remarkably realistic simulation, posing questions like “How can I get the current time in C#?” At one point, it even generated a question about Fortran.

In a discussion on Reddit’s programming forum, Dwyer confirmed that Stack Roboflow had even spontaneously started generating the tell-tale tics of a real post — right down to including a sentence that starts with “EDIT: ” at the top of a question.

“It also learned to thank random people like Dave and even to sometimes sign its own name… which is apparently Charset o_O.”

On Reddit Dwyer clarified that his ultimate goal had been getting better at predicting the quality of a question — in order to then use that to train the AI to generate better questions. “The quality prediction model would also have other useful applications (for example, a Google Chrome extension that warned you if you were about to post a question on Stack Overflow that was likely to be downvoted).”

Dwyer shared some interesting observations on Twitter — including a word cloud comparing the most popular tags used on both sites.

Dwyer ultimately hopes to try training it on data from other Stack Exchange community Q&A sites.  His website notes that “a model trained on philosophy.stackexchange.com might ‘invent’ some interesting theories.”

In the meantime, he’s now offering free downloads of its pre-trained vocabulary, releasing it under a Creative Commons license — but the page has an optional pay-what-you-want donation button to support his next dream.

“I’m hoping to collect enough to pay for the cloud compute time necessary to train on the full dataset (which I will make available for download as well).”

He’s since added a page with some of the more interesting questions, plus an index of the site’s 100 most recently generated questions. And that page which also includes an Elasticsearch instance for search functionality (if you’re looking for AI-generated nonsense on a particular topic.)

Developer In-Jokes

Who is this guy? Dwyer is the founder of a mobile game company called Hatchlings. One of its apps instantly solves a Sudoku puzzle if you point your phone’s camera at it.

But it’s been fun to watch the larger developer community respond to this simulation of their own community. “There’s something almost unsettling about text that initially appears to follow a sort of internal logic, yet doesn’t,” posted UI designer Clayton Miller on Hacker News, adding “Some of the results read like a programming fever dream.”

Or, as a user named Rosser Schwarz posted, “It’s the Uncanny Valley of text synthesis.”

One commenter who claimed to have reviewed 1,600 edits at Stack Overflow noted that “some of the automatically generated questions are more intelligible than the average Stack Overflow question.”

Another saw a harbinger of a dystopian future. “I wonder if our current discussion boards on the interwebs can survive the coming influx of content like this and the next generations of it that follow.”

But at least one commenter said the questions looked strangely familiar. “This is also what every question looked like when I was new to programming.” Although some reactions included their own sardonic comments on the state of social media.

One Reddit user — a PowerShell and automation enthusiast — joked that the site could be adapted to create a technical interview from hell. “Give someone a few real questions from Stack Overflow, and then slip two or three of these in and see how they try to answer them.”

What Next?

Since its initial release, Dwyer appears to have made some improvements. Questions now have tags (leading to search results for questions with the same word), so you can compare its last 100 answers on specific topics like JavaScript or PHP.

But he still has more ideas in mind — for example, the ability to answer questions. “If I could get this working it’d actually become a useful tool instead of a fun toy.”

His project site shares some another future development goal — teaching it how to code. “I was surprised how quickly the model started to pick up a variety of different coding syntaxes. It seems to know the difference between SQL, C#, and JavaScript and is able to switch contexts based on the content of the questions!

“I think it’d be really neat to try training on some code off of GitHub and seeing if it could start to write code that actually runs.”

WebReduce

• Are we really within five years of flying taxis?
• The world now has robot baristas and robot car valets.
• San Francisco gets its own restaurant with a new hamburger-making robot.
• Humorous tech talk: “The cloud is a scam.”
• After waiting 40 years, “Dungeons and Dragons” suddenly becomes cool and popular.

Feature image by rottonara from Pixabay