[Technology] Elon Musk and Twitter

[jackalbion]

With twitter going, I'm gonna have to moan about Andy Naylor, post sopranos videos and moan to train operators on here.

 

[Guinness Boy]

thats pretty unlikely. its a simple app, with some very clever algorithm in the back, it could and should run with minimal intervention.

Hundreds or maybe thousands of pissed off software engineers and security specialists with a grudge. What could possibly go wrong?

 

[mikeyjh]

Saw a rather good theory for where Musk is going wrong. It uses something called the Clocks and Clouds theory (originates from Karl Popper). Boiling it down, there's two types of "problems" in the world: Clock problems, and Cloud problems. Clock problems are ones where you know when you've solved the problem, you can easily measure the success of the solution, and you can take the solution apart and put it back together again and repeat the solution easily. Just like a clock: the problem is you need to know what the time is. The clock solves the problem, and once built continues to solve the problem - and you can pull it apart, see how it works, put it back together again. Cloud problems are entirely different. You can't just pull apart a cloud and see how it works and put it back together again. Clouds are a whole lot harder to measure, harder to model, and don't permanently solve a problem. Cloud problems will often need to have their solutions revised, and the right solution might differ depending on how is using the solution or when they are using it.

Musk's problem is he's coming from a Clock thinking background. Building cars and rockets are mostly clock problems. Twitter, however, is a Cloud problem. He's dealing with social stuff that can't be solved through Clock thinking. But it's clear from a lot of his posts on Twitter that he's applying the same Clock thinking he applied to cars and rockets to Twitter.

Quite - His "my way or the highway" offer of 3 months pay is also an enormous error, the best, most confident staff are going to say thanks very much, I'm off.....

 

[MJsGhost]

thats pretty unlikely. its a simple app, with some very clever algorithm in the back, it could and should run with minimal intervention.

If that were true, you'd have to wonder why companies like Twitter pay so much for teams of highly paid IT staff to keep their systems up as close to 100% as they can get

 

[Bozza]

thats pretty unlikely. its a simple app, with some very clever algorithm in the back, it could and should run with minimal intervention.

It's really not.

It may look that way as an end user who primarily scans down a feed and makes the odd tweet themself, but there will be many, many facets that you'll be unaware of, not to mention all the back office administrative functions and interfaces, the advertising management engine etc.

And beyond the raw functionality, the sheer scale of twitter brings significant complexity.

If a lot of engineers carrying huge amounts of knowledge and experience walk out the door, it's not inconceivable that the whole platform could begin to creak.

 

[LamieRobertson]

He certainly knows how to motivate staff

Elon Musk: Twitter locks staff out of offices until next week

It comes amid reports that large numbers of workers have resigned from the social media platform.

www.bbc.co.uk

Effective immediately, we are temporarily closing our
office buildings and all badge access will be suspended.
Offices will reopen on Monday, November 21st.
Thank you for your flexibility.
Please continue to comply
with company policy by refraining from discussing
confidential company information on social media, with
the press or elsewhere.
We look forward to working with you on Twitter's exciting
future.

 

[deletebeepbeepbeep]

It is going to go down this weekend, probably once the WC kicks off, and then never come back up.

 

[dejavuatbtn]

He’s about as vile and arrogant as Trump. Who the hell would want to work for either of them. I hope they both implode.

 

[BBassic]

If a lot of engineers carrying huge amounts of knowledge and experience walk out the door, it's not inconceivable that the whole platform could begin to creak.

This is such a hugely misunderstood problem in tech.

Every single place I've worked I've had to deal with some piece of undocumented code written by someone decades ago. If it breaks everything falls over. But you're so busy either trying to stop it from falling over or working on new code (that someone else will have to maintain down the line) that you have no time to actually re-code and fix the damn thing.

And this is at small companies. I dread to think what sort of techbombs are lurking within the Twitter code.

 

[beorhthelm]

If that were true, you'd have to wonder why companies like Twitter pay so much for teams of highly paid IT staff to keep their systems up as close to 100% as they can get

well thats the thing, dont believe they need many. once set up and stable something like twitter needs very little oversight. i often what the legions of staff these companies do, and judging by some ticktoks its day care for millenial/gen z who got a tech degree. actual engineering on the main app would be minimal. i work in company with complex platforms that are supported by a dozen engineers for the core systems. then hundreds of devs for dozens of apps on top, and thousands of product owners, business analysts and project managers trying to break it with changes. when was the last functional change to twitter? question of course is if he can keep that core staff that know how things work, >90% the rest could go without interupting service.

 

[Bozza]

. when was the last functional change to twitter?

There will likely be numerous most days, either so small you never notice or applied to the vast swathes of the platform that you never see.

 

[Guinness Boy]

when was the last functional change to twitter?

Charging for blue ticks will be a functional change. Changing the "promoted tweets" as mentioned this morning on this thread will be a functional change. Getting rid of moderators may require more bot moderation unless he's going full Wild West. Not to mention the people who are needed to close offices and cut off email and network access for staff being got rid of.

And that's before you consider @Bozza 's answer, the likely spaghetti code called out by @BBassic and the fact that Musk seems to be pissing off entirely the wrong people who could probably bring the platform down in less than half an hour. Quite possible there'll be hackers after him too.

 

[Bozza]

The rollout of Twitter Blue / $8-for-a-blue-tick was an unmitigated disaster. Lots of people with lots of knowledge of twitter, how it performs and how bad actors were likely to behave on the platform told Elon this. He ignored them, rolled it out and then had to hurriedly cancel the whole thing as it exploded in his face.

From the excellent Platformer newsletter...

Days before the Nov. 9 launch, the company’s trust and safety team had prepared a seven-page list of recommendations intended to help Musk avoid the most obvious and damaging consequences of his plans for Blue. The document, which was obtained by Platformer, predicts with eerie accuracy some of the events that follow.​

​

“Motivated scammers/bad actors could be willing to pay … to leverage increased amplification to achieve their ends where their upside exceeds the cost,” reads the document’s first recommendation, which the team labeled “P0” to denote a concern in the highest risk category.​

​

“Impersonation of world leaders, advertisers, brand partners, election officials, and other high profile individuals” represented another P0 risk, the team found. “Legacy verification provides a critical signal in enforcing impersonation rules, the loss of which is likely to lead to an increase in impersonation of high-profile accounts on Twitter.”​

​

The team identified several other risks for which Twitter has yet to identify any solutions. For starters, the company lacks any automated way to remove verified badges from user accounts. “Given that we will have a large amount of legacy verified users on the platform (400K Twitter customers), and that we anticipate we’ll need to debadge a large number of legacy verified accounts if they decide not to pay for Blue, this will require high operational lift without investment.”​

​

.....​

​

Despite the warnings, the launch proceeded as planned. A few hours later, with the predictions of the trust and safety team largely realized, Musk belatedly stopped the rollout.​

 

[Audax]

This is such a hugely misunderstood problem in tech.

Every single place I've worked I've had to deal with some piece of undocumented code written by someone decades ago. If it breaks everything falls over. But you're so busy either trying to stop it from falling over or working on new code (that someone else will have to maintain down the line) that you have no time to actually re-code and fix the damn thing.

And this is at small companies. I dread to think what sort of techbombs are lurking within the Twitter code.

well thats the thing, dont believe they need many. once set up and stable something like twitter needs very little oversight. i often what the legions of staff these companies do, and judging by some ticktoks its day care for millenial/gen z who got a tech degree. actual engineering on the main app would be minimal. i work in company with complex platforms that are supported by a dozen engineers for the core systems. then hundreds of devs for dozens of apps on top, and thousands of product owners, business analysts and project managers trying to break it with changes. when was the last functional change to twitter? question of course is if he can keep that core staff that know how things work, >90% the rest could go without interupting service.

@beorhthelm The software to make Twitter work behind the scenes will be a massive tech stack. And unless they've been extremely careful over the years as they've layered in new features, there will be significant interdependencies in the code base. Setting aside Twitter itself, with such a large user base they will also have rather complex and complicated infrastructure to run the site - things like auto-scaling, alerting systems, etc etc. Cut the staff base by 90% (which is what some folks with good sources are suggesting has now become a reality) and it will be impossible that the remaining 10% have all of the knowledge needed to keep Twitter running reliably if something does go wrong.

I don't think Twitter will just explode and go down in a big bang. However, the more staff he drives out the door the more likely a catastrophic failure becomes. They're already seeing small scale problems (eg the 2FA went down, apparently there's issues with notifications, etc). The fewer experience Twitter staff they have, the worse the "brain drain" becomes. New staff coming in will be a) hard to find, and b) lacking any context on how the code works and where the interdependencies are. That runs the risk of a new staffer attempting to fix an emerging issue causing even bigger problems.

Layer in a big world event that could cause a big spike in traffic ... say, a football world cup, or Russia doing something really stupid ... and the risks rise rapidly.

 

[beorhthelm]

Charging for blue ticks will be a functional change.

a business function change, with some new backend system to manage payments etc. decison process leading to apply change to an account. front end app shouldn't need touching. likewise removing moderation doesnt change the app functionality. bad code usually keeps going until someone pokes around with it. could be wrong, just think something as large, long running and simple as twitter wouldn't be so flakey and fall over easily. a bad actor from inside/ex-employment quite likely though.

 

[Audax]

There will likely be numerous most days, either so small you never notice or applied to the vast swathes of the platform that you never see.

Guaranteed. My understanding is that Twitter (like Amazon) made significant use of test-and-learn processes where they would roll out lots of small changes to small numbers of users, see what impact those changes had, and then either revert them or implement them fully.

 

[Audax]

a business function change, with some new backend system to manage payments etc. decison process leading to apply change to an account. front end app shouldn't need touching. likewise removing moderation doesnt change the app functionality. bad code usually keeps going until someone pokes around with it. could be wrong, just think something as large, long running and simple as twitter wouldn't be so flakey and fall over easily. a bad actor from inside/ex-employment quite likely though.

Twitter is a lot more than a FE app. If the BE systems start failing, the FE app will also begin to fail.

For context: I'm a product manager for a team that works on a (smaller scale than Twitter) social media platform. Cannot understate how important it is to retain staff who know how the systems work. If we lost 90% of our staff who run our systems (FE, BE, infra, etc) then we'd not have enough expertise left to cover all areas of responsibility. Our platform would continue to run just fine for a while, but if the lost staff aren't replaced and develop the knowledge they need the risks of an outage would increase markedly over time. Not to mention that actual development would need to *cease* completely because of elevated risk of new releases causing cascading failures and not having the staff/knowledge available to recover.

Twitter is big, and while to the average user from the outside looks pretty simple - it won't be behind the scenes. It'll be complex, complicated, and a lot of the code running it will be old, legacy code that they wouldn't write the way it's written today but simply can't go back and re-write now it exists.

 

[Bozza]

a business function change, with some new backend system to manage payments etc. decison process leading to apply change to an account. front end app shouldn't need touching. likewise removing moderation doesnt change the app functionality. bad code usually keeps going until someone pokes around with it. could be wrong, just think something as large, long running and simple as twitter wouldn't be so flakey and fall over easily. a bad actor from inside/ex-employment quite likely though.

The "simple" bit is where you are going wrong.

One thing I've learned in my career - largely spent in systems development - is every business and system looks really, really simple from the outside.

Share registration: just a list of names, addresses and how many shares each person owns.
Electricity supply: just a list of names, addresses and how much power each home uses.
Childcare vouchers: just a list of parents, kids, nurseries and how much each parent has to spend.
Employee share plans: just a list of names and addresses and how many options each employee has been granted.


Twitter: surely it's just a list of accounts and tweets each one has made, right?

 

[Greg Bobkin]

Todd Boehly is preparing a $450bn bid for it, apparently...

 

[Garry Nelson's Left Foot]

If Musk tanks Twitter he'll lose $44billion, right? How does that affect him? He didn't buy it with all his money did he? Won't the banks, people etc who helped fund it be slightly unhappy if he deliberately ruins it?

On the flip side can't all those people who worked there and/or invented it just build a rival platform?

As you might have guessed I'm neither a finance nor tech genius...