Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Biggest was accidentally omitting a WHERE clause in a DELETE statement when doing some data cleanup in a production database. Fortunately we had a nightly backup, regular snapshots, and the query-logs I could replay, but that was a heart-stopping couple hours.
#DevDiscuss
This thread is a gold mine for tonight's #DevDiscuss topic of mistakes.
https://t.co/dXqwyixswq
Also lots of support and wonderfulness from the community. 🤗
The other big one was when I was a wet-behind-the-ears Jr. dev.
"Sr." dev/lead screamed to cut corners to get it out fast (waterfall deadlines) even if it required undoing later.
Client rejected the fiasco release and our company lost serious face.
1/2
#DevDiscuss
Working on an inventory tracking app for "renting" out our small fleet of Mac's to QA. Decided we won't track VM's so I had the data layer pretend like they didn't exist. Since they didn't exist they must be new and we need to make a record for them every heartbeat! #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Fortunately, she moved to another project, a new lead took over and we worked *with* the customer, incrementally ensuring customer-value-delivery at a regular cadence (later called "agile"), salvaging the company's reputation with the client.
2/2
#DevDiscuss#HappyEndings
My big mistake was writing an infinite loop upon app initialization which:
- Took down our app in prod
- Got us ratelimited suspended by our host
- Triggered a host-wide status alert
I've had better days. #DevDiscuss
I once accidentally created an infinite loop in Frontend JS that kept on calling our api. Brought down an aws elastic beanstalk instance in less then 30 seconded that wouldn’t come back up on its own.
Lesson learnt: throttle the api calls
#DevDiscuss
„Quickly“ git rebasing to clean up commits before pushing.
Completely ruining everything.
git reflog to the rescue.
Once I learned about reflog I stopped fearing git
#DevDiscuss
Writing a script to resize a column on a columnar database and dropping the wrong column halfway through the process. Thank God the backup process was running properly lol #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
@ThePracticalDev 24 hours and 10k+ rows later I finally found my mistake!
I also accidentally hit a key too early once and ran an unconstrained DELETE against a production Jira server while fixing something. Had to go to backup and lost a morning's work.
#DevDiscuss
My favorite is when I typed "ctrl-f" to look for some code, but it didn't work the first time.
Tried again, made my change and pushed it up. (this is before automated tests)
Lttle did I know that an "f" in the code would bring the whole site to it's knees.
#DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
(the mistake to learn from here was, even if you're a Jr. dev, if you know the Sr. dev is bungling the project, speak up, talk to the next level of mgmt up, and CYA with a mile-wide paper-trail. Kept me squeaky-clean through the poo-storm.)
#DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Messed up with college Project in the day of defense. Somehow in the end, the code was fixed jut before the defense started. If we had screwed that Project Defense, we sure had to repeat the Semester. AGAIN. (THANK THAT DIDN'T HAPPEN) 🙄 #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Omitted a try/catch block in a module that ended up rendering a submit button inoperable. Happened right around the time a heavily funded marketing campaign for said page went live. Every subsequent discussion the $ for the amount of business we lost at least doubled. #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Not mine but a teammate of mine was testing package indexing from the filesystem of the new package server we were building out. Try did this by "copying" files from production to the drop folder. 1/2 #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
During my first production deployment at an old job, I somehow locked the database during deployment and had forgotten to take a backup prior to my first deployment. Ouch. My DBA saved me. After that, I started to automate the deployment. #yolo#devdiscuss
I work at a university. The weekend before the application deadline, I broke the part of the Admissions system that marked people's fees as paid. Cards got charged, but it didn't show up. One guy paid 23 times before giving up. Took 5 days for admins to notice 😮 #DevDiscuss
@ThePracticalDev after 800/2000ish packages imported we called it quits and cleaned out the server to prep for roll-out. An hour later (after people got back from lunch) we started getting reports of restore failures. Turns out the poor guy did a move and not a copy! #DevDiscuss
First couple of days on the job as a web dev. Was trying to move some files around on prod and somehow manage to individually replace every file in the docroot with a tar of that file. Prod went haywire, someone promptly took the wheel and managed to untar them. #DevDiscuss
I don't remember exactly ... I had to interact with Parse through their API and funneled that through my usual API consumption where 'I knew' certain errors would n.e.v.e.r happen and yet when they shut down, boom down went anyone who opened the app, #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
When I was doing my first WordPress site, I changed all the table prefixes but forgetting to change them in one code file. This mistake led to a white screen and lots of PHP errors. Luckily with some help from Skillcrush TA's I was able to find the issue and fix it #Devdiscuss
We once had a hardware failure on a server that was hosted by a different company right when AWS was becoming a thing. The server was down for 2 days, because the part was out of stock, and I got to quickly discover how to be an AWS admin. #DevDiscuss
Another good one was clearing a float on a watched class for auto loading new content.
Pegged the data base because every page was bringing back every piece of content we had.
Yep, CSS took it down. #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
I shut the wrong process down on an Apache server. A few minutes later, I got a flurry of IMs and e-mails that prod was down.
Thankfully we found out what happened and it was up again. I was in a cold sweat in the mean time, between time. #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
So I’ve had a few database related screw ups over my career:
1. Tried to add an index to a large table. Locked the table. App went down.
2. Tried to add a default value to a large table. Locked the table. App went down.
#DevDiscuss
My password as a normal user on a 10k+ member forum was hello1. I was given admin privs and changed to a secure password. Forum got reverted to a previous state and I was made an admin again.
But password was also reverted to hello1.
Guess what happened next. 🙈
#DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
3. Tried to delete a large number of rows. Locked the table. App went down.
4. Tried to delete a large number of rows in batches. Exceeded the IOPS rate on AWS instance. Database instance had no disk ops left. App went down.
#DevDiscuss
It really sucks when you screw up; however, screw ups are massive opportunities for learning. Your brain learns so much under stress like that.
Also, databases are hard when your app lives for years and deals with even moderate amounts of data #DevDiscuss
Database related screw ups really stress the need for a nice way to snapshot your db and test on a system that matches production. It’s hard to know what’s gonna happen if production has millions of rows and your test env has 1000. #DevDiscuss
One of our "top coders" decided to "optimize" a cleanup routine. He forgot to remove the DELETE statements (used in testing) from the code : it forked hundreds of DELETEs sql statements across 800 million PRODUCTION records in 200+ tables... #HisCareerEndedThatDay#DevDiscuss
I forgot to take https://t.co/0Fv3NXVfhm out of the example snippet I copied for http->s redirect in htaccess file. Realized what happened after 5 minutes of wondering why the hell the site was redirecting to https://t.co/0Fv3NXVfhm. Not horrible, but funny. #DevDiscuss
i once accidentally wrote an infinite loop, which included code that executed a VERY hefty sql script.
Our webmail product took a complete dump >.> All the webmails, down
#DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
On my 1st job I worked on a JS file over the weekend and replaced it on the server on Monday. Boss went crazy because he also worked on the same file during the wknd so his changes were gone. No backup.
I almost cried.
So weird we didn't use Git back then (8 yrs ago)
#DevDiscuss
I once pushed commits marked for staging server to the production server, on a Friday. CI/CD pipeline did its magic and so did the server. :)
#DevDiscuss
I was told to work on a backup of a CMS site to learn the ropes. I hit "publish" and the destination was still pointed to prod, but with different IDs. It didn't find matching IDs, so it made new ones. Every module on our live site was there in duplicate :x
#DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
I accidentally ran a report job via admin interface for our ecommerce sites in my first week on the job at my last place. It crashed the servers and OPS had to restart. Fixed by removing the ability to run that report from the admin interface. #DevDiscuss
I thought I was being clever inheriting a prod config for an app in a new environment, turns out I was overwriting the app's prod config thus reducing its availability whoops #DevDiscuss
On a 3 letter agency project we had issues in prod. Test server worked fine. Instead of letting us troubleshoot management panicked and made us drop a year's worth of work and made us use SSRS for the frontend which does 100k crud operations a day. #Devdiscuss
Two weeks in at my second company I managed to remove all images from our Solr docstore from which our frontend was built. Slightly important for a real estate service. Luckily I had a competent team who actually kept backups. Always keep backups. #devdiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
#DevDiscuss also this has an even better part 2: i had just started this job and it was thanksgiving week - my new boss was off and hunting in the woods. I didn't know this.
I called him, and he calmly whispered instructions to fix it
In a tree
With a deer like 50 ft away
When I was newer to JS I wrote a thing that did a hefty database call to pull numbers for conference registration. Like 3 second updates, with no caching, so users could see up-to-the-minute numbers. Left for the weekend, as did users, who left their browsers open... #devdiscuss
I wrote a service that got in a bad state and looped an error log around 1am and the support people were up all night over it. I got an ass chewing the next day. #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
Spent a day trying to figure out why my program wouldn’t stop.
Turns out I was passing 500 seconds and not milliseconds to a time function.
Huge facepalm moment 🤦🏽♂️
#DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
@ThePracticalDev A developer dynamically linked the C++ shared libs for our fintech website to a dev NFS share. No one noticed for weeks.
One day I unplugged the workstation under my desk to clean. Prod went down about 30 second later.
It was the dev NFS share. #DevDiscuss
#DevDiscuss "That time I messed up" I was bored on a backlog meeting. I started singing @onedirection's "What Makes You Beautiful." I was not muted on @Webex.
> "Ur insecure. Dunno what for. Ur turning heads when you walk through the DO-OO-OOR"
- "Mute yourself, por faVO-OO-OOR"
Oh, here’s another one. Had some issues with a server, so set up some logging that ran every few minutes. Forgot about it, came back a couple weeks later, server crashed.
Learned about logrotate after that. #DevDiscuss
I made db schema changes (django migrate command) on my dev machine, except that I've exported production credentials in my shell earlier on that I forgot about. Overwrote some db values in production as a result #DevDiscuss
I thought I was copying target binaries into a subdirectory prior to creating a firmware image but I flubbed the destination and was overwriting the build server's binaries. #DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
- I deleted the principal admin from the site
- I performed maintenance tasks in a different server and that server was not in maintenance mode
- I locked the largest table by running a migration
#DevDiscuss
Time for #DevDiscuss
Tonight's topic is "That time I messed up"
So let's here it!
Tell us about a time in your coding life that you messed up.
What happened? Was everything okay in the end?
I was creating a backup of our staging env Weblogic domain for some testing of a change. The file mount was out of space so I went to clean up some space but accidentally deleted the ACTUAL domain without a backup. We had to rebuild the server for ops #DevDiscuss
Oooh! I've got another one!
1/2
I had just started working mongoDB using pymongo. I quickly wrote a script to find all the documents which had a particular key set to "true". Co worker asked me to push that script to our private repo, he wanted to see my script.
#DevDiscuss
This didn't happen to me, but I remember reading this on HN. A new developer was setting up their development machine. Somehow they had a connection to the production database. What they thought was resetting the local database was actually dropping the production db. #DevDiscuss
I untarred a linux config tgz to my home folder on a prod box inside sudo bash. It unzipped to ~/etc/app/myfile.cnf. So I copied the file to the system folder. I decided to remove my local copy of the archive. I MEANT to type rm -rf ~/etc. Guess which char I skipped? #DevDiscuss
I accidentally overwrote the prod web server in a misunderstanding of an option in my IDE, which took down the whole website for 30 minutes to an hour, while netadmin restored from a backup. I don't open the prod web server in my IDE anymore. #DevDiscuss
2/2 So the script is pushed, and it's part of the same repo as the server and not included in the .gitignore file. Turns out, I had overwritten my flask server's complete https://t.co/mtQMSlzA1K and gunicorn was not able to run the Flask instance. :D
#DevDiscuss
Got approval from legal and the VP's today to open source a small utility I wrote! It'll be the second project on my company's github page. Hoping to have it up end of this week or next! #DevDiscuss
